You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-03 - 03:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun May 03, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27519217897,26sleep00-21swapper/019:06:190
276542165152,7sleep10-21swapper/119:08:051
471321630,1sleep20-21swapper/223:51:072
27567216096,34sleep30-21swapper/319:06:583
1362721490,5sleep334-21ksoftirqd/323:34:193
274592139105,24sleep20-21swapper/219:05:332
33092730,1sleep30-21swapper/322:13:453
180172730,1sleep10-21swapper/100:01:051
166872650,1sleep20-21swapper/221:59:552
150232640,2sleep30-21swapper/321:10:013
111182640,2sleep20-21swapper/222:43:452
95372620,7sleep122-21ksoftirqd/100:20:031
95372620,7sleep122-21ksoftirqd/100:20:031
63532610,1sleep10-21swapper/121:26:591
11022610,2sleep10-21swapper/121:23:261
137412600,10sleep122-21ksoftirqd/123:58:321
144532560,2sleep00-21swapper/023:35:000
178602550,2sleep20-21swapper/223:36:502
137352530,1sleep20-21swapper/221:32:432
279212520,1sleep00-21swapper/023:45:040
250642500,2sleep00-21swapper/023:42:220
151002500,2sleep10-21swapper/121:58:191
202072470,2sleep20-21swapper/200:03:202
2807699451,4cyclictest28770-21/usr/sbin/munin21:20:220
133782450,2chrt0-21swapper/223:10:022
2807699420,5cyclictest3809-21ssh23:02:170
2807699414,4cyclictest26864-21sh21:42:310
2807699410,4cyclictest22757-21ssh22:52:050
2807699400,5cyclictest17571-21ssh22:48:350
2807699400,4cyclictest1766-21ssh00:38:330
268182400,1chrt0-21swapper/100:32:481
2807899398,10cyclictest18684-21sh21:12:012
28078993910,19cyclictest26783-21sh00:32:462
28078993810,18cyclictest10780-21open_inodes00:20:222
28078993810,18cyclictest10780-21open_inodes00:20:212
2807899379,14cyclictest0-21swapper/222:04:562
2807899379,14cyclictest0-21swapper/222:04:552
2807899379,14cyclictest0-21swapper/221:39:282
2807899378,12cyclictest6381-21taskset23:28:432
28078993713,9cyclictest14926-21diskmemload23:00:172
28078993711,9cyclictest15416-21chrt22:46:222
2807699372,3cyclictest0-21swapper/020:30:130
2807699372,3cyclictest0-21swapper/020:30:120
2807699371,7cyclictest20452-21sh22:01:580
2807699371,7cyclictest20452-21sh22:01:580
2807699370,6cyclictest22221-21ssh21:15:130
28078993611,9cyclictest27277-21wc22:55:222
28078993611,9cyclictest22059-21ssh21:15:092
28076993610,23cyclictest1132-21readlink19:20:220
2807899359,9cyclictest0-21swapper/223:46:022
2807899359,15cyclictest0-21swapper/223:56:092
2807899357,12cyclictest21720-21ssh22:51:012
28078993511,9cyclictest16307-21ssh22:23:242
28078993511,6cyclictest19938-21unixbench_multi22:25:242
28078993510,9cyclictest0-21swapper/200:28:432
28078993510,10cyclictest14926-21diskmemload22:15:542
28076993511,21cyclictest28279-21cat20:25:100
28076993511,21cyclictest23657-21sed20:15:010
28076993510,22cyclictest11781-21which21:00:230
2807699350,5cyclictest28261-21ls23:20:210
2807699350,5cyclictest16372-21ssh21:35:090
157592350,2sleep334-21ksoftirqd/322:46:433
2807899349,8cyclictest14926-21diskmemload22:32:462
2807899349,15cyclictest25889-21sh00:07:212
2807899349,14cyclictest0-21swapper/223:23:042
2807899349,11cyclictest9394-21ssh23:30:232
2807899348,8cyclictest28201-21ssh21:43:542
2807699341,6cyclictest149262diskmemload00:10:360
28076993412,20cyclictest30344-21sh22:34:050
28076993410,21cyclictest21336-21seq20:09:260
28076993410,21cyclictest13652-21cat23:10:130
2807899339,14cyclictest0-21swapper/223:43:412
2807899338,15cyclictest0-21swapper/220:50:252
2807899337,16cyclictest0-21swapper/200:38:462
2807899337,10cyclictest1339-21sh00:13:422
28078993310,20cyclictest8795-21expr20:55:122
2807699339,21cyclictest29263-21mailstats19:10:220
2807699339,21cyclictest25718-21cat20:20:000
2807699335,25cyclictest6398-21sh23:28:460
2807699335,25cyclictest0-21swapper/023:55:140
2807699332,26cyclictest22802-21ssh00:30:130
2807699331,2cyclictest0-21swapper/023:15:120
28076993312,18cyclictest5871-21sh23:52:180
28076993311,20cyclictest29995-21expr21:45:130
28076993310,20cyclictest11101-21expr19:45:140
2807699330,8cyclictest25919-21/usr/sbin/munin20:20:100
2807699330,7cyclictest28481-21ssh22:56:060
2807699330,4cyclictest7624-21awk19:35:210
2807699330,4cyclictest14926-21diskmemload22:18:500
10812330,3sleep3331rcuc/323:25:003
2807899329,14cyclictest0-21swapper/221:29:252
2807899328,9cyclictest6984-21ssh21:51:492
2807899328,8cyclictest1991-21ssh21:24:212
2807899328,10cyclictest22405-21ssh23:15:542
2807699329,20cyclictest16203-21apt-get00:25:110
2807699329,20cyclictest11389-21unixbench_singl21:30:240
2807699328,21cyclictest3316-21expr00:15:130
2807699322,9cyclictest30746-21apt-get22:10:110
2807699321,5cyclictest8168-21mailstats23:05:220
2807699321,12cyclictest18316-21apt-get22:25:120
28076993211,18cyclictest14926-21diskmemload21:50:170
28076993211,18cyclictest13518-21fschecks_time19:50:160
2807699320,5cyclictest5585-21timerwakeupswit20:45:250
2807699320,4cyclictest636-21runrttasks21:11:330
2807699320,4cyclictest2478-21df_inode19:25:130
2807699320,4cyclictest17202-21ssh22:24:170
2807899319,7cyclictest24061-21ssh22:05:112
2807899317,8cyclictest1823-21sh21:48:172
2807699316,6cyclictest8827-21cpu19:40:130
2807699316,22cyclictest32257-21sed22:35:160
2807699315,23cyclictest12117-21ssh21:55:240
2807699312,3cyclictest0-21swapper/019:15:120
2807699310,7cyclictest27982-21ssh00:09:290
2807699310,4cyclictest4938-21rm21:25:340
264092310,10sleep122-21ksoftirqd/120:20:141
18172310,2chrt0-21swapper/100:14:101
48932300,2chrt0-21swapper/123:51:181
2807899309,6cyclictest11198-21ssh23:08:012
2807899307,15cyclictest0-21swapper/222:10:162
2807699307,7cyclictest4261-21/usr/sbin/munin19:30:190
2807699307,4cyclictest18222-21uname20:00:200
2807699307,20cyclictest26407-21sh22:06:010
2807699302,11cyclictest15639-21apt-get00:00:120
2807699301,9cyclictest14867-21tr23:35:140
2807699301,7cyclictest6017-21ssh22:40:050
2807899298,8cyclictest4134-21ntpq00:15:212
2807899297,6cyclictest9235-21perl19:40:152
2807899296,11cyclictest4498-21ssh22:39:002
2807699292,11cyclictest32372-21apt-get20:35:140
2807699291,9cyclictest636-21runrttasks00:23:380
2807699291,9cyclictest636-21runrttasks00:23:370
2807699291,8cyclictest9682-21/usr/sbin/munin20:55:220
2807699291,7cyclictest7468-21/usr/sbin/munin20:50:210
2807699291,7cyclictest2111-21apt-get20:40:100
2807999280,13cyclictest268172ssh00:32:493
2807899286,7cyclictest0-21swapper/220:32:112
2807899286,7cyclictest0-21swapper/220:32:112
2807899285,8cyclictest0-21swapper/220:05:112
28078992813,12cyclictest17445-21/usr/sbin/munin20:00:122
2807699281,7cyclictest13192-21/usr/sbin/munin21:05:130
2807899277,5cyclictest6182-21apt-get19:35:022
2807899275,8cyclictest0-21swapper/220:25:002
28078992716,8cyclictest1881-21perf20:40:002
28078992714,10cyclictest29606-21sed19:10:252
2807699271,9cyclictest15548-21diskstats19:55:140
28078992615,8cyclictest5236-21tr20:45:222
28078992614,9cyclictest14436-21grep21:05:252
28078992613,10cyclictest16058-21grep19:55:192
2807799261,1cyclictest0-21swapper/119:12:321
2807999252,16cyclictest22270-21ssh00:05:123
2807999252,16cyclictest17245-21diskmemload22:00:123
2807999252,16cyclictest17245-21diskmemload22:00:123
28078992514,9cyclictest14961-21munin-run19:55:002
28078992514,8cyclictest4729-21sort19:30:162
28078992514,8cyclictest23952-21/usr/sbin/munin20:15:122
28078992514,8cyclictest12166-21sed21:00:262
28078992512,10cyclictest29442-21cat20:25:212
285082240,1sleep30-21swapper/319:10:143
2807999243,16cyclictest23940-21sh22:05:033
28078992415,6cyclictest30801-21gltestperf19:15:162
28078992414,7cyclictest32554-21cpuspeed_turbos19:20:132
28078992413,8cyclictest12801-21cat19:49:552
28078992410,11cyclictest22998-21cat20:10:232
28078992312,9cyclictest2458-21cstates19:25:132
28078992312,8cyclictest2292-21expr20:40:122
2807799238,4cyclictest14632ssh22:11:521
28077992314,6cyclictest23158-21ntpq21:15:201
2807999221,4cyclictest0-21swapper/320:50:253
28079992211,6cyclictest0-21swapper/321:30:103
28077992213,6cyclictest278-21systemd-journal19:20:001
28077992210,7cyclictest10605-21missed_timers21:30:181
284012210,1sleep30-21swapper/322:32:063
2807999213,14cyclictest7726-21grep22:40:233
2807999211,5cyclictest3997-21cron20:45:003
28079992113,6cyclictest699-21sed19:20:173
28079992112,7cyclictest8592-21cat20:55:133
2807999211,16cyclictest30554-21df20:30:153
2807999211,16cyclictest30554-21df20:30:143
2807799212,16cyclictest560-21dbus-daemon22:25:011
2807799211,4cyclictest0-21swapper/122:54:441
28077992112,7cyclictest8851-21df20:55:131
28077992112,7cyclictest24577-21cron22:30:001
28077992112,7cyclictest23586-21fw_packets21:40:151
28077992112,7cyclictest19847-21diskstats20:05:141
28077992112,6cyclictest5108-21seq22:39:381
2807999209,3cyclictest99942rm23:06:493
2807999203,12cyclictest15198-21munin-run00:00:013
2807999202,12cyclictest15341-21apt-get21:10:123
2807999202,12cyclictest1375-21apt-get23:25:123
28079992014,3cyclictest10269-21cut00:20:173
28079992014,3cyclictest10269-21cut00:20:163
28079992013,5cyclictest29306-21cron21:45:003
28079992013,4cyclictest30321-21sed21:45:153
28079992012,6cyclictest3167-21wc21:25:133
28079992012,6cyclictest2311-21cat19:25:123
28079992012,6cyclictest10924-21diskstats21:55:143
2807999200,6cyclictest19035-21ssh21:36:203
2807799208,3cyclictest141rcu_preempt22:15:191
2807799208,10cyclictest30194-21ssh22:09:531
2807799202,12cyclictest640-21snmpd20:45:131
2807799201,9cyclictest29428-21ssh00:35:111
28077992013,5cyclictest22-21ksoftirqd/121:50:011
28077992012,6cyclictest31077-21seq22:58:481
28077992012,6cyclictest29966-21cron20:30:001
28077992012,6cyclictest17330-21date20:00:021
28077992011,7cyclictest17176-21expr00:25:201
28077992011,6cyclictest22-21ksoftirqd/122:30:201
2807999198,7cyclictest53050irq/142-enp3s0-22:15:133
2807999193,9cyclictest3074-21ssh00:15:123
2807999193,10cyclictest2306-21apt-get23:50:123
2807999192,14cyclictest27645-21latency_hist23:45:003
2807999192,12cyclictest75-21kswapd021:50:123
28079991914,3cyclictest13304-21grep22:20:253
28079991913,4cyclictest9870-21rt-features19:40:213
28079991913,4cyclictest28770-21/usr/sbin/munin21:20:173
28079991912,5cyclictest13623-21cut19:50:153
28079991912,4cyclictest20672-21grep23:15:143
28079991912,4cyclictest12601-21apt-key21:05:003
28079991911,6cyclictest23041-21grep20:10:243
28079991911,6cyclictest14057-21tune2fs23:10:153
28079991911,6cyclictest11102-21seq19:45:143
28079991910,6cyclictest6971-21diskstats19:35:153
28079991910,6cyclictest20460-21awk20:05:213
2807799198,3cyclictest141rcu_preempt23:45:201
28077991913,4cyclictest14484-21uptime22:45:251
28077991913,3cyclictest9775-21sh22:42:231
28077991912,5cyclictest5507-21ls21:50:251
28077991912,5cyclictest421-21cat23:00:121
28077991912,5cyclictest22573-21open_files20:10:201
28077991912,5cyclictest22-21ksoftirqd/123:45:011
28077991912,4cyclictest27896-21sh00:09:241
28077991911,6cyclictest9616-21cat19:40:181
28077991911,6cyclictest8176-21seq19:35:441
28077991911,6cyclictest6052-21seq19:33:391
28077991911,6cyclictest4417-21awk00:15:251
28077991911,6cyclictest31756-21seq20:30:321
28077991911,6cyclictest31756-21seq20:30:321
28077991911,6cyclictest2823-21ls23:25:221
28077991911,6cyclictest26487-21cron23:20:001
28077991911,6cyclictest1673-21seq20:38:031
28077991911,6cyclictest10827-21expr21:00:141
2807799191,15cyclictest2547-21perl20:40:131
28077991911,5cyclictest14926-21diskmemload23:10:131
28077991910,7cyclictest307-21seq19:20:141
28077991910,7cyclictest20842-21seq21:14:101
35912180,2sleep30-21swapper/322:38:043
2807999189,7cyclictest15723-21netstat23:35:203
2807999189,7cyclictest14926-21diskmemload22:55:553
2807999182,11cyclictest23782-21apt-get20:15:113
2807999181,8cyclictest16714-21ntp_states00:00:213
2807999181,7cyclictest20714-21sensors22:50:223
2807999181,6cyclictest1077-21/usr/sbin/munin23:00:163
28079991813,3cyclictest28533-21fschecks_time23:45:153
28079991812,4cyclictest18950irq/143-ahci[0019:35:003
28079991812,4cyclictest14926-21diskmemload00:11:343
28079991811,4cyclictest23659-21sh21:15:263
28079991810,6cyclictest640-21snmpd00:29:053
2807799187,3cyclictest141rcu_preempt23:05:101
2807799182,11cyclictest6389-21apt-get20:50:121
2807799181,8cyclictest11497-21ssh23:32:121
28077991811,5cyclictest20992-21seq22:02:311
28077991811,5cyclictest20992-21seq22:02:301
28077991811,5cyclictest14914-21ls23:35:131
28077991811,5cyclictest11403-21fw_packets19:45:181
28077991810,6cyclictest3442-21cat19:25:231
28077991810,6cyclictest14530-21sed21:05:251
28077991810,5cyclictest52850irq/140-enp3s0-23:20:031
2807999172,3cyclictest0-21swapper/322:25:103
28079991712,3cyclictest636-21runrttasks20:25:113
28079991712,3cyclictest25919-21/usr/sbin/munin20:20:093
2807999171,14cyclictest29538-21cpuspeed_turbos00:35:123
28079991710,5cyclictest4390-21expr20:45:153
28079991710,5cyclictest30831-21cat19:15:173
28079991710,4cyclictest15197-21acpi19:55:113
2807999170,14cyclictest17767-21tune2fs20:00:143
2807799177,7cyclictest13159-21grep19:50:121
2807799171,11cyclictest16446-21apt-get21:35:131
28077991710,5cyclictest19201-21seq20:04:281
28077991611,3cyclictest25667-21latency_hist20:20:011
28079991510,3cyclictest1221-21timerandwakeup20:35:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional