You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 18:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Apr 23, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2225189998466,16cyclictest37-21ksoftirqd/309:55:023
2225187997447,24cyclictest291rcuc/210:10:012
2225178997221,35cyclictest12-21ksoftirqd/012:40:010
2225189996826,37cyclictest2278126-21sh10:58:073
2225189996625,37cyclictest2291988-21latency_hist11:30:013
2225189996525,35cyclictest37-21ksoftirqd/308:10:013
2225189996523,36cyclictest309-21in:imuxsock07:50:293
2225189996521,38cyclictest2231345-21diskmemload11:50:003
2225178996522,6cyclictest131rcu_preempt09:10:020
2225189996423,36cyclictest2273914-21sh10:48:193
2225189996323,35cyclictest2309625-21sh12:10:403
2225189996322,36cyclictest2320563-21sh12:36:053
222518999624,17cyclictest2244272-21sendmail-msp09:40:003
2225189996221,5cyclictest2318618-21sh12:31:413
2225189996221,11cyclictest37-21ksoftirqd/309:13:043
2225189996218,11cyclictest2300113-1kworker/u9:211:52:173
222517899622,40cyclictest131rcu_preempt10:00:000
222517899622,27cyclictest131rcu_preempt11:11:470
222518999612,17cyclictest2316468-21ssh12:26:383
2225187996113,32cyclictest0-21swapper/210:10:032
2225189996022,32cyclictest2314528-21sh12:22:093
2225189996018,10cyclictest0-21swapper/310:50:333
2225187996028,8cyclictest2250773-21kworker/2:211:30:532
2225187996028,8cyclictest2250773-21kworker/2:211:30:522
2225187996019,10cyclictest30-21ksoftirqd/211:37:212
222518399604,39cyclictest2239974-21ssh09:30:011
2225183996033,20cyclictest275-21dbus-daemon11:45:001
222517899603,27cyclictest131rcu_preempt11:55:000
2225189995939,15cyclictest3467602-21systemd-journal12:19:593
2225189995924,31cyclictest2289832-21latency_hist11:25:033
2225189995921,12cyclictest37-21ksoftirqd/309:33:193
2225189995918,4cyclictest2270251-21latency_hist10:40:023
2225178995940,4cyclictest16850irq/134-800000011:10:020
222518999582,18cyclictest2260431-21ssh10:16:583
2225183995840,12cyclictest3467602-21systemd-journal07:55:001
2225178995838,5cyclictest12-21ksoftirqd/012:30:490
2225178995823,3cyclictest131rcu_preempt12:19:260
2225178995821,32cyclictest2240092-21latency_hist09:30:020
2225178995821,32cyclictest12-21ksoftirqd/010:42:370
2225189995722,31cyclictest2231349-21latency_hist09:10:023
2225189995712,10cyclictest37-21ksoftirqd/308:30:023
2225178995727,14cyclictest2288018-21ssh11:20:380
2225178995720,17cyclictest16850irq/134-800000010:13:000
2225178995719,6cyclictest16850irq/134-800000010:03:500
2225178995719,18cyclictest16850irq/134-800000009:54:010
2225178995719,18cyclictest16850irq/134-800000009:54:000
2225178995719,17cyclictest16850irq/134-800000011:16:240
2225178995718,7cyclictest16850irq/134-800000012:07:230
2225189995624,29cyclictest37-21ksoftirqd/307:15:013
2225189995624,28cyclictest37-21ksoftirqd/310:01:143
2225189995622,29cyclictest2262288-21sh10:21:123
2225189995620,9cyclictest37-21ksoftirqd/310:09:413
2225183995633,15cyclictest275-21dbus-daemon08:20:011
222517899564,16cyclictest2230808-21modprobe09:00:000
2225178995628,13cyclictest2310361-21ssh12:12:360
2225178995626,15cyclictest3467602-21systemd-journal10:57:290
222517899562,3cyclictest131rcu_preempt09:35:020
2225178995622,31cyclictest12-21ksoftirqd/011:56:330
2225178995619,17cyclictest16850irq/134-800000009:21:500
2225178995618,17cyclictest16850irq/134-800000012:25:190
2225178995618,17cyclictest16850irq/134-800000009:11:080
2225189995524,27cyclictest37-21ksoftirqd/309:53:513
2225189995524,27cyclictest37-21ksoftirqd/309:53:513
2225189995517,10cyclictest3467602-21systemd-journal11:11:023
2225189995514,9cyclictest2307193-21cut12:05:023
2225178995536,14cyclictest1-21systemd09:40:010
2225178995520,20cyclictest12-21ksoftirqd/011:40:000
2225178995519,17cyclictest16850irq/134-800000011:31:570
2225178995519,17cyclictest16850irq/134-800000011:31:570
2225178995519,16cyclictest16850irq/134-800000011:04:220
2225178995517,6cyclictest16850irq/134-800000011:29:350
2225178995517,6cyclictest16850irq/134-800000010:38:240
2225178995517,6cyclictest16850irq/134-800000009:15:460
2225178995517,19cyclictest16850irq/134-800000010:52:090
2225178995517,18cyclictest16850irq/134-800000011:45:320
2225178995517,17cyclictest16850irq/134-800000012:04:260
2225178995516,6cyclictest16850irq/134-800000010:31:350
2225178995516,35cyclictest2315643-21latency_hist12:25:010
222518999542,17cyclictest2294280-21ssh11:35:083
222518999542,16cyclictest2231095-21cut09:05:013
222518999542,16cyclictest2231095-21cut09:05:003
2225189995416,10cyclictest37-21ksoftirqd/309:18:133
2225189995414,5cyclictest0-21swapper/311:42:593
2225189995413,9cyclictest2230038-21sshd08:44:253
2225178995417,5cyclictest16850irq/134-800000010:09:030
2225178995417,17cyclictest16850irq/134-800000010:29:270
2225178995416,18cyclictest16850irq/134-800000009:44:100
2225189995324,26cyclictest2226416-21sh07:35:003
2225187995313,8cyclictest131rcu_preempt11:23:002
2225183995342,8cyclictest23-21ksoftirqd/112:23:361
2225178995317,17cyclictest16850irq/134-800000010:23:370
2225178995316,6cyclictest16850irq/134-800000010:16:420
2225178995316,17cyclictest16850irq/134-800000011:42:420
2225178995315,18cyclictest16850irq/134-800000010:45:350
2225189995222,9cyclictest37-21ksoftirqd/311:19:093
222518999521,17cyclictest2237829-21cut09:25:003
2225187995244,5cyclictest2250773-21kworker/2:211:00:212
222518399521,20cyclictest0-21swapper/111:29:211
2225183995211,6cyclictest131rcu_preempt10:20:011
2225178995248,2cyclictest0-21swapper/007:15:010
2225189995130,17cyclictest2230815-21cron09:00:003
2225189995119,9cyclictest37-21ksoftirqd/311:02:083
2225189995119,5cyclictest2267468-21rm10:33:263
2225189995116,6cyclictest37-21ksoftirqd/311:56:153
2225187995144,4cyclictest2230274-21kworker/2:111:05:332
2225187995143,5cyclictest2291891-21kworker/2:011:43:142
2225187995143,5cyclictest2230274-21kworker/2:110:49:152
2225187995143,5cyclictest2230274-21kworker/2:109:55:252
2225187995142,6cyclictest2300550-21kworker/2:111:57:342
2225187995142,6cyclictest2230274-21kworker/2:109:33:012
2225187995142,5cyclictest2250773-21kworker/2:211:46:252
2225183995129,18cyclictest23-21ksoftirqd/107:20:031
222518399511,18cyclictest0-21swapper/109:12:401
2225189995021,8cyclictest37-21ksoftirqd/307:29:313
222518999502,17cyclictest2264925-21ssh10:27:263
222518999502,17cyclictest2248599-21munin-run09:50:003
222518999501,16cyclictest2227902-21cat08:00:033
2225187995042,5cyclictest2230274-21kworker/2:111:16:272
2225187995042,5cyclictest2230274-21kworker/2:109:16:512
2225187995041,5cyclictest2250773-21kworker/2:210:31:512
2225187995041,5cyclictest2239967-21kworker/2:309:45:072
222518799502,3cyclictest131rcu_preempt10:29:292
222518399507,10cyclictest2275101-21kworker/u8:111:07:091
222518399505,41cyclictest2235739-21ssh09:20:021
2225183995016,6cyclictest131rcu_preempt09:00:011
2225189994915,7cyclictest37-21ksoftirqd/310:14:343
222518999491,18cyclictest2283159-21ssh11:09:513
2225187994941,5cyclictest2250773-21kworker/2:212:04:342
2225187994941,5cyclictest2230274-21kworker/2:110:19:302
2225187994941,5cyclictest2230274-21kworker/2:110:00:262
2225187994941,5cyclictest2228004-21kworker/2:008:40:012
2225187994940,5cyclictest2250773-21kworker/2:210:59:462
222518399497,6cyclictest131rcu_preempt09:22:411
2225183994937,8cyclictest23-21ksoftirqd/108:00:011
2225183994923,9cyclictest23-21ksoftirqd/109:36:431
2225183994913,7cyclictest23-21ksoftirqd/112:40:021
222518399491,15cyclictest0-21swapper/109:30:311
2225189994818,5cyclictest2244757-21sh09:40:403
222518799484,3cyclictest131rcu_preempt12:20:152
2225187994841,4cyclictest2250773-21kworker/2:210:23:022
2225187994841,4cyclictest2230274-21kworker/2:110:52:232
2225187994840,5cyclictest2230571-21kworker/2:209:28:302
2225187994840,5cyclictest2230274-21kworker/2:109:23:442
2225187994838,5cyclictest2315628-21kworker/2:012:35:022
2225183994840,6cyclictest23-21ksoftirqd/108:10:251
222518399481,16cyclictest0-21swapper/110:06:181
2225183994810,6cyclictest131rcu_preempt12:30:001
222517899482,41cyclictest2248789-21grep09:50:020
2225187994736,8cyclictest2239967-21kworker/2:309:44:022
2225187994736,6cyclictest2250773-21kworker/2:212:10:302
2225187994730,13cyclictest30-21ksoftirqd/209:55:002
2225187994730,13cyclictest30-21ksoftirqd/209:55:002
222518399479,6cyclictest131rcu_preempt09:45:021
2225183994733,8cyclictest309-21in:imuxsock11:35:011
2225183994733,8cyclictest309-21in:imuxsock11:35:011
2225183994725,14cyclictest275-21dbus-daemon07:30:001
2225183994720,6cyclictest131rcu_preempt11:40:021
2225183994718,5cyclictest131rcu_preempt10:40:011
2225183994715,9cyclictest131rcu_preempt12:05:001
2225183994714,6cyclictest131rcu_preempt09:52:041
2225183994714,6cyclictest131rcu_preempt09:52:031
2225183994711,8cyclictest23-21ksoftirqd/110:30:091
222518399471,15cyclictest0-21swapper/112:34:401
222518399471,13cyclictest0-21swapper/110:50:401
2225178994732,12cyclictest16950irq/224-800000008:10:290
222518999463,13cyclictest2226712-21wc07:40:013
222518999463,13cyclictest2226712-21wc07:40:013
222518999462,16cyclictest2292539-21ssh11:31:153
222518999462,16cyclictest2292539-21ssh11:31:143
2225189994616,4cyclictest2231345-21diskmemload10:42:523
2225187994638,5cyclictest2250773-21kworker/2:211:29:252
2225187994631,11cyclictest3467602-21systemd-journal09:05:012
2225187994631,11cyclictest3467602-21systemd-journal09:05:002
222518399467,8cyclictest23-21ksoftirqd/112:15:011
222518399466,8cyclictest23-21ksoftirqd/108:35:001
2225183994635,8cyclictest23-21ksoftirqd/111:20:011
2225183994613,9cyclictest23-21ksoftirqd/111:24:591
222518399461,18cyclictest0-21swapper/112:19:591
222518399461,15cyclictest0-21swapper/111:58:361
222517899463,4cyclictest131rcu_preempt08:00:030
2225189994534,7cyclictest2227214-21latency_hist07:50:013
2225189994519,9cyclictest37-21ksoftirqd/308:50:023
2225187994537,5cyclictest2250773-21kworker/2:211:54:172
2225187994537,5cyclictest2230274-21kworker/2:110:42:412
2225187994537,5cyclictest2213512-21kworker/2:308:33:272
2225187994536,6cyclictest2230274-21kworker/2:109:39:132
2225187994536,6cyclictest2230274-21kworker/2:109:13:432
2225187994529,12cyclictest2230821-21systemctl09:00:002
222518399458,6cyclictest131rcu_preempt11:00:021
222518399453,4cyclictest131rcu_preempt10:25:021
2225183994532,8cyclictest3467602-21systemd-journal08:26:251
2225183994511,5cyclictest131rcu_preempt09:59:231
222517899452,9cyclictest284-21systemd-logind08:40:010
2225178994525,16cyclictest3467602-21systemd-journal07:35:000
2225189994427,13cyclictest275-21dbus-daemon08:35:003
2225189994412,10cyclictest37-21ksoftirqd/308:17:003
2225187994436,5cyclictest2300550-21kworker/2:112:25:512
2225187994436,5cyclictest2230274-21kworker/2:111:13:252
2225187994436,5cyclictest2228004-21kworker/2:008:45:012
2225187994435,5cyclictest2300550-21kworker/2:112:07:332
2225187994435,5cyclictest2230274-21kworker/2:110:38:372
2225187994428,12cyclictest3467602-21systemd-journal07:15:002
222518399442,5cyclictest131rcu_preempt10:01:571
222518399441,18cyclictest0-21swapper/111:48:191
2225178994427,11cyclictest3467602-21systemd-journal08:50:000
2225187994335,5cyclictest2300550-21kworker/2:112:16:292
222518799432,3cyclictest131rcu_preempt12:40:012
222518399438,6cyclictest131rcu_preempt10:30:001
222518399438,5cyclictest131rcu_preempt11:02:231
2225183994332,7cyclictest2229021-21munin-plugin-st08:25:001
2225183994310,7cyclictest131rcu_preempt07:40:001
2225183994310,7cyclictest131rcu_preempt07:40:001
2225178994325,14cyclictest2226908-21sshd07:40:260
222517899432,36cyclictest2228752-21munin-plugin-st08:20:000
2225189994225,14cyclictest37-21ksoftirqd/307:45:003
222518399425,9cyclictest23-21ksoftirqd/110:14:241
222518399423,5cyclictest131rcu_preempt08:05:001
2225183994232,7cyclictest23-21ksoftirqd/111:54:231
2225183994232,6cyclictest23-21ksoftirqd/107:25:011
2225183994232,6cyclictest23-21ksoftirqd/107:25:011
222518399422,4cyclictest131rcu_preempt08:44:251
2225183994224,13cyclictest1-21systemd07:15:001
2225183994212,5cyclictest131rcu_preempt10:48:261
2225183994211,5cyclictest131rcu_preempt09:45:021
2225178994234,5cyclictest12-21ksoftirqd/007:55:010
2225178994226,12cyclictest2226646-21sshd07:39:260
2225178994226,12cyclictest2226646-21sshd07:39:260
2225189994132,7cyclictest37-21ksoftirqd/312:07:063
222518799411,3cyclictest131rcu_preempt08:10:012
222518399416,5cyclictest131rcu_preempt07:45:011
2225183994123,13cyclictest275-21dbus-daemon08:55:011
222518399412,10cyclictest23-21ksoftirqd/108:40:011
2225183994119,15cyclictest3467602-21systemd-journal07:50:011
2225183994112,5cyclictest131rcu_preempt11:11:251
2225183994110,5cyclictest131rcu_preempt12:05:181
222517899412,8cyclictest311-21rs:main0
2225178994127,10cyclictest2229243-21sshd08:25:410
222518399409,6cyclictest131rcu_preempt08:50:001
2225183994028,9cyclictest23-21ksoftirqd/109:10:021
2225183994026,7cyclictest309-21in:imuxsock09:05:011
2225183994026,7cyclictest309-21in:imuxsock09:05:011
222518399401,5cyclictest0-21swapper/110:43:551
2225178994026,10cyclictest3467602-21systemd-journal08:40:020
2225178994021,15cyclictest2227168-21sshd07:46:250
222517899401,3cyclictest131rcu_preempt08:05:010
222518999391,16cyclictest2240090-21ssh09:30:033
2225187993930,5cyclictest309-21in:imuxsock08:50:012
2225183993928,7cyclictest23-21ksoftirqd/107:35:021
222518999386,6cyclictest131rcu_preempt08:40:013
222518799389,14cyclictest2227684-21kworker/u8:108:30:022
2225187993827,8cyclictest3467602-21systemd-journal07:29:022
2225187993827,8cyclictest2228629-21cut08:15:022
2225187993825,11cyclictest30-21ksoftirqd/208:25:022
2225183993810,5cyclictest131rcu_preempt08:10:011
2225178993830,5cyclictest19350irq/34-mmc108:33:370
2225178993828,6cyclictest19350irq/34-mmc109:02:060
2225178993828,6cyclictest19350irq/34-mmc109:02:060
2225189993722,11cyclictest2225648-21sendmail07:20:003
2225187993727,6cyclictest2230274-21kworker/2:108:55:012
2225187993726,8cyclictest2225601-21sshd07:17:252
222518799372,4cyclictest2227695-21modprobe08:00:002
222518799367,6cyclictest131rcu_preempt09:10:022
2225187993628,5cyclictest2212565-21kworker/2:207:38:252
2225187993628,5cyclictest2212565-21kworker/2:207:38:252
2225187993626,7cyclictest30-21ksoftirqd/208:05:002
2225187993625,8cyclictest30-21ksoftirqd/208:20:012
222517899365,28cyclictest12-21ksoftirqd/008:06:470
2225178993628,5cyclictest309-21in:imuxsock08:50:020
222517899362,5cyclictest2226172-21munin-run07:30:000
222518999354,4cyclictest131rcu_preempt08:55:013
2225189993526,6cyclictest309-21in:imuxsock08:15:023
2225189993516,15cyclictest275-21dbus-daemon08:25:013
222518799352,5cyclictest275-21dbus-daemon07:45:002
222518799352,10cyclictest3467602-21systemd-journal07:10:032
222518799352,10cyclictest3467602-21systemd-journal07:10:022
2225187993516,15cyclictest275-21dbus-daemon07:35:012
2225183993523,9cyclictest2225309-21sed07:10:031
2225183993523,9cyclictest2225309-21sed07:10:021
2225178993529,4cyclictest0-21swapper/008:23:270
222517899352,3cyclictest131rcu_preempt07:25:010
222517899352,3cyclictest131rcu_preempt07:25:010
222517899352,28cyclictest2225256-21wc07:10:020
222517899352,28cyclictest2225256-21wc07:10:020
2225189993419,11cyclictest275-21dbus-daemon08:05:003
2225187993425,5cyclictest309-21in:imuxsock07:55:022
2225187993423,9cyclictest30-21ksoftirqd/207:23:252
2225187993423,9cyclictest30-21ksoftirqd/207:23:252
2225187993323,8cyclictest30-21ksoftirqd/207:50:002
2225189993124,5cyclictest2226001-21latency_hist07:25:023
2225189993124,5cyclictest2226001-21latency_hist07:25:023
2225189992817,7cyclictest2225293-21cat07:10:023
2225189992817,7cyclictest2225293-21cat07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional