You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-13 - 13:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Aug 13, 2026 00:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
233782997626,13cyclictest1-21systemd00:40:012
233782997626,13cyclictest1-21systemd00:40:012
233782997426,5cyclictest290236-21latency_hist22:55:022
233782996921,13cyclictest3467602-21systemd-journal21:57:322
233773996924,30cyclictest12-21ksoftirqd/023:58:360
233782996828,4cyclictest300936-21sh23:17:282
233787996724,37cyclictest302016-21sh23:19:513
233787996720,10cyclictest1-21systemd21:40:023
233782996737,26cyclictest1-21systemd21:35:002
233782996730,6cyclictest326039-21latency_hist00:10:032
233787996629,33cyclictest249817-21sh21:30:163
233787996521,40cyclictest270965-21latency_hist22:15:013
233773996521,10cyclictest131rcu_preempt00:21:270
233787996424,36cyclictest275-21dbus-daemon20:15:003
233782996424,8cyclictest270620-21sshd22:14:282
233782996322,10cyclictest30-21ksoftirqd/219:25:412
23378799622,18cyclictest321237-21systemd00:00:003
23377899622,8cyclictest23-21ksoftirqd/121:20:011
233787996127,30cyclictest1-21systemd23:30:013
233787996124,33cyclictest340515-21latency_hist00:40:023
233787996124,33cyclictest340515-21latency_hist00:40:013
233787996123,8cyclictest37-21ksoftirqd/321:23:413
233787996123,5cyclictest239947-21diskmemload22:49:233
233787996122,11cyclictest37-21ksoftirqd/322:56:363
233782996128,5cyclictest284453-21sh22:43:002
233782996128,5cyclictest284453-21sh22:43:002
233782996123,7cyclictest30-21ksoftirqd/223:55:042
23377399611,22cyclictest16850irq/134-800000022:40:010
233787996024,4cyclictest338048-21sh00:34:553
233787996021,4cyclictest275713-21latency_hist22:25:023
233787996021,4cyclictest275713-21latency_hist22:25:023
233787996020,5cyclictest239947-21diskmemload00:23:093
233787996018,6cyclictest37-21ksoftirqd/300:09:233
233782996023,9cyclictest239012-21sort20:50:022
23377399602,50cyclictest242445-21latency_hist21:15:020
233782995925,10cyclictest298519-21sshd23:12:292
233773995936,18cyclictest275-21dbus-daemon23:05:020
233773995926,15cyclictest305591-21ssh23:27:030
233787995826,28cyclictest281902-21sh22:37:463
233787995825,8cyclictest37-21ksoftirqd/322:41:243
233787995825,8cyclictest37-21ksoftirqd/322:41:243
233787995824,8cyclictest37-21ksoftirqd/323:32:293
233787995821,10cyclictest275-21dbus-daemon19:35:013
233782995826,6cyclictest238216-21latency_hist20:35:042
233773995829,13cyclictest251373-21ssh21:33:480
23378799572,16cyclictest237888-21unin-run20:30:013
23378799572,16cyclictest237888-21unin-run20:30:013
233787995718,5cyclictest235465-21latency_hist19:40:023
233787995713,10cyclictest266130-21systemd-run22:05:003
233787995712,10cyclictest303795-21sshd23:23:293
233782995724,8cyclictest297347-21sed23:10:012
233773995723,16cyclictest266148-21ssh22:05:010
23378799563,13cyclictest3467602-21systemd-journal20:00:003
233787995625,27cyclictest277259-21sh22:28:043
233787995620,4cyclictest247343-21sh21:25:083
233787995618,7cyclictest233971-21/usr/sbin/munin19:15:013
233787995616,10cyclictest275-21dbus-daemon19:45:003
233782995648,5cyclictest227253-21kworker/2:320:10:052
23377899568,6cyclictest131rcu_preempt22:45:291
233778995636,11cyclictest23-21ksoftirqd/123:25:021
233773995628,13cyclictest313157-21ssh23:42:500
233773995628,11cyclictest263316-21ssh21:59:010
233773995622,30cyclictest12-21ksoftirqd/023:06:390
23377399561,39cyclictest335935-21rm00:30:030
233778995536,11cyclictest256675-21ssh21:45:001
233773995528,12cyclictest335350-21ssh00:29:180
233773995528,12cyclictest314748-21ssh23:46:010
23377399551,38cyclictest329341-21ssh00:16:350
233773995513,38cyclictest131rcu_preempt21:25:430
233773995511,39cyclictest1-21systemd20:35:040
233787995437,13cyclictest263694-21cron22:00:013
233787995426,24cyclictest323566-21sh00:04:473
233787995423,5cyclictest328809-21sh00:15:183
233787995422,13cyclictest37-21ksoftirqd/322:33:363
233787995422,13cyclictest37-21ksoftirqd/322:33:353
233787995417,6cyclictest37-21ksoftirqd/322:54:003
233782995445,5cyclictest227253-21kworker/2:323:26:562
233782995425,8cyclictest30-21ksoftirqd/223:40:042
233778995445,6cyclictest23-21ksoftirqd/123:45:021
233773995444,7cyclictest16950irq/224-800000023:15:010
233773995436,4cyclictest16850irq/134-800000022:45:020
233773995436,4cyclictest16850irq/134-800000022:45:020
233773995428,10cyclictest287945-21ssh22:50:060
233773995425,13cyclictest323160-21ssh00:03:570
233773995425,13cyclictest308201-21ssh23:32:360
23377399542,36cyclictest131rcu_preempt21:43:480
233787995324,5cyclictest293444-21sh23:01:443
233787995321,4cyclictest239947-21diskmemload22:17:073
233787995313,12cyclictest1-21systemd19:50:013
233787995313,12cyclictest1-21systemd19:50:013
233782995345,5cyclictest263770-21kworker/2:123:37:322
233782995344,5cyclictest227253-21kworker/2:322:45:332
233778995324,10cyclictest23-21ksoftirqd/100:18:161
233773995327,11cyclictest286587-21ssh22:47:260
23377399531,35cyclictest12-21ksoftirqd/019:20:020
233787995221,5cyclictest239947-21diskmemload22:05:033
23378799522,15cyclictest238412-21modprobe20:40:003
233787995218,7cyclictest37-21ksoftirqd/300:13:553
233782995244,5cyclictest237649-21kworker/2:221:36:262
233782995243,6cyclictest263770-21kworker/2:122:06:132
233782995243,5cyclictest237649-21kworker/2:221:45:022
233782995243,5cyclictest227253-21kworker/2:319:50:012
233782995243,5cyclictest227253-21kworker/2:319:50:012
233778995239,11cyclictest23-21ksoftirqd/122:54:361
233778995234,12cyclictest275-21dbus-daemon22:20:011
233778995217,6cyclictest131rcu_preempt21:10:001
233778995217,6cyclictest131rcu_preempt21:10:001
233773995225,12cyclictest267532-21ssh22:07:500
233787995121,4cyclictest239947-21diskmemload21:53:163
233787995117,11cyclictest275-21dbus-daemon20:25:013
233782995143,5cyclictest263770-21kworker/2:100:14:042
233778995132,15cyclictest3467602-21systemd-journal20:21:511
233778995118,6cyclictest131rcu_preempt20:00:001
23377899511,4cyclictest131rcu_preempt00:00:001
233773995125,21cyclictest278286-21latency_hist22:30:020
23377399511,35cyclictest324115-21ssh00:05:370
233787995018,8cyclictest37-21ksoftirqd/323:47:003
233787995018,11cyclictest37-21ksoftirqd/323:40:553
23378299508,4cyclictest131rcu_preempt23:00:072
23378299508,4cyclictest131rcu_preempt21:19:322
233782995044,3cyclictest263770-21kworker/2:122:27:012
233782995042,5cyclictest328465-21kworker/2:200:29:522
233782995042,5cyclictest328465-21kworker/2:200:21:082
233782995042,5cyclictest227253-21kworker/2:321:51:052
233782995032,14cyclictest309-21in:imuxsock00:30:042
23378299502,6cyclictest314613-21ssh23:45:432
23377899506,8cyclictest23-21ksoftirqd/123:20:021
233778995030,15cyclictest3467602-21systemd-journal21:22:251
233773995039,7cyclictest16950irq/224-800000021:00:000
233787994920,10cyclictest37-21ksoftirqd/321:43:593
233787994916,9cyclictest37-21ksoftirqd/323:06:573
233782994941,5cyclictest263770-21kworker/2:122:59:152
233782994941,5cyclictest263770-21kworker/2:122:38:172
233782994941,5cyclictest263770-21kworker/2:122:19:582
23377899492,10cyclictest23-21ksoftirqd/121:47:181
233773994917,12cyclictest254300-21ssh21:40:010
233787994832,11cyclictest275-21dbus-daemon20:20:023
233787994823,11cyclictest37-21ksoftirqd/323:11:243
233782994840,5cyclictest227253-21kworker/2:321:00:012
23377899488,6cyclictest131rcu_preempt00:24:511
23377899482,5cyclictest131rcu_preempt22:45:001
23377899482,5cyclictest131rcu_preempt22:45:001
233778994817,5cyclictest131rcu_preempt00:02:461
23377899481,16cyclictest0-21swapper/121:11:591
233778994810,5cyclictest131rcu_preempt20:50:001
233773994840,4cyclictest16950irq/224-800000019:10:020
23377399483,40cyclictest311684-21unin-run23:40:000
233773994832,13cyclictest274430-21ssh22:22:050
233773994832,13cyclictest274430-21ssh22:22:040
233773994827,16cyclictest275-21dbus-daemon21:05:000
233773994826,16cyclictest235249-21cron19:40:000
233773994818,14cyclictest301447-21ssh23:18:400
233773994818,13cyclictest318849-21ssh23:55:020
233782994740,5cyclictest227253-21kworker/2:323:58:142
233782994739,5cyclictest237649-21kworker/2:221:27:302
233782994737,6cyclictest328465-21kworker/2:200:19:272
233782994731,11cyclictest275-21dbus-daemon19:55:002
23378299471,9cyclictest258439-21ssh21:48:462
23377899473,8cyclictest23-21ksoftirqd/122:40:001
233778994737,7cyclictest23-21ksoftirqd/123:07:131
233778994736,8cyclictest23-21ksoftirqd/121:30:531
233778994735,6cyclictest238657-21munin-plugin-st20:45:001
233778994735,6cyclictest238657-21munin-plugin-st20:45:001
233778994732,11cyclictest23-21ksoftirqd/122:25:011
233778994732,11cyclictest23-21ksoftirqd/122:25:001
233778994714,6cyclictest131rcu_preempt21:00:001
233778994712,9cyclictest131rcu_preempt23:48:511
233778994712,12cyclictest23-21ksoftirqd/119:40:001
233773994731,12cyclictest270399-21ssh22:13:480
233773994725,7cyclictest246512-21ssh21:23:420
233773994725,7cyclictest242560-21ssh21:15:040
23377399471,7cyclictest239942-21modprobe21:10:000
23377399471,7cyclictest239942-21modprobe21:10:000
233773994717,27cyclictest12-21ksoftirqd/000:37:520
233773994717,27cyclictest12-21ksoftirqd/000:37:520
233787994618,9cyclictest37-21ksoftirqd/300:27:113
233782994639,5cyclictest263770-21kworker/2:122:35:022
233782994639,5cyclictest263770-21kworker/2:122:35:022
233782994638,5cyclictest227253-21kworker/2:323:33:082
233782994637,5cyclictest309-21in:imuxsock20:30:012
233782994637,5cyclictest309-21in:imuxsock20:30:012
233782994637,5cyclictest237649-21kworker/2:220:55:012
23377899468,6cyclictest131rcu_preempt23:52:021
23377899468,17cyclictest131rcu_preempt23:02:281
233778994638,6cyclictest221rcuc/100:08:311
233778994637,7cyclictest23-21ksoftirqd/121:35:421
233778994630,12cyclictest3467602-21systemd-journal19:45:011
233778994616,5cyclictest131rcu_preempt00:35:471
233778994616,5cyclictest131rcu_preempt00:35:461
233778994614,6cyclictest131rcu_preempt00:13:381
233778994613,5cyclictest131rcu_preempt22:01:351
233778994612,11cyclictest23-21ksoftirqd/122:14:441
233778994611,7cyclictest131rcu_preempt22:55:121
233773994634,7cyclictest239947-21diskmemload21:55:030
233773994629,13cyclictest12-21ksoftirqd/020:20:010
23377399461,3cyclictest131rcu_preempt23:20:290
23378799454,3cyclictest150-21pr/ttyS220:35:003
233787994517,9cyclictest37-21ksoftirqd/323:37:103
233787994516,9cyclictest1-21systemd21:10:013
233787994516,9cyclictest1-21systemd21:10:013
23378299454,8cyclictest30-21ksoftirqd/219:23:532
233782994537,5cyclictest263770-21kworker/2:122:20:342
233782994537,5cyclictest263770-21kworker/2:122:20:342
233782994537,5cyclictest227253-21kworker/2:320:05:022
233782994537,5cyclictest227253-21kworker/2:320:05:022
233782994512,6cyclictest131rcu_preempt21:22:082
23377899455,7cyclictest23-21ksoftirqd/100:33:571
233778994526,15cyclictest1-21systemd21:30:011
23377899452,16cyclictest23-21ksoftirqd/119:30:021
233778994512,11cyclictest131rcu_preempt22:30:021
233778994511,9cyclictest23-21ksoftirqd/123:29:081
233778994511,6cyclictest131rcu_preempt20:20:001
233773994538,4cyclictest16950irq/224-800000020:40:000
233773994528,14cyclictest327142-21ssh00:11:580
233773994528,12cyclictest0-21swapper/022:35:010
233773994528,12cyclictest0-21swapper/022:35:010
233773994522,4cyclictest131rcu_preempt21:48:450
23377399451,39cyclictest273322-21latency_hist22:20:000
233782994428,12cyclictest30-21ksoftirqd/220:00:002
233782994410,6cyclictest131rcu_preempt23:20:262
23377899448,9cyclictest23-21ksoftirqd/100:26:171
23377899444,9cyclictest23-21ksoftirqd/119:25:011
233778994433,8cyclictest23-21ksoftirqd/121:50:491
233778994429,10cyclictest234239-21systemctl19:20:001
233778994410,6cyclictest131rcu_preempt23:30:031
233773994438,4cyclictest16950irq/224-800000022:58:340
23378299437,31cyclictest238672-21unin-run20:45:012
23378299437,31cyclictest238672-21unin-run20:45:012
233782994329,10cyclictest235557-21sshd19:45:012
23377899438,5cyclictest131rcu_preempt22:00:011
233778994311,6cyclictest131rcu_preempt23:13:001
233773994322,14cyclictest3467602-21systemd-journal19:45:000
233787994219,8cyclictest37-21ksoftirqd/321:16:343
233782994234,5cyclictest221443-21kworker/2:019:20:022
233782994230,9cyclictest1-21systemd20:40:002
23378299422,9cyclictest0-21swapper/222:01:322
233782994210,4cyclictest131rcu_preempt00:04:062
233778994232,7cyclictest23-21ksoftirqd/120:30:021
233778994232,7cyclictest23-21ksoftirqd/120:30:021
233778994215,8cyclictest23-21ksoftirqd/119:35:021
23377899421,16cyclictest0-21swapper/123:36:531
233773994227,11cyclictest3467602-21systemd-journal20:41:090
233773994227,11cyclictest3467602-21systemd-journal20:41:090
23378799411,3cyclictest131rcu_preempt21:10:513
233782994133,5cyclictest227253-21kworker/2:320:20:592
233782994132,6cyclictest227253-21kworker/2:321:13:272
233782994130,8cyclictest3467602-21systemd-journal19:35:032
233782994127,9cyclictest30-21ksoftirqd/220:10:012
233782994123,12cyclictest3467602-21systemd-journal20:20:002
23377899418,5cyclictest131rcu_preempt20:35:041
23378799404,7cyclictest37-21ksoftirqd/321:49:303
23378799402,12cyclictest317330-21sshd23:51:303
233782994032,5cyclictest227253-21kworker/2:321:05:012
233778994028,9cyclictest23-21ksoftirqd/122:30:551
233778994028,9cyclictest23-21ksoftirqd/122:30:551
233778994028,8cyclictest23-21ksoftirqd/120:10:011
233778994026,10cyclictest23-21ksoftirqd/120:40:001
233773994017,6cyclictest131rcu_preempt19:25:010
23378799395,3cyclictest131rcu_preempt20:50:003
23378799392,4cyclictest131rcu_preempt19:55:013
23378799391,4cyclictest234523-21unin-run19:25:013
233778993928,9cyclictest23-21ksoftirqd/119:55:021
233778993926,10cyclictest23-21ksoftirqd/120:55:031
233778993910,6cyclictest131rcu_preempt22:07:191
233773993933,4cyclictest0-21swapper/019:15:000
23377399393,32cyclictest0-21swapper/020:00:020
233787993825,10cyclictest37-21ksoftirqd/319:30:003
233778993827,7cyclictest23-21ksoftirqd/120:10:511
233778993823,10cyclictest309-21in:imuxsock20:00:041
233778993823,10cyclictest309-21in:imuxsock20:00:041
23377399387,8cyclictest12-21ksoftirqd/020:25:090
23377399387,8cyclictest12-21ksoftirqd/020:25:080
233773993832,4cyclictest16950irq/224-800000019:35:010
233773993826,9cyclictest12-21ksoftirqd/019:30:000
23377399381,3cyclictest131rcu_preempt20:55:020
23377399381,33cyclictest0-21swapper/020:10:010
233787993729,6cyclictest37-21ksoftirqd/320:55:023
23378799371,11cyclictest131rcu_preempt20:10:013
23378299371,6cyclictest0-21swapper/219:35:022
233778993727,8cyclictest23-21ksoftirqd/121:01:321
233778993724,8cyclictest23-21ksoftirqd/119:15:011
233787993627,5cyclictest309-21in:imuxsock21:00:013
23377399362,31cyclictest1-21systemd20:03:500
23377399362,31cyclictest1-21systemd20:03:490
23378799358,4cyclictest131rcu_preempt21:05:003
233782993525,7cyclictest233968-21modprobe19:15:002
233782993523,9cyclictest240051-21sed21:10:032
233782993523,9cyclictest240051-21sed21:10:022
233778993519,10cyclictest235786-21modprobe19:50:001
233778993519,10cyclictest235786-21modprobe19:50:001
233773993529,4cyclictest0-21swapper/020:10:020
23377399351,31cyclictest0-21swapper/020:20:400
23377399351,30cyclictest0-21swapper/020:49:110
233787993423,8cyclictest37-21ksoftirqd/319:18:273
23377399341,4cyclictest235853-21latency_hist19:50:020
23377399341,4cyclictest235853-21latency_hist19:50:020
23377399331,4cyclictest0-21swapper/019:51:320
233787993225,4cyclictest238666-21sh20:45:023
233787993225,4cyclictest238666-21sh20:45:013
233778993222,7cyclictest233779-21latency_hist19:10:021
233787993124,4cyclictest233711-21latency_hist19:10:023
233787993119,9cyclictest3467602-21systemd-journal20:00:023
233787993119,9cyclictest3467602-21systemd-journal20:00:023
23378299291,4cyclictest131rcu_preempt19:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional