You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 08:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Aug 07, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2568260991472,15cyclictest131rcu_preempt20:20:010
2568260998611,46cyclictest16850irq/134-800000023:35:240
2568268997136,29cyclictest3467602-21systemd-journal00:00:002
2568268997134,31cyclictest1-21systemd22:25:022
2568268997134,31cyclictest1-21systemd22:25:012
2568260996829,3cyclictest131rcu_preempt21:59:020
2568260996829,3cyclictest131rcu_preempt21:59:010
2568268996625,5cyclictest2612493-21sh22:29:412
2568268996524,5cyclictest2580501-21sh21:22:212
2568268996524,4cyclictest2620673-21sh22:46:452
256826099658,29cyclictest131rcu_preempt22:10:010
256826099658,29cyclictest131rcu_preempt22:10:000
2568268996424,4cyclictest2613012-21sh22:30:192
2568268996424,4cyclictest2613012-21sh22:30:182
2568268996332,27cyclictest3467602-21systemd-journal22:06:332
2568268996332,27cyclictest3467602-21systemd-journal22:06:322
2568268996325,7cyclictest2623151-21sshd22:52:102
2568268996228,30cyclictest2636941-21sh23:20:502
2568268996122,9cyclictest30-21ksoftirqd/223:04:392
2568260996116,23cyclictest2667573-21ssh00:25:270
2568268996026,12cyclictest30-21ksoftirqd/221:30:592
2568268996024,5cyclictest2574153-21latency_hist21:00:022
256826899602,20cyclictest2598243-21rm22:00:012
256826899602,20cyclictest2598243-21rm22:00:002
2568268996021,7cyclictest30-21ksoftirqd/222:37:192
2568268995924,9cyclictest30-21ksoftirqd/221:25:572
2568268995924,5cyclictest30-21ksoftirqd/221:41:162
2568268995924,4cyclictest2572353-21latency_hist20:25:032
2568266995914,6cyclictest131rcu_preempt22:00:021
2568271995820,27cyclictest37-21ksoftirqd/300:20:033
2568268995821,9cyclictest3467602-21systemd-journal00:19:592
2568260995816,33cyclictest16850irq/134-800000023:29:310
2568260995816,33cyclictest16850irq/134-800000023:29:310
2568260995728,14cyclictest2664272-21ssh00:18:440
2568260995728,14cyclictest2651556-21ssh23:51:350
2568260995721,15cyclictest131rcu_preempt23:35:000
2568260995718,34cyclictest16850irq/134-800000021:44:130
2568268995628,6cyclictest2656190-21sh00:01:282
2568268995623,4cyclictest2668720-21sh00:27:572
2568268995619,11cyclictest30-21ksoftirqd/223:41:342
2568268995618,11cyclictest30-21ksoftirqd/222:04:292
2568260995619,32cyclictest16850irq/134-800000000:00:370
2568260995617,31cyclictest16850irq/134-800000023:45:020
2568260995616,20cyclictest16850irq/134-800000022:25:370
2568260995615,32cyclictest16850irq/134-800000000:10:550
256826099561,45cyclictest0-21swapper/023:10:000
2568260995613,32cyclictest12-21ksoftirqd/021:22:100
2568271995521,22cyclictest0-21swapper/322:39:553
2568268995527,6cyclictest2576770-21sh21:14:482
2568268995527,6cyclictest2576770-21sh21:14:482
2568268995519,9cyclictest30-21ksoftirqd/222:56:042
2568268995519,8cyclictest2641123-21wc23:30:022
2568268995519,8cyclictest2641123-21wc23:30:022
2568260995540,12cyclictest1-21systemd00:10:000
2568260995529,11cyclictest2633174-21ssh23:13:120
2568260995528,12cyclictest2599979-21ssh22:03:230
2568260995527,12cyclictest2604406-21ssh22:12:330
2568260995518,29cyclictest16850irq/134-800000022:22:180
2568260995518,29cyclictest16850irq/134-800000022:22:170
2568260995517,34cyclictest16850irq/134-800000022:32:480
2568260995517,34cyclictest16850irq/134-800000022:32:470
2568260995516,20cyclictest16850irq/134-800000022:43:080
2568260995515,20cyclictest16850irq/134-800000023:02:040
2568260995514,33cyclictest16850irq/134-800000021:26:180
2568271995437,12cyclictest1-21systemd19:40:003
2568268995421,8cyclictest30-21ksoftirqd/222:11:332
2568268995421,10cyclictest30-21ksoftirqd/221:47:372
2568268995421,10cyclictest0-21swapper/200:05:422
2568260995446,4cyclictest16950irq/224-800000019:50:000
256826099544,3cyclictest131rcu_preempt21:34:030
2568260995428,11cyclictest2606967-21ssh22:17:550
2568260995428,11cyclictest2593060-21ssh21:48:480
2568260995427,11cyclictest2653246-21ssh23:55:100
2568260995417,33cyclictest16850irq/134-800000022:59:450
2568260995416,34cyclictest16850irq/134-800000021:52:390
2568260995415,20cyclictest16850irq/134-800000023:20:060
2568260995411,39cyclictest16850irq/134-800000000:22:000
2568271995338,11cyclictest275-21dbus-daemon22:55:013
2568268995344,6cyclictest2612692-21kworker/2:300:12:132
2568268995344,5cyclictest2556211-21kworker/2:319:49:592
256826699534,3cyclictest131rcu_preempt19:50:001
2568260995317,32cyclictest16850irq/134-800000022:45:400
2568260995316,33cyclictest16850irq/134-800000023:46:310
2568260995316,33cyclictest16850irq/134-800000021:11:390
2568260995316,33cyclictest16850irq/134-800000021:11:380
2568260995316,33cyclictest16850irq/134-800000000:35:170
2568260995315,34cyclictest16850irq/134-800000022:51:010
2568260995313,32cyclictest16850irq/134-800000022:40:000
256827199525,9cyclictest37-21ksoftirqd/320:30:013
2568271995232,15cyclictest3467602-21systemd-journal22:00:013
2568271995232,15cyclictest3467602-21systemd-journal22:00:003
2568268995243,6cyclictest2641107-21kworker/2:000:31:532
2568266995239,10cyclictest2598241-21cron22:00:011
2568266995239,10cyclictest2598241-21cron22:00:001
2568260995227,10cyclictest2670191-21ssh00:30:540
2568260995216,32cyclictest16850irq/134-800000021:37:410
2568260995215,33cyclictest16850irq/134-800000021:16:200
2568260995211,21cyclictest16850irq/134-800000023:15:340
2568268995122,5cyclictest2572395-21latency_hist20:30:012
2568268995120,9cyclictest30-21ksoftirqd/200:37:222
2568266995135,12cyclictest3467602-21systemd-journal21:10:001
2568266995126,19cyclictest275-21dbus-daemon22:20:001
2568268995042,5cyclictest2584099-21kworker/2:021:53:192
2568268995042,5cyclictest2572412-21kworker/2:222:43:232
2568268995022,8cyclictest30-21ksoftirqd/221:35:062
2568268995021,6cyclictest30-21ksoftirqd/222:19:362
256826699508,35cyclictest2612656-21grep22:29:591
256826699504,3cyclictest131rcu_preempt00:35:011
2568266995027,10cyclictest2664794-21cron00:20:001
2568266995011,6cyclictest131rcu_preempt23:15:011
2568271994934,11cyclictest3467602-21systemd-journal23:30:003
2568271994934,11cyclictest3467602-21systemd-journal23:30:003
2568271994928,15cyclictest275-21dbus-daemon22:25:013
2568271994928,15cyclictest275-21dbus-daemon22:25:003
2568268994940,5cyclictest2568262-21kworker/2:121:10:022
2568268994923,5cyclictest30-21ksoftirqd/223:10:142
256826699492,3cyclictest131rcu_preempt21:35:021
2568266994916,7cyclictest131rcu_preempt00:00:001
256827199485,7cyclictest37-21ksoftirqd/321:43:163
2568268994840,5cyclictest2612692-21kworker/2:323:53:032
2568268994840,5cyclictest2612692-21kworker/2:300:20:312
2568268994839,6cyclictest2568262-21kworker/2:119:35:012
2568268994829,14cyclictest2645883-21cron23:40:002
256826699484,3cyclictest131rcu_preempt00:35:341
256826699483,3cyclictest131rcu_preempt00:07:261
256826699482,6cyclictest131rcu_preempt23:45:001
256826099482,40cyclictest2570845-21sendmail-msp20:00:010
2568260994815,19cyclictest151rcuc/019:35:030
2568271994732,11cyclictest3467602-21systemd-journal23:10:023
2568271994732,11cyclictest3467602-21systemd-journal21:35:013
256826699474,4cyclictest131rcu_preempt21:45:011
256826699473,3cyclictest131rcu_preempt21:53:311
256826699473,3cyclictest131rcu_preempt00:28:581
256826699472,4cyclictest131rcu_preempt23:54:251
2568271994636,7cyclictest37-21ksoftirqd/323:00:013
256827199463,3cyclictest131rcu_preempt23:24:123
2568268994638,5cyclictest2612692-21kworker/2:323:49:422
2568268994637,5cyclictest2641107-21kworker/2:023:31:292
2568268994629,12cyclictest275-21dbus-daemon21:05:002
2568268994614,11cyclictest3467602-21systemd-journal19:55:002
2568260994639,4cyclictest16950irq/224-800000019:45:010
2568260994620,8cyclictest2574656-21ssh21:10:030
2568271994537,6cyclictest37-21ksoftirqd/321:14:593
2568271994537,6cyclictest37-21ksoftirqd/321:14:583
256827199453,37cyclictest2668957-21sshd00:28:393
2568271994529,12cyclictest275-21dbus-daemon19:30:013
256827199451,4cyclictest131rcu_preempt23:14:593
2568268994536,5cyclictest2556211-21kworker/2:319:40:002
256826699453,3cyclictest131rcu_preempt23:31:071
256826699453,3cyclictest131rcu_preempt22:39:021
2568266994529,11cyclictest275-21dbus-daemon21:15:011
2568266994529,11cyclictest275-21dbus-daemon21:15:011
2568260994518,18cyclictest16850irq/134-800000021:02:250
256827199443,3cyclictest131rcu_preempt23:31:433
256827199443,3cyclictest131rcu_preempt20:00:023
2568271994431,9cyclictest3467602-21systemd-journal23:40:013
2568268994436,5cyclictest2572412-21kworker/2:223:15:132
2568268994436,5cyclictest2572412-21kworker/2:221:15:172
2568268994425,6cyclictest2556211-21kworker/2:320:00:002
2568266994430,7cyclictest309-21in:imuxsock23:40:011
2568266994429,11cyclictest3467602-21systemd-journal20:50:001
2568266994428,12cyclictest3467602-21systemd-journal20:45:001
2568266994428,11cyclictest275-21dbus-daemon22:35:001
2568266994428,11cyclictest275-21dbus-daemon22:35:001
256826699442,4cyclictest131rcu_preempt20:30:021
256826699442,4cyclictest131rcu_preempt00:20:391
256826699441,3cyclictest131rcu_preempt23:25:021
256826699441,3cyclictest131rcu_preempt23:17:361
2568260994417,19cyclictest16850irq/134-800000019:28:100
256826099441,4cyclictest2568270-21date19:10:010
256827199434,3cyclictest131rcu_preempt22:05:293
256827199434,3cyclictest131rcu_preempt22:05:293
2568271994326,13cyclictest275-21dbus-daemon22:45:013
2568271994321,5cyclictest37-21ksoftirqd/321:26:303
2568271994314,3cyclictest150-21pr/ttyS223:56:123
256827199431,3cyclictest131rcu_preempt22:05:013
2568271994311,4cyclictest150-21pr/ttyS223:44:403
2568266994323,14cyclictest2568732-21cron19:20:011
2568266994323,14cyclictest2568732-21cron19:20:001
2568260994324,14cyclictest275-21dbus-daemon20:35:000
256826099432,36cyclictest2568745-21sendmail19:20:010
256826099432,36cyclictest2568745-21sendmail19:20:000
2568271994229,10cyclictest37-21ksoftirqd/323:17:543
2568271994219,5cyclictest37-21ksoftirqd/321:54:433
256827199421,4cyclictest131rcu_preempt21:25:023
2568271994210,6cyclictest131rcu_preempt00:37:293
2568268994234,5cyclictest2612692-21kworker/2:323:05:272
2568266994228,10cyclictest0-21swapper/121:40:011
2568266994222,15cyclictest275-21dbus-daemon23:00:001
2568260994225,12cyclictest275-21dbus-daemon19:35:010
256827199416,7cyclictest131rcu_preempt21:19:303
256827199414,6cyclictest131rcu_preempt22:48:263
2568271994128,9cyclictest3467602-21systemd-journal00:34:103
256827199412,4cyclictest131rcu_preempt22:13:433
2568271994124,11cyclictest275-21dbus-daemon20:10:013
256827199412,3cyclictest131rcu_preempt22:17:513
2568271994123,14cyclictest2653119-21cut23:55:013
2568271994123,13cyclictest275-21dbus-daemon20:05:003
2568271994116,6cyclictest37-21ksoftirqd/319:35:013
256827199411,5cyclictest0-21swapper/323:00:363
256827199411,4cyclictest131rcu_preempt22:30:013
2568271994113,10cyclictest0-21swapper/300:00:043
2568268994128,10cyclictest3467602-21systemd-journal20:15:002
256826699413,3cyclictest131rcu_preempt23:26:481
256826699413,3cyclictest131rcu_preempt23:26:481
256826699413,3cyclictest131rcu_preempt21:21:131
2568266994129,9cyclictest2622133-21sed22:50:011
256826699412,3cyclictest131rcu_preempt00:14:111
256826699411,3cyclictest131rcu_preempt23:01:191
2568260994129,7cyclictest2572397-21munin-plugin-st20:29:590
256826099412,4cyclictest131rcu_preempt20:40:030
256826099412,4cyclictest131rcu_preempt20:40:020
256826099412,4cyclictest131rcu_preempt20:25:020
256826099411,4cyclictest131rcu_preempt20:15:000
256826099411,36cyclictest2573403-21sshd20:45:110
256827199408,5cyclictest131rcu_preempt23:46:313
256827199406,8cyclictest37-21ksoftirqd/300:12:513
256827199404,3cyclictest131rcu_preempt21:00:003
256827199401,4cyclictest131rcu_preempt19:20:023
256827199401,4cyclictest131rcu_preempt19:20:013
256827199401,3cyclictest131rcu_preempt00:08:373
2568268994032,5cyclictest2556211-21kworker/2:319:45:012
2568268994031,5cyclictest2556211-21kworker/2:319:30:012
2568268994027,9cyclictest2568684-21sed19:15:022
2568266994032,5cyclictest2657718-21latency_hist00:05:031
2568266994031,6cyclictest23-21ksoftirqd/121:20:001
2568266994030,6cyclictest309-21in:imuxsock19:40:011
2568266994028,8cyclictest3467602-21systemd-journal19:50:091
2568266994027,9cyclictest0-21swapper/120:20:001
2568266994025,11cyclictest2573692-21munin-plugin-st20:55:011
2568266994025,11cyclictest2573692-21munin-plugin-st20:55:001
256826699402,4cyclictest131rcu_preempt20:05:021
256826699402,3cyclictest131rcu_preempt23:50:011
2568266994020,14cyclictest275-21dbus-daemon19:45:011
256826699401,3cyclictest131rcu_preempt22:43:561
256826099403,6cyclictest2570566-21munin-run19:55:000
256827199395,3cyclictest131rcu_preempt00:22:123
256827199392,3cyclictest131rcu_preempt21:40:003
256827199391,3cyclictest131rcu_preempt22:33:083
256827199391,3cyclictest131rcu_preempt22:33:083
256827199391,3cyclictest131rcu_preempt21:48:233
2568268993926,10cyclictest275-21dbus-daemon19:25:012
2568268993920,15cyclictest3467602-21systemd-journal20:49:092
256826899391,3cyclictest131rcu_preempt20:40:012
256826899391,3cyclictest131rcu_preempt20:40:012
256826699397,24cyclictest3467602-21systemd-journal20:00:001
256826699393,3cyclictest131rcu_preempt22:52:441
2568266993929,8cyclictest23-21ksoftirqd/120:35:011
256826699392,4cyclictest131rcu_preempt21:46:281
2568266993924,8cyclictest309-21in:imuxsock22:25:011
2568266993924,8cyclictest309-21in:imuxsock22:25:001
2568260993925,10cyclictest275-21dbus-daemon20:55:010
2568260993925,10cyclictest275-21dbus-daemon20:55:010
2568260993922,11cyclictest3467602-21systemd-journal19:15:000
256826099391,5cyclictest2573969-21sendmail-msp21:00:010
256827199389,25cyclictest2572665-21unin-run20:35:003
256827199385,4cyclictest131rcu_preempt20:15:003
2568271993825,10cyclictest275-21dbus-daemon20:20:013
256827199382,4cyclictest131rcu_preempt20:40:013
256827199382,4cyclictest131rcu_preempt20:40:003
2568271993823,11cyclictest37-21ksoftirqd/319:15:003
256827199381,3cyclictest131rcu_preempt19:55:023
256826699386,4cyclictest131rcu_preempt23:05:141
2568266993817,4cyclictest23-21ksoftirqd/120:40:021
2568266993817,4cyclictest23-21ksoftirqd/120:40:011
2568266993813,4cyclictest23-21ksoftirqd/122:05:111
2568266993813,4cyclictest23-21ksoftirqd/122:05:101
2568266993810,8cyclictest23-21ksoftirqd/121:28:001
256826099384,3cyclictest131rcu_preempt20:10:020
2568260993813,20cyclictest16850irq/134-800000020:03:090
256827199375,4cyclictest131rcu_preempt19:50:013
2568271993722,10cyclictest275-21dbus-daemon19:25:013
256826899379,19cyclictest2569763-21kworker/u8:020:05:012
2568268993723,10cyclictest1-21systemd20:45:022
256826899371,16cyclictest3467602-21systemd-journal20:19:592
256826699374,3cyclictest131rcu_preempt22:11:101
2568266993725,8cyclictest23-21ksoftirqd/120:25:011
256826099374,4cyclictest131rcu_preempt20:45:000
256827199364,4cyclictest131rcu_preempt21:02:103
256827199363,4cyclictest131rcu_preempt20:55:023
256827199363,4cyclictest131rcu_preempt20:55:013
256827199361,3cyclictest131rcu_preempt20:25:003
2568268993626,7cyclictest30-21ksoftirqd/220:35:012
2568268993619,12cyclictest0-21swapper/220:10:012
256826699363,4cyclictest131rcu_preempt20:10:001
2568266993622,9cyclictest309-21in:imuxsock19:30:001
256826699361,3cyclictest131rcu_preempt21:00:031
256826099364,3cyclictest131rcu_preempt19:25:000
256827199353,4cyclictest131rcu_preempt21:10:023
256827199353,3cyclictest131rcu_preempt20:50:013
2568271993528,4cyclictest2570059-21latency_hist19:45:003
2568268993527,6cyclictest30-21ksoftirqd/220:55:022
2568268993527,6cyclictest30-21ksoftirqd/220:55:012
256826699356,23cyclictest2568496-21unin-run19:15:001
256827199344,3cyclictest131rcu_preempt20:45:003
2568268993425,6cyclictest0-21swapper/219:10:012
2568268993422,9cyclictest275-21dbus-daemon19:20:022
2568268993422,9cyclictest275-21dbus-daemon19:20:012
256826699343,3cyclictest131rcu_preempt19:24:591
256826699342,6cyclictest131rcu_preempt20:59:591
2568266993421,9cyclictest309-21in:imuxsock20:11:101
256826699333,3cyclictest131rcu_preempt19:31:101
2568266993326,4cyclictest2568252-21cyclictest19:10:011
256827199313,4cyclictest131rcu_preempt19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional