You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-15 - 03:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Mar 15, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27078759910880,25cyclictest2758863-21ssh22:50:022
2707875998967,20cyclictest30-21ksoftirqd/221:35:022
2707879997927,12cyclictest37-21ksoftirqd/321:15:013
2707865997618,42cyclictest12-21ksoftirqd/000:13:420
2707879997519,13cyclictest37-21ksoftirqd/322:50:013
270786599731,41cyclictest0-21swapper/000:31:180
2707865997217,22cyclictest2743563-21ssh22:15:300
270786599721,40cyclictest131rcu_preempt22:24:590
2707865996729,30cyclictest12-21ksoftirqd/021:25:020
270786599667,39cyclictest12-21ksoftirqd/022:33:140
2707865996618,44cyclictest12-21ksoftirqd/000:20:000
2707865996617,33cyclictest12-21ksoftirqd/021:49:440
2707865996523,22cyclictest2751950-21kworker/u8:222:51:130
2707865996523,22cyclictest2751950-21kworker/u8:222:51:130
2707870996223,8cyclictest2767535-21ssh23:09:271
2707879996116,6cyclictest131rcu_preempt21:40:013
2707865996118,39cyclictest12-21ksoftirqd/019:59:580
2707870996010,11cyclictest131rcu_preempt00:20:021
2707865996019,6cyclictest16850irq/134-800000021:30:220
2707865996017,35cyclictest12-21ksoftirqd/022:40:030
2707875995943,12cyclictest1-21systemd00:40:012
2707875995943,12cyclictest1-21systemd00:40:012
2707865995917,38cyclictest12-21ksoftirqd/021:51:550
2707865995910,27cyclictest131rcu_preempt00:00:000
270787999587,7cyclictest131rcu_preempt22:15:013
270787999584,7cyclictest131rcu_preempt00:00:003
2707875995842,7cyclictest2736842-21kworker/2:023:26:322
270786599585,44cyclictest12-21ksoftirqd/021:15:200
2707865995848,5cyclictest0-21swapper/023:05:010
2707865995848,5cyclictest0-21swapper/023:05:010
2707865995829,13cyclictest2770449-21ssh23:15:380
2707865995818,36cyclictest16850irq/134-800000021:44:240
2707865995817,33cyclictest16850irq/134-800000022:30:010
2707865995815,39cyclictest12-21ksoftirqd/022:57:250
270787999577,5cyclictest131rcu_preempt22:52:223
270787999577,5cyclictest131rcu_preempt22:52:223
2707879995714,6cyclictest131rcu_preempt23:10:003
2707865995719,34cyclictest16850irq/134-800000022:35:070
2707865995718,34cyclictest16850irq/134-800000023:37:540
2707865995717,35cyclictest16850irq/134-800000000:03:570
2707865995713,5cyclictest12-21ksoftirqd/000:40:020
2707865995713,5cyclictest12-21ksoftirqd/000:40:020
270786599571,30cyclictest131rcu_preempt23:33:100
2707879995634,17cyclictest2765521-21unin-run23:05:013
2707879995634,17cyclictest2765521-21unin-run23:05:013
2707865995628,13cyclictest2738442-21ssh22:04:210
2707865995619,32cyclictest16850irq/134-800000023:08:180
2707865995618,34cyclictest16850irq/134-800000000:25:300
2707865995615,37cyclictest12-21ksoftirqd/023:10:180
2707879995537,13cyclictest275-21dbus-daemon21:45:013
2707879995513,9cyclictest131rcu_preempt22:35:023
2707865995519,5cyclictest16850irq/134-800000021:36:470
2707865995518,6cyclictest16850irq/134-800000022:45:130
2707865995518,6cyclictest16850irq/134-800000021:13:550
2707865995517,34cyclictest16850irq/134-800000023:42:360
2707865995517,34cyclictest16850irq/134-800000021:56:010
2707865995517,34cyclictest16850irq/134-800000000:05:560
2707865995517,33cyclictest16850irq/134-800000023:26:480
2707865995516,31cyclictest16850irq/134-800000021:28:380
2707875995440,5cyclictest2713274-21kworker/2:321:36:462
2707875995437,13cyclictest1-21systemd23:50:012
2707875995437,13cyclictest1-21systemd23:50:012
2707870995436,14cyclictest3467602-21systemd-journal21:55:001
2707865995417,33cyclictest16850irq/134-800000023:47:470
2707865995417,33cyclictest16850irq/134-800000023:47:470
2707865995417,33cyclictest16850irq/134-800000000:20:100
2707865995416,34cyclictest16850irq/134-800000023:20:250
270787999535,13cyclictest37-21ksoftirqd/321:10:013
270787999535,13cyclictest37-21ksoftirqd/321:10:013
2707879995319,23cyclictest131rcu_preempt00:20:023
2707879995315,6cyclictest131rcu_preempt22:00:013
2707875995345,5cyclictest2700599-21kworker/2:121:24:142
2707875995344,5cyclictest2736842-21kworker/2:022:54:472
2707875995344,5cyclictest2736842-21kworker/2:022:54:472
270787599531,37cyclictest2707603-21kworker/u8:319:15:022
2707870995336,12cyclictest1-21systemd23:30:011
2707865995317,33cyclictest16850irq/134-800000023:51:590
2707865995315,34cyclictest16850irq/134-800000022:10:490
270787999528,6cyclictest131rcu_preempt22:29:433
2707875995244,5cyclictest2700599-21kworker/2:122:20:112
2707875995243,6cyclictest2736842-21kworker/2:000:14:142
2707875995236,12cyclictest3467602-21systemd-journal20:28:242
2707870995227,21cyclictest23-21ksoftirqd/100:25:001
2707865995215,33cyclictest16850irq/134-800000022:08:430
2707879995131,15cyclictest3467602-21systemd-journal00:10:003
270787999512,43cyclictest1-21systemd21:00:013
2707875995143,5cyclictest2736842-21kworker/2:022:35:412
2707875995143,5cyclictest2700599-21kworker/2:123:15:012
2707875995135,12cyclictest3467602-21systemd-journal23:57:322
2707870995131,5cyclictest23-21ksoftirqd/100:00:011
2707870995122,9cyclictest23-21ksoftirqd/123:42:571
2707865995135,11cyclictest16950irq/224-800000021:10:000
2707865995135,11cyclictest16950irq/224-800000021:09:590
270787999508,6cyclictest131rcu_preempt23:55:013
2707875995043,5cyclictest2700599-21kworker/2:123:08:252
2707875995043,4cyclictest2736842-21kworker/2:000:09:472
2707875995043,4cyclictest2736842-21kworker/2:000:05:022
2707875995042,5cyclictest2736842-21kworker/2:023:53:582
2707875995042,5cyclictest2736842-21kworker/2:023:36:342
2707875995042,5cyclictest2736842-21kworker/2:022:30:132
2707875995042,5cyclictest2736842-21kworker/2:022:27:472
2707875995042,5cyclictest2713274-21kworker/2:321:26:402
2707875995042,5cyclictest2713274-21kworker/2:321:17:452
2707875995042,5cyclictest2700599-21kworker/2:123:04:152
2707875995042,5cyclictest2700599-21kworker/2:123:04:152
2707875995042,5cyclictest2700599-21kworker/2:120:10:242
2707875995042,5cyclictest2700599-21kworker/2:120:10:242
2707875995041,6cyclictest2736842-21kworker/2:023:23:262
2707870995025,14cyclictest23-21ksoftirqd/122:49:521
2707865995035,10cyclictest3467602-21systemd-journal19:40:000
270786599502,15cyclictest1-21systemd20:10:010
270787999494,6cyclictest37-21ksoftirqd/323:21:383
270787999491,6cyclictest0-21swapper/300:00:203
2707879994912,5cyclictest131rcu_preempt21:25:483
2707875994941,5cyclictest2700599-21kworker/2:120:50:002
2707875994940,5cyclictest2700599-21kworker/2:121:49:282
2707875994936,5cyclictest2700599-21kworker/2:121:41:532
2707875994930,13cyclictest3467602-21systemd-journal22:05:002
270787099493,6cyclictest131rcu_preempt19:35:011
270787099493,6cyclictest131rcu_preempt19:35:001
270787999485,6cyclictest131rcu_preempt22:41:223
2707879994832,11cyclictest275-21dbus-daemon21:35:013
2707879994816,5cyclictest131rcu_preempt00:15:003
2707879994813,6cyclictest131rcu_preempt20:50:003
2707875994840,5cyclictest2736842-21kworker/2:022:40:232
2707875994840,5cyclictest2700599-21kworker/2:121:11:352
2707875994840,5cyclictest2700599-21kworker/2:119:35:232
2707870994834,10cyclictest275-21dbus-daemon19:30:011
2707870994834,10cyclictest1-21systemd22:20:011
270787099481,9cyclictest275-21dbus-daemon22:00:001
2707865994811,5cyclictest131rcu_preempt20:20:000
270787999472,5cyclictest131rcu_preempt23:35:023
2707879994722,5cyclictest131rcu_preempt00:24:323
2707879994713,4cyclictest131rcu_preempt23:25:343
2707879994710,7cyclictest131rcu_preempt23:42:013
2707875994739,5cyclictest2794526-21kworker/2:100:32:302
2707875994739,5cyclictest2700599-21kworker/2:122:00:002
2707875994739,5cyclictest2700599-21kworker/2:120:43:172
2707875994739,5cyclictest2700599-21kworker/2:119:27:192
2707875994738,5cyclictest2736842-21kworker/2:023:41:122
2707870994720,17cyclictest23-21ksoftirqd/100:11:231
2707870994714,17cyclictest23-21ksoftirqd/123:13:331
2707865994718,20cyclictest0-21swapper/020:20:030
270787999465,9cyclictest37-21ksoftirqd/322:40:013
270787999465,15cyclictest37-21ksoftirqd/323:11:483
2707879994630,13cyclictest3467602-21systemd-journal20:09:243
2707879994630,12cyclictest2781164-21cat23:40:003
2707875994638,5cyclictest2700599-21kworker/2:123:15:462
2707875994638,5cyclictest2700599-21kworker/2:122:05:082
2707875994638,5cyclictest2700599-21kworker/2:120:58:312
2707875994638,5cyclictest2700599-21kworker/2:120:19:392
2707875994631,12cyclictest1-21systemd00:30:012
2707870994618,19cyclictest23-21ksoftirqd/122:25:301
2707870994614,20cyclictest131rcu_preempt20:40:011
270786599462,6cyclictest131rcu_preempt20:05:020
2707865994617,17cyclictest12-21ksoftirqd/020:15:020
2707865994617,17cyclictest12-21ksoftirqd/020:15:020
270787999458,5cyclictest131rcu_preempt19:40:013
270787999457,10cyclictest37-21ksoftirqd/321:50:023
270787999456,9cyclictest37-21ksoftirqd/321:16:433
270787999455,5cyclictest131rcu_preempt21:53:243
2707879994535,7cyclictest37-21ksoftirqd/322:21:383
2707879994532,7cyclictest2745583-21latency_hist22:20:033
2707879994522,14cyclictest3467602-21systemd-journal23:00:013
270787999451,9cyclictest0-21swapper/300:32:293
2707879994512,6cyclictest131rcu_preempt20:25:003
2707875994538,4cyclictest2700599-21kworker/2:121:53:372
2707875994537,5cyclictest2736842-21kworker/2:023:34:292
2707875994537,5cyclictest2736842-21kworker/2:022:55:162
2707875994537,5cyclictest2700599-21kworker/2:122:17:062
2707875994533,8cyclictest2710397-21cron20:00:002
270787599451,3cyclictest131rcu_preempt00:18:212
270787099459,4cyclictest131rcu_preempt19:19:241
270787099459,4cyclictest131rcu_preempt19:19:241
2707870994530,11cyclictest2772197-21tr23:20:001
270787999449,9cyclictest37-21ksoftirqd/320:05:013
2707879994433,7cyclictest37-21ksoftirqd/323:15:383
2707879994422,3cyclictest131rcu_preempt22:00:023
270787999441,9cyclictest0-21swapper/322:06:593
270787999441,9cyclictest0-21swapper/300:38:553
270787999441,9cyclictest0-21swapper/300:38:553
2707879994414,5cyclictest131rcu_preempt00:28:243
2707875994436,5cyclictest2700599-21kworker/2:122:12:012
2707875994432,5cyclictest2736842-21kworker/2:000:24:392
2707875994429,11cyclictest1-21systemd20:05:012
2707875994428,12cyclictest1-21systemd20:10:002
270787099447,6cyclictest131rcu_preempt23:40:011
2707870994427,13cyclictest1-21systemd22:25:011
2707870994427,12cyclictest275-21dbus-daemon22:05:011
2707870994422,16cyclictest275-21dbus-daemon19:55:001
2707870994420,7cyclictest23-21ksoftirqd/100:07:251
2707870994415,19cyclictest131rcu_preempt22:10:021
270787099441,3cyclictest131rcu_preempt22:57:251
270787099441,3cyclictest131rcu_preempt21:45:011
270787099441,3cyclictest131rcu_preempt21:10:021
270787099441,3cyclictest131rcu_preempt21:10:021
270787099441,3cyclictest131rcu_preempt21:00:021
270786599446,3cyclictest131rcu_preempt20:30:010
2707865994433,7cyclictest19350irq/34-mmc120:40:010
2707865994425,13cyclictest275-21dbus-daemon19:15:010
2707865994412,7cyclictest131rcu_preempt20:35:000
270787999436,23cyclictest2712473-21modprobe20:40:003
2707879994332,9cyclictest37-21ksoftirqd/320:15:243
2707879994320,18cyclictest275-21dbus-daemon20:15:013
2707879994320,18cyclictest275-21dbus-daemon20:15:013
270787999431,8cyclictest0-21swapper/323:46:413
270787999431,8cyclictest0-21swapper/323:46:413
2707879994317,7cyclictest37-21ksoftirqd/320:00:013
270787599434,5cyclictest131rcu_preempt20:55:002
270787099434,5cyclictest131rcu_preempt23:45:091
270787099434,5cyclictest131rcu_preempt23:45:091
2707870994330,10cyclictest2713957-21cat21:05:011
2707870994329,11cyclictest2743149-21sh22:15:001
2707870994316,14cyclictest23-21ksoftirqd/121:36:231
2707870994314,20cyclictest23-21ksoftirqd/122:42:381
2707870994313,20cyclictest23-21ksoftirqd/123:50:151
2707865994323,5cyclictest12-21ksoftirqd/019:50:020
2707865994321,5cyclictest12-21ksoftirqd/019:25:010
2707865994321,4cyclictest12-21ksoftirqd/021:00:000
2707865994319,4cyclictest12-21ksoftirqd/020:55:010
2707865994319,19cyclictest12-21ksoftirqd/019:20:030
2707865994319,19cyclictest12-21ksoftirqd/019:20:020
270787999429,6cyclictest131rcu_preempt19:30:003
270787999423,5cyclictest131rcu_preempt20:45:003
270787999421,9cyclictest0-21swapper/321:20:393
2707879994213,5cyclictest131rcu_preempt19:35:013
2707879994213,5cyclictest131rcu_preempt19:35:013
2707875994229,9cyclictest275-21dbus-daemon20:35:012
270787099425,6cyclictest23-21ksoftirqd/100:30:011
2707870994234,6cyclictest23-21ksoftirqd/123:21:461
2707870994227,12cyclictest23-21ksoftirqd/100:39:151
2707870994227,12cyclictest23-21ksoftirqd/100:39:151
2707870994221,8cyclictest23-21ksoftirqd/121:15:521
2707865994226,12cyclictest2709088-21sshd19:33:250
2707865994226,12cyclictest2709088-21sshd19:33:240
2707865994225,4cyclictest12-21ksoftirqd/019:45:000
2707865994221,5cyclictest12-21ksoftirqd/020:50:010
2707865994218,20cyclictest12-21ksoftirqd/019:30:000
270787999413,6cyclictest131rcu_preempt20:30:013
2707879994121,13cyclictest275-21dbus-daemon19:45:013
2707879994111,6cyclictest131rcu_preempt19:50:033
2707875994133,5cyclictest2700599-21kworker/2:119:31:302
2707875994133,5cyclictest2700599-21kworker/2:119:31:302
2707875994125,12cyclictest2713778-21cron21:05:002
270787099415,3cyclictest131rcu_preempt20:45:011
270787099413,4cyclictest131rcu_preempt22:50:471
270787099413,4cyclictest131rcu_preempt22:50:471
2707870994129,9cyclictest23-21ksoftirqd/121:20:201
270787099411,9cyclictest2709403-21unin-run19:40:011
2707870994119,16cyclictest275-21dbus-daemon20:30:011
2707870994113,21cyclictest131rcu_preempt22:38:161
2707870994112,7cyclictest131rcu_preempt21:15:001
2707865994120,6cyclictest12-21ksoftirqd/020:45:010
270787999404,25cyclictest284-21systemd-logind19:25:013
2707879994025,10cyclictest37-21ksoftirqd/319:15:013
2707879994010,6cyclictest131rcu_preempt20:55:013
2707875994018,16cyclictest275-21dbus-daemon21:10:012
2707875994018,16cyclictest275-21dbus-daemon21:10:012
2707870994025,12cyclictest1-21systemd20:35:011
2707870994025,10cyclictest3467602-21systemd-journal23:05:011
2707870994025,10cyclictest3467602-21systemd-journal23:05:011
270787099401,9cyclictest2713030-21grep20:50:001
2707870994018,11cyclictest2721358-21ssh21:25:521
2707870994015,14cyclictest23-21ksoftirqd/122:30:471
270787099401,11cyclictest131rcu_preempt00:02:391
270786599401,35cyclictest2710112-21sshd19:50:240
270787999395,6cyclictest131rcu_preempt21:05:003
270787999395,6cyclictest131rcu_preempt19:20:013
270787999395,6cyclictest131rcu_preempt19:20:013
270787999394,5cyclictest131rcu_preempt20:34:593
2707879993910,6cyclictest131rcu_preempt19:10:013
2707875993932,4cyclictest2700599-21kworker/2:119:42:232
2707875993916,7cyclictest131rcu_preempt20:40:012
270787099398,4cyclictest131rcu_preempt20:55:001
270787099391,3cyclictest131rcu_preempt00:30:561
2707865993922,4cyclictest12-21ksoftirqd/021:05:010
2707875993820,14cyclictest275-21dbus-daemon19:50:012
270787099383,3cyclictest131rcu_preempt23:31:241
270787099382,5cyclictest131rcu_preempt20:00:001
2707870993817,8cyclictest23-21ksoftirqd/121:30:151
2707870993816,6cyclictest131rcu_preempt21:48:421
2707870993815,5cyclictest23-21ksoftirqd/119:10:021
270786599389,8cyclictest0-21swapper/019:10:020
2707875993724,9cyclictest2708569-21sshd19:20:252
2707870993726,8cyclictest2711692-21sshd20:22:231
2707870993725,9cyclictest23-21ksoftirqd/119:50:001
270787099372,3cyclictest131rcu_preempt20:15:031
270787099372,3cyclictest131rcu_preempt20:15:031
2707870993710,17cyclictest2708065-21munin-run19:15:001
270787999363,25cyclictest2710149-21munin-plugin-st19:55:003
2707875993628,5cyclictest2700599-21kworker/2:119:20:022
2707875993628,5cyclictest2700599-21kworker/2:119:20:022
2707870993626,7cyclictest23-21ksoftirqd/120:20:001
2707870993621,11cyclictest2708611-21systemd-run19:24:591
270787099361,6cyclictest131rcu_preempt20:10:001
2707870993525,7cyclictest2710838-21sed20:05:031
2707875993426,5cyclictest2707805-21latency_hist19:10:012
2707875993426,5cyclictest2700599-21kworker/2:120:25:022
2707870993422,7cyclictest23-21ksoftirqd/119:40:021
2707875993326,5cyclictest2710147-21/usr/sbin/munin19:54:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional