You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 08:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Mar 17, 2026 00:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
333698099115107,6cyclictest23-21ksoftirqd/123:25:031
333697699113105,6cyclictest12-21ksoftirqd/019:25:010
3336989997935,11cyclictest275-21dbus-daemon20:45:013
3336976997820,32cyclictest16850irq/134-800000000:16:150
3336989997113,41cyclictest0-21swapper/321:20:023
333697699681,3cyclictest0-21swapper/000:00:040
3336984996758,6cyclictest30-21ksoftirqd/220:55:012
333698499666,12cyclictest30-21ksoftirqd/221:10:012
333698499666,12cyclictest30-21ksoftirqd/221:10:012
3336984996619,10cyclictest30-21ksoftirqd/223:35:012
3336980996635,9cyclictest23-21ksoftirqd/100:07:151
3336980996412,15cyclictest23-21ksoftirqd/123:20:001
333698499636,6cyclictest30-21ksoftirqd/222:15:022
333698499635,8cyclictest30-21ksoftirqd/222:20:022
333698499624,7cyclictest30-21ksoftirqd/222:25:002
333698099627,9cyclictest23-21ksoftirqd/121:40:001
333698999613,31cyclictest3383552-21sh22:40:023
3336984996014,13cyclictest30-21ksoftirqd/219:40:022
3336980996017,8cyclictest23-21ksoftirqd/122:45:541
333697699594,34cyclictest131rcu_preempt00:35:020
333697699593,4cyclictest131rcu_preempt23:28:080
333697699593,4cyclictest131rcu_preempt23:28:070
333698499585,10cyclictest30-21ksoftirqd/220:00:002
333698499585,10cyclictest30-21ksoftirqd/220:00:002
3336984995825,8cyclictest30-21ksoftirqd/200:08:552
3336980995815,11cyclictest23-21ksoftirqd/123:45:011
3336976995841,13cyclictest1-21systemd23:05:020
3336976995841,13cyclictest1-21systemd23:05:010
333698499578,9cyclictest30-21ksoftirqd/223:30:012
333698499578,9cyclictest30-21ksoftirqd/223:30:002
333698499576,9cyclictest30-21ksoftirqd/223:20:012
3336984995749,5cyclictest3337722-21kworker/2:121:51:392
3336984995730,11cyclictest30-21ksoftirqd/222:50:002
3336984995722,11cyclictest30-21ksoftirqd/200:30:022
333698499572,15cyclictest30-21ksoftirqd/221:40:002
3336984995713,13cyclictest30-21ksoftirqd/223:00:012
333698099576,6cyclictest23-21ksoftirqd/121:30:031
333698099576,6cyclictest23-21ksoftirqd/121:25:001
333698099574,7cyclictest23-21ksoftirqd/122:40:011
333698099573,11cyclictest23-21ksoftirqd/122:42:091
333697699572,37cyclictest131rcu_preempt21:17:060
333698499569,14cyclictest30-21ksoftirqd/200:39:202
333698499567,10cyclictest30-21ksoftirqd/222:08:302
333698499566,11cyclictest30-21ksoftirqd/200:14:502
333698099566,9cyclictest23-21ksoftirqd/122:50:271
333698099566,9cyclictest23-21ksoftirqd/100:25:021
333698099564,9cyclictest23-21ksoftirqd/121:55:001
3336976995626,15cyclictest3408679-21ssh23:36:160
3336976995626,15cyclictest3408679-21ssh23:36:160
3336976995620,32cyclictest16850irq/134-800000022:59:580
3336976995613,36cyclictest16850irq/134-800000023:45:010
333698499558,9cyclictest30-21ksoftirqd/200:00:042
333698499556,8cyclictest30-21ksoftirqd/221:31:092
333698499556,8cyclictest30-21ksoftirqd/220:45:022
333698499551,8cyclictest3339249-21modprobe19:55:002
333698099556,6cyclictest23-21ksoftirqd/119:15:011
333698099556,14cyclictest23-21ksoftirqd/122:00:021
333698099555,8cyclictest23-21ksoftirqd/121:50:021
333698099554,12cyclictest23-21ksoftirqd/121:45:011
333698099552,9cyclictest23-21ksoftirqd/121:10:511
3336980995527,8cyclictest23-21ksoftirqd/122:00:231
333698099552,13cyclictest23-21ksoftirqd/121:10:031
333698099552,13cyclictest23-21ksoftirqd/121:10:021
333698099551,8cyclictest23-21ksoftirqd/100:13:421
3336980995511,4cyclictest131rcu_preempt23:40:011
3336980995511,4cyclictest131rcu_preempt23:40:001
333698099551,10cyclictest23-21ksoftirqd/122:59:591
333697699554,3cyclictest131rcu_preempt23:24:560
3336976995519,32cyclictest16850irq/134-800000021:10:390
3336976995519,32cyclictest16850irq/134-800000000:26:180
3336989995418,24cyclictest37-21ksoftirqd/319:55:013
333698499548,7cyclictest30-21ksoftirqd/200:20:022
333698499547,6cyclictest30-21ksoftirqd/221:12:052
333698499544,9cyclictest30-21ksoftirqd/223:07:332
3336984995441,9cyclictest30-21ksoftirqd/223:40:002
3336984995441,9cyclictest30-21ksoftirqd/223:40:002
333698499543,12cyclictest30-21ksoftirqd/221:20:032
333698099548,11cyclictest23-21ksoftirqd/119:35:021
333698099547,8cyclictest23-21ksoftirqd/122:09:291
333698099547,10cyclictest23-21ksoftirqd/100:05:011
333698099546,8cyclictest23-21ksoftirqd/100:25:311
333698099542,11cyclictest23-21ksoftirqd/123:15:031
3336976995436,14cyclictest3349836-21ssh21:25:010
3336976995427,18cyclictest3401411-21ssh23:20:020
3336976995426,13cyclictest3379835-21ssh22:31:400
3336976995424,14cyclictest3420091-21ssh00:01:430
3336976995424,14cyclictest3387224-21ssh22:48:330
3336976995419,31cyclictest16850irq/134-800000023:32:540
3336976995419,31cyclictest16850irq/134-800000022:13:470
3336976995419,30cyclictest16850irq/134-800000022:01:100
3336976995418,32cyclictest16850irq/134-800000000:21:410
3336976995418,20cyclictest16850irq/134-800000022:44:590
3336976995418,18cyclictest16850irq/134-800000000:10:010
3336976995417,7cyclictest16850irq/134-800000022:20:100
333698499539,8cyclictest30-21ksoftirqd/222:28:432
333698499539,13cyclictest30-21ksoftirqd/222:35:012
333698499537,7cyclictest30-21ksoftirqd/200:24:052
333698499536,9cyclictest30-21ksoftirqd/223:22:382
333698499536,8cyclictest30-21ksoftirqd/223:45:012
333698499536,11cyclictest30-21ksoftirqd/223:12:592
333698499534,7cyclictest30-21ksoftirqd/219:45:022
333698499534,7cyclictest30-21ksoftirqd/219:45:022
3336984995345,5cyclictest3340843-21kworker/2:321:22:082
333698499533,11cyclictest30-21ksoftirqd/220:15:002
333698499533,11cyclictest30-21ksoftirqd/220:15:002
3336984995311,13cyclictest30-21ksoftirqd/200:00:272
333698499531,10cyclictest30-21ksoftirqd/219:30:012
333698099537,8cyclictest23-21ksoftirqd/100:19:231
333698099536,7cyclictest23-21ksoftirqd/100:38:171
333698099536,5cyclictest131rcu_preempt22:24:381
333698099536,10cyclictest23-21ksoftirqd/122:25:091
333698099535,6cyclictest23-21ksoftirqd/123:57:331
333698099534,12cyclictest23-21ksoftirqd/123:54:591
333698099533,9cyclictest23-21ksoftirqd/120:25:011
333698099533,14cyclictest23-21ksoftirqd/123:50:001
333697699532,36cyclictest131rcu_preempt23:48:130
3336976995319,31cyclictest16850irq/134-800000021:25:250
3336976995319,30cyclictest16850irq/134-800000021:55:090
3336976995318,32cyclictest16850irq/134-800000000:14:520
3336976995318,31cyclictest16850irq/134-800000023:12:370
3336976995318,31cyclictest16850irq/134-800000021:44:060
3336976995317,32cyclictest16850irq/134-800000021:37:520
3336976995317,31cyclictest16850irq/134-800000022:52:140
3336976995317,31cyclictest16850irq/134-800000022:19:160
333698499527,9cyclictest30-21ksoftirqd/223:50:572
333698499527,7cyclictest30-21ksoftirqd/221:58:102
333698499527,10cyclictest30-21ksoftirqd/221:42:062
333698499523,11cyclictest30-21ksoftirqd/223:49:362
3336984995230,8cyclictest30-21ksoftirqd/222:50:142
333698499522,5cyclictest30-21ksoftirqd/222:05:002
3336984995214,8cyclictest131rcu_preempt21:28:212
333698099527,11cyclictest23-21ksoftirqd/100:31:141
333698099527,10cyclictest23-21ksoftirqd/121:00:011
333698099526,11cyclictest221rcuc/122:30:071
333698099525,8cyclictest23-21ksoftirqd/123:28:201
333698099525,8cyclictest23-21ksoftirqd/123:28:201
333698099524,7cyclictest23-21ksoftirqd/123:34:501
333697699523,15cyclictest3437034-21munin-run00:40:010
3336976995227,9cyclictest12-21ksoftirqd/021:55:000
3336976995219,30cyclictest16850irq/134-800000022:28:230
3336976995218,30cyclictest16850irq/134-800000023:50:490
3336976995215,32cyclictest16850irq/134-800000023:06:240
333698499516,10cyclictest30-21ksoftirqd/220:20:022
333698499515,10cyclictest30-21ksoftirqd/222:42:542
3336984995132,14cyclictest275-21dbus-daemon22:40:012
333698499513,13cyclictest30-21ksoftirqd/220:40:022
333698499511,9cyclictest3361060-21munin-run21:50:012
333698499511,11cyclictest30-21ksoftirqd/220:30:022
333698099516,9cyclictest23-21ksoftirqd/123:05:021
333698099516,9cyclictest23-21ksoftirqd/123:05:011
333698099516,11cyclictest23-21ksoftirqd/120:40:011
333698099515,7cyclictest23-21ksoftirqd/122:15:001
333698099514,9cyclictest23-21ksoftirqd/121:17:001
333698099511,9cyclictest3374500-21sendmail-msp22:20:011
3336980995112,10cyclictest23-21ksoftirqd/120:20:011
333698099511,14cyclictest23-21ksoftirqd/120:30:031
3336976995117,30cyclictest16850irq/134-800000022:07:000
3336976995116,31cyclictest16850irq/134-800000021:33:060
3336976995114,7cyclictest16850irq/134-800000022:39:510
333698999508,23cyclictest0-21swapper/300:26:463
333698499506,9cyclictest30-21ksoftirqd/220:25:002
3336984995013,8cyclictest30-21ksoftirqd/219:10:022
3336984995013,7cyclictest131rcu_preempt19:25:032
3336984995012,13cyclictest30-21ksoftirqd/220:10:032
3336984995011,10cyclictest131rcu_preempt00:31:532
333698099508,12cyclictest23-21ksoftirqd/121:30:581
333698099505,12cyclictest23-21ksoftirqd/120:05:021
333698099502,14cyclictest23-21ksoftirqd/123:05:281
333698099502,12cyclictest23-21ksoftirqd/119:40:021
3336976995016,31cyclictest16850irq/134-800000021:47:050
3336989994932,14cyclictest275-21dbus-daemon00:00:003
3336989994912,9cyclictest3424087-21sshd00:10:353
333698499496,10cyclictest30-21ksoftirqd/223:02:072
333698499496,10cyclictest30-21ksoftirqd/223:02:062
333698499495,8cyclictest30-21ksoftirqd/220:50:012
333698099493,8cyclictest23-21ksoftirqd/119:30:001
3336980994911,4cyclictest131rcu_preempt20:45:011
333698499487,8cyclictest30-21ksoftirqd/219:50:012
333698499484,16cyclictest30-21ksoftirqd/219:20:012
3336984994841,4cyclictest3322429-21kworker/2:219:32:012
3336984994840,5cyclictest3337722-21kworker/2:120:58:372
3336984994839,5cyclictest3337722-21kworker/2:121:02:182
333698099488,7cyclictest23-21ksoftirqd/120:50:011
333698099485,8cyclictest23-21ksoftirqd/120:00:001
333698099485,8cyclictest23-21ksoftirqd/120:00:001
3336980994834,11cyclictest3340299-21systemctl20:15:021
3336980994834,11cyclictest3340299-21systemctl20:15:011
3336976994839,4cyclictest12-21ksoftirqd/021:10:030
3336976994839,4cyclictest12-21ksoftirqd/021:10:020
333698999475,5cyclictest131rcu_preempt23:20:023
3336989994730,13cyclictest1-21systemd20:10:003
333698999471,6cyclictest0-21swapper/322:55:363
333698499477,8cyclictest30-21ksoftirqd/219:11:352
3336984994740,4cyclictest3337722-21kworker/2:120:01:402
333698099477,8cyclictest23-21ksoftirqd/120:08:351
3336976994740,5cyclictest12-21ksoftirqd/020:35:010
333698999466,8cyclictest37-21ksoftirqd/300:15:393
333698999465,19cyclictest0-21swapper/321:28:343
3336989994631,12cyclictest37-21ksoftirqd/300:30:263
3336989994630,13cyclictest1-21systemd22:00:013
333698999461,4cyclictest131rcu_preempt22:30:023
3336984994638,5cyclictest3337722-21kworker/2:120:30:422
333698099466,10cyclictest23-21ksoftirqd/119:55:011
333698099465,6cyclictest23-21ksoftirqd/119:25:011
333698099462,8cyclictest23-21ksoftirqd/120:55:031
3336976994639,4cyclictest16950irq/224-800000019:20:010
3336976994630,13cyclictest275-21dbus-daemon20:55:010
333698999459,5cyclictest131rcu_preempt00:01:283
3336989994534,8cyclictest3342878-21sshd21:03:363
3336989994525,15cyclictest275-21dbus-daemon19:35:013
3336989994524,15cyclictest275-21dbus-daemon20:40:023
333698999451,6cyclictest3345339-21munin-run21:15:013
333698999451,5cyclictest131rcu_preempt23:00:113
333698999451,5cyclictest131rcu_preempt23:00:113
333698999451,5cyclictest131rcu_preempt22:15:023
333698099458,5cyclictest131rcu_preempt19:45:011
333698099458,5cyclictest131rcu_preempt19:45:011
333698099456,6cyclictest23-21ksoftirqd/119:10:011
333698099455,10cyclictest23-21ksoftirqd/119:20:021
3336980994512,5cyclictest131rcu_preempt19:50:001
3336976994517,24cyclictest12-21ksoftirqd/020:25:010
333698999445,5cyclictest131rcu_preempt21:21:063
333698999443,3cyclictest131rcu_preempt23:34:513
3336989994432,9cyclictest1-21systemd20:25:023
3336989994427,11cyclictest3376702-21grep22:25:003
3336989994426,13cyclictest275-21dbus-daemon21:50:013
333697699443,35cyclictest3467602-21systemd-journal20:10:000
3336976994425,15cyclictest3340260-21sshd20:12:190
3336976994425,15cyclictest3340260-21sshd20:12:190
333697699442,3cyclictest131rcu_preempt20:45:020
333698999439,7cyclictest131rcu_preempt23:12:203
333698999435,4cyclictest131rcu_preempt22:45:003
333698999433,3cyclictest131rcu_preempt21:00:023
3336989994332,9cyclictest3409612-21sshd23:38:363
3336989994332,9cyclictest3409612-21sshd23:38:353
333698999432,3cyclictest131rcu_preempt19:20:003
333697699438,4cyclictest131rcu_preempt20:20:000
3336976994331,9cyclictest1-21systemd19:50:000
333698999429,6cyclictest131rcu_preempt23:54:423
333698999429,4cyclictest131rcu_preempt23:09:503
333698999426,3cyclictest131rcu_preempt21:42:053
333698999424,3cyclictest131rcu_preempt21:40:013
333698999423,6cyclictest3381218-21cat22:35:013
3336989994231,8cyclictest37-21ksoftirqd/321:54:553
333698999421,3cyclictest131rcu_preempt00:35:023
333698999421,3cyclictest131rcu_preempt00:07:543
3336980994211,4cyclictest131rcu_preempt21:05:001
333698999419,5cyclictest0-21swapper/321:31:363
333698999418,5cyclictest131rcu_preempt22:46:363
333698999417,5cyclictest131rcu_preempt23:49:023
333698999417,3cyclictest131rcu_preempt20:20:013
333698999415,6cyclictest131rcu_preempt19:15:033
333698999413,6cyclictest131rcu_preempt22:07:323
3336989994129,10cyclictest37-21ksoftirqd/323:40:443
333698999412,4cyclictest131rcu_preempt23:23:153
333698999411,4cyclictest131rcu_preempt20:30:003
3336976994127,11cyclictest3338722-21sshd19:42:350
3336976994127,11cyclictest3338722-21sshd19:42:350
333697699411,37cyclictest3341616-21modprobe20:40:000
333698999403,3cyclictest131rcu_preempt20:15:023
333698999403,3cyclictest131rcu_preempt20:15:023
3336989994030,8cyclictest37-21ksoftirqd/322:03:293
3336989994028,9cyclictest37-21ksoftirqd/320:35:013
3336989994026,10cyclictest275-21dbus-daemon19:30:023
333698999402,5cyclictest131rcu_preempt20:55:023
333698999402,3cyclictest131rcu_preempt22:53:003
333698999401,5cyclictest0-21swapper/322:17:343
333698999401,4cyclictest131rcu_preempt00:22:283
3336976994026,9cyclictest3467602-21systemd-journal21:01:350
333697699402,33cyclictest3339525-21sendmail20:00:000
333697699402,33cyclictest3339525-21sendmail20:00:000
3336989993920,14cyclictest3467602-21systemd-journal19:40:033
3336989993920,14cyclictest3467602-21systemd-journal19:40:023
333698099394,8cyclictest23-21ksoftirqd/120:35:001
333697699393,4cyclictest131rcu_preempt19:10:010
3336976993932,4cyclictest16950irq/224-800000020:00:350
333697699392,7cyclictest3338475-21munin-run19:40:000
333698999384,3cyclictest131rcu_preempt20:00:013
333698999384,3cyclictest131rcu_preempt20:00:013
333698999382,4cyclictest131rcu_preempt23:26:363
333698999382,4cyclictest131rcu_preempt23:26:363
333697699385,3cyclictest131rcu_preempt19:30:010
333697699382,4cyclictest131rcu_preempt21:00:020
333698999377,3cyclictest131rcu_preempt19:50:003
333698999373,3cyclictest131rcu_preempt20:05:003
333698999373,3cyclictest131rcu_preempt19:20:353
3336989993727,7cyclictest3338432-21sshd19:35:353
333698999371,3cyclictest131rcu_preempt21:10:003
333698999371,3cyclictest131rcu_preempt21:10:003
333698999371,3cyclictest131rcu_preempt20:50:013
333697699375,3cyclictest131rcu_preempt19:15:010
333697699372,30cyclictest364-21snmpd19:31:450
333697699371,3cyclictest131rcu_preempt20:50:010
3336976993527,5cyclictest0-21swapper/019:53:050
3336976993520,12cyclictest12-21ksoftirqd/020:30:010
333698999314,3cyclictest131rcu_preempt19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional