You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-16 - 12:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Jul 16, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
39559029910657,35cyclictest0-21swapper/123:10:021
395590299985,60cyclictest4043259-21sh00:00:031
395590299854,57cyclictest4040752-21sh23:55:021
395590299767,24cyclictest4056828-21ssh00:28:471
3955902996625,10cyclictest3467602-21systemd-journal22:29:011
3955902996625,10cyclictest3467602-21systemd-journal22:29:011
3955907996540,18cyclictest1-21systemd23:59:592
395589899658,5cyclictest131rcu_preempt00:00:400
3955898996554,6cyclictest16950irq/224-800000020:20:000
3955909996417,9cyclictest37-21ksoftirqd/300:23:323
3955902996428,4cyclictest3990769-21latency_hist22:10:021
3955902996425,10cyclictest23-21ksoftirqd/121:40:001
3955902996324,7cyclictest3956309-21sshd19:15:441
395589899631,46cyclictest3994307-21kworker/u8:122:50:020
395589899631,46cyclictest3994307-21kworker/u8:122:50:020
3955909996213,8cyclictest37-21ksoftirqd/300:10:013
3955898996216,40cyclictest131rcu_preempt23:59:510
395590999618,15cyclictest4002748-21sed22:35:013
3955902996126,11cyclictest23-21ksoftirqd/123:25:051
3955898996123,22cyclictest0-21swapper/019:10:010
3955898996119,36cyclictest16850irq/134-800000023:30:230
395589899611,4cyclictest131rcu_preempt21:59:360
3955902996024,10cyclictest23-21ksoftirqd/121:40:251
3955902996024,10cyclictest23-21ksoftirqd/121:40:241
3955898996038,17cyclictest131rcu_preempt00:09:460
3955898996028,4cyclictest39754421ssh21:37:440
3955909995917,9cyclictest37-21ksoftirqd/300:30:013
3955909995917,11cyclictest37-21ksoftirqd/323:17:413
3955909995912,9cyclictest37-21ksoftirqd/322:55:003
3955902995925,12cyclictest4003088-21ssh22:35:261
3955902995925,12cyclictest4003088-21ssh22:35:261
3955898995929,15cyclictest3992174-21ssh22:12:410
3955898995919,35cyclictest16850irq/134-800000022:53:410
3955909995815,9cyclictest37-21ksoftirqd/319:40:003
3955909995815,9cyclictest37-21ksoftirqd/319:40:003
3955909995812,8cyclictest37-21ksoftirqd/321:36:143
3955898995819,34cyclictest12-21ksoftirqd/022:05:200
3955909995712,11cyclictest37-21ksoftirqd/323:50:093
3955909995710,9cyclictest37-21ksoftirqd/323:00:003
3955909995710,9cyclictest37-21ksoftirqd/323:00:003
3955909995710,12cyclictest37-21ksoftirqd/323:34:443
3955902995727,4cyclictest4000874-21sh22:30:501
3955898995734,4cyclictest12-21ksoftirqd/022:35:160
3955898995734,4cyclictest12-21ksoftirqd/022:35:160
3955898995728,13cyclictest4040097-21ssh23:53:550
395589899572,39cyclictest4018109-21ssh23:07:120
3955898995723,30cyclictest12-21ksoftirqd/023:01:280
3955909995615,8cyclictest37-21ksoftirqd/321:05:003
3955909995615,10cyclictest37-21ksoftirqd/323:25:153
3955907995647,6cyclictest3962112-21kworker/2:321:41:332
3955907995647,6cyclictest3962112-21kworker/2:321:41:332
3955902995623,11cyclictest23-21ksoftirqd/122:25:021
3955898995629,12cyclictest3973077-21ssh21:32:520
3955898995628,13cyclictest4025949-21ssh23:23:570
3955898995628,13cyclictest4021105-21ssh23:13:400
3955898995627,14cyclictest4036586-21ssh23:45:530
3955898995619,32cyclictest16850irq/134-800000021:11:430
395589899561,40cyclictest40234361ssh23:18:370
3955909995514,8cyclictest37-21ksoftirqd/322:20:023
3955909995514,8cyclictest37-21ksoftirqd/322:20:013
3955909995511,11cyclictest37-21ksoftirqd/321:35:023
3955907995547,5cyclictest3962112-21kworker/2:322:33:482
3955907995545,6cyclictest3962112-21kworker/2:322:57:432
3955907995545,6cyclictest3962112-21kworker/2:322:57:432
395590299557,37cyclictest4059911-21sh00:35:011
395589899559,5cyclictest12-21ksoftirqd/000:19:250
3955898995534,15cyclictest3467602-21systemd-journal21:00:000
3955898995529,11cyclictest39710141ssh21:28:350
3955898995519,7cyclictest16850irq/134-800000022:20:570
3955898995519,6cyclictest16850irq/134-800000000:35:160
3955898995519,32cyclictest16850irq/134-800000021:54:230
3955898995519,32cyclictest16850irq/134-800000021:54:230
3955898995519,31cyclictest16850irq/134-800000000:30:180
3955898995519,18cyclictest16850irq/134-800000022:30:010
3955898995519,18cyclictest16850irq/134-800000022:30:010
395590999549,10cyclictest37-21ksoftirqd/300:40:013
395590999547,10cyclictest37-21ksoftirqd/300:00:023
395590999543,9cyclictest37-21ksoftirqd/323:15:003
3955909995438,12cyclictest275-21dbus-daemon22:30:023
3955909995438,12cyclictest275-21dbus-daemon22:30:013
3955909995424,6cyclictest37-21ksoftirqd/323:44:433
3955909995424,6cyclictest37-21ksoftirqd/323:44:423
3955909995422,7cyclictest37-21ksoftirqd/300:34:273
3955909995411,8cyclictest37-21ksoftirqd/322:15:013
3955902995426,11cyclictest4007490-21systemd-run22:45:001
3955898995428,11cyclictest4027819-21ssh23:27:400
3955898995418,32cyclictest16850irq/134-800000023:44:360
3955898995418,32cyclictest16850irq/134-800000023:44:350
3955898995418,32cyclictest16850irq/134-800000022:31:210
3955898995418,32cyclictest16850irq/134-800000000:22:430
3955898995418,31cyclictest16850irq/134-800000021:23:490
3955898995417,32cyclictest16850irq/134-800000023:35:220
3955898995417,32cyclictest16850irq/134-800000000:13:300
3955898995416,33cyclictest16850irq/134-800000022:57:480
3955898995416,33cyclictest16850irq/134-800000022:57:470
395590999539,11cyclictest3978982-21latency_hist21:45:023
395590999539,11cyclictest3978982-21latency_hist21:45:023
3955909995317,7cyclictest4005870-21rm22:41:233
3955909995314,8cyclictest37-21ksoftirqd/321:50:003
3955909995311,9cyclictest0-21swapper/300:15:533
3955907995345,5cyclictest3962112-21kworker/2:322:44:162
3955907995345,5cyclictest3962112-21kworker/2:300:03:232
3955907995344,6cyclictest3955092-21kworker/2:121:26:232
3955907995344,5cyclictest3962112-21kworker/2:322:14:112
3955907995344,5cyclictest3955092-21kworker/2:121:51:542
3955907995344,5cyclictest3955092-21kworker/2:121:51:542
3955907995343,6cyclictest3962112-21kworker/2:321:37:592
3955902995334,12cyclictest3467602-21systemd-journal19:35:001
395589899536,41cyclictest3956808-21cat19:25:020
3955898995335,14cyclictest3467602-21systemd-journal20:50:000
3955898995318,6cyclictest16850irq/134-800000022:00:590
3955898995318,32cyclictest16850irq/134-800000021:44:500
3955898995318,32cyclictest16850irq/134-800000021:44:490
3955898995318,31cyclictest16850irq/134-800000022:42:130
3955898995317,6cyclictest16850irq/134-800000000:25:430
3955898995317,32cyclictest16850irq/134-800000021:46:540
3955898995317,32cyclictest16850irq/134-800000021:15:390
395590999528,7cyclictest37-21ksoftirqd/320:40:013
395590999527,8cyclictest3988021-21ssh22:04:143
3955909995213,28cyclictest4033617-21modprobe23:40:003
3955909995212,7cyclictest37-21ksoftirqd/322:35:173
3955909995212,7cyclictest37-21ksoftirqd/322:35:163
3955907995244,5cyclictest3988959-21kworker/2:023:49:032
3955907995244,5cyclictest3988959-21kworker/2:023:42:352
3955907995244,5cyclictest3988959-21kworker/2:023:42:342
3955907995244,5cyclictest3988959-21kworker/2:023:25:472
3955907995244,5cyclictest3962112-21kworker/2:322:46:572
3955907995244,5cyclictest3962112-21kworker/2:322:46:562
3955907995244,5cyclictest3962112-21kworker/2:300:06:372
3955907995244,5cyclictest3955092-21kworker/2:121:17:592
3955907995243,6cyclictest4045436-21kworker/2:200:16:512
3955907995243,6cyclictest3988959-21kworker/2:023:38:082
3955907995242,7cyclictest4045436-21kworker/2:200:33:312
395590999516,7cyclictest37-21ksoftirqd/321:56:513
395590999515,10cyclictest37-21ksoftirqd/321:19:513
395590999514,8cyclictest37-21ksoftirqd/321:25:013
395590999514,6cyclictest37-21ksoftirqd/321:53:333
395590999514,6cyclictest37-21ksoftirqd/321:53:323
3955907995143,5cyclictest4045436-21kworker/2:200:25:272
3955907995143,5cyclictest4045436-21kworker/2:200:22:502
3955907995143,5cyclictest3962112-21kworker/2:322:05:492
3955907995143,5cyclictest3962112-21kworker/2:321:57:062
3955907995143,5cyclictest3955092-21kworker/2:121:33:352
3955907995142,6cyclictest3955092-21kworker/2:121:10:502
3955907995142,5cyclictest3988959-21kworker/2:023:52:542
3955907995142,5cyclictest3962112-21kworker/2:322:01:172
3955907995141,6cyclictest3988959-21kworker/2:023:24:082
395590299519,33cyclictest3984561-1kworker/u9:222:00:011
3955898995116,31cyclictest16850irq/134-800000022:15:110
3955898995116,31cyclictest16850irq/134-800000022:15:110
395590999507,9cyclictest37-21ksoftirqd/323:04:423
3955909995016,8cyclictest37-21ksoftirqd/323:24:013
3955909995012,9cyclictest0-21swapper/322:21:063
3955907995042,5cyclictest3988959-21kworker/2:023:32:142
3955907995042,5cyclictest3988959-21kworker/2:023:17:152
3955907995042,5cyclictest3988959-21kworker/2:022:18:032
3955907995042,5cyclictest3988959-21kworker/2:022:18:032
3955907995042,5cyclictest3962112-21kworker/2:323:10:212
3955907995042,5cyclictest3962112-21kworker/2:323:08:132
3955907995042,5cyclictest3962112-21kworker/2:322:38:552
3955907995042,5cyclictest3962112-21kworker/2:322:38:542
3955907995042,5cyclictest3955092-21kworker/2:121:48:372
3955907995041,6cyclictest3962112-21kworker/2:300:11:592
3955907995039,6cyclictest3962112-21kworker/2:322:29:042
3955907995039,6cyclictest3962112-21kworker/2:322:29:042
3955902995043,3cyclictest23-21ksoftirqd/123:45:031
3955902995034,12cyclictest275-21dbus-daemon21:25:021
3955902995029,14cyclictest3467602-21systemd-journal21:53:411
3955902995029,14cyclictest3467602-21systemd-journal21:53:411
395589899502,14cyclictest3957394-21munin-plugin-st19:40:010
395589899502,14cyclictest3957394-21munin-plugin-st19:40:000
395590999498,15cyclictest3988844-21ssh22:05:373
3955909994912,9cyclictest0-21swapper/321:11:353
3955907994941,5cyclictest4045436-21kworker/2:200:37:552
3955907994941,5cyclictest3988959-21kworker/2:022:21:222
3955907994941,5cyclictest3962112-21kworker/2:323:00:312
3955907994941,5cyclictest3962112-21kworker/2:322:52:052
3955907994941,5cyclictest3955092-21kworker/2:121:23:162
3955907994941,5cyclictest3955092-21kworker/2:119:51:412
395590999488,8cyclictest37-21ksoftirqd/320:05:013
395590999488,8cyclictest37-21ksoftirqd/320:05:013
3955909994833,11cyclictest1-21systemd20:20:013
3955909994825,18cyclictest3467602-21systemd-journal20:45:003
3955909994816,9cyclictest131rcu_preempt19:45:003
395590999481,12cyclictest3958068-21date19:50:023
3955907994840,5cyclictest3955092-21kworker/2:120:10:022
3955907994840,5cyclictest3955092-21kworker/2:119:35:382
3955907994840,5cyclictest3955092-21kworker/2:119:35:382
3955907994839,5cyclictest3955092-21kworker/2:119:15:022
395590299485,4cyclictest131rcu_preempt23:05:011
395589899489,4cyclictest131rcu_preempt21:01:050
395589899488,35cyclictest3956350-21sh19:20:010
3955898994828,14cyclictest3467602-21systemd-journal21:10:010
3955898994828,14cyclictest3467602-21systemd-journal21:10:010
395590999477,8cyclictest37-21ksoftirqd/321:10:033
395590999477,8cyclictest37-21ksoftirqd/321:10:023
395590999476,8cyclictest37-21ksoftirqd/320:55:013
395590999475,9cyclictest37-21ksoftirqd/320:15:013
395590999474,9cyclictest37-21ksoftirqd/319:15:013
3955909994710,7cyclictest0-21swapper/322:46:233
3955909994710,7cyclictest0-21swapper/322:46:233
3955907994739,5cyclictest3955092-21kworker/2:120:44:582
3955907994739,5cyclictest3955092-21kworker/2:120:25:342
395590299472,17cyclictest4050211-21rm00:15:001
395589899478,5cyclictest131rcu_preempt20:55:020
395589899475,4cyclictest131rcu_preempt20:45:020
3955898994730,12cyclictest3467602-21systemd-journal19:50:010
395590999465,9cyclictest37-21ksoftirqd/319:20:023
3955909994612,8cyclictest0-21swapper/300:14:043
3955909994612,7cyclictest0-21swapper/323:56:573
3955909994610,8cyclictest0-21swapper/323:48:493
3955907994639,4cyclictest3955092-21kworker/2:120:48:422
3955907994638,5cyclictest3955092-21kworker/2:119:26:172
3955902994626,15cyclictest4033635-21systemctl23:39:591
395590299462,23cyclictest221rcuc/100:00:021
3955898994630,11cyclictest275-21dbus-daemon20:15:010
395589899461,41cyclictest39586861sshd20:04:410
395589899461,41cyclictest39586861sshd20:04:410
395590999454,7cyclictest37-21ksoftirqd/323:09:403
395590999451,5cyclictest0-21swapper/319:55:013
3955909994513,7cyclictest0-21swapper/320:26:493
3955909994510,10cyclictest37-21ksoftirqd/321:26:193
3955907994538,4cyclictest3955092-21kworker/2:120:18:122
3955907994537,5cyclictest3955092-21kworker/2:120:37:442
3955907994537,5cyclictest3955092-21kworker/2:120:33:282
3955907994537,5cyclictest3955092-21kworker/2:120:20:072
3955907994537,5cyclictest3955092-21kworker/2:119:57:482
3955907994537,5cyclictest3955092-21kworker/2:119:22:452
3955907994537,5cyclictest3955092-21kworker/2:119:15:262
3955907994536,6cyclictest3955092-21kworker/2:120:51:412
3955902994531,11cyclictest23-21ksoftirqd/121:25:321
3955898994536,6cyclictest12-21ksoftirqd/020:25:010
3955909994411,8cyclictest0-21swapper/320:06:413
3955907994437,4cyclictest3955092-21kworker/2:120:12:222
3955907994436,5cyclictest3955092-21kworker/2:120:55:472
3955907994436,5cyclictest3955092-21kworker/2:119:34:462
3955902994431,10cyclictest23-21ksoftirqd/100:19:371
3955898994427,14cyclictest3956861-21sshd19:28:410
395590999433,11cyclictest37-21ksoftirqd/319:35:003
3955907994336,4cyclictest3955092-21kworker/2:121:07:352
3955907994336,4cyclictest3955092-21kworker/2:121:07:352
3955907994335,5cyclictest3955092-21kworker/2:119:48:482
3955902994331,8cyclictest23-21ksoftirqd/119:40:001
3955902994331,8cyclictest23-21ksoftirqd/119:40:001
3955898994329,11cyclictest3956128-21sed19:15:010
3955907994235,4cyclictest3955092-21kworker/2:120:04:372
3955907994235,4cyclictest3955092-21kworker/2:120:04:372
3955902994231,8cyclictest3960233-21sshd20:34:411
3955902994231,8cyclictest23-21ksoftirqd/100:39:111
3955902994227,12cyclictest23-21ksoftirqd/123:05:361
3955902994227,12cyclictest23-21ksoftirqd/122:20:021
3955902994227,12cyclictest23-21ksoftirqd/122:20:011
3955898994226,12cyclictest3960488-21cron20:40:000
3955898994224,13cyclictest275-21dbus-daemon19:45:000
395590999413,7cyclictest3959831-21date20:25:013
3955909994130,6cyclictest3961536-21sendmail21:00:003
3955898994128,10cyclictest3467602-21systemd-journal19:52:410
3955909994025,10cyclictest309-21in:imuxsock20:00:003
3955909994023,12cyclictest275-21dbus-daemon20:35:003
3955907994032,5cyclictest3955092-21kworker/2:119:44:382
395590299403,3cyclictest131rcu_preempt23:25:021
3955902994030,6cyclictest4054939-21cron00:24:591
3955902994025,11cyclictest309-21in:imuxsock21:35:011
3955902994025,10cyclictest1-21systemd20:25:001
395590299402,12cyclictest3986655-21ssh22:01:061
3955902994012,6cyclictest131rcu_preempt21:47:151
3955898994027,10cyclictest3467602-21systemd-journal20:05:540
3955902993928,7cyclictest4014520-21modprobe23:00:001
3955902993928,7cyclictest4014520-21modprobe23:00:001
3955902993927,10cyclictest23-21ksoftirqd/123:45:011
3955902993927,10cyclictest23-21ksoftirqd/123:45:011
395590299392,3cyclictest131rcu_preempt23:20:021
395590299391,4cyclictest131rcu_preempt21:10:021
395590299391,4cyclictest131rcu_preempt21:10:021
3955902993910,5cyclictest131rcu_preempt21:20:001
3955898993932,4cyclictest16950irq/224-800000020:00:010
3955898993918,18cyclictest12-21ksoftirqd/019:31:420
395590999385,4cyclictest131rcu_preempt19:30:023
3955909993821,12cyclictest37-21ksoftirqd/319:25:003
3955902993827,7cyclictest3956090-21munin-plugin-st19:15:001
3955902993826,8cyclictest3993175-21date22:15:021
3955902993823,10cyclictest309-21in:imuxsock19:46:411
3955902993811,6cyclictest131rcu_preempt21:13:261
395590799374,6cyclictest131rcu_preempt21:05:012
3955902993727,8cyclictest23-21ksoftirqd/120:20:011
3955902993726,7cyclictest3956868-21cron19:30:001
3955902993723,10cyclictest1-21systemd23:35:011
3955902993722,12cyclictest23-21ksoftirqd/122:50:391
3955902993721,11cyclictest309-21in:imuxsock19:45:011
395590299369,6cyclictest131rcu_preempt22:48:131
395590299369,6cyclictest131rcu_preempt22:48:121
395590299369,6cyclictest131rcu_preempt00:05:361
395590299363,4cyclictest131rcu_preempt20:50:011
3955898993626,7cyclictest12-21ksoftirqd/020:30:590
3955902993526,6cyclictest3959985-21cron20:30:001
3955902993525,7cyclictest3959034-21latency_hist20:10:011
3955902993524,8cyclictest3961934-21sed21:05:011
3955902993523,9cyclictest23-21ksoftirqd/120:45:011
3955898993528,4cyclictest3960001-21kworker/u8:120:30:000
395590299344,5cyclictest131rcu_preempt20:00:011
3955902993426,6cyclictest23-21ksoftirqd/120:55:011
3955902993424,6cyclictest3960518-21sendmail20:40:001
3955902993423,8cyclictest3958260-21sed19:55:031
3955902993423,8cyclictest23-21ksoftirqd/119:25:011
395590299328,4cyclictest131rcu_preempt20:57:521
395590299325,4cyclictest131rcu_preempt20:15:021
3955902993222,6cyclictest23-21ksoftirqd/119:10:021
3955902993118,10cyclictest3958736-21date20:05:021
3955902993118,10cyclictest3958736-21date20:05:021
3955909993020,7cyclictest37-21ksoftirqd/320:50:003
395590999283,20cyclictest3955939-21grep19:10:013
395590799274,19cyclictest3956017-21grep19:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional