You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 06:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed Mar 11, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1450998998854,17cyclictest1456948-21kworker/u8:021:40:020
145099899711,37cyclictest131rcu_preempt00:36:300
145099899711,37cyclictest131rcu_preempt00:36:300
1451002996829,9cyclictest23-21ksoftirqd/122:25:461
145099899661,32cyclictest131rcu_preempt22:00:030
1450998996415,40cyclictest12-21ksoftirqd/020:20:020
145100299637,41cyclictest3467718-21lldpd22:55:031
145100299637,41cyclictest3467718-21lldpd22:55:021
1450998996329,13cyclictest131rcu_preempt23:25:220
1450998996329,13cyclictest131rcu_preempt23:25:220
145099899631,26cyclictest131rcu_preempt23:15:010
1450998996241,18cyclictest16850irq/134-800000000:05:010
145099899617,17cyclictest1533187-40mandb00:00:000
145099899615,36cyclictest131rcu_preempt22:54:510
1450998996153,4cyclictest16950irq/224-800000023:55:010
145099899615,19cyclictest284-21systemd-logind21:20:000
1450998996139,18cyclictest16850irq/134-800000021:10:000
1450998996119,4cyclictest131rcu_preempt22:59:110
1450998996119,4cyclictest131rcu_preempt22:59:110
145100299606,41cyclictest1495425-21ssh22:35:021
1450998996026,17cyclictest1482112-21ssh22:05:090
1451006995943,7cyclictest1531257-21kworker/2:000:01:562
1450998995919,6cyclictest16850irq/134-800000000:05:360
1450998995819,6cyclictest16850irq/134-800000023:21:250
1450998995818,32cyclictest16850irq/134-800000021:27:120
1450998995816,32cyclictest16850irq/134-800000022:15:590
1451002995740,13cyclictest3467602-21systemd-journal21:45:181
145099899573,3cyclictest131rcu_preempt20:25:000
145099899572,3cyclictest131rcu_preempt21:34:490
1450998995719,6cyclictest16850irq/134-800000021:41:030
1450998995719,30cyclictest16850irq/134-800000022:13:140
1450998995718,34cyclictest16850irq/134-800000023:46:240
1450998995718,31cyclictest16850irq/134-800000000:18:110
1450998995717,36cyclictest16850irq/134-800000022:28:010
1450998995717,31cyclictest16850irq/134-800000023:35:210
1450998995717,31cyclictest16850irq/134-800000022:35:330
1450998995716,33cyclictest16850irq/134-800000022:22:500
1450998995716,32cyclictest16850irq/134-800000000:34:070
1451006995630,7cyclictest1488551-21kworker/2:122:46:002
1451002995635,10cyclictest23-21ksoftirqd/121:10:001
1450998995619,33cyclictest16850irq/134-800000023:40:490
1450998995618,34cyclictest16850irq/134-800000021:46:560
1450998995618,33cyclictest16850irq/134-800000023:30:380
1450998995617,31cyclictest16850irq/134-800000023:16:270
1450998995617,31cyclictest16850irq/134-800000023:16:270
1450998995617,31cyclictest16850irq/134-800000021:24:120
1450998995617,31cyclictest16850irq/134-800000021:24:110
1450998995616,32cyclictest16850irq/134-800000000:20:230
1450998995616,31cyclictest16850irq/134-800000022:44:490
1450998995616,31cyclictest16850irq/134-800000022:00:490
1450998995616,31cyclictest16850irq/134-800000000:28:080
1450998995517,34cyclictest16850irq/134-800000022:30:540
1450998995517,34cyclictest16850irq/134-800000000:14:150
1450998995516,31cyclictest16850irq/134-800000022:48:550
1450998995512,23cyclictest12-21ksoftirqd/021:55:020
145100699543,6cyclictest30-21ksoftirqd/219:40:022
145100699543,6cyclictest30-21ksoftirqd/219:40:022
1451006995436,12cyclictest1-21systemd23:30:002
1451006995436,12cyclictest1-21systemd23:30:002
1451006995434,7cyclictest1488551-21kworker/2:123:24:572
1451006995431,8cyclictest1488551-21kworker/2:122:31:102
1450998995418,6cyclictest16850irq/134-800000021:13:590
145101099531,4cyclictest131rcu_preempt22:30:023
145100299531,32cyclictest23-21ksoftirqd/121:40:041
1450998995317,7cyclictest16850irq/134-800000023:04:150
1451006995244,5cyclictest1531257-21kworker/2:000:28:092
1451006995243,6cyclictest1531257-21kworker/2:000:33:272
1451006995232,7cyclictest1488551-21kworker/2:123:50:242
1451006995143,5cyclictest1531257-21kworker/2:000:06:582
1451006995143,5cyclictest1488551-21kworker/2:123:14:372
1451006995143,5cyclictest1488551-21kworker/2:122:28:192
1451006995142,5cyclictest1488551-21kworker/2:123:00:022
1451006995142,5cyclictest1488551-21kworker/2:123:00:022
1451006995124,9cyclictest1455766-21date20:40:022
145099899513,6cyclictest1452532-21munin-run19:40:000
145099899513,6cyclictest1452532-21munin-run19:40:000
1450998995112,31cyclictest16850irq/134-800000023:06:500
1450998995112,31cyclictest16850irq/134-800000023:06:500
1451010995032,14cyclictest275-21dbus-daemon20:30:013
145100699504,4cyclictest131rcu_preempt00:00:002
1451006995042,5cyclictest1488551-21kworker/2:123:33:142
1451006995042,5cyclictest1450405-21kworker/2:320:05:012
1451006995042,5cyclictest1450405-21kworker/2:320:05:012
1451006995041,5cyclictest1450405-21kworker/2:321:46:022
1451006995041,5cyclictest1450405-21kworker/2:321:42:182
1451006995023,9cyclictest1450405-21kworker/2:321:34:562
1451002995032,13cyclictest3467602-21systemd-journal23:47:181
1451010994922,9cyclictest37-21ksoftirqd/300:01:353
1451010994918,11cyclictest0-21swapper/322:20:153
145100699494,3cyclictest131rcu_preempt23:20:012
145100699494,3cyclictest131rcu_preempt23:20:002
1451006994941,5cyclictest1488551-21kworker/2:123:35:052
1451006994941,5cyclictest1471147-21kworker/2:022:20:432
1451006994941,5cyclictest1471147-21kworker/2:022:14:552
1451006994940,6cyclictest1531257-21kworker/2:000:24:082
1451006994940,6cyclictest1450405-21kworker/2:321:39:292
1451006994936,8cyclictest1-21systemd19:30:012
145100699493,3cyclictest131rcu_preempt22:20:012
1451006994923,10cyclictest1450405-21kworker/2:321:00:032
145100299495,4cyclictest131rcu_preempt22:40:541
1451002994936,10cyclictest1-21systemd19:15:001
1451006994840,5cyclictest1531257-21kworker/2:000:10:442
1451006994840,5cyclictest1488551-21kworker/2:123:04:422
1451006994840,5cyclictest1488551-21kworker/2:122:54:082
1451006994840,5cyclictest1488551-21kworker/2:122:43:152
1451006994840,5cyclictest1450405-21kworker/2:321:15:002
1451006994840,5cyclictest1450405-21kworker/2:320:13:172
1451006994836,8cyclictest1-21systemd20:50:012
1451006994836,8cyclictest1-21systemd19:55:012
1451002994835,11cyclictest23-21ksoftirqd/122:10:041
1451002994831,12cyclictest23-21ksoftirqd/119:40:011
1451002994831,12cyclictest23-21ksoftirqd/119:40:011
145100699475,4cyclictest131rcu_preempt00:40:012
145100699475,4cyclictest131rcu_preempt00:40:002
1451006994739,5cyclictest1471147-21kworker/2:022:05:102
1451006994739,5cyclictest1471147-21kworker/2:022:03:552
1451006994739,5cyclictest1450405-21kworker/2:321:59:032
1451006994739,5cyclictest1450405-21kworker/2:321:24:142
1451006994739,5cyclictest1450405-21kworker/2:321:24:142
1451006994733,10cyclictest3467602-21systemd-journal20:55:012
145100699472,3cyclictest131rcu_preempt20:20:022
145100699472,3cyclictest131rcu_preempt20:20:022
1451006994721,5cyclictest30-21ksoftirqd/223:50:012
145100699471,3cyclictest131rcu_preempt21:10:012
145100299473,27cyclictest0-21swapper/120:25:001
145099899473,10cyclictest1453570-21munin-run20:00:000
145101099466,35cyclictest1502663-21sshd22:51:173
1451010994636,5cyclictest309-21in:imuxsock22:45:013
1451010994634,9cyclictest37-21ksoftirqd/300:40:033
1451010994634,9cyclictest37-21ksoftirqd/300:40:023
145100699465,4cyclictest131rcu_preempt22:35:172
145100699464,4cyclictest131rcu_preempt00:15:092
1451006994637,6cyclictest1450405-21kworker/2:321:25:452
145100699463,3cyclictest131rcu_preempt23:40:502
145100699463,3cyclictest131rcu_preempt21:20:002
1451006994631,12cyclictest1-21systemd23:10:002
1451006994631,12cyclictest1-21systemd23:10:002
1451006994620,10cyclictest1450405-21kworker/2:321:54:412
145100299464,3cyclictest131rcu_preempt21:55:261
1451002994620,16cyclictest23-21ksoftirqd/100:04:551
145099899462,7cyclictest131rcu_preempt19:10:070
145099899461,12cyclictest1451726-21cut19:20:030
1451010994531,11cyclictest1515270-21cron23:20:003
1451010994531,11cyclictest1515270-21cron23:20:003
145100699451,3cyclictest131rcu_preempt19:20:022
1451002994531,9cyclictest275-21dbus-daemon19:55:021
1451002994528,13cyclictest0-21swapper/100:00:011
145099899453,7cyclictest12-21ksoftirqd/019:50:020
1451010994432,9cyclictest309-21in:imuxsock19:35:013
1451010994428,13cyclictest37-21ksoftirqd/323:05:073
1451010994428,13cyclictest37-21ksoftirqd/323:05:063
1451010994428,11cyclictest1451494-21cron19:20:003
145101099441,11cyclictest1452535-21systemctl19:40:003
145101099441,11cyclictest1452535-21systemctl19:40:003
145100699444,3cyclictest131rcu_preempt19:45:002
145100699442,3cyclictest131rcu_preempt20:45:012
145100699441,4cyclictest131rcu_preempt19:50:012
145100699441,4cyclictest131rcu_preempt19:25:002
145100299448,8cyclictest23-21ksoftirqd/121:12:061
1450998994428,12cyclictest1452243-21sshd19:30:430
145101099437,6cyclictest131rcu_preempt23:25:013
145101099436,10cyclictest37-21ksoftirqd/320:37:343
145101099434,4cyclictest131rcu_preempt21:28:103
1451010994329,11cyclictest275-21dbus-daemon20:55:483
1451010994328,11cyclictest1453034-21sshd19:48:183
145101099432,4cyclictest131rcu_preempt22:09:483
145101099432,3cyclictest131rcu_preempt00:19:123
145101099432,3cyclictest131rcu_preempt00:00:003
1451010994321,8cyclictest37-21ksoftirqd/321:24:133
1451010994321,8cyclictest37-21ksoftirqd/321:24:133
1451006994330,9cyclictest3467602-21systemd-journal20:10:002
145100299431,4cyclictest131rcu_preempt22:12:301
1450998994336,4cyclictest16950irq/224-800000020:53:180
1451010994227,11cyclictest1-21systemd21:35:013
1451010994226,11cyclictest37-21ksoftirqd/322:31:573
145101099422,3cyclictest131rcu_preempt23:47:423
145101099422,3cyclictest131rcu_preempt21:20:003
145101099421,4cyclictest131rcu_preempt22:14:123
145101099421,3cyclictest131rcu_preempt23:37:363
145100699429,6cyclictest131rcu_preempt21:05:022
145100699429,6cyclictest131rcu_preempt21:05:022
145100699425,3cyclictest131rcu_preempt19:10:012
1451006994215,9cyclictest1450405-21kworker/2:319:35:052
145100699421,4cyclictest131rcu_preempt20:30:002
1451002994227,11cyclictest1456666-21cron21:00:001
1451002994225,13cyclictest23-21ksoftirqd/120:00:001
145100299422,3cyclictest131rcu_preempt00:30:001
1451002994214,17cyclictest23-21ksoftirqd/100:11:121
145100299421,3cyclictest131rcu_preempt23:33:391
1450998994221,5cyclictest12-21ksoftirqd/019:15:010
145099899421,37cyclictest0-21swapper/020:40:020
145101099417,5cyclictest131rcu_preempt23:05:013
145101099416,6cyclictest131rcu_preempt23:45:023
145101099414,3cyclictest131rcu_preempt21:55:013
145101099413,4cyclictest131rcu_preempt23:31:183
145101099413,4cyclictest131rcu_preempt22:36:073
1451010994129,9cyclictest37-21ksoftirqd/300:23:133
1451010994129,10cyclictest37-21ksoftirqd/320:00:003
1451010994113,8cyclictest37-21ksoftirqd/321:39:563
145100699411,3cyclictest131rcu_preempt20:35:022
145100699411,3cyclictest131rcu_preempt19:15:002
1451002994124,13cyclictest1477370-21sed21:55:021
145099899415,3cyclictest131rcu_preempt20:35:010
1450998994134,4cyclictest16950irq/224-800000019:45:000
1450998994134,4cyclictest16950irq/224-800000019:23:180
1450998994126,11cyclictest275-21dbus-daemon21:05:010
1450998994126,11cyclictest275-21dbus-daemon21:05:010
145099899412,3cyclictest131rcu_preempt20:50:020
1450998994118,5cyclictest12-21ksoftirqd/019:30:010
145101099408,5cyclictest131rcu_preempt00:06:063
145101099405,3cyclictest131rcu_preempt21:46:113
145101099403,5cyclictest131rcu_preempt00:11:073
145101099403,4cyclictest131rcu_preempt22:47:113
1451010994033,4cyclictest1454341-21latency_hist20:15:003
1451010994029,8cyclictest1456943-21sshd21:04:193
1451010994029,8cyclictest1456943-21sshd21:04:183
1451010994029,6cyclictest309-21in:imuxsock00:35:013
1451010994028,9cyclictest37-21ksoftirqd/322:02:233
1451010994028,9cyclictest1452787-21sshd19:44:183
1451010994026,10cyclictest37-21ksoftirqd/323:13:203
145101099402,3cyclictest131rcu_preempt21:10:003
145101099401,5cyclictest131rcu_preempt20:10:013
1451010994013,5cyclictest37-21ksoftirqd/323:26:253
1451010994013,5cyclictest37-21ksoftirqd/323:26:243
145100299405,8cyclictest23-21ksoftirqd/123:39:511
145100299402,6cyclictest1486990-21modprobe22:16:181
1451002994026,11cyclictest23-21ksoftirqd/123:52:251
1451002994026,11cyclictest23-21ksoftirqd/100:08:501
1451002994025,11cyclictest23-21ksoftirqd/123:21:181
1451002994024,12cyclictest23-21ksoftirqd/100:17:021
145100299402,35cyclictest1468117-21sshd21:34:181
1451002994017,14cyclictest1517984-21sh23:25:341
1451002994017,14cyclictest1517984-21sh23:25:341
1451002994016,13cyclictest23-21ksoftirqd/120:20:001
1451002994016,13cyclictest23-21ksoftirqd/120:20:001
145100299401,3cyclictest131rcu_preempt21:05:011
145100299401,3cyclictest131rcu_preempt21:05:011
1451002994011,5cyclictest131rcu_preempt23:00:051
145100299401,12cyclictest1544057-21ssh00:24:171
1450998994033,4cyclictest16950irq/224-800000019:55:000
145099899402,6cyclictest1454622-21readlink20:20:000
1450998994020,16cyclictest275-21dbus-daemon20:05:010
1450998994020,16cyclictest275-21dbus-daemon20:05:010
145101099396,9cyclictest37-21ksoftirqd/321:14:313
145101099395,4cyclictest131rcu_preempt23:53:443
145101099395,4cyclictest131rcu_preempt21:58:503
145101099394,4cyclictest131rcu_preempt22:17:063
145101099394,3cyclictest131rcu_preempt22:58:443
145101099394,3cyclictest131rcu_preempt22:58:443
1451010993924,11cyclictest1472910-21modprobe21:45:003
145101099391,3cyclictest131rcu_preempt20:25:013
1451002993929,8cyclictest23-21ksoftirqd/120:50:001
1451002993926,9cyclictest1490950-21cut22:25:021
1451002993926,10cyclictest1550456-21sshd00:38:371
1451002993926,10cyclictest1550456-21sshd00:38:361
1451002993924,11cyclictest23-21ksoftirqd/121:24:401
1451002993924,11cyclictest23-21ksoftirqd/121:24:391
145100299392,3cyclictest131rcu_preempt21:20:021
1451002993920,15cyclictest275-21dbus-daemon23:15:001
145099899392,5cyclictest1454345-21munin-run20:15:010
145101099386,5cyclictest131rcu_preempt00:30:003
145101099383,3cyclictest131rcu_preempt19:15:013
1451010993827,6cyclictest37-21ksoftirqd/320:55:003
1451010993824,9cyclictest37-21ksoftirqd/320:20:003
1451010993824,9cyclictest37-21ksoftirqd/320:20:003
1451010993824,8cyclictest1455391-21munin-plugin-st20:35:003
1451010993821,13cyclictest3467602-21systemd-journal19:30:003
1451006993830,5cyclictest1450405-21kworker/2:320:21:452
1451006993823,11cyclictest3467602-21systemd-journal20:00:002
145100299386,6cyclictest131rcu_preempt23:40:341
1451002993824,11cyclictest23-21ksoftirqd/122:02:151
145100299381,8cyclictest1453022-21kworker/u8:119:50:021
1451002993816,10cyclictest23-21ksoftirqd/122:34:541
1451002993815,15cyclictest23-21ksoftirqd/120:40:011
1451002993813,17cyclictest23-21ksoftirqd/100:32:111
1451010993727,8cyclictest37-21ksoftirqd/320:05:023
1451010993727,8cyclictest37-21ksoftirqd/320:05:023
145101099371,3cyclictest131rcu_preempt20:44:183
145101099371,3cyclictest131rcu_preempt19:55:023
145100299372,8cyclictest23-21ksoftirqd/122:54:011
1451002993728,7cyclictest23-21ksoftirqd/123:18:241
1451002993728,7cyclictest23-21ksoftirqd/123:18:231
145099899374,3cyclictest131rcu_preempt20:09:180
1450998993729,5cyclictest1456684-21sendmail21:00:000
1450998993728,6cyclictest19350irq/34-mmc120:25:360
1451010993624,9cyclictest1456264-21cat20:50:023
145100299363,6cyclictest131rcu_preempt20:55:011
1451002993627,6cyclictest1510038-21sh23:08:011
1451002993627,6cyclictest1510038-21sh23:08:011
1451002993623,10cyclictest23-21ksoftirqd/120:15:001
1451002993623,10cyclictest120-21kswapd022:50:011
1451002993612,16cyclictest0-21swapper/121:45:011
145099899367,21cyclictest1455944-21latency_hist20:45:010
145101099353,3cyclictest131rcu_preempt19:25:003
145100299353,4cyclictest131rcu_preempt20:30:011
1451002993524,9cyclictest23-21ksoftirqd/120:45:011
1451002993521,11cyclictest23-21ksoftirqd/121:27:511
1451002993422,8cyclictest23-21ksoftirqd/119:35:021
145100299342,11cyclictest284-21systemd-logind19:30:011
1451002993421,10cyclictest23-21ksoftirqd/119:10:021
145101099332,4cyclictest131rcu_preempt19:10:023
145100299333,3cyclictest131rcu_preempt19:45:011
145100299332,3cyclictest131rcu_preempt19:25:011
145100299332,28cyclictest23-21ksoftirqd/120:33:181
1451002993320,10cyclictest1-21systemd19:20:011
1451002993314,6cyclictest131rcu_preempt20:05:511
145100299322,5cyclictest364-21snmpd20:00:501
145100299322,5cyclictest364-21snmpd20:00:501
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional