You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-02 - 16:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Jun 02, 2026 12:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25434019910998,9cyclictest37-21ksoftirqd/312:20:033
2543401998534,45cyclictest2632826-21sort12:15:023
2543401998172,4cyclictest37-21ksoftirqd/310:20:033
254338699791,3cyclictest0-21swapper/007:15:020
2543401996724,12cyclictest37-21ksoftirqd/311:32:183
2543401996721,14cyclictest3467602-21systemd-journal09:55:013
2543401996426,11cyclictest37-21ksoftirqd/309:58:043
2543401996426,11cyclictest37-21ksoftirqd/309:58:033
254339699636,49cyclictest2567728-21sh09:50:032
2543386996149,10cyclictest2610276-21ssh11:25:020
254339699583,11cyclictest30-21ksoftirqd/210:39:222
2543396995824,15cyclictest30-21ksoftirqd/210:35:022
2543396995824,15cyclictest30-21ksoftirqd/210:35:022
2543386995824,30cyclictest12-21ksoftirqd/011:50:230
2543396995747,7cyclictest2546226-21kworker/2:109:17:192
2543391995746,8cyclictest2543393-21latency_hist07:10:011
2543386995752,2cyclictest0-21swapper/011:00:010
2543386995736,16cyclictest275-21dbus-daemon10:40:000
2543386995717,35cyclictest0-21swapper/008:20:010
2543386995717,35cyclictest0-21swapper/008:20:000
2543396995646,6cyclictest2596984-21kworker/2:011:32:422
2543386995626,14cyclictest2599317-21ssh11:00:140
2543386995626,14cyclictest2556124-21ssh09:24:440
2543386995622,30cyclictest12-21ksoftirqd/010:23:390
2543401995525,11cyclictest37-21ksoftirqd/310:46:283
254339699558,10cyclictest30-21ksoftirqd/208:20:002
254339699558,10cyclictest30-21ksoftirqd/208:20:002
2543396995531,18cyclictest1-21systemd09:40:012
254338699555,3cyclictest131rcu_preempt12:07:530
254338699555,3cyclictest131rcu_preempt12:07:530
2543386995527,13cyclictest2552727-21ssh09:16:550
2543386995526,14cyclictest2631868-21ssh12:13:100
2543386995524,29cyclictest12-21ksoftirqd/009:38:220
2543401995418,9cyclictest275-21dbus-daemon09:05:003
2543396995434,13cyclictest2596422-1kworker/u9:112:15:022
2543386995421,30cyclictest12-21ksoftirqd/009:42:530
2543396995345,5cyclictest2549090-21kworker/2:309:23:322
2543396995343,6cyclictest2599039-21kworker/2:311:15:492
2543396995342,8cyclictest30-21ksoftirqd/210:17:552
2543396995341,8cyclictest30-21ksoftirqd/212:02:232
254339699531,7cyclictest2596869-21rm10:55:022
2543386995325,12cyclictest2606719-21ssh11:17:010
2543386995325,12cyclictest2565492-21ssh09:45:080
254340199521,3cyclictest0-21swapper/310:35:023
254340199521,3cyclictest0-21swapper/310:35:013
254339699527,10cyclictest30-21ksoftirqd/207:10:022
2543396995236,12cyclictest30-21ksoftirqd/210:41:322
2543396995228,17cyclictest275-21dbus-daemon07:50:022
2543396995212,5cyclictest131rcu_preempt10:05:012
2543386995236,12cyclictest1-21systemd09:55:020
2543396995143,5cyclictest2599039-21kworker/2:311:23:072
2543396995143,5cyclictest2596984-21kworker/2:011:05:342
2543396995143,5cyclictest2560783-21kworker/2:210:11:492
2543396995143,5cyclictest2546226-21kworker/2:109:53:362
2543396995142,6cyclictest2546226-21kworker/2:109:58:482
2543396995142,6cyclictest2546226-21kworker/2:109:58:472
2543396995141,6cyclictest2596984-21kworker/2:012:29:582
2543396995141,6cyclictest2596984-21kworker/2:012:29:582
2543396995138,8cyclictest2549555-21cron09:10:002
2543386995145,4cyclictest0-21swapper/008:00:010
2543386995122,14cyclictest2612349-21ssh11:29:570
2543386995117,18cyclictest2576531-21ssh10:10:000
2543386995117,18cyclictest2576531-21ssh10:10:000
2543396995042,5cyclictest2612429-21kworker/2:111:55:282
2543396995042,5cyclictest2596984-21kworker/2:012:06:402
2543396995042,5cyclictest2596984-21kworker/2:012:06:392
2543396995042,5cyclictest2560783-21kworker/2:209:43:242
2543396995042,5cyclictest2549090-21kworker/2:309:00:312
2543396995042,5cyclictest2546226-21kworker/2:109:26:512
2543396995041,6cyclictest30-21ksoftirqd/210:05:572
2543396995041,6cyclictest30-21ksoftirqd/210:05:572
2543396995041,6cyclictest2596984-21kworker/2:012:19:032
2543396995041,5cyclictest2544940-21kworker/2:007:52:132
254339699502,15cyclictest30-21ksoftirqd/211:43:272
2543391995034,11cyclictest275-21dbus-daemon10:10:011
2543391995034,11cyclictest275-21dbus-daemon10:10:001
254339199501,9cyclictest2644031-21cat12:40:021
254339199501,9cyclictest2644031-21cat12:40:021
254338699502,42cyclictest2580968-21sh10:20:000
254338699502,3cyclictest131rcu_preempt09:59:590
254338699502,3cyclictest131rcu_preempt09:59:590
2543386995023,12cyclictest2635400-21ssh12:20:430
2543386995010,31cyclictest16850irq/134-800000010:00:410
2543401994936,9cyclictest1-21systemd07:45:003
2543401994925,18cyclictest275-21dbus-daemon11:20:013
2543396994941,5cyclictest2612429-21kworker/2:112:21:382
2543396994941,5cyclictest2612429-21kworker/2:111:53:192
2543396994941,5cyclictest2599039-21kworker/2:311:26:582
2543396994941,5cyclictest2596984-21kworker/2:012:38:332
2543396994941,5cyclictest2596984-21kworker/2:012:38:332
2543396994941,5cyclictest2596984-21kworker/2:011:47:282
2543396994941,5cyclictest2596984-21kworker/2:011:11:122
2543396994941,5cyclictest2596984-21kworker/2:011:11:112
2543396994941,5cyclictest2549090-21kworker/2:309:14:132
2543396994941,5cyclictest2546226-21kworker/2:111:00:562
2543396994941,5cyclictest2546226-21kworker/2:110:26:302
2543396994941,5cyclictest2546226-21kworker/2:108:54:172
2543396994941,5cyclictest2546226-21kworker/2:108:11:232
2543396994941,5cyclictest2544940-21kworker/2:008:01:052
2543396994941,5cyclictest2544940-21kworker/2:007:59:582
2543396994940,6cyclictest2596984-21kworker/2:012:30:262
2543396994940,6cyclictest2546226-21kworker/2:109:30:552
2543396994940,6cyclictest2546226-21kworker/2:109:30:552
2543396994939,6cyclictest2546226-21kworker/2:110:56:212
2543391994920,20cyclictest23-21ksoftirqd/109:15:001
2543391994913,6cyclictest131rcu_preempt10:50:011
2543391994913,6cyclictest131rcu_preempt09:50:001
2543386994928,15cyclictest275-21dbus-daemon11:15:010
2543386994928,15cyclictest275-21dbus-daemon11:15:000
2543386994926,19cyclictest275-21dbus-daemon09:15:000
2543386994918,27cyclictest2586647-21sh10:32:160
2543386994918,27cyclictest2586647-21sh10:32:150
2543401994830,13cyclictest3467602-21systemd-journal07:20:003
254339699489,6cyclictest131rcu_preempt08:25:022
2543396994840,5cyclictest2596984-21kworker/2:011:37:032
2543396994840,5cyclictest2560783-21kworker/2:210:45:382
2543396994840,5cyclictest2546226-21kworker/2:110:20:382
2543396994840,5cyclictest2546226-21kworker/2:108:55:442
2543396994840,5cyclictest2544940-21kworker/2:008:44:182
2543396994840,5cyclictest2544391-21kworker/2:307:39:552
2543396994840,5cyclictest2534944-21kworker/2:107:30:532
2543391994818,24cyclictest23-21ksoftirqd/112:00:011
2543386994831,13cyclictest2595279-21ssh10:51:290
2543401994733,9cyclictest1-21systemd11:55:023
2543401994732,11cyclictest3467602-21systemd-journal11:47:273
2543401994731,12cyclictest3467602-21systemd-journal10:05:003
2543396994739,5cyclictest2546226-21kworker/2:108:30:572
2543396994739,5cyclictest2546226-21kworker/2:108:25:492
2543396994739,5cyclictest2544940-21kworker/2:008:36:392
2543396994739,5cyclictest2544940-21kworker/2:008:06:242
2543396994739,5cyclictest2544391-21kworker/2:307:44:022
2543396994739,5cyclictest2542719-21kworker/2:207:11:182
2543396994739,5cyclictest2534944-21kworker/2:107:25:402
2543396994739,5cyclictest2534944-21kworker/2:107:20:352
2543396994739,5cyclictest2534944-21kworker/2:107:17:342
2543396994738,5cyclictest2544940-21kworker/2:008:45:262
2543391994730,13cyclictest3467602-21systemd-journal07:25:001
2543391994730,13cyclictest275-21dbus-daemon12:05:011
2543391994730,13cyclictest2607879-21cron11:20:001
254338699476,34cyclictest16950irq/224-800000012:20:020
2543386994729,14cyclictest2617159-21ssh11:40:060
2543386994728,13cyclictest2590862-21ssh10:41:300
254338699472,6cyclictest2544383-21grep07:30:000
2543386994722,8cyclictest12-21ksoftirqd/011:10:010
254338699471,4cyclictest131rcu_preempt12:39:540
254338699471,4cyclictest131rcu_preempt12:39:530
254338699471,15cyclictest2576755-21ssh10:10:050
2543386994627,15cyclictest2615242-21ssh11:35:570
2543386994624,18cyclictest16850irq/134-800000012:26:450
2543386994624,18cyclictest16850irq/134-800000012:26:440
254338699462,40cyclictest2560399-21ssh09:34:150
254338699462,40cyclictest2560399-21ssh09:34:140
254338699462,39cyclictest2620761-21ssh11:48:280
2543386994620,13cyclictest2625374-21ssh11:58:490
2543401994527,12cyclictest37-21ksoftirqd/310:11:273
2543391994533,9cyclictest275-21dbus-daemon10:15:001
2543391994532,10cyclictest2621519-21latency_hist11:50:011
2543391994528,4cyclictest23-21ksoftirqd/109:20:011
2543391994526,15cyclictest3467602-21systemd-journal09:05:001
2543386994535,7cyclictest0-21swapper/012:03:590
254338699453,34cyclictest0-21swapper/010:25:540
2543386994527,14cyclictest2546173-21sshd08:02:270
254340199443,23cyclictest361rcuc/312:10:033
254340199443,23cyclictest361rcuc/312:10:023
2543401994428,12cyclictest1-21systemd12:25:003
2543391994431,9cyclictest3467602-21systemd-journal10:58:281
254339199442,3cyclictest131rcu_preempt11:33:471
2543391994417,23cyclictest23-21ksoftirqd/111:45:031
254339199441,3cyclictest131rcu_preempt08:15:001
2543386994436,5cyclictest16850irq/134-800000011:35:000
2543386994428,12cyclictest2557480-21ssh09:27:280
2543386994426,15cyclictest2639661-21ssh12:30:040
2543386994425,15cyclictest2592608-21ssh10:45:200
254340199437,6cyclictest131rcu_preempt12:40:013
254340199437,6cyclictest131rcu_preempt12:40:003
2543401994329,11cyclictest3467602-21systemd-journal09:08:283
2543401994329,11cyclictest275-21dbus-daemon09:15:013
2543401994328,11cyclictest1-21systemd08:00:003
2543391994329,11cyclictest23-21ksoftirqd/112:25:031
2543391994329,11cyclictest23-21ksoftirqd/112:25:021
2543391994329,10cyclictest2572017-21readlink10:00:011
2543391994329,10cyclictest2572017-21readlink10:00:001
254339199432,12cyclictest23-21ksoftirqd/110:40:001
254339199431,6cyclictest131rcu_preempt09:40:011
2543386994326,13cyclictest3467602-21systemd-journal08:55:020
2543386994319,7cyclictest12-21ksoftirqd/008:59:020
254338699431,10cyclictest2549653-21wc09:10:010
2543401994228,10cyclictest37-21ksoftirqd/311:10:013
2543401994224,13cyclictest2546168-21sshd08:00:263
2543391994232,7cyclictest2603443-21rm11:10:011
2543391994228,11cyclictest23-21ksoftirqd/109:55:021
254339199422,3cyclictest131rcu_preempt12:34:581
2543386994225,12cyclictest2544140-21systemd-run07:25:020
2543401994129,6cyclictest309-21in:imuxsock10:45:023
2543401994127,10cyclictest37-21ksoftirqd/309:50:013
2543401994122,15cyclictest2563008-21latency_hist09:40:013
254339199415,7cyclictest23-21ksoftirqd/108:20:011
254339199415,7cyclictest23-21ksoftirqd/108:20:001
2543391994129,9cyclictest23-21ksoftirqd/109:26:501
2543391994128,9cyclictest275-21dbus-daemon11:15:021
2543391994128,9cyclictest275-21dbus-daemon11:15:011
2543391994127,10cyclictest23-21ksoftirqd/108:40:001
2543391994126,4cyclictest2631555-21sh12:12:271
2543391994125,11cyclictest23-21ksoftirqd/110:21:531
254339199411,5cyclictest131rcu_preempt09:00:011
254340199404,24cyclictest2585590-21ssh10:30:013
2543401994029,9cyclictest37-21ksoftirqd/310:37:193
2543401994023,12cyclictest37-21ksoftirqd/308:30:013
254340199401,3cyclictest131rcu_preempt10:05:143
254340199401,3cyclictest131rcu_preempt10:05:133
254339199407,6cyclictest131rcu_preempt10:20:021
2543391994029,9cyclictest23-21ksoftirqd/110:53:321
2543391994027,10cyclictest23-21ksoftirqd/111:25:371
2543391994026,11cyclictest2544906-21tr07:40:001
2543391994026,11cyclictest23-21ksoftirqd/112:05:111
2543391994026,11cyclictest23-21ksoftirqd/112:05:101
254339199401,6cyclictest131rcu_preempt08:50:001
2543386994024,12cyclictest12-21ksoftirqd/007:20:000
254338699401,3cyclictest131rcu_preempt07:55:000
2543401993925,11cyclictest37-21ksoftirqd/311:05:013
2543401993924,12cyclictest275-21dbus-daemon11:30:003
2543401993921,14cyclictest275-21dbus-daemon11:00:013
254340199391,3cyclictest131rcu_preempt11:13:443
254340199391,3cyclictest131rcu_preempt11:13:443
2543401993910,6cyclictest131rcu_preempt11:40:023
2543391993929,8cyclictest23-21ksoftirqd/111:38:061
2543391993929,7cyclictest2564567-21ssh09:43:281
2543391993926,10cyclictest23-21ksoftirqd/111:04:291
2543391993925,10cyclictest2585562-21sed10:30:011
2543391993923,12cyclictest275-21dbus-daemon11:55:021
2543391993920,15cyclictest1-21systemd07:15:011
2543386993931,4cyclictest16950irq/224-800000008:15:010
254338699392,8cyclictest1-21systemd08:25:000
2543386993924,12cyclictest12-21ksoftirqd/008:47:240
254338699391,7cyclictest2544885-21cron07:40:000
254338699391,34cyclictest2549407-21cat09:05:020
254340199388,6cyclictest131rcu_preempt10:53:103
2543401993827,9cyclictest37-21ksoftirqd/312:00:163
2543401993826,10cyclictest37-21ksoftirqd/312:32:523
2543401993825,9cyclictest37-21ksoftirqd/307:12:133
2543401993823,10cyclictest275-21dbus-daemon08:10:013
2543401993810,6cyclictest131rcu_preempt09:00:023
254339199386,6cyclictest23-21ksoftirqd/107:45:011
254339199385,10cyclictest23-21ksoftirqd/110:40:101
2543391993831,4cyclictest2545668-21munin-plugin-st07:55:001
2543391993828,6cyclictest2549555-21cron09:10:011
2543391993826,9cyclictest23-21ksoftirqd/111:23:321
2543391993824,11cyclictest3467602-21systemd-journal10:03:311
254339199382,3cyclictest131rcu_preempt12:21:561
254339199382,3cyclictest131rcu_preempt12:15:271
2543386993823,11cyclictest12-21ksoftirqd/008:40:020
2543386993819,15cyclictest2546422-21sshd08:06:270
254340199378,5cyclictest3467718-21lldpd11:43:173
2543401993727,6cyclictest309-21in:imuxsock08:55:013
254340199371,3cyclictest131rcu_preempt08:25:013
254340199371,13cyclictest2563710-21ssh09:41:113
2543391993729,6cyclictest23-21ksoftirqd/110:32:111
2543391993729,6cyclictest23-21ksoftirqd/110:32:101
2543391993726,8cyclictest2544363-21sshd07:29:411
2543391993725,10cyclictest23-21ksoftirqd/107:50:001
2543391993723,11cyclictest23-21ksoftirqd/108:05:011
254339199372,12cyclictest2556273-21unin-run09:25:011
2543386993727,6cyclictest2547746-21munin-plugin-st08:35:000
254338699371,7cyclictest2548280-21munin-run08:45:010
254340199363,4cyclictest131rcu_preempt08:40:003
254340199363,3cyclictest131rcu_preempt12:30:023
254340199363,3cyclictest131rcu_preempt12:30:013
2543401993628,5cyclictest2581262-21sh10:20:063
2543401993626,7cyclictest2554054-21systemd-run09:20:013
2543401993625,6cyclictest2544633-21munin-plugin-st07:35:003
2543401993624,10cyclictest37-21ksoftirqd/311:21:443
2543401993622,11cyclictest37-21ksoftirqd/309:31:323
2543401993622,11cyclictest37-21ksoftirqd/309:31:313
2543401993621,11cyclictest37-21ksoftirqd/308:10:043
254340199361,3cyclictest0-21swapper/309:27:223
254339199362,5cyclictest131rcu_preempt08:35:021
2543391993625,8cyclictest23-21ksoftirqd/109:31:411
2543391993625,8cyclictest23-21ksoftirqd/109:31:401
2543391993625,8cyclictest23-21ksoftirqd/108:40:281
2543391993624,10cyclictest23-21ksoftirqd/107:56:271
254339199362,3cyclictest131rcu_preempt08:30:011
254338699361,31cyclictest0-21swapper/007:40:030
254340199354,4cyclictest131rcu_preempt08:20:013
254340199354,4cyclictest131rcu_preempt08:20:013
2543401993525,8cyclictest37-21ksoftirqd/309:23:273
2543401993523,9cyclictest37-21ksoftirqd/311:58:233
2543401993523,9cyclictest37-21ksoftirqd/308:35:003
2543401993523,8cyclictest37-21ksoftirqd/307:54:553
2543401993523,8cyclictest3467602-21systemd-journal07:37:273
254339199358,5cyclictest131rcu_preempt07:16:181
254339199356,6cyclictest131rcu_preempt07:35:001
2543386993525,7cyclictest2547506-21systemctl08:30:010
2543386993520,10cyclictest0-21swapper/007:10:010
254338699351,31cyclictest0-21swapper/007:32:000
254340199342,4cyclictest131rcu_preempt07:50:023
2543401993421,10cyclictest37-21ksoftirqd/308:50:013
254339199344,3cyclictest131rcu_preempt08:55:011
2543401993325,5cyclictest2544294-21latency_hist07:25:023
2543391993324,7cyclictest23-21ksoftirqd/108:10:021
2543386993321,9cyclictest2545444-21sort07:50:020
254340199322,6cyclictest131rcu_preempt07:10:013
2543401993219,10cyclictest275-21dbus-daemon07:30:003
2543401993118,10cyclictest309-21in:imuxsock08:45:013
2543391993122,5cyclictest23-21ksoftirqd/108:25:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional