You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 07:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Sep 08, 2026 00:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
247092799117111,4cyclictest30-21ksoftirqd/220:25:022
24709159910525,39cyclictest2574484-21rm00:34:250
24709159910132,25cyclictest2539370-21ssh23:20:200
247091599991,52cyclictest0-21swapper/021:57:110
2470915999712,63cyclictest2515736-21ssh22:30:580
2470915999327,21cyclictest2490185-21ssh21:37:260
2470915999226,25cyclictest0-21swapper/023:44:540
2470915998718,34cyclictest2383687-1kworker/0:2H22:20:030
2470915998630,30cyclictest0-21swapper/000:30:010
247091599802,73cyclictest2511031-21ssh22:21:000
2470915998025,51cyclictest2551871-21ssh23:46:460
247091599782,48cyclictest2502462-21ssh22:03:030
247091599781,72cyclictest2477144-21diskmemload21:34:000
247091599781,45cyclictest0-21swapper/023:57:490
2470915997813,55cyclictest16850irq/134-800000000:11:010
2470915997725,48cyclictest2568005-21ssh00:20:230
247091599772,39cyclictest2518932-21ssh22:37:520
247091599771,71cyclictest364-21snmpd21:24:270
2470915997617,54cyclictest2504779-21ssh22:07:550
247091599752,44cyclictest2560766-21ssh00:05:190
2470915997520,51cyclictest2527130-21latency_hist22:55:010
247091599751,53cyclictest48-21kcompactd023:05:340
247091599744,65cyclictest0-21swapper/023:40:010
2470915997427,30cyclictest2496557-21ssh21:50:300
247091599742,66cyclictest2522826-21ssh22:45:520
247091599742,41cyclictest2496112-21munin-run21:50:010
247091599741,67cyclictest2472942-21/usr/sbin/munin19:50:020
247091599741,67cyclictest2472942-21/usr/sbin/munin19:50:010
247091599741,2cyclictest0-21swapper/021:43:480
247091599735,40cyclictest2535533-21rm23:12:390
2470927997240,6cyclictest2555633-21kworker/2:200:00:182
247091599724,37cyclictest2475561-21cron20:40:010
247091599721,39cyclictest2484771-21ssh21:25:430
2470915997119,47cyclictest16850irq/134-800000022:28:430
247091599706,60cyclictest2472937-21sh19:50:020
2470915997023,16cyclictest0-21swapper/023:30:090
247091599701,2cyclictest0-21swapper/020:45:180
2470915996924,39cyclictest1-21systemd21:20:010
247091599683,38cyclictest2474653-21cat20:20:020
247091599682,36cyclictest2474832-21munin-run20:25:010
247091599681,62cyclictest2538311-21ssh23:18:260
247091599672,61cyclictest0-21swapper/000:15:580
2470915996712,47cyclictest16850irq/134-800000000:04:570
2470915996620,41cyclictest16850irq/134-800000022:56:010
2470915996530,18cyclictest2554288-21ssh23:51:520
247091599652,29cyclictest2577400-21cut00:40:030
2470915996518,42cyclictest16850irq/134-800000023:26:530
247092399648,10cyclictest23-21ksoftirqd/123:55:021
247091599646,54cyclictest2479527-21munin-run21:15:010
247091599646,54cyclictest2479527-21munin-run21:15:000
247091599641,2cyclictest0-21swapper/023:05:010
247092399637,12cyclictest23-21ksoftirqd/100:03:051
247091599622,55cyclictest2471083-21sshd19:11:170
2470923996118,8cyclictest23-21ksoftirqd/123:03:221
2470915996131,14cyclictest2521754-21ssh22:43:540
247091599612,54cyclictest2471909-21munin-run19:30:000
247091599611,56cyclictest0-21swapper/022:11:090
2470927996041,7cyclictest2486638-21kworker/2:322:00:452
247092399606,7cyclictest23-21ksoftirqd/121:59:591
247091599602,53cyclictest2473693-21sshd20:01:170
247091599601,2cyclictest0-21swapper/021:10:030
247091599592,51cyclictest2472894-21latency_hist19:45:020
2470932995837,16cyclictest3467602-21systemd-journal22:15:003
2470923995814,9cyclictest23-21ksoftirqd/123:39:451
2470915995830,22cyclictest3467602-21systemd-journal21:05:010
247091599582,50cyclictest2472123-21sshd19:30:180
247091599582,50cyclictest2472123-21sshd19:30:170
2470927995742,7cyclictest2496110-21kworker/2:223:21:172
247092399577,8cyclictest23-21ksoftirqd/122:09:121
247092399574,18cyclictest23-21ksoftirqd/123:58:341
247091599572,48cyclictest2475298-21sshd20:33:010
247091599572,48cyclictest2473482-21systemd-run20:00:010
247092799566,5cyclictest131rcu_preempt00:24:182
2470927995648,5cyclictest2555633-21kworker/2:200:06:442
2470927995626,7cyclictest30-21ksoftirqd/221:13:162
2470927995626,7cyclictest30-21ksoftirqd/221:13:152
2470927995612,8cyclictest131rcu_preempt22:45:052
2470923995619,6cyclictest131rcu_preempt23:46:171
2470923995613,6cyclictest131rcu_preempt22:38:591
247091599564,46cyclictest2470911-21cyclictest19:10:010
247091599562,49cyclictest2473958-21sshd20:09:170
247091599561,2cyclictest0-21swapper/020:54:150
2470932995537,13cyclictest3467602-21systemd-journal22:20:013
2470932995533,12cyclictest37-21ksoftirqd/319:55:013
2470927995541,6cyclictest2496110-21kworker/2:222:55:192
2470927995521,8cyclictest30-21ksoftirqd/221:54:052
2470927995514,7cyclictest131rcu_preempt21:35:012
2470923995522,7cyclictest23-21ksoftirqd/122:00:141
2470923995520,6cyclictest131rcu_preempt21:12:401
2470923995520,6cyclictest131rcu_preempt21:12:391
2470923995518,9cyclictest131rcu_preempt00:30:301
2470923995513,8cyclictest131rcu_preempt22:43:051
247092799549,6cyclictest131rcu_preempt21:27:302
247092799549,5cyclictest131rcu_preempt23:37:182
2470927995440,6cyclictest2496110-21kworker/2:222:38:482
2470927995439,7cyclictest2486638-21kworker/2:321:59:542
247092799541,5cyclictest131rcu_preempt00:40:012
247092399545,9cyclictest23-21ksoftirqd/122:15:081
247092399543,9cyclictest23-21ksoftirqd/123:25:001
2470923995419,10cyclictest23-21ksoftirqd/120:55:021
2470923995416,29cyclictest131rcu_preempt23:40:431
2470923995412,6cyclictest131rcu_preempt22:33:511
247091599543,28cyclictest0-21swapper/021:00:030
247091599542,48cyclictest2471657-21munin-run19:25:010
2470932995331,18cyclictest37-21ksoftirqd/320:10:163
2470927995344,6cyclictest2486638-21kworker/2:322:08:232
2470927995336,13cyclictest2471365-21cron19:20:002
2470927995336,13cyclictest2471365-21cron19:20:002
2470927995336,11cyclictest3467602-21systemd-journal21:10:012
247092799533,13cyclictest30-21ksoftirqd/200:00:012
2470915995336,12cyclictest1-21systemd20:15:000
247091599532,46cyclictest2471369-21modprobe19:20:000
247091599532,46cyclictest2471369-21modprobe19:20:000
247091599531,3cyclictest2475818-21sshd20:40:170
247091599531,3cyclictest2475818-21sshd20:40:170
2470927995244,5cyclictest2496110-21kworker/2:222:34:202
2470927995243,6cyclictest2527024-21kworker/2:023:04:172
2470927995243,6cyclictest2496110-21kworker/2:222:17:012
2470927995242,6cyclictest2527024-21kworker/2:023:07:162
2470927995239,6cyclictest2486638-21kworker/2:321:47:352
2470927995230,7cyclictest2496110-21kworker/2:223:26:022
2470927995222,9cyclictest131rcu_preempt22:11:302
247092399527,8cyclictest23-21ksoftirqd/121:28:541
2470923995222,27cyclictest23-21ksoftirqd/120:05:021
2470923995213,9cyclictest23-21ksoftirqd/100:25:011
2470923995213,5cyclictest131rcu_preempt21:45:251
2470923995212,6cyclictest131rcu_preempt22:47:251
2470923995210,7cyclictest131rcu_preempt00:17:551
247091599522,46cyclictest2475054-21sshd20:29:170
2470932995120,18cyclictest37-21ksoftirqd/323:55:023
247092799518,5cyclictest131rcu_preempt21:20:032
2470927995143,5cyclictest2555633-21kworker/2:200:31:542
2470927995143,5cyclictest2539003-21kworker/2:323:47:212
2470927995143,5cyclictest2527024-21kworker/2:023:10:252
2470927995143,5cyclictest2496110-21kworker/2:222:42:262
2470927995143,5cyclictest2236780-21kworker/2:019:30:182
2470927995143,5cyclictest2236780-21kworker/2:019:30:172
2470927995136,6cyclictest2539003-21kworker/2:323:33:462
247092399517,9cyclictest23-21ksoftirqd/123:29:591
247092399517,9cyclictest23-21ksoftirqd/123:07:381
247092399516,9cyclictest23-21ksoftirqd/119:30:021
247092399515,10cyclictest23-21ksoftirqd/121:31:561
247092399513,11cyclictest23-21ksoftirqd/120:10:171
2470932995036,10cyclictest275-21dbus-daemon22:55:003
2470932995036,10cyclictest1-21systemd22:45:003
2470932995035,11cyclictest1-21systemd00:15:013
247093299501,11cyclictest37-21ksoftirqd/322:09:193
247092799508,8cyclictest131rcu_preempt19:39:592
2470927995042,5cyclictest2555633-21kworker/2:200:16:362
2470927995042,5cyclictest2555633-21kworker/2:200:14:182
2470927995042,5cyclictest2539003-21kworker/2:323:41:192
2470927995042,5cyclictest2496110-21kworker/2:223:17:302
2470927995042,5cyclictest2496110-21kworker/2:222:21:282
2470927995041,5cyclictest2464647-21kworker/2:121:44:112
2470927995036,7cyclictest2496110-21kworker/2:222:50:032
2470927995010,5cyclictest131rcu_preempt21:36:052
247092399508,9cyclictest23-21ksoftirqd/121:44:191
247092399508,5cyclictest131rcu_preempt00:13:091
247092399504,10cyclictest23-21ksoftirqd/100:10:001
247092399501,7cyclictest2481903-21modprobe21:19:591
2470923995017,5cyclictest131rcu_preempt22:22:311
2470923995015,9cyclictest131rcu_preempt22:13:531
2470923995014,5cyclictest131rcu_preempt00:36:121
2470923995011,6cyclictest131rcu_preempt22:25:311
247091599503,42cyclictest364-21snmpd19:35:260
247093299499,6cyclictest131rcu_preempt21:05:023
2470932994918,17cyclictest37-21ksoftirqd/323:09:513
2470927994941,5cyclictest2565174-21kworker/2:300:25:582
2470927994941,5cyclictest2546139-21kworker/2:123:54:592
2470927994941,5cyclictest2496110-21kworker/2:222:27:282
2470927994940,6cyclictest2464647-21kworker/2:121:20:582
247092399498,8cyclictest23-21ksoftirqd/121:25:001
247092399494,8cyclictest23-21ksoftirqd/123:34:211
2470923994937,9cyclictest1-21systemd19:25:001
247092399492,9cyclictest0-21swapper/121:50:571
2470923994915,5cyclictest131rcu_preempt23:12:461
2470923994914,5cyclictest131rcu_preempt00:25:151
2470923994912,5cyclictest131rcu_preempt22:59:131
2470923994910,7cyclictest131rcu_preempt23:15:251
2470932994833,11cyclictest3467602-21systemd-journal00:21:173
2470932994829,9cyclictest37-21ksoftirqd/323:05:013
2470932994829,14cyclictest275-21dbus-daemon23:59:593
2470932994823,9cyclictest37-21ksoftirqd/323:35:013
2470932994816,6cyclictest131rcu_preempt22:25:013
2470932994811,8cyclictest131rcu_preempt00:40:003
2470927994840,5cyclictest2464647-21kworker/2:120:27:582
2470927994840,5cyclictest2464647-21kworker/2:119:57:172
2470927994840,5cyclictest2464647-21kworker/2:119:22:022
247092399484,9cyclictest23-21ksoftirqd/122:52:071
2470923994817,6cyclictest131rcu_preempt19:30:171
2470923994817,6cyclictest131rcu_preempt19:30:171
2470923994812,5cyclictest131rcu_preempt21:36:161
247093299476,6cyclictest131rcu_preempt21:45:013
2470932994737,5cyclictest37-21ksoftirqd/322:40:013
2470932994726,9cyclictest2477144-21diskmemload22:34:523
247093299471,7cyclictest0-21swapper/320:30:023
2470927994739,5cyclictest2464647-21kworker/2:121:03:462
2470927994739,5cyclictest2464647-21kworker/2:120:01:322
2470927994739,5cyclictest2464647-21kworker/2:119:54:182
2470927994739,5cyclictest2464647-21kworker/2:119:27:412
2470927994727,7cyclictest2464647-21kworker/2:120:39:102
2470927994711,13cyclictest30-21ksoftirqd/220:50:022
2470923994713,5cyclictest131rcu_preempt20:46:411
247093299464,9cyclictest37-21ksoftirqd/321:22:183
2470932994620,16cyclictest37-21ksoftirqd/300:03:053
247093299461,8cyclictest2472409-21modprobe19:40:003
247093299461,7cyclictest0-21swapper/322:58:283
2470927994639,4cyclictest2464647-21kworker/2:120:11:522
2470927994638,5cyclictest2464647-21kworker/2:120:59:102
2470927994638,5cyclictest2464647-21kworker/2:120:41:422
2470927994638,5cyclictest2464647-21kworker/2:120:41:412
2470927994638,5cyclictest2464647-21kworker/2:120:33:302
2470927994638,5cyclictest2464647-21kworker/2:120:17:152
2470927994638,5cyclictest2464647-21kworker/2:119:49:192
2470927994638,5cyclictest2464647-21kworker/2:119:49:192
2470927994638,5cyclictest2464647-21kworker/2:119:42:192
2470927994638,5cyclictest2464647-21kworker/2:119:12:252
247093299454,10cyclictest37-21ksoftirqd/320:55:003
2470932994536,7cyclictest37-21ksoftirqd/300:18:483
2470932994531,10cyclictest37-21ksoftirqd/319:10:023
2470932994521,20cyclictest37-21ksoftirqd/300:35:023
247093299451,5cyclictest131rcu_preempt00:09:313
2470932994514,6cyclictest131rcu_preempt21:59:593
2470927994537,5cyclictest2464647-21kworker/2:120:08:372
2470923994535,7cyclictest23-21ksoftirqd/119:15:011
247093299443,4cyclictest131rcu_preempt22:03:553
2470932994429,11cyclictest37-21ksoftirqd/321:27:473
2470932994421,19cyclictest37-21ksoftirqd/321:32:523
2470932994420,20cyclictest37-21ksoftirqd/321:14:093
2470932994420,20cyclictest37-21ksoftirqd/321:14:093
2470932994419,21cyclictest37-21ksoftirqd/323:43:333
2470927994437,4cyclictest2464647-21kworker/2:120:45:272
247092399444,8cyclictest23-21ksoftirqd/120:05:021
247092399443,10cyclictest23-21ksoftirqd/119:10:021
247092399441,5cyclictest0-21swapper/119:50:011
247092399441,5cyclictest0-21swapper/119:50:001
247093299432,4cyclictest0-21swapper/319:45:013
247093299431,7cyclictest0-21swapper/323:18:273
2470932994317,14cyclictest37-21ksoftirqd/321:52:343
2470932994314,17cyclictest37-21ksoftirqd/322:27:233
2470932994313,5cyclictest131rcu_preempt23:38:593
247092399439,9cyclictest23-21ksoftirqd/120:36:161
247092399438,13cyclictest23-21ksoftirqd/119:56:301
247092399437,10cyclictest23-21ksoftirqd/120:30:021
2470923994334,6cyclictest23-21ksoftirqd/120:20:011
247092399431,13cyclictest2475305-21sshd20:34:171
2470932994233,5cyclictest2473723-21munin-plugin-st20:05:003
2470932994225,9cyclictest37-21ksoftirqd/319:35:033
2470932994225,9cyclictest37-21ksoftirqd/319:35:023
2470932994213,19cyclictest131rcu_preempt21:17:353
247092399429,7cyclictest131rcu_preempt20:25:021
2470923994234,5cyclictest2477154-21/usr/sbin/munin21:10:011
247093299418,6cyclictest131rcu_preempt21:48:113
247093299417,6cyclictest131rcu_preempt23:45:243
247093299413,6cyclictest131rcu_preempt21:40:023
2470932994131,7cyclictest37-21ksoftirqd/323:11:423
2470932994126,11cyclictest1-21systemd23:30:013
247093299411,5cyclictest0-21swapper/300:26:213
2470932994115,9cyclictest37-21ksoftirqd/322:48:463
247092399414,7cyclictest23-21ksoftirqd/121:01:181
2470923994129,8cyclictest275-21dbus-daemon19:45:001
2470932994025,12cyclictest275-21dbus-daemon19:15:003
2470932993911,15cyclictest37-21ksoftirqd/323:21:363
2470932993827,8cyclictest2477172-21systemd-run21:10:013
2470932993822,11cyclictest37-21ksoftirqd/320:25:013
247093299381,7cyclictest2473952-21sshd20:07:543
2470932993812,5cyclictest131rcu_preempt19:22:433
247092399387,11cyclictest23-21ksoftirqd/120:58:281
247092399385,10cyclictest23-21ksoftirqd/119:20:001
247092399385,10cyclictest23-21ksoftirqd/119:20:001
247093299376,5cyclictest131rcu_preempt20:36:433
247093299376,5cyclictest131rcu_preempt19:29:323
247093299375,5cyclictest131rcu_preempt20:33:353
247093299374,4cyclictest131rcu_preempt19:20:023
247093299374,4cyclictest131rcu_preempt19:20:013
2470932993729,5cyclictest2472938-21latency_hist19:50:023
2470932993729,5cyclictest2472938-21latency_hist19:50:013
2470932993727,7cyclictest37-21ksoftirqd/320:00:013
247092399379,5cyclictest131rcu_preempt19:50:171
2470923993721,11cyclictest23-21ksoftirqd/120:45:011
2470923993721,11cyclictest23-21ksoftirqd/120:45:001
247092399371,6cyclictest0-21swapper/119:39:271
247093299364,5cyclictest131rcu_preempt20:18:273
247093299354,5cyclictest131rcu_preempt20:43:043
247093299354,5cyclictest131rcu_preempt20:43:033
247092799357,18cyclictest131rcu_preempt19:10:022
247093299344,5cyclictest131rcu_preempt20:47:423
247093299341,5cyclictest131rcu_preempt20:57:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional