You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-10 - 18:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Jul 10, 2026 12:45:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2154884999629,12cyclictest2169374-21ssh09:27:150
2154884998718,59cyclictest16850irq/134-800000011:06:200
2154884998616,60cyclictest16850irq/134-800000012:10:520
2154884998528,12cyclictest3467602-21systemd-journal09:19:560
2154884998480,2cyclictest0-21swapper/010:55:020
2154884998480,2cyclictest0-21swapper/010:55:020
2154884998336,44cyclictest16950irq/224-800000012:35:000
215488499832,37cyclictest2246490-21ssh12:09:210
215488499832,37cyclictest2246490-21ssh12:09:210
2154884998129,13cyclictest2251077-21ssh12:18:570
2154884998025,13cyclictest2243171-21ssh12:02:050
2154884998024,13cyclictest2192869-21ssh10:16:250
2154884997924,16cyclictest2190143-21ssh10:10:390
2154884997924,12cyclictest2229654-21ssh11:33:590
215488499791,44cyclictest2208791-21ssh10:50:000
2154884997718,31cyclictest3467602-21systemd-journal12:24:410
2154884997717,54cyclictest16850irq/134-800000010:35:200
2154884997628,14cyclictest2254591-21ssh12:25:570
2154884997618,53cyclictest16850irq/134-800000009:53:410
2154884997524,16cyclictest2226459-21ssh11:27:110
215488499752,36cyclictest311-21rs:main0
2154884997518,6cyclictest16850irq/134-800000011:54:320
2154884997419,50cyclictest16850irq/134-800000009:22:510
2154884997418,51cyclictest16850irq/134-800000010:06:190
215488499741,35cyclictest2195363-21ssh10:21:330
2154884997328,12cyclictest2184610-21ssh09:59:260
2154884997328,12cyclictest2184610-21ssh09:59:260
2154884997317,51cyclictest16850irq/134-800000010:04:350
2154884997316,52cyclictest16850irq/134-800000009:38:040
2154884997316,52cyclictest16850irq/134-800000009:38:040
2154884997315,6cyclictest16850irq/134-800000011:18:090
2154884997218,48cyclictest16850irq/134-800000009:33:580
2154884997217,50cyclictest16850irq/134-800000011:55:300
2154884997217,50cyclictest16850irq/134-800000011:55:300
215488499721,28cyclictest0-21swapper/009:47:120
215488999707,54cyclictest2175409-21ssh09:40:021
215488999707,54cyclictest2175409-21ssh09:40:021
2154884997017,48cyclictest16850irq/134-800000010:40:180
2154884997012,17cyclictest16850irq/134-800000010:25:570
2154894996922,7cyclictest30-21ksoftirqd/210:18:562
2154884996917,6cyclictest16850irq/134-800000010:33:310
215488499682,14cyclictest2219935-21sshd11:13:240
215488499682,14cyclictest2219935-21sshd11:13:240
2154884996815,49cyclictest16850irq/134-800000009:44:550
2154884996814,49cyclictest16850irq/134-800000011:02:150
2154884996814,49cyclictest16850irq/134-800000011:02:150
2154898996618,9cyclictest37-21ksoftirqd/309:50:023
2154894996624,5cyclictest2244258-21sh12:04:442
2154884996619,7cyclictest16850irq/134-800000011:21:350
2154884996615,46cyclictest16850irq/134-800000011:48:310
2154898996511,9cyclictest37-21ksoftirqd/311:30:003
2154894996529,33cyclictest30-21ksoftirqd/209:15:012
2154894996425,6cyclictest2211833-21sh10:56:052
215488499648,17cyclictest16850irq/134-800000010:59:270
2154898996218,10cyclictest37-21ksoftirqd/312:10:213
2154898996214,8cyclictest37-21ksoftirqd/310:10:013
2154894996228,7cyclictest30-21ksoftirqd/211:30:482
2154898996116,11cyclictest37-21ksoftirqd/309:40:013
2154898996116,11cyclictest37-21ksoftirqd/309:40:013
2154898996115,6cyclictest37-21ksoftirqd/308:20:003
2154898996110,12cyclictest37-21ksoftirqd/309:21:193
2154894996123,7cyclictest30-21ksoftirqd/210:34:282
2154894996123,6cyclictest2154633-21kworker/u8:307:45:012
2154894996123,6cyclictest2154633-21kworker/u8:307:45:002
2154894996121,9cyclictest30-21ksoftirqd/212:20:012
215489499611,6cyclictest131rcu_preempt12:10:012
215489499611,6cyclictest131rcu_preempt12:10:012
215488499616,37cyclictest19350irq/34-mmc108:40:020
2154884996140,17cyclictest16850irq/134-800000011:40:010
215489899608,14cyclictest37-21ksoftirqd/309:34:483
2154894996027,8cyclictest30-21ksoftirqd/210:10:282
2154894996024,7cyclictest2155634-21sshd07:22:252
2154894996024,4cyclictest2182046-21sh09:54:012
2154894996023,6cyclictest2158239-21modprobe08:15:002
2154884996014,41cyclictest1-21systemd07:25:010
2154898995915,9cyclictest37-21ksoftirqd/311:55:013
2154894995940,15cyclictest3467602-21systemd-journal08:35:002
2154894995940,15cyclictest3467602-21systemd-journal08:35:002
2154894995924,7cyclictest30-21ksoftirqd/211:51:132
215489499592,25cyclictest2177758-21munin-run09:45:002
2154894995916,10cyclictest3467602-21systemd-journal10:55:012
2154894995916,10cyclictest3467602-21systemd-journal10:55:012
215489899588,12cyclictest37-21ksoftirqd/308:35:033
215489899588,12cyclictest37-21ksoftirqd/308:35:033
2154898995815,9cyclictest37-21ksoftirqd/310:35:013
2154889995838,15cyclictest1-21systemd08:50:011
215488999583,45cyclictest275-21dbus-daemon11:10:001
215488499586,20cyclictest2154883-21wc07:10:010
215489899579,11cyclictest37-21ksoftirqd/309:28:593
2154894995719,10cyclictest30-21ksoftirqd/212:30:462
215488999577,26cyclictest0-21swapper/110:00:031
2154889995734,17cyclictest275-21dbus-daemon07:40:001
215488499574,3cyclictest131rcu_preempt08:35:010
215488499574,3cyclictest131rcu_preempt08:35:010
2154898995612,9cyclictest37-21ksoftirqd/310:15:013
2154898995610,11cyclictest37-21ksoftirqd/307:40:003
2154894995626,11cyclictest2156974-21kworker/2:111:39:522
2154894995625,7cyclictest30-21ksoftirqd/209:57:592
2154894995625,7cyclictest30-21ksoftirqd/209:57:592
2154894995622,5cyclictest2219375-21sh11:12:052
2154894995622,5cyclictest2219375-21sh11:12:052
215489899558,13cyclictest37-21ksoftirqd/309:53:013
2154898995516,8cyclictest37-21ksoftirqd/311:18:583
2154894995523,5cyclictest2170860-21sh09:30:152
2154894995521,11cyclictest275-21dbus-daemon07:30:012
2154894995521,11cyclictest275-21dbus-daemon07:30:012
2154894995520,8cyclictest30-21ksoftirqd/212:10:182
2154894995518,11cyclictest30-21ksoftirqd/211:06:012
2154884995519,32cyclictest16850irq/134-800000011:42:290
215489899549,11cyclictest37-21ksoftirqd/311:09:583
2154894995445,6cyclictest2156974-21kworker/2:108:44:592
2154894995423,7cyclictest30-21ksoftirqd/211:50:002
2154894995421,5cyclictest2175368-21latency_hist09:40:022
2154894995421,5cyclictest2175368-21latency_hist09:40:022
2154894995421,10cyclictest30-21ksoftirqd/209:28:142
2154894995420,6cyclictest2167984-21rm09:24:312
2154894995418,7cyclictest0-21swapper/210:25:012
2154884995417,33cyclictest16850irq/134-800000009:10:100
215489899537,9cyclictest37-21ksoftirqd/309:43:423
215489899536,11cyclictest2199061-21ssh10:29:423
215489899534,28cyclictest2159240-21kworker/u8:008:40:013
2154898995313,8cyclictest37-21ksoftirqd/308:10:013
2154894995344,6cyclictest2156974-21kworker/2:111:01:192
2154894995344,6cyclictest2156974-21kworker/2:111:01:192
2154894995343,5cyclictest2129850-21kworker/2:010:39:422
2154894995325,10cyclictest30-21ksoftirqd/212:37:462
2154894995323,4cyclictest2222170-21sh11:18:062
2154894995320,7cyclictest30-21ksoftirqd/209:49:012
215489499531,30cyclictest30-21ksoftirqd/210:06:452
2154894995311,16cyclictest2242027-21ssh12:00:002
2154894995311,16cyclictest2242027-21ssh12:00:002
2154898995216,9cyclictest37-21ksoftirqd/311:39:063
2154898995213,9cyclictest0-21swapper/311:47:523
2154898995213,7cyclictest37-21ksoftirqd/311:00:013
2154894995243,5cyclictest2129850-21kworker/2:012:20:282
2154894995224,10cyclictest30-21ksoftirqd/210:48:242
215489899516,11cyclictest37-21ksoftirqd/312:23:083
215489899515,8cyclictest37-21ksoftirqd/310:37:193
215489899514,7cyclictest37-21ksoftirqd/312:09:473
215489899514,7cyclictest37-21ksoftirqd/312:09:463
2154898995111,9cyclictest0-21swapper/310:52:033
2154898995111,9cyclictest0-21swapper/310:52:033
2154894995143,5cyclictest2129850-21kworker/2:009:00:022
2154894995143,5cyclictest2129850-21kworker/2:009:00:012
215489499511,25cyclictest2226228-1kworker/u9:211:42:502
2154889995133,12cyclictest275-21dbus-daemon12:40:011
215488499514,9cyclictest2160564-21sh09:00:000
215488499514,9cyclictest2160564-21sh09:00:000
215489899507,7cyclictest37-21ksoftirqd/311:41:133
2154898995011,8cyclictest0-21swapper/310:49:513
2154898995011,7cyclictest37-21ksoftirqd/312:28:253
2154894995041,5cyclictest2129850-21kworker/2:011:27:492
2154894995041,5cyclictest2129850-21kworker/2:008:08:242
2154889995038,9cyclictest23-21ksoftirqd/112:25:171
2154889995026,16cyclictest3467602-21systemd-journal11:20:001
215489899497,6cyclictest37-21ksoftirqd/312:39:153
215489899496,10cyclictest37-21ksoftirqd/312:18:543
215489899495,5cyclictest37-21ksoftirqd/311:55:223
215489899495,5cyclictest37-21ksoftirqd/311:55:223
2154898994917,8cyclictest37-21ksoftirqd/311:00:433
2154898994917,8cyclictest37-21ksoftirqd/311:00:433
2154898994914,11cyclictest37-21ksoftirqd/311:33:103
2154898994910,7cyclictest37-21ksoftirqd/309:17:593
215489499493,3cyclictest131rcu_preempt09:15:492
2154894994920,7cyclictest30-21ksoftirqd/210:29:082
215489499491,4cyclictest131rcu_preempt10:01:162
2154889994917,23cyclictest221rcuc/110:18:061
215489899489,11cyclictest0-21swapper/311:21:413
215489899488,8cyclictest37-21ksoftirqd/310:16:473
215489899486,7cyclictest0-21swapper/310:02:213
215489899483,16cyclictest2157180-21sshd07:53:283
2154898994817,8cyclictest37-21ksoftirqd/312:33:373
2154898994814,7cyclictest37-21ksoftirqd/311:12:013
2154898994814,7cyclictest37-21ksoftirqd/311:12:003
2154894994832,12cyclictest275-21dbus-daemon08:55:002
2154894994822,9cyclictest30-21ksoftirqd/210:42:482
2154889994833,11cyclictest1-21systemd12:35:021
215489899472,12cyclictest2157030-21latency_hist07:50:013
2154898994712,6cyclictest0-21swapper/309:11:553
2154889994734,7cyclictest2201596-21latency_hist10:35:021
2154889994725,16cyclictest3467602-21systemd-journal07:25:001
2154884994730,12cyclictest3467602-21systemd-journal08:00:010
215489899468,9cyclictest0-21swapper/310:25:003
215489899466,7cyclictest37-21ksoftirqd/310:41:583
215489899464,9cyclictest37-21ksoftirqd/312:01:563
2154898994630,11cyclictest3467602-21systemd-journal08:15:003
2154898994612,10cyclictest37-21ksoftirqd/309:57:033
2154898994612,10cyclictest37-21ksoftirqd/309:57:033
215489899461,17cyclictest0-21swapper/308:30:013
2154894994638,5cyclictest2156974-21kworker/2:111:23:212
215489499462,3cyclictest131rcu_preempt07:35:002
2154889994629,13cyclictest275-21dbus-daemon09:15:001
2154898994512,8cyclictest0-21swapper/308:04:203
215489499452,3cyclictest131rcu_preempt08:00:012
2154889994533,9cyclictest2209993-21sshd10:52:241
2154889994533,9cyclictest2209993-21sshd10:52:241
2154889994531,10cyclictest3467602-21systemd-journal09:45:001
2154889994529,11cyclictest3467602-21systemd-journal07:30:011
2154889994529,11cyclictest3467602-21systemd-journal07:30:011
215488499452,3cyclictest131rcu_preempt07:15:010
215488499452,38cyclictest275-21dbus-daemon07:50:000
2154894994428,12cyclictest1-21systemd07:50:012
2154889994435,7cyclictest23-21ksoftirqd/110:46:191
2154889994428,12cyclictest275-21dbus-daemon08:40:011
2154889994426,13cyclictest3467602-21systemd-journal11:35:011
2154889994425,14cyclictest3467602-21systemd-journal11:04:241
2154889994425,14cyclictest3467602-21systemd-journal11:04:241
2154889994421,16cyclictest1-21systemd11:30:011
215488999441,5cyclictest0-21swapper/112:05:391
215488999441,5cyclictest0-21swapper/112:05:391
2154884994411,16cyclictest2154633-21kworker/u8:307:35:010
2154898994333,6cyclictest309-21in:imuxsock08:50:013
2154889994333,6cyclictest23-21ksoftirqd/108:15:001
2154889994329,11cyclictest23-21ksoftirqd/109:25:001
2154889994324,14cyclictest0-21swapper/112:20:001
2154889994324,12cyclictest275-21dbus-daemon08:30:001
2154889994322,17cyclictest3467602-21systemd-journal08:10:011
2154889994317,21cyclictest23-21ksoftirqd/112:10:511
2154889994311,6cyclictest131rcu_preempt10:43:341
2154898994226,3cyclictest0-21swapper/307:20:023
215489499424,7cyclictest131rcu_preempt08:20:002
2154894994228,10cyclictest309-21in:imuxsock12:30:002
215488999423,3cyclictest131rcu_preempt11:48:561
2154889994231,8cyclictest23-21ksoftirqd/111:58:241
2154889994231,8cyclictest23-21ksoftirqd/111:58:241
2154889994231,7cyclictest23-21ksoftirqd/112:25:011
2154889994229,11cyclictest23-21ksoftirqd/111:14:401
2154889994229,11cyclictest23-21ksoftirqd/111:14:401
2154889994228,10cyclictest2165838-21cron09:20:001
215488999421,5cyclictest0-21swapper/110:07:301
215488499425,5cyclictest131rcu_preempt08:25:020
2154884994235,4cyclictest16950irq/224-800000008:15:000
2154884994233,6cyclictest16950irq/224-800000008:47:240
2154898994125,13cyclictest2159792-21sshd08:44:243
2154894994132,5cyclictest2146112-21kworker/2:207:15:012
215489499411,36cyclictest30-21ksoftirqd/208:30:012
215488999418,6cyclictest131rcu_preempt09:55:421
215488999418,6cyclictest131rcu_preempt09:55:421
2154889994131,6cyclictest23-21ksoftirqd/109:35:021
2154889994123,13cyclictest3467602-21systemd-journal11:53:241
215488999411,6cyclictest0-21swapper/109:47:511
215488499414,10cyclictest12-21ksoftirqd/007:19:010
2154884994134,4cyclictest16950irq/224-800000008:40:000
215489499403,6cyclictest131rcu_preempt09:05:002
2154894994029,6cyclictest309-21in:imuxsock07:20:012
2154894994028,9cyclictest2157358-21cat07:55:012
2154894994023,12cyclictest275-21dbus-daemon08:50:012
215489499401,3cyclictest131rcu_preempt08:23:242
2154889994030,6cyclictest23-21ksoftirqd/111:45:021
2154889994027,8cyclictest309-21in:imuxsock09:05:001
215488999401,6cyclictest0-21swapper/111:20:581
2154884994027,9cyclictest2156395-21sshd07:38:270
2154884994025,11cyclictest12-21ksoftirqd/009:05:000
215489899397,8cyclictest0-21swapper/307:33:163
215489899395,11cyclictest37-21ksoftirqd/307:45:023
215489899395,11cyclictest37-21ksoftirqd/307:45:013
2154889993929,7cyclictest23-21ksoftirqd/111:37:121
2154889993928,9cyclictest23-21ksoftirqd/112:01:551
2154889993927,10cyclictest23-21ksoftirqd/109:25:081
2154889993926,10cyclictest3467602-21systemd-journal09:50:031
2154889993922,12cyclictest2161331-21date09:10:031
2154889993922,12cyclictest2161331-21date09:10:031
2154889993921,12cyclictest3467602-21systemd-journal07:55:001
215488499392,6cyclictest2161127-21munin-run09:10:010
215488499392,6cyclictest2161127-21munin-run09:10:010
215488499391,34cyclictest2158692-21cut08:20:020
215489899383,6cyclictest37-21ksoftirqd/308:58:243
215489899383,6cyclictest37-21ksoftirqd/308:58:233
215489899383,11cyclictest37-21ksoftirqd/308:53:493
2154894993827,7cyclictest2161083-21cron09:10:002
2154894993827,7cyclictest2161083-21cron09:10:002
215489499382,3cyclictest131rcu_preempt08:40:002
215489499381,4cyclictest131rcu_preempt07:40:002
215488999384,5cyclictest131rcu_preempt08:20:001
2154889993829,7cyclictest23-21ksoftirqd/108:55:371
2154889993829,7cyclictest23-21ksoftirqd/108:55:371
2154889993827,8cyclictest23-21ksoftirqd/110:57:031
2154889993824,11cyclictest23-21ksoftirqd/110:10:361
215488999381,6cyclictest131rcu_preempt07:50:001
2154889993816,5cyclictest23-21ksoftirqd/110:20:401
2154884993833,3cyclictest16950irq/224-800000008:25:240
2154884993822,12cyclictest12-21ksoftirqd/008:10:010
2154898993724,9cyclictest3467602-21systemd-journal09:05:003
2154898993721,11cyclictest37-21ksoftirqd/309:10:003
2154898993721,11cyclictest37-21ksoftirqd/309:10:003
2154889993726,8cyclictest23-21ksoftirqd/110:25:261
2154889993722,11cyclictest3467602-21systemd-journal08:05:001
215488999371,5cyclictest0-21swapper/110:36:411
2154889993711,22cyclictest2158788-21kworker/1:008:35:011
2154889993711,22cyclictest2158788-21kworker/1:008:35:011
2154884993730,4cyclictest16950irq/224-800000007:44:540
2154884993730,4cyclictest16950irq/224-800000007:44:540
2154898993625,7cyclictest309-21in:imuxsock08:00:003
2154898993617,15cyclictest2158758-21sshd08:24:243
215488999369,7cyclictest131rcu_preempt07:15:231
2154889993624,9cyclictest23-21ksoftirqd/107:15:021
2154884993622,11cyclictest2155924-21systemd-run07:30:010
2154884993622,11cyclictest2155924-21systemd-run07:30:010
2154898993525,5cyclictest309-21in:imuxsock07:15:013
2154898993520,11cyclictest2155660-21munin-run07:25:003
215489899351,7cyclictest2155881-21sshd07:26:243
215489899351,7cyclictest2155881-21sshd07:26:243
2154894993524,8cyclictest2155004-21wc07:10:022
2154884993527,5cyclictest19350irq/34-mmc107:52:010
215488499351,31cyclictest0-21swapper/008:54:090
2154889993427,4cyclictest2159804-21sh08:45:001
2154889993423,8cyclictest2160388-21sed08:55:011
2154889993419,12cyclictest23-21ksoftirqd/107:43:451
2154889993419,12cyclictest23-21ksoftirqd/107:43:451
2154894993325,5cyclictest2157866-21latency_hist08:05:022
2154889993316,11cyclictest2157445-21modprobe08:00:001
215488499332,7cyclictest3467602-21systemd-journal08:01:250
2154889993120,7cyclictest23-21ksoftirqd/108:25:011
2154889993024,4cyclictest2154839-21latency_hist07:10:011
2154898992922,4cyclictest2154839-21latency_hist07:10:023
2154889992821,4cyclictest23-21ksoftirqd/107:33:321
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional