You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 11:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Jun 28, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2244621997221,15cyclictest2263477-21ssh21:37:022
2244616997233,34cyclictest0-21swapper/023:50:010
2244621997118,41cyclictest0-21swapper/219:20:022
2244616997027,38cyclictest1-21systemd00:10:010
224462999684,10cyclictest2283740-21cron22:19:593
2244621996819,14cyclictest2337834-21ssh00:14:042
2244629996624,38cyclictest2300500-21latency_hist22:55:013
224462999662,16cyclictest2281398-21munin-run22:15:013
224462999662,16cyclictest2281398-21munin-run22:15:013
2244621996616,14cyclictest2262055-21ssh21:34:082
224462999654,32cyclictest131rcu_preempt21:25:013
2244621996525,9cyclictest2302276-21ssh22:58:442
2244621996520,14cyclictest2245645-21latency_hist19:30:012
2244629996428,33cyclictest37-21ksoftirqd/322:25:003
2244629996428,33cyclictest37-21ksoftirqd/322:25:003
2244621996449,12cyclictest2324728-21ssh23:45:542
2244621996425,7cyclictest2319767-21ssh23:35:162
2244621996425,7cyclictest2319767-21ssh23:35:162
2244621996419,14cyclictest2272412-21ssh21:55:472
2244621996418,15cyclictest2278180-21rm22:08:092
2244621996418,14cyclictest2315721-21ssh23:27:012
2244621996418,14cyclictest2315721-21ssh23:27:002
2244616996426,19cyclictest2331219-21ssh00:00:010
2244616996426,19cyclictest2331219-21ssh00:00:010
2244621996320,13cyclictest2258884-21ssh21:27:192
2244629996223,34cyclictest37-21ksoftirqd/321:37:133
224462999622,18cyclictest3467602-21systemd-journal21:42:243
224462999622,18cyclictest3467602-21systemd-journal21:42:243
2244629996217,4cyclictest2247044-21latency_hist19:55:023
2244621996250,9cyclictest2266316-21ssh21:43:072
2244621996250,9cyclictest2266316-21ssh21:43:062
2244621996222,11cyclictest2250795-21diskmemload22:25:532
2244621996220,14cyclictest2335348-21ssh00:08:502
2244621996215,13cyclictest2309426-21ssh23:13:542
2244616996224,11cyclictest2253811-21ssh21:16:120
2244629996123,34cyclictest37-21ksoftirqd/323:20:013
2244629996123,34cyclictest37-21ksoftirqd/321:55:153
2244629996121,12cyclictest275-21dbus-daemon22:45:013
2244629996120,34cyclictest0-21swapper/319:20:013
2244629996119,10cyclictest37-21ksoftirqd/321:15:033
2244629996116,9cyclictest2275687-21sshd22:02:543
2244621996148,10cyclictest2328286-21ssh23:53:452
2244621996148,10cyclictest2306423-21ssh23:07:262
2244621996143,14cyclictest2330015-21ssh23:57:152
2244621996143,14cyclictest2330015-21ssh23:57:152
2244621996116,11cyclictest2294941-21ssh22:43:032
2244619996112,11cyclictest23-21ksoftirqd/121:53:481
2244619996112,11cyclictest23-21ksoftirqd/121:53:481
224461699612,54cyclictest48-21kcompactd000:00:180
224462999603,18cyclictest2250189-21latency_hist20:55:023
2244629996018,4cyclictest2325839-21sh23:48:323
2244629996018,4cyclictest2268045-21sh21:46:363
224462999601,16cyclictest2315985-21ssh23:27:343
224462999601,16cyclictest2315985-21ssh23:27:343
2244621996046,10cyclictest2317856-21ssh23:31:272
2244621996043,13cyclictest2292089-21ssh22:37:012
2244621996039,17cyclictest2250545-21munin-run21:05:012
2244621996039,17cyclictest2250545-21munin-run21:05:002
2244621996018,13cyclictest2311655-21ssh23:18:362
224461999601,40cyclictest221rcuc/121:15:021
224461999601,40cyclictest221rcuc/121:15:021
2244616996028,16cyclictest2341752-21ssh00:22:060
224462999593,16cyclictest2288568-21modprobe22:30:003
2244629995927,28cyclictest37-21ksoftirqd/300:05:003
2244629995924,31cyclictest2291669-21sh22:36:093
2244629995922,32cyclictest2246120-21sendmail19:40:003
2244629995913,5cyclictest0-21swapper/300:05:343
2244621995944,12cyclictest2348788-21ssh00:36:562
2244621995944,12cyclictest2275914-21ssh22:03:212
2244621995943,12cyclictest2342782-21ssh00:24:262
2244621995942,13cyclictest2346615-21ssh00:32:252
2244621995942,13cyclictest2331806-21ssh00:01:042
2244621995941,14cyclictest2323434-21ssh23:43:192
2244621995941,14cyclictest2313555-21ssh23:22:222
2244621995941,14cyclictest2289623-21ssh22:31:542
2244621995923,10cyclictest0-21swapper/222:10:572
2244621995923,10cyclictest0-21swapper/222:10:572
2244621995914,13cyclictest2297562-21ssh22:48:502
2244621995912,34cyclictest0-21swapper/220:45:022
224461999597,14cyclictest2330887-21sh23:59:271
224461999597,14cyclictest2330887-21sh23:59:271
224461999593,25cyclictest2313430-21ssh23:22:071
2244616995938,17cyclictest16850irq/134-800000022:55:000
224462999583,19cyclictest2250786-21modprobe21:10:003
2244629995826,28cyclictest2302826-21latency_hist23:00:033
224462999582,20cyclictest150-21pr/ttyS223:04:293
224462999582,20cyclictest150-21pr/ttyS223:04:293
224462999582,15cyclictest2321784-21latency_hist23:40:003
224462999582,15cyclictest2321784-21latency_hist23:40:003
224462999582,14cyclictest2350182-21latency_hist00:40:023
2244629995815,6cyclictest37-21ksoftirqd/322:49:063
224462999581,19cyclictest2277058-21ssh22:05:273
224462999581,16cyclictest2291009-21rm22:35:023
2244621995845,10cyclictest2268246-21ssh21:47:032
2244621995844,11cyclictest2299079-21ssh22:51:482
2244621995842,13cyclictest2284289-21ssh22:20:382
2244621995842,13cyclictest2284289-21ssh22:20:382
2244621995842,13cyclictest2283007-21ssh22:18:122
2244621995842,13cyclictest2256980-21ssh21:22:592
2244621995842,12cyclictest2339057-21ssh00:16:172
2244621995841,14cyclictest2343353-21ssh00:25:222
2244621995841,14cyclictest2343353-21ssh00:25:222
2244621995841,14cyclictest2253941-21ssh21:16:272
2244621995841,13cyclictest2270842-21ssh21:52:412
2244621995841,13cyclictest2270842-21ssh21:52:412
2244621995840,14cyclictest2304245-21ssh23:02:492
2244621995840,14cyclictest2304245-21ssh23:02:492
2244619995815,8cyclictest23-21ksoftirqd/100:37:491
2244619995813,11cyclictest23-21ksoftirqd/123:40:551
2244616995833,16cyclictest3467602-21systemd-journal22:50:010
2244616995828,14cyclictest2265524-21ssh21:41:080
2244616995828,14cyclictest2265524-21ssh21:41:070
224462999572,16cyclictest2249360-21latency_hist20:40:033
2244629995717,6cyclictest2248474-21latency_hist20:25:023
2244621995744,10cyclictest2251627-21ssh21:11:372
2244621995744,10cyclictest2251627-21ssh21:11:372
224461999576,17cyclictest2274931-21sh22:01:001
224461999572,16cyclictest2305164-21ssh23:04:551
224461999572,16cyclictest2305164-21ssh23:04:541
2244619995716,9cyclictest23-21ksoftirqd/121:20:021
2244619995714,25cyclictest2244610-21cyclictest00:05:021
2244619995713,14cyclictest2320102-21sh23:36:111
2244619995713,14cyclictest2320102-21sh23:36:111
2244619995712,11cyclictest23-21ksoftirqd/123:13:581
224461999570,15cyclictest2258873-21ssh21:27:161
2244616995744,9cyclictest16950irq/224-800000023:15:000
224461699573,47cyclictest2293392-21cron22:40:000
2244616995729,13cyclictest2283579-21ssh22:19:310
224461699572,28cyclictest131rcu_preempt23:08:200
2244629995623,29cyclictest2262590-21latency_hist21:35:033
2244629995614,8cyclictest309-21in:imuxsock20:17:003
2244621995642,10cyclictest2246620-21sshd19:47:312
2244621995639,14cyclictest364-21snmpd20:46:382
2244621995636,16cyclictest2246086-21sshd19:37:302
224461999569,9cyclictest23-21ksoftirqd/100:28:231
224461999569,9cyclictest23-21ksoftirqd/100:28:231
224462999555,9cyclictest2331192-21munin-run00:00:013
224462999555,9cyclictest2331192-21munin-run00:00:003
2244629995527,24cyclictest2344961-21sh00:29:063
2244629995527,24cyclictest2344961-21sh00:29:053
2244629995522,28cyclictest2340725-21latency_hist00:20:023
2244629995520,6cyclictest2245984-21latency_hist19:35:033
2244629995514,9cyclictest37-21ksoftirqd/300:23:333
2244619995515,9cyclictest23-21ksoftirqd/121:37:561
2244619995515,8cyclictest23-21ksoftirqd/123:28:431
2244619995515,8cyclictest23-21ksoftirqd/123:28:431
224461699556,32cyclictest2250261-21modprobe21:00:000
2244616995528,11cyclictest2322686-21ssh23:41:340
2244616995526,14cyclictest2270662-21ssh21:52:080
2244616995526,14cyclictest2270662-21ssh21:52:070
2244616995526,13cyclictest2285362-21ssh22:23:130
2244616995526,13cyclictest2285362-21ssh22:23:120
224462999542,17cyclictest2248742-21unin-run20:30:003
2244629995411,10cyclictest2245622-21systemctl19:30:013
2244621995424,25cyclictest2248902-21latency_hist20:30:022
224461999549,15cyclictest2257167-21sh21:23:221
224461999545,14cyclictest2294685-21ssh22:42:321
224461999543,14cyclictest2262160-21sh21:34:221
224461699544,42cyclictest2350259-21cat00:40:020
2244616995442,9cyclictest16850irq/134-800000000:20:010
2244616995441,8cyclictest16850irq/134-800000023:35:000
2244616995425,14cyclictest2316612-21ssh23:29:000
2244616995425,14cyclictest2316612-21ssh23:29:000
2244616995425,14cyclictest2268602-21ssh21:48:000
224462999533,9cyclictest2347712-21modprobe00:35:003
224462999533,11cyclictest2247682-21munin-run20:10:013
224462999533,11cyclictest2247682-21munin-run20:10:013
2244629995317,12cyclictest37-21ksoftirqd/323:10:023
224462999531,18cyclictest2327690-21ssh23:52:193
2244629995311,6cyclictest0-21swapper/320:45:003
2244621995336,12cyclictest364-21snmpd20:55:232
2244619995315,7cyclictest23-21ksoftirqd/122:50:011
2244619995315,10cyclictest23-21ksoftirqd/122:53:171
2244619995310,8cyclictest23-21ksoftirqd/122:31:091
224462999523,17cyclictest2271969-21ssh21:55:023
224462999523,17cyclictest2271969-21ssh21:55:023
224462999523,16cyclictest2251475-21ssh21:11:173
224462999523,16cyclictest2251475-21ssh21:11:163
2244629995225,23cyclictest37-21ksoftirqd/321:26:573
2244629995219,8cyclictest37-21ksoftirqd/319:45:023
2244621995240,9cyclictest2248980-21munin-run20:35:012
2244621995231,17cyclictest364-21snmpd20:16:402
2244621995226,21cyclictest1-21systemd20:55:012
224461999529,8cyclictest23-21ksoftirqd/100:20:041
2244619995237,11cyclictest2293416-21readlink22:40:001
2244619995214,11cyclictest2250795-21diskmemload23:46:051
2244616995238,12cyclictest12-21ksoftirqd/022:45:010
2244616995230,14cyclictest275-21dbus-daemon23:00:020
2244616995227,21cyclictest2259165-21ssh21:27:540
224461699522,36cyclictest2277054-21ssh22:05:280
224462999512,16cyclictest2313668-21ssh23:22:373
2244629995116,8cyclictest37-21ksoftirqd/300:13:233
2244621995143,4cyclictest2250823-21munin-run21:10:012
2244621995143,4cyclictest2246913-21munin-run19:55:012
2244621995137,10cyclictest2245339-21sshd19:23:342
224461999519,10cyclictest23-21ksoftirqd/122:28:081
2244619995111,8cyclictest23-21ksoftirqd/123:32:031
2244616995124,12cyclictest2337549-21ssh00:13:190
2244616995122,14cyclictest2257046-21ssh21:23:060
2244616995122,13cyclictest2321240-21ssh23:38:550
2244616995122,13cyclictest2321240-21ssh23:38:550
224462999504,9cyclictest3467602-21systemd-journal23:15:013
2244629995020,8cyclictest37-21ksoftirqd/321:02:343
2244629995020,8cyclictest37-21ksoftirqd/321:02:333
2244629995014,8cyclictest37-21ksoftirqd/320:04:083
2244621995039,7cyclictest2248441-21sshd20:20:332
2244621995036,11cyclictest364-21snmpd19:56:282
2244621995034,12cyclictest3467602-21systemd-journal20:07:322
2244621995034,12cyclictest3467602-21systemd-journal20:07:312
224461999509,13cyclictest23-21ksoftirqd/123:09:341
224461999508,9cyclictest23-21ksoftirqd/100:09:151
224461699504,3cyclictest131rcu_preempt00:30:230
2244616995043,4cyclictest16950irq/224-800000020:15:000
2244616995041,5cyclictest16950irq/224-800000019:20:000
2244616995039,6cyclictest16850irq/134-800000023:25:010
2244616995018,14cyclictest2312507-21ssh23:20:030
224462999492,16cyclictest2318039-21ssh23:31:483
2244629994918,5cyclictest2323426-21rm23:43:183
2244621994942,4cyclictest2249204-21sshd20:37:352
2244621994931,13cyclictest2247896-21sendmail-mta20:10:122
2244621994925,19cyclictest2245987-21cut19:35:032
2244616994917,28cyclictest12-21ksoftirqd/021:10:010
2244616994915,30cyclictest12-21ksoftirqd/000:25:040
2244616994915,30cyclictest12-21ksoftirqd/000:25:040
224462999483,14cyclictest2247145-21modprobe20:00:013
2244629994818,8cyclictest37-21ksoftirqd/320:56:523
2244629994817,9cyclictest37-21ksoftirqd/319:49:003
224461999488,15cyclictest0-21swapper/100:11:421
2244619994830,15cyclictest2268440-21sshd21:47:341
2244616994820,13cyclictest2279695-21ssh22:11:090
2244616994820,13cyclictest2279695-21ssh22:11:080
2244621994732,12cyclictest2246360-21sshd19:40:342
2244619994728,15cyclictest1-21systemd19:35:011
2244619994710,10cyclictest23-21ksoftirqd/123:15:581
2244616994738,5cyclictest16950irq/224-800000020:20:000
2244616994737,7cyclictest16950irq/224-800000020:00:020
2244616994730,13cyclictest2288013-21ssh22:28:440
2244616994728,16cyclictest2304976-21ssh23:04:310
2244616994728,16cyclictest2304976-21ssh23:04:300
2244616994719,7cyclictest0-21swapper/019:10:020
2244616994627,12cyclictest2264738-21cron21:40:000
2244616994622,20cyclictest12-21ksoftirqd/021:01:430
2244616994622,20cyclictest12-21ksoftirqd/021:01:430
2244616994620,11cyclictest2260393-21ssh21:30:210
2244616994615,27cyclictest0-21swapper/020:50:030
2244629994529,13cyclictest1-21systemd20:35:013
2244629994525,17cyclictest37-21ksoftirqd/320:15:013
2244621994533,9cyclictest364-21snmpd20:00:062
2244621994532,10cyclictest364-21snmpd19:11:112
224461999459,8cyclictest0-21swapper/122:23:431
224461999459,8cyclictest0-21swapper/122:23:431
224461999459,10cyclictest0-21swapper/121:55:561
224461999458,6cyclictest131rcu_preempt21:43:111
224461999458,6cyclictest131rcu_preempt21:43:111
224461999458,13cyclictest2326885-21ssh23:50:271
224461999457,5cyclictest131rcu_preempt22:19:211
2244619994532,10cyclictest3467602-21systemd-journal19:47:341
2244619994525,15cyclictest2339639-21ssh00:17:391
224461699458,34cyclictest12-21ksoftirqd/021:59:190
2244616994531,10cyclictest1-21systemd20:55:010
2244616994529,13cyclictest2252846-21ssh21:14:280
2244616994529,13cyclictest2252846-21ssh21:14:270
2244616994521,11cyclictest2290331-21ssh22:33:300
2244619994426,15cyclictest2249477-21sshd20:40:331
2244619994416,8cyclictest23-21ksoftirqd/122:58:371
2244616994414,6cyclictest12-21ksoftirqd/023:50:470
224461699441,39cyclictest0-21swapper/019:40:030
2244619994327,11cyclictest275-21dbus-daemon21:10:011
2244619994326,13cyclictest2247647-21sshd20:05:341
2244619994326,13cyclictest2247647-21sshd20:05:331
2244619994325,14cyclictest2345914-21ssh00:30:471
2244619994325,14cyclictest2250228-21sshd20:55:331
2244616994327,13cyclictest2274865-21ssh22:00:510
2244616994325,14cyclictest2247663-21sshd20:08:340
2244616994325,14cyclictest2247663-21sshd20:08:340
224461699432,37cyclictest2246392-21latency_hist19:45:010
224461999426,11cyclictest23-21ksoftirqd/120:29:331
2244619994230,9cyclictest3467602-21systemd-journal20:20:341
2244619994229,9cyclictest1-21systemd19:20:011
2244619994229,10cyclictest1-21systemd20:20:001
2244619994228,11cyclictest2248975-21cron20:35:001
2244619994223,15cyclictest2279740-21ssh22:11:161
2244619994223,15cyclictest2279740-21ssh22:11:161
224461999421,10cyclictest23-21ksoftirqd/122:07:331
2244616994224,14cyclictest1-21systemd20:45:010
224461699421,37cyclictest0-21swapper/019:35:010
224462999411,35cyclictest37-21ksoftirqd/319:15:013
2244619994128,10cyclictest2247418-21cron20:05:001
224461699415,33cyclictest12-21ksoftirqd/019:25:020
2244619994011,9cyclictest2245496-21latency_hist19:25:011
2244629993918,6cyclictest37-21ksoftirqd/319:22:533
2244621993920,14cyclictest2244560-21latency_hist19:10:022
2244616993926,6cyclictest309-21in:imuxsock20:40:000
2244619993826,9cyclictest23-21ksoftirqd/119:45:021
2244619993825,9cyclictest2245612-21munin-run19:30:011
2244619993825,10cyclictest2247118-21sshd19:56:341
2244619993825,10cyclictest2244677-21latency_hist19:10:021
2244619993823,11cyclictest3467602-21systemd-journal20:50:021
2244619993822,12cyclictest48-21kcompactd021:00:121
2244619993822,12cyclictest48-21kcompactd021:00:121
2244616993817,17cyclictest275-21dbus-daemon19:15:010
224461699381,10cyclictest1-21systemd19:48:340
224462999372,10cyclictest2244745-21grep19:10:023
2244619993722,11cyclictest2246863-21sshd19:50:341
2244616993723,11cyclictest12-21ksoftirqd/020:35:010
2244616993715,17cyclictest275-21dbus-daemon20:05:010
2244619993623,10cyclictest2247902-21sshd20:11:341
2244619993620,12cyclictest2244831-21/usr/sbin/munin19:15:021
2244616993622,10cyclictest364-21snmpd19:51:280
224461699361,31cyclictest0-21swapper/019:30:020
2244619993526,6cyclictest23-21ksoftirqd/120:55:031
2244619993521,9cyclictest48-21kcompactd020:36:541
2244616993525,7cyclictest12-21ksoftirqd/020:25:010
2244619993414,16cyclictest2246149-21sh19:40:001
2244616993322,9cyclictest12-21ksoftirqd/020:30:010
2244629993120,8cyclictest37-21ksoftirqd/320:45:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional