You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-08 - 14:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Wed Apr 08, 2026 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2308310996453,7cyclictest2352494-21ssh09:20:011
2308311995924,19cyclictest2454085-21aten_r2power_fr11:30:222
2308311995645,8cyclictest1056-21mta-sts-daemon07:20:362
2308310995540,12cyclictest2385071-21sendmail10:00:011
2308310995540,12cyclictest2385071-21sendmail10:00:011
2308311995222,15cyclictest2339666-21unixbench_multi08:50:422
2308309995226,23cyclictest2421641-21aten_r2power_vo10:45:220
2308309995124,4cyclictest2374717-21head09:45:460
2308310995039,7cyclictest2364716-21munin-plugin-st09:35:011
2308309995029,18cyclictest2432764-21latency_hist11:00:020
2308309994925,22cyclictest2349046-21cpuspeed09:15:230
230831199487,24cyclictest2504740-21sh12:34:472
230831199487,24cyclictest2504740-21sh12:34:472
2308309994829,15cyclictest0-21swapper/009:13:540
2308309994823,3cyclictest0-21swapper/011:45:320
2308309994814,8cyclictest2336038-21/usr/sbin/munin08:40:280
2308309994737,6cyclictest151ktimers/009:30:340
2308309994722,22cyclictest2444709-21timerandwakeup11:15:440
2308311994638,4cyclictest0-21swapper/209:30:012
230831199462,36cyclictest2318180-21ntpq07:40:332
2308311994620,21cyclictest0-21swapper/211:25:362
2308311994620,21cyclictest0-21swapper/211:25:362
230831199461,40cyclictest2330213-21ntpq08:20:332
2308309994622,4cyclictest0-21swapper/012:05:010
230831199452,35cyclictest2321219-21ntpq07:50:332
2308310994537,4cyclictest0-21swapper/111:50:331
2308310994535,6cyclictest421-21dbus-daemon10:50:011
2308309994537,6cyclictest2417098-21sendmail10:40:010
2308309994535,4cyclictest2326270-21kworker/u8:0-nfsiod08:13:590
2308309994519,5cyclictest1431064-21cups-browsed09:51:000
230830999451,13cyclictest2386237-21awk10:00:330
2308311994436,5cyclictest2356564-21munin-plugin-st09:25:012
2308311994434,6cyclictest2497036-21latency_hist12:25:032
2308311994434,6cyclictest2497036-21latency_hist12:25:032
2308310994437,3cyclictest0-21swapper/110:20:011
2308310994436,5cyclictest2338316-21systemd-run08:50:011
2308310994431,9cyclictest2381032-21sh09:55:021
2308310994423,4cyclictest171rcu_preempt07:30:371
2308309994437,5cyclictest2493341-21aten_r2power_en12:20:210
2308309994437,5cyclictest2493341-21aten_r2power_en12:20:210
230830999442,19cyclictest2454710-21ssh11:30:330
2308309994421,18cyclictest2342820-21kworker/u8:8+rpciod10:05:380
2308311994333,4cyclictest0-21swapper/212:05:372
2308311994325,15cyclictest2508855-21cron12:40:002
230831199432,34cyclictest2336149-21ntpq08:40:302
230831199431,35cyclictest2316776-21ntpq07:35:352
230831099434,27cyclictest2454805-21ssh11:30:371
230830999433,15cyclictest428-21mta-sts-daemon09:40:370
2308309994320,14cyclictest12350irq/32-eth009:35:320
2308311994232,6cyclictest421-21dbus-daemon09:14:392
2308311994231,7cyclictest420-21avahi-daemon09:58:092
2308311994231,7cyclictest420-21avahi-daemon09:58:092
2308311994229,10cyclictest2444177-21missed_timers11:15:292
2308311994228,10cyclictest2365615-21latency09:35:292
230831199422,36cyclictest2474158-21ntpq11:55:362
230831199422,35cyclictest2325642-21ntpq08:05:292
230831199422,35cyclictest2325642-21ntpq08:05:292
230831199421,2cyclictest0-21swapper/212:25:472
230831199421,2cyclictest0-21swapper/212:25:472
230831199421,10cyclictest2310771-21ntpq07:15:372
2308310994233,5cyclictest2461108-21munin-run11:40:001
2308310994232,6cyclictest1431061-21cupsd08:30:241
2308309994217,6cyclictest1431064-21cups-browsed11:56:490
2308311994124,14cyclictest0-21swapper/210:35:012
2308311994124,13cyclictest2327498-21xargs08:10:422
230831199412,35cyclictest2490129-21ntpq12:15:382
230831199412,35cyclictest2333149-21ntpq08:30:302
230831199412,34cyclictest2437438-21ntpq11:05:372
230831199412,33cyclictest2324199-21ntpq08:00:332
2308311994120,18cyclictest0-21swapper/209:55:012
230831199411,9cyclictest2460082-21ssh11:37:532
230831199411,33cyclictest2315244-21ntpq07:30:342
230831199411,10cyclictest2351550-21ssh09:18:182
230831199410,10cyclictest2401537-21ssh10:20:192
2308310994135,3cyclictest0-21swapper/112:00:371
2308310994133,5cyclictest2494593-21grep12:20:441
2308310994133,5cyclictest2494593-21grep12:20:441
2308310994132,6cyclictest2485689-21if_wlan012:10:271
2308310994129,9cyclictest2342193-21ntp_kernel_pll_09:00:341
2308310994129,9cyclictest2342193-21ntp_kernel_pll_09:00:341
2308309994118,6cyclictest2308314-21systemd-run07:10:010
2308309994112,9cyclictest12350irq/32-eth011:00:340
2308309994112,9cyclictest12350irq/32-eth011:00:340
230831199403,32cyclictest2440732-21ntpq11:10:312
2308311994030,6cyclictest2481112-21latency_hist12:05:032
230831199402,34cyclictest2337678-21ntpq08:45:322
230831199402,30cyclictest2466093-21ssh11:45:312
230831199402,28cyclictest2361838-21ssh09:30:352
2308311994021,15cyclictest2406865-21grep10:25:472
230831199401,4cyclictest2370071-21ntpq09:40:352
230831199401,4cyclictest2328644-21ntpq08:15:312
230831199401,3cyclictest2398263-21ntpq10:15:322
230831199401,35cyclictest2343365-21if_err_eth009:05:262
230831199401,35cyclictest2343365-21if_err_eth009:05:262
230831199401,34cyclictest2396236-21ssh10:13:162
230831199401,11cyclictest2485954-21ntpq12:10:312
230831199401,11cyclictest2322766-21ntpq07:55:362
230831199401,10cyclictest2447771-21ntpq11:20:302
230831199400,10cyclictest2427410-21ssh10:51:582
230831099404,27cyclictest2366027-21ssh09:35:401
2308310994035,2cyclictest0-21swapper/110:40:011
2308310994029,7cyclictest2345775-21mailstats09:10:421
2308310994028,9cyclictest2358108-21switchtime09:25:431
2308309994032,4cyclictest0-21swapper/010:10:330
2308309994020,3cyclictest0-21swapper/010:15:440
230831199391,9cyclictest2434846-21ssh11:01:462
230831199391,9cyclictest2434846-21ssh11:01:462
230831199391,9cyclictest2389417-21ssh10:05:092
230831199391,5cyclictest2340642-21ntpq08:55:302
230831199391,5cyclictest2340642-21ntpq08:55:302
230831199391,4cyclictest2462372-21ntp_states11:40:372
230831199391,33cyclictest2414304-21ntpq10:35:352
230831199391,13cyclictest2470228-21ntpq11:50:382
230831199391,11cyclictest2376917-21ssh09:49:572
230831199391,10cyclictest2342192-21/usr/sbin/munin09:00:332
230831199391,10cyclictest2342192-21/usr/sbin/munin09:00:332
230831199390,9cyclictest2421066-21ssh10:45:002
2308310993929,6cyclictest420-21avahi-daemon10:42:461
2308309993923,8cyclictest12350irq/32-eth010:50:330
230831199381,4cyclictest2331677-21ntpq08:25:332
230831199381,34cyclictest2430949-21ssh10:56:172
230831199381,32cyclictest2313306-21diskstats07:25:252
230831199381,10cyclictest2387292-21ssh10:01:282
2308310993832,3cyclictest0-21swapper/110:25:461
2308310993830,5cyclictest1-21systemd10:35:021
2308310993829,6cyclictest421-21dbus-daemon10:50:411
230831099381,29cyclictest2466104-21sh11:45:311
2308309993829,7cyclictest2492863-21plymouth12:20:010
230831199371,8cyclictest2423035-21ssh10:46:052
230831199371,31cyclictest2334318-21/usr/sbin/munin08:35:262
230831199370,10cyclictest2309194-21ntpq07:10:332
2308310993730,3cyclictest0-21swapper/112:40:001
2308310993729,5cyclictest2497415-21aten_r2power_po12:25:231
2308310993729,5cyclictest2497415-21aten_r2power_po12:25:231
2308310993729,4cyclictest0-21swapper/112:19:001
2308310993728,6cyclictest2373979-21needreboot09:45:321
2308310993727,7cyclictest421-21dbus-daemon11:13:451
230831099372,4cyclictest171rcu_preempt07:35:351
230831099372,28cyclictest2474111-21ssh11:55:341
230831099372,28cyclictest2353782-21ssh09:20:381
2308309993727,7cyclictest2465001-21cron11:45:000
2308309993717,4cyclictest1051-21mta-sts-daemon10:20:260
230831199361,4cyclictest2319532-21memory07:45:272
2308310993628,4cyclictest0-21swapper/111:20:311
2308310993626,6cyclictest261-21systemd-journal07:45:011
2308310993617,15cyclictest271ktimers/108:25:321
2308309993629,3cyclictest0-21swapper/011:10:380
2308309993624,9cyclictest0-21swapper/011:35:440
2308310993527,5cyclictest2330921-21latency_hist08:25:021
2308310993527,5cyclictest2324794-21cron08:05:011
2308310993527,4cyclictest1051-21mta-sts-daemon10:10:361
2308310993527,4cyclictest0-21swapper/110:05:271
2308310993526,6cyclictest421-21dbus-daemon12:33:291
2308310993526,6cyclictest421-21dbus-daemon12:33:291
2308310993526,6cyclictest2311343-21/usr/sbin/munin07:20:011
2308309993529,3cyclictest0-21swapper/008:55:010
2308310993427,4cyclictest2434510-21kworker/u8:1-nfsiod11:20:011
2308310993425,6cyclictest271ktimers/108:40:311
2308310993425,6cyclictest2341322-21systemd-run09:00:011
2308310993425,6cyclictest2341322-21systemd-run09:00:011
2308309993428,4cyclictest2340308-21fschecks_time08:55:240
2308309993428,4cyclictest2340308-21fschecks_time08:55:240
2308309993428,3cyclictest0-21swapper/011:20:380
2308310993325,5cyclictest261-21systemd-journal10:25:011
2308310993323,7cyclictest1431061-21cupsd10:58:541
2308309993326,5cyclictest420-21avahi-daemon12:06:200
2308309993325,5cyclictest2485252-21/usr/sbin/munin12:10:230
2308309993321,10cyclictest2498316-21mailstats12:25:420
2308309993321,10cyclictest2498316-21mailstats12:25:420
2308309993319,10cyclictest421-21dbus-daemon11:08:040
230830999331,26cyclictest2418393-21ssh10:40:370
230831199321,4cyclictest2308323-21munin-run07:10:012
2308310993224,5cyclictest2322311-21fschecks_time07:55:251
2308310993224,5cyclictest2309682-21grep07:10:441
2308310993223,6cyclictest0-21swapper/112:10:021
2308309993227,2cyclictest0-21swapper/011:55:020
2308309993225,5cyclictest2410840-21unixbench_multi10:30:440
2308309993223,6cyclictest420-21avahi-daemon07:20:430
2308309993219,11cyclictest0-21swapper/012:35:010
2308309993219,11cyclictest0-21swapper/012:35:010
2308309993217,5cyclictest2344317-21ssh09:10:010
2308309993217,5cyclictest2344317-21ssh09:10:010
2308309993217,10cyclictest12350irq/32-eth009:25:310
2308310993123,5cyclictest2389116-21cron10:05:001
2308310993123,4cyclictest0-21swapper/109:40:391
230831099311,5cyclictest171rcu_preempt07:20:341
2308309993122,7cyclictest2508792-21sshd12:39:540
2308310993024,3cyclictest0-21swapper/107:50:011
2308310993023,3cyclictest0-21swapper/111:40:371
2308310993022,5cyclictest2436260-21cron11:05:001
2308310993022,5cyclictest2436260-21cron11:05:001
2308310993022,5cyclictest2339339-21processes08:50:381
2308309993020,8cyclictest2451008-21latency11:25:280
2308309993020,8cyclictest2451008-21latency11:25:280
230831099298,17cyclictest2344299-21diskmemload11:05:331
230831099292,5cyclictest171rcu_preempt07:50:371
2308310992923,3cyclictest0-21swapper/109:10:001
2308310992923,3cyclictest0-21swapper/109:10:001
230831099291,23cyclictest2451231-21rm11:25:331
230831099291,23cyclictest2451231-21rm11:25:331
2308309992924,3cyclictest2338298-21munin-plugin-st08:50:010
2308309992923,4cyclictest420-21avahi-daemon08:15:430
2308309992823,3cyclictest2342793-21cron09:05:000
2308309992823,3cyclictest2342793-21cron09:05:000
2308309992822,3cyclictest0-21swapper/009:55:440
2308309992822,3cyclictest0-21swapper/009:55:440
2308309992821,5cyclictest151ktimers/007:10:370
2308309992820,5cyclictest2320486-21latency_hist07:50:020
2308309992815,9cyclictest2405835-21/usr/sbin/munin10:25:260
230831099275,15cyclictest428-21mta-sts-daemon07:25:371
2308310992719,5cyclictest2325100-21aten_r2power_en08:05:211
2308310992719,5cyclictest2325100-21aten_r2power_en08:05:211
2308310992719,5cyclictest1051-21mta-sts-daemon08:10:221
230830999271,21cyclictest2353583-21ssh09:20:310
2308310992621,3cyclictest1792018-21kworker/u8:4-nfsiod07:10:011
2308310992616,8cyclictest2334631-21ntp_kernel_err08:35:311
2308309992620,3cyclictest2324160-21ntpq08:00:310
2308309992616,8cyclictest151ktimers/008:05:330
2308309992616,8cyclictest151ktimers/008:05:330
230830999251,9cyclictest2332786-21df08:30:240
2308309992518,5cyclictest1431064-21cups-browsed08:26:100
2308309992517,5cyclictest2315814-21systemd-run07:35:000
2308309992516,6cyclictest0-21swapper/007:35:340
2308310992418,3cyclictest0-21swapper/108:15:431
2308309992418,4cyclictest2330803-21cron08:25:010
2308309992418,4cyclictest0-21swapper/008:00:010
2308309992411,9cyclictest1055-21mta-sts-daemon08:35:310
230830999237,12cyclictest421-21dbus-daemon07:18:310
230830999232,16cyclictest2318217-21ntpq07:40:350
2308309992216,4cyclictest2311335-21kworker/u8:17+nfsiod07:30:010
230830999214,13cyclictest12350irq/32-eth007:50:360
230831299181,9cyclictest0-21swapper/312:00:303
230831299163,8cyclictest0-21swapper/311:25:323
230831299163,8cyclictest0-21swapper/311:25:323
230831299161,8cyclictest0-21swapper/312:20:333
230831299161,8cyclictest0-21swapper/312:20:333
230831299161,8cyclictest0-21swapper/312:15:313
230831299161,7cyclictest0-21swapper/311:15:313
230831299161,7cyclictest0-21swapper/307:30:353
230831299161,6cyclictest0-21swapper/308:35:343
230831299152,8cyclictest0-21swapper/311:05:333
230831299152,8cyclictest0-21swapper/311:00:343
230831299152,8cyclictest0-21swapper/311:00:343
230831299152,7cyclictest0-21swapper/311:50:343
230831299151,8cyclictest0-21swapper/312:30:333
230831299151,8cyclictest0-21swapper/312:30:333
230831299151,8cyclictest0-21swapper/309:55:313
230831299151,8cyclictest0-21swapper/309:55:313
230831299151,8cyclictest0-21swapper/309:35:383
230831299151,7cyclictest0-21swapper/308:25:333
230831299151,7cyclictest0-21swapper/307:45:323
230831299151,6cyclictest0-21swapper/307:25:333
230831299151,6cyclictest0-21swapper/307:20:293
230831299143,8cyclictest0-21swapper/310:00:333
230831299142,6cyclictest0-21swapper/308:13:573
230831299141,8cyclictest0-21swapper/310:39:543
230831299141,8cyclictest0-21swapper/308:50:303
230831299141,7cyclictest0-21swapper/311:35:323
230831299141,7cyclictest0-21swapper/309:30:363
230831299141,7cyclictest0-21swapper/309:25:333
230831299141,7cyclictest0-21swapper/308:30:333
230831299141,5cyclictest0-21swapper/308:45:303
230831299141,5cyclictest0-21swapper/308:40:313
230831299141,5cyclictest0-21swapper/307:15:303
230831299141,5cyclictest0-21swapper/307:10:323
230831299140,7cyclictest0-21swapper/308:15:363
230831299140,6cyclictest0-21swapper/309:05:343
230831299140,6cyclictest0-21swapper/309:05:343
230831299135,6cyclictest0-21swapper/309:15:393
230831299133,6cyclictest0-21swapper/309:45:363
230831299132,7cyclictest0-21swapper/310:40:343
230831299132,7cyclictest0-21swapper/310:05:403
230831299132,6cyclictest0-21swapper/312:05:223
230831299132,6cyclictest0-21swapper/309:20:273
230831299131,8cyclictest0-21swapper/309:40:313
230831299131,7cyclictest0-21swapper/311:40:313
230831299131,7cyclictest0-21swapper/310:50:303
230831299131,7cyclictest0-21swapper/310:15:343
230831299131,7cyclictest0-21swapper/310:10:333
230831299131,7cyclictest0-21swapper/309:10:313
230831299131,6cyclictest0-21swapper/309:00:313
230831299131,6cyclictest0-21swapper/309:00:313
230831299131,6cyclictest0-21swapper/308:00:323
230831299131,6cyclictest0-21swapper/307:35:303
230831299122,6cyclictest0-21swapper/311:45:253
230831299122,6cyclictest0-21swapper/309:00:003
230831299122,6cyclictest0-21swapper/309:00:003
230831299121,8cyclictest0-21swapper/310:30:223
230831299121,7cyclictest0-21swapper/312:10:273
230831299121,7cyclictest0-21swapper/311:55:343
230831299121,7cyclictest0-21swapper/310:55:303
230831299121,7cyclictest0-21swapper/310:25:413
230831299121,7cyclictest0-21swapper/310:20:343
230831299121,5cyclictest0-21swapper/308:20:303
230831299112,6cyclictest0-21swapper/311:30:363
230831299112,5cyclictest0-21swapper/312:35:253
230831299111,7cyclictest0-21swapper/312:25:233
230831299111,7cyclictest0-21swapper/312:25:233
230831299111,6cyclictest0-21swapper/310:45:223
230831299111,6cyclictest0-21swapper/309:50:253
230831299111,5cyclictest0-21swapper/307:55:303
230831299102,6cyclictest0-21swapper/311:20:223
230831299102,5cyclictest0-21swapper/311:10:253
230831299102,5cyclictest0-21swapper/308:05:303
230831299102,5cyclictest0-21swapper/308:05:303
23083129993,5cyclictest0-21swapper/307:50:313
23083129991,4cyclictest0-21swapper/307:45:023
23083129990,3cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional