You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-06 - 21:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Mon Apr 06, 2026 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2311773996253,6cyclictest2481297-21munin-plugin-st11:55:000
2311773996022,16cyclictest12350irq/32-eth009:40:310
2311775995847,4cyclictest0-21swapper/210:25:352
2311775995847,4cyclictest0-21swapper/210:25:352
231177599541,5cyclictest171rcu_preempt10:30:312
2311775995335,15cyclictest2384757-21cron09:55:002
2311775995127,21cyclictest2489302-21cron12:05:012
2311774995143,5cyclictest2441254-21kworker/u8:8-xprtiod12:00:011
231177499510,23cyclictest28-21ksoftirqd/110:50:341
2311775995024,23cyclictest0-21swapper/209:25:012
2311774995040,6cyclictest2348613-21tr09:10:251
2311773994940,7cyclictest0-21swapper/009:35:010
2311775994822,5cyclictest421-21dbus-daemon09:43:002
2311774994838,7cyclictest1-21systemd07:10:011
2311774994836,7cyclictest1457231-21cups-browsed11:36:531
2311774994738,4cyclictest0-21swapper/110:20:361
2311774994736,7cyclictest2311772-21cyclictest07:45:331
2311775994626,15cyclictest0-21swapper/211:35:332
2311775994621,6cyclictest1051-21mta-sts-daemon12:13:252
2311775994618,5cyclictest2342118-21aten2.4-expect08:50:212
2311774994633,9cyclictest421-21dbus-daemon09:25:241
2311775994539,3cyclictest0-21swapper/209:45:362
2311775994539,3cyclictest0-21swapper/209:45:362
231177599452,22cyclictest2365409-21ssh09:30:362
2311775994520,3cyclictest0-21swapper/211:15:312
2311775994518,6cyclictest2423025-21unixbench_singl10:40:442
2311774994535,7cyclictest2316292-21systemd-run07:25:011
2311774994534,8cyclictest2344772-21sh09:00:011
2311774994523,16cyclictest271ktimers/108:15:331
2311773994514,28cyclictest12450irq/33-eth009:45:320
2311773994514,28cyclictest12450irq/33-eth009:45:320
2311774994437,4cyclictest0-21swapper/109:55:431
2311774994435,6cyclictest2493820-21aten_r2power_fr12:10:221
2311775994322,3cyclictest0-21swapper/211:55:392
2311775994322,3cyclictest0-21swapper/210:55:012
2311774994335,5cyclictest2489337-21systemd-run12:05:011
2311774994331,6cyclictest0-21swapper/109:20:351
2311774994329,10cyclictest2442664-21mailstats11:05:401
231177499432,29cyclictest2434501-21ssh10:55:381
2311775994232,7cyclictest1457231-21cups-browsed11:00:352
231177499422,8cyclictest171rcu_preempt10:15:461
2311773994235,4cyclictest2422594-21proc_pri10:40:390
2311775994135,3cyclictest0-21swapper/212:40:012
2311775994131,5cyclictest0-21swapper/210:10:342
231177599412,22cyclictest2418438-21ssh10:35:342
231177499413,32cyclictest2414566-21ssh10:30:351
2311773994134,4cyclictest2353223-21grep09:15:400
2311773994133,5cyclictest1457231-21cups-browsed10:24:160
2311775994034,3cyclictest0-21swapper/210:20:012
2311775994033,4cyclictest2322673-21kworker/u8:1-xprtiod08:25:362
2311775994033,4cyclictest2322673-21kworker/u8:1-xprtiod08:25:362
2311775994031,4cyclictest0-21swapper/211:20:322
2311775994030,7cyclictest2316292-21systemd-run07:25:012
2311774994033,3cyclictest0-21swapper/111:20:011
231177499403,30cyclictest2377859-21ssh09:45:381
231177499403,30cyclictest2377859-21ssh09:45:381
2311774994032,5cyclictest2389653-21uniq10:00:261
2311774994016,16cyclictest271ktimers/108:35:311
2311773994034,4cyclictest1-21systemd11:30:010
2311774993930,5cyclictest0-21swapper/108:25:361
2311774993930,5cyclictest0-21swapper/108:25:361
2311774993928,7cyclictest1457231-21cups-browsed10:45:101
2311773993933,4cyclictest2326744-21munin-run08:00:010
2311773993932,4cyclictest0-21swapper/011:20:260
2311775993830,5cyclictest2449270-21systemd-run11:15:012
2311775993830,5cyclictest2347791-21latency_hist09:10:022
2311775993829,6cyclictest2321093-21aten_r2power_fr07:40:222
2311774993830,5cyclictest0-21swapper/110:10:281
2311774993829,5cyclictest2501332-21munin-plugin-st12:20:011
2311774993827,8cyclictest2369910-21timerwakeupswit09:35:421
231177499382,30cyclictest2470584-21grep11:40:371
2311774993811,7cyclictest0-21swapper/107:35:301
2311773993830,5cyclictest1457231-21cups-browsed12:04:480
2311775993727,6cyclictest1457231-21cups-browsed11:28:382
231177499373,24cyclictest2347750-21diskmemload11:50:341
2311774993731,3cyclictest0-21swapper/107:20:011
2311774993729,5cyclictest2384787-21ssh09:55:011
2311774993728,5cyclictest2368343-21latency_hist09:35:021
2311774993728,5cyclictest2325239-21cron07:55:011
2311774993727,7cyclictest2462567-21/usr/sbin/munin11:30:391
2311774993726,7cyclictest421-21dbus-daemon09:15:071
2311774993710,21cyclictest421-21dbus-daemon08:50:321
2311773993732,3cyclictest2320740-21cron07:40:010
2311773993728,6cyclictest1457231-21cups-browsed12:30:400
2311773993724,11cyclictest2509443-21latency_hist12:30:020
2311773993719,12cyclictest12350irq/32-eth012:05:310
2311775993629,3cyclictest0-21swapper/210:05:022
2311775993626,6cyclictest420-21avahi-daemon11:53:482
2311775993626,6cyclictest420-21avahi-daemon09:14:122
2311775993625,8cyclictest2434137-21latency10:55:282
2311774993629,3cyclictest0-21swapper/112:30:321
2311774993628,5cyclictest2345288-21fschecks_time09:00:231
2311774993628,5cyclictest2340406-21latency_hist08:45:031
2311774993627,7cyclictest261-21systemd-journal11:50:011
2311774993627,6cyclictest2376521-21idleruntime-cro09:45:011
2311774993626,6cyclictest2461336-21latency_hist11:30:011
231177499362,26cyclictest2438363-21ssh11:00:321
2311773993627,7cyclictest2517393-21kthreadd12:40:010
2311775993529,4cyclictest2311772-21cyclictest09:35:432
2311775993527,5cyclictest2513364-21sh12:35:012
2311775993527,5cyclictest2344758-21sendmail09:00:012
2311775993526,6cyclictest2320800-21latency_hist07:40:022
2311774993526,6cyclictest2409722-21aten_r2power_cu10:25:211
2311774993526,6cyclictest2409722-21aten_r2power_cu10:25:211
2311773993527,5cyclictest420-21avahi-daemon11:37:370
2311773993524,8cyclictest2442272-21needreboot11:05:290
2311773993516,14cyclictest12350irq/32-eth010:50:310
231177599347,18cyclictest428-21mta-sts-daemon08:35:322
2311775993424,7cyclictest2445261-21munin-run11:10:022
2311774993427,4cyclictest0-21swapper/107:10:351
2311774993426,5cyclictest2327280-21fschecks_count08:00:251
2311774993423,8cyclictest695-21cron11:15:011
2311774993421,8cyclictest2347750-21diskmemload10:40:351
2311773993428,4cyclictest2331246-21cron08:15:000
2311773993427,4cyclictest2372393-21grep09:40:010
2311775993326,4cyclictest1051-21mta-sts-daemon09:15:082
2311775993324,6cyclictest2490764-21mailstats12:05:422
2311774993327,3cyclictest0-21swapper/110:35:331
2311774993313,5cyclictest171rcu_preempt10:05:321
231177399333,25cyclictest2498511-21ssh12:15:350
2311773993328,3cyclictest2317753-21cron07:30:000
2311773993328,3cyclictest2317753-21cron07:30:000
2311773993327,4cyclictest2390720-21head10:00:450
2311773993327,3cyclictest0-21swapper/010:45:380
2311773993327,3cyclictest0-21swapper/009:10:220
2311773993325,5cyclictest2409681-21/usr/sbin/munin10:25:240
2311773993325,5cyclictest2409681-21/usr/sbin/munin10:25:240
2311773993324,5cyclictest1457231-21cups-browsed09:50:310
2311773993324,4cyclictest0-21swapper/008:35:300
2311775993224,3cyclictest0-21swapper/210:45:362
2311775993221,6cyclictest0-21swapper/212:25:322
2311775993219,7cyclictest2462415-21sh11:30:332
2311774993226,3cyclictest2340481-21kworker/u8:4-xprtiod08:45:241
2311774993226,3cyclictest0-21swapper/112:35:261
2311774993226,3cyclictest0-21swapper/112:25:011
2311774993220,8cyclictest420-21avahi-daemon12:25:211
231177499321,24cyclictest2454526-21ssh11:20:391
231177399322,9cyclictest2418449-21ssh10:35:340
2311773993227,3cyclictest2360044-21munin-plugin-st09:25:010
2311773993226,4cyclictest2397926-21/usr/sbin/munin10:10:250
2311773993226,3cyclictest0-21swapper/011:15:420
2311773993222,7cyclictest2474188-21latency11:45:270
2311775993123,5cyclictest2473360-21latency_hist11:45:022
2311775993123,5cyclictest2346249-21cron09:05:012
2311775993122,7cyclictest2337264-21sh08:35:012
231177499312,21cyclictest1457231-21cups-browsed08:30:331
2311773993126,2cyclictest0-21swapper/012:10:390
2311773993125,3cyclictest0-21swapper/011:30:220
2311773993124,5cyclictest2323768-21systemd-run07:50:010
2311773993124,4cyclictest151ktimers/008:20:320
2311773993122,7cyclictest0-21swapper/011:45:010
231177399312,22cyclictest659-21snmpd09:55:360
2311773993121,6cyclictest0-21swapper/010:55:220
2311773993115,11cyclictest12350irq/32-eth010:30:320
231177599306,14cyclictest2361199-21ssh09:25:332
231177599303,21cyclictest2474414-21ssh11:45:322
2311775993023,3cyclictest0-21swapper/210:20:312
2311775993022,5cyclictest2316731-21cpuspeed07:25:222
2311775993022,5cyclictest2316731-21cpuspeed07:25:222
231177499304,22cyclictest2490519-21ssh12:05:351
2311774993022,5cyclictest2347889-21latency_hist09:10:021
2311773993025,3cyclictest2329865-21latency_hist08:10:020
2311773993025,2cyclictest0-21swapper/012:20:250
2311773993024,4cyclictest420-21avahi-daemon08:55:290
2311773993024,4cyclictest2347783-21ssh09:10:020
2311773993023,3cyclictest0-21swapper/009:25:380
2311773993019,8cyclictest12350irq/32-eth008:00:350
2311773993015,12cyclictest1457231-21cups-browsed11:10:400
2311775992922,5cyclictest1055-21mta-sts-daemon10:05:432
2311775992922,5cyclictest0-21swapper/212:20:242
2311775992922,4cyclictest1054-21mta-sts-daemon09:55:282
2311774992921,5cyclictest439-21systemd-logind08:25:011
2311773992923,3cyclictest0-21swapper/010:15:490
2311773992921,6cyclictest2482315-21missed_timers11:55:290
2311773992921,6cyclictest2345653-21ntp_kernel_pll_09:00:310
231177599286,7cyclictest171rcu_preempt08:15:332
2311775992820,3cyclictest0-21swapper/212:15:392
2311774992820,5cyclictest2319180-21unixbench_singl07:30:431
2311773992822,4cyclictest2314799-21systemd-run07:20:010
2311773992818,8cyclictest261-21systemd-journal08:19:090
2311773992814,9cyclictest12350irq/32-eth011:00:350
231177599273,15cyclictest428-21mta-sts-daemon08:00:312
2311775992721,4cyclictest2311772-21cyclictest08:10:352
231177399272,21cyclictest2393737-21diskstats10:05:240
2311775992620,3cyclictest0-21swapper/207:30:382
2311775992620,3cyclictest0-21swapper/207:20:012
231177499264,5cyclictest171rcu_preempt07:40:381
231177499262,19cyclictest1457231-21cups-browsed08:05:371
2311774992618,5cyclictest2331242-21cron08:15:001
2311774992618,5cyclictest2325473-21/usr/sbin/munin07:55:231
2311774992614,9cyclictest2317141-21ntp_kernel_pll_07:25:311
2311774992614,9cyclictest2317141-21ntp_kernel_pll_07:25:311
2311773992621,3cyclictest2313136-21unixbench_multi07:10:420
2311773992619,5cyclictest1457228-21cupsd07:50:240
2311775992520,2cyclictest0-21swapper/208:00:012
2311775992519,3cyclictest1054-21mta-sts-daemon07:45:292
2311775992519,3cyclictest0-21swapper/208:10:022
2311775992518,4cyclictest2340623-21aten_r2power_po08:45:222
2311773992519,4cyclictest0-21swapper/007:35:000
2311773992517,6cyclictest2340926-21latency08:45:270
2311775992418,3cyclictest0-21swapper/208:25:012
2311775992416,6cyclictest440-21thd07:50:332
231177399249,11cyclictest12350irq/32-eth008:25:360
231177399249,11cyclictest12350irq/32-eth008:25:360
2311773992418,4cyclictest2307894-21kworker/u8:12+xprtiod07:40:430
2311773992416,6cyclictest2343183-21unixbench_singl08:50:420
231177399240,17cyclictest151ktimers/007:20:320
231177599231,17cyclictest2312779-21ps07:10:372
231177699223,10cyclictest0-21swapper/312:10:323
231177599222,11cyclictest428-21mta-sts-daemon08:40:322
2311773992216,3cyclictest1051-21mta-sts-daemon08:40:340
231177699214,9cyclictest0-21swapper/309:25:353
231177699213,12cyclictest0-21swapper/310:35:363
231177399211,15cyclictest2336662-21ntpq08:30:330
231177699203,8cyclictest0-21swapper/311:55:313
231177699202,9cyclictest0-21swapper/311:10:393
231177699202,9cyclictest0-21swapper/310:30:363
231177699202,10cyclictest0-21swapper/312:20:353
231177699194,8cyclictest0-21swapper/311:40:303
231177699192,10cyclictest0-21swapper/310:10:343
231177599192,12cyclictest2311732-21/usr/sbin/munin07:10:012
231177699185,7cyclictest0-21swapper/310:05:353
231177699184,8cyclictest0-21swapper/311:00:393
231177699184,7cyclictest0-21swapper/311:50:373
231177699183,9cyclictest0-21swapper/311:35:303
231177699183,9cyclictest0-21swapper/310:20:303
231177699183,10cyclictest0-21swapper/312:15:353
231177699182,9cyclictest0-21swapper/312:30:333
231177699182,11cyclictest0-21swapper/310:00:313
231177699174,7cyclictest0-21swapper/309:15:323
231177699174,5cyclictest0-21swapper/307:10:333
231177699173,9cyclictest0-21swapper/309:50:313
231177699173,8cyclictest0-21swapper/309:35:353
231177699173,8cyclictest0-21swapper/309:30:313
231177699172,9cyclictest0-21swapper/310:50:323
231177699172,9cyclictest0-21swapper/310:45:353
231177699172,9cyclictest0-21swapper/309:55:293
231177699172,9cyclictest0-21swapper/309:20:303
231177699171,9cyclictest0-21swapper/311:20:193
231177699171,8cyclictest0-21swapper/311:30:333
231177399171,7cyclictest2311906-21latency_hist07:10:020
231177699163,8cyclictest0-21swapper/312:35:353
231177699163,8cyclictest0-21swapper/311:45:333
231177699163,8cyclictest0-21swapper/309:45:323
231177699163,8cyclictest0-21swapper/309:45:323
231177699162,9cyclictest0-21swapper/312:25:313
231177699162,9cyclictest0-21swapper/310:55:313
231177699162,9cyclictest0-21swapper/310:40:353
231177699162,9cyclictest0-21swapper/309:40:283
231177699162,10cyclictest0-21swapper/312:05:263
231177699162,10cyclictest0-21swapper/310:15:423
231177699161,9cyclictest0-21swapper/311:25:333
231177699161,6cyclictest0-21swapper/307:35:303
231177699161,10cyclictest0-21swapper/310:25:383
231177699161,10cyclictest0-21swapper/310:25:383
231177699153,8cyclictest0-21swapper/312:00:333
231177699153,7cyclictest0-21swapper/309:10:313
231177699152,9cyclictest0-21swapper/311:05:323
231177699151,7cyclictest0-21swapper/308:40:353
231177699143,7cyclictest0-21swapper/311:15:333
231177699143,7cyclictest0-21swapper/308:50:323
231177699142,8cyclictest0-21swapper/309:10:013
231177699142,7cyclictest0-21swapper/309:00:293
231177699142,7cyclictest0-21swapper/309:00:013
231177699142,6cyclictest0-21swapper/308:10:303
231177699142,6cyclictest0-21swapper/308:00:333
231177699131,7cyclictest0-21swapper/308:40:023
231177699131,7cyclictest0-21swapper/308:25:253
231177699131,7cyclictest0-21swapper/308:25:253
231177699131,6cyclictest0-21swapper/308:20:253
231177699131,6cyclictest0-21swapper/307:25:303
231177699131,6cyclictest0-21swapper/307:25:303
231177699124,5cyclictest0-21swapper/307:55:343
231177699122,6cyclictest0-21swapper/307:45:313
231177699121,7cyclictest0-21swapper/307:40:433
231177699121,7cyclictest0-21swapper/307:15:323
231177699121,6cyclictest0-21swapper/307:20:373
231177699112,6cyclictest0-21swapper/308:30:333
231177699112,6cyclictest0-21swapper/308:15:213
231177699112,6cyclictest0-21swapper/307:50:223
231177699111,7cyclictest0-21swapper/307:30:353
231177699111,6cyclictest0-21swapper/308:45:313
231177699111,6cyclictest0-21swapper/308:05:083
231177699101,5cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional