You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-30 - 19:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Tue Jun 30, 2026 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3484033996660,4cyclictest3589501-21kworker/u8:9-nfsiod11:27:502
3484031996146,9cyclictest151ktimers/010:20:350
3484031996141,16cyclictest3604432-40mandb11:27:490
3484032995542,5cyclictest0-21swapper/110:50:371
3484033995042,5cyclictest3553535-21kworker/u8:1-xprtiod11:40:012
3484033994734,10cyclictest3527527-21mailstats09:20:442
3484031994620,19cyclictest12650irq/32-eth009:50:450
3484031994620,19cyclictest12650irq/32-eth009:50:450
3484031994618,11cyclictest3598842-21sh11:19:110
3484031994614,13cyclictest3640359-21ssh12:26:300
3484033994535,6cyclictest3497533-21systemd-run07:55:012
3484032994522,16cyclictest271ktimers/111:20:461
3484032994436,5cyclictest3496038-21systemd-run07:50:011
3484031994427,12cyclictest3501376-21ntpq08:05:330
3484031994418,9cyclictest3528298-21sh09:22:010
3484033994336,4cyclictest3484052-21latency_hist07:10:012
3484033994336,4cyclictest3484052-21latency_hist07:10:012
3484033994333,7cyclictest2679292-21cups-browsed09:05:302
3484031994336,5cyclictest567-21gdbus08:40:430
3484031994331,5cyclictest3554725-21ntpq10:05:310
3484031994328,10cyclictest3498434-21ntpq07:55:350
3484031994319,8cyclictest3577313-21sh10:42:580
3484033994234,5cyclictest3591710-21unixbench_singl11:05:452
3484032994233,6cyclictest487-21rs:main1
348403299423,30cyclictest3612389-21ssh11:40:401
3484031994236,4cyclictest3626770-21aten_r2power_fr12:05:220
3484031994229,10cyclictest3568941-21cron10:30:000
3484031994229,10cyclictest3568941-21cron10:30:000
3484031994226,10cyclictest3615452-21ps11:45:400
3484031994224,6cyclictest3645595-21ntpq12:35:410
3484031994218,7cyclictest3520022-21diskmemload10:02:520
3484031994217,9cyclictest3496611-21if_eth007:50:280
3484033994134,4cyclictest3494519-21cron07:45:012
3484033994130,5cyclictest0-21swapper/207:10:382
3484033994130,5cyclictest0-21swapper/207:10:382
3484033994129,9cyclictest3513332-21needreboot08:45:302
3484032994133,5cyclictest3513049-21fschecks_time08:45:261
3484032994132,4cyclictest0-21swapper/110:45:371
3484032994130,8cyclictest694-21cron07:15:001
3484032994130,8cyclictest694-21cron07:15:001
3484032994127,11cyclictest3540109-21unixbench_multi09:40:471
3484032994127,11cyclictest3540109-21unixbench_multi09:40:471
3484031994129,9cyclictest265-21systemd-journal09:15:010
3484031994127,8cyclictest3504471-21ntpq08:15:380
3484031994126,12cyclictest3516423-21ntpq08:55:400
3484031994125,12cyclictest3530330-21ntpq09:25:370
3484031994122,16cyclictest3519407-21ntpq09:05:370
3484031994118,17cyclictest0-21swapper/009:00:320
3484031994117,9cyclictest3535087-21sh09:34:060
3484031994114,8cyclictest3621295-21ntpq11:55:330
348403399404,28cyclictest958-21sendmail-mta07:45:342
3484033994033,4cyclictest3547641-21ssh09:55:012
3484033994033,4cyclictest3547641-21ssh09:55:012
3484033994024,10cyclictest351ktimers/209:00:362
348403399402,31cyclictest3554824-21ssh10:05:362
3484032994032,5cyclictest3620713-21sh11:55:241
3484032994032,5cyclictest3617376-21latency_hist11:50:021
3484032994032,4cyclictest0-21swapper/112:00:241
3484031994027,8cyclictest3510366-21ntpq08:35:320
3484031994027,8cyclictest3495367-21ntpq07:45:310
3484031994026,9cyclictest3618268-21ntpq11:50:320
3484031994025,12cyclictest3499874-21ntpq08:00:320
3484031994018,17cyclictest0-21swapper/008:10:350
3484031994018,10cyclictest3520022-21diskmemload12:32:390
3484031994016,7cyclictest3520022-21diskmemload12:01:440
3484031994015,9cyclictest3583327-21ssh10:52:540
3484031994015,9cyclictest3569405-21sh10:30:250
3484031994015,17cyclictest0-21swapper/007:20:320
3484031994015,17cyclictest0-21swapper/007:20:320
3484033993931,5cyclictest3579075-21ssh10:45:372
3484033993931,5cyclictest3510954-21unixbench_singl08:35:462
3484033993929,6cyclictest1003-21mta-sts-daemon10:31:282
3484033993929,6cyclictest0-21swapper/212:30:422
3484033993928,7cyclictest3536064-21fschecks_time09:35:272
3484032993931,3cyclictest0-21swapper/110:25:001
3484032993930,6cyclictest3638446-21cron12:25:011
3484032993929,6cyclictest3502032-21systemd-run08:10:011
3484032993929,6cyclictest3496040-21kworker/u8:16+xprtiod09:20:011
3484032993929,4cyclictest0-21swapper/111:05:341
3484032993928,7cyclictest421-21dbus-daemon09:27:541
3484031993929,4cyclictest3507380-21ntpq08:25:310
3484031993926,8cyclictest3514975-21ntpq08:50:380
3484031993926,8cyclictest3513394-21ntpq08:45:320
3484031993926,8cyclictest3508963-21ntpq08:30:350
3484031993924,11cyclictest3489378-21ntpq07:25:320
3484031993919,6cyclictest3608025-21ssh11:34:350
3484031993916,7cyclictest3520022-21diskmemload10:55:080
3484031993915,10cyclictest3543380-21sh09:46:150
3484031993915,10cyclictest3543380-21sh09:46:150
3484033993830,5cyclictest3500468-21sed08:00:452
3484033993829,6cyclictest3526207-21sh09:20:012
3484032993830,5cyclictest3535390-21cron09:35:011
3484032993830,3cyclictest0-21swapper/110:00:011
3484032993826,9cyclictest3496884-21ntp_kernel_pll_07:50:341
3484031993831,4cyclictest3579541-21unixbench-2d10:45:460
3484031993826,8cyclictest3572959-21ntpq10:35:370
3484031993823,12cyclictest3484926-21ntpq07:10:340
3484031993823,12cyclictest3484926-21ntpq07:10:340
3484031993819,15cyclictest3540297-21ssh09:41:090
3484031993819,15cyclictest3540297-21ssh09:41:090
3484031993815,8cyclictest3558578-21ssh10:11:060
3484033993731,2cyclictest0-21swapper/210:25:012
3484033993729,5cyclictest421-21dbus-daemon08:25:012
3484033993729,5cyclictest421-21dbus-daemon08:25:012
3484033993728,6cyclictest3532353-21systemd-run09:30:012
348403299375,16cyclictest1001-21mta-sts-daemon08:35:391
3484032993729,5cyclictest3608171-21munin-run11:35:011
3484032993729,4cyclictest3496040-21kworker/u8:16-nfsiod10:30:011
3484032993729,4cyclictest3496040-21kworker/u8:16-nfsiod10:30:011
3484032993723,8cyclictest3588101-21sh11:00:361
3484031993728,6cyclictest3486465-21ntp_states07:15:380
3484031993727,7cyclictest3561275-21/usr/sbin/munin10:15:460
3484031993727,7cyclictest3561275-21/usr/sbin/munin10:15:460
3484031993725,9cyclictest3613121-21ssh11:41:300
3484031993725,7cyclictest3492493-21ntpq07:35:390
3484031993724,10cyclictest3600237-21/usr/sbin/munin11:20:400
3484031993724,10cyclictest3491151-21sensors_volt07:30:460
3484031993719,15cyclictest3537070-21uptime09:35:450
3484031993715,8cyclictest3520022-21diskmemload12:22:080
3484031993714,7cyclictest3520022-21diskmemload12:12:200
348403399367,13cyclictest351ktimers/212:05:382
3484033993627,6cyclictest3553841-21latency_hist10:05:022
3484032993628,5cyclictest3620293-21systemd-run11:55:011
3484032993628,4cyclictest0-21swapper/109:00:381
3484032993626,6cyclictest3561370-21timerwakeupswit10:15:471
3484032993626,6cyclictest3561370-21timerwakeupswit10:15:471
3484032993625,8cyclictest3499831-21needreboot08:00:301
3484031993630,4cyclictest3550651-21munin-plugin-st10:00:000
3484031993626,7cyclictest3493881-21ntpq07:40:320
3484031993624,9cyclictest3523747-21/usr/sbin/munin09:15:270
3484031993622,11cyclictest3589917-21ssh11:04:320
3484031993618,14cyclictest3591151-21ntpq11:05:350
3484031993612,11cyclictest3594134-21/usr/sbin/munin11:10:370
3484033993529,3cyclictest0-21swapper/212:00:012
3484033993528,3cyclictest0-21swapper/211:50:022
3484033993527,5cyclictest3548964-21mailstats09:55:472
3484033993523,9cyclictest3487019-21munin-plugin-st07:20:002
3484032993528,4cyclictest3599120-21munin-run11:20:011
3484032993526,5cyclictest1-21systemd08:00:011
3484032993525,7cyclictest439-21systemd-logind08:25:011
3484032993525,7cyclictest439-21systemd-logind08:25:011
3484031993526,6cyclictest3506003-21ntp_states08:20:380
3484031993526,6cyclictest3506003-21ntp_states08:20:380
3484031993523,8cyclictest3633429-21ntpq12:15:320
348403399343,28cyclictest3585056-21ssh10:55:342
3484033993426,5cyclictest3647498-21munin-plugin-st12:40:012
3484033993424,7cyclictest2679292-21cups-browsed12:18:032
3484033993423,8cyclictest3562867-21sendmail10:20:002
3484033993423,8cyclictest3562867-21sendmail10:20:002
3484033993423,8cyclictest3492321-21needreboot07:35:312
3484032993426,5cyclictest3587028-21idleruntime-cro11:00:011
3484032993426,5cyclictest3527171-21needreboot09:20:311
3484032993426,4cyclictest1001-21mta-sts-daemon10:42:131
348403299342,24cyclictest3603214-21ssh11:25:371
3484031993425,6cyclictest3609167-21ntpq11:35:310
348403399333,24cyclictest3539582-21sh09:40:362
348403399333,24cyclictest3539582-21sh09:40:362
3484033993326,4cyclictest1003-21mta-sts-daemon11:20:212
3484033993325,5cyclictest3491400-21unixbench_multi07:30:482
3484033993324,6cyclictest420-21avahi-daemon09:48:512
3484033993324,6cyclictest420-21avahi-daemon09:48:512
3484033993319,8cyclictest1000-21mta-sts-daemon11:15:372
3484032993324,6cyclictest3626990-21uniq12:05:251
3484032993320,8cyclictest0-21swapper/110:10:021
348403299331,27cyclictest3594153-21ssh11:10:381
348403399327,21cyclictest351ktimers/212:00:372
348403399327,12cyclictest36-21ksoftirqd/208:55:012
348403399326,9cyclictest36-21ksoftirqd/208:40:382
3484033993226,3cyclictest0-21swapper/211:55:022
3484033993225,4cyclictest3516993-21cron09:00:012
3484033993223,5cyclictest3503533-21systemd-run08:15:012
3484033993223,4cyclictest0-21swapper/211:10:392
348403399322,21cyclictest3576068-21ssh10:40:372
3484032993225,4cyclictest0-21swapper/108:35:021
3484032993224,5cyclictest3640028-21unixbench-2d12:25:461
3484032993223,6cyclictest420-21avahi-daemon12:33:491
3484032993223,4cyclictest0-21swapper/112:12:081
3484032993222,7cyclictest421-21dbus-daemon10:38:371
348403299322,24cyclictest3551686-21ssh10:00:321
3484032993218,10cyclictest567-21gdbus10:10:091
3484031993222,7cyclictest3484043-21munin-run07:10:010
3484031993222,7cyclictest3484043-21munin-run07:10:010
3484033993124,5cyclictest3630585-21wc12:10:402
3484033993124,4cyclictest3573510-21grep10:35:482
3484033993123,4cyclictest0-21swapper/211:00:372
3484033993117,3cyclictest171rcu_preempt11:30:122
3484033993115,3cyclictest171rcu_preempt09:10:372
348403399311,20cyclictest3557961-21ssh10:10:412
3484032993124,4cyclictest1003-21mta-sts-daemon09:11:151
3484032993122,6cyclictest3538475-21munin-run09:40:011
3484032993122,5cyclictest3645155-21fschecks_time12:35:301
3484032993121,7cyclictest3520020-21cron09:10:011
3484032993119,8cyclictest3545489-21missed_timers09:50:351
3484032993119,8cyclictest3545489-21missed_timers09:50:351
348403399309,8cyclictest36-21ksoftirqd/212:25:342
3484033993025,2cyclictest0-21swapper/208:00:012
3484033993023,3cyclictest341rcuc/208:33:232
3484033993023,3cyclictest0-21swapper/208:20:012
3484033993022,5cyclictest0-21swapper/210:25:342
3484033993022,5cyclictest0-21swapper/210:25:342
3484033993017,3cyclictest171rcu_preempt07:25:352
348403299305,16cyclictest3633587-21grep12:15:381
3484032993023,4cyclictest3542817-21mailstats09:45:401
3484032993023,4cyclictest3542817-21mailstats09:45:401
3484032993023,4cyclictest3489172-21kernelversion07:25:291
3484032993022,5cyclictest3505053-21systemd-run08:20:011
3484032993022,5cyclictest3493956-21ntp_offset07:40:361
3484032993022,5cyclictest0-21swapper/107:20:001
348403399296,9cyclictest36-21ksoftirqd/208:25:322
348403399295,8cyclictest36-21ksoftirqd/210:53:222
348403399294,14cyclictest36-21ksoftirqd/212:20:342
348403399292,10cyclictest36-21ksoftirqd/209:33:552
348403299291,24cyclictest3609207-21ssh11:35:331
3484032992911,15cyclictest271ktimers/107:30:421
3484033992821,5cyclictest351ktimers/211:40:332
3484033992816,3cyclictest171rcu_preempt08:05:282
3484033992813,3cyclictest171rcu_preempt07:20:412
3484033992813,3cyclictest171rcu_preempt07:20:412
3484032992821,5cyclictest0-21swapper/107:40:011
3484032992820,5cyclictest1003-21mta-sts-daemon07:20:411
3484032992820,5cyclictest1003-21mta-sts-daemon07:20:411
348403299273,19cyclictest3569966-21ntpq10:30:341
3484032992721,3cyclictest0-21swapper/107:10:011
3484032992721,3cyclictest0-21swapper/107:10:011
3484032992719,5cyclictest3516896-21unixbench_multi08:55:501
3484032992716,8cyclictest3502938-21ntp_offset08:10:331
3484032992618,5cyclictest3512406-21tail08:40:441
3484032992614,9cyclictest3507293-21missed_timers08:25:291
348403499192,10cyclictest0-21swapper/309:15:363
348403499173,8cyclictest0-21swapper/311:30:363
348403499172,8cyclictest0-21swapper/310:10:353
348403499162,8cyclictest0-21swapper/312:25:393
348403499162,7cyclictest0-21swapper/312:20:353
348403499161,7cyclictest0-21swapper/311:45:363
348403499153,8cyclictest0-21swapper/309:40:363
348403499153,8cyclictest0-21swapper/309:40:363
348403499152,8cyclictest0-21swapper/311:55:353
348403499152,8cyclictest0-21swapper/311:05:323
348403499151,8cyclictest0-21swapper/309:55:393
348403499151,8cyclictest0-21swapper/309:50:383
348403499151,8cyclictest0-21swapper/309:50:383
348403499151,7cyclictest0-21swapper/309:30:363
348403499142,8cyclictest0-21swapper/311:27:523
348403499142,7cyclictest0-21swapper/312:05:273
348403499142,5cyclictest0-21swapper/307:20:403
348403499142,5cyclictest0-21swapper/307:20:403
348403499141,8cyclictest0-21swapper/310:50:313
348403499141,8cyclictest0-21swapper/310:00:343
348403499141,7cyclictest0-21swapper/312:15:303
348403499141,7cyclictest0-21swapper/310:55:333
348403499141,7cyclictest0-21swapper/310:45:403
348403499141,7cyclictest0-21swapper/309:35:323
348403499141,7cyclictest0-21swapper/308:10:293
348403499141,5cyclictest0-21swapper/311:50:123
348403499133,7cyclictest0-21swapper/311:40:343
348403499133,6cyclictest0-21swapper/312:30:423
348403499132,7cyclictest0-21swapper/312:00:263
348403499132,7cyclictest0-21swapper/311:10:333
348403499132,7cyclictest0-21swapper/310:25:333
348403499132,7cyclictest0-21swapper/310:25:333
348403499132,7cyclictest0-21swapper/310:08:123
348403499132,6cyclictest0-21swapper/309:20:263
348403499132,6cyclictest0-21swapper/309:10:253
348403499132,6cyclictest0-21swapper/308:45:023
348403499131,9cyclictest0-21swapper/311:15:223
348403499131,7cyclictest0-21swapper/312:35:373
348403499131,7cyclictest0-21swapper/311:00:353
348403499131,7cyclictest0-21swapper/309:25:363
348403499131,7cyclictest0-21swapper/308:40:023
348403499131,6cyclictest0-21swapper/308:20:333
348403499131,6cyclictest0-21swapper/308:20:333
348403499123,6cyclictest0-21swapper/310:15:313
348403499123,6cyclictest0-21swapper/310:15:313
348403499122,7cyclictest0-21swapper/311:35:393
348403499122,4cyclictest0-21swapper/307:10:333
348403499122,4cyclictest0-21swapper/307:10:333
348403499121,7cyclictest0-21swapper/311:20:363
348403499121,7cyclictest0-21swapper/310:35:333
348403499121,6cyclictest0-21swapper/309:45:323
348403499121,6cyclictest0-21swapper/309:45:323
348403499121,6cyclictest0-21swapper/309:05:413
348403499121,6cyclictest0-21swapper/308:15:353
348403499121,6cyclictest0-21swapper/307:40:333
348403499121,6cyclictest0-21swapper/307:25:403
348403499112,6cyclictest0-21swapper/312:10:223
348403499112,6cyclictest0-21swapper/310:20:373
348403499112,5cyclictest0-21swapper/307:40:013
348403499111,5cyclictest0-21swapper/309:00:323
348403499111,5cyclictest0-21swapper/308:50:333
348403499111,5cyclictest0-21swapper/308:05:363
348403499111,5cyclictest0-21swapper/307:30:343
348403499102,5cyclictest0-21swapper/310:40:253
348403499102,4cyclictest0-21swapper/308:55:373
348403499101,6cyclictest0-21swapper/310:30:253
348403499101,5cyclictest0-21swapper/308:25:273
348403499101,5cyclictest0-21swapper/307:50:333
348403499100,6cyclictest0-21swapper/308:00:323
34840349993,4cyclictest0-21swapper/308:30:333
34840349992,5cyclictest0-21swapper/308:45:263
34840349992,5cyclictest0-21swapper/307:10:013
34840349992,5cyclictest0-21swapper/307:10:013
34840349991,5cyclictest0-21swapper/307:55:333
34840349991,5cyclictest0-21swapper/307:45:273
34840349991,4cyclictest0-21swapper/307:15:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional