You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 01:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sun Apr 05, 2026 00:43:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3378051995845,10cyclictest695-21cron23:20:000
3378051995648,6cyclictest439-21systemd-logind20:35:010
3378053994941,5cyclictest3471944-21fschecks_count22:20:232
3378051994934,12cyclictest3487492-21ssh22:40:000
3378051994932,13cyclictest0-21swapper/023:00:380
3378052994833,12cyclictest3487976-21aten_r2power_en22:40:201
3378051994819,25cyclictest3549811-21ssh23:55:360
3378052994738,6cyclictest695-21cron20:15:011
3378052994637,5cyclictest3540441-21systemd-run23:45:011
3378053994538,3cyclictest0-21swapper/223:00:022
3378052994539,4cyclictest3518527-21kworker/u8:3-xprtiod00:40:011
337805199456,33cyclictest3525395-21ssh23:25:350
3378051994538,5cyclictest3427562-21/usr/sbin/munin21:25:350
3378053994438,3cyclictest0-21swapper/222:05:012
3378053994436,5cyclictest3577177-21systemd-run00:30:022
3378053994434,6cyclictest3554436-21unixbench_singl00:00:432
3378053994427,12cyclictest1430195-21cups-browsed21:23:232
3378052994436,5cyclictest3455033-21latency_hist22:00:021
3378052994435,6cyclictest1-21systemd00:30:011
3378051994332,7cyclictest0-21swapper/022:50:300
3378052994234,5cyclictest3459121-21latency_hist22:05:021
3378051994233,5cyclictest3422305-21kworker/u8:2+xprtiod21:35:010
3378053994134,4cyclictest0-21swapper/200:05:442
3378052994132,5cyclictest3533120-21kernelversion23:35:261
3378052994132,5cyclictest3400544-21munin-plugin-st20:25:011
3378051994134,5cyclictest3471968-21gltestperf22:20:240
3378051994133,5cyclictest3438565-21sendmail21:40:000
3378051994133,5cyclictest3438565-21sendmail21:40:000
337805199413,32cyclictest3415193-21ssh21:10:320
3378051994132,7cyclictest3471209-21cron22:20:010
3378051994132,6cyclictest548-21gdbus00:02:130
3378051994115,19cyclictest3419440-21ssh21:15:370
3378053994034,3cyclictest0-21swapper/222:35:432
3378052994031,6cyclictest0-21swapper/121:00:011
3378052994031,5cyclictest1056-21mta-sts-daemon22:49:261
3378052994028,5cyclictest0-21swapper/122:25:311
3378052994028,5cyclictest0-21swapper/122:25:311
3378051994033,5cyclictest3385554-21systemd-run19:35:010
3378051994028,9cyclictest3423799-21switchtime21:20:410
3378051994028,9cyclictest1430177-21cupsd20:00:010
3378051994015,19cyclictest421-21dbus-daemon21:50:310
3378053993932,4cyclictest3454940-21systemd-run22:00:012
3378053993923,8cyclictest3541153-21ssh23:45:252
3378052993932,4cyclictest3443394-21fschecks_time21:45:231
3378052993932,3cyclictest0-21swapper/121:40:001
3378052993932,3cyclictest0-21swapper/121:40:001
3378052993931,4cyclictest0-21swapper/121:20:011
3378051993928,8cyclictest548-21gdbus23:07:220
337805199392,27cyclictest3578301-21ssh00:30:350
3378051993921,12cyclictest12350irq/32-eth023:20:340
337805199391,31cyclictest151ktimers/019:20:300
3378053993832,3cyclictest0-21swapper/221:45:002
3378053993830,5cyclictest3549882-21proc_pri23:55:392
3378053993828,7cyclictest421-21dbus-daemon21:35:512
3378053993828,7cyclictest421-21dbus-daemon21:35:512
3378052993830,5cyclictest3472914-21unixbench-2d22:20:421
3378052993829,6cyclictest420-21avahi-daemon23:04:261
3378051993831,4cyclictest3422319-21kworker/u8:5+rpciod23:10:330
3378051993830,5cyclictest3558367-21timerwakeupswit00:05:430
3378051993829,7cyclictest3393848-21needreboot20:00:290
337805199382,30cyclictest3480548-21ssh22:30:320
337805399377,22cyclictest3529277-21ssh23:30:292
3378053993730,4cyclictest3568986-21ssh00:20:012
3378053993726,7cyclictest421-21dbus-daemon21:17:532
3378052993729,5cyclictest1054-21mta-sts-daemon20:10:011
3378052993728,6cyclictest3557483-21grep00:05:251
3378052993725,9cyclictest3516403-21/usr/sbin/munin23:15:211
3378053993627,6cyclictest3582851-21unixbench-2d00:35:422
3378053993627,6cyclictest1430177-21cupsd22:13:152
3378053993626,6cyclictest261-21systemd-journal23:53:552
337805399361,27cyclictest3476362-21ssh22:25:282
337805399361,27cyclictest3476362-21ssh22:25:282
3378052993630,3cyclictest0-21swapper/100:20:011
3378052993628,5cyclictest1054-21mta-sts-daemon23:20:561
337805299361,30cyclictest3439693-21ssh21:40:291
3378051993629,5cyclictest12350irq/32-eth023:45:320
3378051993628,6cyclictest1430195-21cups-browsed22:42:590
337805199362,25cyclictest151ktimers/020:45:300
3378053993527,5cyclictest3450824-21munin-plugin-st21:55:002
3378053993527,5cyclictest3431370-21missed_timers21:30:292
337805399352,29cyclictest3578443-21ssh00:30:402
3378052993527,5cyclictest3581168-21idleruntime-cro00:35:011
3378052993524,8cyclictest3554042-21mailstats00:00:381
3378051993528,4cyclictest3564987-21latency_hist00:15:020
3378051993527,4cyclictest0-21swapper/000:22:460
3378051993526,6cyclictest3545637-21ssh23:50:340
3378051993521,11cyclictest3532261-21cron23:35:010
3378051993512,17cyclictest3552625-21cupsd00:25:310
3378053993426,5cyclictest3381052-21tr19:20:012
3378052993427,4cyclictest0-21swapper/119:20:001
3378052993426,5cyclictest3393029-21idleruntime-cro20:00:011
3378052993426,5cyclictest3388530-21cron19:45:001
3378052993426,5cyclictest3386059-21fschecks_time19:35:231
3378052993425,6cyclictest3411559-21fschecks_time21:00:251
3378052993424,6cyclictest1430195-21cups-browsed22:10:191
337805299342,26cyclictest3549756-21ssh23:55:341
3378051993426,5cyclictest439-21systemd-logind20:40:010
3378051993419,6cyclictest12350irq/32-eth023:35:340
3378051993419,10cyclictest151ktimers/020:55:300
3378051993417,12cyclictest12350irq/32-eth000:35:340
337805399335,20cyclictest3492917-21ssh22:45:392
3378053993326,3cyclictest0-21swapper/223:20:002
3378053993325,5cyclictest3516011-21cron23:15:002
3378053993324,6cyclictest3395967-21unixbench_singl20:05:432
337805399331,22cyclictest3480464-21diskmemload22:30:292
3378052993323,6cyclictest1430177-21cupsd22:15:381
3378051993327,4cyclictest3568996-21munin-run00:20:010
3378051993326,5cyclictest548-21gdbus20:50:400
3378051993326,5cyclictest261-21systemd-journal20:45:010
3378051993326,3cyclictest0-21swapper/022:10:310
3378051993325,5cyclictest3492177-21aten_r2power_vo22:45:230
3378051993324,6cyclictest3413052-21fschecks_count21:05:250
3378051993323,8cyclictest3396926-21ntp_kernel_pll_20:10:320
3378051993323,6cyclictest3537058-21grep23:40:260
3378051993321,9cyclictest3439819-21ntpq21:40:340
3378053993226,3cyclictest0-21swapper/222:20:012
3378053993225,4cyclictest3562377-21xargs00:10:422
3378053993224,5cyclictest1-21systemd23:40:012
3378053993223,7cyclictest3570040-21needreboot00:20:302
337805299323,20cyclictest3414029-21diskmemload23:45:321
3378052993224,5cyclictest3483922-21aten_r2power_fr22:35:211
3378052993223,5cyclictest0-21swapper/120:50:011
3378051993222,4cyclictest12350irq/32-eth022:00:370
3378053993122,6cyclictest3488629-21ntp_kernel_pll_22:40:302
337805299316,18cyclictest3570165-21ssh00:20:351
337805299313,24cyclictest3500868-21ssh22:55:311
3378052993123,5cyclictest695-21cron20:35:001
3378052993123,5cyclictest1-21systemd19:15:011
3378052993122,7cyclictest3383844-21timerwakeupswit19:25:411
3378052993122,7cyclictest3383844-21timerwakeupswit19:25:411
337805299312,23cyclictest3423549-21ssh21:20:351
3378052993121,7cyclictest3414426-21/usr/sbin/munin21:10:381
3378051993123,6cyclictest3503828-21sendmail23:00:010
3378051993121,7cyclictest3459390-21acpi22:05:200
337805399304,12cyclictest3414029-21diskmemload21:13:592
3378053993025,2cyclictest0-21swapper/219:55:012
3378053993022,5cyclictest1430195-21cups-browsed20:11:342
3378053993022,4cyclictest0-21swapper/222:50:322
3378052993024,3cyclictest0-21swapper/120:45:011
3378052993024,3cyclictest0-21swapper/100:10:271
3378052993023,4cyclictest1056-21mta-sts-daemon21:05:411
337805299302,21cyclictest3447901-21ssh21:50:321
3378051993024,3cyclictest0-21swapper/022:30:020
3378051993024,3cyclictest0-21swapper/022:30:020
337805399293,21cyclictest3525313-21ssh23:25:312
3378053992923,3cyclictest0-21swapper/223:20:232
3378053992923,3cyclictest0-21swapper/221:00:012
3378053992923,3cyclictest0-21swapper/220:20:012
3378053992922,4cyclictest3406451-21head20:40:402
3378053992921,5cyclictest0-21swapper/219:40:012
3378053992911,14cyclictest351ktimers/221:05:352
3378052992923,4cyclictest3378050-21cyclictest23:30:301
3378052992921,5cyclictest1056-21mta-sts-daemon22:51:281
3378052992920,6cyclictest3430989-21aten_r2power_vo21:30:231
3378051992925,3cyclictest0-21swapper/021:55:290
3378051992921,6cyclictest3379508-21sshd19:12:540
3378051992920,5cyclictest3443058-21/usr/sbin/munin21:45:280
337805399285,16cyclictest624-21lldpd20:45:302
337805399284,11cyclictest3414029-21diskmemload21:46:042
3378053992823,2cyclictest0-21swapper/219:35:002
3378053992821,3cyclictest0-21swapper/219:45:012
3378053992820,5cyclictest420-21avahi-daemon20:25:422
3378053992820,5cyclictest420-21avahi-daemon20:25:422
3378053992820,5cyclictest3412460-21unixbench_singl21:00:432
3378052992823,2cyclictest0-21swapper/120:55:011
3378052992821,4cyclictest3479796-21telnet22:30:201
3378052992821,4cyclictest3381320-21aten_r2power_cu19:20:191
3378052992820,5cyclictest3545115-21cpu23:50:241
3378052992820,5cyclictest3525882-21tail23:25:431
3378052992820,5cyclictest3513687-21tail23:10:431
3378052992819,6cyclictest1430195-21cups-browsed22:09:241
3378051992819,5cyclictest3388523-21kworker/u8:7+xprtiod20:25:420
3378051992819,5cyclictest3388523-21kworker/u8:7+xprtiod20:25:420
3378051992817,8cyclictest0-21swapper/019:50:250
3378051992816,9cyclictest421-21dbus-daemon19:20:010
3378051992816,9cyclictest3412686-21latency_hist21:05:020
3378051992813,6cyclictest14-21ksoftirqd/020:05:340
3378053992722,2cyclictest0-21swapper/223:45:022
3378053992721,3cyclictest1054-21mta-sts-daemon21:29:252
3378053992720,3cyclictest0-21swapper/223:00:352
3378053992719,5cyclictest3404900-21unixbench_multi20:35:412
3378053992719,5cyclictest3384054-21systemd-run19:30:012
3378053992719,5cyclictest3384054-21systemd-run19:30:012
3378053992718,6cyclictest3403428-21unixbench_multi20:30:422
3378052992722,2cyclictest0-21swapper/120:20:011
3378052992721,3cyclictest0-21swapper/123:05:231
3378052992720,4cyclictest1051-21mta-sts-daemon19:30:351
3378052992719,5cyclictest3403921-21aten_r2power_vo20:35:211
3378052992719,5cyclictest1054-21mta-sts-daemon21:25:131
337805199274,4cyclictest14-21ksoftirqd/020:15:290
3378053992619,4cyclictest1056-21mta-sts-daemon19:20:372
3378053992618,3cyclictest0-21swapper/222:05:112
3378053992617,6cyclictest516-21in:imuxsock20:25:012
3378052992618,5cyclictest3394729-21latency_hist20:05:031
3378051992621,2cyclictest0-21swapper/019:40:010
3378053992520,2cyclictest0-21swapper/223:05:302
3378053992518,4cyclictest3393852-21needreboot20:00:292
337805199251,10cyclictest3389066-21fschecks_time19:45:230
3378053992418,3cyclictest0-21swapper/220:55:012
3378053992417,4cyclictest3393019-21latency_hist20:00:012
3378052992419,2cyclictest0-21swapper/120:30:001
3378052992419,2cyclictest0-21swapper/120:30:001
337805199245,14cyclictest151ktimers/019:40:350
3378051992415,6cyclictest3388523-21kworker/u8:7-xprtiod20:20:240
337805499233,12cyclictest0-21swapper/322:50:303
337805199231,10cyclictest3383786-21timerandwakeup19:25:400
337805199231,10cyclictest3383786-21timerandwakeup19:25:400
337805499223,10cyclictest0-21swapper/321:50:323
337805499222,13cyclictest0-21swapper/322:05:343
3378052992215,4cyclictest3391119-21mailstats19:50:381
337805499213,9cyclictest0-21swapper/323:20:313
3378053992112,6cyclictest3378629-21awk19:10:242
337805399211,14cyclictest3389586-21ps19:45:362
337805299212,14cyclictest3389507-21ntpq19:45:341
337805499204,8cyclictest0-21swapper/300:20:383
337805499203,11cyclictest0-21swapper/323:30:323
337805499203,11cyclictest0-21swapper/322:40:343
3378053992012,4cyclictest3378062-21systemd-run19:10:012
337805199200,15cyclictest3378189-21wc19:10:020
337805499194,8cyclictest0-21swapper/321:15:343
337805499194,7cyclictest0-21swapper/321:10:303
337805499194,10cyclictest0-21swapper/322:20:323
337805499193,10cyclictest0-21swapper/323:40:333
337805499193,10cyclictest0-21swapper/323:05:333
337805499193,10cyclictest0-21swapper/322:15:353
337805499193,10cyclictest0-21swapper/322:10:323
337805499193,10cyclictest0-21swapper/300:25:273
337805499193,10cyclictest0-21swapper/300:10:283
337805499192,11cyclictest0-21swapper/321:55:323
337805499192,10cyclictest0-21swapper/323:45:333
337805499192,10cyclictest0-21swapper/321:45:363
337805499191,10cyclictest0-21swapper/323:50:353
3378052991913,3cyclictest0-21swapper/119:10:011
337805499183,8cyclictest0-21swapper/323:10:313
337805499183,8cyclictest0-21swapper/321:30:303
337805499183,10cyclictest0-21swapper/300:30:233
337805499182,9cyclictest0-21swapper/300:05:313
337805499182,8cyclictest0-21swapper/323:35:283
337805499182,8cyclictest0-21swapper/320:10:303
337805499181,10cyclictest0-21swapper/322:45:313
337805499173,9cyclictest0-21swapper/323:15:333
337805499173,8cyclictest0-21swapper/321:25:343
337805499173,8cyclictest0-21swapper/300:35:283
337805499173,7cyclictest0-21swapper/319:45:343
337805499172,9cyclictest0-21swapper/321:20:313
337805499172,9cyclictest0-21swapper/300:15:313
337805499172,10cyclictest0-21swapper/323:25:303
337805499172,10cyclictest0-21swapper/322:35:303
337805499172,10cyclictest0-21swapper/321:40:323
337805499172,10cyclictest0-21swapper/321:35:333
337805499172,10cyclictest0-21swapper/321:35:333
337805499165,7cyclictest0-21swapper/322:55:303
337805499164,6cyclictest0-21swapper/323:55:323
337805499163,8cyclictest0-21swapper/323:00:323
337805499162,9cyclictest0-21swapper/300:00:313
337805499162,7cyclictest0-21swapper/319:35:363
337805499161,9cyclictest0-21swapper/322:30:293
337805499161,8cyclictest0-21swapper/320:05:313
337805499153,8cyclictest0-21swapper/320:30:333
337805499153,7cyclictest0-21swapper/322:00:323
337805499153,6cyclictest0-21swapper/319:10:303
337805499152,7cyclictest0-21swapper/322:25:193
337805499152,7cyclictest0-21swapper/322:25:193
337805499152,7cyclictest0-21swapper/319:25:333
337805499152,7cyclictest0-21swapper/319:25:333
337805499152,6cyclictest0-21swapper/320:25:333
337805499152,6cyclictest0-21swapper/320:25:333
337805499152,6cyclictest0-21swapper/319:30:313
337805499151,7cyclictest0-21swapper/320:50:023
337805499142,7cyclictest0-21swapper/320:00:323
337805499142,7cyclictest0-21swapper/319:15:313
337805499142,6cyclictest0-21swapper/321:05:333
337805499134,5cyclictest0-21swapper/320:15:293
337805499132,6cyclictest0-21swapper/320:55:303
337805499132,6cyclictest0-21swapper/320:20:293
337805499131,6cyclictest0-21swapper/319:20:283
337805499122,6cyclictest0-21swapper/320:40:283
337805499121,6cyclictest0-21swapper/320:50:313
337805499121,6cyclictest0-21swapper/320:35:303
337805499112,6cyclictest0-21swapper/319:40:293
337805499111,6cyclictest0-21swapper/321:00:303
337805499111,6cyclictest0-21swapper/319:55:263
337805499102,5cyclictest0-21swapper/319:50:333
337805499102,5cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional