You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 10:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Tue Jul 07, 2026 00:43:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3094175995337,12cyclictest3136213-21sshd21:20:400
3094176995143,5cyclictest3116960-21systemd-run20:30:011
3094177995041,6cyclictest3127927-21/usr/sbin/munin21:05:292
3094176994942,4cyclictest3186891-21cron22:45:011
3094176994739,5cyclictest3128952-21systemd-run21:10:021
3094177994637,5cyclictest3135076-21systemctl21:20:012
3094176994637,6cyclictest3130325-21switchtime21:10:481
3094177994537,5cyclictest1-21systemd21:30:022
3094177994536,6cyclictest3192983-21munin-run22:55:012
3094176994536,6cyclictest3144274-21ssh21:35:011
3094177994335,5cyclictest3163218-21fschecks_count22:05:292
3094177994335,5cyclictest3163218-21fschecks_count22:05:292
3094176994336,4cyclictest1001-21mta-sts-daemon20:30:401
3094177994233,6cyclictest3147323-21systemctl21:40:012
3094176994234,5cyclictest3214312-21systemd-run23:30:011
3094176994232,6cyclictest3244730-21ssh00:20:011
3094175994234,5cyclictest1-21systemd21:40:020
3094177994133,5cyclictest3187277-21aten_r2power_cu22:45:262
3094177994132,4cyclictest0-21swapper/219:20:012
3094176994132,6cyclictest3177799-21unin-run22:30:011
3094176994131,6cyclictest265-21systemd-journal00:33:081
3094175994133,5cyclictest3166852-21sendmail_mailqu22:10:420
3094177994032,5cyclictest3112432-21munin-run20:15:012
3094176994033,4cyclictest3170255-21unixbench_multi22:15:491
3094176994032,5cyclictest3193059-21latency_hist22:55:021
3094175994033,4cyclictest1003-21mta-sts-daemon22:30:020
3094175994032,6cyclictest0-21swapper/021:35:010
3094177993931,5cyclictest3220362-21cron23:40:012
3094176993933,3cyclictest0-21swapper/121:40:011
3094176993930,6cyclictest421-21dbus-daemon00:10:371
3094176993929,6cyclictest3232609-21cups-browsed00:05:181
3094175993933,4cyclictest3253889-21systemd-run00:35:020
3094175993932,4cyclictest3208206-21basename23:20:010
3094177993831,4cyclictest3200718-21unixbench_multi23:05:482
3094177993831,4cyclictest3200718-21unixbench_multi23:05:482
3094177993830,5cyclictest3236591-21needreboot00:05:342
3094177993829,6cyclictest265-21systemd-journal00:25:012
3094176993832,3cyclictest0-21swapper/122:05:381
3094176993832,3cyclictest0-21swapper/122:05:381
309417599389,24cyclictest3212328-21ssh23:25:390
3094175993829,6cyclictest421-21dbus-daemon20:25:010
3094177993730,3cyclictest0-21swapper/220:25:422
3094177993729,5cyclictest421-21dbus-daemon20:00:022
3094177993729,5cyclictest3186909-21awk22:45:012
3094177993728,6cyclictest3196669-21/usr/sbin/munin23:00:292
3094177993728,6cyclictest3196669-21/usr/sbin/munin23:00:292
3094177993727,6cyclictest1370083-21cupsd21:31:542
3094177993726,8cyclictest3227784-21mailstats23:50:452
3094176993731,3cyclictest3167910-21kworker/u8:2-xprtiod22:15:001
3094176993731,3cyclictest0-21swapper/122:00:011
3094176993730,5cyclictest3101945-21sendmail19:40:001
3094176993730,5cyclictest3101945-21sendmail19:40:001
3094176993729,5cyclictest3196021-21sendmail23:00:011
3094175993731,3cyclictest0-21swapper/021:45:260
3094175993728,7cyclictest3250838-21/usr/sbin/munin00:30:020
3094175993727,7cyclictest3148312-21needreboot21:40:350
3094175993725,9cyclictest3156907-21aten_r2power_fr21:55:280
3094175993721,13cyclictest3245181-21telnet00:20:250
3094175993718,14cyclictest151ktimers/020:15:350
3094177993629,4cyclictest3153427-21systemd-run21:50:022
3094177993628,5cyclictest3232638-21latency_hist00:00:022
3094177993628,5cyclictest3132118-21latency_hist21:15:022
3094177993625,4cyclictest1001-21mta-sts-daemon00:18:282
3094176993630,2cyclictest0-21swapper/100:40:011
3094176993628,5cyclictest3095922-21cron19:20:011
3094175993628,5cyclictest3255709-21systemd-run00:40:020
3094177993528,4cyclictest3138316-21latency_hist21:25:022
3094177993528,4cyclictest1000-21mta-sts-daemon23:20:572
3094177993527,5cyclictest3177811-21latency_hist22:30:022
3094177993527,5cyclictest3114477-21fschecks_time20:20:312
3094177993526,6cyclictest1370086-21cups-browsed19:35:362
3094177993526,6cyclictest1370086-21cups-browsed19:35:362
3094177993525,7cyclictest3205140-21cron23:15:002
3094176993529,3cyclictest0-21swapper/123:10:521
3094176993528,4cyclictest3250827-21cron00:30:011
3094176993526,6cyclictest1003-21mta-sts-daemon22:45:571
3094175993529,4cyclictest3139614-21timerandwakeup21:25:460
3094175993528,4cyclictest3211261-21idleruntime-cro23:25:010
3094175993528,4cyclictest3156557-21latency_hist21:55:020
3094175993527,5cyclictest3127372-21head21:00:520
3094175993525,6cyclictest420-21avahi-daemon22:01:140
3094175993523,8cyclictest3191369-21switchtime22:50:470
3094177993428,3cyclictest0-21swapper/200:40:012
3094177993427,5cyclictest421-21dbus-daemon20:55:012
3094177993427,3cyclictest0-21swapper/222:35:412
3094177993424,7cyclictest3167157-21grep22:10:462
3094176993428,3cyclictest0-21swapper/122:20:411
3094176993426,3cyclictest0-21swapper/122:35:271
3094175993429,3cyclictest0-21swapper/023:40:010
3094175993428,4cyclictest3223421-21latency_hist23:45:010
3094175993424,7cyclictest420-21avahi-daemon20:25:530
3094175993423,8cyclictest3224233-21kernelversion23:45:310
3094177993325,5cyclictest3215735-21threads23:30:482
3094177993325,5cyclictest3172356-21fschecks_count22:20:292
3094177993324,6cyclictest420-21avahi-daemon23:28:242
3094176993328,2cyclictest0-21swapper/123:20:011
3094176993326,4cyclictest3232543-21munin-run00:00:001
3094176993326,4cyclictest1001-21mta-sts-daemon21:51:251
3094176993324,6cyclictest1370086-21cups-browsed19:20:351
3094176993323,7cyclictest3218647-21mailstats23:35:451
3094175993327,4cyclictest3174818-21latency_hist22:25:020
3094175993322,4cyclictest3095788-21unixbench_multi19:15:460
3094175993315,12cyclictest12650irq/32-eth000:10:420
3094177993226,4cyclictest3094174-21cyclictest21:40:352
3094177993226,3cyclictest0-21swapper/223:40:312
3094177993226,3cyclictest0-21swapper/200:10:322
3094177993224,6cyclictest3112419-21kworker/u8:2+rpciod20:20:012
3094177993224,5cyclictest1370086-21cups-browsed19:25:212
3094176993226,3cyclictest0-21swapper/100:25:011
3094176993225,4cyclictest1002-21mta-sts-daemon23:09:151
3094176993225,4cyclictest1002-21mta-sts-daemon23:09:151
3094176993225,3cyclictest0-21swapper/123:50:011
3094176993222,7cyclictest1370086-21cups-browsed22:34:261
3094175993225,4cyclictest0-21swapper/023:10:410
3094175993224,6cyclictest0-21swapper/019:15:010
3094175993222,8cyclictest3199078-21cron23:05:000
3094175993222,8cyclictest3199078-21cron23:05:000
3094175993217,4cyclictest3117550-21if_wlan020:30:320
309417599321,24cyclictest3242840-21ssh00:15:400
3094177993125,3cyclictest0-21swapper/222:35:012
3094177993124,4cyclictest3234229-21grep00:00:482
3094177993122,6cyclictest421-21dbus-daemon19:50:022
3094177993121,7cyclictest3160420-21missed_timers22:00:332
3094176993125,3cyclictest0-21swapper/119:30:401
3094176993125,3cyclictest0-21swapper/119:30:401
3094176993123,5cyclictest3108471-21fschecks_time20:00:301
3094176993123,5cyclictest3097966-21fschecks_count19:25:331
3094176993122,5cyclictest1370086-21cups-browsed21:17:581
3094176993121,8cyclictest3160329-21latency22:00:321
3094175993123,5cyclictest3101961-21sendmail-msp19:40:010
3094175993123,5cyclictest3101961-21sendmail-msp19:40:010
3094175993122,6cyclictest3124414-21cron20:55:000
3094175993121,8cyclictest3110255-21needreboot20:05:340
3094177993022,5cyclictest1001-21mta-sts-daemon23:18:042
3094177993015,10cyclictest1370086-21cups-browsed19:50:392
3094177993014,5cyclictest3128933-21diskmemload22:19:192
309417699304,12cyclictest3128933-21diskmemload21:29:141
3094176993024,4cyclictest3094174-21cyclictest21:45:341
3094176993024,3cyclictest0-21swapper/123:05:011
3094176993024,3cyclictest0-21swapper/123:05:011
3094176993024,3cyclictest0-21swapper/121:40:361
3094176993022,6cyclictest0-21swapper/121:20:401
3094176993021,6cyclictest421-21dbus-daemon20:50:011
3094176993021,6cyclictest421-21dbus-daemon20:50:011
309417599306,18cyclictest665-21snmpd22:05:410
309417599306,18cyclictest665-21snmpd22:05:410
3094175993023,5cyclictest0-21swapper/022:40:010
3094175993023,4cyclictest3217364-21systemd23:35:020
3094175993023,4cyclictest1003-21mta-sts-daemon21:10:130
3094175993023,4cyclictest0-21swapper/000:00:340
3094175993021,6cyclictest3121309-21tail20:40:470
3094175993019,8cyclictest3098255-21needreboot19:25:370
3094177992923,3cyclictest0-21swapper/221:50:262
3094177992921,5cyclictest3096406-21cpuspeed19:20:292
3094177992920,6cyclictest443-21gpm20:35:332
3094177992920,6cyclictest420-21avahi-daemon00:30:462
3094177992920,4cyclictest0-21swapper/220:40:422
3094177992919,7cyclictest3157919-21timerandwakeup21:55:492
3094176992923,3cyclictest0-21swapper/119:55:011
3094176992923,3cyclictest0-21swapper/100:05:011
3094176992922,4cyclictest3214861-21cpu23:30:281
3094176992921,5cyclictest3226921-21aten2.4-expect23:50:271
3094176992920,7cyclictest3125569-21switchtime20:55:481
3094175992923,4cyclictest3124950-21fschecks_time20:55:310
3094175992922,3cyclictest0-21swapper/022:40:370
309417599291,23cyclictest665-21snmpd23:56:030
309417799284,5cyclictest171rcu_preempt23:45:392
3094177992821,4cyclictest1003-21mta-sts-daemon22:58:582
3094177992820,5cyclictest1002-21mta-sts-daemon21:00:012
3094177992811,13cyclictest3248301-21aten2.4-expect00:25:282
3094176992821,4cyclictest3209837-21unixbench_multi23:20:491
3094176992821,4cyclictest3110827-21unixbench_multi20:05:481
3094176992821,4cyclictest3102475-21fschecks_time19:40:301
309417599286,16cyclictest3227505-21ssh23:50:370
309417599282,20cyclictest3194092-21ssh22:55:420
3094175992821,5cyclictest420-21avahi-daemon20:00:450
3094175992820,4cyclictest0-21swapper/021:10:020
3094175992816,9cyclictest0-21swapper/022:50:010
3094175992814,10cyclictest3133331-21mailstats21:15:450
3094177992721,3cyclictest0-21swapper/220:45:372
3094177992721,3cyclictest0-21swapper/220:45:372
3094177992719,4cyclictest1000-21mta-sts-daemon19:30:482
3094177992719,4cyclictest1000-21mta-sts-daemon19:30:482
309417699273,19cyclictest3221406-21ssh23:40:351
3094176992721,3cyclictest0-21swapper/120:00:011
3094176992720,4cyclictest1003-21mta-sts-daemon20:10:451
3094176992719,5cyclictest3113935-21sendmail20:20:011
3094176992718,7cyclictest3126775-21ntp_kernel_err21:00:361
3094175992718,7cyclictest3106359-21unixbench_singl19:50:470
3094175992718,6cyclictest3170015-21switchtime22:15:470
309417599271,22cyclictest3200116-21ntpq23:05:350
309417599271,22cyclictest3200116-21ntpq23:05:350
3094177992616,7cyclictest694-21cron21:05:002
3094176992618,5cyclictest3103957-21fschecks_count19:45:271
3094175992620,3cyclictest0-21swapper/022:35:020
3094175992619,4cyclictest3107857-21unixbench_singl19:55:510
3094175992619,4cyclictest3104877-21unixbench_singl19:45:450
3094175992614,7cyclictest12650irq/32-eth020:45:390
3094175992614,7cyclictest12650irq/32-eth020:45:390
309417599261,19cyclictest1370086-21cups-browsed19:30:400
309417599261,19cyclictest1370086-21cups-browsed19:30:400
3094177992519,3cyclictest0-21swapper/220:05:012
3094177992515,8cyclictest3102817-21ntp_kernel_pll_19:40:382
3094176992516,6cyclictest3120704-21missed_timers20:40:331
3094175992521,2cyclictest0-21swapper/000:05:300
309417799245,3cyclictest3110818-21sed20:05:482
3094176992416,5cyclictest3115635-21latency_hist20:25:021
309417599241,16cyclictest12650irq/32-eth020:10:400
309417799231,17cyclictest3117898-21ntpq20:30:412
309417699235,14cyclictest1370086-21cups-browsed20:35:361
3094176992317,4cyclictest0-21swapper/120:50:491
3094175992315,5cyclictest0-21swapper/020:40:020
3094175992215,3cyclictest0-21swapper/019:40:350
3094177992113,5cyclictest3094417-21cron19:15:012
3094177992112,6cyclictest567-21gdbus19:10:012
3094176992113,5cyclictest3094188-21systemd-run19:10:011
3094175992117,3cyclictest3094174-21cyclictest19:20:070
3094176992011,6cyclictest1370086-21cups-browsed19:12:531
309417899192,7cyclictest0-21swapper/320:25:433
309417899171,5cyclictest0-21swapper/319:35:373
309417899171,5cyclictest0-21swapper/319:35:373
309417899163,5cyclictest0-21swapper/322:00:413
309417899162,5cyclictest0-21swapper/323:50:373
309417899161,8cyclictest0-21swapper/322:30:433
309417599168,5cyclictest567-21gdbus19:10:010
309417899152,6cyclictest0-21swapper/300:30:413
309417899151,8cyclictest0-21swapper/300:15:433
309417899151,6cyclictest0-21swapper/322:10:373
309417899151,5cyclictest0-21swapper/320:00:353
309417899151,5cyclictest0-21swapper/319:20:383
309417899143,6cyclictest0-21swapper/321:45:393
309417899142,6cyclictest0-21swapper/322:35:383
309417899141,8cyclictest0-21swapper/321:55:283
309417899141,7cyclictest0-21swapper/300:00:283
309417899141,6cyclictest0-21swapper/323:00:363
309417899141,6cyclictest0-21swapper/323:00:363
309417899141,6cyclictest0-21swapper/320:30:363
309417899141,6cyclictest0-21swapper/319:40:353
309417899141,5cyclictest0-21swapper/320:40:413
309417899133,7cyclictest0-21swapper/321:10:403
309417899133,6cyclictest0-21swapper/321:20:013
309417899132,6cyclictest0-21swapper/321:20:383
309417899132,6cyclictest0-21swapper/321:05:413
309417899132,6cyclictest0-21swapper/320:55:413
309417899132,5cyclictest0-21swapper/322:05:373
309417899132,5cyclictest0-21swapper/322:05:373
309417899132,5cyclictest0-21swapper/320:15:383
309417899131,8cyclictest0-21swapper/323:25:463
309417899131,7cyclictest0-21swapper/323:10:463
309417899131,5cyclictest0-21swapper/320:35:353
309417899131,5cyclictest0-21swapper/319:30:383
309417899131,5cyclictest0-21swapper/319:30:383
309417899123,6cyclictest0-21swapper/323:55:363
309417899123,5cyclictest0-21swapper/323:15:403
309417899122,7cyclictest0-21swapper/323:25:013
309417899122,6cyclictest0-21swapper/323:45:293
309417899122,6cyclictest0-21swapper/323:05:373
309417899122,6cyclictest0-21swapper/323:05:373
309417899122,5cyclictest0-21swapper/321:50:253
309417899122,5cyclictest0-21swapper/319:25:473
309417899121,7cyclictest0-21swapper/321:40:273
309417899121,6cyclictest0-21swapper/323:35:423
309417899121,6cyclictest0-21swapper/321:39:413
309417899121,6cyclictest0-21swapper/319:15:363
309417899121,6cyclictest0-21swapper/300:10:313
309417899121,5cyclictest0-21swapper/320:50:373
309417899113,6cyclictest0-21swapper/322:45:353
309417899112,5cyclictest0-21swapper/322:50:273
309417899112,5cyclictest0-21swapper/322:40:303
309417899112,5cyclictest0-21swapper/321:25:263
309417899112,5cyclictest0-21swapper/300:40:013
309417899112,4cyclictest0-21swapper/322:55:303
309417899111,6cyclictest0-21swapper/323:40:273
309417899111,6cyclictest0-21swapper/323:30:313
309417899111,6cyclictest0-21swapper/322:15:443
309417899111,6cyclictest0-21swapper/321:30:333
309417899111,6cyclictest0-21swapper/319:50:393
309417899111,6cyclictest0-21swapper/300:05:353
309417899102,5cyclictest0-21swapper/322:25:283
309417899101,6cyclictest0-21swapper/319:45:423
309417899101,6cyclictest0-21swapper/300:25:273
309417899101,5cyclictest0-21swapper/322:20:373
309417899101,5cyclictest0-21swapper/320:05:363
309417899101,5cyclictest0-21swapper/319:56:203
30941789993,3cyclictest0-21swapper/320:45:363
30941789993,3cyclictest0-21swapper/320:45:363
30941789991,5cyclictest0-21swapper/321:00:353
30941789991,5cyclictest0-21swapper/320:20:283
30941789991,4cyclictest0-21swapper/319:14:053
30941789991,4cyclictest0-21swapper/300:25:033
30941789981,4cyclictest0-21swapper/320:10:353
30941789971,3cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional