You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 20:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Thu Nov 20, 2025 12:39:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3528892998260,14cyclictest3554660-21aten_r2power_cu08:39:030
352889299613,49cyclictest2667052-21cups-browsed11:09:130
3528892995712,32cyclictest3655331-21ssh11:24:130
3528893995648,5cyclictest3584792-21idleruntime-cro09:40:011
3528894995143,5cyclictest3702228-21unixbench_multi12:29:202
3528893995139,9cyclictest3628171-21sh10:44:181
3528893994931,14cyclictest3701663-21sh12:29:091
3528892994840,5cyclictest3635185-21mailstats10:54:200
3528892994835,6cyclictest12550irq/33-eth011:29:150
3528894994638,5cyclictest3639090-21latency_hist11:00:022
3528894994638,5cyclictest3639090-21latency_hist11:00:022
3528893994638,5cyclictest3560417-21munin-plugin-st08:55:011
352889299464,7cyclictest3549348-21ntpq08:19:170
352889499454,30cyclictest2667052-21cups-browsed10:54:172
3528894994537,5cyclictest3533393-21munin-plugin-st07:25:012
352889299453,34cyclictest3611294-21ssh10:19:130
3528892994523,14cyclictest12450irq/32-eth010:14:180
3528892994436,5cyclictest3649224-21grep11:14:250
3528892994435,5cyclictest3631805-21sendmail_mailqu10:49:180
3528892994431,11cyclictest3617857-21kernelversion10:29:060
352889499433,33cyclictest3607983-21ssh10:14:182
3528894994332,6cyclictest0-21swapper/208:39:152
352889399434,32cyclictest3697993-21ssh12:24:121
3528893994335,5cyclictest3590413-21fschecks_time09:49:081
3528893994335,5cyclictest3542393-21systemd-run07:55:011
352889299431,15cyclictest3614503-21irqstats10:24:060
3528894994234,5cyclictest3546901-21systemd-run08:10:022
3528894994231,7cyclictest2667052-21cups-browsed11:53:182
3528893994234,5cyclictest3665960-21munin-run11:40:011
3528893994233,5cyclictest3602069-21latency_hist10:05:021
352889299422,7cyclictest3628020-21ntpq10:44:110
3528892994226,10cyclictest12450irq/32-eth012:29:080
352889299422,14cyclictest3561319-21ntpq08:59:340
352889299422,14cyclictest3561319-21ntpq08:59:340
3528892994217,21cyclictest0-21swapper/007:35:010
352889299421,16cyclictest3697916-21ssh12:24:090
3528894994132,6cyclictest1-21systemd11:50:022
3528893994130,8cyclictest3693613-21aten_r2power_vo12:19:041
3528893994130,8cyclictest3693613-21aten_r2power_vo12:19:041
352889299413,9cyclictest3541769-21ntpq07:54:100
352889299413,8cyclictest3552266-21ntpq08:29:330
352889299412,6cyclictest3556891-21ntpq08:44:110
352889299412,16cyclictest3564394-21ntp_states09:09:130
352889299412,16cyclictest3564394-21ntp_states09:09:130
352889299411,17cyclictest3590994-21ps09:49:220
352889299411,14cyclictest3638162-21ntpq10:59:150
352889299411,14cyclictest3638162-21ntpq10:59:150
352889299411,12cyclictest3676138-21ntpq11:54:190
352889299411,11cyclictest3660914-21ssh11:32:170
352889299411,11cyclictest3624796-21ntpq10:39:140
352889299411,10cyclictest3544886-21ntpq08:04:160
352889499406,28cyclictest3669049-21ssh11:44:092
352889499403,29cyclictest3628118-21ssh10:44:142
3528894994031,5cyclictest0-21swapper/209:20:012
3528894994029,8cyclictest266-21systemd-journal09:52:172
352889499402,32cyclictest3580268-21ssh09:34:112
352889499402,32cyclictest3580268-21ssh09:34:112
3528893994032,5cyclictest3691363-21cron12:15:011
3528893994031,6cyclictest3625730-21systemd-run10:40:021
3528893994031,6cyclictest1-21systemd12:35:011
3528893994031,6cyclictest1-21systemd09:05:011
3528893994031,6cyclictest1-21systemd09:05:011
3528893994031,5cyclictest3659754-21cron11:30:011
3528892994033,5cyclictest3593879-21/usr/sbin/munin09:54:210
352889299402,6cyclictest3672663-21ntpq11:49:180
3528892994019,18cyclictest2667052-21cups-browsed08:23:010
352889299401,15cyclictest3678554-21ssh11:58:170
352889299401,14cyclictest3577204-21ntpq09:29:130
352889299401,14cyclictest3577204-21ntpq09:29:130
352889299401,10cyclictest3600118-21ssh10:03:320
3528894993931,5cyclictest3691378-21munin-plugin-st12:15:012
3528894993929,7cyclictest3663312-21cron11:35:012
3528894993928,8cyclictest3614695-21needreboot10:24:082
352889499392,29cyclictest3577171-21ssh09:29:122
352889499392,29cyclictest3577171-21ssh09:29:122
3528893993927,9cyclictest3668996-21needreboot11:44:081
3528893993924,9cyclictest591-21sshd12:39:181
352889299394,22cyclictest12450irq/32-eth009:34:150
352889299394,22cyclictest12450irq/32-eth009:34:150
3528892993927,7cyclictest151ktimers/012:09:120
3528892993915,21cyclictest266-21systemd-journal12:00:180
3528892993915,21cyclictest266-21systemd-journal12:00:180
352889299391,4cyclictest3546260-21ntpq08:09:080
3528892993913,24cyclictest3530367-21cron07:15:010
352889299391,10cyclictest3652040-21ntpq11:19:150
352889299391,10cyclictest3572720-21ssh09:22:170
352889299391,10cyclictest3547853-21ntpq08:14:150
352889499383,25cyclictest657-21snmpd09:04:122
352889499383,25cyclictest657-21snmpd09:04:122
3528893993831,4cyclictest3680662-21latency_hist12:00:021
3528892993832,3cyclictest0-21swapper/009:44:040
352889299382,9cyclictest3690446-21ntpq12:14:160
352889299382,7cyclictest3604485-21ntpq10:09:100
3528892993816,19cyclictest0-21swapper/011:04:230
3528892993812,2cyclictest0-21swapper/009:59:170
352889299381,17cyclictest3571375-21awk09:19:330
352889299381,14cyclictest3694309-21ntpq12:19:180
352889299381,14cyclictest3694309-21ntpq12:19:180
352889299381,14cyclictest3543410-21ps07:59:180
352889299381,11cyclictest3703054-21ssh12:30:200
352889299381,11cyclictest3665935-21ntpq11:40:000
352889299381,11cyclictest3538879-21ntpq07:44:160
352889299381,11cyclictest3538879-21ntpq07:44:160
3528892993810,2cyclictest0-21swapper/007:19:160
352889499373,28cyclictest3686925-21ssh12:09:182
3528894993731,3cyclictest0-21swapper/209:14:172
3528894993731,3cyclictest0-21swapper/209:14:172
3528894993729,5cyclictest453-21systemd-logind08:50:012
3528894993729,5cyclictest3575162-21latency_hist09:25:022
3528894993727,6cyclictest0-21swapper/211:40:012
3528893993729,5cyclictest421-21avahi-daemon11:24:171
3528893993729,5cyclictest3543756-21unixbench_multi07:59:231
352889299379,5cyclictest0-21swapper/007:39:210
3528892993729,5cyclictest3567423-21missed_timers09:14:080
3528892993729,5cyclictest3567423-21missed_timers09:14:080
3528892993713,22cyclictest0-21swapper/010:34:030
352889299371,13cyclictest3669354-21sshd11:44:180
352889299371,11cyclictest3554229-21unixbench-2d08:34:230
352889299371,11cyclictest3539903-21/usr/sbin/munin07:49:050
352889299371,10cyclictest3562679-21memory09:04:060
352889299371,10cyclictest3562679-21memory09:04:060
3528894993628,5cyclictest3632506-21cron10:50:012
3528894993627,6cyclictest3543901-21systemd-run08:00:022
3528893993627,6cyclictest3682959-21fschecks_time12:04:051
3528893993627,6cyclictest3682959-21fschecks_time12:04:051
3528893993622,10cyclictest3587390-21sshd09:44:171
352889299368,25cyclictest0-21swapper/012:39:000
3528892993629,5cyclictest3583773-21ntp_kernel_err09:39:070
3528892993611,4cyclictest3560381-21unixbench_singl08:54:230
3528894993526,5cyclictest956-21mta-sts-daemon11:58:562
3528894993524,8cyclictest3655061-21kernelversion11:24:082
3528893993527,5cyclictest3652828-21ssh11:20:011
3528893993527,5cyclictest3561915-21munin-plugin-st09:00:011
3528893993527,5cyclictest3561915-21munin-plugin-st09:00:011
3528893993527,5cyclictest3534836-21unixbench_singl07:29:211
3528893993526,6cyclictest421-21avahi-daemon08:39:011
3528893993525,6cyclictest3640858-21/usr/sbin/munin11:04:011
3528894993425,6cyclictest0-21swapper/209:10:012
3528894993425,6cyclictest0-21swapper/209:10:012
3528894993422,9cyclictest3622131-21unixbench_multi10:34:242
3528894993422,8cyclictest3652207-21mailstats11:19:202
352889499342,26cyclictest3583821-21ssh09:39:092
352889399344,20cyclictest3676028-21ssh11:54:141
3528893993427,4cyclictest3615705-21kworker/u8:2-xprtiod10:25:521
3528893993427,4cyclictest3615354-21unixbench_multi10:24:221
3528893993427,4cyclictest3546950-21latency_hist08:10:021
3528893993427,4cyclictest0-21swapper/110:19:201
3528893993426,5cyclictest3635316-21wc10:54:231
352889299342,11cyclictest422-21dbus-daemon07:22:170
3528892993413,2cyclictest0-21swapper/008:49:080
3528894993327,3cyclictest0-21swapper/209:44:192
3528894993326,4cyclictest957-21mta-sts-daemon07:34:142
3528894993325,5cyclictest3705228-21fschecks_count12:34:042
3528894993325,5cyclictest3644954-21needreboot11:09:072
3528894993324,6cyclictest491-21gdbus08:19:202
3528894993324,6cyclictest1-21systemd12:20:022
3528894993324,6cyclictest1-21systemd12:20:022
3528893993327,3cyclictest0-21swapper/108:30:021
3528893993326,3cyclictest0-21swapper/110:14:271
3528893993323,6cyclictest2667052-21cups-browsed11:30:431
3528893993321,8cyclictest421-21avahi-daemon10:46:261
352889299331,13cyclictest0-21swapper/007:10:030
352889299331,11cyclictest3534319-21ntpq07:29:110
3528894993224,5cyclictest957-21mta-sts-daemon10:04:102
3528894993223,6cyclictest2667052-21cups-browsed11:01:382
3528894993222,6cyclictest3648898-21mailstats11:14:222
3528894993222,5cyclictest0-21swapper/210:29:122
352889399324,19cyclictest3574234-21ssh09:24:161
3528893993226,3cyclictest0-21swapper/109:20:011
3528893993226,3cyclictest0-21swapper/107:44:101
3528893993226,3cyclictest0-21swapper/107:44:101
3528893993225,3cyclictest0-21swapper/107:14:201
3528893993224,5cyclictest3648514-21needreboot11:14:101
3528893993222,7cyclictest3638710-21unixbench_singl10:59:241
3528893993222,7cyclictest3638710-21unixbench_singl10:59:241
3528894993125,3cyclictest0-21swapper/211:30:012
3528894993124,4cyclictest0-21swapper/208:55:012
3528894993123,3cyclictest0-21swapper/212:25:012
3528894993122,6cyclictest3561800-21unixbench_multi08:59:442
3528894993122,6cyclictest3561800-21unixbench_multi08:59:442
352889399313,21cyclictest3597589-21ssh09:59:111
352889399313,21cyclictest3577121-21ssh09:29:091
352889399313,21cyclictest3577121-21ssh09:29:091
3528893993125,3cyclictest0-21swapper/112:09:221
3528893993122,6cyclictest2667052-21cups-browsed09:12:561
3528893993122,6cyclictest2667052-21cups-browsed09:12:561
352889399312,23cyclictest3645696-21awk11:09:241
3528894993024,3cyclictest0-21swapper/208:05:022
3528894993024,3cyclictest0-21swapper/207:45:012
3528894993024,3cyclictest0-21swapper/207:45:012
3528894993020,7cyclictest453-21systemd-logind07:30:012
3528894993020,7cyclictest3605002-21timerwakeupswit10:09:212
352889499301,23cyclictest3597613-21ssh09:59:122
3528893993025,2cyclictest0-21swapper/107:25:011
3528893993023,4cyclictest3526234-21kworker/u8:3-rpciod07:35:011
3528893993023,4cyclictest0-21swapper/110:09:061
352889399301,23cyclictest3564921-21diskmemload09:54:301
352889499293,20cyclictest657-21snmpd08:14:152
3528894992923,3cyclictest0-21swapper/212:40:012
3528894992923,3cyclictest0-21swapper/207:20:012
3528894992922,4cyclictest0-21swapper/210:19:242
3528894992921,5cyclictest3553992-21systemctl08:34:202
3528894992921,5cyclictest3550747-21ntp_kernel_err08:24:112
3528894992920,5cyclictest3530379-21munin-plugin-st07:15:012
352889399293,19cyclictest657-21snmpd08:44:101
3528893992921,5cyclictest3672098-21cpuspeed11:49:051
3528894992820,5cyclictest477-21rs:main2
3528894992818,7cyclictest266-21systemd-journal07:39:082
352889399286,9cyclictest657-21snmpd10:33:461
3528894992721,3cyclictest3557659-21kworker/u8:13-xprtiod12:03:362
3528894992721,3cyclictest3557659-21kworker/u8:13-xprtiod12:03:362
3528893992718,6cyclictest3531767-21tail07:19:241
3528893992717,7cyclictest956-21mta-sts-daemon08:23:131
3528893992717,7cyclictest3540891-21sh07:50:011
3528893992717,6cyclictest3579540-21/usr/sbin/munin09:34:071
3528893992717,6cyclictest3579540-21/usr/sbin/munin09:34:071
352889499269,12cyclictest422-21dbus-daemon08:44:072
3528894992620,3cyclictest0-21swapper/210:39:202
352889499261,17cyclictest3540381-21sshd07:49:172
3528893992619,4cyclictest3547350-21cpuspeed08:14:021
3528893992519,4cyclictest955-21mta-sts-daemon08:19:031
3528893992415,6cyclictest3564868-21unixbench_singl09:09:201
3528893992415,6cyclictest3564868-21unixbench_singl09:09:201
3528893992415,5cyclictest698-21cron08:45:031
3528894992317,3cyclictest956-21mta-sts-daemon07:54:122
352889399237,10cyclictest271ktimers/108:04:151
352889399232,15cyclictest3553760-21ntpq08:34:111
3528894992213,6cyclictest955-21mta-sts-daemon08:27:472
3528893992211,8cyclictest1-21systemd07:40:011
3528893992015,2cyclictest0-21swapper/107:10:021
352889599182,9cyclictest0-21swapper/309:44:173
352889599173,9cyclictest0-21swapper/312:09:113
352889599172,8cyclictest0-21swapper/308:39:033
352889599172,6cyclictest0-21swapper/310:09:093
352889599163,7cyclictest0-21swapper/310:14:153
352889599162,9cyclictest0-21swapper/310:59:173
352889599162,9cyclictest0-21swapper/310:59:173
352889599162,7cyclictest0-21swapper/312:14:213
352889599161,9cyclictest0-21swapper/310:44:103
352889599161,7cyclictest0-21swapper/308:19:183
352889499167,6cyclictest453-21systemd-logind07:10:012
352889599153,6cyclictest0-21swapper/309:39:123
352889599152,9cyclictest0-21swapper/312:34:123
352889599152,8cyclictest0-21swapper/311:24:153
352889599152,8cyclictest0-21swapper/310:04:103
352889599152,8cyclictest0-21swapper/308:09:093
352889599152,7cyclictest0-21swapper/311:39:573
352889599151,8cyclictest0-21swapper/308:54:113
352889599151,7cyclictest0-21swapper/312:04:003
352889599151,7cyclictest0-21swapper/312:04:003
352889599151,6cyclictest0-21swapper/307:44:103
352889599151,6cyclictest0-21swapper/307:44:103
352889599144,7cyclictest0-21swapper/310:29:093
352889599143,6cyclictest0-21swapper/312:39:183
352889599142,7cyclictest0-21swapper/311:59:033
352889599142,7cyclictest0-21swapper/310:19:123
352889599141,8cyclictest0-21swapper/312:29:113
352889599141,8cyclictest0-21swapper/310:54:133
352889599141,7cyclictest0-21swapper/312:24:123
352889599141,7cyclictest0-21swapper/311:54:023
352889599141,7cyclictest0-21swapper/311:29:193
352889599141,6cyclictest0-21swapper/308:44:073
352889599141,6cyclictest0-21swapper/308:14:103
352889599141,6cyclictest0-21swapper/307:49:123
352889599141,3cyclictest0-21swapper/309:29:123
352889599141,3cyclictest0-21swapper/309:29:123
352889599133,7cyclictest0-21swapper/309:34:113
352889599133,7cyclictest0-21swapper/309:34:113
352889599133,6cyclictest0-21swapper/309:49:183
352889599132,7cyclictest0-21swapper/311:19:143
352889599132,7cyclictest0-21swapper/311:04:273
352889599132,6cyclictest0-21swapper/310:39:053
352889599132,6cyclictest0-21swapper/310:34:063
352889599131,8cyclictest0-21swapper/311:34:093
352889599131,7cyclictest0-21swapper/311:09:113
352889599131,6cyclictest0-21swapper/308:24:173
352889599131,6cyclictest0-21swapper/307:54:153
352889599131,6cyclictest0-21swapper/307:39:103
352889599131,6cyclictest0-21swapper/307:34:103
352889599131,6cyclictest0-21swapper/307:14:123
352889599131,3cyclictest0-21swapper/311:44:093
352889599124,6cyclictest0-21swapper/309:54:003
352889599122,6cyclictest0-21swapper/310:45:183
352889599122,5cyclictest0-21swapper/307:20:013
352889599121,7cyclictest0-21swapper/311:14:073
352889599121,7cyclictest0-21swapper/310:24:003
352889599121,7cyclictest0-21swapper/309:59:093
352889599121,7cyclictest0-21swapper/309:24:093
352889599121,7cyclictest0-21swapper/309:20:013
352889599121,7cyclictest0-21swapper/309:14:133
352889599121,7cyclictest0-21swapper/309:14:133
352889599121,6cyclictest0-21swapper/309:09:103
352889599121,6cyclictest0-21swapper/309:09:103
352889599121,6cyclictest0-21swapper/309:04:103
352889599121,6cyclictest0-21swapper/309:04:103
352889599121,6cyclictest0-21swapper/307:59:133
352889599112,5cyclictest0-21swapper/307:30:013
352889599111,6cyclictest0-21swapper/312:19:013
352889599111,6cyclictest0-21swapper/312:19:013
352889599111,6cyclictest0-21swapper/311:45:353
352889599111,6cyclictest0-21swapper/308:29:323
352889599111,5cyclictest0-21swapper/308:04:103
352889599103,5cyclictest0-21swapper/308:59:333
352889599103,5cyclictest0-21swapper/308:59:333
352889599103,5cyclictest0-21swapper/308:35:013
352889599101,5cyclictest0-21swapper/307:24:083
35288959992,5cyclictest0-21swapper/308:49:083
35288959991,4cyclictest0-21swapper/307:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional