You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 11:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Sun Nov 23, 2025 00:39:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
98056995542,10cyclictest0-21swapper/123:59:151
98055995041,7cyclictest422-21dbus-daemon19:55:010
98057994834,9cyclictest0-21swapper/222:59:212
98057994834,9cyclictest0-21swapper/222:59:212
98056994839,6cyclictest453-21systemd-logind20:25:011
98056994834,11cyclictest283943-21sh00:29:121
98056994834,11cyclictest283943-21sh00:29:121
98055994836,10cyclictest12550irq/33-eth023:14:120
98056994732,10cyclictest224289-21ssh23:09:121
98056994730,14cyclictest159636-21missed_timers21:44:101
98056994635,8cyclictest137406-21mailstats21:14:211
98057994537,5cyclictest274151-21cron00:15:012
98057994537,5cyclictest122055-21systemd-run20:30:012
98057994535,6cyclictest2465701-21cups-browsed23:09:182
98056994537,5cyclictest270614-21true00:10:011
98056994431,10cyclictest1-21systemd21:35:011
98055994437,4cyclictest217694-21sendmail23:00:010
98055994437,4cyclictest217694-21sendmail23:00:010
98057994334,6cyclictest474-21in:imuxsock22:11:422
98055994336,5cyclictest141771-21ssh21:20:010
98055994336,5cyclictest141771-21ssh21:20:010
98055994324,14cyclictest151ktimers/000:19:130
98056994234,5cyclictest172120-21ssh22:00:011
98056994232,7cyclictest164210-21head21:49:261
98056994231,8cyclictest216844-21sh22:59:191
98056994231,8cyclictest216844-21sh22:59:191
98057994135,3cyclictest0-21swapper/223:24:192
98057994134,4cyclictest221476-21debian-sa123:05:012
98057994134,4cyclictest191237-21latency_hist22:25:022
98057994122,3cyclictest171rcu_preempt22:29:182
98056994134,3cyclictest0-21swapper/121:35:301
98056994132,4cyclictest0-21swapper/122:09:191
98057994032,5cyclictest152655-21unixbench_multi21:34:232
98056994032,5cyclictest288581-21cron00:35:001
98056994032,5cyclictest114538-21cron20:05:011
98056994020,9cyclictest271ktimers/123:54:101
98056994013,8cyclictest28-21ksoftirqd/121:19:181
98056994013,8cyclictest28-21ksoftirqd/121:19:181
98055994034,4cyclictest152310-21mailstats21:34:190
98055994026,8cyclictest151ktimers/021:09:160
98057993933,3cyclictest0-21swapper/222:00:012
98057993931,5cyclictest187339-21cron22:20:012
98057993931,5cyclictest148929-21tail21:29:292
98057993924,11cyclictest263320-21cups-browsed00:04:132
98056993931,4cyclictest0-21swapper/122:26:431
98056993930,6cyclictest422-21dbus-daemon23:25:021
98056993928,8cyclictest127796-21timerandwakeup20:49:231
98056993927,9cyclictest251482-21timerwakeupswit23:44:241
98055993932,5cyclictest1-21systemd22:00:010
98055993932,4cyclictest292104-21systemctl00:40:010
98055993927,9cyclictest0-21swapper/000:00:020
98057993830,5cyclictest246731-21fschecks_count23:39:042
98057993830,5cyclictest110070-21latency_hist19:50:022
98057993830,4cyclictest0-21swapper/222:34:142
98057993828,7cyclictest422-21dbus-daemon19:10:012
98057993827,7cyclictest957-21mta-sts-daemon23:52:262
98056993829,6cyclictest955-21mta-sts-daemon20:19:171
98056993829,6cyclictest190114-21needreboot22:24:101
98056993825,9cyclictest134033-21diskmemload22:19:061
98055993825,10cyclictest0-21swapper/023:09:100
98057993729,5cyclictest1-21systemd21:10:022
98057993729,4cyclictest0-21swapper/223:49:152
98057993729,4cyclictest0-21swapper/223:49:152
98057993728,6cyclictest474-21in:imuxsock20:16:172
98057993728,6cyclictest263320-21cups-browsed00:08:312
98056993730,5cyclictest111424-21grep19:54:231
98056993728,6cyclictest957-21mta-sts-daemon23:04:191
98056993725,6cyclictest0-21swapper/123:14:161
9805699372,29cyclictest657-21snmpd22:14:201
98055993726,8cyclictest248130-21sendmail23:40:020
98055993723,8cyclictest12450irq/32-eth022:54:170
98055993723,8cyclictest12450irq/32-eth022:54:170
9805599372,27cyclictest287762-21ssh00:34:170
98055993718,14cyclictest151ktimers/019:34:170
9805799368,22cyclictest209253-21ssh22:49:142
98057993628,5cyclictest134033-21diskmemload22:39:262
9805699363,25cyclictest273196-21ssh00:14:131
9805699363,24cyclictest205569-21ssh22:44:191
98056993627,6cyclictest209190-21ntp_kernel_err22:49:121
98056993627,5cyclictest698-21cron21:10:011
98055993630,4cyclictest221475-21idleruntime-cro23:05:010
98055993627,6cyclictest103098-21fschecks_time19:29:070
98055993627,4cyclictest0-21swapper/022:14:160
98055993622,4cyclictest0-21swapper/019:25:000
98055993622,4cyclictest0-21swapper/019:25:000
98057993528,4cyclictest131054-21kworker/u8:10-nfsiod21:00:012
98057993526,6cyclictest1690079-21ntpd22:54:152
98057993526,6cyclictest1690079-21ntpd22:54:152
98057993525,5cyclictest0-21swapper/200:34:152
98057993524,8cyclictest129515-21cron20:55:012
9805699354,25cyclictest167453-21ssh21:54:171
98056993529,3cyclictest0-21swapper/120:40:011
98056993529,3cyclictest0-21swapper/120:40:011
98056993526,6cyclictest232976-21latency_hist23:20:021
98055993528,4cyclictest185689-21aten2.4-expect22:19:030
98055993527,5cyclictest2465701-21cups-browsed21:46:220
98055993526,7cyclictest98007-21/usr/sbin/munin19:10:020
98057993429,2cyclictest0-21swapper/200:00:022
98057993428,3cyclictest0-21swapper/221:19:222
98057993428,3cyclictest0-21swapper/221:19:222
98057993427,4cyclictest957-21mta-sts-daemon23:42:322
98057993427,3cyclictest0-21swapper/223:34:142
98057993427,3cyclictest0-21swapper/222:04:092
98057993426,5cyclictest132575-21latency_hist21:05:022
98057993426,5cyclictest1-21systemd21:55:012
98057993426,4cyclictest0-21swapper/223:14:112
98057993425,6cyclictest239913-21xargs23:29:252
98057993425,5cyclictest0-21swapper/221:39:162
98056993428,3cyclictest0-21swapper/121:05:011
98056993426,5cyclictest2465701-21cups-browsed23:46:581
98056993426,5cyclictest2465701-21cups-browsed23:46:581
98056993425,5cyclictest175946-21munin-plugin-st22:05:011
98056993424,7cyclictest277791-21idleruntime-cro00:20:011
9805699342,24cyclictest213010-21ssh22:54:171
9805699342,24cyclictest213010-21ssh22:54:171
98055993428,3cyclictest0-21swapper/022:29:160
98055993427,3cyclictest0-21swapper/000:29:150
98055993427,3cyclictest0-21swapper/000:29:150
98057993325,4cyclictest0-21swapper/221:44:162
98057993315,3cyclictest171rcu_preempt00:39:192
98056993327,3cyclictest959-21mta-sts-daemon21:24:181
98056993327,3cyclictest959-21mta-sts-daemon21:24:181
98056993327,3cyclictest0-21swapper/119:20:011
98056993327,3cyclictest0-21swapper/119:20:011
98056993326,3cyclictest0-21swapper/123:34:151
98056993325,5cyclictest291271-21ntpq00:39:171
98056993325,5cyclictest0-21swapper/121:00:011
9805699331,27cyclictest197862-21ssh22:34:151
9805599335,23cyclictest258632-21ssh23:54:170
9805599333,24cyclictest235872-21ssh23:24:200
98055993328,2cyclictest0-21swapper/000:10:010
98055993325,5cyclictest239641-21/usr/sbin/munin23:29:190
98055993324,5cyclictest0-21swapper/000:14:140
9805799321,25cyclictest284051-21sh00:29:142
9805799321,25cyclictest284051-21sh00:29:142
9805699329,5cyclictest28-21ksoftirqd/120:54:141
9805699325,17cyclictest437-21mta-sts-daemon19:14:201
9805699325,17cyclictest437-21mta-sts-daemon19:14:201
98056993224,5cyclictest263320-21cups-browsed00:02:091
98056993224,4cyclictest959-21mta-sts-daemon22:35:491
98056993221,8cyclictest117474-21head20:14:241
98056993212,4cyclictest28-21ksoftirqd/120:09:131
98055993226,4cyclictest99558-21systemd-run19:15:010
98055993226,4cyclictest99558-21systemd-run19:15:010
98055993214,11cyclictest151ktimers/021:54:140
98057993123,5cyclictest114538-21cron20:05:012
98057993123,5cyclictest105571-21latency_hist19:35:022
98057993120,8cyclictest178848-21ntp_kernel_pll_22:09:152
98056993124,4cyclictest125520-21cpuspeed20:44:041
98056993123,5cyclictest106081-21fschecks_time19:39:061
9805699312,22cyclictest148293-21ssh21:29:141
98056993121,8cyclictest101916-21ntp_kernel_err19:24:121
98056993121,8cyclictest101916-21ntp_kernel_err19:24:121
98056993113,11cyclictest0-21swapper/100:24:171
98056993113,11cyclictest0-21swapper/100:24:171
9805599319,18cyclictest149011-21users21:29:300
98055993124,5cyclictest491-21gdbus19:59:230
98055993124,3cyclictest1550ktimers/023:34:160
98055993123,5cyclictest175937-21cron22:05:010
98055993122,6cyclictest421-21avahi-daemon00:02:090
9805599312,24cyclictest599-21lldpd21:20:480
9805599312,24cyclictest599-21lldpd21:20:480
98057993025,2cyclictest0-21swapper/219:40:022
98057993024,3cyclictest0-21swapper/221:49:072
98057993024,3cyclictest0-21swapper/221:15:022
98057993022,5cyclictest126572-21latency_hist20:45:012
98057993022,5cyclictest112892-21unixbench-2d19:59:272
98057993020,7cyclictest232859-21idleruntime-cro23:20:012
98056993020,7cyclictest247514-21switchtime23:39:231
9805699301,24cyclictest239542-21ssh23:29:151
9805599306,21cyclictest12550irq/33-eth022:49:130
98055993025,3cyclictest114533-21kworker/u8:11-rpciod20:15:010
98055993023,5cyclictest0-21swapper/000:25:010
98055993023,5cyclictest0-21swapper/000:25:010
98055993022,5cyclictest255353-21grep23:49:260
98055993022,5cyclictest255353-21grep23:49:260
9805599302,22cyclictest231997-21ssh23:19:180
98057992923,3cyclictest0-21swapper/219:44:132
98057992922,5cyclictest98054-21cyclictest20:15:012
98055992923,4cyclictest114458-21tail20:04:260
98055992922,5cyclictest179828-21latency_hist22:10:020
98055992920,6cyclictest0-21swapper/023:44:240
98055992914,13cyclictest151ktimers/022:44:170
98057992822,3cyclictest0-21swapper/220:40:012
98057992822,3cyclictest0-21swapper/220:40:012
98057992821,3cyclictest0-21swapper/221:24:162
98057992821,3cyclictest0-21swapper/221:24:162
98057992820,5cyclictest422-21dbus-daemon00:20:022
98057992820,5cyclictest280472-21ntp_kernel_err00:24:142
98057992820,5cyclictest280472-21ntp_kernel_err00:24:142
98057992819,7cyclictest116204-21latency_hist20:10:032
98057992815,9cyclictest205519-21sh22:44:162
98055992822,4cyclictest421-21avahi-daemon19:44:210
98055992822,3cyclictest956-21mta-sts-daemon22:33:320
98055992821,5cyclictest155275-21aten_r2power_fr21:39:050
98055992820,2cyclictest0-21swapper/022:39:040
98055992810,12cyclictest151ktimers/020:09:130
98057992721,3cyclictest0-21swapper/220:49:032
98057992720,4cyclictest959-21mta-sts-daemon20:34:192
98057992720,4cyclictest959-21mta-sts-daemon20:34:192
9805699276,4cyclictest28-21ksoftirqd/120:29:131
9805699274,15cyclictest437-21mta-sts-daemon19:34:121
98056992723,3cyclictest98054-21cyclictest19:44:111
98056992721,4cyclictest112991-21head19:59:281
98056992712,3cyclictest171rcu_preempt19:29:211
98055992723,2cyclictest0-21swapper/020:30:010
98055992720,5cyclictest0-21swapper/020:24:130
98055992718,6cyclictest0-21swapper/020:20:010
9805599271,20cyclictest159731-21sh21:44:130
98057992619,4cyclictest110514-21cpuspeed19:54:032
98057992618,5cyclictest1-21systemd20:25:012
98057992613,10cyclictest103489-21awk19:29:162
9805699264,16cyclictest955-21mta-sts-daemon19:49:111
98055992620,3cyclictest0-21swapper/021:14:050
98055992619,5cyclictest2465678-21cupsd19:15:310
98055992619,5cyclictest2465678-21cupsd19:15:310
98055992616,8cyclictest0-21swapper/022:24:120
98057992517,5cyclictest421-21avahi-daemon19:10:102
98057992517,5cyclictest421-21avahi-daemon19:10:102
9805699254,4cyclictest28-21ksoftirqd/120:34:071
9805699254,4cyclictest28-21ksoftirqd/120:34:071
9805599256,2cyclictest171rcu_preempt20:54:220
98055992417,4cyclictest127893-21unixbench-2d20:49:240
98055992415,7cyclictest130352-21needreboot20:59:100
98055992317,4cyclictest122458-21aten2.4-expect20:34:030
98055992317,4cyclictest122458-21aten2.4-expect20:34:030
98055992317,3cyclictest0-21swapper/020:45:010
98055992316,5cyclictest105782-21/usr/sbin/munin19:39:260
98055992312,3cyclictest151ktimers/020:39:130
98055992312,3cyclictest151ktimers/020:39:130
98057992216,4cyclictest957-21mta-sts-daemon19:18:382
98057992216,4cyclictest957-21mta-sts-daemon19:18:382
98057992215,3cyclictest0-21swapper/219:24:152
98057992215,3cyclictest0-21swapper/219:24:152
98056992212,3cyclictest171rcu_preempt19:10:021
98055992217,3cyclictest151ktimers/019:49:160
98055992113,4cyclictest98054-21cyclictest21:04:240
9805899172,9cyclictest0-21swapper/300:29:053
9805899172,9cyclictest0-21swapper/300:29:053
9805899172,9cyclictest0-21swapper/300:04:233
9805899172,8cyclictest0-21swapper/300:14:193
9805899171,9cyclictest0-21swapper/323:19:153
9805899165,6cyclictest0-21swapper/323:14:163
9805899162,8cyclictest0-21swapper/322:49:133
9805899162,8cyclictest0-21swapper/322:04:183
9805899162,8cyclictest0-21swapper/321:59:153
9805899162,7cyclictest0-21swapper/322:24:133
9805899162,7cyclictest0-21swapper/321:28:443
9805899161,8cyclictest0-21swapper/321:49:133
9805899161,8cyclictest0-21swapper/300:24:143
9805899161,8cyclictest0-21swapper/300:24:143
9805899153,9cyclictest0-21swapper/321:34:123
9805899153,7cyclictest0-21swapper/300:34:143
9805899152,8cyclictest0-21swapper/321:44:193
9805899152,7cyclictest0-21swapper/322:14:153
9805899152,6cyclictest0-21swapper/321:09:173
9805899151,9cyclictest0-21swapper/322:59:213
9805899151,9cyclictest0-21swapper/322:59:213
9805899151,8cyclictest0-21swapper/323:24:183
9805899151,8cyclictest0-21swapper/321:54:153
9805899151,8cyclictest0-21swapper/319:34:163
9805899151,7cyclictest0-21swapper/323:29:123
9805899143,7cyclictest0-21swapper/323:49:083
9805899143,7cyclictest0-21swapper/323:49:083
9805899143,4cyclictest0-21swapper/320:34:143
9805899143,4cyclictest0-21swapper/320:34:143
9805899142,7cyclictest0-21swapper/322:54:123
9805899142,7cyclictest0-21swapper/322:54:123
9805899142,7cyclictest0-21swapper/322:44:133
9805899142,7cyclictest0-21swapper/322:29:123
9805899142,6cyclictest0-21swapper/319:24:153
9805899142,6cyclictest0-21swapper/319:24:153
9805899142,6cyclictest0-21swapper/319:14:153
9805899142,6cyclictest0-21swapper/319:14:153
9805899142,5cyclictest0-21swapper/320:04:153
9805899142,5cyclictest0-21swapper/319:44:133
9805899141,9cyclictest0-21swapper/322:36:453
9805899141,9cyclictest0-21swapper/322:19:073
9805899141,8cyclictest0-21swapper/323:51:003
9805899141,8cyclictest0-21swapper/322:09:113
9805899141,8cyclictest0-21swapper/321:19:173
9805899141,8cyclictest0-21swapper/321:19:173
9805899141,7cyclictest0-21swapper/300:39:153
9805899141,6cyclictest0-21swapper/320:29:173
9805899141,6cyclictest0-21swapper/320:19:163
9805899141,6cyclictest0-21swapper/319:54:153
9805899132,7cyclictest0-21swapper/323:35:323
9805899132,7cyclictest0-21swapper/320:24:103
9805899132,7cyclictest0-21swapper/300:10:013
9805899132,6cyclictest0-21swapper/323:44:063
9805899132,6cyclictest0-21swapper/319:40:013
9805899132,6cyclictest0-21swapper/300:00:033
9805899132,4cyclictest0-21swapper/319:29:123
9805899131,7cyclictest0-21swapper/323:04:133
9805899131,6cyclictest0-21swapper/320:54:123
9805899124,5cyclictest0-21swapper/321:14:143
9805899121,8cyclictest0-21swapper/323:09:013
9805899121,7cyclictest0-21swapper/323:34:013
9805899121,7cyclictest0-21swapper/322:34:123
9805899121,6cyclictest0-21swapper/321:20:293
9805899121,6cyclictest0-21swapper/321:20:293
9805899121,6cyclictest0-21swapper/320:59:143
9805899121,6cyclictest0-21swapper/320:49:113
9805899114,5cyclictest0-21swapper/320:09:133
9805899112,7cyclictest0-21swapper/321:39:093
9805899112,7cyclictest0-21swapper/320:44:023
9805899112,6cyclictest0-21swapper/321:05:013
9805899112,6cyclictest0-21swapper/300:19:123
9805899111,6cyclictest0-21swapper/320:14:063
9805899111,6cyclictest0-21swapper/320:00:013
9805899111,6cyclictest0-21swapper/319:49:133
9805899111,6cyclictest0-21swapper/319:19:063
9805899111,6cyclictest0-21swapper/319:19:063
980589993,4cyclictest0-21swapper/320:39:123
980589993,4cyclictest0-21swapper/320:39:123
980589991,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional