You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 22:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Mon Mar 23, 2026 12:43:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1983610995745,7cyclictest2012979-21sshd08:45:242
1983609994738,6cyclictest2071709-21latency_hist10:10:021
1983610994536,6cyclictest2123051-21cut11:10:012
1983609994531,11cyclictest2162357-21sh11:55:231
1983609994334,6cyclictest1090850-21cups-browsed12:00:311
1983610994234,5cyclictest2199517-21systemd-run12:40:012
1983610994135,3cyclictest0-21swapper/211:10:232
1983610994133,5cyclictest1998584-21munin-run08:00:012
1983610994130,7cyclictest1090850-21cups-browsed11:35:212
1983610994130,7cyclictest1090850-21cups-browsed11:35:212
1983609994133,5cyclictest2045677-21basename09:40:001
1983609994133,5cyclictest2045677-21basename09:40:001
1983609994132,6cyclictest2118771-21ssh11:05:011
1983610994032,5cyclictest421-21dbus-daemon09:55:012
1983610994032,5cyclictest2144269-21munin-run11:35:002
1983610994032,5cyclictest2144269-21munin-run11:35:002
1983609993931,5cyclictest2012105-21cut08:45:011
1983610993830,5cyclictest2161320-21systemd-run11:55:012
1983610993830,5cyclictest2161320-21systemd-run11:55:012
1983610993830,5cyclictest2080240-21awk10:20:012
1983610993827,8cyclictest1987452-21ntp_kernel_err07:20:212
1983609993830,5cyclictest2072957-21mailstats10:10:311
1983608993832,4cyclictest695-21cron11:15:010
1983608993832,4cyclictest2088821-21systemd-run10:30:020
1983608993822,14cyclictest2010642-21latency_hist08:40:010
1983610993728,7cyclictest2041331-21/usr/sbin/munin09:35:012
1983610993727,6cyclictest2098211-21kernelversion10:40:192
1983609993729,5cyclictest421-21dbus-daemon11:30:011
1983609993729,5cyclictest421-21dbus-daemon11:30:011
1983609993729,5cyclictest1090850-21cups-browsed10:44:181
1983609993728,6cyclictest1-21systemd11:55:011
1983609993728,6cyclictest1-21systemd11:55:011
1983610993629,4cyclictest2178289-21munin-run12:15:012
1983610993628,5cyclictest420-21avahi-daemon09:40:552
1983610993628,5cyclictest420-21avahi-daemon09:40:552
1983610993628,5cyclictest2004647-21latency_hist08:20:012
1983610993627,6cyclictest2071589-21/usr/sbin/munin10:10:022
1983609993628,5cyclictest1056-21mta-sts-daemon10:20:151
1983609993627,6cyclictest2128687-21mailstats11:15:321
1983609993626,6cyclictest420-21avahi-daemon11:45:181
1983609993626,6cyclictest420-21avahi-daemon11:45:181
1983609993624,9cyclictest2111452-21collect210:55:311
1983608993628,5cyclictest2199517-21systemd-run12:40:010
1983608993626,8cyclictest2019597-21diskmemload12:30:260
1983608993626,7cyclictest2142275-21sshd11:31:230
1983608993626,7cyclictest2142275-21sshd11:31:230
1983610993528,4cyclictest2151323-21kworker/u8:14-xprtiod11:45:322
1983610993528,4cyclictest2151323-21kworker/u8:14-xprtiod11:45:322
1983610993526,6cyclictest421-21dbus-daemon09:20:012
1983610993526,6cyclictest421-21dbus-daemon09:20:012
1983609993527,5cyclictest2024996-21ntpq09:15:231
1983609993527,5cyclictest2024996-21ntpq09:15:231
1983609993527,5cyclictest2015040-21head08:50:371
1983609993526,7cyclictest2093107-21sh10:35:021
1983609993526,6cyclictest1-21systemd12:40:011
1983609993525,6cyclictest1-21systemd11:15:011
1983608993530,3cyclictest2161289-21sh11:55:000
1983608993530,3cyclictest2161289-21sh11:55:000
1983610993426,5cyclictest2000253-21latency_hist08:05:022
1983610993426,4cyclictest1055-21mta-sts-daemon10:35:302
1983610993426,4cyclictest1055-21mta-sts-daemon10:35:302
1983609993428,4cyclictest1983607-21cyclictest12:08:021
1983609993428,3cyclictest0-21swapper/109:15:031
1983609993427,4cyclictest2186769-21cron12:25:011
1983609993426,5cyclictest1-21systemd08:05:021
1983609993423,8cyclictest2107505-21timerwakeupswit10:50:351
1983608993427,5cyclictest2019625-21systemd-run09:10:020
1983608993426,6cyclictest695-21cron09:15:010
1983610993325,5cyclictest2152792-21latency_hist11:45:012
1983610993325,5cyclictest2152792-21latency_hist11:45:012
1983610993325,5cyclictest2132273-21ssh11:20:172
1983610993325,5cyclictest2132273-21ssh11:20:172
1983610993325,5cyclictest2105904-21latency_hist10:50:002
1983610993325,5cyclictest1090850-21cups-browsed10:31:282
1983610993324,6cyclictest548-21gdbus10:58:052
1983609993325,5cyclictest2175447-21sendmail_mailtr12:10:321
1983609993325,5cyclictest1985086-21cron07:15:001
1983609993325,5cyclictest1985086-21cron07:15:001
1983609993324,6cyclictest1090850-21cups-browsed09:21:421
1983609993324,6cyclictest1090850-21cups-browsed09:21:421
1983608993327,4cyclictest2080673-21aten_r2power_en10:20:140
1983608993327,4cyclictest2028971-21fschecks_count09:20:170
1983608993327,4cyclictest2028971-21fschecks_count09:20:170
1983608993327,4cyclictest1-21systemd08:00:010
1983608993327,4cyclictest0-21swapper/011:00:010
1983608993325,5cyclictest2028470-21latency_hist09:20:020
1983608993325,5cyclictest2028470-21latency_hist09:20:020
1983608993314,11cyclictest171rcu_preempt09:25:200
1983608993314,11cyclictest171rcu_preempt09:25:200
1983610993228,2cyclictest0-21swapper/211:01:012
1983610993226,3cyclictest0-21swapper/212:00:312
1983610993225,4cyclictest2019950-21acpi09:10:132
1983610993225,4cyclictest1054-21mta-sts-daemon09:25:172
1983610993225,4cyclictest1054-21mta-sts-daemon09:25:172
1983610993224,5cyclictest1984989-21tail07:10:352
1983610993224,5cyclictest1984989-21tail07:10:352
1983610993223,6cyclictest516-21in:imuxsock10:55:002
1983610993221,8cyclictest2170614-21kernelversion12:05:172
1983609993226,3cyclictest0-21swapper/109:50:141
1983609993225,5cyclictest2144289-21systemd-run11:35:011
1983609993225,5cyclictest2144289-21systemd-run11:35:011
1983609993225,5cyclictest0-21swapper/110:45:241
1983609993224,5cyclictest2194419-21ssh12:33:321
1983609993224,5cyclictest2034360-21unixbench_multi09:25:381
1983609993224,5cyclictest2034360-21unixbench_multi09:25:381
1983609993224,5cyclictest2009611-21fschecks_count08:35:181
1983608993227,3cyclictest1994325-21/usr/sbin/munin07:45:230
1983608993226,4cyclictest421-21dbus-daemon10:10:010
1983608993226,4cyclictest2056071-21unixbench_multi09:50:370
1983608993226,4cyclictest1-21systemd11:25:010
1983608993226,4cyclictest1-21systemd11:25:010
1983608993225,5cyclictest2062471-21sh09:59:070
1983608993224,5cyclictest2037321-21/usr/sbin/munin09:30:170
1983608993222,8cyclictest1985961-21ntp_kernel_pll_07:15:230
1983610993125,3cyclictest0-21swapper/209:40:002
1983610993125,3cyclictest0-21swapper/209:40:002
1983610993124,4cyclictest2180023-21unixbench_singl12:15:382
1983610993124,4cyclictest2131591-21ssh11:20:012
1983610993124,3cyclictest0-21swapper/212:30:242
1983610993123,5cyclictest2017113-21fschecks_count09:00:172
1983610993123,5cyclictest1090850-21cups-browsed08:30:442
1983610993121,7cyclictest2062990-21/usr/sbin/munin10:00:022
1983610993121,7cyclictest2051001-21needreboot09:45:212
1983609993125,3cyclictest0-21swapper/110:05:011
1983609993123,5cyclictest421-21dbus-daemon09:05:021
1983609993123,5cyclictest2077194-21proc_pri10:15:311
1983609993123,5cyclictest2046271-21cpuspeed09:40:151
1983609993123,5cyclictest2046271-21cpuspeed09:40:151
1983609993123,5cyclictest1055-21mta-sts-daemon08:20:351
1983609993123,5cyclictest1055-21mta-sts-daemon08:20:351
1983609993123,4cyclictest0-21swapper/109:30:241
1983609993122,6cyclictest1090850-21cups-browsed12:25:161
1983608993125,4cyclictest421-21dbus-daemon11:45:010
1983608993125,4cyclictest421-21dbus-daemon11:45:010
1983608993125,4cyclictest420-21avahi-daemon07:30:350
1983608993124,5cyclictest420-21avahi-daemon10:44:510
1983608993122,7cyclictest2001611-21sh08:10:010
1983608993122,6cyclictest2105906-21sh10:50:000
198361099304,12cyclictest2019597-21diskmemload10:15:012
1983610993024,3cyclictest0-21swapper/212:30:012
1983610993024,3cyclictest0-21swapper/208:55:002
1983610993022,5cyclictest2140056-21munin-plugin-st11:30:002
1983610993022,5cyclictest2140056-21munin-plugin-st11:30:002
1983610993022,5cyclictest2015612-21fschecks_time08:55:182
1983610993021,6cyclictest2085741-21grep10:25:282
1983609993024,3cyclictest0-21swapper/111:05:311
1983609993024,3cyclictest0-21swapper/110:30:011
1983609993024,3cyclictest0-21swapper/107:20:001
1983609993021,6cyclictest2004576-40mandb08:19:021
1983608993025,3cyclictest1055-21mta-sts-daemon10:50:290
1983608993024,4cyclictest2114825-21/usr/sbin/munin11:00:370
1983608993023,5cyclictest2183820-21proc_pri12:20:290
1983608993021,6cyclictest0-21swapper/007:25:010
1983608993020,4cyclictest2009110-21systemd-run08:35:010
1983608993019,4cyclictest2013348-21timerandwakeup08:45:340
1983610992921,5cyclictest1-21systemd07:45:022
1983609992923,3cyclictest0-21swapper/109:45:541
1983609992922,4cyclictest2002839-21timerandwakeup08:10:341
1983609992922,3cyclictest0-21swapper/110:35:191
1983609992922,3cyclictest0-21swapper/110:35:191
1983609992919,8cyclictest1989012-21ntp_kernel_pll_07:25:261
1983608992925,2cyclictest0-21swapper/011:40:000
1983608992925,2cyclictest0-21swapper/011:40:000
1983608992923,4cyclictest2174740-21fschecks_count12:10:170
1983608992923,4cyclictest2054566-21latency_hist09:50:020
1983608992923,4cyclictest2019597-21diskmemload11:18:180
1983608992923,3cyclictest2114509-21kworker/u8:7-rpciod11:05:300
1983610992823,2cyclictest0-21swapper/210:00:272
1983610992822,3cyclictest0-21swapper/208:45:002
1983610992822,2cyclictest0-21swapper/209:24:392
1983610992822,2cyclictest0-21swapper/209:24:392
1983610992820,5cyclictest1992464-21unixbench_multi07:35:412
1983610992819,7cyclictest2080928-21sh10:20:182
1983609992822,3cyclictest0-21swapper/110:00:011
1983609992821,5cyclictest1055-21mta-sts-daemon07:30:231
1983609992820,5cyclictest2178774-21aten_r2power_po12:15:161
1983609992819,5cyclictest0-21swapper/107:20:211
1983608992822,4cyclictest427-21polkitd11:28:340
1983608992822,4cyclictest427-21polkitd11:28:340
1983608992822,4cyclictest2156669-21ssh11:49:200
1983608992822,4cyclictest2156669-21ssh11:49:200
1983608992822,4cyclictest1055-21mta-sts-daemon08:50:310
1983608992822,3cyclictest0-21swapper/012:00:010
1983608992821,5cyclictest2166066-21telnet12:00:150
1983608992820,6cyclictest2171208-21mailstats12:05:300
1983608992820,6cyclictest1090850-21cups-browsed10:30:220
1983610992722,2cyclictest0-21swapper/212:00:002
1983610992721,4cyclictest1056-21mta-sts-daemon07:25:162
1983610992721,3cyclictest0-21swapper/208:25:002
1983610992721,3cyclictest0-21swapper/208:25:002
1983610992719,5cyclictest2010590-21ls08:40:002
1983609992723,2cyclictest0-21swapper/111:35:351
1983609992723,2cyclictest0-21swapper/111:35:351
1983609992722,2cyclictest0-21swapper/108:45:241
1983609992721,3cyclictest0-21swapper/111:20:301
1983609992721,3cyclictest0-21swapper/111:20:301
1983608992721,4cyclictest2015439-21aten_r2power_po08:55:150
1983608992720,5cyclictest1090846-21cupsd09:37:120
1983608992720,5cyclictest1090846-21cupsd09:37:120
1983608992720,5cyclictest1056-21mta-sts-daemon08:44:070
1983608992710,14cyclictest12450irq/33-eth010:00:270
1983610992622,2cyclictest0-21swapper/207:55:012
1983610992620,3cyclictest0-21swapper/207:20:002
1983610992617,6cyclictest1990985-21grep07:30:352
1983609992620,3cyclictest1997224-21kworker/u8:4-xprtiod08:05:171
1983609992620,3cyclictest0-21swapper/109:00:011
1983609992618,5cyclictest1998595-21idleruntime-cro08:00:011
1983609992618,5cyclictest1992593-21idleruntime-cro07:40:011
1983609992616,7cyclictest2149230-21fschecks_time11:40:181
1983609992616,7cyclictest2149230-21fschecks_time11:40:181
1983608992620,4cyclictest2097389-21sh10:40:000
1983608992620,4cyclictest2097389-21sh10:40:000
1983608992620,4cyclictest2049992-21cron09:45:000
1983608992620,4cyclictest2049992-21cron09:45:000
1983608992619,3cyclictest0-21swapper/007:10:250
1983608992619,3cyclictest0-21swapper/007:10:250
1983608992618,5cyclictest1056-21mta-sts-daemon12:19:130
198361099253,19cyclictest2019597-21diskmemload12:22:092
1983610992521,2cyclictest0-21swapper/209:10:012
1983610992518,4cyclictest1995505-21unixbench_multi07:45:362
1983609992520,2cyclictest0-21swapper/107:50:011
1983609992518,4cyclictest1998618-21kworker/u8:9-nfsiod09:05:231
1983609992517,5cyclictest1996121-21fschecks_time07:50:171
1983609992515,7cyclictest1993982-21unixbench_multi07:40:361
1983608992520,3cyclictest2073275-21unixbench_multi10:10:350
1983608992520,3cyclictest1983607-21cyclictest12:29:510
1983608992520,3cyclictest1054-21mta-sts-daemon08:18:490
1983608992519,4cyclictest2076237-21/usr/sbin/munin10:15:170
1983608992519,4cyclictest0-21swapper/007:30:010
1983610992417,4cyclictest1055-21mta-sts-daemon08:10:342
1983610992416,5cyclictest1983669-21latency_hist07:10:012
1983608992420,3cyclictest1971308-21kworker/u8:10-xprtiod07:40:270
1983608992416,2cyclictest0-21swapper/008:30:020
1983609992317,3cyclictest0-21swapper/108:30:011
1983609992316,4cyclictest0-21swapper/108:30:221
198360899237,13cyclictest151ktimers/008:10:250
198360899232,16cyclictest1996562-21ntp_states07:50:290
198360899222,15cyclictest1992052-21ntp_states07:35:340
198360899222,14cyclictest2017482-21ntpq09:00:250
1983610992116,3cyclictest1985926-21kworker/u8:0-xprtiod08:25:332
1983610992115,3cyclictest0-21swapper/208:05:262
1983609992113,5cyclictest1983613-21systemd-run07:10:011
198360899217,11cyclictest421-21dbus-daemon08:03:240
198361199203,7cyclictest0-21swapper/311:25:233
198361199203,7cyclictest0-21swapper/311:25:233
1983608992014,4cyclictest0-21swapper/007:10:010
198360899201,14cyclictest2005481-21ntpq08:20:230
198360899201,14cyclictest2005481-21ntpq08:20:230
198361199173,8cyclictest0-21swapper/311:50:253
198361199173,8cyclictest0-21swapper/311:50:253
198361199172,9cyclictest0-21swapper/312:30:233
198361199172,8cyclictest0-21swapper/309:30:303
198361199168,4cyclictest0-21swapper/309:48:593
198361199161,8cyclictest0-21swapper/310:50:223
198361199161,8cyclictest0-21swapper/308:19:033
198361199161,7cyclictest0-21swapper/309:35:223
198361199161,7cyclictest0-21swapper/309:35:223
198361199161,10cyclictest0-21swapper/309:20:293
198361199161,10cyclictest0-21swapper/309:20:293
198361199158,4cyclictest0-21swapper/307:10:013
198361199152,8cyclictest0-21swapper/311:15:233
198361199151,7cyclictest0-21swapper/312:00:243
198361199151,6cyclictest0-21swapper/309:00:243
198361199151,6cyclictest0-21swapper/308:20:223
198361199151,6cyclictest0-21swapper/308:20:223
198361199151,6cyclictest0-21swapper/307:40:253
198361199147,4cyclictest0-21swapper/312:25:233
198361199146,4cyclictest0-21swapper/308:55:183
198361199142,7cyclictest0-21swapper/312:20:213
198361199141,8cyclictest0-21swapper/311:20:233
198361199141,8cyclictest0-21swapper/311:20:233
198361199141,6cyclictest0-21swapper/308:50:263
198361199141,6cyclictest0-21swapper/308:00:303
198361199137,4cyclictest0-21swapper/311:30:243
198361199137,4cyclictest0-21swapper/311:30:243
198361199136,4cyclictest0-21swapper/312:35:143
198361199136,4cyclictest0-21swapper/312:05:253
198361199136,4cyclictest0-21swapper/311:55:163
198361199136,4cyclictest0-21swapper/311:45:153
198361199136,4cyclictest0-21swapper/311:45:153
198361199136,4cyclictest0-21swapper/310:45:123
198361199136,4cyclictest0-21swapper/310:30:133
198361199136,4cyclictest0-21swapper/310:20:263
198361199136,4cyclictest0-21swapper/310:00:373
198361199136,4cyclictest0-21swapper/309:05:253
198361199136,4cyclictest0-21swapper/307:50:183
198361199136,4cyclictest0-21swapper/307:30:323
198361199136,4cyclictest0-21swapper/307:30:013
198361199135,4cyclictest0-21swapper/309:15:323
198361199135,4cyclictest0-21swapper/309:15:323
198361199132,6cyclictest0-21swapper/309:50:143
198361199132,6cyclictest0-21swapper/307:35:363
198361199131,8cyclictest0-21swapper/311:00:183
198361199131,8cyclictest0-21swapper/310:55:243
198361199131,8cyclictest0-21swapper/309:29:243
198361199131,8cyclictest0-21swapper/309:29:243
198361199131,7cyclictest0-21swapper/312:15:143
198361199131,7cyclictest0-21swapper/310:35:283
198361199131,7cyclictest0-21swapper/310:35:283
198361199131,7cyclictest0-21swapper/310:25:283
198361199131,7cyclictest0-21swapper/310:05:253
198361199131,6cyclictest0-21swapper/309:55:123
198361199131,6cyclictest0-21swapper/308:35:263
198361199127,3cyclictest0-21swapper/312:10:153
198361199127,3cyclictest0-21swapper/311:05:163
198361199127,3cyclictest0-21swapper/308:40:233
198361199127,3cyclictest0-21swapper/308:10:133
198361199127,3cyclictest0-21swapper/307:10:133
198361199127,3cyclictest0-21swapper/307:10:133
198361199126,3cyclictest0-21swapper/311:40:153
198361199126,3cyclictest0-21swapper/311:40:153
198361199126,3cyclictest0-21swapper/311:35:133
198361199126,3cyclictest0-21swapper/311:35:133
198361199126,3cyclictest0-21swapper/309:40:133
198361199126,3cyclictest0-21swapper/309:40:133
198361199126,3cyclictest0-21swapper/309:10:133
198361199126,3cyclictest0-21swapper/308:30:153
198361199122,5cyclictest0-21swapper/308:25:133
198361199121,7cyclictest0-21swapper/311:10:173
198361199121,7cyclictest0-21swapper/310:40:133
198361199121,7cyclictest0-21swapper/310:10:213
198361199121,6cyclictest0-21swapper/308:45:233
198361199121,6cyclictest0-21swapper/308:05:323
198361199116,3cyclictest0-21swapper/310:15:033
198361199116,3cyclictest0-21swapper/307:45:143
198361199116,3cyclictest0-21swapper/307:20:173
198361199116,3cyclictest0-21swapper/307:15:143
198361199106,3cyclictest0-21swapper/307:55:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional