You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 16:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Aug 26, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
280692238163,24sleep30-21swapper/307:06:393
280912234162,58sleep00-21swapper/007:06:560
280462203166,25sleep20-21swapper/207:06:222
263612202164,25sleep10-21swapper/107:05:041
760021470,3sleep32843499cyclictest10:00:173
319532570,4sleep011-21rcuc/012:10:160
68472550,1sleep10-21swapper/107:30:221
233992550,2sleep223403-21fschecks_count11:50:162
242852540,2sleep20-21swapper/208:10:212
266262520,4sleep22843099cyclictest11:55:262
2843499491,38cyclictest0-21swapper/311:53:233
163512460,3sleep20-21swapper/211:35:022
2842099441,5cyclictest1384-21kworker/0:208:35:210
28420993911,13cyclictest29431-21awk12:05:010
28420993911,13cyclictest29431-21awk12:05:010
28424993814,14cyclictest11589-21cut12:35:231
2842499378,21cyclictest14634-21snmpd09:44:001
2842099371,16cyclictest31385-21kworker/0:312:15:190
2842099370,16cyclictest2422-21kworker/0:107:15:210
2842499368,20cyclictest17468-21chrt10:21:111
2842499361,3cyclictest0-21swapper/109:14:471
2843499351,15cyclictest0-21swapper/310:42:093
28430993513,3cyclictest2607-21/usr/sbin/munin09:50:182
28430993511,3cyclictest7002-21/usr/sbin/munin10:00:162
28430993511,3cyclictest5742-21/usr/sbin/munin07:30:242
28430993511,3cyclictest1334-21/usr/sbin/munin07:20:252
28430993510,4cyclictest29525-21/usr/sbin/munin10:50:162
28430993510,2cyclictest18981-21df_inode11:40:152
2842499351,25cyclictest0-21swapper/107:27:531
28424993512,20cyclictest6414-21df12:25:131
28424993512,12cyclictest29427-21awk12:05:011
28424993512,12cyclictest29427-21awk12:05:011
2842099351,14cyclictest0-21swapper/009:52:050
2842099351,14cyclictest0-21swapper/008:47:250
2842099350,14cyclictest16019-21kworker/0:011:35:270
2843499341,13cyclictest24740-21if_err_enp1s009:25:173
2843099343,7cyclictest2966-40dnf11:04:332
2843099341,5cyclictest18498-21grep10:25:152
28430993412,3cyclictest3546-21/usr/sbin/munin07:25:192
28430993412,3cyclictest15182-21/usr/sbin/munin09:05:202
28430993411,3cyclictest11016-21latency07:40:192
2843099341,13cyclictest0-21swapper/209:12:492
2842499349,10cyclictest30715-21ntp_states12:05:221
28424993411,19cyclictest17152-21munin-run09:10:011
2842099346,13cyclictest10267-21perf12:35:010
2842099341,15cyclictest12633-21kworker/0:209:07:370
2843499331,14cyclictest0-21swapper/307:15:333
2843099332,13cyclictest25503-21sed09:25:262
2843099332,13cyclictest2282-21grep12:15:172
2843099331,6cyclictest17153-21diskmemload12:25:022
28430993312,3cyclictest29976-21expr12:05:142
2843099331,10cyclictest0-21swapper/210:36:432
28430993310,3cyclictest3429-21cat11:05:122
2842499338,10cyclictest9199-21memory12:30:231
2842499331,24cyclictest0-21swapper/111:16:471
2842099339,11cyclictest8979-21idleruntime-cro10:05:010
2842099331,14cyclictest0-21swapper/010:36:370
2843499321,12cyclictest0-21swapper/310:21:013
2843099329,3cyclictest8755-21pmu-power11:15:252
2843099324,3cyclictest86550irq/126-enp1s0-09:45:232
2843099324,3cyclictest86550irq/126-enp1s0-09:45:232
2843099321,5cyclictest31827-21gltestperf10:55:172
2842499326,14cyclictest3292-21hwlatdetect09:50:171
2842499325,21cyclictest30000-21kworker/u8:110:07:591
28424993214,15cyclictest10912-21ntp_states11:20:221
28424993213,7cyclictest16437-21/usr/sbin/munin11:35:161
28424993213,6cyclictest14531-21/usr/sbin/munin07:50:251
28424993211,6cyclictest7940-21/usr/sbin/munin07:35:291
28424993211,6cyclictest7940-21/usr/sbin/munin07:35:291
2842099321,17cyclictest0-21swapper/011:44:530
2842099321,14cyclictest5077-21kworker/0:011:10:230
2842099321,13cyclictest24952-21kworker/0:012:05:130
2842099321,12cyclictest0-21swapper/007:39:170
2842099321,12cyclictest0-21swapper/007:39:170
2842099321,11cyclictest24779-21cstates10:40:130
2843499314,7cyclictest17153-21diskmemload11:04:333
2843499311,15cyclictest0-21swapper/309:03:173
2843499311,14cyclictest0-21swapper/311:19:293
2843099319,4cyclictest759-21dbus-daemon10:20:092
2843099315,3cyclictest27516-21awk08:20:012
2843099313,3cyclictest86550irq/126-enp1s0-11:20:202
2843099312,4cyclictest14634-21snmpd09:17:302
2843099312,3cyclictest3942-21latency_hist08:40:012
2843099311,4cyclictest0-21swapper/210:16:522
2843099311,3cyclictest29705-21kworker/2:007:35:452
2843099311,3cyclictest29705-21kworker/2:007:35:452
28430993112,3cyclictest5543-21cat08:40:262
28430993112,3cyclictest5543-21cat08:40:262
28430993110,3cyclictest8285-21/usr/sbin/munin12:30:192
28430993110,3cyclictest7668-21cat08:45:242
2842499319,11cyclictest25907-21hddtemp_smartct11:55:161
2842499315,20cyclictest30000-21kworker/u8:110:10:521
28424993113,15cyclictest14589-21diskstats11:30:161
28424993112,10cyclictest19967-21latency_hist10:30:031
2842499310,16cyclictest0-21swapper/111:52:111
2842099311,16cyclictest0-21swapper/009:55:170
2842099311,14cyclictest27206-21munin-run12:00:010
2842099311,14cyclictest0-21swapper/009:36:450
2842099311,14cyclictest0-21swapper/009:32:370
2842099310,14cyclictest5919-21kworker/0:008:53:150
2842099310,14cyclictest16280-21kworker/0:107:58:350
2842099310,13cyclictest26388-21kworker/0:010:46:390
188162310,2sleep30-21swapper/310:25:183
2843499301,22cyclictest0-21swapper/309:35:273
2843499301,12cyclictest0-21swapper/309:46:293
2843499301,12cyclictest0-21swapper/309:46:293
2843099304,5cyclictest836-21runrttasks08:02:392
2843099304,3cyclictest86550irq/126-enp1s0-10:40:162
2843099303,6cyclictest30573-21/usr/sbin/munin09:40:152
2843099301,7cyclictest5452-21/usr/sbin/munin11:10:182
2843099301,5cyclictest10536-21latency_hist08:55:012
2843099301,2cyclictest0-21swapper/207:55:172
2843099301,27cyclictest0-21swapper/208:08:152
28430993011,3cyclictest32145-21expr12:10:182
28430993011,3cyclictest21987-21expr09:20:142
2843099301,13cyclictest0-21swapper/208:55:032
28430993010,3cyclictest32346-21sed07:15:222
28430993010,3cyclictest20925-21cpuspeed_turbos11:45:182
2842499308,13cyclictest8741-21cstates08:50:141
2842499308,11cyclictest19995-21ntp_states08:00:231
2842499307,8cyclictest1980-21/usr/sbin/munin08:35:211
2842499307,12cyclictest13573-21tail11:25:271
2842499305,19cyclictest0-21swapper/111:00:261
28424993020,7cyclictest6067-21smart_sda09:55:291
2842499301,3cyclictest0-21swapper/107:21:031
28424993012,15cyclictest14420-21irqstats10:15:221
28424993011,11cyclictest23198-21cat08:10:011
28424993010,17cyclictest28140-21perf09:35:011
2842099301,13cyclictest6159-21awk08:45:010
2842099301,13cyclictest6159-21awk08:45:010
2842099301,13cyclictest25214-21kworker/0:308:15:380
2842099301,13cyclictest10209-21kworker/0:212:37:150
2842099300,8cyclictest18594-21kworker/0:009:15:210
2842099300,13cyclictest18594-21kworker/0:009:11:350
2842099300,12cyclictest485-21kworker/0:111:00:570
2842099300,11cyclictest7321-21kworker/0:111:29:150
2842099300,11cyclictest0-21swapper/012:21:250
28434992916,3cyclictest0-21swapper/311:20:233
2843499291,10cyclictest0-21swapper/308:00:293
2843099293,11cyclictest32-21ksoftirqd/210:09:232
2843099292,25cyclictest32-21ksoftirqd/208:22:192
2843099291,5cyclictest28076-21gltestperf12:00:172
2843099291,5cyclictest28076-21gltestperf12:00:172
2843099291,4cyclictest0-21swapper/211:37:442
2843099291,4cyclictest0-21swapper/211:25:222
2843099291,2cyclictest0-21swapper/212:25:142
2843099291,2cyclictest0-21swapper/209:05:012
2843099291,22cyclictest0-21swapper/208:34:332
2843099291,11cyclictest0-21swapper/208:29:132
28430992910,3cyclictest20509-21expr10:30:192
2842499299,11cyclictest4039-21if_err_enp2s011:05:171
2842499298,11cyclictest6311-21cat08:45:011
2842499298,11cyclictest6311-21cat08:45:011
2842499297,8cyclictest20839-21/usr/sbin/munin11:45:281
2842499296,12cyclictest21837-21cpu09:20:141
28424992921,5cyclictest17796-21ntp_states07:55:231
28424992920,5cyclictest31106-21smart_sda07:15:031
28424992920,5cyclictest28056-21smart_sda10:45:271
28424992915,11cyclictest24943-21diskstats10:40:161
28424992912,4cyclictest31198-21/usr/sbin/munin10:55:181
2842499291,10cyclictest0-21swapper/108:22:071
2842099291,14cyclictest485-21kworker/0:110:57:270
2842099291,14cyclictest0-21swapper/010:26:450
2842099291,14cyclictest0-21swapper/010:05:570
2842099291,12cyclictest485-21kworker/0:111:05:530
2842099291,12cyclictest0-21swapper/011:23:090
28420992910,16cyclictest14316-21basename07:50:020
2842099290,15cyclictest16019-21kworker/0:011:46:010
2842099290,13cyclictest14933-21kworker/0:109:25:130
2842099290,11cyclictest3509-21kworker/0:112:28:390
2843499281,3cyclictest0-21swapper/311:55:413
2843499281,11cyclictest0-21swapper/310:48:213
2843099281,5cyclictest17153-21diskmemload10:49:352
2843099281,3cyclictest0-21swapper/209:36:232
2843099281,17cyclictest0-21swapper/207:45:072
28430992811,3cyclictest26277-21expr09:30:182
2843099281,11cyclictest0-21swapper/207:54:252
2842499289,11cyclictest1687-21/usr/sbin/munin12:15:241
2842499288,7cyclictest1446-21perf12:15:011
2842499281,9cyclictest0-21swapper/108:33:191
28424992819,5cyclictest6712-21smart_sda11:10:271
28424992813,12cyclictest30665-21if_enp1s008:25:171
28424992813,12cyclictest20208-21/usr/sbin/munin10:30:341
28424992812,13cyclictest4743-21sort12:20:201
28424992811,4cyclictest23482-21cpuspeed_turbos08:10:141
28424992811,14cyclictest23324-21missed_timers10:35:251
28424992810,15cyclictest759-21dbus-daemon09:46:371
28424992810,15cyclictest759-21dbus-daemon09:46:371
2842499280,22cyclictest0-21swapper/109:15:131
2842099281,17cyclictest22505-21kworker/0:311:54:390
2842099281,12cyclictest21865-21kworker/0:210:33:590
2842099281,11cyclictest14933-21kworker/0:109:22:010
2843499271,9cyclictest28506-21ntp_states12:00:213
2843499271,9cyclictest28506-21ntp_states12:00:213
28434992717,3cyclictest0-21swapper/308:50:213
2843499271,13cyclictest0-21swapper/312:11:093
2843499271,12cyclictest325042sleep309:44:393
28434992710,4cyclictest3647-21crond12:20:013
2843499270,10cyclictest0-21swapper/307:25:073
2843099271,2cyclictest86550irq/126-enp1s0-09:59:022
28430992711,3cyclictest28675-21cpu07:10:132
2843099271,12cyclictest0-21swapper/212:38:422
2842499279,15cyclictest19628-21grep11:40:211
2842499279,15cyclictest12734-21latency_hist09:00:021
2842499278,7cyclictest0-21swapper/108:15:181
2842499278,16cyclictest15153-21cat09:05:021
2842499277,9cyclictest17153-21diskmemload09:39:551
28424992712,12cyclictest29006-21/usr/sbin/munin10:50:131
28424992711,4cyclictest24314-21diskstats09:25:151
28424992710,6cyclictest17153-21diskmemload10:01:391
2842099271,12cyclictest27437-21kworker/0:008:21:380
2842099271,11cyclictest26388-21kworker/0:010:50:270
2842099271,10cyclictest32196-21kworker/0:209:40:550
2842099270,11cyclictest0-21swapper/010:15:570
2842099270,10cyclictest7321-21kworker/0:111:31:150
2843499261,3cyclictest0-21swapper/307:13:053
2843499261,12cyclictest0-21swapper/309:14:453
2843499261,12cyclictest0-21swapper/308:35:013
2843499261,11cyclictest0-21swapper/311:47:573
28434992610,4cyclictest0-21swapper/310:05:293
28424992617,6cyclictest13556-21/usr/sbin/munin07:45:241
2842499261,10cyclictest0-21swapper/107:43:351
2842099261,12cyclictest0-21swapper/007:23:170
2842099261,11cyclictest12633-21kworker/0:209:02:510
2842099261,10cyclictest12081-21kworker/0:007:52:330
2842099260,15cyclictest0-21swapper/010:10:170
2843499259,3cyclictest0-21swapper/309:05:113
2843499251,16cyclictest0-21swapper/311:14:133
2843499251,14cyclictest5155-21diskstats09:55:173
2843499251,14cyclictest23317-21memory10:35:233
2843499251,11cyclictest0-21swapper/311:40:533
2843499251,11cyclictest0-21swapper/310:10:053
2843499251,11cyclictest0-21swapper/308:19:413
2843099250,10cyclictest32-21ksoftirqd/210:12:252
28424992517,5cyclictest6720-21diskstats08:45:141
2842099259,8cyclictest0-21swapper/008:55:570
2842099259,8cyclictest0-21swapper/008:26:540
2842099251,18cyclictest0-21swapper/011:19:250
2842099251,13cyclictest0-21swapper/007:25:330
2842099250,11cyclictest15529-21kworker/0:310:21:070
2843499243,5cyclictest0-21swapper/312:30:313
2843499241,9cyclictest0-21swapper/312:37:413
2843499241,4cyclictest0-21swapper/308:11:233
2843499241,14cyclictest0-21swapper/312:20:173
2843499241,14cyclictest0-21swapper/311:35:013
2843499241,13cyclictest20880-21wc09:15:253
2843499241,13cyclictest0-21swapper/309:50:153
2843499241,12cyclictest1459-21cpuspeed_turbos07:20:153
2843499241,11cyclictest0-21swapper/309:21:173
2843499240,13cyclictest0-21swapper/307:35:273
2843499240,13cyclictest0-21swapper/307:35:273
28424992414,7cyclictest29113-21tune2fs07:10:161
2842099242,8cyclictest2422-21kworker/0:107:30:110
2842099241,10cyclictest0-21swapper/008:03:330
2843499239,3cyclictest0-21swapper/307:55:153
2843499231,9cyclictest0-21swapper/311:35:513
2843499231,14cyclictest0-21swapper/309:30:093
28434992310,3cyclictest0-21swapper/308:25:133
2842099236,4cyclictest25328-21awk08:15:010
2842099236,3cyclictest0-21swapper/008:31:010
2842099231,15cyclictest0-21swapper/009:46:210
2842099231,15cyclictest0-21swapper/009:46:210
2842099231,13cyclictest2422-21kworker/0:107:10:190
2842099231,12cyclictest0-21swapper/008:06:290
34922220,4sleep339-21rcuc/311:05:123
2843499226,3cyclictest11583-21sed07:40:253
2843499224,5cyclictest0-21swapper/307:33:533
2843499224,3cyclictest30716-21taskset10:51:533
2843499221,8cyclictest0-21swapper/312:29:333
28434992212,2cyclictest86550irq/126-enp1s0-12:07:033
2843499221,12cyclictest0-21swapper/310:16:253
2843499218,4cyclictest0-21swapper/310:56:243
2843499215,3cyclictest86850irq/129-enp1s0-10:30:273
2843499211,13cyclictest0-21swapper/308:56:533
2843499210,10cyclictest0-21swapper/311:28:493
2842099210,12cyclictest9563-21kworker/0:307:41:310
2843499201,7cyclictest0-21swapper/307:53:293
2843499201,11cyclictest32143-21/usr/sbin/munin08:30:233
2843499201,10cyclictest0-21swapper/308:46:373
2843499190,9cyclictest0-21swapper/308:22:373
2843499190,11cyclictest0-21swapper/308:05:273
2843499182,3cyclictest0-21swapper/308:44:493
2843499182,3cyclictest0-21swapper/308:44:493
2843499181,8cyclictest12612-21expr07:45:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional