You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-14 - 21:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jul 14, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
192162238164,25sleep20-21swapper/207:07:132
191312235158,24sleep00-21swapper/007:06:060
192272234163,57sleep30-21swapper/307:07:223
174782234163,58sleep10-21swapper/107:05:111
19530991171,114cyclictest0-21swapper/209:23:492
195309910197,3cyclictest0-21swapper/208:13:362
319702700,7sleep2311ktimersoftd/207:35:192
304172680,2sleep00-21swapper/007:30:230
1952399391,14cyclictest0-21swapper/109:56:441
1951499358,12cyclictest7614-21idleruntime-cro07:55:010
1951499358,12cyclictest7614-21idleruntime-cro07:55:010
1952399331,6cyclictest0-21swapper/110:33:321
1952399331,13cyclictest0-21swapper/107:15:161
1952399322,14cyclictest6588-21df09:05:161
1952399321,9cyclictest0-21swapper/111:06:001
1952399321,9cyclictest0-21swapper/111:05:591
1952399321,13cyclictest0-21swapper/110:25:401
1951499325,12cyclictest6657-21perf10:20:000
1952399311,5cyclictest0-21swapper/110:55:401
1952399311,16cyclictest0-21swapper/112:19:161
1952399311,13cyclictest0-21swapper/112:35:521
1952399311,13cyclictest0-21swapper/111:10:121
1952399311,11cyclictest0-21swapper/111:21:211
1952399301,6cyclictest0-21swapper/109:45:001
1952399301,12cyclictest0-21swapper/109:28:041
1952399301,10cyclictest0-21swapper/110:43:301
1953099292,10cyclictest24807-40dnf12:14:332
1953099292,10cyclictest24807-40dnf12:14:332
1952399291,5cyclictest0-21swapper/107:35:531
1952399291,13cyclictest0-21swapper/112:28:201
1952399291,13cyclictest0-21swapper/109:35:461
1952399291,11cyclictest0-21swapper/108:05:201
1952399291,10cyclictest0-21swapper/111:52:071
86442280,3sleep2311ktimersoftd/209:10:132
1952399281,18cyclictest0-21swapper/110:50:281
1952399281,14cyclictest0-21swapper/112:03:161
1952399281,11cyclictest0-21swapper/110:21:321
1951499286,10cyclictest1637-21perf08:55:010
1952399271,13cyclictest0-21swapper/111:01:561
1952399271,12cyclictest0-21swapper/109:31:201
1952399271,11cyclictest0-21swapper/112:30:581
1952399271,11cyclictest0-21swapper/110:48:041
1952399271,10cyclictest0-21swapper/109:22:121
1952399270,11cyclictest0-21swapper/110:11:221
1952399270,10cyclictest0-21swapper/110:35:461
1953599261,23cyclictest0-21swapper/308:40:233
1953099261,4cyclictest581-21systemd-journal10:40:012
1952399263,12cyclictest0-21swapper/111:39:351
1952399261,9cyclictest0-21swapper/109:10:261
1952399261,9cyclictest0-21swapper/108:39:121
1952399261,11cyclictest0-21swapper/112:21:481
1952399261,11cyclictest0-21swapper/110:18:181
1952399261,11cyclictest0-21swapper/108:29:451
1952399261,10cyclictest0-21swapper/109:47:481
1952399260,9cyclictest0-21swapper/108:32:111
1951499263,15cyclictest0-21swapper/010:55:560
107822260,4sleep05346-21kworker/0:308:00:180
312962250,2sleep232-21ksoftirqd/211:15:172
1953599251,16cyclictest14634-21snmpd11:18:463
1952399259,3cyclictest27114-21ntp_states09:50:241
1952399251,9cyclictest0-21swapper/111:27:161
1952399251,9cyclictest0-21swapper/107:51:401
1952399251,9cyclictest0-21swapper/107:51:401
1952399251,7cyclictest0-21swapper/107:58:001
1952399251,17cyclictest0-21swapper/110:04:161
1952399251,17cyclictest0-21swapper/110:04:161
1952399251,11cyclictest0-21swapper/111:48:181
1952399251,11cyclictest0-21swapper/108:40:321
1952399251,10cyclictest0-21swapper/112:06:161
1952399251,10cyclictest0-21swapper/111:30:031
1951499257,11cyclictest0-21swapper/011:05:130
1951499257,11cyclictest0-21swapper/011:05:130
1951499251,16cyclictest0-21swapper/009:56:260
1953599241,21cyclictest0-21swapper/310:06:133
1953599241,12cyclictest842-21lldpd07:43:113
1953099241,6cyclictest0-21swapper/212:26:142
1952399247,9cyclictest0-21swapper/109:18:321
1952399242,19cyclictest24483-21users12:10:311
1952399242,19cyclictest24483-21users12:10:311
1952399241,6cyclictest0-21swapper/108:04:261
1952399241,15cyclictest0-21swapper/108:19:121
1952399241,12cyclictest0-21swapper/108:14:341
1952399241,11cyclictest0-21swapper/110:08:281
1951499242,5cyclictest31791-21munin-run08:50:010
1951499241,21cyclictest0-21swapper/010:45:490
1951499241,17cyclictest0-21swapper/012:12:260
1951499241,17cyclictest0-21swapper/012:12:260
1951499241,17cyclictest0-21swapper/009:11:520
1951499241,15cyclictest0-21swapper/012:38:520
1951499241,15cyclictest0-21swapper/009:39:450
1951499240,22cyclictest0-21swapper/011:52:340
1951499240,22cyclictest0-21swapper/011:21:040
1951499240,22cyclictest0-21swapper/009:27:550
1953599232,7cyclictest11275-21/usr/sbin/munin10:30:243
1953599231,7cyclictest0-21swapper/310:12:553
1953599231,11cyclictest14634-21snmpd08:53:173
1953599231,10cyclictest17250-21/usr/sbin/munin09:30:183
1953099236,7cyclictest780-21rngd08:59:142
1953099236,7cyclictest780-21rngd08:59:142
1952399231,9cyclictest0-21swapper/109:01:221
1952399231,6cyclictest0-21swapper/107:45:071
1952399231,19cyclictest2315-21ntp_states07:40:251
1952399231,16cyclictest0-21swapper/111:15:221
1952399231,10cyclictest0-21swapper/107:25:401
1951499238,9cyclictest0-21swapper/007:18:330
1951499238,4cyclictest10673-21egrep09:15:130
1951499236,9cyclictest0-21swapper/012:00:200
19514992317,3cyclictest23842-21/usr/sbin/munin09:45:250
19514992317,3cyclictest14435-21/usr/sbin/munin08:10:270
19514992314,3cyclictest0-21swapper/011:26:220
1951499231,15cyclictest86550irq/126-enp1s0-12:15:270
1951499231,15cyclictest0-21swapper/012:29:050
1951499231,15cyclictest0-21swapper/011:16:050
1951499231,15cyclictest0-21swapper/010:35:410
1951499231,15cyclictest0-21swapper/010:25:430
1951499231,15cyclictest0-21swapper/010:11:020
1951499231,15cyclictest0-21swapper/009:40:530
1951499231,14cyclictest0-21swapper/011:36:230
1951499231,14cyclictest0-21swapper/011:34:280
1951499231,14cyclictest0-21swapper/009:30:190
1951499231,10cyclictest11884-21/usr/sbin/munin11:45:270
1953599227,4cyclictest7547-21hddtemp_smartct10:20:163
1953599225,9cyclictest27706-21/usr/sbin/munin11:05:223
1953599225,9cyclictest27706-21/usr/sbin/munin11:05:223
1953599223,16cyclictest28796-21tail12:20:273
1953599221,6cyclictest14777-21kworker/u8:309:51:003
1953599221,4cyclictest0-21swapper/311:52:343
1953599221,3cyclictest0-21swapper/310:45:493
1953599221,3cyclictest0-21swapper/310:36:363
1953599221,11cyclictest26736-21sshd07:24:333
1953599221,10cyclictest8456-21/usr/sbin/munin09:10:283
1953599221,10cyclictest3096-21/usr/sbin/munin11:25:163
1953599221,10cyclictest14634-21snmpd09:43:283
1953599221,10cyclictest14634-21snmpd08:08:083
1953599221,10cyclictest14634-21snmpd07:28:003
1953099223,5cyclictest0-21swapper/212:20:142
1953099221,8cyclictest0-21swapper/210:18:472
19530992210,3cyclictest18944-21fschecks_time12:00:172
1952399221,16cyclictest0-21swapper/108:58:041
1952399221,16cyclictest0-21swapper/108:58:041
1952399221,15cyclictest29993-21if_err_enp2s007:30:171
1952399221,15cyclictest0-21swapper/108:50:431
1952399221,13cyclictest0-21swapper/107:24:421
1952399221,13cyclictest0-21swapper/107:10:181
1952399220,17cyclictest0-21swapper/108:50:001
1951499227,8cyclictest0-21swapper/011:56:540
1951499227,10cyclictest0-21swapper/011:40:430
1951499226,9cyclictest0-21swapper/009:52:490
1951499226,4cyclictest0-21swapper/012:06:420
1951499222,3cyclictest24847-21cut11:00:160
1951499221,15cyclictest0-21swapper/010:08:080
1951499221,14cyclictest0-21swapper/012:31:580
1951499221,14cyclictest0-21swapper/012:23:490
1951499221,14cyclictest0-21swapper/010:54:560
1951499221,14cyclictest0-21swapper/010:42:030
1953599219,3cyclictest14634-21snmpd10:42:383
1953599216,3cyclictest7828-21/usr/sbin/munin07:55:123
1953599215,8cyclictest7491-21/usr/sbin/munin11:35:223
1953599215,4cyclictest12800-21missed_timers11:45:203
1953599214,8cyclictest1850-21/usr/sbin/munin08:55:173
1953599214,8cyclictest1850-21/usr/sbin/munin08:55:173
1953599212,2cyclictest86550irq/126-enp1s0-12:31:323
1953599211,9cyclictest6439-21pmu-power11:30:263
19535992119,1cyclictest0-21swapper/308:20:273
1953599211,5cyclictest26427-21kworker/u8:111:21:043
19535992115,3cyclictest30558-21cut10:00:133
19535992115,3cyclictest30558-21cut10:00:133
1953599211,3cyclictest0-21swapper/311:10:583
1953599211,3cyclictest0-21swapper/309:27:553
1953599211,12cyclictest842-21lldpd09:05:123
1953599211,10cyclictest25651-21fschecks_time12:15:203
1953099216,4cyclictest1838-21sed10:05:282
1953099213,2cyclictest86850irq/129-enp1s0-10:40:322
1953099213,15cyclictest31424-21grep10:00:202
1953099213,15cyclictest31424-21grep10:00:202
1953099212,9cyclictest0-21swapper/212:05:142
1953099211,8cyclictest0-21swapper/209:50:432
1953099211,7cyclictest0-21swapper/211:35:552
1953099211,7cyclictest0-21swapper/210:54:162
1953099211,7cyclictest0-21swapper/210:30:142
1953099211,7cyclictest0-21swapper/209:55:202
1953099211,6cyclictest0-21swapper/211:41:122
1953099211,6cyclictest0-21swapper/209:18:582
1953099211,4cyclictest0-21swapper/212:30:182
1953099211,4cyclictest0-21swapper/210:55:522
1952399210,15cyclictest0-21swapper/111:55:141
1952399210,13cyclictest0-21swapper/108:24:421
1951499211,14cyclictest0-21swapper/010:30:500
1951499211,14cyclictest0-21swapper/009:20:450
1951499210,2cyclictest86550irq/126-enp1s0-10:00:270
1951499210,2cyclictest86550irq/126-enp1s0-10:00:270
1951499210,15cyclictest0-21swapper/008:35:200
1953599206,7cyclictest3439-21/usr/sbin/munin07:45:263
1953599203,8cyclictest29822-21/usr/sbin/munin08:45:273
1953599202,9cyclictest25434-21users09:45:283
1953599202,6cyclictest1859-21df_inode12:35:153
1953599202,5cyclictest21501-21irqstats12:05:193
1953599202,3cyclictest0-21swapper/309:20:463
1953599201,8cyclictest14634-21snmpd09:03:193
1953599201,6cyclictest18823-21diskstats12:00:163
1953599201,6cyclictest0-21swapper/312:13:073
1953599201,6cyclictest0-21swapper/312:13:073
19535992013,3cyclictest0-21swapper/310:55:293
1953599201,16cyclictest29210-21/usr/sbin/munin07:30:173
1953599201,10cyclictest25088-21gltestperf11:00:173
1953599201,10cyclictest0-21swapper/312:28:013
1953599200,11cyclictest0-21swapper/308:15:143
1953099205,3cyclictest26499-21runrttasks08:35:252
1953099201,7cyclictest0-21swapper/212:15:172
1953099201,7cyclictest0-21swapper/208:45:192
1953099201,6cyclictest0-21swapper/207:10:112
1953099201,5cyclictest0-21swapper/211:48:382
1953099201,4cyclictest0-21swapper/211:04:282
1953099200,6cyclictest20791-1kworker/u9:211:59:122
1952399201,11cyclictest10949-21smart_sda11:40:251
1951499209,3cyclictest12233-21/usr/sbin/munin08:05:150
1951499206,2cyclictest0-21swapper/011:14:020
1951499201,7cyclictest27018-21/usr/sbin/munin07:25:150
1951499201,11cyclictest6255-21/usr/sbin/munin09:05:230
1951499201,11cyclictest31408-21/usr/sbin/munin07:35:270
215392191,2sleep00-21swapper/007:13:100
1953599196,3cyclictest10024-21/usr/sbin/munin08:00:173
1953599195,4cyclictest792-21avahi-daemon07:15:293
1953599195,4cyclictest18073-21chrt11:57:583
1953599194,4cyclictest21743-21basename08:25:183
1953599193,9cyclictest10989-21df09:15:143
1953599191,9cyclictest0-21swapper/310:27:083
1953599191,5cyclictest29300-21ntp_states09:55:213
1953599191,15cyclictest5634-21/usr/sbin/munin07:50:213
1953599191,15cyclictest5634-21/usr/sbin/munin07:50:213
1953599191,15cyclictest4674-21/usr/sbin/munin10:15:273
1953099195,4cyclictest79932cut10:20:222
1953099194,3cyclictest3697-21grep11:25:192
1953099194,3cyclictest12050irq/123-ahci[0012:35:242
1953099193,3cyclictest12050irq/123-ahci[0011:10:012
1953099193,3cyclictest12050irq/123-ahci[0011:10:012
1953099192,7cyclictest0-21swapper/211:34:042
1953099192,7cyclictest0-21swapper/211:10:192
1953099192,15cyclictest0-21swapper/210:46:032
1953099191,7cyclictest0-21swapper/209:32:472
1953099191,6cyclictest6021-21dump-pmu-power09:05:012
1953099191,6cyclictest15400-21sh09:25:172
1953099191,5cyclictest3208-21munin-run07:45:012
1953099191,5cyclictest2813-21sh10:10:152
19530991913,4cyclictest15634-21head11:50:282
19530991912,4cyclictest7105-21cat09:05:212
1953099191,16cyclictest11492-21grep08:00:262
1953099191,15cyclictest24376-21fschecks_time09:45:162
1951499194,3cyclictest7543-21hddtemp_smartct10:20:150
1951499191,6cyclictest23001-21munin-run08:30:010
19514991916,2cyclictest0-21swapper/007:45:200
1951499191,10cyclictest4048-21/usr/sbin/munin09:00:170
5802181,13sleep2620-21ntpq08:50:232
1953599184,4cyclictest31408-21/usr/sbin/munin07:35:153
1953599183,3cyclictest14435-21/usr/sbin/munin08:10:183
1953599182,5cyclictest10035-21diskstats11:40:153
1953599181,8cyclictest0-21swapper/309:35:383
1953599181,15cyclictest20757-21awk07:10:193
1953599181,11cyclictest0-21swapper/310:50:073
1953099185,3cyclictest836-21runrttasks07:34:172
1953099184,3cyclictest86550irq/126-enp1s0-07:20:192
1953099184,11cyclictest21650-21/usr/sbin/munin09:40:222
1953099183,3cyclictest86550irq/126-enp1s0-11:20:202
1953099182,7cyclictest0-21swapper/208:32:082
1953099182,3cyclictest19442-21/usr/sbin/munin09:35:142
1953099181,7cyclictest8980-21awk07:55:232
1953099181,7cyclictest0-21swapper/210:25:372
1953099181,4cyclictest22218-21missed_timers07:15:032
1953099181,4cyclictest17231-21grep08:15:162
19530991811,3cyclictest28268-21gltestperf08:40:192
1951499181,9cyclictest0-21swapper/008:34:420
1951499181,6cyclictest842-21lldpd07:42:410
1951499181,5cyclictest9927-21cat08:00:010
1951499181,15cyclictest0-21swapper/008:20:170
19514991811,4cyclictest2837-21needreboot08:55:200
19514991811,4cyclictest2837-21needreboot08:55:200
1951499180,17cyclictest0-21swapper/008:44:360
1953599171,5cyclictest25431-21/usr/sbin/munin08:35:173
19535991711,3cyclictest23585-21cat08:30:163
1953099171,9cyclictest0-21swapper/208:05:172
1953099171,5cyclictest4993-21grep07:45:292
1953099171,5cyclictest0-21swapper/207:50:172
1953099171,5cyclictest0-21swapper/207:50:172
19530991712,3cyclictest0-21swapper/208:20:132
19530991711,3cyclictest27158-21tr07:25:162
1951499171,3cyclictest86550irq/126-enp1s0-07:20:210
1953099161,14cyclictest0-21swapper/208:26:532
19514991612,3cyclictest0-21swapper/008:15:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional