You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 18:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Nov 19, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
158612239161,25sleep00-21swapper/007:07:150
159862234162,59sleep20-21swapper/207:08:502
159932204167,24sleep30-21swapper/307:08:553
157442202164,25sleep10-21swapper/107:05:431
1883321490,4sleep01616699cyclictest08:25:230
116942640,2sleep20-21swapper/211:50:222
187602610,3sleep018761-21fschecks_time07:15:010
33902550,4sleep11617299cyclictest11:30:251
16166993511,11cyclictest8663-21munin-run08:05:010
271922310,3sleep339-21rcuc/308:45:173
1618399305,10cyclictest4894-21diskmemload09:41:413
1616699302,13cyclictest13703-21basename09:30:010
1617299293,18cyclictest86750irq/128-enp1s0-09:35:221
1617299281,6cyclictest11148-21strings10:35:241
1616699282,14cyclictest13067-21munin-run08:15:010
233862270,3sleep081ktimersoftd/011:05:160
1618399272,15cyclictest5103-21memory11:35:203
1618399272,15cyclictest5103-21memory11:35:203
1617299271,8cyclictest16616-21kworker/u8:011:17:511
16172992715,4cyclictest822-21cut09:00:161
1616699275,11cyclictest0-21swapper/010:35:010
1616699275,11cyclictest0-21swapper/010:35:010
1617999261,8cyclictest11309-21pmu-power10:35:272
1617299262,17cyclictest0-21swapper/112:00:471
16172992612,6cyclictest29768-21kworker/u8:008:25:271
1618399255,17cyclictest25621-21latency_hist07:30:013
1618399251,11cyclictest7322-21/usr/sbin/munin09:15:213
1617299252,4cyclictest13693-21munin-run09:30:001
1617299252,17cyclictest0-21swapper/107:49:081
1617299250,7cyclictest0-21swapper/109:31:381
1617299250,19cyclictest0-21swapper/108:52:361
1617299250,18cyclictest0-21swapper/111:20:111
32082240,4sleep339-21rcuc/311:30:243
1618399241,8cyclictest28186-21ntp_states10:00:233
1618399241,14cyclictest15462-21memory10:45:203
1618399240,3cyclictest0-21swapper/311:20:203
1617299242,16cyclictest0-21swapper/110:53:361
1617299241,17cyclictest0-21swapper/107:21:321
108872240,2sleep10-21swapper/109:20:261
108872240,2sleep10-21swapper/109:20:261
166332230,1sleep10-21swapper/107:10:131
1618399239,6cyclictest0-21swapper/311:19:093
1617999231,5cyclictest0-21swapper/209:58:112
1617999231,18cyclictest20005-21grep08:30:152
1617299234,10cyclictest86750irq/128-enp1s0-09:08:521
1617299232,15cyclictest86850irq/129-enp1s0-08:43:011
16172992315,2cyclictest0-21swapper/111:46:081
16172992314,3cyclictest0-21swapper/108:02:201
1617299231,16cyclictest0-21swapper/112:27:071
1617299231,16cyclictest0-21swapper/107:58:221
1617299231,16cyclictest0-21swapper/107:18:211
1617299231,15cyclictest0-21swapper/110:25:121
1617299231,12cyclictest10856-21dump-pmu-power08:10:011
1617299231,12cyclictest0-21swapper/111:55:281
1617299231,11cyclictest0-21swapper/110:57:431
1617299230,16cyclictest0-21swapper/107:30:171
1616699233,2cyclictest0-21swapper/012:22:000
1616699230,5cyclictest12519-21kworker/u8:112:00:180
1618399222,12cyclictest1704-21df_inode10:15:153
1618399221,9cyclictest0-21swapper/310:13:263
16183992216,3cyclictest14458-21latency_hist10:45:023
1618399221,13cyclictest0-21swapper/310:06:323
1618399220,3cyclictest0-21swapper/312:28:293
1617999226,4cyclictest3263-21df09:05:152
1617999221,6cyclictest0-21swapper/210:14:542
1617299227,4cyclictest0-21swapper/110:01:531
1617299227,3cyclictest0-21swapper/110:22:141
1617299222,15cyclictest0-21swapper/112:18:401
1617299222,15cyclictest0-21swapper/111:52:401
1617299222,15cyclictest0-21swapper/110:35:001
1617299222,15cyclictest0-21swapper/110:35:001
1617299221,8cyclictest4546-21/usr/sbin/munin11:35:151
1617299221,8cyclictest4546-21/usr/sbin/munin11:35:151
1617299221,3cyclictest23-21ksoftirqd/107:50:231
1617299221,3cyclictest0-21swapper/110:15:471
1617299221,15cyclictest0-21swapper/112:30:471
1617299221,15cyclictest0-21swapper/109:45:181
1617299221,15cyclictest0-21swapper/108:39:581
1617299221,15cyclictest0-21swapper/108:20:581
1617299221,11cyclictest0-21swapper/109:17:331
1617299220,6cyclictest0-21swapper/110:07:211
1617299220,10cyclictest784-21NetworkManager08:18:571
1616699222,9cyclictest0-21swapper/011:25:180
1616699222,11cyclictest14634-21snmpd08:53:050
1616699221,4cyclictest0-21swapper/011:11:330
1618399214,4cyclictest17402-21switchtime09:35:263
1618399212,7cyclictest29067-21hddtemp_smartct12:30:183
1618399211,5cyclictest7349-21users10:25:303
1618399211,4cyclictest9229-21gltestperf11:45:183
1618399211,13cyclictest29768-21kworker/u8:009:56:183
1618399211,13cyclictest0-21swapper/309:52:213
1618399211,12cyclictest6862-21chrt09:12:113
1618399211,12cyclictest0-21swapper/312:11:143
1618399211,11cyclictest21793-21/usr/sbin/munin12:15:143
1617999211,3cyclictest0-21swapper/212:31:102
1617999210,8cyclictest0-21swapper/211:05:302
1617299216,5cyclictest88050irq/133-enp2s0-11:40:241
1617299212,15cyclictest0-21swapper/108:12:011
1617299212,13cyclictest86750irq/128-enp1s0-11:25:371
1617299211,2cyclictest0-21swapper/112:23:161
1617299211,15cyclictest0-21swapper/111:06:591
1617299211,15cyclictest0-21swapper/108:56:441
1617299211,14cyclictest0-21swapper/107:40:081
1617299211,13cyclictest29768-21kworker/u8:009:10:271
1617299211,12cyclictest0-21swapper/109:50:161
1617299211,11cyclictest13924-21unixbench_singl10:40:271
1617299211,10cyclictest0-21swapper/111:02:181
16172992110,3cyclictest31861-21df10:10:151
16172992110,3cyclictest26179-21smart_sda09:55:261
1617299210,4cyclictest0-21swapper/111:10:131
1616699218,3cyclictest21154-21head12:10:270
1616699217,3cyclictest2979-21strings11:30:200
1616699211,5cyclictest0-21swapper/010:45:020
1616699211,2cyclictest0-21swapper/010:15:240
1616699211,17cyclictest23428-21munin-run07:25:000
1616699211,13cyclictest0-21swapper/010:36:140
1616699211,10cyclictest28437-21grep07:35:170
1618399209,3cyclictest11391-21head11:50:193
1618399206,7cyclictest29723-21kworker/3:107:41:293
1618399203,7cyclictest28678-21/usr/sbin/munin08:50:123
1618399203,14cyclictest10961-21interrupts10:35:223
1618399203,10cyclictest0-21swapper/311:27:573
1618399201,7cyclictest3731-21ls10:20:143
1618399201,4cyclictest0-21swapper/312:38:043
1618399201,4cyclictest0-21swapper/312:24:253
1618399201,4cyclictest0-21swapper/311:02:093
1618399201,12cyclictest0-21swapper/311:43:113
1618399201,12cyclictest0-21swapper/309:29:013
16183992010,7cyclictest41-21ksoftirqd/308:23:463
1617999204,3cyclictest27273-21grep10:00:142
1617999203,6cyclictest17112-21strings09:35:212
1617999201,8cyclictest876-21systemd-logind12:40:012
1617999201,8cyclictest0-21swapper/210:45:462
1617999201,6cyclictest0-21swapper/209:30:162
1617999201,6cyclictest0-21swapper/208:18:582
1617999201,5cyclictest25201-21memory08:40:222
16179992015,3cyclictest4894-21diskmemload09:27:222
16179992013,3cyclictest1367-21gltestperf09:00:202
1617999201,16cyclictest0-21swapper/208:55:152
1617999201,10cyclictest0-21swapper/211:12:112
1617299206,4cyclictest0-21swapper/110:46:011
1617299203,11cyclictest0-21swapper/112:05:091
1617299202,16cyclictest0-21swapper/108:48:231
1617299201,15cyclictest0-21swapper/109:41:111
1617299201,15cyclictest0-21swapper/107:36:211
1617299201,15cyclictest0-21swapper/107:28:061
1617299201,10cyclictest0-21swapper/112:35:261
1617299201,10cyclictest0-21swapper/112:10:571
1616699209,6cyclictest0-21swapper/008:44:560
1616699209,3cyclictest27759-21/usr/sbin/munin11:15:270
1616699207,3cyclictest784-21NetworkManager09:58:070
1616699204,11cyclictest86550irq/126-enp1s0-10:40:030
1616699202,6cyclictest0-21swapper/012:05:280
1616699201,8cyclictest4894-21diskmemload10:23:170
1616699201,6cyclictest0-21swapper/010:00:060
1616699201,5cyclictest0-21swapper/011:50:120
1616699201,4cyclictest0-21swapper/011:55:090
1616699201,12cyclictest0-21swapper/009:15:160
1616699201,11cyclictest5763-21/usr/sbin/munin10:25:290
1616699201,11cyclictest32445-21/usr/sbin/munin07:45:130
1616699201,11cyclictest18952-21/usr/sbin/munin10:55:210
1616699201,11cyclictest16117-21/usr/sbin/munin09:35:250
1616699201,10cyclictest31759-21wc12:35:270
1616699201,10cyclictest10512-21needreboot09:20:220
1616699201,10cyclictest10512-21needreboot09:20:220
176292191,15sleep017762-21timerandwakeup07:10:240
1618399198,3cyclictest17394-21/usr/sbin/munin12:05:213
1618399197,3cyclictest0-21swapper/310:51:483
1618399195,6cyclictest0-21swapper/312:03:403
1618399193,3cyclictest86550irq/126-enp1s0-09:20:503
1618399193,3cyclictest86550irq/126-enp1s0-09:20:503
1618399191,9cyclictest8350-21tune2fs10:30:163
1618399191,9cyclictest8350-21tune2fs10:30:163
1618399191,9cyclictest16703-21df07:10:143
1618399191,9cyclictest13792-21irqstats11:55:203
1618399191,3cyclictest0-21swapper/310:55:563
16183991912,4cyclictest4953-21grep07:55:163
1618399191,15cyclictest14060-21tr09:30:143
1618399191,12cyclictest0-21swapper/311:12:103
1617999198,3cyclictest5527-21latency_hist10:25:012
1617999193,5cyclictest0-21swapper/211:21:362
1617999191,7cyclictest8377-21grep10:30:162
1617999191,7cyclictest8377-21grep10:30:162
1617999191,7cyclictest361-21grep07:45:152
1617999191,7cyclictest0-21swapper/207:50:202
1617999191,6cyclictest30136-21irqstats10:05:192
1617999191,6cyclictest0-21swapper/211:39:192
1617999191,6cyclictest0-21swapper/211:39:192
1617999191,6cyclictest0-21swapper/210:19:102
1617999191,6cyclictest0-21swapper/207:31:212
16179991916,2cyclictest0-21swapper/211:49:022
1617999191,4cyclictest22162-21taskset09:46:032
1617999191,3cyclictest28928-21ps11:15:252
1617999191,3cyclictest0-21swapper/211:43:162
1617999191,3cyclictest0-21swapper/208:50:192
16179991913,3cyclictest23006-21unixbench_singl07:20:272
16172991910,3cyclictest0-21swapper/108:33:511
1616699196,4cyclictest12735-1kworker/u9:212:30:330
1616699192,8cyclictest780-21rngd11:48:400
1616699191,9cyclictest29959-21/usr/sbin/munin11:20:180
1616699191,9cyclictest16009-21sed09:35:020
1616699191,9cyclictest0-21swapper/007:34:360
1616699191,8cyclictest18309-21unixbench_singl10:50:280
1616699191,8cyclictest0-21swapper/011:35:200
1616699191,8cyclictest0-21swapper/011:35:200
1616699191,6cyclictest29768-21kworker/u8:008:23:510
1616699191,12cyclictest0-21swapper/011:43:340
1616699191,11cyclictest0-21swapper/012:16:450
1616699191,11cyclictest0-21swapper/009:45:210
1616699191,10cyclictest13290-21/usr/sbin/munin08:15:220
67832181,2sleep20-21swapper/209:11:102
264382180,4sleep31618399cyclictest07:30:183
202312181,3sleep00-21swapper/009:44:280
171322181,3sleep20-21swapper/212:04:442
1618399186,4cyclictest24537-21cstates08:40:163
1618399185,3cyclictest4214-21switchtime09:05:263
1618399184,8cyclictest0-21swapper/307:20:173
1618399183,8cyclictest0-21swapper/311:06:343
1618399182,6cyclictest1052-21grep07:45:203
1618399181,9cyclictest21749-21sshd08:33:463
1618399181,9cyclictest13368-21cat08:15:153
1618399181,6cyclictest18768-21awk08:25:223
1618399181,5cyclictest20912-21fschecks_count09:45:173
16183991812,3cyclictest22432-21df08:35:153
1618399180,9cyclictest0-21swapper/308:10:363
1617999185,4cyclictest86850irq/129-enp1s0-09:40:292
1617999183,5cyclictest22430-21df_inode08:35:152
1617999183,3cyclictest7043-21switchtime10:25:282
1617999182,6cyclictest86750irq/128-enp1s0-11:55:182
1617999182,13cyclictest32754-21gltestperf11:25:172
1617999181,8cyclictest0-21swapper/211:32:122
1617999181,7cyclictest12353-21/usr/sbin/munin10:40:162
1617999181,6cyclictest21503-21diskstats11:00:172
1617999181,6cyclictest0-21swapper/212:16:012
1617999181,6cyclictest0-21swapper/210:52:222
1617999181,5cyclictest0-21swapper/207:13:392
1617999181,3cyclictest7519-21cstates09:15:132
1617999181,3cyclictest0-21swapper/207:41:282
16179991813,3cyclictest23765-21cut07:25:142
16179991813,3cyclictest20655-21ntp_offset12:10:222
1617999181,15cyclictest0-21swapper/209:21:352
1617999181,15cyclictest0-21swapper/209:21:352
1616699187,2cyclictest31805-21unixbench_singl07:40:270
1616699182,2cyclictest86550irq/126-enp1s0-08:55:220
1616699182,10cyclictest86550irq/126-enp1s0-09:13:400
1616699181,5cyclictest14634-21snmpd10:11:060
16166991812,2cyclictest86550irq/126-enp1s0-10:08:290
1616699181,11cyclictest0-21swapper/012:26:430
1616699181,10cyclictest28403-21sshd08:49:360
39732170,3sleep00-21swapper/007:51:360
1618399171,4cyclictest0-21swapper/307:50:183
1618399171,3cyclictest28811-21if_enp2s007:35:193
1618399171,15cyclictest0-21swapper/307:17:213
1618399171,14cyclictest7854-21ps08:00:263
16183991711,3cyclictest2245-21sed09:00:293
16183991710,4cyclictest9938-21ntpq08:05:213
1617999177,3cyclictest0-21swapper/208:12:342
1617999176,2cyclictest86550irq/126-enp1s0-12:05:162
1617999172,6cyclictest8890-21/usr/sbin/munin08:05:132
1617999171,9cyclictest0-21swapper/208:45:122
1617999171,6cyclictest6685-21/usr/sbin/munin08:00:272
1617999171,6cyclictest4894-21diskmemload10:56:562
1617999171,6cyclictest22718-21/usr/sbin/munin09:50:152
1617999171,6cyclictest0-21swapper/212:21:402
1617999171,4cyclictest0-21swapper/212:25:202
16179991712,3cyclictest4792-21expr07:55:132
16179991712,3cyclictest20449-21processes07:15:352
16179991711,3cyclictest18036-21diskstats08:25:142
1616699171,9cyclictest0-21swapper/007:55:130
1616699171,8cyclictest0-21swapper/009:50:030
1616699171,6cyclictest3343-21fschecks_count09:05:160
16166991712,3cyclictest23420-21threads08:35:260
16166991712,3cyclictest21155-21/usr/sbin/munin11:00:270
1616699171,14cyclictest0-21swapper/008:08:280
1618399161,3cyclictest31477-21gltestperf08:55:183
16179991611,3cyclictest836-21runrttasks08:24:512
16179991610,3cyclictest28234-21cstates07:35:142
1616699161,3cyclictest0-21swapper/007:26:190
16166991612,3cyclictest0-21swapper/008:30:120
16166991611,3cyclictest2434-21seq09:01:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional