You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 05:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jul 07, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
192542239164,24sleep30-21swapper/319:07:213
193672217165,38sleep10-21swapper/119:08:481
194162203166,24sleep20-21swapper/219:09:252
193552203165,25sleep00-21swapper/019:08:390
990621740,4sleep01955499cyclictest23:40:150
1956599108104,3cyclictest0-21swapper/221:40:172
82962630,2sleep00-21swapper/022:20:270
19565996158,2cyclictest0-21swapper/221:38:222
206382590,2sleep20-21swapper/222:50:152
228762540,4sleep011-21rcuc/000:10:010
19565994744,2cyclictest0-21swapper/223:10:422
19565994744,2cyclictest0-21swapper/223:10:422
289172460,3sleep339-21rcuc/300:20:283
19565993532,2cyclictest0-21swapper/223:41:322
1955499351,14cyclictest0-21swapper/019:20:130
1955499344,14cyclictest16094-21munin-run23:55:020
1955499331,8cyclictest0-21swapper/020:40:300
1955499330,14cyclictest16737-21kworker/0:021:40:260
1956199321,19cyclictest0-21swapper/123:32:351
1955499328,11cyclictest28670-21munin-run23:10:010
1955499312,13cyclictest30663-21kworker/0:019:30:200
1955499311,15cyclictest27863-21kworker/0:121:56:480
1955499310,14cyclictest19531-21kworker/0:122:49:240
1955499301,13cyclictest9123-21vmstat23:35:260
1955499300,13cyclictest4166-21kworker/0:022:15:240
280702290,2sleep30-21swapper/323:05:303
1956199290,19cyclictest0-21swapper/121:32:541
1955499295,11cyclictest20519-21idleruntime-cro00:05:010
1955499292,14cyclictest5437-21munin-run19:50:010
1955499291,13cyclictest20832-21munin-run20:25:010
1955499291,11cyclictest22952-21kworker/0:120:37:000
1955499290,10cyclictest24073-21kworker/0:023:00:160
274312280,4sleep339-21rcuc/319:25:163
1956599281,9cyclictest4070-21kworker/2:121:20:102
1955499283,12cyclictest2914-21basename23:25:010
1955499281,16cyclictest23624-21kworker/0:222:13:480
1955499281,10cyclictest30888-21kworker/0:119:40:260
1956199272,14cyclictest0-21swapper/122:49:261
1956199271,7cyclictest0-21swapper/122:15:181
1955499278,16cyclictest16432-21munin-run20:15:010
1955499271,15cyclictest17665-21perf22:45:020
1955499271,10cyclictest19531-21kworker/0:122:54:560
1955499270,17cyclictest1013-21kworker/0:200:35:560
1955499270,12cyclictest22390-21kworker/0:000:11:300
154782270,4sleep011-21rcuc/022:40:020
19570992617,4cyclictest0-21swapper/321:27:293
1957099261,17cyclictest0-21swapper/300:30:133
1955499263,17cyclictest30512-21kworker/0:223:14:360
1955499263,17cyclictest30512-21kworker/0:223:14:360
1955499262,12cyclictest19132-21kworker/0:221:38:180
1955499261,13cyclictest10439-21kworker/0:221:20:260
1955499261,12cyclictest1013-21kworker/0:200:32:360
1956199253,4cyclictest1355550kworker/u8:123:57:201
1956199251,3cyclictest3771-21smart_sda22:10:231
1955499253,6cyclictest5230-1kworker/u9:100:27:210
1955499253,12cyclictest30100-21kworker/0:022:00:340
1955499252,2cyclictest15744-21kworker/0:123:59:080
1955499252,11cyclictest24073-21kworker/0:022:59:120
1955499252,10cyclictest16737-21kworker/0:021:33:540
1955499252,10cyclictest11648-21kworker/0:023:48:420
1955499251,9cyclictest32396-21kworker/0:122:08:360
19554992511,2cyclictest30663-21kworker/0:019:19:260
19554992511,2cyclictest29325-21kworker/0:220:45:240
1955499251,11cyclictest2659-21kworker/0:023:28:120
1955499251,10cyclictest8589-21kworker/0:322:26:420
1955499250,12cyclictest31716-21kworker/0:020:55:220
19565992421,2cyclictest0-21swapper/219:13:462
1956199249,2cyclictest86750irq/128-enp1s0-23:07:231
1956199242,3cyclictest14835-21kworker/u8:221:15:471
19561992414,3cyclictest0-21swapper/122:05:041
1956199241,17cyclictest0-21swapper/123:17:001
1956199241,17cyclictest0-21swapper/122:30:251
1956199241,10cyclictest22740-21ntp_states21:40:231
1956199240,8cyclictest23-21ksoftirqd/121:58:571
1955499242,14cyclictest2659-21kworker/0:023:30:180
1955499242,10cyclictest27335-21kworker/0:100:24:000
1955499241,9cyclictest30663-21kworker/0:019:25:360
1955499241,6cyclictest16793-21kworker/u8:121:29:170
1955499241,10cyclictest31716-21kworker/0:020:50:320
1955499240,10cyclictest22390-21kworker/0:000:17:240
1955499240,10cyclictest22390-21kworker/0:000:17:240
19570992316,2cyclictest0-21swapper/321:00:173
19570992316,2cyclictest0-21swapper/321:00:173
19570992312,2cyclictest5394-21/usr/sbin/munin22:15:193
1956599231,7cyclictest29540-21cstates19:30:152
1956199232,6cyclictest0-21swapper/122:22:001
19561992314,3cyclictest0-21swapper/123:42:341
19561992314,3cyclictest0-21swapper/121:05:541
19561992314,3cyclictest0-21swapper/120:26:331
19561992314,3cyclictest0-21swapper/119:21:241
1956199231,19cyclictest20764-21hddtemp_smartct22:50:151
1956199231,17cyclictest0-21swapper/100:04:491
1956199231,16cyclictest0-21swapper/100:29:481
1955499239,3cyclictest3735-21kworker/0:321:01:420
1955499239,3cyclictest3735-21kworker/0:321:01:420
1955499232,12cyclictest30512-21kworker/0:223:15:540
19554992312,3cyclictest5990-21kworker/0:121:10:200
1955499231,14cyclictest8212-21kworker/0:021:15:280
1955499231,13cyclictest23624-21kworker/0:221:46:180
1955499230,13cyclictest10889-21kworker/0:022:30:160
1957099229,4cyclictest7253-21diskstats22:20:163
1957099228,2cyclictest0-21swapper/321:10:203
1957099224,3cyclictest22245-21unixbench_multi00:05:273
1957099221,5cyclictest0-21swapper/321:24:193
1957099221,13cyclictest0-21swapper/321:55:023
1957099220,9cyclictest0-21swapper/322:30:193
1957099220,20cyclictest0-21swapper/321:18:183
1956599226,4cyclictest11893-21sshd20:03:232
1956199222,2cyclictest86750irq/128-enp1s0-21:36:181
1956199222,18cyclictest0-21swapper/119:38:541
1956199222,15cyclictest0-21swapper/120:39:501
1956199222,13cyclictest86750irq/128-enp1s0-21:54:051
1956199221,6cyclictest0-21swapper/121:12:111
1956199221,5cyclictest0-21swapper/122:58:211
19561992214,3cyclictest0-21swapper/123:28:051
19561992213,3cyclictest0-21swapper/123:37:371
19561992213,3cyclictest0-21swapper/120:34:321
19561992213,2cyclictest14634-21snmpd00:36:451
1956199221,18cyclictest27382-21diskstats19:25:141
1956199221,16cyclictest0-21swapper/123:50:561
1956199221,16cyclictest0-21swapper/121:25:381
1956199221,16cyclictest0-21swapper/100:21:141
1956199221,15cyclictest0-21swapper/123:00:051
1956199221,15cyclictest0-21swapper/122:36:201
1956199221,15cyclictest0-21swapper/122:00:221
1956199221,15cyclictest0-21swapper/120:23:131
1956199221,15cyclictest0-21swapper/120:06:161
1956199221,15cyclictest0-21swapper/119:17:341
1956199221,15cyclictest0-21swapper/100:30:131
1956199221,12cyclictest501-21memory20:50:191
1956199221,11cyclictest30487-21users23:10:261
1956199221,11cyclictest30487-21users23:10:261
19561992210,3cyclictest10164-21ntp_states22:25:221
1956199220,16cyclictest0-21swapper/120:10:521
1955499229,4cyclictest7497-21kworker/0:219:55:220
1955499229,4cyclictest7497-21kworker/0:219:55:220
1955499229,3cyclictest30663-21kworker/0:019:14:380
1955499221,12cyclictest642-21kworker/0:020:02:520
1955499220,9cyclictest0-21swapper/021:50:280
1955499220,12cyclictest20635-21kworker/0:020:25:200
1957099214,3cyclictest0-21swapper/323:37:093
1957099213,3cyclictest0-21swapper/300:37:313
1957099211,9cyclictest0-21swapper/321:45:213
1957099211,9cyclictest0-21swapper/320:25:233
19570992115,3cyclictest21236-21sshd21:37:303
1957099211,14cyclictest0-21swapper/323:49:043
1957099211,14cyclictest0-21swapper/323:26:383
1957099211,12cyclictest10850-21pmu-power23:40:253
1957099210,14cyclictest0-21swapper/321:44:063
1956599214,10cyclictest0-21swapper/222:49:472
1956599212,9cyclictest12009-21basename22:30:192
1956599211,9cyclictest0-21swapper/222:27:112
19565992117,2cyclictest2326-21ntpq19:40:252
1956599211,6cyclictest6393-21ntp_states23:30:242
1956599211,6cyclictest23435-21pmu-power22:55:242
1956599211,5cyclictest14485-21needreboot22:35:212
1956599211,5cyclictest0-21swapper/221:18:182
1956599211,3cyclictest0-21swapper/223:54:202
1956199219,3cyclictest13644-21if_enp2s021:20:231
1956199216,6cyclictest22942-21/usr/sbin/munin00:10:191
19561992117,3cyclictest0-21swapper/119:51:591
19561992114,2cyclictest0-21swapper/120:41:521
19561992114,2cyclictest0-21swapper/119:34:161
19561992113,3cyclictest0-21swapper/122:43:461
19561992113,2cyclictest0-21swapper/123:45:121
1956199211,15cyclictest0-21swapper/121:46:091
1956199211,15cyclictest0-21swapper/119:49:391
1956199211,15cyclictest0-21swapper/100:07:571
19561992111,3cyclictest0-21swapper/120:45:161
1956199210,15cyclictest0-21swapper/121:03:281
1956199210,15cyclictest0-21swapper/121:03:281
1956199210,15cyclictest0-21swapper/120:19:231
1956199210,15cyclictest0-21swapper/119:55:051
1956199210,15cyclictest0-21swapper/119:55:051
1955499219,11cyclictest0-21swapper/019:40:010
1955499218,3cyclictest22952-21kworker/0:120:33:580
1955499211,10cyclictest12007-21kworker/0:220:05:100
321532200,2sleep30-21swapper/322:01:283
1957099207,7cyclictest0-21swapper/322:35:043
1957099205,12cyclictest5015-21unixbench_singl19:45:263
1957099204,2cyclictest41-21ksoftirqd/321:30:283
1957099202,15cyclictest9741-21gltestperf22:25:183
1957099201,6cyclictest25073-21cat00:15:013
1957099201,6cyclictest0-21swapper/322:43:143
19570992015,3cyclictest0-21swapper/300:03:473
1957099201,4cyclictest31056-21unixbench_multi00:25:343
1957099201,4cyclictest0-21swapper/322:14:233
1957099201,3cyclictest20907-21irqstats22:50:183
1957099201,11cyclictest1879-21/usr/sbin/munin20:55:203
1957099201,10cyclictest25143-21/usr/sbin/munin00:15:243
1957099201,10cyclictest25143-21/usr/sbin/munin00:15:243
1957099201,10cyclictest0-21swapper/319:36:083
1957099200,9cyclictest0-21swapper/323:30:293
1956599205,7cyclictest0-21swapper/220:26:502
1956599205,3cyclictest20032-21wc20:20:242
1956599204,3cyclictest86850irq/129-enp1s0-19:24:232
1956599202,5cyclictest3161-21users00:35:312
1956599201,6cyclictest0-21swapper/223:45:182
1956599201,5cyclictest30804-21/usr/sbin/munin22:00:152
1956599201,5cyclictest26701-21/usr/sbin/munin23:05:182
19565992014,3cyclictest4114-21gltestperf19:45:162
1956599201,3cyclictest0-21swapper/223:21:012
1956599201,12cyclictest26057-21memory00:15:202
1956599201,12cyclictest26057-21memory00:15:202
1956599200,6cyclictest31022-21grep00:25:342
1956199209,3cyclictest2263-21sed23:20:271
1956199203,7cyclictest1879-21/usr/sbin/munin20:55:121
1956199202,17cyclictest0-21swapper/119:12:101
1956199201,3cyclictest0-21swapper/119:41:531
1956199201,10cyclictest12027-21latency_hist20:05:011
1956199200,15cyclictest0-21swapper/100:17:341
1956199200,15cyclictest0-21swapper/100:17:331
1955499208,3cyclictest3735-21kworker/0:321:05:340
1955499206,2cyclictest14217-21kworker/0:120:16:460
199612191,2sleep320079-21rt-features20:20:243
1957099194,3cyclictest26353-21unixbench_multi19:20:273
1957099193,3cyclictest86850irq/129-enp1s0-20:30:203
1957099192,6cyclictest0-21swapper/323:55:163
19570991912,4cyclictest1849-21/usr/sbin/munin23:20:223
1957099191,10cyclictest22644-21/usr/sbin/munin19:15:223
1957099191,10cyclictest0-21swapper/323:16:013
1957099190,6cyclictest0-21swapper/322:56:263
1957099190,3cyclictest86750irq/128-enp1s0-19:33:473
1956599195,3cyclictest86550irq/126-enp1s0-22:18:232
1956599193,3cyclictest0-21swapper/221:31:042
1956599192,5cyclictest0-21swapper/220:52:232
1956599192,3cyclictest17708-21ntpq20:15:212
1956599191,9cyclictest0-21swapper/200:05:162
1956599191,7cyclictest0-21swapper/223:28:002
1956599191,7cyclictest0-21swapper/222:10:522
1956599191,7cyclictest0-21swapper/221:11:352
1956599191,6cyclictest25757-21smart_sda23:00:282
1956599191,6cyclictest0-21swapper/222:45:012
1956599191,6cyclictest0-21swapper/221:53:002
1956599191,5cyclictest27341-21/usr/sbin/munin00:20:182
1956599191,11cyclictest0-21swapper/223:16:442
1957099188,3cyclictest0-21swapper/323:11:343
1957099188,3cyclictest0-21swapper/323:11:343
1957099184,3cyclictest27280-21sshd20:38:243
1957099184,3cyclictest1606-21chrt19:40:173
1957099181,9cyclictest27653-21/usr/sbin/munin20:40:273
1957099181,5cyclictest20816-21ntp_states19:10:213
1957099181,4cyclictest0-21swapper/323:00:213
1957099181,3cyclictest14645-21/usr/sbin/munin23:50:173
19570991812,3cyclictest7857-21/usr/sbin/munin19:55:233
19570991812,3cyclictest7857-21/usr/sbin/munin19:55:233
1957099181,15cyclictest0-21swapper/320:15:073
1957099181,14cyclictest26650-21perl21:50:173
19570991811,4cyclictest1-21systemd22:05:233
1956599184,3cyclictest86850irq/129-enp1s0-23:55:152
1956599181,8cyclictest0-21swapper/222:06:492
1956599181,8cyclictest0-21swapper/221:55:182
1956599181,8cyclictest0-21swapper/200:30:162
1956599181,8cyclictest0-21swapper/200:00:242
1956599181,4cyclictest28999-21latency_hist19:30:022
1956599181,4cyclictest25452-21/usr/sbin/munin20:35:142
19565991812,3cyclictest21857-21munin-node19:15:042
1956599181,15cyclictest0-21swapper/221:45:062
1956599181,10cyclictest19547-21cyclictest22:22:592
1956599181,10cyclictest0-21swapper/200:11:132
133262181,14sleep213338-21ntp_states20:05:302
35502170,2sleep20-21swapper/220:56:242
1957099179,3cyclictest0-21swapper/319:52:473
1957099173,9cyclictest0-21swapper/322:46:583
1957099172,12cyclictest6283-21/usr/sbin/munin21:05:153
1957099171,14cyclictest0-21swapper/320:00:463
1957099171,13cyclictest12423-21grep20:05:203
1956599172,8cyclictest86850irq/129-enp1s0-19:50:242
1956599171,5cyclictest8629-21ls23:35:212
1956599171,5cyclictest16665-21users21:25:292
19565991712,3cyclictest6563-21ls21:05:142
19565991712,3cyclictest4230-21grep21:00:162
19565991712,3cyclictest4230-21grep21:00:162
19565991712,3cyclictest14605-21tr20:10:142
19565991711,4cyclictest8223-21/usr/sbin/munin19:55:172
19565991711,4cyclictest8223-21/usr/sbin/munin19:55:172
1956599171,14cyclictest27999-21diskstats20:40:182
19565991711,3cyclictest31797-21fschecks_count19:35:162
1955499178,2cyclictest0-21swapper/019:50:140
1957099161,4cyclictest0-21swapper/320:50:103
1957099161,3cyclictest30450-21gltestperf20:45:193
19570991611,3cyclictest14595-21expr20:10:143
19565991611,3cyclictest23577-21expr20:30:142
1956599157,6cyclictest32-21ksoftirqd/220:45:042
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional