You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 06:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Nov 27, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
67532240162,25sleep00-21swapper/019:08:400
67892237163,21sleep30-21swapper/319:09:063
65712227194,21sleep10-21swapper/119:06:191
68472216162,41sleep20-21swapper/219:09:502
841021710,4sleep0694399cyclictest19:10:240
529621710,4sleep2695799cyclictest20:15:242
1485921530,4sleep3696499cyclictest23:11:243
1021821510,3sleep2695799cyclictest00:15:182
78422540,2sleep10-21swapper/122:55:241
6957994845,2cyclictest0-21swapper/220:35:272
6957994643,2cyclictest0-21swapper/223:04:272
694399453,28cyclictest16633-21/usr/sbin/munin19:30:190
694399412,28cyclictest16912-21latency_hist22:05:020
694399400,29cyclictest0-21swapper/021:55:140
694399391,25cyclictest0-21swapper/000:15:150
694399391,11cyclictest20814-21awk19:40:020
694399382,26cyclictest16366-21df00:30:120
694399382,26cyclictest16366-21df00:30:120
694399381,25cyclictest11898-21/usr/sbin/munin20:40:120
694399371,27cyclictest23226-21/usr/sbin/munin19:45:120
694399371,27cyclictest1348-21sed22:40:220
694399371,23cyclictest6565-21/usr/sbin/munin22:55:200
694399363,16cyclictest19875-21cpuspeed_turbos23:25:140
694399363,16cyclictest19875-21cpuspeed_turbos23:25:140
694399353,21cyclictest86850irq/129-enp1s0-20:20:110
694399345,25cyclictest0-21swapper/022:24:190
694399342,3cyclictest87950irq/132-enp2s0-20:33:330
10752340,5sleep230-21rcuc/223:55:152
694399338,13cyclictest25429-21/usr/sbin/munin19:50:190
694399335,24cyclictest28044-21grep22:30:140
694399331,3cyclictest86550irq/126-enp1s0-23:35:200
694399331,3cyclictest86550irq/126-enp1s0-19:15:140
694399331,30cyclictest3851-21kworker/0:022:50:240
694399331,28cyclictest8670-21sort21:45:150
6943993312,18cyclictest15148-21dump-pmu-power23:15:000
694399331,19cyclictest21297-21sort22:15:020
694399331,19cyclictest0-21swapper/022:45:120
694399321,3cyclictest21031-21/usr/sbin/munin19:40:130
694399321,28cyclictest18492-21/usr/sbin/munin20:55:200
694399321,20cyclictest2624-21open_files21:30:210
694399321,20cyclictest1669-21ntp_states23:55:200
694399312,26cyclictest4845-21grep21:35:220
6943993119,2cyclictest24792-21kworker/0:221:16:150
694399311,5cyclictest30773-21/usr/sbin/munin23:50:140
6943993113,15cyclictest29591-21dump-pmu-power20:00:010
694399311,26cyclictest7010-21ntp_states21:40:210
694399311,26cyclictest3395-21unixbench_multi20:10:270
694399311,26cyclictest30123-21unixbench_singl23:45:240
694399311,24cyclictest20321-21kworker/0:021:08:530
694399311,15cyclictest27723-21perf22:30:000
694399310,27cyclictest14257-21taskset20:45:120
6964993011,4cyclictest86550irq/126-enp1s0-00:20:163
694399301,4cyclictest10338-21/usr/sbin/munin21:50:230
694399301,3cyclictest12054-21/usr/sbin/munin23:05:200
694399301,18cyclictest2814-21/usr/sbin/munin00:00:140
6943993010,17cyclictest16330-21cpu20:50:120
6943993010,16cyclictest10534-21sshd23:02:190
694399300,6cyclictest0-21swapper/019:25:200
694399300,2cyclictest86850irq/129-enp1s0-00:10:110
694399300,1cyclictest0-21swapper/021:00:130
694399300,1cyclictest0-21swapper/020:00:240
696499291,25cyclictest0-21swapper/320:40:173
696499291,19cyclictest0-21swapper/323:35:273
694399298,18cyclictest18464-21unixbench_multi22:05:260
694399298,18cyclictest16064-21hwlatdetect23:15:150
694399292,24cyclictest31090-21users21:20:260
694399292,23cyclictest27961-21users23:40:240
694399292,23cyclictest18642-21ntp_states23:20:200
694399291,4cyclictest18461-21expr00:35:130
6943992914,12cyclictest1164-21grep20:05:260
6943992914,12cyclictest1164-21grep20:05:260
694399291,3cyclictest23234-21/usr/sbin/munin23:30:230
694399291,25cyclictest26013-21/usr/sbin/munin21:10:190
694399291,24cyclictest4747-21hddtemp_smartct20:15:180
694399291,24cyclictest13605-21dump-pmu-power00:25:000
694399290,12cyclictest20831-21kworker/0:022:15:130
694399290,12cyclictest20831-21kworker/0:022:15:130
221562290,3sleep3401ktimersoftd/321:00:263
696499281,20cyclictest0-21swapper/322:42:573
696499280,11cyclictest0-21swapper/300:11:213
694399289,16cyclictest11849-21date20:40:020
694399287,18cyclictest13860-21cpu00:25:130
694399281,6cyclictest30133-21/usr/sbin/munin22:35:140
694399281,5cyclictest24870-21diskmemload21:25:130
694399281,25cyclictest9022-21kworker/0:219:21:020
694399281,23cyclictest5673-21sed00:05:150
696499272,5cyclictest780-21rngd00:26:053
696499271,9cyclictest0-21swapper/319:42:453
696499271,8cyclictest0-21swapper/322:58:533
696499271,15cyclictest0-21swapper/322:36:093
694399272,22cyclictest8874-21basename20:30:010
296142270,4sleep20-21swapper/221:20:122
696499265,3cyclictest0-21swapper/321:05:593
695799255,7cyclictest24870-21diskmemload23:11:232
695799251,5cyclictest0-21swapper/222:48:392
695299255,9cyclictest780-21rngd23:45:551
695299250,9cyclictest0-21swapper/123:36:271
696499249,3cyclictest0-21swapper/319:29:193
696499247,2cyclictest0-21swapper/323:05:173
696499245,3cyclictest0-21swapper/320:12:073
696499244,3cyclictest0-21swapper/322:20:233
696499243,4cyclictest86550irq/126-enp1s0-22:50:193
6952992417,2cyclictest0-21swapper/119:29:581
6952992413,3cyclictest12709-21ntp_states00:20:231
695299241,21cyclictest0-21swapper/120:21:491
695299241,15cyclictest0-21swapper/123:12:531
695299240,5cyclictest0-21swapper/121:05:191
696499239,4cyclictest401ktimersoftd/321:45:023
696499239,2cyclictest0-21swapper/322:30:373
696499237,2cyclictest86550irq/126-enp1s0-22:10:133
696499234,3cyclictest0-21swapper/319:59:513
696499233,17cyclictest1290-21munin-plugin-st21:30:013
696499232,3cyclictest0-21swapper/323:41:293
696499232,10cyclictest17915-21df23:20:133
696499231,9cyclictest0-21swapper/320:46:153
696499231,9cyclictest0-21swapper/300:01:433
696499231,3cyclictest0-21swapper/321:51:473
696499231,11cyclictest0-21swapper/320:05:273
696499231,11cyclictest0-21swapper/320:05:273
695299239,13cyclictest86750irq/128-enp1s0-21:33:521
695299235,2cyclictest0-21swapper/119:40:121
695299233,6cyclictest22552-1kworker/u9:123:55:231
695299231,19cyclictest7828-21hddtemp_smartct19:10:161
695299231,13cyclictest11890-21memory23:05:171
695299230,16cyclictest0-21swapper/122:32:271
695299230,15cyclictest0-21swapper/121:59:221
696499229,4cyclictest86550irq/126-enp1s0-21:10:263
696499222,2cyclictest0-21swapper/322:00:183
696499221,6cyclictest0-21swapper/321:22:353
696499221,13cyclictest0-21swapper/320:21:513
696499221,11cyclictest0-21swapper/320:18:573
695799224,9cyclictest0-21swapper/223:18:202
695299225,7cyclictest0-21swapper/119:30:241
695299221,6cyclictest0-21swapper/123:19:151
695299221,18cyclictest85792chrt20:26:311
695299221,18cyclictest10686-21diskstats21:50:131
695299221,13cyclictest24893-21timerandwakeup22:20:241
695299221,12cyclictest0-21swapper/121:30:001
202442221,3sleep20-21swapper/220:57:212
696499219,3cyclictest3507-21latency_hist21:35:023
696499214,9cyclictest24870-21diskmemload23:57:393
696499214,8cyclictest0-21swapper/323:04:043
696499212,10cyclictest23226-21/usr/sbin/munin19:45:203
696499211,3cyclictest0-21swapper/319:54:513
696499210,8cyclictest0-21swapper/300:31:553
696499210,8cyclictest0-21swapper/300:31:553
6957992115,4cyclictest20366-21grep23:25:182
6957992115,4cyclictest20366-21grep23:25:182
695799211,2cyclictest86750irq/128-enp1s0-22:06:002
695299218,7cyclictest0-21swapper/121:46:451
695299212,11cyclictest30572-21ntp_states21:20:201
695299212,10cyclictest30192-21fschecks_count20:00:161
6952992116,2cyclictest0-21swapper/119:55:151
6952992114,3cyclictest0-21swapper/122:05:481
695299211,3cyclictest0-21swapper/120:10:131
6952992113,3cyclictest0-21swapper/123:21:141
695299211,2cyclictest0-21swapper/100:35:231
6952992112,3cyclictest0-21swapper/119:16:091
695299211,19cyclictest0-21swapper/120:50:211
695299211,18cyclictest0-21swapper/122:00:241
695299211,15cyclictest0-21swapper/119:22:281
695299211,11cyclictest20474-21/usr/sbin/munin23:25:191
695299211,11cyclictest20474-21/usr/sbin/munin23:25:191
695299211,10cyclictest0-21swapper/122:36:351
695299210,15cyclictest0-21swapper/122:42:491
695299210,14cyclictest0-21swapper/123:04:151
696499206,2cyclictest9-21rcu_preempt21:15:233
696499206,2cyclictest0-21swapper/320:37:553
696499205,9cyclictest0-21swapper/323:33:313
696499205,8cyclictest41-21ksoftirqd/300:35:163
696499205,4cyclictest24870-21diskmemload21:36:373
696499203,3cyclictest0-21swapper/319:37:313
696499202,16cyclictest39-21rcuc/323:16:243
696499201,9cyclictest0-21swapper/319:20:163
696499201,6cyclictest86550irq/126-enp1s0-21:55:133
696499201,3cyclictest41-21ksoftirqd/300:05:133
696499201,16cyclictest30931-21tr23:50:153
696499201,12cyclictest0-21swapper/300:15:293
696499201,11cyclictest16633-21/usr/sbin/munin19:30:143
696499201,10cyclictest14634-21snmpd23:26:273
696499201,10cyclictest14634-21snmpd23:26:273
696499201,10cyclictest0-21swapper/322:05:243
696499200,10cyclictest0-21swapper/323:46:113
695799203,5cyclictest0-21swapper/223:06:562
6957992015,3cyclictest16679-21cut00:30:162
6957992015,3cyclictest16679-21cut00:30:162
695799201,11cyclictest0-21swapper/222:19:292
695799201,11cyclictest0-21swapper/222:19:292
695299204,8cyclictest21-21rcuc/122:51:221
695299203,2cyclictest86750irq/128-enp1s0-00:25:221
695299201,6cyclictest0-21swapper/121:42:291
6952992016,3cyclictest0-21swapper/120:50:011
695299201,18cyclictest0-21swapper/122:45:151
695299201,18cyclictest0-21swapper/121:04:351
695299201,17cyclictest0-21swapper/120:31:171
695299201,17cyclictest0-21swapper/120:06:491
695299201,17cyclictest0-21swapper/120:06:491
695299201,10cyclictest6037-21latency_hist20:20:011
695299201,10cyclictest29471-21cat21:20:011
695299201,10cyclictest27074-21latency_hist21:15:011
695299201,10cyclictest22786-21irqstats23:30:181
695299200,14cyclictest0-21swapper/123:53:261
96122190,15sleep30-21swapper/320:30:123
90962191,3sleep10-21swapper/100:13:411
696499198,4cyclictest0-21swapper/322:46:143
696499194,2cyclictest0-21swapper/320:51:033
696499193,3cyclictest0-21swapper/321:46:453
696499192,9cyclictest86850irq/129-enp1s0-20:01:233
696499191,11cyclictest0-21swapper/322:28:063
696499191,10cyclictest10041-21/usr/sbin/munin19:15:163
695799194,5cyclictest1653-40dnf20:10:002
695799194,5cyclictest1653-40dnf20:10:002
695799192,9cyclictest0-21swapper/200:28:292
695799191,7cyclictest20807-21perf19:40:012
695799191,5cyclictest0-21swapper/222:35:322
6957991914,3cyclictest23711-21irqstats21:05:172
6957991912,5cyclictest11478-21timerwakeupswit19:15:252
695799191,12cyclictest0-21swapper/222:26:282
695799191,11cyclictest0-21swapper/200:03:522
695799191,10cyclictest0-21swapper/222:12:312
695299191,17cyclictest0-21swapper/123:44:191
695299191,17cyclictest0-21swapper/119:52:451
695299191,17cyclictest0-21swapper/100:33:271
695299191,17cyclictest0-21swapper/100:33:271
695299191,16cyclictest0-21swapper/120:55:031
695299191,16cyclictest0-21swapper/120:35:271
695299191,16cyclictest0-21swapper/100:15:551
695299191,10cyclictest0-21swapper/122:27:191
695299190,6cyclictest0-21swapper/121:35:321
261612190,4sleep230-21rcuc/223:40:012
696499187,3cyclictest22327-21strings22:15:193
696499187,3cyclictest22327-21strings22:15:193
696499181,9cyclictest14634-21snmpd19:10:393
696499181,1cyclictest0-21swapper/320:26:273
6964991811,4cyclictest18614-21tr20:55:113
695799189,2cyclictest86750irq/128-enp1s0-23:42:412
695799186,3cyclictest12050irq/123-ahci[0021:55:022
695799182,6cyclictest14652-21fschecks_time20:45:142
695799182,2cyclictest0-21swapper/222:50:182
695799181,7cyclictest0-21swapper/221:42:402
695799181,7cyclictest0-21swapper/221:10:282
695799181,7cyclictest0-21swapper/200:10:452
695799181,6cyclictest0-21swapper/221:38:412
695799181,5cyclictest9612-21runrttasks20:30:102
6957991813,3cyclictest10108-21dump-pmu-power21:50:022
6957991812,4cyclictest2347-21fschecks_time20:10:162
6957991812,3cyclictest18028-21cat19:30:232
6957991811,4cyclictest12875-21needreboot20:40:152
695299183,9cyclictest0-21swapper/100:05:441
6952991815,2cyclictest0-21swapper/119:46:171
695299181,16cyclictest0-21swapper/122:15:421
695299181,16cyclictest0-21swapper/122:15:421
695299181,16cyclictest0-21swapper/122:10:441
695299181,16cyclictest0-21swapper/119:39:121
695299181,16cyclictest0-21swapper/100:00:061
287572180,4sleep2695799cyclictest22:30:212
144182181,2sleep20-21swapper/221:59:072
136032181,3sleep10-21swapper/120:41:411
695799176,3cyclictest86550irq/126-enp1s0-23:30:142
695799172,8cyclictest0-21swapper/221:33:212
695799172,4cyclictest22552-1kworker/u9:119:56:512
695799171,8cyclictest0-21swapper/223:20:172
695799171,7cyclictest8486-21sshd20:25:202
695799171,7cyclictest7153-21/usr/sbin/munin19:10:132
695799171,7cyclictest0-21swapper/200:20:162
695799171,6cyclictest0-21swapper/221:00:172
695799171,5cyclictest0-21swapper/200:05:282
695799171,4cyclictest28917-21diskstats23:45:142
6957991712,3cyclictest7092-21kernelversion20:20:172
6957991712,3cyclictest32032-21expr21:25:122
6957991712,3cyclictest30603-21if_err_enp1s020:00:182
6957991712,3cyclictest28113-21irqstats21:15:172
6957991712,3cyclictest26499-21ntp_states19:50:212
6957991712,3cyclictest15151-21fschecks_count22:00:172
695799171,15cyclictest0-21swapper/219:26:052
6957991711,3cyclictest31065-21expr23:50:162
695799163,3cyclictest0-21swapper/200:35:172
695799161,9cyclictest0-21swapper/222:58:242
695799161,6cyclictest22293-21smart_sda19:40:232
695799161,6cyclictest0-21swapper/222:40:232
695799161,14cyclictest0-21swapper/219:46:302
6957991611,3cyclictest23718-21cat22:20:132
6957991611,3cyclictest16297-21/usr/sbin/munin20:50:212
6957991611,3cyclictest12310-21wc19:20:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional