You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 13:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Nov 21, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
204222236162,24sleep20-21swapper/219:07:002
204562235162,59sleep00-21swapper/019:07:260
205352205166,25sleep10-21swapper/119:08:291
203522203167,23sleep30-21swapper/319:06:083
296721860,4sleep12075399cyclictest23:20:211
1464721700,4sleep02074899cyclictest20:05:240
56372620,3sleep011-21rcuc/021:00:140
237052510,2sleep19-21rcu_preempt20:25:271
39952500,4sleep12075399cyclictest19:40:231
2074899506,13cyclictest4132-21idleruntime-cro23:25:000
171012490,1sleep20-21swapper/221:25:172
2074899418,16cyclictest3518-21munin-run22:10:020
2074899411,14cyclictest28022-21basename19:25:000
2074899388,14cyclictest329-21munin-run00:30:000
2074899372,11cyclictest21812-21perf00:05:010
2074899349,12cyclictest27722-21awk23:05:010
20753993112,9cyclictest20357-21kworker/u8:019:55:221
2075399311,13cyclictest0-21swapper/119:21:191
2074899317,11cyclictest24010-21munin-run00:10:010
20753993015,6cyclictest0-21swapper/123:17:031
2075399301,17cyclictest19795-40updatedb00:00:101
2075399300,15cyclictest0-21swapper/119:33:271
2075399300,15cyclictest0-21swapper/119:33:271
2074899302,13cyclictest22041-21munin-run20:25:000
2075399281,10cyclictest0-21swapper/122:55:191
2074899283,11cyclictest0-21swapper/023:00:020
2074899282,11cyclictest30604-21kworker/0:320:45:150
2074899281,15cyclictest17377-21munin-run23:55:010
2075399271,9cyclictest0-21swapper/122:43:511
2074899279,15cyclictest22928-21awk19:15:000
2074899272,11cyclictest31741-21kworker/0:323:15:010
20748992713,11cyclictest15169-21dump-pmu-power23:50:010
20748992710,3cyclictest20501-21kworker/0:219:15:210
2075399261,20cyclictest86750irq/128-enp1s0-21:07:391
2074899260,10cyclictest20058-21kworker/0:220:25:130
93912250,3sleep10-21swapper/123:35:191
2075399251,3cyclictest18276-21munin-run21:30:011
2075399251,18cyclictest0-21swapper/100:30:331
20753992511,7cyclictest21013-21kworker/u8:022:50:061
2074899258,3cyclictest22604-21kworker/0:321:45:170
2074899252,9cyclictest4086-21kworker/0:123:35:330
2074899251,14cyclictest11334-21kworker/0:021:15:160
20748992511,3cyclictest27402-21kworker/0:123:07:300
2076499244,9cyclictest0-21swapper/323:16:453
2075399242,9cyclictest13514-21df23:45:131
20753992418,3cyclictest29529-21sed20:40:191
2075399240,18cyclictest0-21swapper/100:05:451
2074899249,3cyclictest11334-21kworker/0:021:14:460
2074899242,9cyclictest21615-21kworker/0:300:12:050
2074899241,7cyclictest4999-21kworker/0:321:09:010
20748992411,3cyclictest8120-21seq22:20:030
218652230,1sleep10-21swapper/119:10:201
2076499237,9cyclictest28643-21idleruntime23:05:203
2076499231,6cyclictest0-21swapper/321:42:393
2076499231,4cyclictest0-21swapper/321:20:013
2075999231,7cyclictest0-21swapper/200:35:222
2075399239,2cyclictest14708-21mailstats20:05:231
2075399231,3cyclictest7306-21timerandwakeup22:15:261
2074899239,4cyclictest371-21kworker/0:220:57:410
2074899232,9cyclictest20501-21kworker/0:219:35:580
2074899232,11cyclictest30139-21kworker/0:319:33:100
2074899232,11cyclictest30139-21kworker/0:319:33:100
2074899231,9cyclictest31158-21kworker/0:322:12:210
2074899231,10cyclictest26118-21kworker/0:120:41:090
2074899231,10cyclictest26118-21kworker/0:120:39:170
2076499228,9cyclictest0-21swapper/321:36:053
2076499228,9cyclictest0-21swapper/321:36:053
2076499221,3cyclictest3402-21kworker/u8:122:15:203
2076499221,10cyclictest14634-21snmpd00:36:093
2075399222,15cyclictest0-21swapper/100:17:341
20753992213,3cyclictest0-21swapper/100:38:341
2075399221,19cyclictest0-21swapper/123:10:141
2075399221,19cyclictest0-21swapper/119:25:241
2075399221,12cyclictest13867-21cat22:30:251
2074899229,3cyclictest21615-21kworker/0:300:18:290
2074899228,3cyclictest13822-21kworker/0:221:30:220
2074899226,2cyclictest13822-21kworker/0:221:23:230
2074899222,9cyclictest31741-21kworker/0:323:15:500
2074899222,9cyclictest15318-21kworker/0:020:14:380
2074899221,9cyclictest8809-21kworker/0:119:55:500
2074899221,9cyclictest32496-21kworker/0:000:32:530
2074899221,9cyclictest1991-21kworker/0:019:44:220
2074899221,7cyclictest13105-21kworker/0:220:03:580
20748992217,3cyclictest11665-21cat22:25:270
2074899221,15cyclictest13822-21kworker/0:221:36:500
2074899221,15cyclictest13822-21kworker/0:221:36:500
2074899221,14cyclictest5500-21kworker/0:122:30:150
2074899221,10cyclictest27959-21kworker/0:019:28:060
20748992210,2cyclictest5500-21kworker/0:122:39:490
20748992210,2cyclictest5500-21kworker/0:122:39:490
2074899220,10cyclictest0-21swapper/019:52:540
213332211,3sleep321513-21cat22:50:133
2076499216,3cyclictest14634-21snmpd20:40:243
2076499215,5cyclictest9012-21latency22:20:223
2076499211,9cyclictest13362cut22:05:013
2076499211,6cyclictest0-21swapper/323:24:493
20764992114,4cyclictest25849-21tr23:00:123
2076499211,11cyclictest842-21lldpd20:25:233
2075999212,6cyclictest27352-21memory00:15:212
2075399218,10cyclictest0-21swapper/121:39:561
2075399218,10cyclictest0-21swapper/121:39:561
2075399216,3cyclictest0-21swapper/120:37:311
2075399216,3cyclictest0-21swapper/119:35:231
2075399212,3cyclictest0-21swapper/122:27:541
2075399212,15cyclictest0-21swapper/123:51:351
2075399212,12cyclictest88150irq/134-enp2s0-19:16:011
2075399211,9cyclictest0-21swapper/100:27:171
2075399211,8cyclictest86850irq/129-enp1s0-23:00:211
2075399211,8cyclictest0-21swapper/121:40:101
20753992117,3cyclictest0-21swapper/121:15:381
20753992117,3cyclictest0-21swapper/120:56:081
20753992115,4cyclictest0-21swapper/123:55:271
20753992113,3cyclictest0-21swapper/100:22:251
2075399211,18cyclictest9-21rcu_preempt23:40:261
2075399211,18cyclictest9-21rcu_preempt23:40:261
2075399211,18cyclictest0-21swapper/123:30:231
2075399211,18cyclictest0-21swapper/122:37:171
2075399211,18cyclictest0-21swapper/122:37:171
2075399211,15cyclictest0-21swapper/100:14:221
2075399211,10cyclictest30863-21timerandwakeup21:55:271
2075399211,10cyclictest0-21swapper/120:18:111
2075399211,10cyclictest0-21swapper/120:13:391
2075399210,18cyclictest25698-21gunzip20:30:291
2074899218,3cyclictest26118-21kworker/0:120:34:380
2074899218,3cyclictest12895-21kworker/0:123:56:290
2074899217,3cyclictest22343-21kworker/0:121:56:050
2074899211,9cyclictest17578-21kworker/0:120:16:500
2074899211,7cyclictest2759-21kworker/0:120:54:450
2074899211,12cyclictest0-21swapper/021:50:130
2074899211,11cyclictest31158-21kworker/0:322:00:250
2074899211,10cyclictest30139-21kworker/0:319:45:420
20748992110,3cyclictest23816-21memory21:40:200
120402210,4sleep230-21rcuc/220:00:192
9352201,2sleep10-21swapper/122:00:501
228072201,2sleep20-21swapper/219:13:482
2076499201,9cyclictest0-21swapper/322:57:393
2076499201,11cyclictest3745-21/usr/sbin/munin22:10:243
2076499201,11cyclictest30134-21/usr/sbin/munin23:10:203
2076499201,11cyclictest19758-21taskset20:19:113
2076499201,11cyclictest14634-21snmpd00:23:563
2076499201,10cyclictest8872-21cpuspeed_turbos23:35:153
2076499201,10cyclictest15400-21/usr/sbin/munin23:50:243
2076499201,10cyclictest0-21swapper/320:09:543
2075999202,3cyclictest0-21swapper/222:23:042
2075999201,5cyclictest0-21swapper/222:39:502
2075999201,5cyclictest0-21swapper/222:39:502
20759992015,3cyclictest20935-21pmu-power00:00:242
2075999201,4cyclictest0-21swapper/222:57:042
20759992014,4cyclictest17960-21ntp_kernel_err22:40:212
2075399207,3cyclictest21119-21idleruntime-cro22:50:011
2075399205,3cyclictest0-21swapper/119:49:231
2075399202,3cyclictest0-21swapper/121:33:001
2075399202,3cyclictest0-21swapper/120:04:011
2075399201,9cyclictest0-21swapper/121:00:111
20753992017,2cyclictest0-21swapper/123:05:121
20753992016,3cyclictest0-21swapper/122:12:501
2075399201,3cyclictest0-21swapper/121:45:291
2075399201,18cyclictest0-21swapper/121:52:381
2075399201,17cyclictest0-21swapper/122:07:361
2075399201,17cyclictest0-21swapper/120:51:251
2075399201,10cyclictest10999-21sed21:10:241
2075399200,18cyclictest0-21swapper/120:20:481
2074899207,2cyclictest21079-21kworker/0:222:50:170
20748992010,3cyclictest3402-21kworker/u8:122:40:270
2074899200,9cyclictest4647-21kworker/0:100:38:450
194022200,2sleep20-21swapper/220:15:282
128532200,2sleep30-21swapper/322:30:153
3432191,2sleep30-21swapper/320:47:273
2076499198,3cyclictest10336-21/usr/sbin/munin22:25:183
2076499195,3cyclictest0-21swapper/321:05:193
2076499194,9cyclictest22550-40dnf19:10:323
2076499193,3cyclictest2232-21sh19:40:013
2076499192,7cyclictest22691-21hddtemp_smartct00:05:163
2076499191,9cyclictest9767-21cpuspeed_turbos21:10:123
2076499191,9cyclictest0-21swapper/321:27:103
2076499191,7cyclictest0-21swapper/321:51:323
2076499191,7cyclictest0-21swapper/300:16:573
2076499191,6cyclictest0-21swapper/320:11:413
2076499191,5cyclictest18626-21cpuspeed_turbos21:30:143
2076499191,12cyclictest0-21swapper/322:44:593
2076499191,10cyclictest569-21/usr/sbin/munin00:30:223
2076499191,10cyclictest3102-21/usr/sbin/munin20:55:273
2076499191,10cyclictest0-21swapper/321:57:263
2075999193,6cyclictest0-21swapper/221:45:232
2075999191,8cyclictest22176-21grep00:05:122
2075999191,8cyclictest19572-21sleep00:00:012
2075999191,7cyclictest31468-21munin-run22:00:012
2075999191,7cyclictest0-21swapper/221:20:572
2075999191,6cyclictest0-21swapper/222:13:282
2075999191,5cyclictest0-21swapper/223:53:472
2075999191,4cyclictest0-21swapper/221:37:542
2075999191,4cyclictest0-21swapper/221:37:542
20759991912,4cyclictest22633-21grep20:25:162
2075999191,11cyclictest0-21swapper/221:17:172
2075999191,10cyclictest0-21swapper/223:16:292
2075999190,6cyclictest0-21swapper/223:45:242
2075399192,16cyclictest0-21swapper/122:23:391
2075399191,17cyclictest0-21swapper/121:24:431
2075399191,17cyclictest0-21swapper/119:50:261
2074899198,3cyclictest28635-21/usr/sbin/munin00:20:160
2074899196,3cyclictest6050-21kworker/0:323:42:530
2074899196,3cyclictest6050-21kworker/0:323:42:530
2074899195,3cyclictest18842-21kworker/0:022:47:570
20748991913,3cyclictest4960-21grep23:25:160
2074899191,10cyclictest4086-21kworker/0:123:30:190
2074899190,10cyclictest16050-21kworker/0:121:25:220
69062180,1sleep20-21swapper/223:30:152
27712181,2sleep30-21swapper/320:53:573
2076499187,3cyclictest784-21NetworkManager20:21:383
2076499186,3cyclictest86550irq/126-enp1s0-22:05:243
2076499183,12cyclictest4661-21/usr/sbin/munin19:45:223
2076499183,10cyclictest41-21ksoftirqd/300:13:463
2076499182,8cyclictest86550irq/126-enp1s0-21:20:253
2076499182,10cyclictest41-21ksoftirqd/323:47:573
2076499181,9cyclictest31568-21tr19:30:233
2076499181,9cyclictest31568-21tr19:30:233
2076499181,9cyclictest0-21swapper/323:55:493
2076499181,7cyclictest27065-21latency_hist21:50:023
2076499181,4cyclictest14634-21snmpd20:00:163
20764991813,3cyclictest6330-21awk21:00:213
20764991812,3cyclictest25056-21rt-features19:15:253
20764991811,4cyclictest4517-21tr23:25:133
2076499181,11cyclictest41-21ksoftirqd/322:47:443
2076499180,8cyclictest0-21swapper/323:40:153
2076499180,8cyclictest0-21swapper/323:40:153
2075999183,9cyclictest0-21swapper/221:31:022
2075999183,5cyclictest0-21swapper/223:43:142
2075999183,5cyclictest0-21swapper/223:43:142
2075999181,8cyclictest3021-21fschecks_time19:40:122
2075999181,7cyclictest4102-21netstat20:55:262
2075999181,6cyclictest856-21/usr/sbin/munin20:50:252
2075999181,6cyclictest0-21swapper/222:00:142
2075999181,6cyclictest0-21swapper/221:11:172
2075999181,3cyclictest31741-21memory00:25:192
2075999181,3cyclictest297272open_inodes00:20:252
2075999181,3cyclictest0-21swapper/222:25:322
20759991813,3cyclictest26807-21cut20:35:122
20759991813,3cyclictest23151-21munin-node19:15:052
2075399181,2cyclictest0-21swapper/120:45:031
2075399181,16cyclictest0-21swapper/123:26:301
2074899187,3cyclictest5500-21kworker/0:122:18:290
142242181,3sleep20-21swapper/222:31:382
312862170,1sleep20-21swapper/223:10:232
2076499172,6cyclictest0-21swapper/319:44:153
2076499171,9cyclictest0-21swapper/323:30:153
2076499171,6cyclictest14731-21/usr/sbin/munin22:35:153
2076499171,6cyclictest14731-21/usr/sbin/munin22:35:153
2076499171,5cyclictest0-21swapper/300:28:273
2076499171,3cyclictest86550irq/126-enp1s0-20:30:223
20764991711,4cyclictest848-21sssd_nss00:00:243
20764991711,3cyclictest27690-21ntpq20:35:213
2076499171,10cyclictest41-21ksoftirqd/319:56:173
2075999175,4cyclictest14634-21snmpd22:18:402
2075999171,9cyclictest0-21swapper/223:00:122
2075999171,7cyclictest9641-21/usr/sbin/munin19:55:162
2075999171,4cyclictest0-21swapper/222:05:242
20759991712,3cyclictest8106-21expr21:05:162
20759991712,3cyclictest27639-21df_inode21:50:152
20759991712,3cyclictest22056-21if_enp1s022:50:182
20759991712,2cyclictest2335-21cpuspeed_turbos23:20:152
2075999171,10cyclictest0-21swapper/222:47:532
2075999170,15cyclictest0-21swapper/200:30:382
104152171,2sleep20-21swapper/223:36:092
2076499161,9cyclictest0-21swapper/319:52:053
2076499161,6cyclictest26057-21cpu19:20:113
20764991612,2cyclictest28898-21grep19:25:163
2075999169,4cyclictest24400-21grep00:10:142
2075999169,4cyclictest1-21systemd21:00:092
2075999165,3cyclictest12050irq/123-ahci[0019:25:492
2075999162,6cyclictest7660-21interrupts19:50:172
2075999161,6cyclictest31975-21memory20:45:222
2075999161,5cyclictest0-21swapper/223:25:162
2075999161,5cyclictest0-21swapper/223:05:452
2075999161,5cyclictest0-21swapper/220:10:152
2075999161,4cyclictest0-21swapper/220:07:412
2075999161,3cyclictest0-21swapper/221:41:492
20759991612,3cyclictest0-21swapper/219:24:122
20759991611,3cyclictest5475-21kernelversion19:45:172
20759991611,3cyclictest28972-21grep20:40:152
20759991611,3cyclictest24556-21ls20:30:182
20759991611,3cyclictest1342-21open_inodes19:35:232
20759991611,3cyclictest1-21systemd19:30:092
20759991611,3cyclictest1-21systemd19:30:092
2075999160,6cyclictest0-21swapper/220:20:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional