You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 07:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Dec 03, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14942239162,25sleep00-21swapper/019:09:170
14282237164,24sleep20-21swapper/219:08:262
13752237163,59sleep10-21swapper/119:07:451
13772236165,58sleep30-21swapper/319:07:473
1669998785,1cyclictest0-21swapper/222:04:232
16852580,3sleep00-21swapper/021:35:120
165999381,10cyclictest7575-21/usr/sbin/munin20:35:140
1659993712,2cyclictest19825-21sensors_temp21:00:250
1659993712,2cyclictest19825-21sensors_temp21:00:250
1659993712,22cyclictest8848-21/usr/sbin/munin23:05:180
165999371,11cyclictest28192-21sshd23:47:200
165999361,10cyclictest6955-21/usr/sbin/munin19:20:250
1659993610,12cyclictest21811-21munin-run23:35:010
167499350,2cyclictest88250irq/135-enp2s0-00:20:253
165999355,14cyclictest13316-21munin-run19:35:000
1659993512,3cyclictest11107-21cat20:40:240
165999351,13cyclictest2899-21/usr/sbin/munin00:05:290
1659993510,3cyclictest28945-21/usr/sbin/munin20:10:160
167499341,24cyclictest0-21swapper/322:27:513
1659993410,2cyclictest13902-21/usr/sbin/munin00:30:200
167499331,22cyclictest0-21swapper/322:44:393
165999338,3cyclictest7293-21grep23:00:170
165999334,14cyclictest1376-21perf21:35:010
165999331,6cyclictest0-21swapper/021:20:170
1659993313,17cyclictest15246-21seq19:36:460
1659993312,3cyclictest21408-21/usr/sbin/munin22:20:130
165999331,13cyclictest5785-21munin-run21:45:000
165999331,13cyclictest18891-21kworker/0:222:14:310
167499325,11cyclictest5787-21perf21:45:003
167499321,12cyclictest0-21swapper/320:00:313
167499320,26cyclictest0-21swapper/323:53:433
165999322,6cyclictest31926-40dnf20:15:190
165999321,5cyclictest0-21swapper/021:45:140
167499315,8cyclictest86550irq/126-enp1s0-22:50:203
167499314,11cyclictest780-21rngd20:06:333
167499310,24cyclictest0-21swapper/323:37:433
167499310,14cyclictest0-21swapper/319:18:233
165999319,3cyclictest27137-21crond21:20:010
165999319,10cyclictest15233-21perf23:20:010
165999318,3cyclictest836-21runrttasks21:25:160
165999316,5cyclictest0-21swapper/000:25:170
165999314,8cyclictest30325-21cpuspeed_turbos22:40:130
165999313,13cyclictest24307-21munin-run20:00:010
165999313,13cyclictest24307-21munin-run20:00:010
165999312,22cyclictest404-21kworker/0:120:28:110
165999311,5cyclictest14170-21/usr/sbin/munin20:50:140
1659993112,3cyclictest8909-21seq00:15:290
1659993112,3cyclictest8909-21seq00:15:290
1659993112,16cyclictest31911-21sh23:55:220
1659993111,3cyclictest955-21/usr/sbin/munin20:20:130
1659993111,3cyclictest26523-21perf20:05:010
1659993110,3cyclictest27018-21expr20:05:140
1659993110,3cyclictest19468-21seq19:45:250
1659993110,2cyclictest836-21runrttasks20:34:570
165999310,4cyclictest12050irq/123-ahci[0019:20:010
167499305,15cyclictest0-21swapper/321:15:233
167499304,11cyclictest780-21rngd19:36:073
165999309,2cyclictest32736-21df22:45:130
165999308,3cyclictest836-21runrttasks00:23:460
165999308,3cyclictest25419-21seq22:28:030
165999308,3cyclictest12959-21entropy22:00:140
1659993012,3cyclictest30003-21wc22:40:010
165999301,13cyclictest10190-21awk21:55:010
165999301,10cyclictest4361-21ls22:55:010
1659993010,3cyclictest22050-21seq19:54:280
167499295,10cyclictest780-21rngd23:18:313
166499291,21cyclictest0-21swapper/121:20:261
165999299,3cyclictest24271-21cpu23:40:120
165999299,3cyclictest18410-21if_err_enp1s023:25:160
165999298,4cyclictest0-21swapper/020:45:280
165999298,3cyclictest836-21runrttasks21:09:050
165999291,2cyclictest0-21swapper/022:15:220
1659992911,3cyclictest29282-21sed23:50:180
1659992911,3cyclictest17436-21threads00:35:260
165999291,12cyclictest6169-21sshd00:10:200
1659992910,3cyclictest16163-21sort22:05:240
1659992910,3cyclictest11238-21cut21:55:180
167499284,15cyclictest780-21rngd20:45:133
167499282,17cyclictest0-21swapper/320:53:433
167499281,12cyclictest0-21swapper/319:10:153
167499281,11cyclictest0-21swapper/319:47:593
167499280,13cyclictest0-21swapper/321:54:233
1664992818,3cyclictest0-21swapper/100:12:131
165999289,3cyclictest11863-21uname23:10:180
165999289,3cyclictest11863-21uname23:10:180
165999288,3cyclictest836-21runrttasks19:44:190
165999287,3cyclictest10056-21load19:25:190
1659992810,3cyclictest22738-21idleruntime23:35:180
167499276,10cyclictest86650irq/127-enp1s0-19:54:353
167499275,9cyclictest780-21rngd19:40:353
167499275,6cyclictest842-21lldpd23:45:203
167499273,9cyclictest86650irq/127-enp1s0-19:28:073
167499273,15cyclictest0-21swapper/300:18:133
167499273,15cyclictest0-21swapper/300:18:133
167499271,10cyclictest0-21swapper/322:58:233
167499271,10cyclictest0-21swapper/320:35:113
167499270,9cyclictest0-21swapper/322:45:513
167499270,11cyclictest0-21swapper/323:30:153
1664992720,4cyclictest10576-21ntp_states00:20:221
165999278,3cyclictest17764-21cat20:55:240
165999274,3cyclictest86750irq/128-enp1s0-00:00:180
165999270,2cyclictest86550irq/126-enp1s0-22:55:270
167499265,11cyclictest0-21swapper/300:35:163
167499263,6cyclictest86550irq/126-enp1s0-22:15:203
167499263,11cyclictest0-21swapper/321:12:043
1674992613,3cyclictest32189-21timerandwakeup23:55:253
167499261,17cyclictest20564-21cat21:05:013
167499261,17cyclictest20564-21cat21:05:013
167499261,12cyclictest0-21swapper/323:02:073
166499267,11cyclictest0-21swapper/123:32:391
166499261,15cyclictest0-21swapper/123:42:121
165999262,6cyclictest86550irq/126-enp1s0-23:20:050
165999261,10cyclictest24009-21ntpq21:10:210
167499252,13cyclictest22330-21/usr/sbin/munin19:55:273
167499252,13cyclictest22330-21/usr/sbin/munin19:55:273
167499252,11cyclictest0-21swapper/322:38:333
1674992512,4cyclictest0-21swapper/321:23:033
167499251,10cyclictest0-21swapper/321:35:033
167499250,16cyclictest30497-21missed_timers21:25:213
167499250,14cyclictest0-21swapper/323:40:253
167499250,11cyclictest0-21swapper/322:22:313
167499250,11cyclictest0-21swapper/300:32:553
166999253,6cyclictest26219-40dnf21:15:232
1669992521,3cyclictest0-21swapper/220:41:402
167499243,12cyclictest0-21swapper/320:26:033
167499242,11cyclictest780-21rngd20:11:393
167499241,11cyclictest32350-21perl21:30:163
167499241,10cyclictest0-21swapper/320:30:513
167499241,10cyclictest0-21swapper/319:20:033
167499241,10cyclictest0-21swapper/300:13:513
167499240,21cyclictest6642-21gltestperf21:45:163
167499240,13cyclictest0-21swapper/323:21:193
166499242,10cyclictest13458-21ls22:00:191
166499241,17cyclictest0-21swapper/122:27:451
166499241,17cyclictest0-21swapper/121:27:351
166499241,16cyclictest0-21swapper/121:56:281
166499241,16cyclictest0-21swapper/100:17:331
166499241,16cyclictest0-21swapper/100:17:331
166499241,16cyclictest0-21swapper/100:07:591
166499241,15cyclictest13843-21latency_hist00:30:011
166499241,15cyclictest0-21swapper/121:34:081
114722240,2sleep20-21swapper/221:55:212
167499236,2cyclictest0-21swapper/300:00:233
167499232,9cyclictest0-21swapper/323:25:533
167499232,9cyclictest0-21swapper/320:20:033
1674992315,5cyclictest17602-21find20:55:233
167499231,10cyclictest0-21swapper/322:10:573
167499230,12cyclictest0-21swapper/322:10:013
167499230,12cyclictest0-21swapper/320:15:433
167499230,10cyclictest13794-21latency_hist00:30:013
166999234,4cyclictest12050irq/123-ahci[0000:00:012
166999232,2cyclictest32-21ksoftirqd/222:10:212
166499232,14cyclictest86750irq/128-enp1s0-21:44:231
166499231,3cyclictest175592sleep119:43:131
166499231,19cyclictest836-21runrttasks22:48:341
166499231,19cyclictest1979-21forks21:35:151
166499231,16cyclictest0-21swapper/123:02:451
166499231,15cyclictest0-21swapper/122:05:531
166499231,11cyclictest0-21swapper/123:15:331
165999234,3cyclictest12050irq/123-ahci[0022:35:010
167499228,5cyclictest0-21swapper/322:02:203
167499227,6cyclictest0-21swapper/323:06:543
167499221,9cyclictest0-21swapper/321:56:513
167499221,9cyclictest0-21swapper/300:05:533
167499220,9cyclictest0-21swapper/322:30:163
167499220,10cyclictest113382awk20:40:263
166999225,7cyclictest29520-21unixbench_multi22:35:292
166999221,5cyclictest0-21swapper/200:12:142
166499228,3cyclictest19735-21sed23:30:011
166499225,9cyclictest25667-21fschecks_time21:15:151
166499222,11cyclictest31121-21memory22:40:191
1664992217,3cyclictest0-21swapper/119:50:241
166499221,6cyclictest0-21swapper/123:20:141
1664992216,3cyclictest26726-21memory22:30:181
1664992215,2cyclictest0-21swapper/119:55:171
1664992215,2cyclictest0-21swapper/119:55:171
166499221,19cyclictest0-21swapper/119:10:201
166499221,18cyclictest8558-21diskstats21:50:161
166499221,18cyclictest24642-21cpuspeed_turbos20:00:131
166499221,18cyclictest0-21swapper/122:20:131
166499221,15cyclictest0-21swapper/120:10:021
166499221,15cyclictest0-21swapper/119:37:571
166499221,12cyclictest19811-21grep22:15:171
166499220,3cyclictest0-21swapper/122:58:041
167499215,4cyclictest12050irq/123-ahci[0019:35:013
167499210,9cyclictest0-21swapper/323:12:413
167499210,9cyclictest0-21swapper/323:12:413
166999218,3cyclictest86850irq/129-enp1s0-19:26:522
166999216,3cyclictest6016-21smart_sda19:15:252
1669992118,2cyclictest0-21swapper/200:16:062
1669992118,2cyclictest0-21swapper/200:16:062
1669992115,3cyclictest32664-21grep20:15:282
166999211,4cyclictest24794-21perl23:40:162
166499212,11cyclictest18909-21df_inode21:00:151
166499212,11cyclictest18909-21df_inode21:00:151
1664992114,4cyclictest23592-21grep21:10:161
1664992114,2cyclictest0-21swapper/119:46:161
166499211,18cyclictest0-21swapper/123:10:471
166499211,18cyclictest0-21swapper/123:10:471
166499211,15cyclictest0-21swapper/120:19:471
166499211,11cyclictest5154-21munin-run20:30:011
166499211,10cyclictest9549-21munin-run20:40:011
166499210,9cyclictest16446-21diskstats00:35:151
166499210,12cyclictest0-21swapper/119:18:071
167499201,8cyclictest210512cat21:05:133
166999204,4cyclictest21746-21cat21:05:202
166999203,4cyclictest31546-21munin-run21:30:012
166999203,4cyclictest29802-21ls20:10:212
166999202,7cyclictest30-21rcuc/219:20:262
166999201,6cyclictest11704-21forks19:30:152
166999201,11cyclictest0-21swapper/223:03:442
166499209,3cyclictest28977-21df23:50:161
166499208,8cyclictest0-21swapper/123:35:281
166499208,7cyclictest0-21swapper/100:04:261
166499206,8cyclictest0-21swapper/123:48:401
1664992016,3cyclictest0-21swapper/123:57:261
1664992016,3cyclictest0-21swapper/121:49:241
1664992015,2cyclictest6382-21netstat20:30:201
166499201,1cyclictest0-21swapper/122:35:291
166499201,18cyclictest0-21swapper/120:51:451
166499201,18cyclictest0-21swapper/120:47:421
166499201,18cyclictest0-21swapper/120:07:301
166499201,18cyclictest0-21swapper/119:22:171
166499201,17cyclictest0-21swapper/122:10:481
166499201,17cyclictest0-21swapper/120:55:031
166499201,10cyclictest3571-21switchtime22:50:251
166499201,10cyclictest0-21swapper/100:30:131
166499200,15cyclictest0-21swapper/120:21:151
165999207,2cyclictest3642-21kworker/0:119:14:450
166999195,3cyclictest12050irq/123-ahci[0022:55:012
166999194,4cyclictest2320-21idleruntime21:35:172
166999193,3cyclictest28748-21timerandwakeup21:20:262
166999191,9cyclictest0-21swapper/220:25:212
166999191,7cyclictest581-21systemd-journal21:05:012
166999191,7cyclictest581-21systemd-journal21:05:012
166999191,7cyclictest23372-21kworker/u8:022:40:282
166999191,6cyclictest0-21swapper/219:59:462
166999191,6cyclictest0-21swapper/219:59:452
166999191,6cyclictest0-21swapper/200:00:422
1669991914,3cyclictest11151-21tr23:10:132
1669991914,3cyclictest11151-21tr23:10:132
1669991913,4cyclictest12574-21tail20:45:182
166999191,12cyclictest0-21swapper/223:37:272
166499193,3cyclictest10627-21unixbench-2d19:25:261
166499191,17cyclictest0-21swapper/123:05:021
166499191,17cyclictest0-21swapper/119:33:371
166499191,10cyclictest14634-21snmpd20:40:481
166999186,3cyclictest12526-21kernelversion00:25:202
166999183,6cyclictest32-21ksoftirqd/220:05:272
166999181,9cyclictest0-21swapper/221:41:162
166999181,7cyclictest0-21swapper/222:25:432
166999181,7cyclictest0-21swapper/222:23:262
166999181,6cyclictest3814-21memory00:05:212
166999181,6cyclictest20444-21grep23:30:172
166999181,5cyclictest0-21swapper/200:40:032
1669991813,2cyclictest9603-21cpuspeed_turbos00:20:132
1669991812,3cyclictest8969-21cat20:35:282
1669991812,3cyclictest14235-21df00:30:142
166999181,10cyclictest0-21swapper/223:06:562
166999181,10cyclictest0-21swapper/222:55:222
166999180,7cyclictest0-21swapper/220:30:142
1664991815,2cyclictest0-21swapper/121:05:121
166999174,2cyclictest86550irq/126-enp1s0-23:47:172
166999171,7cyclictest17690-21cat19:44:552
166999171,6cyclictest26802-21needreboot22:30:202
166999171,6cyclictest0-21swapper/223:18:472
166999171,5cyclictest1592-21unixbench_singl22:45:272
166999171,4cyclictest0-21swapper/221:13:522
1669991712,3cyclictest6635-21gltestperf21:45:162
1669991712,3cyclictest31953-21cstates21:30:132
1669991712,3cyclictest20470-21sensors_temp22:15:272
1669991711,3cyclictest19259-21cat19:45:242
166999171,13cyclictest15398-21tr22:05:152
166999171,10cyclictest0-21swapper/220:23:112
166999163,4cyclictest17975-21df23:25:122
166999162,11cyclictest15087-21unixbench_singl19:35:282
166999161,9cyclictest0-21swapper/220:54:512
166999161,6cyclictest20739-21grep19:50:192
166999161,6cyclictest0-21swapper/223:20:512
1669991612,3cyclictest0-21swapper/220:02:202
1669991612,2cyclictest7973-21latency_hist21:50:012
1669991612,2cyclictest28741-21tr23:50:152
1669991611,3cyclictest17967-21vmstat20:55:272
1669991610,3cyclictest2155-21cat19:10:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional