You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 06:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Sep 18, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
824823030,5sleep230-21rcuc/219:05:262
88402215144,57sleep30-21swapper/319:09:303
88072205166,26sleep10-21swapper/119:09:041
86172204166,25sleep00-21swapper/019:06:360
183282540,2sleep10-21swapper/123:15:191
898399395,21cyclictest780-21rngd20:00:152
898399393,31cyclictest6473-40dnf00:01:032
898399391,7cyclictest31988-21timerwakeupswit23:45:272
897799371,19cyclictest0-21swapper/120:56:141
898399363,6cyclictest25871-40dnf21:00:132
898399361,5cyclictest25019-21latency_hist19:45:022
897199367,13cyclictest11520-21awk21:45:000
898399354,3cyclictest12050irq/123-ahci[0022:15:022
898399352,30cyclictest16014-21taskset21:55:232
898399351,5cyclictest16760-21memory00:25:202
898399351,22cyclictest23676-21/usr/sbin/munin20:55:242
898399342,28cyclictest20187-21unixbench_multi19:30:272
898399341,7cyclictest3866-21cat20:10:022
898399341,7cyclictest21436-21processes00:35:232
898399341,5cyclictest31543-21users22:30:282
898399341,5cyclictest14260-21/usr/sbin/munin19:20:182
8983993410,20cyclictest4567-21/usr/sbin/munin20:10:172
898399338,22cyclictest21826-21df_inode20:50:182
898399336,14cyclictest12050irq/123-ahci[0021:35:022
898399332,4cyclictest0-21swapper/200:11:022
898399332,3cyclictest11094-21sshd19:14:352
898399331,4cyclictest86550irq/126-enp1s0-00:05:162
8983993312,18cyclictest2903-21missed_timers22:40:212
898399327,22cyclictest25253-21/usr/sbin/munin19:45:252
898399324,3cyclictest86750irq/128-enp1s0-00:35:022
898399323,3cyclictest86750irq/128-enp1s0-23:25:192
898399323,2cyclictest86550irq/126-enp1s0-23:15:262
898399321,4cyclictest10349-21irqstats21:40:192
8983993211,18cyclictest20528-21sendmail_mailtr20:45:242
181182320,3sleep20-21swapper/222:00:252
898399312,5cyclictest12399-40dnf23:00:372
898399312,3cyclictest31571-40dnf19:59:372
898399311,4cyclictest7004-21memory20:15:202
898399311,4cyclictest3361-21/usr/sbin/munin23:55:192
898399311,4cyclictest3361-21/usr/sbin/munin23:55:192
898399311,3cyclictest16198-21wc20:35:252
8983993111,17cyclictest29650-21/usr/sbin/munin23:40:262
8983993110,18cyclictest23939-21grep22:15:162
897799311,14cyclictest0-21swapper/121:35:181
897799311,11cyclictest15695-21awk21:55:201
897199314,12cyclictest4928-21idleruntime-cro21:30:000
898799301,13cyclictest0-21swapper/322:04:073
898399309,18cyclictest11824-21threads20:25:252
898399308,19cyclictest6647-21cat22:50:132
898399308,19cyclictest6647-21cat22:50:132
898399308,19cyclictest28799-21ntp_kernel_pll_22:25:202
898399308,19cyclictest12777-21ntp_kernel_pll_21:45:192
898399308,19cyclictest12777-21ntp_kernel_pll_21:45:192
898399306,3cyclictest29055-21kworker/2:220:22:102
898399303,3cyclictest19301-21/usr/sbin/munin22:05:162
898399302,4cyclictest24899-21memory23:30:172
898399302,3cyclictest12050irq/123-ahci[0023:00:022
898399302,3cyclictest12050irq/123-ahci[0019:55:022
898399301,6cyclictest8312-21needreboot21:35:222
898399301,4cyclictest13648-21/usr/sbin/munin00:20:142
898399301,3cyclictest800-21/usr/sbin/munin22:35:232
898399301,3cyclictest16251-21/usr/sbin/munin23:10:232
897799309,3cyclictest86650irq/127-enp1s0-19:36:201
898399299,17cyclictest3326-21cut21:25:162
898399298,18cyclictest836-21runrttasks23:09:592
898399292,3cyclictest12832-21/usr/sbin/munin19:15:192
898399292,24cyclictest27522-21latency_hist22:25:022
898399291,5cyclictest16787-21df19:25:112
898399291,4cyclictest1780-21/usr/sbin/munin21:20:202
898399291,4cyclictest13716-21/usr/sbin/munin20:30:202
898399291,3cyclictest5289-21/usr/sbin/munin22:45:232
898399291,3cyclictest32011-21/usr/sbin/munin21:15:242
898399291,3cyclictest29410-21idleruntime21:10:142
898399291,3cyclictest27523-21/usr/sbin/munin23:35:272
898399291,3cyclictest19761-21/usr/sbin/munin23:20:142
8983992912,14cyclictest28474-21sh21:10:022
897799291,8cyclictest0-21swapper/122:50:131
897799291,8cyclictest0-21swapper/122:50:131
897199294,12cyclictest14662-21munin-run20:35:000
898799288,3cyclictest0-21swapper/319:50:253
898799281,12cyclictest0-21swapper/323:45:053
898799281,12cyclictest0-21swapper/322:39:403
898399287,18cyclictest21019-21cstates19:35:132
898399281,4cyclictest17252-21/usr/sbin/munin20:40:132
898399281,4cyclictest17252-21/usr/sbin/munin20:40:132
898399281,3cyclictest12075-21/usr/sbin/munin00:15:202
898399281,3cyclictest1070-21hddtemp_smartct23:50:172
897799283,6cyclictest780-21rngd21:07:561
897799281,18cyclictest14060-21ntp_states23:05:231
8971992811,14cyclictest27544-21perf22:25:010
898799271,9cyclictest0-21swapper/321:37:053
8987992715,6cyclictest25968-21ls21:00:123
8987992711,3cyclictest3189-21cstates21:25:133
898399272,3cyclictest14323-21munin-run21:55:012
897799273,3cyclictest0-21swapper/119:40:181
8987992615,6cyclictest6473-40dnf00:01:023
8987992611,3cyclictest14634-21snmpd22:55:593
898799261,12cyclictest25165-21seq22:18:063
8987992610,3cyclictest12688-21/usr/sbin/munin20:30:253
897799264,8cyclictest0-21swapper/120:17:281
897199261,12cyclictest10265-21munin-run20:25:000
898799251,9cyclictest0-21swapper/320:02:093
898799251,14cyclictest0-21swapper/321:55:233
897799254,3cyclictest0-21swapper/119:30:221
897799252,3cyclictest27251-21ntp_states23:35:231
897199259,14cyclictest29433-21awk19:55:000
897199251,12cyclictest31928-21munin-run22:35:000
897199251,12cyclictest30685-21munin-run21:15:010
897199251,12cyclictest11233-21awk00:15:010
897199250,11cyclictest0-21swapper/019:50:000
898799247,3cyclictest226902chrt19:38:273
8987992412,3cyclictest0-21swapper/321:09:413
8987992410,8cyclictest0-21swapper/300:09:513
897799246,3cyclictest2023-21awk23:50:281
897799243,9cyclictest0-21swapper/121:30:171
897799242,13cyclictest5498-21hddtemp_smartct00:00:141
897799241,15cyclictest0-21swapper/123:22:261
897799241,11cyclictest0-21swapper/121:20:301
897199248,14cyclictest0-21swapper/023:55:000
897199241,11cyclictest25973-21awk23:35:010
8971992410,13cyclictest0-21swapper/000:25:020
898799239,3cyclictest0-21swapper/323:13:053
898799237,11cyclictest0-21swapper/320:21:313
898799233,9cyclictest28477-21diskmemload23:26:373
898799231,14cyclictest0-21swapper/323:42:033
898799231,13cyclictest0-21swapper/322:30:183
898799231,11cyclictest496-21jbd2/dm-0-820:40:123
898799231,11cyclictest496-21jbd2/dm-0-820:40:113
897799234,3cyclictest0-21swapper/122:35:221
897799231,6cyclictest6158-21munin-run22:50:011
897799231,6cyclictest0-21swapper/122:16:371
897799231,6cyclictest0-21swapper/100:23:351
897799231,11cyclictest0-21swapper/100:10:241
897199236,14cyclictest31634-21idleruntime-cro20:00:010
8971992317,3cyclictest25945-21cpuspeed_turbos21:00:120
8971992316,4cyclictest6205-21ntp_states21:30:230
898799229,3cyclictest0-21swapper/322:08:233
898799228,2cyclictest0-21swapper/323:20:213
898799227,3cyclictest0-21swapper/321:34:353
898799226,3cyclictest0-21swapper/319:10:103
898799222,3cyclictest0-21swapper/322:40:283
898799221,6cyclictest31240-21df21:15:143
898799221,12cyclictest0-21swapper/300:16:193
8987992210,4cyclictest186332tr00:30:173
897799221,7cyclictest0-21swapper/119:51:561
8977992215,2cyclictest0-21swapper/119:46:481
8977992213,2cyclictest17261-21latency_hist23:15:021
897799221,18cyclictest23261-21grep23:25:271
897799221,15cyclictest0-21swapper/119:59:531
897799221,12cyclictest9985-21irqstats19:10:181
897799220,6cyclictest0-21swapper/123:57:201
897799220,6cyclictest0-21swapper/123:57:201
898799219,3cyclictest0-21swapper/319:20:473
898799214,3cyclictest0-21swapper/322:45:203
8987992116,3cyclictest2735-21tr23:55:133
8987992116,3cyclictest2735-21tr23:55:133
8987992114,6cyclictest0-21swapper/320:59:273
898799211,11cyclictest6384-21/usr/sbin/munin22:50:213
898799211,11cyclictest6384-21/usr/sbin/munin22:50:203
8987992110,3cyclictest0-21swapper/323:15:363
897799216,2cyclictest7917-21latency00:05:221
897799214,3cyclictest12950-21/usr/sbin/munin21:45:221
897799214,3cyclictest12950-21/usr/sbin/munin21:45:221
897799212,5cyclictest86650irq/127-enp1s0-21:10:341
8977992115,3cyclictest19295-21df22:05:161
8977992114,2cyclictest0-21swapper/120:20:331
8977992114,2cyclictest0-21swapper/100:25:551
897799211,3cyclictest29342-21users22:25:261
8977992113,3cyclictest0-21swapper/121:25:021
8977992113,3cyclictest0-21swapper/100:31:271
897799211,15cyclictest0-21swapper/123:31:271
897799211,15cyclictest0-21swapper/121:15:471
897799211,15cyclictest0-21swapper/121:03:101
897799211,15cyclictest0-21swapper/120:41:251
897799211,15cyclictest0-21swapper/120:41:251
897799211,15cyclictest0-21swapper/119:26:261
897799211,15cyclictest0-21swapper/100:36:381
897799211,15cyclictest0-21swapper/100:15:141
897799211,13cyclictest9218-21gltestperf22:55:181
897799211,11cyclictest28492-21tr23:40:141
897799210,16cyclictest0-21swapper/120:30:081
897199213,10cyclictest8582-21/usr/sbin/munin22:55:190
8971992114,4cyclictest7038-21tail22:50:150
8971992114,4cyclictest7038-21tail22:50:150
897199211,11cyclictest15310-21ntp_kernel_pll_19:20:210
898799206,5cyclictest0-21swapper/321:46:063
898799206,5cyclictest0-21swapper/321:46:063
898799204,3cyclictest0-21swapper/322:26:283
898799202,2cyclictest0-21swapper/300:11:113
898799202,14cyclictest28477-21diskmemload23:35:203
898799201,9cyclictest0-21swapper/319:18:293
898799201,6cyclictest0-21swapper/300:37:513
8987992015,3cyclictest27216-21idleruntime-cro19:50:003
898799201,3cyclictest0-21swapper/321:22:153
898799201,18cyclictest0-21swapper/322:21:543
898799201,11cyclictest9540-21/usr/sbin/munin21:40:193
898799201,10cyclictest3896-21/usr/sbin/munin20:10:253
898799201,10cyclictest25125-21grep23:30:203
898799200,9cyclictest20271-21strings20:45:193
898799200,8cyclictest0-21swapper/320:09:353
897799209,3cyclictest18724-21latency_hist22:05:021
897799204,6cyclictest86850irq/129-enp1s0-19:17:381
897799203,4cyclictest15670-21ls19:20:261
897799202,3cyclictest23161-21taskset20:51:281
897799202,15cyclictest11377-21grep23:00:181
897799201,15cyclictest0-21swapper/122:32:441
897799201,15cyclictest0-21swapper/120:01:471
897799201,11cyclictest5347-21sed20:10:271
897799200,19cyclictest0-21swapper/121:44:481
897799200,15cyclictest0-21swapper/123:45:461
897799200,14cyclictest86750irq/128-enp1s0-22:21:381
897199209,3cyclictest4185-21/usr/sbin/munin22:45:200
897199209,3cyclictest19721-21ntp_states19:30:220
897199203,5cyclictest2796350kworker/u8:100:00:120
897199201,5cyclictest3503-1kworker/u9:223:35:510
897199201,4cyclictest0-21swapper/023:20:350
8971992014,4cyclictest23761-21munin-run23:30:010
897199201,11cyclictest12995-21/usr/sbin/munin23:05:140
897199201,10cyclictest0-21swapper/020:06:350
898799195,5cyclictest0-21swapper/323:07:093
898799194,4cyclictest0-21swapper/321:11:333
898799194,2cyclictest86550irq/126-enp1s0-00:20:213
8987991915,3cyclictest0-21swapper/319:58:593
8987991914,3cyclictest24628-21grep19:40:253
8987991913,3cyclictest16836-21head00:25:193
898799191,11cyclictest0-21swapper/322:12:123
898799191,10cyclictest6091-21/usr/sbin/munin20:15:243
898799190,11cyclictest0-21swapper/321:50:433
898799190,10cyclictest0-21swapper/320:25:113
897799191,4cyclictest0-21swapper/122:11:521
897799191,2cyclictest0-21swapper/120:25:521
8977991912,3cyclictest2128-21tr22:40:141
897799191,17cyclictest0-21swapper/121:50:381
897799191,10cyclictest0-21swapper/120:05:261
897799190,2cyclictest0-21swapper/120:35:231
897799190,14cyclictest0-21swapper/120:45:061
897199198,3cyclictest1458-21sshd22:36:350
897199196,4cyclictest0-21swapper/021:35:230
897199191,8cyclictest0-21swapper/023:44:570
897199191,6cyclictest0-21swapper/000:37:580
8971991914,3cyclictest28381-21gltestperf22:25:140
897199191,16cyclictest18541-21kworker/0:222:13:310
897199191,12cyclictest13347-21sshd21:45:360
897199191,12cyclictest13347-21sshd21:45:360
897199191,10cyclictest19281-21/usr/sbin/munin20:45:190
897199191,10cyclictest17086-21/usr/sbin/munin20:40:140
897199191,10cyclictest17086-21/usr/sbin/munin20:40:140
897199191,10cyclictest15845-21/usr/sbin/munin00:25:200
898799188,2cyclictest0-21swapper/320:54:153
898799182,8cyclictest0-21swapper/319:26:493
898799182,6cyclictest0-21swapper/323:00:233
898799181,7cyclictest0-21swapper/323:52:553
898799180,3cyclictest0-21swapper/320:37:133
897199189,3cyclictest86750irq/128-enp1s0-20:12:280
897199181,7cyclictest0-21swapper/022:43:210
897199181,4cyclictest0-21swapper/022:06:030
8971991812,3cyclictest24597-21sendmail_mailst22:15:230
897199181,14cyclictest15305-21grep20:35:150
190672180,2sleep00-21swapper/023:16:230
898799171,4cyclictest18657-21/usr/sbin/munin19:30:133
897199176,3cyclictest86750irq/128-enp1s0-00:30:180
897199176,3cyclictest14634-21snmpd20:25:330
897199173,4cyclictest0-21swapper/023:56:000
897199173,4cyclictest0-21swapper/023:56:000
897199172,5cyclictest15613-21ntp_states21:55:190
897199172,2cyclictest28495-21awk21:10:010
897199171,7cyclictest9173-21/usr/sbin/munin19:10:250
897199171,5cyclictest0-21swapper/021:50:090
897199171,4cyclictest16747-21/usr/sbin/munin22:00:170
897199171,15cyclictest0-21swapper/019:42:060
897199171,15cyclictest0-21swapper/000:08:370
8971991711,3cyclictest426-21idleruntime-cro21:20:010
8971991711,3cyclictest30586-21/usr/sbin/munin23:45:180
8971991711,3cyclictest13414-21latency_hist00:20:020
8971991711,3cyclictest11419-21gltestperf23:00:170
254462171,3sleep00-21swapper/020:57:330
231872170,2sleep00-21swapper/020:51:470
897199165,6cyclictest0-21swapper/020:00:100
897199165,3cyclictest16483-21cpu19:25:100
897199162,10cyclictest11603-21grep19:15:030
897199161,4cyclictest1059-21df21:20:120
8971991611,3cyclictest22172-21cat19:35:250
8971991610,3cyclictest15537-21sh23:10:150
8971991510,3cyclictest6820-21if_enp1s020:15:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional