You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 19:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Dec 03, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
154822238163,25sleep10-21swapper/107:06:031
157242236161,25sleep30-21swapper/307:09:113
138302205167,25sleep00-21swapper/007:05:080
138322201164,24sleep20-21swapper/207:05:112
1589599106103,2cyclictest0-21swapper/211:26:062
84632610,5sleep31590199cyclictest08:05:023
116942600,4sleep230-21rcuc/208:10:192
181492590,2sleep00-21swapper/007:15:010
220832570,5sleep01588499cyclictest12:15:160
204612560,4sleep21589599cyclictest08:30:182
174112540,1sleep20-21swapper/207:10:252
238472470,2sleep20-21swapper/208:40:012
156622460,1sleep10-21swapper/108:20:171
1588499381,23cyclictest27565-21kworker/0:012:27:460
1588499377,3cyclictest86850irq/129-enp1s0-09:20:270
1588499377,3cyclictest2686-21/usr/sbin/munin09:05:190
15884993712,12cyclictest17232-21awk08:25:000
15884993616,17cyclictest27291-21chrt08:45:200
15884993611,22cyclictest4291-21df11:35:120
1588499357,3cyclictest22480-21/usr/sbin/munin09:50:150
1588499351,30cyclictest2670-21cut07:50:150
1588499351,10cyclictest18711-21/usr/sbin/munin10:55:140
1588499351,10cyclictest0-21swapper/011:59:440
15884993510,2cyclictest25912-21df07:30:140
1588499347,4cyclictest24398-21/usr/sbin/munin11:05:250
1588499347,13cyclictest9070-21perf09:20:010
1588499342,4cyclictest19362-21/usr/sbin/munin12:10:190
15884993412,19cyclictest1098-21/usr/sbin/munin10:15:210
1588499341,10cyclictest0-21swapper/010:50:000
15884993410,21cyclictest11171-21tail11:50:150
281752330,3sleep31590199cyclictest12:30:113
1588499339,2cyclictest9915-21/usr/sbin/munin10:35:260
1588499339,2cyclictest9915-21/usr/sbin/munin10:35:260
1588499337,11cyclictest0-21swapper/009:05:010
1588499336,12cyclictest30430-21basename08:55:000
1588499331,6cyclictest0-21swapper/010:13:320
1589199321,24cyclictest0-21swapper/112:30:141
1588499327,3cyclictest18085-21/usr/sbin/munin09:40:240
1588499326,4cyclictest848-21sssd_nss11:25:150
1588499326,4cyclictest0-21swapper/012:35:170
1588499325,8cyclictest4848-21gltestperf07:55:150
1588499322,19cyclictest21411-21kworker/0:108:44:490
1588499321,4cyclictest29727-21/usr/sbin/munin11:20:210
15884993213,16cyclictest14758-21seq08:16:540
15884993211,18cyclictest836-21runrttasks10:32:580
1588499321,10cyclictest14634-21snmpd11:33:390
15884993210,5cyclictest0-21swapper/010:00:260
15884993210,3cyclictest17852-21hwlatdetect12:05:220
1588499319,4cyclictest12059-21fgrep08:10:230
1588499319,3cyclictest17146-21gltestperf10:50:160
1588499316,12cyclictest20951-21munin-run07:20:010
1588499312,5cyclictest29090-21/usr/sbin/munin10:05:280
15884993111,17cyclictest21807-21sed09:45:260
1588499310,9cyclictest0-21swapper/011:10:180
1590199301,13cyclictest0-21swapper/307:24:113
1589199301,23cyclictest0-21swapper/109:16:131
1588499308,4cyclictest848-21sssd_nss09:10:210
1588499307,3cyclictest32263-21grep07:45:130
1588499304,7cyclictest0-21swapper/011:40:120
1588499304,7cyclictest0-21swapper/011:40:120
1588499304,5cyclictest848-21sssd_nss09:25:150
1588499304,3cyclictest12050irq/123-ahci[0012:35:010
1588499301,11cyclictest24397-21netstat07:25:210
1588499301,11cyclictest24397-21netstat07:25:210
1588499301,10cyclictest14634-21snmpd10:28:260
15884993010,17cyclictest8371-21idleruntime-cro08:05:000
1589199291,3cyclictest12689-21cut11:55:021
1589199291,22cyclictest0-21swapper/109:48:051
1588499298,3cyclictest25236-21grep12:20:280
1588499297,3cyclictest28787-21sensors11:15:250
1588499297,3cyclictest17446-21/usr/sbin/munin08:25:120
1588499297,3cyclictest17446-21/usr/sbin/munin08:25:120
1588499296,3cyclictest31249-21grep08:55:160
1588499294,3cyclictest88050irq/133-enp2s0-10:40:280
1588499294,3cyclictest86550irq/126-enp1s0-08:05:150
1588499291,9cyclictest15056-21cut12:00:120
1588499291,4cyclictest20915-21/usr/sbin/munin11:00:250
1588499291,3cyclictest29971-21/usr/sbin/munin07:40:190
15884992912,14cyclictest9613-21mailstats11:45:280
1588499291,13cyclictest23820-21munin-run08:40:000
15884992910,3cyclictest28237-21cut07:35:180
1590199285,14cyclictest780-21rngd08:23:363
1590199281,11cyclictest0-21swapper/308:14:483
1590199280,19cyclictest0-21swapper/307:47:443
1589199281,22cyclictest0-21swapper/112:16:511
1588499286,4cyclictest4557-21sendmail_mailst10:20:220
1588499286,4cyclictest24682-21/usr/sbin/munin09:55:190
1588499286,4cyclictest16923-21ntpq09:35:230
1588499283,7cyclictest14212-21fschecks_time09:30:160
1588499283,7cyclictest14212-21fschecks_time09:30:160
1590199271,11cyclictest0-21swapper/307:14:563
1590199270,12cyclictest0-21swapper/310:18:403
15895992724,2cyclictest0-21swapper/208:43:142
1589199271,17cyclictest0-21swapper/111:56:511
15891992711,4cyclictest2339-21perl11:30:161
1588499279,4cyclictest20308-21sed08:30:150
1588499271,8cyclictest22670-21unixbench_multi07:20:260
1590199260,20cyclictest0-21swapper/312:38:153
1589199261,22cyclictest581-21systemd-journal08:00:011
1589199261,16cyclictest0-21swapper/108:42:151
1589199261,11cyclictest0-21swapper/110:47:361
277132251,3sleep127839-21users08:45:271
1590199259,8cyclictest0-21swapper/309:15:043
1590199257,9cyclictest0-21swapper/308:05:383
1590199257,3cyclictest0-21swapper/308:53:523
1590199251,12cyclictest0-21swapper/308:34:083
1590199251,10cyclictest0-21swapper/311:25:443
1590199250,13cyclictest0-21swapper/307:54:503
1590199250,11cyclictest0-21swapper/310:35:143
1590199250,11cyclictest0-21swapper/310:35:143
1590199250,10cyclictest6782sleep310:13:323
15891992513,3cyclictest0-21swapper/108:05:261
1589199251,22cyclictest0-21swapper/107:30:211
1589199251,21cyclictest25734-21munin-run12:25:021
1589199251,16cyclictest0-21swapper/110:04:201
1589199251,16cyclictest0-21swapper/109:52:031
1589199251,16cyclictest0-21swapper/109:27:511
1589199251,15cyclictest0-21swapper/109:10:081
1589199251,15cyclictest0-21swapper/109:01:311
1589199251,11cyclictest6577-21fschecks_count11:40:161
1589199251,11cyclictest6577-21fschecks_count11:40:161
1589199251,10cyclictest10265-21sed09:20:201
1590199241,14cyclictest0-21swapper/309:27:123
1590199241,12cyclictest0-21swapper/311:51:443
1590199241,12cyclictest0-21swapper/311:30:243
1590199241,10cyclictest0-21swapper/308:39:203
1589599241,4cyclictest0-21swapper/211:55:022
1589599241,3cyclictest0-21swapper/212:39:012
1589199249,4cyclictest0-21swapper/112:00:231
1589199242,8cyclictest0-21swapper/108:29:401
1589199242,8cyclictest0-21swapper/108:29:401
1589199242,5cyclictest3660-21df_inode10:20:131
15891992413,3cyclictest0-21swapper/108:39:231
1589199241,21cyclictest0-21swapper/111:36:481
1589199241,17cyclictest0-21swapper/108:30:301
1589199241,15cyclictest0-21swapper/112:27:441
1589199241,15cyclictest0-21swapper/112:11:001
1589199241,15cyclictest0-21swapper/111:46:371
1589199241,15cyclictest0-21swapper/110:11:231
1589199241,15cyclictest0-21swapper/110:08:371
1589199241,15cyclictest0-21swapper/108:18:491
1589199241,15cyclictest0-21swapper/108:12:361
1589199241,15cyclictest0-21swapper/108:01:591
1589199241,11cyclictest7484-21latency_hist10:30:011
1589199241,11cyclictest31935-21latency_hist07:45:021
133252240,4sleep232-21ksoftirqd/210:40:242
1590199237,3cyclictest0-21swapper/310:04:583
1590199236,3cyclictest0-21swapper/307:15:243
1590199234,3cyclictest0-21swapper/308:46:003
1590199232,10cyclictest780-21rngd10:47:303
1590199231,9cyclictest0-21swapper/311:23:283
15901992313,4cyclictest780-21rngd10:57:543
1590199231,11cyclictest23979-21grep09:50:263
1590199231,11cyclictest0-21swapper/309:45:043
1590199231,11cyclictest0-21swapper/309:44:443
1590199231,10cyclictest0-21swapper/308:55:363
1589599235,6cyclictest28116-21sh07:35:162
15895992317,3cyclictest8117-21ntpq09:15:202
15895992314,6cyclictest29247-21/usr/sbin/munin12:30:222
1589199238,4cyclictest29877-21cat07:40:021
1589199238,3cyclictest25640-21missed_timers09:55:221
1589199237,5cyclictest1448-21diskstats10:15:131
1589199237,3cyclictest31280-21gltestperf08:55:171
1589199232,9cyclictest17860-21munin-run09:40:011
15891992314,2cyclictest0-21swapper/111:12:341
15891992313,7cyclictest28476-21missed_timers11:15:221
1589199231,20cyclictest15471-21kworker/u8:010:35:491
1589199231,20cyclictest15471-21kworker/u8:010:35:491
1589199231,20cyclictest0-21swapper/110:53:241
1589199231,19cyclictest8981-21smart_sda10:30:251
1589199231,10cyclictest21264-21runrttasks11:00:151
1589199231,10cyclictest0-21swapper/112:09:021
1590199229,4cyclictest0-21swapper/309:11:203
1590199228,7cyclictest0-21swapper/311:08:583
1590199228,3cyclictest780-21rngd08:16:163
1590199228,3cyclictest0-21swapper/312:00:323
1590199221,8cyclictest0-21swapper/309:08:583
1590199221,11cyclictest0-21swapper/311:42:343
1590199221,11cyclictest0-21swapper/311:42:343
1590199221,10cyclictest0-21swapper/309:22:483
1590199221,10cyclictest0-21swapper/307:31:043
1589599223,10cyclictest23121-21/usr/sbin/munin11:05:242
1589599221,6cyclictest21510-21grep11:00:172
15895992214,5cyclictest18407-21/usr/sbin/munin12:05:282
1589199226,3cyclictest17670-21users07:10:261
1589199221,19cyclictest0-21swapper/111:05:411
1589199221,19cyclictest0-21swapper/109:05:161
1589199221,19cyclictest0-21swapper/107:28:201
1589199221,19cyclictest0-21swapper/107:28:201
1589199221,19cyclictest0-21swapper/107:20:451
1590199218,3cyclictest0-21swapper/311:37:283
1590199217,3cyclictest0-21swapper/308:44:243
1590199216,9cyclictest25442-40dnf12:21:343
1590199216,7cyclictest0-21swapper/308:26:263
1590199216,7cyclictest0-21swapper/308:26:263
1590199215,3cyclictest0-21swapper/307:55:443
1590199213,3cyclictest0-21swapper/310:20:343
1590199211,9cyclictest0-21swapper/310:29:563
1590199211,9cyclictest0-21swapper/307:42:523
1590199211,8cyclictest0-21swapper/312:14:103
1590199211,7cyclictest0-21swapper/311:14:163
15901992110,3cyclictest29496-21awk11:20:023
1590199210,7cyclictest0-21swapper/311:56:363
1590199210,11cyclictest0-21swapper/312:16:243
1589599219,3cyclictest836-21runrttasks11:37:182
1589599213,6cyclictest4893-21/usr/sbin/munin09:10:242
15895992111,8cyclictest24190-21kworker/u8:207:45:062
1589599211,12cyclictest0-21swapper/211:55:152
1589599211,12cyclictest0-21swapper/211:32:192
1589599211,11cyclictest6503-21diskstats11:40:152
1589599211,11cyclictest6503-21diskstats11:40:152
1589599211,11cyclictest26273-21users09:55:292
1589199212,16cyclictest29362-21memory08:50:191
1589199211,19cyclictest0-21swapper/111:21:271
1589199211,19cyclictest0-21swapper/110:42:351
1589199211,19cyclictest0-21swapper/107:46:001
1589199211,18cyclictest0-21swapper/111:25:211
15891992111,2cyclictest0-21swapper/109:40:441
1589199210,14cyclictest0-21swapper/109:33:311
1589199210,14cyclictest0-21swapper/109:33:311
284532200,1sleep30-21swapper/307:35:203
1590199208,3cyclictest0-21swapper/310:30:083
1590199207,3cyclictest0-21swapper/310:51:563
1590199204,3cyclictest0-21swapper/311:04:083
1590199203,3cyclictest0-21swapper/309:03:363
1590199202,3cyclictest0-21swapper/307:25:563
1590199202,3cyclictest0-21swapper/307:25:563
1590199201,7cyclictest0-21swapper/310:05:083
15901992010,2cyclictest86850irq/129-enp1s0-09:30:143
15901992010,2cyclictest86850irq/129-enp1s0-09:30:143
1590199200,13cyclictest0-21swapper/309:39:523
1589599202,5cyclictest0-21swapper/207:29:072
1589599202,5cyclictest0-21swapper/207:29:072
1589599202,3cyclictest0-21swapper/209:08:182
1589599201,6cyclictest19964-21which10:55:232
1589599201,6cyclictest11503-21users10:35:282
1589599201,6cyclictest11503-21users10:35:282
1589599201,5cyclictest0-21swapper/209:45:132
1589599201,13cyclictest0-21swapper/211:17:262
1589599201,13cyclictest0-21swapper/210:29:032
1589599201,11cyclictest15836-21cat12:00:182
1589599201,11cyclictest1-21systemd09:20:252
1589599201,11cyclictest0-21swapper/212:19:212
1589599201,11cyclictest0-21swapper/210:20:412
1589599201,10cyclictest14754-21ntp_states09:30:242
1589599201,10cyclictest14754-21ntp_states09:30:242
1589599201,10cyclictest0-21swapper/211:45:302
1589199208,2cyclictest2312-21expr07:50:131
1589199201,18cyclictest0-21swapper/110:55:081
1590199196,2cyclictest0-21swapper/309:57:523
1590199195,7cyclictest0-21swapper/310:44:123
1590199192,3cyclictest0-21swapper/311:48:563
1589599195,6cyclictest0-21swapper/209:39:442
1589599195,3cyclictest12050irq/123-ahci[0008:05:142
1589599193,6cyclictest30-21rcuc/212:20:282
1589599193,3cyclictest16362-21pmu-power08:20:252
1589599191,8cyclictest0-21swapper/207:20:132
1589599191,7cyclictest0-21swapper/208:57:402
1589599191,6cyclictest30217-21pmu-power10:05:242
1589599191,6cyclictest28796-21diskstats08:50:152
1589599191,6cyclictest20894-40dnf07:19:322
1589599191,6cyclictest19696-21df12:10:152
1589599191,6cyclictest15050-21cat10:45:172
1589599191,6cyclictest0-21swapper/212:27:392
1589599191,12cyclictest0-21swapper/211:12:152
1589599191,11cyclictest0-21swapper/211:20:032
1589599191,11cyclictest0-21swapper/209:51:312
1589599191,10cyclictest31288-21/usr/sbin/munin10:10:172
1589599191,10cyclictest14634-21snmpd10:33:282
1589599191,10cyclictest0-21swapper/210:16:262
1589199198,3cyclictest0-21swapper/112:36:391
15891991915,3cyclictest0-21swapper/107:16:471
1590199185,3cyclictest0-21swapper/312:27:563
1590199180,9cyclictest0-21swapper/312:07:563
1589599181,9cyclictest4661-21diskmemload09:42:492
1589599181,5cyclictest6750-21diskstats08:00:172
1589599181,5cyclictest11827-21diskstats09:25:142
1589599181,4cyclictest4534-21df07:55:142
1589599181,4cyclictest18256-21irqstats08:25:182
1589599181,4cyclictest18256-21irqstats08:25:182
1589599181,10cyclictest0-21swapper/210:01:122
1589599173,3cyclictest12050irq/123-ahci[0008:50:022
1589599171,6cyclictest0-21swapper/208:15:102
1589599171,4cyclictest30884-21memory07:40:192
1589599171,4cyclictest16861-21df_inode10:50:142
15895991712,3cyclictest374-21/usr/sbin/munin09:00:162
1589599161,4cyclictest25599-21cpu07:30:132
15895991611,3cyclictest2139-21expr07:50:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional