You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 16:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Wed Nov 26, 2025 12:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
166472240163,25sleep00-21swapper/007:06:030
168732235160,24sleep30-21swapper/307:09:003
166442233164,56sleep20-21swapper/207:06:012
168332231157,25sleep10-21swapper/107:08:291
535021790,4sleep31705999cyclictest10:20:203
1162121550,2sleep01703899cyclictest09:20:230
51321430,4sleep21705299cyclictest11:25:022
1903921050,4sleep31705999cyclictest12:05:163
1705299103100,2cyclictest0-21swapper/210:05:542
292612670,4sleep339-21rcuc/307:35:133
292612670,4sleep339-21rcuc/307:35:133
22352650,4sleep32239-21unixbench_singl11:25:263
211192590,4sleep339-21rcuc/307:15:213
17052995249,2cyclictest0-21swapper/207:22:402
326302350,4sleep21705299cyclictest07:40:262
1703899329,11cyclictest18985-21perf09:40:010
1703899311,15cyclictest1052-21munin-run12:40:010
1703899306,11cyclictest22739-21munin-run08:35:000
238392280,3sleep223841-21proc_pri12:15:242
1704699275,10cyclictest780-21rngd09:05:161
1704699272,15cyclictest0-21swapper/111:05:331
17046992720,3cyclictest4212-21munin-run10:20:011
1703899271,13cyclictest2021-21idleruntime-cro10:15:010
1703899270,12cyclictest23375-21awk09:50:010
1703899270,12cyclictest23375-21awk09:50:010
1704699263,11cyclictest0-21swapper/111:23:001
1704699261,17cyclictest0-21swapper/108:50:231
1704699251,15cyclictest4850-21fschecks_count10:20:161
205382241,3sleep020810-21/usr/sbin/munin12:10:130
1705999242,3cyclictest12050irq/123-ahci[0011:00:003
1704699242,16cyclictest0-21swapper/108:26:071
1704699242,15cyclictest21-21rcuc/112:30:151
1704699241,20cyclictest20161-21df10:55:131
1704699241,17cyclictest0-21swapper/112:13:031
1704699241,16cyclictest0-21swapper/109:36:141
1704699241,15cyclictest0-21swapper/109:01:491
1703899249,3cyclictest28632-21/usr/sbin/munin11:15:230
1703899242,11cyclictest941-21/usr/sbin/munin07:45:170
17038992412,7cyclictest848-21sssd_nss08:04:240
145852240,5sleep232-21ksoftirqd/209:30:012
1705999237,7cyclictest0-21swapper/309:59:283
1705999237,6cyclictest0-21swapper/312:10:293
1704699234,9cyclictest23-21ksoftirqd/111:15:011
1704699232,15cyclictest0-21swapper/111:16:281
17046992315,4cyclictest22901-21grep09:45:281
17046992315,4cyclictest22901-21grep09:45:281
1704699231,4cyclictest14634-21snmpd11:59:061
1704699231,4cyclictest14634-21snmpd11:59:061
17046992314,3cyclictest0-21swapper/107:19:311
1704699231,20cyclictest0-21swapper/110:11:331
1704699231,17cyclictest0-21swapper/107:23:091
1704699231,12cyclictest0-21swapper/112:29:171
1703899232,10cyclictest0-21swapper/011:00:260
1705999228,8cyclictest0-21swapper/309:44:493
1705999227,6cyclictest0-21swapper/311:08:233
1705999226,5cyclictest0-21swapper/311:43:393
1705999221,6cyclictest27670-21gltestperf12:25:153
1705999221,5cyclictest0-21swapper/309:20:143
1705299226,7cyclictest32-21ksoftirqd/212:30:152
17046992218,3cyclictest0-21swapper/109:12:541
1704699221,18cyclictest22713-21cpu12:15:131
1704699221,18cyclictest0-21swapper/110:05:211
1704699221,16cyclictest0-21swapper/111:42:121
1704699221,16cyclictest0-21swapper/108:46:231
1704699221,15cyclictest0-21swapper/109:28:491
1704699221,15cyclictest0-21swapper/108:30:091
1704699221,10cyclictest0-21swapper/108:10:271
17046992210,6cyclictest3983-21kworker/u8:010:46:221
1704699220,17cyclictest0-21swapper/109:30:191
1703899221,11cyclictest14289-21ntp_states10:40:230
237062211,2sleep221624-21kworker/u8:011:01:232
1705999218,3cyclictest5767-21diskmemload12:04:283
1705999218,3cyclictest0-21swapper/307:25:363
1705999217,7cyclictest0-21swapper/310:54:393
1705999217,7cyclictest0-21swapper/310:54:393
1705999217,5cyclictest0-21swapper/311:10:213
1705999216,6cyclictest0-21swapper/310:37:533
1705999216,4cyclictest0-21swapper/312:36:283
1705999211,5cyclictest0-21swapper/311:38:123
1705999211,12cyclictest0-21swapper/310:10:143
1705999211,11cyclictest2582-21diskstats10:15:143
17059992110,8cyclictest8573-21sshd10:29:493
17059992110,3cyclictest23129-21pmu-power11:00:223
1705999210,7cyclictest0-21swapper/310:40:053
1705999210,6cyclictest0-21swapper/309:10:483
1705299211,6cyclictest0-21swapper/207:14:282
1704699215,10cyclictest0-21swapper/112:06:591
1704699212,3cyclictest0-21swapper/109:53:071
1704699212,17cyclictest0-21swapper/108:37:381
17046992117,3cyclictest0-21swapper/108:41:011
17046992117,3cyclictest0-21swapper/108:41:011
17046992117,3cyclictest0-21swapper/107:28:011
1704699211,6cyclictest0-21swapper/111:27:401
17046992116,3cyclictest26409-21grep09:55:161
17046992113,2cyclictest0-21swapper/107:45:131
1704699211,18cyclictest0-21swapper/112:01:281
1704699211,15cyclictest0-21swapper/112:20:541
1704699211,15cyclictest0-21swapper/111:39:591
1704699211,15cyclictest0-21swapper/111:31:141
1704699211,15cyclictest0-21swapper/108:58:011
1704699211,15cyclictest0-21swapper/108:16:161
1704699211,15cyclictest0-21swapper/108:09:561
1704699211,15cyclictest0-21swapper/107:53:051
1704699211,15cyclictest0-21swapper/107:43:301
1704699211,15cyclictest0-21swapper/107:31:351
1704699211,13cyclictest0-21swapper/109:42:541
1704699211,12cyclictest19162-21unixbench_singl10:50:261
1704699211,12cyclictest19162-21unixbench_singl10:50:261
1704699211,11cyclictest7904-21diskstats08:00:151
1704699211,10cyclictest0-21swapper/112:39:521
1704699211,10cyclictest0-21swapper/110:30:331
1704699210,15cyclictest0-21swapper/107:55:071
1703899211,11cyclictest30830-21/usr/sbin/munin11:20:120
1703899211,11cyclictest15422-21/usr/sbin/munin10:45:230
268232200,4sleep339-21rcuc/312:24:553
1705999209,3cyclictest10094-21smart_sda10:30:263
1705999208,5cyclictest0-21swapper/309:45:263
1705999208,5cyclictest0-21swapper/309:45:263
1705999207,4cyclictest0-21swapper/309:19:473
1705999206,4cyclictest0-21swapper/309:51:553
1705999206,3cyclictest12050irq/123-ahci[0009:00:013
1705999205,3cyclictest86850irq/129-enp1s0-08:50:263
1705999204,3cyclictest4037-21wc11:30:213
1705999201,8cyclictest0-21swapper/309:35:233
1705999201,7cyclictest0-21swapper/310:45:123
1705999201,6cyclictest30143-21sed11:15:253
17059992015,3cyclictest11737-21basename08:10:013
1705999201,10cyclictest0-21swapper/311:57:303
1705999201,10cyclictest0-21swapper/311:57:303
1705299206,4cyclictest18920-21sshd09:39:382
1705299202,6cyclictest31794-21df12:35:132
1705299201,9cyclictest0-21swapper/209:43:202
1705299201,7cyclictest0-21swapper/210:50:232
1705299201,7cyclictest0-21swapper/210:50:232
1705299201,5cyclictest0-21swapper/210:10:192
1704699208,9cyclictest28627-21gltestperf10:00:151
1704699206,13cyclictest7587-21kworker/u8:109:19:201
1704699201,9cyclictest0-21swapper/110:42:561
17046992016,3cyclictest0-21swapper/109:20:071
1704699201,18cyclictest0-21swapper/108:22:051
1704699201,17cyclictest0-21swapper/110:37:261
1704699201,14cyclictest0-21swapper/107:36:341
1704699201,14cyclictest0-21swapper/107:36:341
1704699201,11cyclictest13709-21cut11:55:011
1704699201,11cyclictest0-21swapper/111:02:311
1704699201,10cyclictest0-21swapper/110:26:471
1704699200,15cyclictest0-21swapper/111:48:081
1703899205,7cyclictest5767-21diskmemload12:25:130
1703899205,5cyclictest24224-21kworker/u8:111:05:340
1703899201,9cyclictest842-21lldpd11:35:210
1703899201,9cyclictest0-21swapper/009:59:460
1703899201,7cyclictest0-21swapper/012:22:380
1703899201,10cyclictest3384-21sshd09:02:500
1703899201,10cyclictest19196-21/usr/sbin/munin09:40:170
1703899201,10cyclictest14634-21snmpd07:13:120
204202191,2sleep00-21swapper/008:28:460
198152191,2sleep20-21swapper/210:55:042
1705999197,4cyclictest0-21swapper/310:08:543
1705999197,2cyclictest86550irq/126-enp1s0-09:26:103
1705999196,4cyclictest0-21swapper/312:16:333
1705999196,3cyclictest784-21NetworkManager08:49:433
1705999191,6cyclictest2521-21head07:45:273
1705999191,6cyclictest0-21swapper/312:31:323
1705999191,10cyclictest7556-21/usr/sbin/munin08:00:213
1705999191,10cyclictest27986-21/usr/sbin/munin10:00:173
1705999191,10cyclictest22958-21/usr/sbin/munin08:35:153
1705299199,3cyclictest86850irq/129-enp1s0-07:45:132
1705299194,5cyclictest8938-21cut10:30:142
1705299194,10cyclictest25841-40dnf11:05:332
1705299192,2cyclictest86750irq/128-enp1s0-11:32:572
1705299192,14cyclictest21711-21find12:10:232
1705299191,7cyclictest0-21swapper/209:20:142
1705299191,6cyclictest0-21swapper/207:35:122
1705299191,6cyclictest0-21swapper/207:35:122
1705299191,4cyclictest30-21rcuc/208:01:252
17052991913,3cyclictest18612-21sh12:05:142
17046991912,2cyclictest0-21swapper/107:11:571
1703899199,3cyclictest6144-21sshd10:21:490
1703899198,2cyclictest0-21swapper/010:55:490
1703899195,3cyclictest18795-21proc_pri10:50:230
1703899195,3cyclictest18795-21proc_pri10:50:230
1703899193,7cyclictest26436-21/usr/sbin/munin11:10:260
1703899192,8cyclictest14159-21/usr/sbin/munin08:15:200
1703899191,9cyclictest0-21swapper/011:26:490
1703899191,9cyclictest0-21swapper/010:15:490
1703899191,8cyclictest0-21swapper/008:09:300
1703899191,4cyclictest28588-21grep10:00:150
17038991912,4cyclictest27149-21/usr/sbin/munin07:30:140
1703899191,16cyclictest0-21swapper/009:50:230
1703899191,11cyclictest0-21swapper/012:32:370
77812181,2sleep20-21swapper/209:12:482
5802181,2sleep30-21swapper/307:44:253
1705999185,4cyclictest24321-21perf07:25:013
1705999182,8cyclictest17250-21/usr/sbin/munin07:10:193
1705999181,6cyclictest0-21swapper/309:33:083
1705999181,3cyclictest0-21swapper/311:20:183
17059991813,3cyclictest20764-21/usr/sbin/munin08:30:283
17059991813,3cyclictest12018-21entropy11:50:153
1705999181,14cyclictest14159-21/usr/sbin/munin08:15:133
1705999181,10cyclictest25163-21/usr/sbin/munin08:40:283
1705999181,10cyclictest25163-21/usr/sbin/munin08:40:283
1705999181,10cyclictest0-21swapper/311:46:403
1705299187,3cyclictest86550irq/126-enp1s0-09:45:142
1705299187,3cyclictest86550irq/126-enp1s0-09:45:142
1705299182,7cyclictest28711-21cpuspeed_turbos11:15:132
1705299181,8cyclictest28167-21timerwakeupswit07:30:252
1705299181,8cyclictest0-21swapper/209:30:142
1705299181,7cyclictest2431-21kernelversion09:00:172
1705299181,7cyclictest0-21swapper/212:20:202
1705299181,6cyclictest0-21swapper/210:25:032
1705299181,3cyclictest0-21swapper/210:41:452
17052991813,3cyclictest0-21swapper/211:40:152
17052991812,3cyclictest836-21runrttasks08:35:222
17052991812,3cyclictest4434-21/usr/sbin/munin10:20:462
17052991812,3cyclictest26375-21processes08:40:262
17052991812,3cyclictest26375-21processes08:40:262
17052991812,3cyclictest24907-21cat09:50:242
1705299181,10cyclictest0-21swapper/211:29:062
1703899188,2cyclictest86550irq/126-enp1s0-12:15:270
1703899186,3cyclictest86750irq/128-enp1s0-09:25:160
1703899185,5cyclictest792-21avahi-daemon08:36:010
1703899181,8cyclictest5357-21/usr/sbin/munin07:55:190
1703899181,7cyclictest0-21swapper/010:31:310
1703899181,6cyclictest0-21swapper/009:32:400
1703899181,5cyclictest18274-21/usr/sbin/munin12:05:180
1703899181,5cyclictest0-21swapper/007:15:250
1703899181,4cyclictest31703-21grep10:05:260
17038991812,3cyclictest8531-21seq11:40:250
17038991812,3cyclictest25146-21date08:40:020
17038991812,3cyclictest25146-21date08:40:010
17038991812,3cyclictest16477-21grep12:00:150
1703899180,15cyclictest26517-21awk07:30:000
132722181,2sleep20-21swapper/211:50:362
1705999173,5cyclictest18679-21tr08:25:123
1705999171,8cyclictest5762-21idleruntime-cro09:10:013
1705999171,5cyclictest11963-21/usr/sbin/munin08:10:143
1705999171,4cyclictest26733-21/usr/sbin/munin07:30:133
1705999171,3cyclictest16360-21/usr/sbin/munin08:20:243
17059991712,3cyclictest6867-21sed07:55:263
1705299173,5cyclictest32-21ksoftirqd/211:49:292
1705299172,5cyclictest16112-21sed10:45:172
1705299171,7cyclictest0-21swapper/208:33:022
1705299171,6cyclictest29323-21latency_hist08:50:012
1705299171,6cyclictest20142-21/usr/sbin/munin07:15:142
1705299171,5cyclictest0-21swapper/210:00:172
1705299171,4cyclictest0-21swapper/212:02:082
1705299171,4cyclictest0-21swapper/211:35:092
1705299171,3cyclictest0-21swapper/208:20:592
17052991712,3cyclictest3957-21cat07:50:172
17052991712,3cyclictest30347-21if_err_enp2s008:50:172
17052991712,3cyclictest14003-21tr11:55:132
17052991712,3cyclictest14003-21tr11:55:132
1705299171,14cyclictest0-21swapper/211:10:052
17052991711,3cyclictest8431-21expr09:15:142
17052991711,3cyclictest3143-21load10:15:182
17052991711,3cyclictest25600-21ntpq07:25:192
1703899176,3cyclictest86750irq/128-enp1s0-09:14:350
1703899172,6cyclictest19356-21munin-node07:35:270
1703899172,6cyclictest19356-21munin-node07:35:270
1703899172,12cyclictest4211-21cut09:05:140
1703899171,6cyclictest0-21swapper/011:45:290
1703899171,5cyclictest18443-21sort08:25:000
1703899171,3cyclictest15488-21sshd11:55:490
1703899171,3cyclictest15488-21sshd11:55:490
1703899171,2cyclictest32353-21tail08:55:160
17038991712,3cyclictest23256-21missed_timers07:20:210
17038991711,3cyclictest6971-21sh10:25:140
17038991711,3cyclictest3345-21cstates07:50:110
17038991711,3cyclictest3222-21entropy11:30:140
17038991711,3cyclictest31141-21users08:50:260
1703899170,15cyclictest0-21swapper/007:40:070
127732171,2sleep00-21swapper/010:37:200
1705999161,4cyclictest1585-21/usr/sbin/munin09:00:153
17059991610,3cyclictest3163-21/usr/sbin/munin07:50:163
1705299169,4cyclictest1-21systemd08:05:492
1705299161,9cyclictest0-21swapper/209:05:162
1705299161,6cyclictest15075-21memory08:15:192
17052991612,2cyclictest13032-21ntp_states08:10:222
17052991611,3cyclictest28655-21users12:25:262
17052991611,3cyclictest18843-21ls08:25:132
17052991611,3cyclictest11382-21entropy10:35:152
1703899161,6cyclictest8198-21/usr/sbin/munin09:15:250
1703899161,14cyclictest0-21swapper/008:14:160
17038991611,3cyclictest27692-21df08:45:140
1703899160,9cyclictest0-21swapper/011:50:190
1705299159,3cyclictest836-21runrttasks07:55:122
1705299151,5cyclictest0-21swapper/209:55:072
17052991510,3cyclictest32334-21gltestperf08:55:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional