You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 14:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Nov 23, 2025 12:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
193622238164,24sleep30-21swapper/307:08:223
193462237164,60sleep20-21swapper/207:08:102
175092206166,26sleep10-21swapper/107:05:061
193292202164,25sleep00-21swapper/007:07:550
19593997874,3cyclictest0-21swapper/209:22:232
241302580,3sleep011-21rcuc/007:15:250
19593994240,1cyclictest0-21swapper/208:07:442
253802360,12sleep341-21ksoftirqd/308:35:013
1959399362,20cyclictest13052-21kworker/2:008:37:492
1958899361,8cyclictest0-21swapper/110:27:411
1958899361,26cyclictest0-21swapper/111:14:071
1959399352,18cyclictest13052-21kworker/2:009:28:132
1958899341,14cyclictest0-21swapper/111:52:411
1958899331,15cyclictest11105-21diskstats09:15:131
1958899331,13cyclictest0-21swapper/112:34:311
1958399335,13cyclictest19967-21munin-run10:50:010
1959399321,12cyclictest13052-21kworker/2:012:00:292
1959399321,12cyclictest13052-21kworker/2:012:00:292
1958899321,13cyclictest0-21swapper/108:24:331
1958899320,8cyclictest0-21swapper/111:17:531
1958899316,14cyclictest0-21swapper/109:12:311
1958899311,9cyclictest0-21swapper/110:46:431
1958899311,14cyclictest0-21swapper/110:11:331
1958899311,12cyclictest0-21swapper/110:42:111
1958899311,11cyclictest5703-21sed07:50:011
1958899310,8cyclictest0-21swapper/110:52:151
1958399315,12cyclictest3013-21idleruntime-cro11:25:020
1958399312,15cyclictest8327-21diskmemload10:35:350
1959399302,10cyclictest13052-21kworker/2:008:50:212
1959399302,10cyclictest13052-21kworker/2:008:50:212
1958899301,14cyclictest0-21swapper/111:49:431
1958899301,14cyclictest0-21swapper/111:38:411
1958899301,14cyclictest0-21swapper/109:21:351
1958899301,14cyclictest0-21swapper/107:52:031
1958899301,13cyclictest0-21swapper/112:29:031
1958899301,12cyclictest0-21swapper/109:59:511
1958899301,12cyclictest0-21swapper/107:17:291
1958899301,11cyclictest0-21swapper/111:40:491
1958899301,11cyclictest0-21swapper/110:56:411
1958899301,11cyclictest0-21swapper/110:56:411
1958899300,24cyclictest0-21swapper/112:36:451
1958899300,13cyclictest0-21swapper/109:43:451
1959399292,17cyclictest13052-21kworker/2:010:20:452
1959399292,16cyclictest13052-21kworker/2:007:37:092
19588992912,3cyclictest1312-21/usr/sbin/munin07:40:141
1958899291,22cyclictest0-21swapper/112:21:151
1958899291,14cyclictest3573-21diskstats11:25:131
1958899291,13cyclictest0-21swapper/112:19:211
1958899291,12cyclictest0-21swapper/112:09:031
1958899291,11cyclictest0-21swapper/110:30:111
1958899291,11cyclictest0-21swapper/107:25:161
1958899291,10cyclictest0-21swapper/109:05:591
1958899290,13cyclictest0-21swapper/110:38:411
1958899290,11cyclictest0-21swapper/108:37:071
1958399294,12cyclictest16510-21awk08:15:010
1959399281,19cyclictest13052-21kworker/2:010:44:532
1958899281,13cyclictest0-21swapper/110:00:211
1958899281,12cyclictest0-21swapper/111:04:231
1958899281,12cyclictest0-21swapper/109:36:311
1958899281,12cyclictest0-21swapper/108:29:531
1958899281,11cyclictest0-21swapper/111:58:351
1958899281,11cyclictest0-21swapper/111:21:011
1958899281,11cyclictest0-21swapper/111:05:471
1958899281,11cyclictest0-21swapper/110:15:531
1958899281,11cyclictest0-21swapper/109:32:491
1958899281,10cyclictest8327-21diskmemload12:00:531
1958899281,10cyclictest8327-21diskmemload12:00:531
1958899281,10cyclictest0-21swapper/112:11:391
1958899280,12cyclictest0-21swapper/110:06:131
1958899280,12cyclictest0-21swapper/108:05:171
1958399285,11cyclictest11172-21awk10:30:010
1959399272,11cyclictest13052-21kworker/2:009:40:052
1959399272,10cyclictest13052-21kworker/2:011:14:212
19593992716,3cyclictest13052-21kworker/2:011:28:372
1959399271,12cyclictest13052-21kworker/2:008:15:252
1959399270,10cyclictest13052-21kworker/2:012:36:452
1958899271,13cyclictest0-21swapper/109:45:411
1958899271,11cyclictest0-21swapper/108:54:091
1958899271,11cyclictest0-21swapper/108:54:091
1958899271,10cyclictest0-21swapper/111:30:251
19588992710,3cyclictest31236-40dnf07:34:521
1959399262,14cyclictest13052-21kworker/2:011:55:172
1959399262,12cyclictest13052-21kworker/2:009:36:532
1959399262,10cyclictest13052-21kworker/2:007:10:452
1959399261,18cyclictest13052-21kworker/2:011:21:572
1959399261,12cyclictest13052-21kworker/2:008:34:212
1959399261,10cyclictest13052-21kworker/2:007:23:092
1958899261,13cyclictest0-21swapper/107:45:451
1958899261,12cyclictest0-21swapper/110:21:431
1958899261,11cyclictest0-21swapper/109:01:571
1958899260,11cyclictest0-21swapper/109:54:131
1958899260,11cyclictest0-21swapper/109:54:131
1958399261,13cyclictest9598-21munin-run11:40:020
1958399260,12cyclictest23740-21idleruntime-cro09:45:020
1959399252,10cyclictest13052-21kworker/2:010:17:412
19593992513,2cyclictest13052-21kworker/2:012:19:252
1959399251,21cyclictest11979-21grep09:15:232
1959399251,13cyclictest13052-21kworker/2:010:35:332
19593992511,3cyclictest13052-21kworker/2:007:58:372
1959399250,12cyclictest13052-21kworker/2:010:07:012
1958899251,9cyclictest0-21swapper/107:12:351
1958899251,13cyclictest0-21swapper/109:26:311
1958899251,12cyclictest0-21swapper/108:14:051
1958399259,14cyclictest7-21ksoftirqd/008:40:020
1958399253,10cyclictest29423-21basename12:25:020
1959399248,2cyclictest13052-21kworker/2:010:25:252
1959399242,13cyclictest13052-21kworker/2:012:21:412
1959399242,13cyclictest13052-21kworker/2:011:31:252
1959399242,13cyclictest13052-21kworker/2:010:57:252
1959399242,13cyclictest13052-21kworker/2:010:57:252
1959399242,11cyclictest13052-21kworker/2:011:03:492
1959399242,11cyclictest13052-21kworker/2:009:31:412
1959399242,10cyclictest13052-21kworker/2:012:34:452
19593992417,2cyclictest13052-21kworker/2:010:48:292
1959399241,11cyclictest13052-21kworker/2:012:05:332
1959399241,11cyclictest13052-21kworker/2:011:17:412
1958899241,16cyclictest0-21swapper/108:44:351
1958899241,14cyclictest0-21swapper/107:22:531
1958399246,10cyclictest12956-21/usr/sbin/munin09:20:230
1959899231,6cyclictest0-21swapper/309:25:113
1959399232,9cyclictest13052-21kworker/2:009:58:292
1959399232,8cyclictest13052-21kworker/2:010:31:492
1959399232,13cyclictest13052-21kworker/2:011:47:332
1959399231,10cyclictest13052-21kworker/2:010:01:012
1958899231,6cyclictest0-21swapper/108:47:451
1958899231,6cyclictest0-21swapper/108:18:131
1958899231,12cyclictest0-21swapper/107:35:251
1958899231,11cyclictest0-21swapper/107:55:071
1958399231,2cyclictest86550irq/126-enp1s0-11:18:250
1958399231,12cyclictest22175-21perf10:55:010
1958399230,11cyclictest18709-21perf08:20:010
256862222,3sleep32051-1kworker/u9:011:00:263
1959899221,9cyclictest129332latency_hist09:20:023
1959899221,3cyclictest0-21swapper/310:38:123
1959899221,15cyclictest0-21swapper/309:50:193
1959899221,15cyclictest0-21swapper/309:50:193
1959399226,4cyclictest13052-21kworker/2:008:22:072
1959399221,9cyclictest13052-21kworker/2:011:50:212
19593992211,2cyclictest13052-21kworker/2:009:09:412
1958899221,15cyclictest0-21swapper/108:56:171
1958899220,13cyclictest0-21swapper/108:30:271
1958899220,13cyclictest0-21swapper/108:02:291
1958399227,14cyclictest0-21swapper/012:00:010
1958399226,6cyclictest0-21swapper/010:43:320
1958399223,15cyclictest9056-21head07:55:240
1958399221,12cyclictest21345-21fschecks_time12:05:160
1958399221,10cyclictest0-21swapper/011:28:210
1959899212,10cyclictest0-21swapper/308:45:213
1959899211,11cyclictest6341-21memory11:30:193
1959899210,11cyclictest11574-21kworker/u8:212:11:253
1959399219,4cyclictest4565-21fschecks_count09:00:142
1959399219,3cyclictest13052-21kworker/2:012:11:492
1959399212,10cyclictest13052-21kworker/2:010:52:052
1959399211,13cyclictest13052-21kworker/2:010:13:252
1959399211,11cyclictest13052-21kworker/2:011:42:132
1959399211,11cyclictest13052-21kworker/2:007:20:012
1958399219,3cyclictest23383-21strings10:55:210
1958399219,3cyclictest23383-21strings10:55:210
1958399218,3cyclictest0-21swapper/010:23:380
1958399217,2cyclictest2975-21kworker/0:207:45:210
1958399217,2cyclictest2975-21kworker/0:207:45:210
1958399217,13cyclictest26137-21head11:00:290
1958399216,9cyclictest0-21swapper/010:18:150
1958399216,5cyclictest0-21swapper/010:05:030
1958399215,2cyclictest24339-21grep09:45:160
1958399215,13cyclictest26874-21munin-run07:25:020
1958399211,9cyclictest0-21swapper/008:55:280
1958399211,11cyclictest30586-21users11:10:280
1958399211,11cyclictest10898-21ntp_states11:40:230
315722201,2sleep20-21swapper/208:46:092
1959899207,5cyclictest1113-21expr11:20:123
1959899207,3cyclictest15241-21basename08:10:173
1959899207,3cyclictest0-21swapper/311:29:533
1959899205,13cyclictest39-21rcuc/308:20:273
1959899204,4cyclictest8245-21sshd09:09:103
1959899203,14cyclictest27102-21head08:35:263
1959899201,9cyclictest0-21swapper/307:38:403
1959899201,5cyclictest14000-21perf11:50:003
1959899201,10cyclictest9508-21users07:55:283
1959899201,10cyclictest30565-21cpu10:00:113
19598992010,7cyclictest5411-21systemd-journal07:49:073
19598992010,7cyclictest5411-21systemd-journal07:49:073
1959399202,12cyclictest13052-21kworker/2:008:13:012
1958399209,6cyclictest0-21swapper/007:29:560
1958399209,3cyclictest1640-21sshd08:54:080
1958399209,3cyclictest1640-21sshd08:54:080
1958399208,3cyclictest0-21swapper/007:35:130
1958399201,8cyclictest0-21swapper/012:26:200
19583992013,4cyclictest24487-21cat12:10:270
1958399201,11cyclictest12326-21/usr/sbin/munin08:05:240
1958399201,10cyclictest0-21swapper/009:34:240
1959899198,3cyclictest0-21swapper/310:49:043
1959899198,2cyclictest30235-21gltestperf12:25:143
1959899194,12cyclictest39-21rcuc/308:50:163
1959899194,12cyclictest39-21rcuc/308:50:163
1959899192,5cyclictest26521-21kworker/u8:211:35:253
1959899191,9cyclictest6894-21grep07:50:233
1959899191,9cyclictest21701-21unixbench_multi10:50:263
1959899191,9cyclictest0-21swapper/309:10:103
19598991913,3cyclictest836-21runrttasks10:20:193
1959899191,12cyclictest86550irq/126-enp1s0-07:14:163
1959899191,10cyclictest31832-21/usr/sbin/munin12:30:243
1959899191,10cyclictest14634-21snmpd10:27:383
1959899191,10cyclictest0-21swapper/311:54:193
1959399193,4cyclictest10895-21kworker/u8:109:10:212
19593991914,3cyclictest29947-21gltestperf07:30:152
19593991914,2cyclictest13052-21kworker/2:011:07:412
1959399191,10cyclictest13052-21kworker/2:008:29:252
1958399196,6cyclictest86550irq/126-enp1s0-12:02:210
1958399196,6cyclictest86550irq/126-enp1s0-12:02:210
1958399192,14cyclictest0-21swapper/009:15:190
1958399191,9cyclictest27895-21kworker/0:009:54:520
1958399191,9cyclictest27895-21kworker/0:009:54:510
1958399191,8cyclictest2559-21gunzip07:40:250
1958399191,8cyclictest0-21swapper/008:27:440
1958399191,7cyclictest26521-21kworker/u8:211:31:200
1958399191,7cyclictest0-21swapper/008:30:130
1958399191,5cyclictest17128-21munin-run09:30:020
1958399191,10cyclictest8014-21sshd09:06:110
1958399191,10cyclictest27726-21/usr/sbin/munin08:40:240
1958399191,10cyclictest14226-21/usr/sbin/munin11:50:150
175032191,3sleep30-21swapper/310:41:553
204782180,2sleep30-21swapper/312:03:283
204782180,2sleep30-21swapper/312:03:283
1959899186,2cyclictest29640-21switchtime09:55:253
1959899184,3cyclictest86550irq/126-enp1s0-12:35:163
1959899182,3cyclictest16588-21/usr/sbin/munin11:55:123
1959899181,8cyclictest0-21swapper/310:18:333
1959899181,7cyclictest4261-21cut09:00:123
1959899181,5cyclictest0-21swapper/310:30:123
1959899181,3cyclictest86550irq/126-enp1s0-11:12:333
1959899181,3cyclictest28178-21taskset12:20:173
19598991813,3cyclictest4059-21unixbench-2d10:10:273
19598991812,3cyclictest1032-21gltestperf10:05:153
19598991812,3cyclictest10117-21/usr/sbin/munin08:00:183
1959899181,16cyclictest0-21swapper/310:55:283
1959899181,16cyclictest0-21swapper/310:55:283
1959899181,14cyclictest21755-21/usr/sbin/munin09:40:233
19598991810,5cyclictest848-21sssd_nss08:25:183
1959399182,10cyclictest13052-21kworker/2:009:45:012
1959399182,10cyclictest13052-21kworker/2:007:26:452
1959399181,4cyclictest0-21swapper/212:25:262
1959399181,10cyclictest13052-21kworker/2:008:56:132
1958399188,2cyclictest12023-21/usr/sbin/munin11:45:180
1958399182,8cyclictest0-21swapper/010:03:020
1958399181,9cyclictest8327-21diskmemload11:07:140
1958399181,9cyclictest0-21swapper/010:33:070
1958399181,9cyclictest0-21swapper/010:13:570
1958399181,10cyclictest0-21swapper/012:31:130
120242181,2sleep20-21swapper/208:04:142
75632171,2sleep00-21swapper/007:53:320
295732171,2sleep30-21swapper/308:43:413
1959899173,6cyclictest0-21swapper/311:42:063
1959899172,7cyclictest39-21rcuc/309:45:213
1959899172,6cyclictest848-21sssd_nss07:30:203
1959899172,12cyclictest41-21ksoftirqd/307:40:183
1959899171,7cyclictest0-21swapper/311:17:133
1959899171,6cyclictest20163-21gltestperf09:35:163
1959899171,6cyclictest0-21swapper/312:06:063
1959899171,6cyclictest0-21swapper/309:30:173
1959899171,3cyclictest0-21swapper/308:15:353
19598991712,3cyclictest1951-21/usr/sbin/munin08:55:153
1959899171,15cyclictest0-21swapper/307:28:273
1959899170,7cyclictest28330-21unixbench_multi11:05:413
1959399171,6cyclictest7964-21df11:35:142
1959399171,6cyclictest0-21swapper/209:51:372
1959399171,6cyclictest0-21swapper/209:51:372
1958399178,3cyclictest0-21swapper/012:35:140
1958399174,3cyclictest0-21swapper/009:57:060
1958399172,12cyclictest25238-21/usr/sbin/munin12:15:140
1958399171,7cyclictest0-21swapper/007:10:420
1958399171,6cyclictest0-21swapper/009:35:200
19583991713,3cyclictest0-21swapper/009:03:160
1958399171,2cyclictest9916-21timerandwakeup09:10:270
19583991712,3cyclictest10213-21cpuspeed_turbos08:00:110
1958399171,10cyclictest0-21swapper/008:49:530
19583991710,4cyclictest29405-21cut07:30:120
205642161,2sleep00-21swapper/008:20:540
1959899165,3cyclictest86550irq/126-enp1s0-09:20:213
1959899163,12cyclictest39-21rcuc/307:20:243
1959899162,8cyclictest0-21swapper/307:15:123
1959899161,6cyclictest0-21swapper/312:15:243
19598991612,3cyclictest0-21swapper/308:06:523
19593991612,3cyclictest0-21swapper/208:41:172
19593991611,3cyclictest836-21runrttasks07:45:082
19593991611,3cyclictest836-21runrttasks07:45:082
19593991611,3cyclictest1589-21cat07:40:142
19593991610,2cyclictest13052-21kworker/2:007:50:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional