You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-11 - 14:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Apr 11, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
296182240162,26sleep00-21swapper/007:08:050
278012237163,23sleep30-21swapper/307:05:043
297222234164,57sleep20-21swapper/207:09:292
294592233158,24sleep10-21swapper/107:06:021
81412630,11sleep230-21rcuc/212:25:162
230872500,4sleep121-21rcuc/110:30:251
33172460,4sleep339-21rcuc/312:15:133
33172460,4sleep339-21rcuc/312:15:133
275982450,2sleep10-21swapper/108:15:161
2986399431,34cyclictest0-21swapper/209:40:112
2985399389,13cyclictest10413-21basename10:05:000
2987099351,18cyclictest0-21swapper/311:07:183
2987099351,18cyclictest0-21swapper/311:07:183
2987099341,24cyclictest0-21swapper/312:30:303
2987099321,23cyclictest0-21swapper/309:57:323
2987099321,22cyclictest0-21swapper/311:32:083
2987099321,12cyclictest0-21swapper/309:21:083
2985399325,12cyclictest8192-21munin-run10:00:000
2985399324,13cyclictest24550-21awk08:10:010
2987099311,13cyclictest0-21swapper/307:27:373
2987099311,13cyclictest0-21swapper/307:27:363
2987099310,23cyclictest0-21swapper/311:53:413
2987099310,14cyclictest0-21swapper/312:05:263
2987099301,10cyclictest0-21swapper/310:40:493
2987099291,12cyclictest0-21swapper/308:54:253
2985399294,12cyclictest30183-21munin-run10:50:010
2987099281,9cyclictest0-21swapper/310:46:253
2987099281,19cyclictest0-21swapper/308:12:253
2987099281,18cyclictest0-21swapper/308:37:013
2987099281,18cyclictest0-21swapper/308:37:013
2987099281,16cyclictest0-21swapper/312:22:543
2987099281,11cyclictest0-21swapper/309:19:563
2987099281,10cyclictest0-21swapper/308:46:213
2987099280,17cyclictest0-21swapper/311:16:223
2987099280,12cyclictest0-21swapper/310:14:373
29863992818,5cyclictest0-21swapper/210:03:202
2987099271,11cyclictest0-21swapper/311:25:493
2987099271,10cyclictest2883-21ntpq09:45:203
2987099271,10cyclictest0-21swapper/312:10:253
2987099270,19cyclictest0-21swapper/307:35:573
2987099270,12cyclictest0-21swapper/310:03:293
2987099270,11cyclictest0-21swapper/309:54:443
2987099270,10cyclictest0-21swapper/309:11:533
2985799272,10cyclictest13499-21hwlatdetect10:10:171
2985399272,20cyclictest87350irq/126-enp1s0-09:39:260
2987099261,11cyclictest18570-21diskmemload10:51:103
2987099261,11cyclictest0-21swapper/312:35:583
2987099261,11cyclictest0-21swapper/308:31:133
2987099261,10cyclictest0-21swapper/311:13:533
2987099261,10cyclictest0-21swapper/307:13:413
2987099260,9cyclictest0-21swapper/311:44:243
2987099260,8cyclictest0-21swapper/308:59:453
2987099260,12cyclictest0-21swapper/310:20:213
2987099260,11cyclictest0-21swapper/308:24:293
2987099260,11cyclictest0-21swapper/307:47:213
2986399261,12cyclictest0-21swapper/208:12:412
2985799261,16cyclictest0-21swapper/112:34:451
2985799261,12cyclictest0-21swapper/107:10:241
2985399269,2cyclictest21294-21smart_sda11:40:230
2985399265,17cyclictest13215-21munin-run11:25:010
2987099251,9cyclictest28878-21grep12:00:123
2987099251,12cyclictest0-21swapper/311:21:263
2987099251,11cyclictest0-21swapper/307:43:333
2987099251,10cyclictest0-21swapper/311:56:533
2987099251,10cyclictest0-21swapper/309:42:363
2987099251,10cyclictest0-21swapper/308:41:253
2987099251,10cyclictest0-21swapper/308:06:253
2987099250,9cyclictest0-21swapper/311:02:143
2987099250,11cyclictest0-21swapper/310:36:493
2987099250,10cyclictest0-21swapper/311:35:093
2987099250,10cyclictest0-21swapper/307:20:323
2985799252,20cyclictest5091-21ntp_states09:50:231
2985799252,19cyclictest31689-40dnf09:39:251
2985799251,21cyclictest9145-21latency_hist07:35:021
2985799251,21cyclictest8144-21diskstats08:45:141
2985799251,21cyclictest802-21sssd_be11:40:131
2985799251,21cyclictest27910-21df09:30:131
2985799251,21cyclictest19316-21cat10:25:021
2985799251,21cyclictest16320-21wc10:15:251
2985799251,21cyclictest15974-21df_inode11:30:151
2985799250,18cyclictest0-21swapper/111:55:191
29853992514,3cyclictest87550irq/128-enp1s0-12:35:160
2985399251,12cyclictest23582-21munin-run10:35:010
2985399250,12cyclictest16999-21awk10:20:010
2987099248,3cyclictest0-21swapper/311:47:443
2987099241,9cyclictest0-21swapper/307:58:273
2987099241,13cyclictest32276-21memory08:25:173
2987099241,11cyclictest0-21swapper/310:09:213
2987099241,10cyclictest0-21swapper/312:28:583
29870992410,3cyclictest0-21swapper/308:01:013
2987099240,9cyclictest0-21swapper/309:27:083
2985799241,20cyclictest22171-21chrt08:02:561
2985799241,20cyclictest21799-21irqstats09:15:181
2985799241,20cyclictest16595-21/usr/sbin/munin09:05:171
2985799241,11cyclictest5945-21diskstats08:40:141
2985799241,11cyclictest25152-21latency_hist09:25:011
2985799241,11cyclictest18641-21irqstats11:35:171
2985799241,10cyclictest0-21swapper/109:40:111
2985399246,15cyclictest967-21perf08:30:010
2987099236,3cyclictest0-21swapper/307:50:473
2987099231,7cyclictest0-21swapper/310:16:423
2987099231,11cyclictest0-21swapper/308:15:133
2987099230,9cyclictest0-21swapper/309:35:163
2987099230,11cyclictest0-21swapper/310:57:583
2986399237,3cyclictest141762basename09:00:002
2985799231,19cyclictest13480-21sed08:55:241
2985799231,11cyclictest8428-21cat12:25:191
2985799231,11cyclictest4755-21munin-run07:25:011
2985799231,11cyclictest29040-21expr12:00:121
2985799231,11cyclictest26449-21ntp_states09:25:221
2985799231,10cyclictest1087-21date08:30:001
2985399233,11cyclictest13439-21/usr/sbin/munin11:25:150
2985399231,13cyclictest0-21swapper/011:02:240
2987099221,9cyclictest0-21swapper/309:04:413
2987099221,9cyclictest0-21swapper/307:17:433
2987099221,10cyclictest0-21swapper/310:32:453
2987099221,10cyclictest0-21swapper/307:33:133
2987099220,9cyclictest0-21swapper/310:28:373
2986399226,8cyclictest87350irq/126-enp1s0-09:35:122
2986399221,13cyclictest0-21swapper/209:20:262
29863992210,2cyclictest0-21swapper/208:20:592
29857992217,3cyclictest0-21swapper/112:12:021
2985799221,3cyclictest9717-21df07:35:141
2985799221,20cyclictest0-21swapper/110:02:071
2985799221,19cyclictest0-21swapper/111:48:101
2985799221,19cyclictest0-21swapper/110:58:421
2985799221,19cyclictest0-21swapper/110:37:441
2985799221,19cyclictest0-21swapper/109:12:501
2985799221,19cyclictest0-21swapper/109:03:591
2985799221,19cyclictest0-21swapper/108:36:531
2985799221,19cyclictest0-21swapper/108:36:531
2985799221,19cyclictest0-21swapper/108:09:061
2985799221,14cyclictest0-21swapper/112:20:431
2985799221,13cyclictest0-21swapper/108:34:421
2985799221,10cyclictest8993-21cat11:15:021
2985799221,10cyclictest30241-21ntpq08:20:231
2985799221,10cyclictest20630-21rt-features10:25:251
2985799220,15cyclictest0-21swapper/111:29:021
2985399225,14cyclictest2228-21basename11:00:010
2985399224,3cyclictest22540-21cat08:05:020
2985399223,11cyclictest27305-21/usr/sbin/munin11:55:160
2985399221,18cyclictest3175-21munin-run08:35:010
2985399221,13cyclictest0-21swapper/010:35:520
2987099216,3cyclictest27595-21grep09:30:103
2987099211,11cyclictest0-21swapper/309:06:293
2986399217,3cyclictest87350irq/126-enp1s0-07:16:502
2986399215,8cyclictest12050irq/123-ahci[0011:35:022
2986399213,9cyclictest31688-21sshd07:10:522
2986399213,15cyclictest8904-21ls08:45:222
2986399211,7cyclictest32735-21cpuspeed_turbos10:55:132
2986399211,6cyclictest29430-21gltestperf12:00:152
2985799215,12cyclictest579-21systemd-journal10:55:001
2985799211,9cyclictest25756-21strings08:10:201
2985799211,19cyclictest0-21swapper/112:16:091
2985799211,19cyclictest0-21swapper/112:16:091
2985799211,18cyclictest0-21swapper/111:16:331
2985799211,18cyclictest0-21swapper/111:01:101
2985799211,11cyclictest0-21swapper/110:45:181
2985799211,10cyclictest20140-21perf08:00:011
29857992110,3cyclictest0-21swapper/112:36:181
2985799210,19cyclictest0-21swapper/112:05:221
2985399218,4cyclictest87350irq/126-enp1s0-09:25:230
2985399213,11cyclictest0-21swapper/009:50:490
2985399212,11cyclictest3401-21/usr/sbin/munin08:35:150
2985399212,11cyclictest3401-21/usr/sbin/munin08:35:150
2985399212,11cyclictest15970-21/usr/sbin/munin07:50:250
2985399211,7cyclictest22094-21/usr/sbin/munin09:15:230
2985399211,5cyclictest0-21swapper/010:52:150
29853992114,4cyclictest11206-40dnf07:38:220
29853992113,2cyclictest87350irq/126-enp1s0-10:10:240
289492210,4sleep02985399cyclictest09:30:240
2986399206,4cyclictest0-21swapper/210:52:262
2986399206,3cyclictest87350irq/126-enp1s0-10:25:202
2986399204,9cyclictest0-21swapper/212:08:162
2986399201,9cyclictest0-21swapper/210:24:432
2986399201,7cyclictest3254-21irqstats11:00:212
2986399201,7cyclictest17204-21kworker/u8:010:35:252
2986399201,7cyclictest0-21swapper/210:47:472
2986399201,6cyclictest8020-21ps11:10:212
2986399201,6cyclictest26969-21diskstats11:55:142
2986399201,6cyclictest11536-21memory10:05:192
2986399201,5cyclictest0-21swapper/212:21:162
2986399201,4cyclictest0-21swapper/209:30:162
2986399201,12cyclictest0-21swapper/211:38:542
2986399201,12cyclictest0-21swapper/211:05:192
2986399201,12cyclictest0-21swapper/211:05:192
2986399201,10cyclictest7390-21sshd08:42:502
29857992015,3cyclictest0-21swapper/110:05:071
2985799201,18cyclictest0-21swapper/110:41:221
2985799201,18cyclictest0-21swapper/109:56:541
2985799201,18cyclictest0-21swapper/108:54:131
2985799201,18cyclictest0-21swapper/107:18:501
2985799201,16cyclictest17562-21users07:50:281
2985799201,16cyclictest12493-21memory07:40:191
2985799201,10cyclictest0-21swapper/109:47:411
2985799200,18cyclictest0-21swapper/111:50:241
2985399209,3cyclictest14388-21/usr/sbin/munin09:00:220
2985399209,3cyclictest12191-21/usr/sbin/munin08:55:230
2985399205,6cyclictest0-21swapper/010:41:380
2985399201,8cyclictest84802sed07:30:230
2985399201,5cyclictest0-21swapper/010:20:310
29853992014,3cyclictest5314-21gltestperf11:05:170
29853992014,3cyclictest5314-21gltestperf11:05:170
2985399201,11cyclictest11607-21sshd08:50:330
2985399201,11cyclictest10622-21/usr/sbin/munin10:05:180
2986399195,4cyclictest13554-21perf07:49:562
2986399194,4cyclictest19752-21sed09:10:182
2986399193,9cyclictest0-21swapper/211:53:092
2986399193,9cyclictest0-21swapper/209:56:022
2986399193,9cyclictest0-21swapper/208:35:072
2986399193,9cyclictest0-21swapper/208:35:062
2986399191,7cyclictest15993-21cpu07:50:142
2986399191,6cyclictest4105-21kworker/u8:011:15:152
2986399191,6cyclictest21618-21users11:40:252
2986399191,6cyclictest2156-21df09:45:142
2986399191,6cyclictest1755-21perl08:30:172
2986399191,6cyclictest0-21swapper/212:15:172
2986399191,6cyclictest0-21swapper/212:15:172
2986399191,5cyclictest579-21systemd-journal12:15:012
2986399191,5cyclictest4372-21users07:20:262
2986399191,5cyclictest11581-21sh11:20:142
2986399191,12cyclictest0-21swapper/212:37:372
2986399191,12cyclictest0-21swapper/210:30:372
2986399191,12cyclictest0-21swapper/209:25:212
2986399191,12cyclictest0-21swapper/208:02:192
2986399191,11cyclictest0-21swapper/212:32:592
2986399191,11cyclictest0-21swapper/207:25:102
2986399191,11cyclictest0-21swapper/207:25:102
2986399191,10cyclictest11575-21/usr/sbin/munin07:40:182
2985799191,3cyclictest11937-21grep11:20:181
2985799191,2cyclictest0-21swapper/111:08:291
2985799191,2cyclictest0-21swapper/111:08:291
2985799191,17cyclictest0-21swapper/107:48:371
2985799191,17cyclictest0-21swapper/107:27:471
2985799191,17cyclictest0-21swapper/107:27:471
2985399197,4cyclictest0-21swapper/012:13:390
2985399194,2cyclictest87350irq/126-enp1s0-11:45:460
2985399193,6cyclictest4142-21ntp_states12:15:210
2985399193,6cyclictest4142-21ntp_states12:15:210
2985399193,3cyclictest81ktimersoftd/011:30:280
2985399192,6cyclictest87350irq/126-enp1s0-10:25:190
2985399191,6cyclictest2711-21cat09:45:190
29853991912,4cyclictest28927-21cpuspeed_turbos12:00:110
2985399191,11cyclictest7796-21/usr/sbin/munin08:45:150
2985399191,10cyclictest0-21swapper/012:33:510
2985399191,10cyclictest0-21swapper/011:37:510
318272181,2sleep00-21swapper/007:12:390
2986399183,4cyclictest15101-21cpuspeed_turbos10:15:132
2986399181,9cyclictest87350irq/126-enp1s0-10:40:242
2986399181,9cyclictest0-21swapper/208:53:232
2986399181,7cyclictest0-21swapper/211:49:372
2986399181,6cyclictest0-21swapper/209:05:122
2986399181,5cyclictest4756-21if_enp1s009:50:172
2986399181,5cyclictest22919-21diskstats08:05:152
2986399181,5cyclictest13787-21diskstats11:25:142
2986399181,5cyclictest10447-21ntp_states07:35:222
2986399181,11cyclictest0-21swapper/209:15:172
2986399181,11cyclictest0-21swapper/208:27:262
2986399181,10cyclictest0-21swapper/210:11:532
2986399180,9cyclictest14774-21fschecks_count09:00:142
2985399186,5cyclictest87550irq/128-enp1s0-07:45:120
2985399185,3cyclictest7-21ksoftirqd/009:40:140
2985399183,6cyclictest11-21rcuc/007:55:250
2985399183,2cyclictest18570-21diskmemload12:25:150
2985399183,12cyclictest488-21/usr/sbin/munin07:15:210
2985399181,9cyclictest0-21swapper/011:51:230
2985399181,4cyclictest6847-21/usr/sbin/munin11:10:110
29853991813,3cyclictest18556-21idleruntime-cro09:10:020
2985399181,15cyclictest0-21swapper/009:20:190
207062181,2sleep00-21swapper/009:14:280
2986399171,6cyclictest0-21swapper/207:30:262
2986399171,4cyclictest19235-21ntp_states07:55:222
2985399175,2cyclictest11-21rcuc/012:20:230
2985399171,9cyclictest10116-21taskset11:15:220
2985399171,13cyclictest29159-21/usr/sbin/munin08:20:170
2986399161,5cyclictest0-21swapper/208:15:172
2985399162,2cyclictest11905-21expr07:40:120
2985399161,4cyclictest4981-21/usr/sbin/munin07:25:230
2985399161,4cyclictest4981-21/usr/sbin/munin07:25:220
2985399161,4cyclictest0-21swapper/008:40:190
29853991611,3cyclictest24766-21/usr/sbin/munin08:10:130
2985399161,12cyclictest2784-21/usr/sbin/munin07:20:250
2985399160,9cyclictest0-21swapper/012:08:510
2985399151,3cyclictest0-21swapper/008:15:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional