You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 18:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Jun 28, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
278232239164,24sleep20-21swapper/207:06:582
278012238161,25sleep00-21swapper/007:06:400
278662230172,48sleep30-21swapper/307:07:323
277202203167,24sleep10-21swapper/107:05:361
247432680,1sleep30-21swapper/311:55:013
28154996356,2cyclictest0-21swapper/212:13:212
60222600,4sleep06020-21grep07:30:150
309722590,1sleep330973-21cat08:25:253
240722540,4sleep230-21rcuc/209:25:152
239642520,4sleep12815099cyclictest11:50:221
223142450,2sleep10-21swapper/110:35:141
26132440,20sleep22815499cyclictest12:15:232
28154993835,2cyclictest0-21swapper/212:23:162
2815499371,6cyclictest6417-21hddtemp_smartct12:25:172
28154993634,1cyclictest0-21swapper/208:35:282
2815499361,5cyclictest26978-21fschecks_time10:45:152
28154993611,22cyclictest25708-21irqstats11:55:162
2815499355,3cyclictest87550irq/128-enp1s0-07:35:182
2815499353,5cyclictest16550-40dnf09:06:252
2815499353,5cyclictest16550-40dnf09:06:252
2815099351,27cyclictest0-21swapper/107:15:151
2815499342,3cyclictest1626-21memory11:00:192
2815499341,4cyclictest21335-21latency_hist09:20:012
28154993412,19cyclictest5125-21/usr/sbin/munin11:10:202
2815499333,5cyclictest0-21swapper/209:41:282
2815499331,5cyclictest31950-21sh10:55:202
2815499331,28cyclictest14524-21grep11:30:172
28154993312,18cyclictest27445-21/usr/sbin/munin08:20:132
28154993311,19cyclictest7318-21/usr/sbin/munin11:15:182
28154993311,19cyclictest11715-21/usr/sbin/munin11:25:242
28154993311,19cyclictest11715-21/usr/sbin/munin11:25:242
110902330,3sleep02814299cyclictest07:40:230
2815499327,4cyclictest10040-21/usr/sbin/munin11:20:142
2815499321,5cyclictest14360-21sshd09:01:262
28154993210,19cyclictest20857-21/usr/sbin/munin08:05:172
213102320,3sleep321302-21fschecks_count08:05:143
2815499319,18cyclictest9902-21sh12:35:012
2815499318,3cyclictest17573-21unixbench-2d11:35:242
2815499312,4cyclictest9243-21diskstats10:05:132
2815499311,6cyclictest0-21swapper/210:15:462
28154993112,16cyclictest30417-21ls12:05:232
28154993111,17cyclictest22085-21/usr/sbin/munin10:35:252
28154993111,17cyclictest14593-21df07:50:132
28154993111,16cyclictest29985-21switchtime10:50:242
28154993110,18cyclictest10471-21/usr/sbin/munin08:55:132
28154993110,17cyclictest3646-21cat11:05:182
2815499308,4cyclictest0-21swapper/209:30:222
2815499307,3cyclictest24950-21/usr/sbin/munin10:40:152
2815499304,3cyclictest12252-21/usr/sbin/munin10:10:222
2815499302,5cyclictest4945-21ntp_states08:40:222
2815499302,4cyclictest9338-21ntp_states08:50:192
2815499302,4cyclictest7503-21irqstats10:00:182
2815499301,6cyclictest24678-21munin-run11:55:012
2815499301,5cyclictest8048-21perf08:50:002
2815499301,5cyclictest19986-21threads08:00:232
2815499301,4cyclictest30329-21/usr/sbin/munin08:25:162
2815099300,15cyclictest0-21swapper/110:22:521
2815499291,5cyclictest5635-21/usr/sbin/munin07:30:142
2815499291,5cyclictest12060-21/usr/sbin/munin07:45:242
2815499291,4cyclictest19668-21munin-run10:30:012
2815499291,25cyclictest11308-21chrt07:40:252
2815099291,12cyclictest0-21swapper/112:35:001
2815099291,12cyclictest0-21swapper/110:52:511
2815099291,12cyclictest0-21swapper/108:44:081
2815099290,23cyclictest0-21swapper/111:24:111
28142992911,15cyclictest6472-21munin-run10:00:010
141392290,1sleep30-21swapper/307:50:023
2816099281,10cyclictest0-21swapper/312:36:233
2816099281,10cyclictest0-21swapper/312:36:233
2815499289,16cyclictest28581-21expr07:10:132
2815499287,3cyclictest22458-21grep09:20:182
2815499287,3cyclictest1051-21/usr/sbin/munin07:20:232
2815499285,4cyclictest910-21systemd-logind07:57:262
2815499285,4cyclictest910-21systemd-logind07:57:262
2815499284,3cyclictest367-21needreboot08:30:202
2815499282,4cyclictest808-21snmpd12:35:142
2815499282,4cyclictest808-21snmpd12:35:142
2815499282,4cyclictest20700-21irqstats10:30:182
2815499281,5cyclictest25594-21df_inode08:15:132
2815499281,4cyclictest25025-21munin-run08:15:012
2815499281,4cyclictest18309-21/usr/sbin/munin11:40:232
28154992810,15cyclictest29043-21uname09:35:192
2815099281,9cyclictest0-21swapper/112:38:081
2815099281,9cyclictest0-21swapper/112:38:081
2815099281,13cyclictest0-21swapper/111:17:271
2815099281,10cyclictest0-21swapper/110:13:591
2815099281,10cyclictest0-21swapper/108:30:471
28142992810,15cyclictest17477-21awk10:25:010
6582270,2sleep30-21swapper/311:00:113
2815499275,18cyclictest0-21swapper/211:45:162
2815499273,3cyclictest5120-21gltestperf09:55:172
2815499272,4cyclictest27880-21/usr/sbin/munin12:00:172
2815499271,6cyclictest0-21swapper/209:47:442
2815499271,4cyclictest3622-21diskstats07:25:162
2815499271,4cyclictest17889-21irqstats09:10:172
28154992710,14cyclictest4272-21perf09:55:002
28154992710,14cyclictest16241-21if_enp2s010:20:162
2815099271,12cyclictest0-21swapper/111:55:511
2815099271,12cyclictest0-21swapper/108:35:351
2815099271,12cyclictest0-21swapper/108:24:191
2815099271,11cyclictest0-21swapper/109:17:471
2815099271,11cyclictest0-21swapper/109:12:071
2815099270,15cyclictest0-21swapper/112:18:471
2815099270,14cyclictest0-21swapper/111:43:271
2814299278,16cyclictest13702-21basename11:30:010
2814299278,16cyclictest13702-21basename11:30:010
2814299271,12cyclictest26267-21perf10:45:010
2814299270,13cyclictest0-21swapper/011:10:010
2815499265,3cyclictest0-21swapper/207:15:122
2815099261,13cyclictest16852-21diskmemload11:07:511
2815099261,12cyclictest0-21swapper/110:00:191
2815099260,13cyclictest0-21swapper/108:08:471
2815099260,11cyclictest0-21swapper/111:14:391
2816099251,22cyclictest0-21swapper/308:53:263
2815099251,9cyclictest0-21swapper/111:46:551
2815099251,9cyclictest0-21swapper/111:30:191
2815099251,9cyclictest0-21swapper/107:41:111
2815099251,14cyclictest0-21swapper/112:12:391
2815099251,13cyclictest0-21swapper/107:23:161
2815099251,10cyclictest0-21swapper/107:29:511
2815099250,12cyclictest0-21swapper/112:07:511
2815099250,11cyclictest0-21swapper/109:21:431
28142992511,4cyclictest31139-40dnf12:08:240
2814299251,13cyclictest2083-21awk09:50:010
2816099249,3cyclictest1064-21needreboot09:45:193
2816099241,20cyclictest16852-21diskmemload10:25:213
2815099247,4cyclictest0-21swapper/110:45:131
2815099241,16cyclictest0-21swapper/112:25:091
2815099241,15cyclictest0-21swapper/110:30:411
2815099241,15cyclictest0-21swapper/109:42:361
2815099241,13cyclictest0-21swapper/108:49:081
2815099241,13cyclictest0-21swapper/107:48:231
2815099241,12cyclictest0-21swapper/111:00:271
2815099241,12cyclictest0-21swapper/110:59:511
2815099241,11cyclictest0-21swapper/112:01:591
2815099240,3cyclictest0-21swapper/108:15:151
2815099240,13cyclictest0-21swapper/109:06:511
2815099240,13cyclictest0-21swapper/109:06:511
2815099240,11cyclictest0-21swapper/111:27:551
2815099240,11cyclictest0-21swapper/111:27:551
2816099239,3cyclictest31512-21/usr/sbin/munin12:10:223
2816099239,3cyclictest24897-21/usr/sbin/munin11:55:173
2816099238,3cyclictest897-21timerwakeupswit08:30:263
2816099231,6cyclictest0-21swapper/310:11:153
2816099231,11cyclictest24279-21/usr/sbin/munin10:40:243
2816099231,11cyclictest2294-21/usr/sbin/munin09:50:123
2816099230,21cyclictest0-21swapper/311:10:253
2816099230,21cyclictest0-21swapper/310:20:193
2816099230,21cyclictest0-21swapper/308:35:193
2815099231,9cyclictest0-21swapper/109:26:391
2815099231,16cyclictest0-21swapper/112:25:011
2815099231,16cyclictest0-21swapper/109:02:311
2815099231,13cyclictest0-21swapper/109:37:591
2815099231,11cyclictest0-21swapper/109:30:181
2815099230,6cyclictest0-21swapper/109:47:091
2815099230,14cyclictest0-21swapper/110:27:201
2815099230,13cyclictest0-21swapper/111:36:431
2814299237,5cyclictest0-21swapper/011:16:430
2816099221,10cyclictest31239-21/usr/sbin/munin07:15:233
2816099221,10cyclictest30260-21/usr/sbin/munin09:40:223
2816099221,10cyclictest18309-21/usr/sbin/munin11:40:233
2816099221,10cyclictest12670-21/usr/sbin/munin09:00:143
2816099220,9cyclictest0-21swapper/312:26:433
2816099220,21cyclictest0-21swapper/307:35:163
2816099220,20cyclictest0-21swapper/310:05:223
2816099220,20cyclictest0-21swapper/308:15:183
2816099220,20cyclictest0-21swapper/308:10:193
2815099222,5cyclictest808-21snmpd07:50:181
2815099221,3cyclictest8468-21cat08:50:111
2815099221,16cyclictest0-21swapper/108:14:551
2815099221,16cyclictest0-21swapper/107:56:221
2815099221,16cyclictest0-21swapper/107:56:221
2815099221,15cyclictest0-21swapper/107:39:161
2815099220,15cyclictest0-21swapper/110:07:011
2815099220,14cyclictest0-21swapper/110:15:371
2815099220,12cyclictest0-21swapper/107:31:231
2814299227,6cyclictest0-21swapper/012:36:010
2814299227,6cyclictest0-21swapper/012:36:010
2814299227,4cyclictest0-21swapper/012:15:020
2814299225,14cyclictest11-21rcuc/009:35:010
2814299220,9cyclictest0-21swapper/010:10:290
92232210,2sleep00-21swapper/012:30:230
2816099218,4cyclictest839-21lldpd12:04:443
2816099217,3cyclictest3272-21/usr/sbin/munin07:25:183
2816099215,3cyclictest16221-21crond07:55:023
2816099211,8cyclictest7972-21wc10:00:243
2816099211,12cyclictest17076-21/usr/sbin/munin09:10:233
2816099211,10cyclictest839-21lldpd10:17:133
2816099210,10cyclictest0-21swapper/309:23:243
28150992117,3cyclictest0-21swapper/108:28:351
2814299212,11cyclictest3878-21/usr/sbin/munin08:40:200
2814299211,9cyclictest18315-21fgrep09:10:230
2814299211,17cyclictest5280-21cpuspeed_turbos11:10:130
2814299211,11cyclictest622-21/usr/sbin/munin11:00:140
2814299211,11cyclictest3902-21sshd09:50:270
2816099207,3cyclictest808-21snmpd09:38:423
2816099207,3cyclictest0-21swapper/311:49:453
2816099205,4cyclictest0-21swapper/309:06:243
2816099205,4cyclictest0-21swapper/309:06:243
2816099201,8cyclictest808-21snmpd07:33:173
2816099201,6cyclictest13051-21threads11:25:263
2816099201,6cyclictest13051-21threads11:25:263
2816099201,13cyclictest0-21swapper/307:40:213
2816099201,10cyclictest0-21swapper/311:15:133
2816099200,19cyclictest0-21swapper/307:55:233
2816099200,19cyclictest0-21swapper/307:55:233
2816099200,10cyclictest0-21swapper/307:20:163
2815099206,8cyclictest0-21swapper/109:57:221
2815099201,8cyclictest0-21swapper/109:52:301
28150992012,3cyclictest0-21swapper/107:10:111
2815099201,10cyclictest10677-21cstates08:55:141
2815099201,10cyclictest0-21swapper/108:00:201
2815099200,14cyclictest0-21swapper/110:41:561
2814299205,6cyclictest0-21swapper/012:27:120
28142992013,5cyclictest0-21swapper/009:20:230
2814299201,10cyclictest18763-21ntp_states10:25:230
2814299201,10cyclictest0-21swapper/011:57:430
2814299201,10cyclictest0-21swapper/011:20:030
2814299200,10cyclictest0-21swapper/010:57:590
310652191,2sleep30-21swapper/312:07:263
2816099195,3cyclictest3541-21/usr/sbin/munin12:20:153
2816099193,4cyclictest7840-21seq08:47:333
2816099192,8cyclictest0-21swapper/310:35:123
2816099191,6cyclictest18649-21/usr/sbin/munin08:00:193
2816099191,6cyclictest13916-21/usr/sbin/munin11:30:273
2816099191,3cyclictest16600-21grep11:35:153
28160991912,4cyclictest28680-21/usr/sbin/munin08:20:233
2816099191,11cyclictest0-21swapper/310:56:073
2816099190,1cyclictest0-21swapper/310:45:153
2814299198,3cyclictest27158-21sshd08:19:270
2814299197,3cyclictest0-21swapper/011:42:310
2814299196,6cyclictest0-21swapper/011:37:280
2814299196,11cyclictest0-21swapper/007:50:010
2814299195,2cyclictest87350irq/126-enp1s0-09:18:220
28142991913,4cyclictest14023-21cpuspeed_turbos11:30:140
28142991913,3cyclictest13434-21tr10:15:110
2814299191,10cyclictest21646-21sshd10:32:250
2814299191,10cyclictest20857-21/usr/sbin/munin08:05:160
2814299191,10cyclictest18649-21/usr/sbin/munin08:00:100
2814299191,10cyclictest16560-21sshd09:06:260
2814299191,10cyclictest16560-21sshd09:06:260
2814299190,9cyclictest0-21swapper/012:01:370
56042181,2sleep30-21swapper/308:42:033
2816099187,2cyclictest0-21swapper/311:20:193
2816099185,3cyclictest12050irq/123-ahci[0009:30:023
2816099184,3cyclictest19276-21/usr/sbin/munin09:15:133
2816099184,3cyclictest16852-21diskmemload11:05:193
2816099183,3cyclictest808-21snmpd08:58:353
2816099182,4cyclictest8850-21memory12:30:173
2816099181,5cyclictest25859-21/usr/sbin/munin09:30:193
28160991812,3cyclictest827-21runrttasks09:56:383
28160991812,3cyclictest30653-21turbostat10:55:013
2816099181,10cyclictest0-21swapper/310:32:513
2816099180,9cyclictest0-21swapper/307:10:253
2816099180,17cyclictest0-21swapper/312:17:433
2814299188,2cyclictest0-21swapper/012:24:250
2814299186,3cyclictest0-21swapper/009:29:200
2814299182,6cyclictest18245-1kworker/u9:210:36:470
2814299181,6cyclictest0-21swapper/010:07:150
2814299180,8cyclictest0-21swapper/009:38:000
2814299180,11cyclictest0-21swapper/008:10:240
32692170,2sleep00-21swapper/007:25:110
2814299171,9cyclictest0-21swapper/012:18:480
2814299171,7cyclictest0-21swapper/008:20:140
2814299171,5cyclictest8281-21sshd10:00:270
28142991713,3cyclictest0-21swapper/007:37:020
2814299171,15cyclictest0-21swapper/008:55:160
2814299171,14cyclictest7135-21ntpq08:45:210
2814299171,14cyclictest0-21swapper/010:47:020
28142991711,3cyclictest32171-21expr08:30:140
28142991711,3cyclictest2126-21ntp_offset07:20:200
2814299171,13cyclictest16852-21diskmemload11:45:130
2814299171,10cyclictest0-21swapper/011:51:510
2814299171,10cyclictest0-21swapper/009:40:110
28142991710,4cyclictest29906-21grep10:50:220
2814299165,3cyclictest1983-21cstates08:35:170
2814299161,6cyclictest0-21swapper/008:25:140
2814299161,14cyclictest0-21swapper/008:50:320
2814299161,14cyclictest0-21swapper/007:16:020
28142991611,3cyclictest28348-21acpi07:10:130
28142991611,3cyclictest18406-21seq07:59:570
28142991611,3cyclictest18406-21seq07:59:570
2814299161,12cyclictest14256-21/usr/sbin/munin07:50:250
28142991610,3cyclictest827-21runrttasks09:01:390
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional