You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-05 - 15:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Mon Jan 05, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
245312231160,58sleep00-21swapper/007:08:190
243822205168,25sleep10-21swapper/107:06:221
244102203166,25sleep20-21swapper/207:06:432
243792201165,24sleep30-21swapper/307:06:203
1397221360,2sleep22476399cyclictest11:35:232
131692590,2sleep10-21swapper/111:35:151
150102500,1sleep3401ktimersoftd/310:25:173
2477099491,34cyclictest0-21swapper/312:05:193
24770994328,3cyclictest0-21swapper/308:40:173
2476399432,30cyclictest23475-40dnf08:15:412
2476399421,30cyclictest15021-21latency_hist08:00:012
2475399422,24cyclictest10950-21kworker/0:109:12:410
24770994128,7cyclictest0-21swapper/310:32:373
2476399411,29cyclictest2023-21grep09:55:182
2476399411,20cyclictest0-21swapper/212:34:092
24770993923,7cyclictest0-21swapper/309:06:003
2476399391,27cyclictest31657-21latency_hist11:05:012
2476399391,27cyclictest11651-21cstates09:05:112
2476399386,21cyclictest12050irq/123-ahci[0009:30:012
2476399381,28cyclictest4273-21/usr/sbin/munin07:35:252
2476399381,26cyclictest3726-21ntpq08:45:252
86942370,4sleep38699-21needreboot10:10:213
24770993723,7cyclictest0-21swapper/308:12:173
24770993723,7cyclictest0-21swapper/308:12:173
2477099371,14cyclictest0-21swapper/309:04:423
2476399371,25cyclictest16495-21/usr/sbin/munin10:30:282
2476399371,24cyclictest5639-40dnf11:16:552
2475399372,13cyclictest1663-21kworker/0:107:30:240
2477099361,14cyclictest0-21swapper/312:37:123
2476399361,25cyclictest2346-21/usr/sbin/munin12:25:272
2476399361,24cyclictest15882-21/usr/sbin/munin09:15:212
2475399361,29cyclictest5138-21kworker/0:110:05:380
2475399361,29cyclictest28454-21kworker/0:010:03:060
2475399361,23cyclictest6805-21kworker/0:009:00:390
2477099351,13cyclictest0-21swapper/308:08:233
2477099351,12cyclictest484-21ntpq11:05:223
2477099351,11cyclictest0-21swapper/309:23:063
2477099351,11cyclictest0-21swapper/309:23:063
2476399353,22cyclictest801-21snmpd07:12:392
2476399351,25cyclictest25285-21/usr/sbin/munin10:50:192
2476399351,24cyclictest14860-21processes09:10:232
2476399351,23cyclictest8648-21chrt12:39:262
2476399351,21cyclictest27950-21open_files09:40:192
2475399352,14cyclictest7817-21kworker/0:111:23:120
2475399351,27cyclictest22077-21kworker/0:209:32:040
2475399351,12cyclictest4438-21kworker/0:108:50:210
169292350,11sleep011-21rcuc/011:45:020
2477099341,13cyclictest0-21swapper/310:46:183
24763993414,17cyclictest21526-21gltestperf10:40:172
2476399341,20cyclictest30752-21idleruntime07:20:172
2476399341,18cyclictest16264-21sh10:30:012
2476399340,20cyclictest24803-21cut09:35:132
2475399342,22cyclictest12289-21kworker/0:211:49:520
2475399341,12cyclictest12099-21kworker/0:210:25:210
24770993325,3cyclictest0-21swapper/307:58:253
24770993323,3cyclictest0-21swapper/307:53:093
2477099331,13cyclictest0-21swapper/311:00:563
2477099331,13cyclictest0-21swapper/309:57:213
2477099331,13cyclictest0-21swapper/307:14:433
2476399335,18cyclictest12050irq/123-ahci[0011:30:012
2476399333,24cyclictest7707-21/usr/sbin/munin10:10:242
2476399332,27cyclictest32084-40dnf12:17:412
2476399332,21cyclictest87250irq/129-enp1s0-07:40:122
2476399332,16cyclictest26026-21sh08:25:022
2476399331,28cyclictest23906-21/usr/sbin/munin10:45:182
2476399331,28cyclictest12121-21smart_sda07:50:232
2476399331,21cyclictest86950irq/126-enp1s0-10:05:012
2476399331,20cyclictest9904-21/usr/sbin/munin10:15:232
2475399332,24cyclictest25280-21kworker/0:010:50:430
2475399331,26cyclictest25408-21kworker/0:212:06:090
2475399331,11cyclictest30045-21kworker/0:112:34:010
2475399331,11cyclictest28454-21kworker/0:009:45:180
2477099321,9cyclictest0-21swapper/309:35:093
2475399329,2cyclictest32304-21kworker/0:008:40:210
2475399328,2cyclictest5138-21kworker/0:110:17:210
2475399321,8cyclictest12289-21kworker/0:211:38:000
2475399321,24cyclictest6090-21kworker/0:207:42:240
2475399321,21cyclictest21969-21sshd09:26:410
294702310,4sleep341-21ksoftirqd/311:00:023
2477099310,13cyclictest0-21swapper/312:31:473
2477099310,11cyclictest0-21swapper/307:29:523
2476399318,20cyclictest31263-21/usr/sbin/munin09:50:252
2476399311,26cyclictest32586-21diskstats07:25:142
2476399311,22cyclictest31888-21/usr/sbin/munin11:05:232
2476399311,21cyclictest10266-21awk07:45:272
2476399311,20cyclictest17742-21head11:45:152
2476399311,18cyclictest13241-21tr10:20:242
2476399310,20cyclictest26410-21grep12:05:152
2475399319,3cyclictest31513-21kworker/0:211:06:410
2475399316,4cyclictest24639-21kworker/0:007:15:230
2475399312,17cyclictest31513-21kworker/0:211:15:120
2475399312,17cyclictest28454-21kworker/0:009:52:020
2475399312,11cyclictest14802-21kworker/0:207:57:420
2475399311,23cyclictest19302-21kworker/0:008:08:380
24770993018,3cyclictest0-21swapper/307:21:503
2477099301,14cyclictest0-21swapper/307:39:283
24770993010,4cyclictest4982-21ntpq11:15:213
24763993014,13cyclictest18179-21grep09:20:122
24763993014,13cyclictest18179-21grep09:20:122
2476399301,26cyclictest20766-21/usr/sbin/munin11:50:272
2476399301,26cyclictest16175-21/usr/sbin/munin11:40:252
2476399301,22cyclictest0-21swapper/212:04:062
2476399301,21cyclictest835-21runrttasks12:20:282
2476399301,17cyclictest28430-21cstates12:10:152
2476399301,16cyclictest1817-21cpuspeed_turbos11:10:122
2476399300,14cyclictest86950irq/126-enp1s0-07:33:182
2475899303,15cyclictest23475-40dnf08:15:431
2475399304,12cyclictest10302-21perf11:30:010
2475399301,10cyclictest28454-21kworker/0:009:41:380
24770992917,3cyclictest0-21swapper/308:51:213
24770992912,4cyclictest13054-21df_inode11:35:133
24770992910,3cyclictest0-21swapper/312:19:033
2476399294,14cyclictest87250irq/129-enp1s0-11:55:192
2476399292,19cyclictest801-21snmpd09:48:082
2476399291,25cyclictest904-21sssd_nss10:05:222
2476399291,25cyclictest7751-21hddtemp_smartct08:55:162
2476399291,25cyclictest27847-21/usr/sbin/munin07:15:132
2476399291,24cyclictest20334-21hddtemp_smartct08:10:162
2476399291,24cyclictest20334-21hddtemp_smartct08:10:162
24763992910,16cyclictest28552-21ntp_states10:55:222
24758992916,6cyclictest0-21swapper/111:16:561
2475399291,17cyclictest20808-21kworker/0:111:59:120
2475399291,16cyclictest12099-21kworker/0:210:24:110
24753992911,3cyclictest2724-21kworker/0:111:13:000
2475399291,11cyclictest5138-21kworker/0:110:14:310
2475399291,11cyclictest17667-21kworker/0:009:20:320
2475399291,11cyclictest17667-21kworker/0:009:20:320
2477099281,8cyclictest0-21swapper/310:04:253
24770992814,3cyclictest0-21swapper/308:46:373
2477099281,18cyclictest0-21swapper/309:45:163
24770992811,3cyclictest0-21swapper/312:22:133
2477099281,11cyclictest21946-21ntpq10:40:223
2477099280,3cyclictest0-21swapper/311:57:463
2476399283,22cyclictest16323-21ntp_states08:00:212
2476399282,23cyclictest7189-21ntp_states11:20:212
2476399281,24cyclictest31323-21hddtemp_smartct08:35:162
2476399281,24cyclictest29560-21/usr/sbin/munin08:30:242
2476399281,24cyclictest24323-21sshd09:33:402
2476399281,24cyclictest2423-21sshd08:44:412
2476399281,24cyclictest19366-21hddtemp_smartct10:35:172
2476399281,24cyclictest19366-21hddtemp_smartct10:35:172
2476399281,24cyclictest11588-21/usr/sbin/munin11:30:232
2476399281,24cyclictest10085-21irqstats09:00:172
2476399281,23cyclictest26810-21fschecks_time08:25:162
2475399282,15cyclictest24350-21kworker/0:109:39:540
2475399281,8cyclictest22782-21kworker/0:210:46:250
2475399281,17cyclictest9403-21kworker/0:007:51:500
2475399280,12cyclictest529-21kworker/0:109:56:400
24770992713,3cyclictest31641-21strings08:35:193
24770992713,3cyclictest14747-21ntp_states09:10:223
2477099271,12cyclictest0-21swapper/311:14:383
2477099270,13cyclictest0-21swapper/310:15:053
2476399271,23cyclictest5681-21/usr/sbin/munin08:50:162
2476399271,14cyclictest12050irq/123-ahci[0008:05:172
2475899279,10cyclictest28490-21cpu08:30:131
2475899275,13cyclictest0-21swapper/112:35:151
2475399275,4cyclictest21525-21kworker/0:108:18:290
2475399272,9cyclictest20100-21kworker/0:107:10:140
2475399272,8cyclictest29586-21kworker/0:107:21:210
2475399271,9cyclictest28763-21kworker/0:111:00:270
2475399271,9cyclictest25408-21kworker/0:212:04:110
2475399271,6cyclictest4000-21kworker/0:012:26:210
24753992710,3cyclictest20808-21kworker/0:111:52:080
2477099264,15cyclictest14418-21kworker/u8:111:49:233
2477099261,8cyclictest0-21swapper/309:32:033
24770992613,3cyclictest0-21swapper/308:56:303
24770992612,4cyclictest28185-21sh07:15:133
2477099261,14cyclictest0-21swapper/311:28:133
24770992611,3cyclictest0-21swapper/312:02:453
2477099261,11cyclictest0-21swapper/311:51:393
2477099261,11cyclictest0-21swapper/307:45:513
2477099260,11cyclictest0-21swapper/308:03:233
2475399262,4cyclictest9403-21kworker/0:007:48:400
2475399262,12cyclictest23755-21kworker/0:208:31:290
2475399262,10cyclictest12289-21kworker/0:211:32:560
2475399261,9cyclictest20415-21kworker/0:010:37:590
2475399261,9cyclictest20415-21kworker/0:010:37:590
24753992612,2cyclictest9403-21kworker/0:008:00:560
2475399261,13cyclictest10950-21kworker/0:109:07:330
2477099252,8cyclictest16886-21munin-run11:45:003
24770992512,3cyclictest0-21swapper/308:31:493
24770992510,3cyclictest0-21swapper/310:06:453
2477099250,17cyclictest0-21swapper/309:18:433
2475399256,4cyclictest30045-21kworker/0:112:16:430
2475399252,10cyclictest6805-21kworker/0:008:56:050
2475399252,10cyclictest25280-21kworker/0:010:58:390
2475399251,9cyclictest23755-21kworker/0:208:20:590
2475399251,12cyclictest1864-21awk07:30:010
24753992510,2cyclictest6233-21kworker/0:212:38:070
200902250,2sleep30-21swapper/310:35:263
200902250,2sleep30-21swapper/310:35:263
2477099245,2cyclictest0-21swapper/312:25:153
2477099245,14cyclictest86950irq/126-enp1s0-12:11:333
2477099244,2cyclictest87150irq/128-enp1s0-08:19:293
24770992412,3cyclictest0-21swapper/309:40:373
24770992412,3cyclictest0-21swapper/309:25:223
24770992410,3cyclictest0-21swapper/310:51:263
2475899244,10cyclictest11914-21ntpq07:50:191
24758992413,2cyclictest0-21swapper/108:45:181
2475899241,16cyclictest0-21swapper/109:00:181
2475899240,12cyclictest0-21swapper/111:08:451
2477099239,3cyclictest3140-21ntp_states07:30:223
2477099231,9cyclictest24068-21/usr/sbin/munin08:20:113
2477099231,10cyclictest31448-21ls09:50:133
2475899237,4cyclictest0-21swapper/110:26:591
2475899231,6cyclictest0-21swapper/112:05:091
2475899231,3cyclictest24610-21memory12:00:191
2475899231,16cyclictest0-21swapper/110:44:011
2475399239,3cyclictest11150-21kworker/0:309:16:290
2475399237,2cyclictest22782-21kworker/0:210:44:510
2475399235,6cyclictest1735-21kworker/0:212:22:390
2475399235,2cyclictest12099-21kworker/0:210:30:090
24753992311,3cyclictest27799-21kworker/0:012:13:210
24753992311,3cyclictest1663-21kworker/0:107:36:080
24753992310,3cyclictest2185-21kworker/0:208:45:510
2477099229,3cyclictest0-21swapper/307:42:543
2477099226,3cyclictest0-21swapper/311:33:473
2477099220,2cyclictest0-21swapper/310:23:083
2477099220,10cyclictest0-21swapper/311:20:593
24758992215,1cyclictest0-21swapper/109:25:161
2475899221,4cyclictest0-21swapper/111:25:221
2475899221,3cyclictest835-21runrttasks10:20:141
2475899221,18cyclictest30381-21df_inode07:20:131
2475899221,16cyclictest0-21swapper/112:33:171
2475899221,16cyclictest0-21swapper/112:25:371
2475899221,16cyclictest0-21swapper/111:48:331
2475899221,15cyclictest0-21swapper/112:21:071
2475899221,10cyclictest0-21swapper/109:08:091
24758992210,9cyclictest22009-21kworker/u8:109:43:101
2475399229,2cyclictest19302-21kworker/0:008:12:300
2475399229,2cyclictest19302-21kworker/0:008:12:300
2475399225,4cyclictest32304-21kworker/0:008:37:190
2477099219,3cyclictest0-21swapper/308:27:033
2475899217,7cyclictest0-21swapper/109:15:201
2475899216,3cyclictest2965-21kworker/u8:010:08:201
24758992114,1cyclictest0-21swapper/108:44:541
24758992113,3cyclictest0-21swapper/112:14:041
24758992113,3cyclictest0-21swapper/110:35:481
24758992113,3cyclictest0-21swapper/110:35:481
24758992113,1cyclictest0-21swapper/112:15:201
2475899211,18cyclictest0-21swapper/110:30:211
2475899211,17cyclictest27250-21turbostat.cron10:55:011
2475899211,17cyclictest24568-21sed10:45:261
2475899211,17cyclictest0-21swapper/111:20:121
2475899211,15cyclictest0-21swapper/111:31:181
2475899211,15cyclictest0-21swapper/109:35:511
2475899211,15cyclictest0-21swapper/109:20:461
2475899211,15cyclictest0-21swapper/109:20:461
2475899211,15cyclictest0-21swapper/108:27:161
24758992111,3cyclictest19995-21df08:10:141
24758992111,3cyclictest19995-21df08:10:141
2475899211,12cyclictest24070-21sa109:30:271
2475899211,11cyclictest28965-21pmu-power07:15:231
2475899210,16cyclictest0-21swapper/111:11:521
2475899210,10cyclictest0-21swapper/107:25:271
2475899209,3cyclictest9914-21cat10:15:131
2475899208,3cyclictest7728-21smart_sda07:40:241
2475899208,3cyclictest0-21swapper/111:41:351
2475899203,11cyclictest87150irq/128-enp1s0-10:55:211
2475899201,3cyclictest0-21swapper/109:10:211
24758992013,4cyclictest31236-21sed11:00:271
24758992013,2cyclictest0-21swapper/108:52:331
2475899201,17cyclictest0-21swapper/107:55:201
2475899201,17cyclictest0-21swapper/107:45:131
2475899201,17cyclictest0-21swapper/107:32:061
2475899201,11cyclictest30136-21ntp_states09:45:221
2475899201,11cyclictest24981-21memory08:20:211
2475899201,10cyclictest32739-21cat08:40:021
2475899200,15cyclictest0-21swapper/108:06:241
2475399208,3cyclictest23755-21kworker/0:208:25:310
2475899199,3cyclictest87150irq/128-enp1s0-10:10:181
2475899198,5cyclictest835-21runrttasks09:57:111
24758991912,2cyclictest0-21swapper/111:53:211
2475899191,15cyclictest7993-21memory08:55:181
2475899191,10cyclictest4567-21expr07:35:121
2475899190,17cyclictest0-21swapper/108:01:291
2475899181,9cyclictest0-21swapper/111:55:111
24758991813,3cyclictest32103-21kernelversion09:50:201
2475899181,2cyclictest0-21swapper/107:11:591
24758991812,3cyclictest3645-21df_inode10:00:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional