You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 08:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Tue Dec 30, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30533702123115,5sleep20-21swapper/219:08:562
3053411211758,55sleep30-21swapper/319:09:293
305317628164,9sleep10-21swapper/119:06:301
304833727157,6sleep00-21swapper/019:05:010
3053584992623,1cyclictest0-21swapper/323:05:003
3053584992617,4cyclictest3139394-21taskset20:30:003
305358499241,19cyclictest3065800-21taskset19:17:373
305358499237,7cyclictest3240114-21turbostat22:05:003
3053584992312,4cyclictest31134642sleep320:05:123
305358499229,3cyclictest0-21swapper/319:44:543
30562052210,2sleep10-21swapper/119:10:181
305358499211,8cyclictest3366549-21turbostat00:05:003
3053584992118,1cyclictest0-21swapper/322:00:013
3053584992118,1cyclictest0-21swapper/320:40:273
33294922201,4sleep03329495-21sshd23:26:380
33294922201,4sleep03329495-21sshd23:26:370
305358499207,2cyclictest0-21swapper/320:04:243
305358499206,3cyclictest3176685-21turbostat21:05:003
3053584992017,1cyclictest0-21swapper/300:25:183
305358499201,6cyclictest228-1kworker/3:1H+kblockd19:10:083
305358499201,15cyclictest3334961-21turbostat23:35:003
305358499197,10cyclictest15850irq/127-ahci[0000:00:17.0]21:09:553
3053584991916,1cyclictest0-21swapper/323:55:283
3053584991916,1cyclictest0-21swapper/322:20:003
3053584991916,1cyclictest0-21swapper/300:20:183
3053584991915,2cyclictest0-21swapper/323:35:153
3053584991915,2cyclictest0-21swapper/321:10:123
305358499191,14cyclictest3102680-21turbostat19:55:003
305358499191,14cyclictest0-21swapper/321:15:133
305358499191,13cyclictest0-21swapper/300:05:123
305358499191,11cyclictest3355698-21taskset23:51:533
33140562183,10sleep03314072-21munin-run23:15:000
305358499188,2cyclictest15850irq/127-ahci[0000:00:17.0]20:50:083
305358499187,4cyclictest32248472sleep321:50:133
3053584991815,1cyclictest0-21swapper/322:55:183
3053584991815,1cyclictest0-21swapper/320:15:213
3053584991814,2cyclictest0-21swapper/322:20:293
3053584991814,2cyclictest0-21swapper/322:10:153
305358499181,13cyclictest3224338-21sleep21:49:593
305358499181,13cyclictest3145173-21chrt20:35:123
305358499181,13cyclictest0-21swapper/321:35:123
305358499181,13cyclictest0-21swapper/300:30:133
305358499181,13cyclictest0-21swapper/300:10:133
305358499181,12cyclictest3208528-21chrt21:34:533
305358499181,12cyclictest3171453-21sleep21:00:013
3053584991811,2cyclictest0-21swapper/320:11:173
305358499181,11cyclictest3202917-21taskset21:25:433
305358499181,11cyclictest3202917-21taskset21:25:433
305358499181,10cyclictest3266598-21taskset22:30:013
305358499181,10cyclictest3129163-21taskset20:20:063
305358499180,13cyclictest3097447-21turbostat19:50:003
33968762171,10sleep13397012-21/usr/sbin/munin00:30:201
30704232170,2sleep10-21swapper/119:20:241
3053584991714,1cyclictest0-21swapper/320:35:003
3053584991714,1cyclictest0-21swapper/319:35:193
3053584991713,2cyclictest0-21swapper/323:40:173
3053584991713,2cyclictest0-21swapper/322:50:003
305358499171,13cyclictest3403591-21sleep00:40:003
305358499171,13cyclictest0-21swapper/322:50:203
305358499171,12cyclictest3350719-21sleep23:50:003
305358499171,12cyclictest0-21swapper/319:25:133
305358499170,14cyclictest0-21swapper/322:40:273
31918482161,10sleep03191965-21gzip21:15:230
305358499162,9cyclictest3214419-21taskset21:40:143
305358499161,9cyclictest3377302-21taskset00:15:063
305358499161,9cyclictest3313881-21taskset23:12:453
305358499161,9cyclictest3307796-21taskset23:05:203
305358499161,9cyclictest3271854-21taskset22:35:063
305358499161,9cyclictest3265993-21taskset22:25:323
305358499161,9cyclictest3160394-21taskset20:46:433
305358499161,9cyclictest3070571-21taskset19:20:263
305358499161,10cyclictest3314577-21taskset23:15:143
305358499161,10cyclictest3081511-21taskset19:31:353
305358499160,13cyclictest0-21swapper/323:20:193
305358499160,13cyclictest0-21swapper/321:20:103
305358499160,12cyclictest0-21swapper/322:05:143
33235942151,5sleep13323639-21ntpq23:20:221
32884462151,4sleep13288562-21irqstats22:50:161
32832062151,9sleep23283408-21cat22:45:162
32442782151,9sleep03244363-21ld22:05:220
32074842151,9sleep13207534-21ntpq21:30:191
31671332151,5sleep23168741-21cut20:55:172
30863012151,3sleep00-21swapper/019:35:270
305358499150,13cyclictest0-21swapper/323:25:133
305358499150,13cyclictest0-21swapper/323:25:133
305358499150,12cyclictest0-21swapper/319:55:163
33760242141,4sleep23376070-21ntpq00:10:212
32724232141,9sleep13273237-21grep22:35:171
32390932141,8sleep03239143-21ntpq22:00:210
31757022141,8sleep03175755-21ld21:00:230
33339752131,7sleep03334027-21ld23:30:230
32627082130,8sleep03264396-21grep22:25:180
31830952131,7sleep23184521-21kthreadcore21:10:162
30915332131,7sleep13091753-21sed19:40:251
32456332121,5sleep13245652-21apt-get22:10:121
32298112120,2sleep00-21swapper/021:55:070
32193622120,8sleep03219366-21find21:45:090
31930162122,5sleep23193035-21apt-get21:20:102
31142222121,7sleep03115821-21cut20:05:170
30545772121,6sleep03056202-21kthreadcore19:10:180
33988662110,7sleep23399080-21smartctl00:35:182
33970132110,7sleep23397057-21collect200:30:212
33100382111,6sleep23311107-21kthreadcore23:10:172
32757442111,6sleep03275865-21ntpq22:35:200
31348902111,6sleep03135059-21cut20:25:170
31300872111,6sleep13131739-21kthreadcore20:20:171
31184612111,4sleep23118509-21munin-run20:10:002
31029272111,2sleep1171rcu_preempt19:55:121
30768332111,5sleep03077093-21df_inode19:30:140
33655902101,4sleep13365647-21ntp_states00:00:221
33521282101,4sleep23353797-21cut23:50:172
33198632101,4sleep23319987-21perl23:20:152
33150332101,5sleep23316623-21grep23:15:182
32985982101,4sleep03298801-21diskstats23:00:130
32774572101,4sleep13277715-21df_inode22:40:141
32706632101,4sleep03270716-21ntp_states22:30:230
32666122101,3sleep20-21swapper/222:30:132
32391972101,5sleep23239277-21tr22:00:232
32245692101,4sleep13224588-21apt-get21:50:111
32141192101,4sleep03214138-21apt-get21:40:120
31808082101,4sleep03180896-21ntpq21:05:210
31422842101,4sleep23143451-21memory20:30:162
31344032101,4sleep13134659-21df20:25:141
31317422101,4sleep23132960-21memory20:20:182
31230852101,4sleep23123302-21users20:10:282
30979922101,5sleep03097697-21/usr/sbin/munin19:50:140
30964192101,4sleep23096465-21ntpq19:45:192
30882712105,2sleep03089308-21kthreadcore19:40:160
3402869291,4sleep13402944-21grep00:35:291
3370898290,2sleep10-21swapper/100:05:231
3367338291,4sleep23367741-21grep00:05:162
3345174290,1sleep10-21swapper/123:41:461
3336920293,2sleep23338676-21kthreadcore23:35:182
3323543291,3sleep03323592-21ntpq23:20:210
3309585291,4sleep13310037-21grep23:10:161
3308778291,3sleep13308799-21munin-run23:10:001
3302824290,1sleep20-21swapper/223:00:272
3297361291,5sleep00-21swapper/022:55:240
3297258291,4sleep23297307-21ntpq22:55:222
3291981291,4sleep23292030-21ntpq22:50:202
3260029291,4sleep13260118-21ntpq22:20:221
3259296291,4sleep03260028-21cat22:20:210
3244364291,3sleep13244406-21ntpq22:05:231
3228586291,4sleep23228641-21modprobe21:50:202
3220788291,4sleep23222484-21grep21:45:152
3198803291,4sleep13199148-21cat21:25:181
3198803291,4sleep13199148-21cat21:25:181
3192008291,1sleep20-21swapper/221:15:252
3186443291,3sleep13186494-21ntpq21:10:181
3182426291,3sleep03181937-21apt-get21:10:020
3176997291,3sleep23177215-21df_inode21:05:152
3166423291,3sleep13166442-21apt-get20:55:121
3161448291,4sleep23161772-21tr20:50:162
3156989293,3sleep20-21swapper/220:45:192
3147851291,4sleep03149004-21memory20:35:190
3145151291,3sleep23145170-21apt-get20:35:132
3129717291,4sleep03130085-21grep20:20:160
3128886291,4sleep03128953-21systemd-run20:20:000
3122742290,1sleep10-21swapper/120:10:231
3118746291,4sleep03118902-21cstates20:10:140
3112203291,4sleep03112252-21ntpq20:00:200
3085405291,4sleep13085897-21needreboot19:35:201
3065291291,4sleep03065371-21timerandwakeup19:15:270
3058754291,4sleep23059139-21munin-node19:15:012
3399177281,3sleep0221rcuc/000:35:200
3398031281,3sleep03398049-21munin-run00:35:000
3387849281,2sleep20-21swapper/200:25:122
3387420280,1sleep10-21swapper/100:22:551
3386898280,4sleep20-21swapper/200:20:272
3381324281,3sleep23381377-21modprobe00:15:222
3362087280,4sleep03363470-21grep00:00:170
3350949280,3sleep13350968-21apt-get23:50:121
3346263281,3sleep20-21swapper/223:45:162
3334952280,1sleep10-21swapper/123:35:011
3329074280,1sleep20-21swapper/223:25:242
3329074280,1sleep20-21swapper/223:25:242
3326176282,3sleep10-21swapper/123:25:151
3326176282,3sleep10-21swapper/123:25:151
3314835280,4sleep134-21ksoftirqd/123:15:171
3307753281,3sleep23307794-21ntpq23:05:202
3293783281,3sleep13294180-21kthreadcore22:55:171
3292089281,3sleep03292174-21ps22:50:230
3287115280,4sleep10-21swapper/122:45:271
3281532281,3sleep2421rcuc/222:40:232
3277435281,2sleep00-21swapper/022:40:130
3254724280,1sleep20-21swapper/222:15:182
3250927280,4sleep00-21swapper/022:15:120
3249434280,1sleep00-21swapper/022:10:180
3242024280,5sleep23243747-21cut22:05:202
3235092281,2sleep13235350-21df_inode22:00:131
3230386281,3sleep23231415-21cut21:55:172
3213835281,3sleep0553-21dbus-daemon21:40:000
3203026280,1sleep20-21swapper/221:27:082
3203026280,1sleep20-21swapper/221:27:082
3176398280,5sleep134-21ksoftirqd/121:01:381
3171068280,4sleep00-21swapper/020:55:390
3165196281,3sleep03165373-21timerwakeupswit20:50:260
3155706280,3sleep13155724-21apt-get20:45:141
3155409280,1sleep00-21swapper/020:44:370
3139615281,2sleep03139634-21apt-get20:30:100
3138662281,3sleep23138735-21threads20:25:262
3109861281,3sleep13111603-21grep20:00:171
3107907281,3sleep0800-21snmpd19:59:470
3092766281,3sleep00-21swapper/019:45:130
3065099280,4sleep13065217-21grep19:15:241
3392822270,3sleep11-21systemd00:30:001
3392632272,2sleep03053578-21cyclictest00:27:390
3382298270,1sleep00-21swapper/000:19:370
3361208271,3sleep03361401-21latency_hist00:00:010
3359992271,3sleep13360132-21netstat23:55:211
3356206270,2sleep20-21swapper/223:55:142
3355690270,1sleep00-21swapper/023:51:470
3334669270,1sleep20-21swapper/223:31:292
3273238270,3sleep23274926-21cut22:35:182
3256681271,2sleep20-21swapper/222:20:192
3219358270,1sleep10-21swapper/121:45:061
3218946270,1sleep20-21swapper/221:42:592
3209090271,3sleep10-21swapper/121:35:161
3202388271,3sleep00-21swapper/021:25:250
3202388271,3sleep00-21swapper/021:25:250
3197973270,4sleep10-21swapper/121:24:371
3187743270,2sleep13187762-21apt-get21:15:121
3150449270,2sleep10-21swapper/120:40:131
3113083270,1sleep20-21swapper/220:03:312
3098205271,3sleep13098302-21ps19:50:151
3097718271,2sleep20-21swapper/219:50:122
3096277271,2sleep13096418-21ntpq19:45:181
3086954270,1sleep20-21swapper/219:39:342
3080684271,3sleep20-21swapper/219:30:202
3071549270,2sleep00-21swapper/019:25:130
3066271270,2sleep20-21swapper/219:20:132
3381233261,2sleep13381322-21ntpq00:15:211
3372024260,4sleep00-21swapper/000:10:070
3365253260,2sleep23365481-21strings00:00:192
3345690260,1sleep10-21swapper/123:45:111
3345271260,1sleep20-21swapper/223:42:592
3340449260,1sleep00-21swapper/023:40:050
3308684260,4sleep00-21swapper/023:08:500
3298511260,1sleep10-21swapper/123:00:101
3271309260,4sleep10-21swapper/122:31:191
3266050260,1sleep10-21swapper/122:26:151
3265219261,1sleep20-21swapper/222:25:202
3255624260,1sleep10-21swapper/122:17:291
3246360261,2sleep20-21swapper/222:10:172
3214954260,2sleep10-21swapper/121:40:171
3207304260,2sleep03207478-21strings21:30:180
3203539260,2sleep20-21swapper/221:30:122
3181705260,1sleep10-21swapper/121:07:311
3176333260,1sleep20-21swapper/221:00:462
3165018261,2sleep10-21swapper/120:50:231
3151584261,2sleep20-21swapper/220:40:182
3140638261,2sleep10-21swapper/120:30:151
3123933260,2sleep20-21swapper/220:15:132
3103500261,2sleep20-21swapper/219:55:172
3091877260,3sleep20-21swapper/219:40:382
3076411260,1sleep20-21swapper/219:28:192
3076322260,4sleep10-21swapper/119:27:101
3386495250,1sleep00-21swapper/000:20:210
3377004250,3sleep10-21swapper/100:14:291
3369390250,1sleep00-21swapper/000:05:180
3345981250,1sleep00-21swapper/023:45:140
3335664250,1sleep00-21swapper/023:35:160
3209273250,1sleep20-21swapper/221:35:162
3197028250,2sleep00-21swapper/021:20:200
3156091250,2sleep00-21swapper/020:45:170
3149914250,3sleep10-21swapper/120:36:521
3127726250,2sleep10-21swapper/120:15:201
3118264250,3sleep10-21swapper/120:07:291
3081527252,1sleep10-21swapper/119:31:471
3057886250,1sleep20-21swapper/219:10:232
3330301240,1sleep10-21swapper/123:30:151
3314617240,1sleep00-21swapper/023:15:160
3282528240,1sleep00-21swapper/022:45:020
3229817240,1sleep10-21swapper/121:55:121
3224901240,1sleep00-21swapper/021:50:140
3066551240,1sleep00-21swapper/019:20:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional