You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-21 - 17:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Aug 21, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
370322000,10sleep20-21swapper/207:05:122
51582176102,29sleep30-21swapper/307:06:103
52582175103,28sleep10-21swapper/107:07:291
53562166100,59sleep00-21swapper/007:08:450
2570121060,10sleep012-21ksoftirqd/011:57:320
55602770,5sleep15565-21unixbench_multi10:30:241
49612670,2sleep00-21swapper/009:42:240
85332650,1sleep30-21swapper/312:32:583
272682550,8sleep334-21ksoftirqd/309:59:203
278092540,1sleep20-21swapper/211:35:152
241012530,3sleep334-21ksoftirqd/311:32:263
326862520,1sleep30-21swapper/312:26:293
265232520,3sleep326524-21latency_hist09:35:003
17412520,1sleep00-21swapper/009:15:350
300162510,2sleep230014-21cpuspeed_turbos12:25:112
184042500,2sleep00-21swapper/011:51:590
313112490,1sleep20-21swapper/210:50:122
325252450,2sleep30-21swapper/308:10:173
135532410,1sleep30-21swapper/311:48:523
74082370,3chrt0-21swapper/110:08:251
155252370,1sleep30-21swapper/311:02:063
5731993412,7cyclictest24127-21which10:20:232
5731993412,5cyclictest16645-21cat07:35:002
5731993411,6cyclictest8562-21expr11:45:142
5732993314,16cyclictest1848-21sh09:15:413
5732993312,4cyclictest21574-21sh11:06:233
573299321,2cyclictest0-21swapper/307:23:533
5732993212,5cyclictest15825-21basename11:50:183
573299321,1cyclictest0-21swapper/307:33:203
5732993211,6cyclictest5141-21seq12:05:553
5731993213,16cyclictest25417-21sh09:10:062
5731993210,6cyclictest22122-21tune2fs09:55:142
5731993210,6cyclictest12477-21find08:40:102
573299319,5cyclictest8034-21sed11:45:013
573299318,5cyclictest766-21basename11:15:163
573299317,6cyclictest18382-21sed10:40:173
5732993112,3cyclictest11599-21sh10:59:553
5732993111,4cyclictest24053-21sed07:50:183
5732993111,4cyclictest13837-21sh12:36:333
573199319,6cyclictest26979-21acpi09:35:102
573199318,6cyclictest9259-21timerandwakeup08:30:242
5731993113,6cyclictest8249-21diskstats07:15:152
5731993112,4cyclictest2975-21sh11:40:222
573199311,18cyclictest0-21swapper/207:23:522
327592310,2chrt0-21swapper/110:26:521
201642310,2chrt0-21swapper/111:53:431
573299308,8cyclictest26947-21awk11:10:213
573299301,27cyclictest0-21swapper/307:18:163
573199307,7cyclictest17173-21grep07:35:172
573199307,7cyclictest17173-21grep07:35:162
573199302,8cyclictest3919-21apt-get10:30:132
5731993012,3cyclictest2431-21sh12:04:572
5731993011,3cyclictest30265-21sh10:49:382
573299299,5cyclictest6502-21cat07:10:163
573299299,5cyclictest3616-21sed08:20:003
573299298,7cyclictest32629-1kworker/u9:010:05:213
573299297,6cyclictest32641-21rm12:02:433
5732992912,3cyclictest9805-21sh09:21:543
5732992912,3cyclictest20691-21sh12:17:593
5732992912,3cyclictest20691-21sh12:17:593
5732992912,3cyclictest13365-21sh09:49:063
5732992911,4cyclictest18722-21sh11:28:493
5732992911,3cyclictest910-21sh10:50:583
5732992911,3cyclictest7077-21sh10:31:503
5732992911,3cyclictest13700-21sh10:13:003
573299291,10cyclictest15785-21ssh09:26:083
5732992910,6cyclictest34-21ksoftirqd/309:00:243
5732992910,3cyclictest9623-21/usr/sbin/munin12:10:173
5732992910,3cyclictest19918-21sh10:17:293
573199298,5cyclictest9865-21grep10:10:142
573199298,5cyclictest28653-21ls10:00:132
573199298,5cyclictest23445-21cat07:50:122
573199298,4cyclictest8901-21seq10:57:132
573199297,5cyclictest26671-21load11:10:192
573199297,4cyclictest7163-21gltestperf09:20:172
5731992921,6cyclictest0-21swapper/207:29:472
5731992912,3cyclictest18713-21sh09:52:422
573199291,14cyclictest5661-21wc12:30:232
5731992910,3cyclictest15788-21sh12:38:292
573299289,4cyclictest5779-21sed08:25:003
5732992822,4cyclictest0-21swapper/307:29:463
573299281,5cyclictest108162/usr/sbin/munin10:35:233
5732992811,4cyclictest25090-21diskmemload11:58:543
5732992811,3cyclictest2903-21sh10:29:263
573299281,10cyclictest14389-21latency_hist08:45:003
573299280,14cyclictest21011-21cat09:00:003
573199289,3cyclictest18152-21ssh11:51:422
573199288,5cyclictest3764-21cat08:20:122
573199288,4cyclictest25009-21sh11:33:212
573199287,9cyclictest52950irq/141-enp3s0-08:55:172
573199287,6cyclictest28-21ksoftirqd/212:05:232
573199286,8cyclictest0-21swapper/211:55:542
573199281,3cyclictest25090-21diskmemload10:17:142
5731992811,3cyclictest9380-21sh11:21:122
5731992811,3cyclictest29205-21sh12:24:442
5731992810,3cyclictest19616-21sh12:16:542
5731992810,3cyclictest19616-21sh12:16:532
5731992810,3cyclictest12291-21sh10:35:232
153412280,2sleep00-21swapper/012:14:210
573299279,3cyclictest27025-21sh10:46:223
573299279,3cyclictest1498-21sh10:04:163
573299271,9cyclictest1274-21/usr/sbin/munin09:40:293
5732992711,3cyclictest25090-21diskmemload12:23:333
5732992710,4cyclictest25090-21diskmemload09:13:553
573199279,6cyclictest8518-21perl09:45:152
573199279,6cyclictest24153-21pluginstate09:05:212
573199279,3cyclictest22534-21unixbench_singl09:00:252
573199278,6cyclictest11633-21cat08:35:242
573199277,7cyclictest0-21swapper/212:13:452
573199274,9cyclictest0-21swapper/211:00:422
573199271,24cyclictest0-21swapper/208:29:502
5731992712,3cyclictest31193-21unixbench_multi10:25:242
5731992712,3cyclictest25090-21diskmemload09:44:092
573199271,13cyclictest14880-21ls09:25:262
573199271,10cyclictest27274-21ls08:00:012
5731992710,4cyclictest307-21expr11:15:122
5731992710,3cyclictest24888-21sh09:33:322
5731992710,3cyclictest21216-21sh10:42:182
5731992710,3cyclictest19841-21sh11:29:552
5730992723,2cyclictest0-21swapper/107:20:321
573099271,25cyclictest0-21swapper/107:19:371
572999271,24cyclictest0-21swapper/008:29:440
572999271,24cyclictest0-21swapper/007:53:410
572999271,24cyclictest0-21swapper/007:33:520
572999271,24cyclictest0-21swapper/007:22:290
572999271,24cyclictest0-21swapper/007:18:410
573299269,3cyclictest12307-21sh11:24:073
5732992610,3cyclictest25090-21diskmemload11:36:093
5732992610,3cyclictest15262-21gltestperf09:50:153
5731992610,3cyclictest2258-21sh09:16:062
5730992623,2cyclictest0-21swapper/107:26:141
573099262,13cyclictest75-21kswapd010:50:111
573099261,24cyclictest0-21swapper/108:29:081
573099261,24cyclictest0-21swapper/107:34:091
573099261,1cyclictest0-21swapper/107:54:241
572999261,23cyclictest0-21swapper/008:23:310
5729992611,6cyclictest13510-21expr09:25:140
572999260,24cyclictest0-21swapper/007:25:300
573299258,4cyclictest18950irq/143-ahci[0008:50:013
573299251,8cyclictest27538-21apt-get08:00:123
573299251,23cyclictest0-21swapper/308:29:513
5732992510,3cyclictest636-21runrttasks07:57:573
573199257,4cyclictest0-21swapper/208:15:152
573199252,8cyclictest18849-21apt-get11:05:112
573199251,11cyclictest14631-21apt-get08:45:122
573199251,10cyclictest5941-21ssh10:06:582
573099251,1cyclictest0-21swapper/107:55:091
5729992518,5cyclictest8884-21fschecks_time11:45:160
40922250,11sleep122-21ksoftirqd/111:41:081
573199241,9cyclictest21107-21apt-get07:45:132
173792240,3sleep10-21swapper/112:15:191
173792240,3sleep10-21swapper/112:15:181
573099232,14cyclictest10229-21apt-get10:35:031
5730992314,6cyclictest610-21ssh10:03:241
5729992316,5cyclictest6730-21sh10:31:300
5729992316,5cyclictest11690-21sh11:00:000
5729992314,6cyclictest12477-21diskstats11:00:130
215122230,2sleep10-21swapper/110:42:371
176502230,1sleep10-21swapper/110:15:241
126492230,0sleep0131rcuc/010:35:300
573199229,3cyclictest30532-21netstat08:05:182
573099222,15cyclictest9420-21ssh10:10:121
572999227,5cyclictest3103-21cpuspeed_turbos12:05:130
5729992211,7cyclictest28050-21ntpq09:35:200
572999221,10cyclictest25207-21http11:10:000
230412220,9sleep122-21ksoftirqd/112:20:041
204932220,1sleep00-21swapper/010:41:350
83022211,5sleep08316-21diskstats08:30:140
573299215,11cyclictest26401-21apt-get09:35:043
5732992113,6cyclictest11602-21sed08:35:253
573099212,13cyclictest75-21kswapd011:00:091
573099212,13cyclictest6878-21apt-get11:20:121
5730992111,8cyclictest4059-21cut12:05:231
5729992119,1cyclictest0-21swapper/009:07:020
5729992119,1cyclictest0-21swapper/007:55:110
5729992111,6cyclictest52950irq/141-enp3s0-09:20:250
5732992011,6cyclictest278-21systemd-journal08:31:573
5731992017,2cyclictest0-21swapper/208:02:392
573099207,10cyclictest0-21swapper/112:35:211
573099202,12cyclictest75-21kswapd012:30:091
573099202,12cyclictest29886-21apt-get12:25:111
5730992018,1cyclictest0-21swapper/108:20:401
5730992014,4cyclictest17280-21grep08:50:171
5730992011,7cyclictest8312-21df_inode08:30:131
5730992011,7cyclictest26431-21cat09:10:171
5730992011,5cyclictest52750irq/139-enp3s0-09:33:111
5729992014,4cyclictest32531-21sh10:26:380
5729992014,4cyclictest28354-21sh11:11:150
5729992014,4cyclictest12-21ksoftirqd/011:40:010
5729992013,4cyclictest24591-21sh09:33:130
5729992012,6cyclictest29408-21ntp_kernel_pll_10:00:190
5729992012,6cyclictest2784-21cat10:05:130
5729992012,6cyclictest2693-21ls12:05:010
5729992012,6cyclictest22354-21awk09:55:160
5729992012,6cyclictest16595-21cat08:50:010
5729992012,6cyclictest16121-21apt10:15:100
5729992011,7cyclictest10558-21cron12:35:000
5729992011,6cyclictest30177-21cat12:25:110
5729992011,6cyclictest21567-21perl07:45:150
5729992010,6cyclictest52950irq/141-enp3s0-11:25:120
573299192,2cyclictest141rcu_preempt10:21:363
5732991913,3cyclictest24372-21switchtime09:05:233
5732991912,5cyclictest20301-21sed07:40:243
5731991912,5cyclictest32076-21cat08:10:132
5731991911,6cyclictest19944-21ls07:40:202
573099199,8cyclictest560-21dbus-daemon07:15:011
573099199,7cyclictest23899-21ntpq10:20:201
573099192,11cyclictest640-21snmpd11:25:121
573099191,8cyclictest25090-21diskmemload10:46:161
573099191,7cyclictest0-21swapper/109:45:011
5730991911,6cyclictest20957-21head11:30:151
5730991911,5cyclictest22-21ksoftirqd/109:35:101
5730991910,6cyclictest2642-21tr08:15:231
5730991910,6cyclictest22616-21cat11:55:201
5730991910,6cyclictest22514-21unixbench_singl09:00:241
572999199,7cyclictest32716-21ntpq08:10:200
572999199,7cyclictest32602-21grep11:15:110
572999195,3cyclictest12215-21cron08:40:010
572999192,12cyclictest16770-21apt-get08:50:130
5729991914,3cyclictest22304-21threads09:00:240
5729991913,4cyclictest52950irq/141-enp3s0-11:32:080
5729991913,4cyclictest29405-21sh10:48:470
5729991913,4cyclictest10689-21mailstats10:10:220
5729991912,5cyclictest7926-21seq09:45:100
5729991911,6cyclictest23080-21cat10:20:130
5729991911,6cyclictest20619-21seq08:57:570
5729991910,7cyclictest24503-21python312:20:230
573099188,8cyclictest10397-21ssh12:10:161
573099188,7cyclictest22-21ksoftirqd/111:45:131
573099182,11cyclictest26687-21apt-get11:35:031
573099182,11cyclictest14694-21apt-get09:50:121
573099182,10cyclictest13403-21ssh09:25:151
573099181,7cyclictest25090-21diskmemload09:56:401
5730991813,3cyclictest7954-21/usr/sbin/munin09:45:211
5730991813,3cyclictest6066-21sh10:55:161
5730991812,4cyclictest25185-21dump-pmu-power11:14:551
5730991811,5cyclictest653-21cat09:15:191
5730991811,5cyclictest17437-21sed07:35:191
5730991811,5cyclictest17437-21sed07:35:181
5730991811,4cyclictest636-21runrttasks12:01:041
5730991810,6cyclictest689-21head11:15:151
5730991810,6cyclictest20500-21seq08:56:251
573099180,4cyclictest13411-21sh08:40:211
572999187,6cyclictest52950irq/141-enp3s0-12:35:110
5729991813,3cyclictest427-21sh10:50:300
5729991813,3cyclictest20766-21sh09:54:450
5729991812,4cyclictest20286-21sh12:17:350
5729991812,4cyclictest20286-21sh12:17:340
5729991811,5cyclictest6093-21df07:10:120
5729991811,5cyclictest30629-21ls08:05:200
5729991811,5cyclictest2426-21ntpq08:15:190
5729991811,5cyclictest12652-21expr08:40:120
5729991811,4cyclictest1665-21sh11:40:110
572999181,13cyclictest6907-21ssh11:20:120
5729991810,6cyclictest26818-21sensors09:10:220
5729991810,5cyclictest28840-21cat08:00:240
5729991810,5cyclictest17889-21ls07:35:250
5729991810,5cyclictest17889-21ls07:35:240
573299179,6cyclictest30141-21gltestperf08:05:153
5732991712,3cyclictest21082-21/usr/sbin/munin07:45:233
573199178,6cyclictest6538-21missed_timers07:10:172
573099179,6cyclictest560-21dbus-daemon08:14:591
573099179,6cyclictest23795-21basename09:05:171
573099179,6cyclictest20242-21wc07:40:241
573099171,13cyclictest21637-21head07:45:161
5730991710,5cyclictest6834-21expr09:20:131
5730991710,5cyclictest15530-21ntpq08:45:191
5730991710,4cyclictest18708-21sh11:05:031
5730991710,4cyclictest10137-21apt-key08:35:001
572999171,11cyclictest18968-21apt-get07:40:110
5732991611,3cyclictest636-21runrttasks07:35:153
5732991611,3cyclictest636-21runrttasks07:35:153
5731991611,3cyclictest17276-21gltestperf08:50:162
573099168,6cyclictest560-21dbus-daemon08:05:001
573299151,12cyclictest0-21swapper/308:50:073
5730991512,2cyclictest0-21swapper/108:05:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional