You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 20:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Sat Nov 29, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29077159913430,1338cyclictest33776042sleep312:25:023
29077159912530,1250cyclictest32391272sleep311:00:013
2907715992840,281cyclictest3303786-21turbostat11:44:543
30226602420,2sleep0151ktimers/008:35:210
33283882221,5sleep03328421-21apt-get11:55:130
2907715991916,1cyclictest0-21swapper/312:05:013
290771599185,9cyclictest30847642sleep309:23:003
290771599185,10cyclictest2922394-21turbostat07:20:003
2907715991815,1cyclictest0-21swapper/312:20:183
31565142170,4sleep03156522-21ssh10:05:420
31565142170,4sleep03156522-21ssh10:05:410
290771599171,13cyclictest3181465-21taskset10:22:383
32570752161,5sleep20-21swapper/211:10:202
290771599161,5cyclictest32871542sleep311:29:353
290771599161,3cyclictest0-21swapper/311:20:133
290771599161,3cyclictest0-21swapper/310:55:143
290771599161,3cyclictest0-21swapper/308:55:133
2907715991613,1cyclictest0-21swapper/311:30:193
2907715991613,1cyclictest0-21swapper/310:50:163
2907715991613,1cyclictest0-21swapper/310:45:243
2907715991613,1cyclictest0-21swapper/310:15:253
2907715991613,1cyclictest0-21swapper/308:25:173
290771599161,12cyclictest3191977-21taskset10:30:173
290771599161,12cyclictest3117842-21turbostat09:45:003
290771599161,12cyclictest0-21swapper/312:35:133
290771599151,3cyclictest0-21swapper/312:15:153
290771599151,3cyclictest0-21swapper/312:05:143
290771599151,3cyclictest0-21swapper/311:10:143
290771599151,3cyclictest0-21swapper/311:05:143
290771599151,3cyclictest0-21swapper/309:45:123
290771599151,3cyclictest0-21swapper/308:35:143
290771599151,3cyclictest0-21swapper/308:20:133
290771599151,3cyclictest0-21swapper/307:40:133
2907715991513,1cyclictest0-21swapper/309:32:473
2907715991513,1cyclictest0-21swapper/309:15:083
2907715991513,1cyclictest0-21swapper/308:52:553
290771599151,2cyclictest0-21swapper/310:10:153
290771599151,2cyclictest0-21swapper/310:10:143
290771599151,2cyclictest0-21swapper/309:50:143
290771599151,2cyclictest0-21swapper/307:50:123
290771599151,2cyclictest0-21swapper/307:50:123
290771599151,2cyclictest0-21swapper/307:35:173
290771599151,2cyclictest0-21swapper/307:20:153
2907715991512,1cyclictest0-21swapper/309:56:473
2907715991512,1cyclictest0-21swapper/309:56:473
2907715991512,1cyclictest0-21swapper/309:03:413
290771599151,11cyclictest3336141-21turbostat11:59:593
2907715991511,1cyclictest0-21swapper/308:10:013
290771599151,10cyclictest3150260-21turbostat10:05:013
290771599151,10cyclictest2973625-21chrt08:00:133
290771599151,10cyclictest2967275-21chrt07:55:133
33105742141,4sleep20-21swapper/211:42:052
290771599141,9cyclictest2910034-21taskset07:10:173
290771599141,9cyclictest0-21swapper/311:15:123
290771599141,3cyclictest0-21swapper/312:10:133
290771599141,3cyclictest0-21swapper/310:25:143
290771599141,3cyclictest0-21swapper/310:05:143
290771599141,3cyclictest0-21swapper/310:05:133
290771599141,3cyclictest0-21swapper/309:35:133
290771599141,3cyclictest0-21swapper/309:05:153
290771599141,3cyclictest0-21swapper/308:10:153
290771599141,2cyclictest0-21swapper/312:32:593
290771599141,2cyclictest0-21swapper/311:50:133
290771599141,2cyclictest0-21swapper/311:45:153
290771599141,2cyclictest0-21swapper/310:40:143
290771599141,2cyclictest0-21swapper/310:35:133
290771599141,2cyclictest0-21swapper/308:45:143
290771599141,2cyclictest0-21swapper/308:40:133
290771599141,2cyclictest0-21swapper/307:48:363
290771599141,2cyclictest0-21swapper/307:30:133
2907715991412,1cyclictest0-21swapper/308:16:323
2907715991411,1cyclictest0-21swapper/307:27:083
290771599140,5cyclictest3085896-21taskset09:25:053
290771599140,3cyclictest0-21swapper/311:35:133
290771599140,2cyclictest0-21swapper/308:32:413
290771599131,4cyclictest0-21swapper/309:10:133
190898130,9rtkit-daemon3126956-21irqcore09:50:170
30724062122,4sleep23074422-21kthreadcore09:15:192
29075542120,3sleep30-21swapper/307:09:423
190898120,8rtkit-daemon3269176-21sed11:15:251
190898120,7rtkit-daemon3352097-21ssh12:09:300
190898120,4rtkit-daemon667-21snmpd10:58:420
190898120,4rtkit-daemon473-21gmain10:29:170
33202672111,6sleep03320289-21apt-get11:50:130
30771312111,7sleep03077139-21ssh09:19:060
30304592111,7sleep13030588-21irqcore08:45:161
29865612110,8sleep12987064-21cut08:10:191
29735892111,5sleep22973622-21apt-get08:00:132
190898110,7rtkit-daemon3386050-21apt-get12:30:140
190898110,7rtkit-daemon3164887-21ssh10:11:290
190898110,7rtkit-daemon3164887-21ssh10:11:280
190898110,4rtkit-daemon3245586-21ssh11:01:490
190898110,3rtkit-daemon667-21snmpd07:08:251
33425142100,3sleep00-21swapper/012:00:580
32796592101,5sleep03279811-21df11:25:170
29984102101,3sleep22998648-21sed08:20:002
29162992101,5sleep12916321-21apt-get07:15:121
190898100,7rtkit-daemon2869580-21kworker/0:2+kacpid10:46:510
190898100,6rtkit-daemon3303367-21ssh11:39:002
190898100,5rtkit-daemon3228357-21ntp_states10:50:210
190898100,4rtkit-daemon3205778-21ssh10:38:020
190898100,3rtkit-daemon2914074-21munin-node12:18:320
190898100,2rtkit-daemon3293538-21ssh11:30:421
3294272290,6sleep03294280-21ssh11:32:250
3182760291,4sleep13182797-21apt-get10:25:141
3042938291,4sleep13043087-21df_inode08:55:131
3042902291,4sleep03042935-21apt-get08:55:120
3040208290,1sleep00-21swapper/008:50:210
3030123291,4sleep23030145-21apt-get08:45:142
3017688291,6sleep23017827-21gltestperf08:35:162
2948425291,4sleep12948458-21apt-get07:40:131
2908295291,5sleep12908881-21/usr/sbin/munin07:10:171
19089890,5rtkit-daemon438-21wpa_supplicant09:09:200
19089890,5rtkit-daemon3087024-21irqrtprio09:25:161
19089890,5rtkit-daemon2960623-21taskset07:48:401
19089890,4rtkit-daemon3383518-21ssh12:25:290
19089890,4rtkit-daemon3027466-21cut08:40:181
19089890,3rtkit-daemon3212837-21ssh10:40:390
19089890,3rtkit-daemon3067010-21ntp_states09:10:200
19089890,1rtkit-daemon0-21swapper/012:13:130
3400585286,1sleep2433-21hwrng12:37:112
3375920281,1sleep2171rcu_preempt12:22:162
3368337280,1sleep20-21swapper/212:18:282
3352935281,4sleep13352974-21apt-get12:10:131
3327233280,3sleep10-21swapper/111:53:111
3318799280,3sleep20-21swapper/211:47:222
3293660280,3sleep23293668-21ssh11:30:592
3238846280,5sleep13238909-21sendmail-msp11:00:011
3229100281,4sleep23229108-21ssh10:51:072
3193028280,5sleep03195017-21kthreadcore10:30:190
3181721281,4sleep03181729-21ssh10:23:130
3172613280,3sleep10-21swapper/110:15:411
3131934281,4sleep13131942-21ssh09:50:311
3108487280,1sleep10-21swapper/109:37:011
3085955281,3sleep23085992-21apt-get09:25:132
3077864281,4sleep23077896-21cat09:20:112
3068536280,3sleep13068542-21ssh09:12:441
3056437281,4sleep13057331-21grep09:05:191
3049237281,4sleep13049389-21df_inode09:00:151
3045553281,4sleep23047689-21grep08:55:182
2960990281,3sleep02961022-21apt-get07:50:140
2960990281,3sleep02961022-21apt-get07:50:130
2955877280,5sleep02957038-21grep07:45:160
2941470281,4sleep12941489-21munin-run07:35:001
2933028281,4sleep22933885-21sort07:25:192
19089880,5rtkit-daemon473-21gmain10:17:102
19089880,5rtkit-daemon1928-21sendmail-mta12:21:340
19089880,5rtkit-daemon1928-21sendmail-mta09:42:341
19089880,5rtkit-daemon1928-21sendmail-mta08:25:251
19089880,4rtkit-daemon667-21snmpd08:37:301
19089880,4rtkit-daemon667-21snmpd07:28:481
19089880,4rtkit-daemon438-21wpa_supplicant11:21:172
19089880,4rtkit-daemon438-21wpa_supplicant07:50:161
19089880,4rtkit-daemon438-21wpa_supplicant07:50:151
19089880,4rtkit-daemon3310658-21ssh11:42:181
19089880,4rtkit-daemon3148549-21ssh10:00:550
19089880,4rtkit-daemon3142093-21gpgv10:00:010
19089880,4rtkit-daemon3142093-21gpgv10:00:000
19089880,4rtkit-daemon3093982-21ssh09:30:111
19089880,4rtkit-daemon3084958-21ssh09:23:251
19089880,3rtkit-daemon3286486-21ssh11:28:021
19089880,1rtkit-daemon0-21swapper/211:53:382
19089880,1rtkit-daemon0-21swapper/211:25:242
19089880,1rtkit-daemon0-21swapper/111:14:301
19089880,1rtkit-daemon0-21swapper/012:38:540
3384324270,4sleep13384332-21ssh12:27:281
3369419270,5sleep13369583-21ssh12:20:151
3342329270,4sleep13342337-21ssh12:00:361
3297439271,3sleep03298643-21grep11:35:180
3278065270,1sleep00-21swapper/011:22:030
3263421270,2sleep20-21swapper/211:15:122
3239625270,4sleep134-21ksoftirqd/111:00:171
3239237271,3sleep20-21swapper/211:00:132
3231067271,3sleep23231108-21apt-get10:55:132
3214961270,3sleep13215000-21apt-get10:45:141
3195918271,3sleep23196048-21ssh10:30:212
3190205270,1sleep20-21swapper/210:29:192
3172726270,1sleep00-21swapper/010:15:550
3165958270,1sleep10-21swapper/110:14:051
3165958270,1sleep10-21swapper/110:14:041
3158890270,4sleep23159094-21ssh10:10:172
3158890270,4sleep23159094-21ssh10:10:162
3149244270,1sleep20-21swapper/210:02:362
3134442271,3sleep20-21swapper/209:55:182
3134442271,3sleep20-21swapper/209:55:172
3111555270,4sleep03112990-21grep09:40:170
3109641270,5sleep03109648-21sh09:39:440
3101688270,1sleep00-21swapper/009:35:000
3092914270,4sleep03092922-21ssh09:28:100
3061824270,4sleep23061978-21df_inode09:10:132
3005120270,4sleep03005151-21uniq08:25:160
3003994271,1sleep00-21swapper/008:20:220
2990018270,4sleep02991127-21memory08:10:220
2978817270,4sleep02978942-21python308:00:240
2961181270,5sleep22961305-21gltestperf07:50:162
2961181270,5sleep22961305-21gltestperf07:50:152
2903146270,4sleep02905119-21cut07:05:180
2902040270,4sleep22902183-21irqcore07:05:172
19089870,4rtkit-daemon674-21lldpd11:45:021
19089870,4rtkit-daemon674-21lldpd07:22:011
19089870,4rtkit-daemon671-21ntpd11:06:201
19089870,4rtkit-daemon667-21snmpd09:48:061
19089870,4rtkit-daemon667-21snmpd08:18:581
19089870,4rtkit-daemon421-21avahi-daemon08:03:411
19089870,4rtkit-daemon299-21systemd-journal09:17:571
19089870,4rtkit-daemon2914074-21munin-node08:06:121
19089870,4rtkit-daemon2914074-21munin-node07:37:521
19089870,3rtkit-daemon667-21snmpd07:55:551
19089870,3rtkit-daemon3012686-21kthreadcore08:30:161
19089870,3rtkit-daemon1928-21sendmail-mta10:20:342
19089870,3rtkit-daemon0-21swapper/108:21:261
19089870,1rtkit-daemon0-21swapper/009:02:300
3400855260,1sleep10-21swapper/112:37:461
3392251260,1sleep20-21swapper/212:31:352
3383916260,1sleep20-21swapper/212:26:302
3359444260,1sleep20-21swapper/212:12:062
3347331260,4sleep10-21swapper/112:05:211
3335002260,3sleep13335010-21ssh11:57:181
3331107260,4sleep23333171-21kthreadcore11:55:192
3309656260,3sleep03309881-21ssh11:40:270
3252557260,4sleep23252606-21ssh11:05:222
3247256261,2sleep03247293-21apt-get11:05:120
3221582260,1sleep20-21swapper/210:47:202
3213109260,1sleep20-21swapper/210:41:232
3206410260,1sleep20-21swapper/210:39:372
3206391260,1sleep10-21swapper/110:39:281
3197103260,1sleep10-21swapper/110:31:401
3175130261,2sleep10-21swapper/110:20:181
3142599261,3sleep13142653-21ssh10:00:141
3133196260,1sleep20-21swapper/209:53:302
3124176260,3sleep03124184-21ssh09:46:020
3102535260,3sleep23103530-21irqstats09:35:182
3052203260,4sleep20-21swapper/209:00:202
3036411261,2sleep13036444-21apt-get08:50:141
3029509260,1sleep20-21swapper/208:40:322
2941511260,3sleep22941485-21apt-get07:35:022
2922648261,2sleep20-21swapper/207:20:152
2914285260,3sleep22916010-21cut07:15:012
2908882261,3sleep22910032-21kthreadcore07:10:182
19089860,3rtkit-daemon671-21ntpd08:47:140
3393303250,3sleep10-21swapper/112:33:581
3361253250,2sleep10-21swapper/112:15:151
3344859252,2sleep20-21swapper/212:05:172
3336123250,3sleep20-21swapper/212:00:012
3214643250,1sleep10-21swapper/110:44:531
3157303250,2sleep13157311-21ssh10:07:351
3157303250,2sleep13157311-21ssh10:07:341
3136453251,2sleep13138459-21grep09:55:211
3136453251,2sleep13138459-21grep09:55:201
3124467250,3sleep23124473-21ssh09:46:402
3116144250,1sleep20-21swapper/209:41:002
3084036250,3sleep00-21swapper/009:21:130
3057333250,3sleep23059384-21grep09:05:202
3042085250,1sleep20-21swapper/208:51:112
3016918250,1sleep00-21swapper/008:30:520
3005354250,3sleep23005974-21sed08:25:172
3004374250,1sleep20-21swapper/208:21:022
2992407250,1sleep00-21swapper/008:15:100
2985552250,1sleep20-21swapper/208:05:312
2973216250,1sleep00-21swapper/007:58:520
2954938251,2sleep22955127-21idleruntime07:45:152
2954401250,1sleep20-21swapper/207:44:222
2936407250,3sleep22937321-21kthreadcore07:30:162
2928384250,1sleep00-21swapper/007:21:330
2922324250,1sleep00-21swapper/007:19:360
3313658240,1sleep00-21swapper/011:45:190
3298644240,2sleep13300693-21kthreadcore11:35:191
3271976240,2sleep10-21swapper/111:20:161
3264913240,2sleep03266327-21cut11:15:180
3223110240,2sleep10-21swapper/110:50:141
3157481240,2sleep20-21swapper/210:08:042
3157481240,2sleep20-21swapper/210:08:042
3100642240,2sleep23100650-21ssh09:32:292
3025293241,1sleep00-21swapper/008:40:180
2986198240,2sleep22986247-21/usr/sbin/munin08:10:162
2967239240,2sleep20-21swapper/207:55:132
2947762240,3sleep00-21swapper/007:36:240
2939357240,2sleep00-21swapper/007:30:180
2929892240,2sleep00-21swapper/007:25:180
3255276230,1sleep00-21swapper/011:10:060
3015602230,1sleep20-21swapper/208:30:182
2980081230,1sleep00-21swapper/008:05:160
2953774230,1sleep00-21swapper/007:40:240
2913237230,1sleep00-21swapper/007:10:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional