You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 04:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Tue Nov 25, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31182899913930,1389cyclictest35805712sleep300:20:013
31182899912740,1270cyclictest33941402sleep322:24:573
31182899911461,1142cyclictest33291832sleep321:45:013
31182899911370,1133cyclictest31713592sleep319:50:023
3118289992420,1cyclictest0-21swapper/300:19:593
190898220,5rtkit-daemon438-21wpa_supplicant23:51:562
3118289992118,1cyclictest0-21swapper/300:10:003
3118289992017,1cyclictest0-21swapper/322:00:013
190898190,7rtkit-daemon3277796-21python321:10:240
34027802171,13sleep13402813-21/usr/sbin/munin22:30:131
311828999177,7cyclictest3208869-21turbostat20:19:593
3118289991714,1cyclictest0-21swapper/323:30:153
3118289991713,2cyclictest3369694-21turbostat22:10:013
311828999166,7cyclictest35476392sleep323:59:513
311828999161,11cyclictest3410316-21turbostat22:35:003
311828999161,11cyclictest3133441-21taskset19:20:133
311828999155,7cyclictest36028372sleep300:31:003
3118289991513,1cyclictest0-21swapper/323:49:083
3118289991513,1cyclictest0-21swapper/323:44:053
3118289991513,1cyclictest0-21swapper/323:44:043
3118289991513,1cyclictest0-21swapper/319:15:013
3118289991513,0cyclictest0-21swapper/321:02:573
3118289991512,1cyclictest0-21swapper/323:38:353
3118289991512,1cyclictest0-21swapper/323:38:353
3118289991512,1cyclictest0-21swapper/323:10:263
3118289991512,1cyclictest0-21swapper/320:00:243
311828999151,11cyclictest3593397-21taskset00:25:183
311828999151,11cyclictest3532538-21taskset23:50:163
311828999151,11cyclictest3402294-21turbostat22:30:013
190898150,5rtkit-daemon3456207-21ntp_states23:00:220
3118289991412,0cyclictest0-21swapper/321:13:313
3118289991411,1cyclictest0-21swapper/322:50:583
3118289991411,1cyclictest0-21swapper/321:30:153
3118289991411,1cyclictest0-21swapper/320:55:443
3118289991411,1cyclictest0-21swapper/319:25:233
3118289991411,1cyclictest0-21swapper/300:36:223
311828999141,10cyclictest3571290-21chrt00:13:283
311828999141,10cyclictest3457721-21taskset23:02:473
311828999141,10cyclictest3446119-21chrt22:55:183
311828999141,10cyclictest3432801-21taskset22:46:003
311828999141,10cyclictest3410899-21taskset22:35:153
311828999141,10cyclictest3385705-21taskset22:19:123
311828999141,10cyclictest3320452-21taskset21:39:443
311828999141,10cyclictest3246584-21turbostat20:50:003
311828999141,10cyclictest3198434-21taskset20:10:173
311828999141,10cyclictest3152005-21taskset19:31:593
311828999140,9cyclictest31776672sleep319:55:123
311828999140,11cyclictest3359645-21taskset22:00:273
311828999140,11cyclictest3326282-21chrt21:40:213
33270422131,2sleep10-21swapper/121:40:361
311828999135,6cyclictest34257982sleep322:43:323
311828999134,7cyclictest3164686-21taskset19:43:213
311828999133,8cyclictest3220898-21taskset20:25:243
311828999131,9cyclictest3554231-21taskset00:01:183
311828999131,9cyclictest3464756-21chrt23:05:283
311828999131,9cyclictest3342667-21taskset21:50:183
311828999131,9cyclictest3290229-21taskset21:20:173
311828999131,9cyclictest3265942-21taskset21:05:023
311828999131,9cyclictest3237008-21taskset20:40:183
311828999131,9cyclictest3190298-21taskset20:05:123
311828999131,8cyclictest34913032sleep323:25:133
3118289991312,0cyclictest0-21swapper/321:16:303
311828999131,10cyclictest3227887-21taskset20:35:013
3118289991310,1cyclictest0-21swapper/319:45:143
311828999130,10cyclictest3299627-21taskset21:25:183
311828999130,10cyclictest3158537-21turbostat19:40:013
311828999130,10cyclictest3124163-21taskset19:10:383
31180622130,3sleep30-21swapper/319:09:003
35940262121,8sleep23594157-21python300:25:222
32522742120,1sleep23252518-21users20:50:242
31237012121,2sleep20-21swapper/219:10:232
311828999124,6cyclictest3375983-21taskset22:11:023
311828999124,6cyclictest3209312-21taskset20:20:153
311828999123,7cyclictest3475062-21taskset23:15:023
311828999121,9cyclictest3226776-21taskset20:30:203
311828999121,8cyclictest3246851-21taskset20:50:123
311828999120,9cyclictest3483185-21taskset23:20:023
190898120,8rtkit-daemon3498530-21ssh23:28:400
190898120,7rtkit-daemon3572007-21systemd-run00:15:010
33776922111,6sleep03377763-21apt-config22:15:010
190898110,7rtkit-daemon3547387-21ssh23:59:180
190898110,4rtkit-daemon3088517-21kworker/0:0+kacpid20:06:330
190898110,2rtkit-daemon3152975-21irqcore19:35:182
33514172100,3sleep23351423-21ssh21:55:472
190898100,7rtkit-daemon473-21gmain22:48:181
190898100,6rtkit-daemon3393019-21ssh22:22:230
190898100,5rtkit-daemon3559186-21grep00:05:170
190898100,5rtkit-daemon3480340-21/usr/sbin/munin23:15:210
190898100,5rtkit-daemon3362100-21df_inode22:05:120
190898100,5rtkit-daemon3285682-21ntp_states21:15:220
190898100,5rtkit-daemon3209515-21hddtemp_smartct20:20:160
190898100,3rtkit-daemon3402224-21ssh22:29:540
3602281291,5sleep13602539-21grep00:30:251
3561435293,3sleep23561528-21mailstats00:05:232
3426324290,6sleep03426326-21ssh22:44:460
3412045290,6sleep13413137-21grep22:35:181
3393049294,3sleep23271985-21diskmemload22:22:282
3367089291,5sleep13367174-21ntp_states22:05:201
3347941290,6sleep13350061-21cut21:55:181
3337626290,6sleep13337817-21ssh21:50:121
3267411291,5sleep13269021-21kthreadcore21:05:171
3250501290,1sleep1321rcuc/120:50:181
3245649291,5sleep13245722-21ntp_states20:45:211
3209121291,4sleep13209153-21apt-get20:20:141
3208086291,5sleep13208221-21/usr/sbin/munin20:15:251
3183946291,4sleep03183979-21apt-get20:00:130
3173866290,6sleep03175982-21cut19:50:200
3171827291,5sleep23172734-21sed19:50:172
19089890,6rtkit-daemon473-21gmain22:40:142
19089890,6rtkit-daemon421-21avahi-daemon21:32:192
19089890,5rtkit-daemon438-21wpa_supplicant23:23:560
19089890,5rtkit-daemon3587221-21chrt00:22:161
19089890,5rtkit-daemon3577567-21ntp_kernel_pll_00:15:210
19089890,5rtkit-daemon3326347-21needreboot21:40:212
19089890,5rtkit-daemon3319313-21ssh21:37:032
19089890,5rtkit-daemon3237346-21kthreadcore20:40:180
19089890,4rtkit-daemon3506329-21ssh23:32:580
19089890,4rtkit-daemon3259160-21seq20:56:380
19089890,4rtkit-daemon3252169-21run-parts20:50:220
19089890,3rtkit-daemon421-21avahi-daemon20:16:230
19089890,3rtkit-daemon3474064-21ssh23:13:160
19089890,3rtkit-daemon3344847-21ssh21:54:142
19089890,3rtkit-daemon1928-21sendmail-mta00:36:272
3572513280,5sleep13572721-21hddtemp_smartct00:15:161
3545972280,6sleep2667-21snmpd23:56:052
3466327281,5sleep1667-21snmpd23:09:041
3457245280,3sleep13457251-21ssh23:01:421
3443741281,4sleep13444354-21uname22:55:161
3417499280,1sleep00-21swapper/022:38:010
3394478281,3sleep23394512-21apt-get22:25:132
3353849281,3sleep13353886-21apt-get22:00:111
3335972280,5sleep13335979-21sh21:47:291
3310233281,4sleep03310482-21users21:30:250
3288744281,4sleep03288946-21hddtemp_smartct21:20:150
3286238280,5sleep23286246-21ssh21:15:322
3163894281,3sleep03163973-21ntp_states19:40:210
3152699281,4sleep03152898-21hddtemp_smartct19:35:170
3138767280,6sleep13139019-21sed19:20:241
3112185281,3sleep20-21swapper/219:05:142
19089880,5rtkit-daemon671-21ntpd23:42:220
19089880,5rtkit-daemon671-21ntpd23:42:210
19089880,4rtkit-daemon671-21ntpd20:45:200
19089880,4rtkit-daemon667-21snmpd22:36:022
19089880,4rtkit-daemon667-21snmpd22:31:000
19089880,4rtkit-daemon667-21snmpd22:10:592
19089880,4rtkit-daemon438-21wpa_supplicant22:55:342
19089880,4rtkit-daemon438-21wpa_supplicant20:25:470
19089880,4rtkit-daemon438-21wpa_supplicant19:32:442
19089880,4rtkit-daemon3384341-21ssh22:15:580
19089880,4rtkit-daemon3124747-21munin-node22:45:302
19089880,4rtkit-daemon1928-21sendmail-mta19:43:572
19089880,3rtkit-daemon3335182-21ssh21:45:392
19089880,3rtkit-daemon3141212-21irqrtprio19:25:162
19089880,1rtkit-daemon0-21swapper/219:55:042
19089880,1rtkit-daemon0-21swapper/219:49:032
19089880,1rtkit-daemon0-21swapper/122:51:141
19089880,1rtkit-daemon0-21swapper/023:37:230
19089880,1rtkit-daemon0-21swapper/023:37:230
19089880,1rtkit-daemon0-21swapper/023:09:110
19089880,1rtkit-daemon0-21swapper/022:56:450
3605344270,1sleep10-21swapper/100:35:151
3593853270,4sleep03593967-21sed00:25:200
3564397270,4sleep23564177-21/usr/sbin/munin00:10:162
3554618271,1sleep20-21swapper/200:02:142
3522525270,1sleep10-21swapper/123:43:141
3522525270,1sleep10-21swapper/123:43:141
3506579270,4sleep23506587-21ssh23:33:372
3498333270,4sleep23498342-21sshd23:28:162
3452434270,4sleep23453825-21cut23:00:192
3432810270,1sleep00-21swapper/022:46:020
3391462270,4sleep13391607-21ssh22:20:191
3385670270,4sleep23385679-21ssh22:19:072
3384208270,5sleep13384210-21ssh22:15:421
3302928271,4sleep23302936-21ssh21:26:462
3296567271,2sleep13296602-21apt-get21:25:131
3278863270,1sleep20-21swapper/221:12:152
3271056271,3sleep03271127-21ntp_states21:05:200
3234201270,3sleep13234234-21apt-get20:40:131
3221648271,3sleep20-21swapper/220:30:112
3220577271,3sleep13220701-21python320:25:231
3203168270,5sleep23203285-21ps20:15:182
3196718270,4sleep03196917-21hddtemp_smartct20:10:150
3190614270,5sleep13190939-21sed20:05:141
3185404270,5sleep23187103-21grep20:00:192
3177704270,4sleep03177853-21df_inode19:55:130
3165070270,3sleep13165103-21apt-get19:45:121
3124705271,3sleep03125241-21memory19:15:020
19089870,4rtkit-daemon674-21lldpd22:54:502
19089870,4rtkit-daemon667-21snmpd21:20:562
19089870,4rtkit-daemon473-21gmain22:13:141
19089870,4rtkit-daemon473-21gmain20:35:140
19089870,4rtkit-daemon473-21gmain19:15:502
19089870,4rtkit-daemon3124747-21munin-node20:32:120
19089870,4rtkit-daemon3124747-21munin-node00:25:501
19089870,3rtkit-daemon671-21ntpd22:06:332
19089870,3rtkit-daemon667-21snmpd21:00:010
19089870,3rtkit-daemon3124747-21munin-node23:53:040
19089870,3rtkit-daemon3124747-21munin-node23:47:532
19089870,3rtkit-daemon3124747-21munin-node21:41:570
19089870,3rtkit-daemon3124747-21munin-node00:00:300
19089870,3rtkit-daemon3088517-21kworker/0:0+kacpid21:27:380
19089870,3rtkit-daemon3088517-21kworker/0:0+kacpid19:05:400
19089870,3rtkit-daemon1223-21cron23:43:002
19089870,3rtkit-daemon1223-21cron23:43:002
19089870,1rtkit-daemon0-21swapper/219:21:302
3605127260,4sleep03605315-21idleruntime00:35:140
3603268260,1sleep20-21swapper/200:32:042
3602170260,1sleep00-21swapper/000:30:250
3570513260,1sleep10-21swapper/100:11:431
3547526260,1sleep10-21swapper/123:59:371
3538567260,4sleep13271985-21diskmemload23:52:441
3530836260,1sleep00-21swapper/023:48:340
3507443260,1sleep10-21swapper/123:35:041
3507443260,1sleep10-21swapper/123:35:031
3501725260,4sleep13503750-21cut23:30:191
3483446260,3sleep13483567-21fschecks_time23:20:141
3482301260,1sleep10-21swapper/123:18:371
3475122261,2sleep20-21swapper/223:15:142
3473827260,3sleep23473835-21ssh23:12:492
3467034260,3sleep13467071-21apt-get23:10:121
3434787261,2sleep03434826-21apt-get22:50:110
3424914260,1sleep10-21swapper/122:41:271
3394803260,4sleep13394475-21/usr/sbin/munin22:25:161
3361156260,1sleep00-21swapper/022:04:030
3346811261,2sleep00-21swapper/021:55:170
3335831260,1sleep00-21swapper/021:47:120
3312782260,3sleep03312815-21apt-get21:35:140
3312148260,4sleep13312154-21ssh21:34:191
3288947260,3sleep13289339-21irqcore21:20:161
3265372260,1sleep20-21swapper/221:00:502
3233063261,3sleep23227901-21/usr/sbin/munin20:35:212
3227645260,4sleep13227703-21latency_hist20:35:001
3215633261,2sleep2667-21snmpd20:25:162
3196518260,2sleep13196551-21apt-get20:10:121
3152499260,3sleep13152532-21apt-get19:35:141
3145367261,2sleep10-21swapper/119:25:241
3125242261,2sleep10-21swapper/119:15:031
3119902261,2sleep10-21swapper/119:10:181
3118453260,2sleep03118486-21apt-get19:10:130
3581922250,2sleep00-21swapper/000:20:200
3580057250,2sleep23580570-21ssh00:20:012
3579928250,1sleep20-21swapper/200:19:452
3548026250,2sleep10-21swapper/100:00:131
3497710250,3sleep13497718-21ssh23:26:511
3489751250,3sleep23489759-21ssh23:22:102
3459016250,2sleep20-21swapper/223:05:122
3355124250,3sleep23355381-21kthreadcore22:00:162
3339678250,2sleep00-21swapper/021:50:160
3282644251,2sleep13284652-21cut21:15:191
3278621250,3sleep10-21swapper/121:11:371
3261167251,2sleep10-21swapper/121:00:171
3259178250,1sleep20-21swapper/220:56:532
3227900250,2sleep10-21swapper/120:35:121
3210101250,2sleep20-21swapper/220:20:192
3197499250,3sleep23198433-21cat20:10:172
3189956250,1sleep10-21swapper/120:04:551
3183483250,1sleep10-21swapper/119:57:421
3152232250,1sleep00-21swapper/019:34:530
3139314250,1sleep00-21swapper/019:23:190
19089850,1rtkit-daemon0-21swapper/123:45:231
3556829240,1sleep10-21swapper/100:05:171
3514484240,2sleep23514492-21ssh23:38:142
3514484240,2sleep23514492-21ssh23:38:142
3409429240,2sleep23409435-21ssh22:32:542
3314271240,2sleep10-21swapper/121:35:191
3259424240,2sleep13259521-21latency_hist21:00:001
3246431240,3sleep20-21swapper/220:48:192
3235655240,2sleep23237007-21kthreadcore20:40:182
3190940240,2sleep20-21swapper/220:05:162
3169902240,2sleep03170145-21needreboot19:45:190
3151563240,2sleep10-21swapper/119:30:251
3112488240,2sleep10-21swapper/119:05:171
3266142230,1sleep20-21swapper/221:05:142
3175985230,1sleep10-21swapper/119:50:211
3160253230,1sleep10-21swapper/119:40:191
3145651230,1sleep00-21swapper/019:25:440
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional