You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 06:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8.osadl.org (updated Tue Dec 02, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
48510221040,1sleep20-21swapper/223:48:482
2991382300,1sleep20-21swapper/221:54:272
73938992826,1cyclictest0-21swapper/323:50:003
73938992825,1cyclictest0-21swapper/321:00:013
73938992522,1cyclictest0-21swapper/320:40:013
7393899229,1cyclictest0-21swapper/322:50:003
73938992219,1cyclictest0-21swapper/320:10:003
7393899219,9cyclictest0-21swapper/321:10:223
7393899211,9cyclictest477075-21taskset23:43:563
73938992119,1cyclictest0-21swapper/321:55:003
73938992118,2cyclictest0-21swapper/322:00:223
7393899211,15cyclictest170037-21turbostat20:25:003
7393899209,1cyclictest0-21swapper/323:20:173
7393899209,1cyclictest0-21swapper/323:05:213
7393899209,0cyclictest0-21swapper/300:33:403
73938992017,1cyclictest0-21swapper/300:00:143
7393899201,15cyclictest468527-21taskset23:37:513
7393899201,15cyclictest307427-21turbostat22:00:003
7393899201,14cyclictest267446-21chrt21:35:123
7393899201,13cyclictest207950-21taskset20:55:013
73938992011,1cyclictest0-21swapper/320:33:353
5662502201,11sleep1566254-21ssh00:38:541
7393899197,7cyclictest3403412sleep322:20:103
7393899191,14cyclictest79987-21taskset19:12:583
7393899191,14cyclictest525985-21taskset00:14:183
7393899191,14cyclictest512436-21taskset00:05:163
7393899191,14cyclictest446110-21taskset23:25:173
7393899191,14cyclictest370930-21chrt22:36:163
7393899191,14cyclictest364470-21turbostat22:35:003
73938991910,1cyclictest0-21swapper/322:50:183
73938991910,0cyclictest0-21swapper/323:59:123
7393899190,14cyclictest113619-21turbostat19:40:003
7393899188,1cyclictest0-21swapper/320:25:263
7393899186,6cyclictest3787612sleep322:40:453
7393899186,6cyclictest2571482sleep321:25:433
73938991815,1cyclictest0-21swapper/321:05:223
73938991811,5cyclictest0-21swapper/323:12:393
7393899181,14cyclictest249661-21chrt21:21:583
7393899181,13cyclictest459486-21taskset23:30:303
7393899181,13cyclictest411026-21chrt23:00:403
7393899181,13cyclictest404888-21turbostat23:00:003
7393899181,13cyclictest282449-21taskset21:43:103
7393899180,14cyclictest265303-21chrt21:30:513
7393899176,7cyclictest2413302sleep321:16:263
7393899176,5cyclictest1574232sleep320:14:103
7393899175,6cyclictest1950352sleep320:40:353
73938991714,1cyclictest0-21swapper/323:18:223
73938991714,1cyclictest0-21swapper/322:15:163
73938991714,1cyclictest0-21swapper/300:20:003
7393899171,13cyclictest290222-21taskset21:47:303
7393899171,13cyclictest201346-21taskset20:46:073
7393899171,12cyclictest88201-21chrt19:15:543
7393899171,12cyclictest562213-21taskset00:35:183
7393899171,12cyclictest548671-21taskset00:25:323
7393899171,12cyclictest548671-21taskset00:25:323
7393899171,12cyclictest355713-21taskset22:28:223
7393899171,12cyclictest132078-21chrt19:51:013
7393899168,6cyclictest0-21swapper/319:20:133
7393899168,1cyclictest0-21swapper/320:00:153
7393899161,12cyclictest477839-21chrt23:45:093
7393899161,12cyclictest323183-21chrt22:08:303
7393899161,12cyclictest220131-21taskset21:00:503
7393899161,12cyclictest138384-21taskset19:56:503
7393899161,12cyclictest125948-21taskset19:48:003
7393899161,12cyclictest100863-21taskset19:27:363
7393899161,11cyclictest330269-21taskset22:11:073
7393899160,13cyclictest534769-21chrt00:20:113
7393899151,11cyclictest113878-21taskset19:40:113
7393899151,10cyclictest106415-21taskset19:30:213
7393899150,12cyclictest163447-21taskset20:16:213
678722150,2sleep30-21swapper/319:05:133
5428462151,4sleep2542880-21apt-get00:25:122
5428462151,4sleep2542880-21apt-get00:25:122
5238382150,12sleep1523961-21python300:10:241
5104022150,8sleep2510410-21ssh00:05:042
4277382151,7sleep2427746-21ssh23:12:022
147598150,2rtkit-daemon0-21swapper/000:13:380
1282112141,10sleep0130220-21kthreadcore19:50:190
147598130,9rtkit-daemon232905-21ssh21:10:332
729982121,6sleep0214-21plymouthd19:05:220
4860912120,9sleep2485602-21apt-get23:50:012
2676002120,8sleep0267696-21fschecks_time21:35:140
2496982120,9sleep0249706-21ssh21:22:040
1835242120,4sleep2184471-21cut20:35:172
1768402121,8sleep1176899-21sort20:30:151
147598120,8rtkit-daemon392522-21cut22:50:182
147598120,7rtkit-daemon468931-21ssh23:38:422
147598120,6rtkit-daemon1540-21sendmail-mta20:21:502
803942111,7sleep281885-21kthreadcore19:15:022
738042111,7sleep20-21swapper/219:10:012
5525182110,4sleep1554470-21tr00:30:161
3996962111,7sleep2401711-21tr22:55:192
2258392111,7sleep1225895-21ntp_states21:05:211
1538032110,8sleep1155874-21kthreadcore20:10:191
147598110,7rtkit-daemon456758-21kthreadcore23:30:192
147598110,7rtkit-daemon1540-21sendmail-mta23:07:352
147598110,7rtkit-daemon118856-21/usr/sbin/munin19:40:202
147598110,4rtkit-daemon80407-21munin-node22:59:460
5604612100,7sleep20-21swapper/200:35:182
4308282100,2sleep1431954-21kthreadcore23:15:181
3872012100,7sleep0387209-21ssh22:46:350
3388602100,7sleep1338868-21ssh22:17:191
2899122101,7sleep1289914-21ssh21:46:411
2756912101,5sleep2275861-21cut21:40:172
2081762101,4sleep1208209-21apt-get20:55:161
1772102100,8sleep00-21swapper/020:30:180
147598100,6rtkit-daemon226742-21diskmemload23:27:332
147598100,5rtkit-daemon494469-21diskstats23:55:150
147598100,3rtkit-daemon214-21plymouthd00:27:330
147598100,3rtkit-daemon214-21plymouthd00:27:330
147598100,2rtkit-daemon372064-21ssh22:38:512
1215482100,7sleep0123199-21grep19:45:180
1039882100,7sleep1106100-21cut19:30:201
88261291,5sleep1720-21snmpd19:16:411
72396292,4sleep172873-21grep19:05:191
533014290,2sleep1533022-21ssh00:16:491
509636290,7sleep1509644-21ssh00:03:511
494277292,4sleep2494294-21apt-get23:55:142
478319290,5sleep1479076-21cut23:45:161
412286290,3sleep2214-21plymouthd23:03:372
330523290,7sleep2330530-21sh22:11:432
309194291,5sleep2310297-21grep22:00:172
306116290,2sleep2214-21plymouthd21:57:002
14759890,6rtkit-daemon515-21gmain22:17:400
14759890,6rtkit-daemon515-21gmain20:04:012
14759890,5rtkit-daemon313-21systemd-udevd23:42:420
14759890,5rtkit-daemon313080-21ntp_states22:00:200
14759890,5rtkit-daemon22110-21packagekitd21:33:320
14759890,5rtkit-daemon214-21plymouthd22:41:082
14759890,4rtkit-daemon486381-21df_inode23:50:130
14759890,4rtkit-daemon226052-21/usr/sbin/munin21:05:232
14759890,4rtkit-daemon214-21plymouthd20:58:402
14759890,4rtkit-daemon148221-21kthreadcore20:05:232
14759890,4rtkit-daemon134416-21grep19:55:182
14759890,4rtkit-daemon107698-21ls19:35:132
14759890,2rtkit-daemon214-21plymouthd22:23:050
100228290,1sleep20-21swapper/219:25:242
468973280,1sleep00-21swapper/023:38:490
451835280,1sleep00-21swapper/023:26:270
444338281,1sleep00-21swapper/023:22:580
421690280,2sleep0171rcu_preempt23:10:160
364795281,3sleep1364832-21apt-get22:35:141
348185281,4sleep2348132-21apt-get22:25:012
323710280,6sleep1323714-21ssh22:09:421
267410281,3sleep2267443-21apt-get21:35:122
241208281,4sleep0241216-21ssh21:16:100
14759880,5rtkit-daemon214-21plymouthd22:27:100
14759880,4rtkit-daemon730-21ntpd23:03:570
14759880,4rtkit-daemon316409-21cut22:05:140
14759880,4rtkit-daemon313-21systemd-udevd20:52:422
14759880,4rtkit-daemon295-21systemd-journal21:27:590
14759880,4rtkit-daemon22110-21packagekitd22:10:120
14759880,4rtkit-daemon214-21plymouthd23:45:370
14759880,4rtkit-daemon214-21plymouthd00:07:480
14759880,4rtkit-daemon194740-21ls20:40:232
14759880,4rtkit-daemon179075-21grep20:30:192
14759880,4rtkit-daemon1-21systemd21:59:550
14759880,4rtkit-daemon1-21systemd21:01:092
14759880,4rtkit-daemon1-21systemd00:05:000
14759880,3rtkit-daemon730-21ntpd23:22:012
14759880,3rtkit-daemon541128-21ssh00:21:510
14759880,3rtkit-daemon427096-21kworker/u16:3-xprtiod23:18:512
14759880,2rtkit-daemon175449-21ntp_states20:25:232
137942280,5sleep1138102-21timerwakeupswit19:55:251
94680270,1sleep00-21swapper/019:23:420
82515271,3sleep00-21swapper/019:15:120
558070270,1sleep00-21swapper/000:33:320
548307270,4sleep1548556-21grep00:25:231
548307270,4sleep1548556-21grep00:25:231
518322270,2sleep2214-21plymouthd00:10:012
518237271,3sleep1518320-21latency_hist00:10:001
508951270,5sleep2508959-21ssh00:02:162
476878271,4sleep2476886-21ssh23:43:322
420958270,4sleep1421183-21sed23:10:011
420230270,4sleep0420238-21ssh23:08:200
372866270,1sleep00-21swapper/022:40:060
363551270,2sleep1214-21plymouthd22:32:501
362586270,1sleep00-21swapper/022:30:360
362539271,4sleep2720-21snmpd22:30:262
347859271,3sleep2347867-21ssh22:24:292
299785271,3sleep1299840-21/usr/sbin/munin21:55:151
250924271,1sleep10-21swapper/121:24:531
189352270,3sleep1189385-21apt-get20:40:131
189074270,4sleep1189248-21sed20:40:011
170687270,1sleep10-21swapper/120:25:181
164202271,3sleep0164299-21fschecks_time20:20:160
157749271,3sleep2157784-21apt-get20:15:132
14759870,4rtkit-daemon761-21gmain21:50:590
14759870,4rtkit-daemon515-21gmain19:47:292
14759870,4rtkit-daemon515-21gmain19:20:292
14759870,4rtkit-daemon1-21systemd21:40:250
14759870,3rtkit-daemon899-21cron20:48:002
14759870,3rtkit-daemon75348-21sed19:10:162
14759870,3rtkit-daemon242667-21ssh21:19:291
14759870,3rtkit-daemon241272-21taskset21:16:172
14759870,3rtkit-daemon214-21plymouthd19:31:132
14759870,3rtkit-daemon1-21systemd19:50:472
14759870,2rtkit-daemon214-21plymouthd00:16:120
14759870,2rtkit-daemon0-21swapper/222:45:172
107597271,3sleep10-21swapper/119:35:131
101347270,4sleep0101496-21df_inode19:30:150
100797274,2sleep180407-21munin-node19:26:471
79458261,2sleep00-21swapper/019:10:230
74111261,2sleep10-21swapper/119:10:121
535579261,3sleep2536436-21kthreadcore00:20:152
528076261,3sleep20-21swapper/200:15:182
501062260,3sleep1501070-21ssh23:57:501
485587261,2sleep10-21swapper/123:50:001
468631260,3sleep1468639-21ssh23:38:061
458955260,3sleep0459069-21ssh23:30:240
457819260,3sleep1458766-21sed23:30:211
451904260,1sleep10-21swapper/123:26:361
427251261,2sleep1214-21plymouthd23:10:531
411452260,1sleep10-21swapper/123:01:391
388139260,1sleep10-21swapper/122:48:411
327016260,3sleep10-21swapper/122:10:181
316076260,3sleep2315588-21apt-get22:05:022
289967260,4sleep00-21swapper/021:46:480
289150260,3sleep2289366-21users21:45:262
275101261,3sleep11-21systemd21:40:001
257630260,4sleep1169943-21kworker/u16:2+rpciod21:26:531
250617260,1sleep20-21swapper/221:24:142
233767260,1sleep00-21swapper/021:12:300
226121260,3sleep0226365-21users21:05:240
207766260,1sleep00-21swapper/020:53:100
201898260,1sleep10-21swapper/120:50:071
201442260,1sleep00-21swapper/020:47:210
195642261,2sleep10-21swapper/120:45:141
195114260,1sleep00-21swapper/020:41:360
183240260,4sleep0183376-21head20:35:150
169872260,1sleep10-21swapper/120:23:101
163412260,1sleep00-21swapper/020:15:550
163043260,3sleep1163217-21timerwakeupswit20:15:241
157473260,1sleep00-21swapper/020:14:470
14759860,3rtkit-daemon515-21gmain20:10:052
139109260,3sleep1139309-21hddtemp_smartct20:00:171
119841260,1sleep00-21swapper/019:44:260
100780260,1sleep00-21swapper/019:26:340
542183250,3sleep10-21swapper/100:24:191
476501250,1sleep10-21swapper/123:42:351
429393250,2sleep00-21swapper/023:15:130
397069250,3sleep1397077-21ssh22:55:101
395201250,3sleep10-21swapper/122:51:211
394182251,2sleep0394295-21ntp_kernel_pll_22:50:200
379871250,3sleep10-21swapper/122:43:161
348952251,2sleep10-21swapper/122:25:161
345655251,2sleep10-21swapper/122:20:241
332232250,3sleep20-21swapper/222:15:042
276946250,3sleep1278727-21cut21:40:201
264751250,2sleep20-21swapper/221:30:232
257666250,2sleep2257674-21ssh21:26:592
232937250,3sleep10-21swapper/121:10:381
220211250,1sleep10-21swapper/121:01:531
220210250,3sleep00-21swapper/021:01:520
214009250,3sleep00-21swapper/020:58:110
174072250,3sleep0175254-21memory20:25:210
151008250,3sleep10-21swapper/120:07:311
145195250,2sleep00-21swapper/020:05:170
138632250,2sleep00-21swapper/020:00:000
126354250,3sleep10-21swapper/119:50:081
119777250,1sleep10-21swapper/119:43:381
90204240,1sleep10-21swapper/119:20:181
559012240,2sleep0226742-21diskmemload00:35:080
558086240,2sleep2558094-21ssh00:33:342
443467240,2sleep10-21swapper/123:20:511
371255240,2sleep00-21swapper/022:37:020
307759240,2sleep1307798-21apt-get22:00:121
262149240,1sleep10-21swapper/121:30:181
140519240,2sleep00-21swapper/020:00:200
112503240,2sleep0112676-21strings19:35:200
291587230,1sleep10-21swapper/121:50:101
120132230,1sleep10-21swapper/119:45:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional