You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-04 - 11:05
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rackdslot3.osadl.org (updated Mon Dec 04, 2023 00:45:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
776521700,7sleep01514899cyclictest21:10:340
8255911680,5ptp4l401ktimersoftd/319:09:033
309022770,3sleep030907-21needreboot23:30:440
57698750,48rtkit-daemon559-21gmain19:05:081
197022700,4sleep11131-21lldpd19:20:001
825591690,1ptp4l391rcuc/300:34:343
326372690,5sleep125-21ksoftirqd/122:20:291
15149996356,4cyclictest25-21ksoftirqd/100:15:161
57698610,20rtkit-daemon1131-21lldpd21:35:321
275522610,4sleep11123-21snmpd23:25:181
15149996154,4cyclictest25-21ksoftirqd/119:50:221
15149996153,5cyclictest25-21ksoftirqd/123:40:201
15149996152,5cyclictest25-21ksoftirqd/120:15:201
1515199592,8cyclictest31209-21cat22:20:133
15149995951,5cyclictest25-21ksoftirqd/120:40:171
15149995951,5cyclictest25-21ksoftirqd/119:35:201
1463225833,13sleep09-21ksoftirqd/019:07:030
1515199572,9cyclictest18011-21latency_hist23:05:003
116112570,3sleep21515099cyclictest20:10:242
57698560,11rtkit-daemon0-21swapper/021:39:370
1515199562,8cyclictest2811-21ls23:40:283
15149995649,4cyclictest25-21ksoftirqd/123:35:151
15149995649,4cyclictest25-21ksoftirqd/123:05:221
1515199551,9cyclictest19436-21chrt19:18:143
1515199541,7cyclictest401ktimersoftd/319:45:173
1515199532,9cyclictest5298-21processes19:55:343
1515199514,7cyclictest41-21ksoftirqd/322:50:183
1515199513,6cyclictest41-21ksoftirqd/319:50:353
1461825122,17sleep20-21swapper/219:06:522
1515199500,9cyclictest401ktimersoftd/300:15:333
15149995043,4cyclictest25-21ksoftirqd/121:45:001
1515199496,8cyclictest41-21ksoftirqd/321:48:413
1515199496,8cyclictest0-21swapper/320:49:333
1515199494,8cyclictest0-21swapper/321:35:003
1515199494,6cyclictest41-21ksoftirqd/319:10:253
1515199493,8cyclictest41-21ksoftirqd/319:35:393
1515199486,9cyclictest41-21ksoftirqd/300:13:253
1515199486,8cyclictest41-21ksoftirqd/320:55:393
1515199486,5cyclictest41-21ksoftirqd/323:26:533
1515199486,10cyclictest41-21ksoftirqd/300:09:403
1515199484,6cyclictest401ktimersoftd/320:05:293
1515199482,6cyclictest41-21ksoftirqd/323:55:233
1515199480,7cyclictest41-21ksoftirqd/320:19:003
1515199477,4cyclictest41-21ksoftirqd/321:54:023
1515199476,9cyclictest41-21ksoftirqd/321:09:523
1515199476,8cyclictest41-21ksoftirqd/323:11:183
1515199476,8cyclictest41-21ksoftirqd/321:44:503
1515199476,7cyclictest41-21ksoftirqd/321:13:313
1515199476,10cyclictest41-21ksoftirqd/323:49:233
1515199475,8cyclictest41-21ksoftirqd/322:35:093
1515199474,8cyclictest41-21ksoftirqd/320:39:333
1515199474,8cyclictest41-21ksoftirqd/320:39:323
57698460,3rtkit-daemon575-21rtkit-daemon22:50:232
1515199466,8cyclictest41-21ksoftirqd/322:59:163
1515199466,8cyclictest41-21ksoftirqd/322:59:163
1515199466,8cyclictest41-21ksoftirqd/322:48:593
1515199466,8cyclictest41-21ksoftirqd/321:16:183
1515199466,8cyclictest41-21ksoftirqd/320:52:353
1515199466,8cyclictest41-21ksoftirqd/319:27:163
1515199466,8cyclictest41-21ksoftirqd/300:37:083
1515199464,7cyclictest41-21ksoftirqd/320:25:163
1515199460,7cyclictest401ktimersoftd/323:15:453
15149994640,3cyclictest25-21ksoftirqd/121:05:001
57698450,2rtkit-daemon575-21rtkit-daemon22:26:082
1515199457,8cyclictest41-21ksoftirqd/321:21:173
1515199457,7cyclictest41-21ksoftirqd/322:05:003
1515199457,7cyclictest41-21ksoftirqd/300:21:373
1515199456,9cyclictest41-21ksoftirqd/319:42:433
1515199456,8cyclictest41-21ksoftirqd/322:15:213
1515199456,8cyclictest41-21ksoftirqd/321:34:193
1515199456,8cyclictest41-21ksoftirqd/321:03:053
1515199456,8cyclictest41-21ksoftirqd/320:12:543
1515199456,8cyclictest41-21ksoftirqd/319:33:593
1515199456,8cyclictest41-21ksoftirqd/319:20:013
1515199456,8cyclictest41-21ksoftirqd/319:20:013
1515199456,7cyclictest41-21ksoftirqd/322:11:493
1515199456,10cyclictest41-21ksoftirqd/320:44:483
1515199455,8cyclictest41-21ksoftirqd/323:05:543
1515199455,8cyclictest41-21ksoftirqd/321:56:403
1515199454,6cyclictest41-21ksoftirqd/300:26:213
1515199454,5cyclictest0-21swapper/323:35:143
1515199453,8cyclictest41-21ksoftirqd/321:27:583
1515199453,7cyclictest8654-21sed22:40:273
1515199452,8cyclictest41-21ksoftirqd/320:31:473
1515199450,6cyclictest41-21ksoftirqd/322:32:393
1515199450,6cyclictest401ktimersoftd/320:00:203
15150994538,4cyclictest33-21ksoftirqd/223:15:262
15149994538,4cyclictest25-21ksoftirqd/120:50:211
15149994538,3cyclictest25-21ksoftirqd/100:15:001
15149994536,6cyclictest25-21ksoftirqd/100:05:231
57698440,3rtkit-daemon575-21rtkit-daemon23:46:362
1515199446,8cyclictest41-21ksoftirqd/323:32:583
1515199446,8cyclictest41-21ksoftirqd/323:21:173
1515199446,8cyclictest41-21ksoftirqd/322:05:403
1515199445,8cyclictest41-21ksoftirqd/322:26:223
1515199445,7cyclictest41-21ksoftirqd/300:01:033
1515199443,7cyclictest41-21ksoftirqd/323:50:323
15149994438,4cyclictest25-21ksoftirqd/123:00:001
15149994438,4cyclictest25-21ksoftirqd/123:00:001
15149994438,3cyclictest25-21ksoftirqd/122:20:011
15149994436,4cyclictest25-21ksoftirqd/119:55:361
1515199435,7cyclictest41-21ksoftirqd/320:21:233
15150994335,5cyclictest33-21ksoftirqd/221:30:192
15150994326,10cyclictest35192irq/90-enp1s0-T19:40:192
15149994337,4cyclictest25-21ksoftirqd/121:50:231
15149994336,4cyclictest25-21ksoftirqd/120:05:281
114952430,8sleep21515099cyclictest00:05:272
15149994235,4cyclictest25-21ksoftirqd/123:25:011
15149994235,4cyclictest25-21ksoftirqd/121:29:591
15149994235,4cyclictest25-21ksoftirqd/121:20:011
15149994235,4cyclictest25-21ksoftirqd/120:25:191
15148994235,4cyclictest9-21ksoftirqd/020:50:200
15149994136,3cyclictest25-21ksoftirqd/122:25:341
15149994136,3cyclictest25-21ksoftirqd/121:00:001
15149994135,3cyclictest25-21ksoftirqd/121:15:001
15149994135,3cyclictest25-21ksoftirqd/120:40:011
15149994135,3cyclictest25-21ksoftirqd/120:40:001
15149994135,3cyclictest25-21ksoftirqd/100:25:001
15149994135,3cyclictest25-21ksoftirqd/100:05:001
15149994134,4cyclictest25-21ksoftirqd/123:30:371
15149994134,4cyclictest25-21ksoftirqd/123:20:011
1514899413,6cyclictest1131-21lldpd22:05:370
15148994135,3cyclictest26886-21kworker/0:022:20:300
1515099403,3cyclictest121rcu_preempt21:09:442
15149994034,4cyclictest25-21ksoftirqd/122:30:311
15149994034,4cyclictest25-21ksoftirqd/119:30:211
15149994034,3cyclictest25-21ksoftirqd/122:00:231
15149994033,4cyclictest25-21ksoftirqd/121:30:301
15149994025,12cyclictest25-21ksoftirqd/121:45:531
15148994033,4cyclictest9-21ksoftirqd/020:20:000
15150993934,3cyclictest33-21ksoftirqd/219:10:332
15150993933,3cyclictest33-21ksoftirqd/223:35:312
15150993932,4cyclictest33-21ksoftirqd/219:27:002
1514999397,3cyclictest121rcu_preempt23:49:261
15149993934,3cyclictest25-21ksoftirqd/100:30:451
15149993933,4cyclictest25-21ksoftirqd/119:40:261
15149993933,3cyclictest25-21ksoftirqd/123:50:341
15149993933,3cyclictest25-21ksoftirqd/122:40:001
15149993933,3cyclictest25-21ksoftirqd/119:20:271
15149993933,3cyclictest25-21ksoftirqd/119:20:261
15149993932,4cyclictest25-21ksoftirqd/100:25:331
15148993932,4cyclictest9-21ksoftirqd/000:40:000
15148993921,8cyclictest571-21rs:main0
1514899391,6cyclictest14464-21cpuspeed21:30:210
15150993832,4cyclictest33-21ksoftirqd/222:00:192
15150993832,4cyclictest33-21ksoftirqd/219:35:392
15150993832,4cyclictest33-21ksoftirqd/200:15:262
15150993832,3cyclictest33-21ksoftirqd/223:07:392
15149993833,3cyclictest25-21ksoftirqd/122:40:361
15149993833,3cyclictest25-21ksoftirqd/119:50:011
15149993832,4cyclictest25-21ksoftirqd/123:02:221
15149993832,3cyclictest25-21ksoftirqd/121:10:011
15149993832,3cyclictest25-21ksoftirqd/120:00:401
15149993832,3cyclictest25-21ksoftirqd/119:14:591
15149993831,4cyclictest25-21ksoftirqd/121:24:591
15149993831,4cyclictest25-21ksoftirqd/120:10:231
15149993831,4cyclictest11963-21irqrtprio22:50:251
15148993832,4cyclictest9-21ksoftirqd/020:45:300
15148993832,4cyclictest9-21ksoftirqd/000:18:170
199822370,8sleep01514899cyclictest21:40:330
15150993732,3cyclictest33-21ksoftirqd/222:30:402
15150993731,4cyclictest33-21ksoftirqd/220:50:282
15150993730,4cyclictest33-21ksoftirqd/223:21:522
15150993730,4cyclictest33-21ksoftirqd/222:18:372
15150993730,4cyclictest33-21ksoftirqd/219:58:032
15149993730,4cyclictest25-21ksoftirqd/121:57:181
15149993729,5cyclictest25-21ksoftirqd/122:46:061
15148993730,4cyclictest9-21ksoftirqd/023:40:410
1514899371,3cyclictest121rcu_preempt20:29:030
15150993631,3cyclictest33-21ksoftirqd/200:30:292
15150993630,4cyclictest33-21ksoftirqd/223:25:472
15150993629,4cyclictest33-21ksoftirqd/220:35:352
15150993629,4cyclictest33-21ksoftirqd/220:35:352
15149993631,3cyclictest25-21ksoftirqd/120:45:241
15149993631,3cyclictest25-21ksoftirqd/120:25:001
15149993630,4cyclictest25-21ksoftirqd/119:25:331
15149993629,4cyclictest25-21ksoftirqd/123:15:001
15149993629,4cyclictest25-21ksoftirqd/122:15:011
15149993629,4cyclictest25-21ksoftirqd/122:08:471
15148993630,4cyclictest9-21ksoftirqd/023:20:520
15148993630,4cyclictest9-21ksoftirqd/022:50:270
15148993630,4cyclictest9-21ksoftirqd/022:47:200
15148993630,4cyclictest9-21ksoftirqd/020:57:000
15148993630,4cyclictest9-21ksoftirqd/020:35:390
15148993630,4cyclictest9-21ksoftirqd/020:35:380
15148993630,3cyclictest9-21ksoftirqd/019:55:270
15148993630,3cyclictest9-21ksoftirqd/019:46:560
15148993630,3cyclictest9-21ksoftirqd/019:15:210
1514899360,3cyclictest121rcu_preempt23:15:100
15150993530,3cyclictest33-21ksoftirqd/219:25:012
15150993530,3cyclictest33-21ksoftirqd/219:25:002
15150993529,4cyclictest33-21ksoftirqd/222:10:012
15150993529,4cyclictest33-21ksoftirqd/200:28:392
15150993528,4cyclictest33-21ksoftirqd/221:55:052
15150993528,4cyclictest33-21ksoftirqd/221:25:332
15149993530,3cyclictest25-21ksoftirqd/120:30:391
15149993529,3cyclictest25-21ksoftirqd/123:55:241
15149993528,4cyclictest25-21ksoftirqd/100:35:081
15148993530,3cyclictest9-21ksoftirqd/022:40:000
15148993530,3cyclictest9-21ksoftirqd/019:20:290
15148993530,3cyclictest9-21ksoftirqd/019:20:280
15148993529,4cyclictest9-21ksoftirqd/020:30:340
15148993529,3cyclictest9-21ksoftirqd/022:15:240
15148993529,3cyclictest9-21ksoftirqd/000:09:040
15148993528,4cyclictest9-21ksoftirqd/020:20:180
1514899350,6cyclictest1100-21runrttasks22:55:520
1514899350,6cyclictest1100-21runrttasks22:55:510
57698340,5rtkit-daemon1131-21lldpd21:50:240
15150993428,4cyclictest33-21ksoftirqd/221:00:342
15150993428,3cyclictest33-21ksoftirqd/220:15:552
15150993427,4cyclictest33-21ksoftirqd/222:58:422
15150993427,4cyclictest33-21ksoftirqd/222:58:412
15150993427,4cyclictest33-21ksoftirqd/220:55:212
15150993427,4cyclictest33-21ksoftirqd/220:45:462
15150993426,4cyclictest33-21ksoftirqd/222:44:202
15150993424,8cyclictest33-21ksoftirqd/220:05:382
15148993428,4cyclictest9-21ksoftirqd/021:30:000
15148993428,3cyclictest9-21ksoftirqd/021:47:520
15148993427,4cyclictest9-21ksoftirqd/023:10:220
15148993427,4cyclictest9-21ksoftirqd/000:20:550
15150993328,3cyclictest33-21ksoftirqd/223:33:052
15150993327,4cyclictest33-21ksoftirqd/221:21:162
15150993327,4cyclictest33-21ksoftirqd/219:15:322
15150993326,5cyclictest33-21ksoftirqd/223:41:412
15150993326,4cyclictest33-21ksoftirqd/221:17:172
15150993326,4cyclictest33-21ksoftirqd/200:01:252
15148993328,3cyclictest9-21ksoftirqd/020:00:310
15148993327,4cyclictest9-21ksoftirqd/020:40:200
15148993327,4cyclictest9-21ksoftirqd/020:10:370
15148993327,4cyclictest9-21ksoftirqd/000:31:100
15148993327,3cyclictest9-21ksoftirqd/021:20:010
15148993326,4cyclictest9-21ksoftirqd/022:00:170
215022323,8sleep233-21ksoftirqd/221:45:142
15150993227,3cyclictest33-21ksoftirqd/223:00:362
15150993226,4cyclictest33-21ksoftirqd/220:00:392
15150993226,3cyclictest33-21ksoftirqd/221:36:292
15150993225,4cyclictest33-21ksoftirqd/222:36:332
15150993225,4cyclictest33-21ksoftirqd/221:14:452
15148993226,4cyclictest9-21ksoftirqd/023:05:270
15148993226,3cyclictest9-21ksoftirqd/019:28:080
15148993225,4cyclictest9-21ksoftirqd/021:00:380
15148993224,5cyclictest9-21ksoftirqd/023:37:220
15150993126,3cyclictest33-21ksoftirqd/221:50:312
15150993126,3cyclictest33-21ksoftirqd/219:30:362
15150993125,3cyclictest33-21ksoftirqd/200:13:132
15150993124,4cyclictest33-21ksoftirqd/200:00:002
15150993123,5cyclictest33-21ksoftirqd/221:43:562
15148993126,3cyclictest9-21ksoftirqd/021:05:270
15148993125,4cyclictest9-21ksoftirqd/000:12:350
15148993125,3cyclictest9-21ksoftirqd/022:30:230
15148993125,3cyclictest9-21ksoftirqd/000:25:500
15148993124,4cyclictest9-21ksoftirqd/023:54:480
15148993124,4cyclictest9-21ksoftirqd/022:44:590
15148993123,5cyclictest9-21ksoftirqd/019:41:110
15150993024,4cyclictest33-21ksoftirqd/223:50:292
15150993024,4cyclictest33-21ksoftirqd/220:26:112
15148993024,4cyclictest9-21ksoftirqd/021:58:160
15148993024,3cyclictest9-21ksoftirqd/023:47:250
15148993023,4cyclictest9-21ksoftirqd/019:13:370
15150992923,4cyclictest33-21ksoftirqd/222:12:582
15150992922,4cyclictest33-21ksoftirqd/220:33:292
15150992922,4cyclictest33-21ksoftirqd/200:37:272
15148992923,4cyclictest9-21ksoftirqd/019:50:400
15148992922,4cyclictest9-21ksoftirqd/022:26:250
2107422821,3sleep233-21ksoftirqd/223:10:262
15150992823,3cyclictest33-21ksoftirqd/220:22:482
15148992822,4cyclictest9-21ksoftirqd/023:27:020
15148992822,4cyclictest9-21ksoftirqd/020:08:300
15148992822,4cyclictest9-21ksoftirqd/019:30:350
15148992821,4cyclictest9-21ksoftirqd/022:10:170
15148992721,3cyclictest9-21ksoftirqd/000:01:470
15150992619,4cyclictest33-21ksoftirqd/222:24:002
15150992619,4cyclictest33-21ksoftirqd/219:50:012
15150992518,4cyclictest33-21ksoftirqd/222:46:332
15150992417,4cyclictest33-21ksoftirqd/219:47:262
15148992417,4cyclictest9-21ksoftirqd/000:00:000
15150992318,3cyclictest33-21ksoftirqd/220:42:442
15150992318,3cyclictest33-21ksoftirqd/200:21:502
15148992315,5cyclictest9-21ksoftirqd/019:39:470
1514899221,9cyclictest11775-21latency_hist21:25:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional