You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-10-04 - 05:17
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack6slot3.osadl.org (updated Wed Oct 04, 2023 00:46:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1008326934,0sleep40-21swapper/419:05:264
1057926632,0sleep20-21swapper/219:07:002
1047425842,0sleep30-21swapper/319:05:323
1057425740,0sleep60-21swapper/619:06:566
1055025741,0sleep50-21swapper/519:06:395
1051625639,0sleep70-21swapper/719:06:107
877625539,0sleep10-21swapper/119:05:061
1066025235,0sleep00-21swapper/019:08:110
11113992524,0cyclictest13531-21kworker/0:200:07:270
11113992424,0cyclictest13531-21kworker/0:221:12:440
11113992423,0cyclictest13531-21kworker/0:223:15:180
11113992423,0cyclictest13531-21kworker/0:222:40:210
11113992423,0cyclictest13531-21kworker/0:222:39:490
11113992423,0cyclictest13531-21kworker/0:221:46:340
11113992423,0cyclictest13531-21kworker/0:220:54:580
11113992323,0cyclictest20687-21kworker/0:119:11:480
11113992323,0cyclictest13531-21kworker/0:223:47:280
11113992323,0cyclictest13531-21kworker/0:223:13:380
11113992322,0cyclictest13531-21kworker/0:223:31:570
11113992322,0cyclictest13531-21kworker/0:222:57:340
11113992322,0cyclictest13531-21kworker/0:222:22:370
11113992322,0cyclictest13531-21kworker/0:222:04:180
11113992322,0cyclictest13531-21kworker/0:221:30:280
11113992322,0cyclictest13531-21kworker/0:220:37:140
11113992322,0cyclictest13531-21kworker/0:220:21:080
11113992322,0cyclictest13531-21kworker/0:220:03:240
11113992222,0cyclictest13531-21kworker/0:222:05:580
11113992222,0cyclictest13531-21kworker/0:219:45:380
11113992222,0cyclictest13531-21kworker/0:219:30:080
11113992222,0cyclictest13531-21kworker/0:219:27:540
11113992221,0cyclictest13531-21kworker/0:222:17:130
11113992221,0cyclictest13531-21kworker/0:220:05:040
11113992221,0cyclictest13531-21kworker/0:200:22:580
11113992121,0cyclictest13531-21kworker/0:221:28:480
11113992121,0cyclictest13531-21kworker/0:220:56:380
11113992120,0cyclictest13531-21kworker/0:220:19:200
11113992120,0cyclictest12-21rcu_preempt21:55:290
11113991817,0cyclictest13531-21kworker/0:220:31:040
11133991513,0cyclictest11343-21expr19:10:183
11133991310,0cyclictest1999350irq/28-enp2s0f021:32:313
1112099130,0cyclictest0-21swapper/121:10:311
1112699120,0cyclictest0-21swapper/223:14:532
1112699120,0cyclictest0-21swapper/222:29:102
1112699120,0cyclictest0-21swapper/221:35:222
1112699120,0cyclictest0-21swapper/221:26:182
1112699120,0cyclictest0-21swapper/200:17:012
1112699110,0cyclictest3857-21ls22:20:222
1112699110,0cyclictest0-21swapper/223:26:162
1112699110,0cyclictest0-21swapper/223:05:242
1112699110,0cyclictest0-21swapper/222:40:192
1112699110,0cyclictest0-21swapper/222:38:192
1112699110,0cyclictest0-21swapper/222:33:022
1112699110,0cyclictest0-21swapper/221:32:432
1112699110,0cyclictest0-21swapper/221:20:192
1112699110,0cyclictest0-21swapper/220:53:002
1112699110,0cyclictest0-21swapper/220:45:262
1112699110,0cyclictest0-21swapper/220:37:492
1112699110,0cyclictest0-21swapper/220:05:232
1112699110,0cyclictest0-21swapper/219:27:022
1112699110,0cyclictest0-21swapper/219:20:592
1112699110,0cyclictest0-21swapper/200:25:562
1112699104,0cyclictest648-21idleruntime-cro19:55:012
1112699103,0cyclictest9634-21cat21:50:262
1112699100,0cyclictest0-21swapper/223:35:302
1112699100,0cyclictest0-21swapper/223:30:152
1112699100,0cyclictest0-21swapper/221:16:262
1112699100,0cyclictest0-21swapper/219:41:412
1112699100,0cyclictest0-21swapper/200:07:292
199935090,0irq/28-enp2s0f011189-21ssh22:29:533
111269996,0cyclictest0-21swapper/223:58:002
111269996,0cyclictest0-21swapper/223:58:002
111269994,0cyclictest831-21runrttasks19:37:242
111269994,0cyclictest4221-21diskmemload21:55:052
111269993,0cyclictest9656-21sort23:40:202
111269993,0cyclictest4710-21cat21:45:242
111269993,0cyclictest3677-21cstates20:00:182
111269993,0cyclictest28814-21tr19:45:172
111269993,0cyclictest17951-21tr20:30:182
111269992,0cyclictest31719-21cpuspeed_turbos21:40:202
111269992,0cyclictest3133-21sed21:05:222
111269990,0cyclictest12858-21grep00:20:182
111269990,0cyclictest0-21swapper/223:21:142
111269990,0cyclictest0-21swapper/223:18:032
111269990,0cyclictest0-21swapper/222:13:222
111269990,0cyclictest0-21swapper/221:14:222
111269990,0cyclictest0-21swapper/220:56:182
111269990,0cyclictest0-21swapper/220:16:512
111269990,0cyclictest0-21swapper/200:10:382
71180,0ktimersoftd/70-21swapper/723:05:247
71180,0ktimersoftd/70-21swapper/700:17:017
609980,0migration/614985-21gltestperf19:15:196
55180,0ktimersoftd/50-21swapper/500:17:015
39180,0ktimersoftd/30-21swapper/323:14:533
23180,0ktimersoftd/10-21swapper/119:27:021
199975080,0irq/31-enp2s0f0549-21systemd-journal21:56:394
199955080,0irq/30-enp2s0f00-21swapper/521:14:285
199935080,0irq/28-enp2s0f032495-21irqstats21:40:243
199935080,0irq/28-enp2s0f023550-21chrt23:19:503
199935080,0irq/28-enp2s0f00-21swapper/321:59:163
111549987,0cyclictest0-21swapper/721:34:317
111439980,0cyclictest4064-21processes22:20:255
111379987,0cyclictest0-21swapper/420:15:174
111379980,0cyclictest13104-21df00:20:194
111269983,0cyclictest22261-21sort22:05:262
111269982,0cyclictest2643-21sed19:55:282
111269982,0cyclictest14607-21fschecks_time23:45:192
111269981,0cyclictest8172-21cat20:10:002
111269981,0cyclictest13638-21cat19:15:012
111269981,0cyclictest11874-21gltestperf19:10:212
111269980,0cyclictest820-21snmpd19:34:472
111269980,0cyclictest32386-21cat21:00:192
111269980,0cyclictest30053-21df_abs22:50:172
111269980,0cyclictest27710-21ssh22:47:052
111269980,0cyclictest27294-21wc00:00:012
111269980,0cyclictest26182-21cpuspeed_turbos00:35:192
111269980,0cyclictest1972-21cstates22:55:202
111269980,0cyclictest19198-21head23:50:212
111269980,0cyclictest16101-21grep20:25:242
111269980,0cyclictest0-21swapper/223:01:112
111269980,0cyclictest0-21swapper/222:15:062
111269980,0cyclictest0-21swapper/222:00:252
111269980,0cyclictest0-21swapper/220:40:042
111269980,0cyclictest0-21swapper/220:20:172
111269980,0cyclictest0-21swapper/200:30:442
111139983,0cyclictest9102-21ssh23:40:000
10180,0ktimersoftd/00-21swapper/000:17:020
71170,0ktimersoftd/70-21swapper/723:14:547
71170,0ktimersoftd/70-21swapper/722:02:357
71170,0ktimersoftd/70-21swapper/721:58:397
71170,0ktimersoftd/70-21swapper/721:26:197
71170,0ktimersoftd/70-21swapper/720:53:017
71170,0ktimersoftd/70-21swapper/720:45:267
71170,0ktimersoftd/70-21swapper/719:27:027
71170,0ktimersoftd/70-21swapper/700:29:297
689970,0migration/77489-21cstates22:25:187
63170,0ktimersoftd/60-21swapper/623:05:256
63170,0ktimersoftd/60-21swapper/622:42:006
63170,0ktimersoftd/60-21swapper/622:38:206
63170,0ktimersoftd/60-21swapper/622:33:016
63170,0ktimersoftd/60-21swapper/622:29:526
63170,0ktimersoftd/60-21swapper/621:58:396
63170,0ktimersoftd/60-21swapper/621:32:426
63170,0ktimersoftd/60-21swapper/619:27:026
63170,0ktimersoftd/60-21swapper/600:29:286
63170,0ktimersoftd/60-21swapper/600:17:026
609970,0migration/610016-21df_abs23:40:226
55170,0ktimersoftd/50-21swapper/522:33:025
55170,0ktimersoftd/50-21swapper/521:58:385
55170,0ktimersoftd/50-21swapper/520:53:005
47170,0ktimersoftd/40-21swapper/422:40:184
47170,0ktimersoftd/40-21swapper/420:53:014
39170,0ktimersoftd/30-21swapper/322:42:003
39170,0ktimersoftd/30-21swapper/322:20:233
23170,0ktimersoftd/10-21swapper/122:40:181
23170,0ktimersoftd/10-21swapper/121:32:431
23170,0ktimersoftd/10-21swapper/120:45:271
23170,0ktimersoftd/10-21swapper/100:29:291
23170,0ktimersoftd/10-21swapper/100:17:021
209970,0migration/114645-21taskset22:32:491
199985070,0irq/32-enp2s0f08001-21ssh23:00:361
199985070,0irq/32-enp2s0f027712-21ssh23:24:141
199985070,0irq/32-enp2s0f00-21swapper/123:46:561
199975070,0irq/31-enp2s0f00-21swapper/423:51:364
199975070,0irq/31-enp2s0f00-21swapper/423:30:354
199975070,0irq/31-enp2s0f00-21swapper/423:14:294
199975070,0irq/31-enp2s0f00-21swapper/422:33:384
199975070,0irq/31-enp2s0f00-21swapper/419:40:234
199955070,0irq/30-enp2s0f00-21swapper/523:45:465
199955070,0irq/30-enp2s0f00-21swapper/523:39:485
199955070,0irq/30-enp2s0f00-21swapper/523:21:305
199955070,0irq/30-enp2s0f00-21swapper/523:06:245
199955070,0irq/30-enp2s0f00-21swapper/522:45:255
199955070,0irq/30-enp2s0f00-21swapper/522:19:365
199955070,0irq/30-enp2s0f00-21swapper/522:07:395
199955070,0irq/30-enp2s0f00-21swapper/521:41:185
199955070,0irq/30-enp2s0f00-21swapper/521:37:395
199955070,0irq/30-enp2s0f00-21swapper/521:25:505
199955070,0irq/30-enp2s0f00-21swapper/521:24:595
199955070,0irq/30-enp2s0f00-21swapper/500:36:375
199955070,0irq/30-enp2s0f00-21swapper/500:33:455
199955070,0irq/30-enp2s0f00-21swapper/500:25:015
199955070,0irq/30-enp2s0f00-21swapper/500:23:335
199955070,0irq/30-enp2s0f00-21swapper/500:07:195
199955070,0irq/30-enp2s0f00-21swapper/500:00:185
199945070,0irq/29-enp2s0f031294-21grep22:15:197
199935070,0irq/28-enp2s0f024536-21ssh23:20:163
199935070,0irq/28-enp2s0f011952-21cstates22:30:173
199935070,0irq/28-enp2s0f00-21swapper/323:59:013
199935070,0irq/28-enp2s0f00-21swapper/323:59:013
199935070,0irq/28-enp2s0f00-21swapper/323:53:073
199935070,0irq/28-enp2s0f00-21swapper/323:42:423
199935070,0irq/28-enp2s0f00-21swapper/323:38:143
199935070,0irq/28-enp2s0f00-21swapper/323:34:003
199935070,0irq/28-enp2s0f00-21swapper/323:26:463
199935070,0irq/28-enp2s0f00-21swapper/323:09:133
199935070,0irq/28-enp2s0f00-21swapper/323:01:073
199935070,0irq/28-enp2s0f00-21swapper/322:59:053
199935070,0irq/28-enp2s0f00-21swapper/322:47:313
199935070,0irq/28-enp2s0f00-21swapper/322:15:433
199935070,0irq/28-enp2s0f00-21swapper/322:10:173
199935070,0irq/28-enp2s0f00-21swapper/322:07:393
199935070,0irq/28-enp2s0f00-21swapper/321:39:083
199935070,0irq/28-enp2s0f00-21swapper/320:10:333
111549976,0cyclictest0-21swapper/723:57:167
111549976,0cyclictest0-21swapper/723:57:167
111549976,0cyclictest0-21swapper/723:50:567
111549976,0cyclictest0-21swapper/721:47:087
111549976,0cyclictest0-21swapper/721:36:257
111549975,0cyclictest15790-21sh00:23:137
111549970,0cyclictest0-21swapper/723:42:427
111549970,0cyclictest0-21swapper/700:00:247
111439976,0cyclictest0-21swapper/519:55:255
111439970,0cyclictest0-21swapper/522:51:015
111379977,0cyclictest0-21swapper/421:22:474
111379976,0cyclictest0-21swapper/423:02:104
111379976,0cyclictest0-21swapper/421:25:274
111379970,0cyclictest0-21swapper/423:45:484
111379970,0cyclictest0-21swapper/423:43:544
111379970,0cyclictest0-21swapper/422:10:204
111379970,0cyclictest0-21swapper/400:19:024
111339976,0cyclictest0-21swapper/321:14:223
111339970,0cyclictest0-21swapper/322:50:043
111209975,0cyclictest9648-21cpuspeed_turbos23:40:211
111139973,0cyclictest0-21swapper/021:55:010
10170,0ktimersoftd/04221-21diskmemload23:42:590
71160,0ktimersoftd/70-21swapper/722:40:077
71160,0ktimersoftd/70-21swapper/720:20:487
63160,0ktimersoftd/60-21swapper/621:35:216
63160,0ktimersoftd/60-21swapper/621:13:536
63160,0ktimersoftd/60-21swapper/600:09:056
609960,0migration/627547-21cpuspeed_turbos20:50:176
55160,0ktimersoftd/50-21swapper/523:30:155
55160,0ktimersoftd/50-21swapper/522:40:195
47160,0ktimersoftd/44550-21ssh22:20:314
47160,0ktimersoftd/428608-21ssh00:37:114
47160,0ktimersoftd/40-21swapper/423:08:344
47160,0ktimersoftd/40-21swapper/422:29:524
47160,0ktimersoftd/40-21swapper/420:05:234
47160,0ktimersoftd/40-21swapper/419:20:584
449960,0migration/431630-21taskset20:57:264
449960,0migration/427628-21taskset22:46:564
449960,0migration/41157-21timerwakeupswit00:05:264
39160,0ktimersoftd/30-21swapper/322:00:483
39160,0ktimersoftd/30-21swapper/321:15:373
39160,0ktimersoftd/30-21swapper/321:05:103
39160,0ktimersoftd/30-21swapper/320:55:533
39160,0ktimersoftd/30-21swapper/320:53:003
39160,0ktimersoftd/30-21swapper/320:40:053
39160,0ktimersoftd/30-21swapper/320:35:273
39160,0ktimersoftd/30-21swapper/320:33:003
39160,0ktimersoftd/30-21swapper/320:00:343
39160,0ktimersoftd/30-21swapper/319:58:073
39160,0ktimersoftd/30-21swapper/319:35:013
39160,0ktimersoftd/30-21swapper/319:27:033
39160,0ktimersoftd/30-21swapper/319:22:003
39160,0ktimersoftd/30-21swapper/300:35:433
39160,0ktimersoftd/30-21swapper/300:30:543
23160,0ktimersoftd/10-21swapper/123:38:081
23160,0ktimersoftd/10-21swapper/121:29:261
23160,0ktimersoftd/10-21swapper/121:00:401
23160,0ktimersoftd/10-21swapper/119:41:401
199985060,0irq/32-enp2s0f00-21swapper/123:55:211
199985060,0irq/32-enp2s0f00-21swapper/123:55:201
199985060,0irq/32-enp2s0f00-21swapper/123:25:191
199985060,0irq/32-enp2s0f00-21swapper/121:43:451
199985060,0irq/32-enp2s0f00-21swapper/100:38:161
199985060,0irq/32-enp2s0f00-21swapper/100:10:461
199985060,0irq/32-enp2s0f00-21swapper/100:08:461
199985060,0irq/32-enp2s0f00-21swapper/100:04:321
199975060,0irq/31-enp2s0f023484-21ssh00:30:464
199975060,0irq/31-enp2s0f021640-21ssh23:15:584
199975060,0irq/31-enp2s0f00-21swapper/423:59:294
199975060,0irq/31-enp2s0f00-21swapper/423:59:294
199975060,0irq/31-enp2s0f00-21swapper/422:18:594
199975060,0irq/31-enp2s0f00-21swapper/421:47:284
199955060,0irq/30-enp2s0f015613-21df_inode23:10:215
199955060,0irq/30-enp2s0f00-21swapper/523:56:395
199955060,0irq/30-enp2s0f00-21swapper/523:56:385
199955060,0irq/30-enp2s0f00-21swapper/523:50:245
199955060,0irq/30-enp2s0f00-21swapper/523:40:235
199955060,0irq/30-enp2s0f00-21swapper/523:00:185
199955060,0irq/30-enp2s0f00-21swapper/522:57:195
199955060,0irq/30-enp2s0f00-21swapper/522:39:555
199955060,0irq/30-enp2s0f00-21swapper/522:11:415
199955060,0irq/30-enp2s0f00-21swapper/522:02:405
199955060,0irq/30-enp2s0f00-21swapper/521:50:085
199955060,0irq/30-enp2s0f00-21swapper/521:45:515
199955060,0irq/30-enp2s0f00-21swapper/521:30:595
199955060,0irq/30-enp2s0f00-21swapper/521:15:225
199955060,0irq/30-enp2s0f00-21swapper/520:22:395
199955060,0irq/30-enp2s0f00-21swapper/500:11:215
199945060,0irq/29-enp2s0f011159-21rm21:15:577
199945060,0irq/29-enp2s0f00-21swapper/723:45:327
199945060,0irq/29-enp2s0f00-21swapper/723:36:147
199945060,0irq/29-enp2s0f00-21swapper/723:26:097
199945060,0irq/29-enp2s0f00-21swapper/723:15:207
199945060,0irq/29-enp2s0f00-21swapper/722:32:097
199945060,0irq/29-enp2s0f00-21swapper/721:50:167
199945060,0irq/29-enp2s0f00-21swapper/721:44:217
199945060,0irq/29-enp2s0f00-21swapper/721:22:377
199945060,0irq/29-enp2s0f00-21swapper/721:14:397
199945060,0irq/29-enp2s0f00-21swapper/721:08:317
199945060,0irq/29-enp2s0f00-21swapper/720:35:327
199945060,0irq/29-enp2s0f00-21swapper/719:38:387
199945060,0irq/29-enp2s0f00-21swapper/700:36:327
199935060,0irq/28-enp2s0f00-21swapper/323:45:563
199935060,0irq/28-enp2s0f00-21swapper/322:36:493
199935060,0irq/28-enp2s0f00-21swapper/321:50:233
199935060,0irq/28-enp2s0f00-21swapper/321:45:013
199935060,0irq/28-enp2s0f00-21swapper/321:25:163
199935060,0irq/28-enp2s0f00-21swapper/321:21:163
199935060,0irq/28-enp2s0f00-21swapper/300:26:063
199935060,0irq/28-enp2s0f00-21swapper/300:20:293
199935060,0irq/28-enp2s0f00-21swapper/300:16:033
199935060,0irq/28-enp2s0f00-21swapper/300:11:033
199935060,0irq/28-enp2s0f00-21swapper/300:05:413
169960,0migration/06780-21fw_forwarded_lo23:00:220
169960,0migration/029814-21ssh23:25:260
169960,0migration/026217-21missed_timers22:45:260
169960,0migration/017175-21ssh21:24:020
169960,0migration/014093-21runrttasks23:08:510
169960,0migration/010183-21taskset20:13:360
169960,0migration/010017-21irqstats21:15:260
111549966,0cyclictest711ktimersoftd/722:53:347
111549966,0cyclictest0-21swapper/700:07:227
111549965,0cyclictest23834-21sh22:06:157
111549964,0cyclictest18763-21ls19:24:597
111549960,0cyclictest0-21swapper/722:24:117
111549960,0cyclictest0-21swapper/722:13:577
111549960,0cyclictest0-21swapper/719:44:537
111509966,0cyclictest0-21swapper/622:45:226
111509965,0cyclictest32436-21smart_sda19:50:316
111509965,0cyclictest31571-21chrt20:56:376
111509964,0cyclictest8894-21cut20:10:246
111509964,0cyclictest831-21runrttasks21:15:226
111509964,0cyclictest831-21runrttasks19:47:456
111509964,0cyclictest3656-21sh23:33:026
111509964,0cyclictest14964-21sh00:21:316
111509960,0cyclictest4221-21diskmemload22:13:426
111509960,0cyclictest0-21swapper/622:55:056
111509960,0cyclictest0-21swapper/619:13:056
111439965,0cyclictest14507-21tr19:15:175
111439960,0cyclictest30908-21ssh23:26:415
111439960,0cyclictest22041-21ssh23:16:475
111379964,0cyclictest4221-21diskmemload22:59:564
111379964,0cyclictest31660-21sh23:28:104
111379964,0cyclictest20230-21latency_hist22:40:004
111379960,0cyclictest0-21swapper/421:35:254
111379960,0cyclictest0-21swapper/421:03:014
111379960,0cyclictest0-21swapper/419:36:574
111379960,0cyclictest0-21swapper/400:00:424
111339965,0cyclictest0-21swapper/300:00:143
111209966,0cyclictest0-21swapper/121:15:221
111209965,0cyclictest7090-21pmu-power20:05:261
111209965,0cyclictest25032-21sh00:33:581
111209965,0cyclictest21076-21switchtime23:15:301
111209965,0cyclictest16173-21ssh23:10:271
111209965,0cyclictest13830-21ssh23:08:191
111209964,0cyclictest28072-21ssh22:10:451
111209964,0cyclictest24214-21rm22:07:011
111209960,0cyclictest0-21swapper/123:52:141
111209960,0cyclictest0-21swapper/123:31:451
111209960,0cyclictest0-21swapper/122:59:121
111209960,0cyclictest0-21swapper/122:22:341
111209960,0cyclictest0-21swapper/121:39:581
111209960,0cyclictest0-21swapper/121:22:331
111209960,0cyclictest0-21swapper/120:58:091
111209960,0cyclictest0-21swapper/120:42:511
111209960,0cyclictest0-21swapper/119:50:331
111209960,0cyclictest0-21swapper/119:35:561
111209960,0cyclictest0-21swapper/119:21:231
111139964,0cyclictest27882-21idleruntime21:35:220
111139960,0cyclictest0-21swapper/023:51:450
10160,0ktimersoftd/00-21swapper/020:45:260
10160,0ktimersoftd/00-21swapper/019:20:580
71150,0ktimersoftd/70-21swapper/723:30:147
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional