You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-22 - 05:06
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Sat Jun 22, 2024 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
112899102102,0cyclictest83150irq/26-0000:00:22:14:392
103829690,0sleep00-21swapper/018:50:570
82325041,0sleep30-21swapper/318:48:443
105724233,0sleep10-21swapper/118:51:101
91723828,0sleep20-21swapper/218:49:352
35160,0ktimersoftd/30-21swapper/321:35:143
35160,0ktimersoftd/30-21swapper/321:01:013
339960,0migration/37770-21switchtime18:51:423
339960,0migration/326172-21kthreadcore00:01:353
339960,0migration/324285-21sh23:22:043
339960,0migration/321948-21kernelversion20:11:333
339960,0migration/321297-21munin-node23:01:463
339960,0migration/316861-21sh22:24:293
339960,0migration/314469-21sh21:07:213
329960,0watchdog/32277-21sh21:38:043
29925060,0irq/32-eth0-rx-634-21diskstats22:16:263
29925060,0irq/32-eth0-rx-1329-21irqstats23:11:303
259960,0migration/227228-21idleruntime23:26:342
11289960,0cyclictest0-21swapper/222:33:282
35150,0ktimersoftd/30-21swapper/321:27:213
339950,0migration/38758-21munin-node23:31:373
339950,0migration/38692-21kthreadcore22:56:313
339950,0migration/36117-21taskset20:45:243
339950,0migration/35407-21kthreadcore19:36:313
339950,0migration/3528-21sh22:31:453
339950,0migration/35195-21switchtime21:01:473
339950,0migration/33504-21idleruntime20:21:343
339950,0migration/328399-21sh22:49:183
339950,0migration/323882-21missed_timers22:26:333
339950,0migration/323309-21runrttasks00:19:353
339950,0migration/322706-21switchtime21:11:463
339950,0migration/320041-21latency_hist23:41:113
339950,0migration/319937-21kthreadcore19:01:333
339950,0migration/318942-21munin-node21:46:433
339950,0migration/316968-21fschecks_count19:46:273
339950,0migration/313809-21diskmemload22:03:303
339950,0migration/313620-21kthreadcore20:26:343
339950,0migration/311913-21diskmemload23:51:523
329950,0watchdog/327805-21grep19:31:243
329950,0watchdog/324592-21fschecks_time22:11:283
329950,0watchdog/316747-21cut23:36:313
329950,0watchdog/312736-21python20:46:483
329950,0watchdog/311160-21ssh21:43:083
329950,0watchdog/30-21swapper/320:01:483
329950,0watchdog/30-21swapper/319:56:323
329950,0watchdog/30-21swapper/318:56:363
29935050,0irq/33-eth0-tx-8913-21kthreadcore22:01:312
29935050,0irq/33-eth0-tx-8620-21munin-node23:16:352
29935050,0irq/33-eth0-tx-29935-21chrt00:03:552
29935050,0irq/33-eth0-tx-27630-21cpu21:36:252
29935050,0irq/33-eth0-tx-249502chrt23:23:472
29935050,0irq/33-eth0-tx-21975-21ssh00:00:592
29935050,0irq/33-eth0-tx-20951-21cut21:51:112
29935050,0irq/33-eth0-tx-18203-21ssh23:36:522
29935050,0irq/33-eth0-tx-14849-21kthreadcore22:21:352
29935050,0irq/33-eth0-tx-1240-21munin-node21:21:312
29935050,0irq/33-eth0-tx-0-21swapper/223:35:252
29935050,0irq/33-eth0-tx-0-21swapper/019:40:320
29935050,0irq/33-eth0-tx-0-21swapper/019:31:430
29925050,0irq/32-eth0-rx-9886-21sh22:36:533
29925050,0irq/32-eth0-rx-7795-21fschecks_count00:11:283
29925050,0irq/32-eth0-rx-6086-21tune2fs19:16:233
29925050,0irq/32-eth0-rx-4191-21sh23:49:333
29925050,0irq/32-eth0-rx-29419-21ssh22:51:223
29925050,0irq/32-eth0-rx-27325-21if_eth123:26:353
29925050,0irq/32-eth0-rx-0-21swapper/323:16:453
29925050,0irq/32-eth0-rx-0-21swapper/322:07:533
29925050,0irq/32-eth0-rx-0-21swapper/321:58:243
29925050,0irq/32-eth0-rx-0-21swapper/321:51:363
29925050,0irq/32-eth0-rx-0-21swapper/321:21:413
29925050,0irq/32-eth0-rx-0-21swapper/319:27:463
29925050,0irq/32-eth0-rx-0-21swapper/300:06:363
259950,0migration/28593-21kthreadcore22:36:332
259950,0migration/232066-21kthreadcore19:31:332
259950,0migration/231769-21cpuspeed_turbos20:41:242
259950,0migration/228896-21switchtime20:56:462
259950,0migration/217927-21kthreadcore19:01:322
259950,0migration/21142-21kthreadcore23:46:372
249950,0watchdog/224832-21cut19:06:322
249950,0watchdog/222749-21ssh00:17:352
249950,0watchdog/20-21swapper/223:51:312
179950,0migration/15242-21taskset22:55:291
179950,0migration/129996-21taskset20:59:031
179950,0migration/127660-21aten_r1power_en20:16:271
179950,0migration/126063-21kthreadcore23:06:291
179950,0migration/125775-21taskset23:25:571
179950,0migration/123596-21munin-node00:01:321
179950,0migration/122220-21kthreadcore21:31:351
179950,0migration/116159-21hwlatdetect22:06:301
179950,0migration/115390-21kthreadcore21:46:351
179950,0migration/11131-21kthreadcore00:06:291
169950,0watchdog/12702-21sort20:21:281
169950,0watchdog/119167-21ssh23:39:231
149950,0migration/09982-21fschecks_time19:41:290
149950,0migration/09930-21kthreadcore21:06:310
149950,0migration/07652-21proc_pri21:21:400
149950,0migration/06385-21kthreadcore21:41:360
149950,0migration/06127-21idleruntime23:51:280
149950,0migration/06012-21aten_r1power_po20:01:230
149950,0migration/05312-21proc_pri20:41:420
149950,0migration/04661-21sh23:50:510
149950,0migration/04549-21kthreadcore18:51:330
149950,0migration/032231-21runrttasks21:18:110
149950,0migration/029101-21kthreadcore22:11:350
149950,0migration/02709-21latency_hist22:36:110
149950,0migration/026107-21kthreadcore20:56:350
149950,0migration/024878-21kthreadcore21:51:370
149950,0migration/021648-21kthreadcore21:11:360
149950,0migration/020722-21kthreadcore19:01:330
149950,0migration/020128-21cpuspeed_turbos19:26:240
149950,0migration/019353-21kthreadcore23:21:350
149950,0migration/017142-21fschecks_count19:46:270
149950,0migration/01695-21kthreadcore21:56:330
149950,0migration/014174-21kthreadcore22:41:360
149950,0migration/012350-21mdstat22:01:330
149950,0migration/01084-21kthreadcore19:56:320
11289955,0cyclictest0-21swapper/223:13:592
11289955,0cyclictest0-21swapper/221:57:412
11289955,0cyclictest0-21swapper/221:28:222
11289955,0cyclictest0-21swapper/221:11:012
11289954,0cyclictest62952sleep221:03:552
11279955,0cyclictest0-21swapper/119:56:371
11279950,0cyclictest0-21swapper/121:24:541
11279950,0cyclictest0-21swapper/120:11:451
8315040,0irq/26-0000:00:1246-21date18:51:122
8315040,0irq/26-0000:00:0-21swapper/219:56:262
8315040,0irq/26-0000:00:0-21swapper/219:26:332
339940,0migration/329099-21kthreadcore20:36:363
339940,0migration/317431-21kthreadcore20:06:323
329940,0watchdog/30-21swapper/323:56:233
329940,0watchdog/30-21swapper/320:51:363
329940,0watchdog/30-21swapper/320:31:323
329940,0watchdog/30-21swapper/319:51:323
329940,0watchdog/30-21swapper/319:06:323
29935040,0irq/33-eth0-tx-5378-21aten2.4-expect19:16:200
29935040,0irq/33-eth0-tx-16158-21sed22:06:302
29935040,0irq/33-eth0-tx-0-21swapper/223:42:452
29935040,0irq/33-eth0-tx-0-21swapper/223:07:092
29935040,0irq/33-eth0-tx-0-21swapper/223:01:212
29935040,0irq/33-eth0-tx-0-21swapper/222:51:282
29935040,0irq/33-eth0-tx-0-21swapper/222:16:262
29935040,0irq/33-eth0-tx-0-21swapper/221:52:342
29935040,0irq/33-eth0-tx-0-21swapper/221:16:482
29935040,0irq/33-eth0-tx-0-21swapper/220:53:042
29935040,0irq/33-eth0-tx-0-21swapper/200:12:072
29935040,0irq/33-eth0-tx-0-21swapper/020:31:430
29935040,0irq/33-eth0-tx-0-21swapper/019:00:250
29925040,0irq/32-eth0-rx-0-21swapper/323:06:393
29925040,0irq/32-eth0-rx-0-21swapper/322:41:163
29925040,0irq/32-eth0-rx-0-21swapper/321:17:103
259940,0migration/29369-21kthreadcore22:56:312
259940,0migration/28118-21kthreadcore21:41:382
259940,0migration/25181-21latency_hist19:16:122
259940,0migration/230488-21kthreadcore20:16:392
259940,0migration/228003-21kthreadcore20:36:362
259940,0migration/225016-21kthreadcore22:46:342
259940,0migration/223802-21aten_r1power_en19:51:212
259940,0migration/218681-21kthreadcore20:06:332
259940,0migration/217773-21kthreadcore19:21:352
249940,0watchdog/29720-21tr20:01:362
249940,0watchdog/23335-21perl20:21:322
249940,0watchdog/219424-21fschecks_time21:31:282
249940,0watchdog/213193-21seq20:48:042
249940,0watchdog/20-21swapper/219:16:322
249940,0watchdog/20-21swapper/200:06:232
179940,0migration/19181-21kthreadcore19:16:321
179940,0migration/18966-21cpuspeed_turbos18:56:271
179940,0migration/17920-21taskset23:15:061
179940,0migration/16254-21latency_hist20:46:121
179940,0migration/14344-21kthreadcore22:16:351
179940,0migration/130156-21kthreadcore21:36:371
179940,0migration/127769-21kthreadcore22:11:351
179940,0migration/127237-21kthreadcore20:36:351
179940,0migration/125451-21fschecks_count21:16:281
179940,0migration/123796-21munin-node19:51:211
179940,0migration/122373-21kthreadcore20:31:321
179940,0migration/117624-21if_eth019:46:301
179940,0migration/114149-21kthreadcore23:16:381
179940,0migration/113634-21idleruntime19:21:281
179940,0migration/113066-21kthreadcore22:21:341
169940,0watchdog/16461-21cut23:51:311
169940,0watchdog/129074-21ssh22:51:041
169940,0watchdog/127804-21tune2fs19:31:241
169940,0watchdog/115354-21kthreadcore21:26:361
169940,0watchdog/114740-21kthreadcore20:06:321
169940,0watchdog/113464-21ssh22:58:521
169940,0watchdog/10-21swapper/123:59:391
169940,0watchdog/10-21swapper/122:01:321
169940,0watchdog/10-21swapper/120:51:361
169940,0watchdog/10-21swapper/119:37:241
169940,0watchdog/10-21swapper/119:26:361
169940,0watchdog/10-21swapper/119:06:321
149940,0migration/09310-21fschecks_count23:16:290
149940,0migration/08404-21taskset22:20:080
149940,0migration/027865-21kthreadcore23:26:380
149940,0migration/027645-21aten_r1power_en20:16:270
149940,0migration/02694-21kthreadcore00:06:300
149940,0migration/025153-21fschecks_time20:36:290
149940,0migration/024292-21kthreadcore20:11:340
149940,0migration/023868-21missed_timers22:26:330
149940,0migration/014236-21kthreadcore20:26:340
149940,0migration/01047-21irqcore23:11:290
11289943,0cyclictest9743-21awk18:56:312
11289943,0cyclictest21498-21awk20:11:292
11289943,0cyclictest18928-21awk22:26:282
11289943,0cyclictest18101-21awk20:31:272
11289943,0cyclictest17528-21sort19:46:292
11289943,0cyclictest17392-21awk21:11:322
11289943,0cyclictest12907-21awk22:41:312
11289943,0cyclictest10368-21awk19:41:332
11279944,0cyclictest0-21swapper/123:41:261
11279943,0cyclictest8496-21awk00:11:321
11279943,0cyclictest6959-21awk20:01:311
11279943,0cyclictest4894-21awk21:41:311
11279943,0cyclictest4152-21awk22:36:281
11279943,0cyclictest3599-21lldpd23:32:411
11279943,0cyclictest3599-21lldpd23:27:101
11279943,0cyclictest3599-21lldpd23:01:391
11279943,0cyclictest3599-21lldpd20:49:371
11279943,0cyclictest23871-21snmpd21:02:391
11279943,0cyclictest22336-21awk21:51:331
11279943,0cyclictest18955-21awk22:26:281
11279943,0cyclictest17295-21awk21:11:311
11279943,0cyclictest16843-21awk00:16:291
11279943,0cyclictest16828-21sadc20:31:121
11279943,0cyclictest12941-21awk22:41:311
11279940,0cyclictest0-21swapper/122:33:011
11269943,0cyclictest7512-21awk20:46:340
11269943,0cyclictest3615-21awk23:31:270
11269943,0cyclictest3599-21lldpd19:06:330
11269943,0cyclictest3262-21irqbalance22:46:150
11269943,0cyclictest3262-21irqbalance00:13:250
11269943,0cyclictest28571-21awk21:36:340
11269943,0cyclictest24951-21awk19:51:280
11269943,0cyclictest23871-21snmpd23:46:090
11269943,0cyclictest23871-21snmpd23:39:390
11269943,0cyclictest23871-21snmpd23:05:240
11269943,0cyclictest23871-21snmpd21:34:090
11269943,0cyclictest14871-21awk20:51:320
11269943,0cyclictest14854-21awk23:56:310
329930,0watchdog/30-21swapper/319:41:283
329930,0watchdog/30-21swapper/319:15:203
29935030,0irq/33-eth0-tx-0-21swapper/019:21:210
29925030,0irq/32-eth0-rx-0-21swapper/319:25:293
249930,0watchdog/20-21swapper/219:36:242
169930,0watchdog/118884-21kthreadcore19:01:321
169930,0watchdog/113477-21diskmemload21:06:241
169930,0watchdog/10-21swapper/118:55:081
11289932,0cyclictest10995-21awk20:26:312
11279933,0cyclictest31529-21awk19:11:301
11279932,0cyclictest30821-21perl21:56:261
11279932,0cyclictest29820-21awk23:46:331
11279932,0cyclictest12507-21grep19:41:371
11269932,0cyclictest8835-21sensors20:21:460
11269932,0cyclictest6590-21tune2fs22:56:250
11269932,0cyclictest3599-21lldpd21:28:380
11269932,0cyclictest31616-21awk19:11:310
11269932,0cyclictest25865-21grep00:01:340
11269932,0cyclictest23871-21snmpd21:01:240
11269932,0cyclictest17543-21cut21:46:350
11269932,0cyclictest16267-21awk22:06:320
11269932,0cyclictest16132-21sed00:16:230
11269932,0cyclictest10197-21awk22:21:300
11269930,0cyclictest3677-21diskmemload22:51:450
11269930,0cyclictest0-21swapper/023:09:350
329920,0watchdog/30-21swapper/320:16:163
11269921,0cyclictest13181-21aten_r1power_cu20:06:220
11269920,0cyclictest2885-21latency_hist22:36:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional