You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-10 - 18:18
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Sun Dec 10, 2023 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2914624435,0sleep10-21swapper/107:01:441
2912724132,0sleep30-21swapper/307:01:303
2899723827,0sleep20-21swapper/206:59:562
2892023423,0sleep00-21swapper/006:59:020
29372991515,0cyclictest3536-21snmpd08:46:150
29377991414,0cyclictest25114-21sensors09:48:281
293849983,0cyclictest28844-21awk12:23:212
293849983,0cyclictest26150-21awk09:13:212
293849973,0cyclictest5353-21awk12:28:202
293849970,0cyclictest27882-21kthreadcore10:48:252
4160,0ktimersoftd/00-21swapper/008:53:170
339960,0migration/36490-21sh10:37:563
339960,0migration/332364-21sh09:15:543
339960,0migration/328634-21sh10:29:083
29925060,0irq/32-eth0-rx-6399-21sh11:50:463
293849965,0cyclictest0-21swapper/212:01:272
293849965,0cyclictest0-21swapper/211:24:142
293849965,0cyclictest0-21swapper/209:03:182
293849962,0cyclictest8662-21awk11:53:202
293849960,0cyclictest0-21swapper/211:20:062
293849960,0cyclictest0-21swapper/209:54:272
293849960,0cyclictest0-21swapper/209:45:052
293849960,0cyclictest0-21swapper/207:18:272
149960,0migration/09125-21df_abs09:03:180
149960,0migration/025575-21kthreadcore11:43:220
149960,0migration/024179-21irqrtprio08:28:200
339950,0migration/3983-21aten_r1power_en09:18:153
339950,0migration/3777-21taskset08:54:053
339950,0migration/35763-21sh10:18:223
339950,0migration/34711-21fschecks_count12:28:173
339950,0migration/330896-21kthreadcore12:23:263
339950,0migration/330884-21taskset10:49:573
339950,0migration/327683-21kernelversion09:33:253
339950,0migration/327476-21cpuspeed_turbos09:53:173
339950,0migration/327253-21taskset11:25:393
339950,0migration/326172-21taskset12:19:393
339950,0migration/325026-21munin-node08:03:273
339950,0migration/320974-21fschecks_time10:08:183
339950,0migration/320687-21kthreadcore10:43:213
339950,0migration/320484-21diskmemload11:38:463
339950,0migration/319742-21gltestperf07:18:273
339950,0migration/317894-21munin-node07:13:313
339950,0migration/316973-21munin-node09:08:173
339950,0migration/311704-21latency_hist10:02:563
329950,0watchdog/38806-21expr10:58:123
329950,0watchdog/331677-21cut10:33:203
329950,0watchdog/329576-21kthreadcore07:23:333
329950,0watchdog/329572-21chrt10:13:203
329950,0watchdog/325457-21port_sshpriv08:48:283
329950,0watchdog/323232-21kthreadcore09:48:243
329950,0watchdog/32041-21sh11:13:083
329950,0watchdog/31802-21df_abs08:58:203
329950,0watchdog/310871-21munin-node07:53:323
29935050,0irq/33-eth0-tx-26237-21munin-node08:08:220
29935050,0irq/33-eth0-tx-0-21swapper/007:56:050
29935050,0irq/33-eth0-tx-0-21swapper/007:30:590
29935050,0irq/33-eth0-tx-0-21swapper/007:08:300
29925050,0irq/32-eth0-rx-3336-21munin-node09:58:203
29925050,0irq/32-eth0-rx-29546-21cut12:03:233
29925050,0irq/32-eth0-rx-28881-21grep11:08:223
29925050,0irq/32-eth0-rx-18820-21cut10:23:233
29925050,0irq/32-eth0-rx-10468-21idleruntime09:23:203
29925050,0irq/32-eth0-rx-0-21swapper/312:13:263
29925050,0irq/32-eth0-rx-0-21swapper/311:33:163
29925050,0irq/32-eth0-rx-0-21swapper/311:31:003
29925050,0irq/32-eth0-rx-0-21swapper/311:18:173
29925050,0irq/32-eth0-rx-0-21swapper/311:03:203
29925050,0irq/32-eth0-rx-0-21swapper/310:54:353
29925050,0irq/32-eth0-rx-0-21swapper/309:03:243
293849955,0cyclictest0-21swapper/208:58:342
293849955,0cyclictest0-21swapper/207:38:382
293849954,0cyclictest0-21swapper/210:17:332
293849952,0cyclictest9531-21awk08:18:242
293849952,0cyclictest3599-21lldpd11:03:122
293849950,0cyclictest0-21swapper/210:11:062
293849950,0cyclictest0-21swapper/210:07:442
293849950,0cyclictest0-21swapper/209:24:162
293729955,0cyclictest0-21swapper/011:20:230
293729955,0cyclictest0-21swapper/011:06:020
293729950,0cyclictest0-21swapper/010:17:380
293729950,0cyclictest0-21swapper/009:37:260
293729950,0cyclictest0-21swapper/009:16:190
27150,0ktimersoftd/20-21swapper/211:16:502
179950,0migration/18918-21kthreadcore11:33:211
179950,0migration/128287-21latency_hist11:27:561
179950,0migration/121180-21munin-node07:38:301
179950,0migration/120931-21kthreadcore12:18:201
169950,0watchdog/10-21swapper/112:28:241
149950,0migration/09102-21gltestperf07:33:210
149950,0migration/07626-21timerwakeupswit09:18:330
149950,0migration/07356-21kthreadcore08:38:260
149950,0migration/07173-21kthreadcore12:28:240
149950,0migration/06966-21kthreadcore11:33:200
149950,0migration/05259-21kthreadcore08:13:240
149950,0migration/03915-21kthreadcore10:53:250
149950,0migration/03709-21sh11:30:400
149950,0migration/03289-21taskset09:58:050
149950,0migration/032385-21timerandwakeup07:23:400
149950,0migration/032147-21sh11:48:160
149950,0migration/030959-21hwlatdetect08:33:200
149950,0migration/030632-21sh10:49:140
149950,0migration/028931-21taskset10:30:010
149950,0migration/028292-21kthreadcore07:43:250
149950,0migration/026748-21taskset11:24:190
149950,0migration/02458-21kthreadcore07:03:240
149950,0migration/021915-21kthreadcore07:18:310
149950,0migration/02164-21fschecks_time12:08:200
149950,0migration/019416-21munin-node10:23:250
149950,0migration/019199-21kthreadcore10:43:210
149950,0migration/018200-21sh09:47:040
149950,0migration/017294-21gltestperf11:58:180
149950,0migration/014113-21timerandwakeup11:53:320
149950,0migration/013671-21taskset10:39:300
149950,0migration/012994-21latency_hist10:22:550
8315040,0irq/26-0000:00:0-21swapper/307:33:253
339940,0migration/34591-21kthreadcore09:38:263
339940,0migration/34016-21kthreadcore07:28:293
339940,0migration/33853-21kthreadcore12:08:243
339940,0migration/324142-21diskmemload12:01:193
339940,0migration/323413-21sh11:43:183
339940,0migration/317455-21kthreadcore08:43:203
339940,0migration/314452-21kthreadcore07:58:283
329940,0watchdog/33360-21sendmail_mailqu07:03:293
329940,0watchdog/30-21swapper/308:13:243
329940,0watchdog/30-21swapper/307:38:283
29935040,0irq/33-eth0-tx-0-21swapper/212:19:552
29935040,0irq/33-eth0-tx-0-21swapper/209:39:512
29935040,0irq/33-eth0-tx-0-21swapper/008:18:270
29935040,0irq/33-eth0-tx-0-21swapper/008:03:320
29935040,0irq/33-eth0-tx-0-21swapper/007:38:200
29925040,0irq/32-eth0-rx-11294-21gltestperf09:43:203
29925040,0irq/32-eth0-rx-0-21swapper/311:53:223
29925040,0irq/32-eth0-rx-0-21swapper/309:28:223
29925040,0irq/32-eth0-rx-0-21swapper/307:08:273
293849944,0cyclictest0-21swapper/211:40:192
293849944,0cyclictest0-21swapper/210:54:272
293849943,0cyclictest4539-21awk09:58:202
293849943,0cyclictest3599-21lldpd10:41:362
293849943,0cyclictest3599-21lldpd10:27:592
293849943,0cyclictest3599-21lldpd09:33:012
293849943,0cyclictest3599-21lldpd07:13:002
293849943,0cyclictest3536-21snmpd09:51:502
293849943,0cyclictest3536-21snmpd09:28:502
293849943,0cyclictest3536-21snmpd08:33:502
293849943,0cyclictest3536-21snmpd08:31:202
293849943,0cyclictest3536-21snmpd08:03:342
293849943,0cyclictest3536-21snmpd07:03:042
293849943,0cyclictest3262-21irqbalance12:05:302
293849943,0cyclictest3262-21irqbalance10:23:002
293849943,0cyclictest3262-21irqbalance08:53:002
293849943,0cyclictest3262-21irqbalance08:48:202
293849943,0cyclictest3262-21irqbalance08:43:302
293849943,0cyclictest3262-21irqbalance08:38:302
293849943,0cyclictest3262-21irqbalance08:23:102
293849943,0cyclictest3262-21irqbalance08:15:402
293849943,0cyclictest3262-21irqbalance07:58:502
293849943,0cyclictest3262-21irqbalance07:53:202
293849943,0cyclictest3262-21irqbalance07:44:002
293849943,0cyclictest3262-21irqbalance07:33:002
293849943,0cyclictest3262-21irqbalance07:28:402
293849943,0cyclictest3262-21irqbalance07:23:002
293849943,0cyclictest3262-21irqbalance07:08:002
293849943,0cyclictest31350-21awk10:33:182
293849943,0cyclictest30910-21awk07:48:252
293849943,0cyclictest27291-21awk08:08:252
293849943,0cyclictest23511-21awk11:43:192
293849943,0cyclictest0-21swapper/210:18:202
293779944,0cyclictest2546-21nvme07:03:241
293779944,0cyclictest0-21swapper/111:38:161
293779944,0cyclictest0-21swapper/109:53:071
293779943,0cyclictest5927-21awk08:38:231
293779943,0cyclictest5475-21awk07:08:241
293779943,0cyclictest3536-21snmpd12:04:061
293779943,0cyclictest3262-21irqbalance11:13:401
293779943,0cyclictest3262-21irqbalance10:28:101
293779943,0cyclictest3262-21irqbalance10:11:001
293779943,0cyclictest31049-21awk08:33:201
293779943,0cyclictest3080-21awk09:38:241
293779943,0cyclictest20076-21awk08:03:201
293779943,0cyclictest14381-21awk10:23:201
293779943,0cyclictest13210-21awk08:43:171
293779943,0cyclictest12975-21awk10:03:171
293779940,0cyclictest0-21swapper/112:26:501
293729943,0cyclictest3536-21snmpd10:35:210
293729943,0cyclictest3262-21irqbalance12:03:000
293729943,0cyclictest3262-21irqbalance10:08:100
293729943,0cyclictest30789-21awk07:48:220
293729943,0cyclictest20069-21awk12:18:170
293729943,0cyclictest18929-21awk09:28:210
293729943,0cyclictest10501-21awk09:23:200
249940,0watchdog/22041-21kthreadcore11:48:202
249940,0watchdog/20-21swapper/210:58:122
249940,0watchdog/20-21swapper/209:08:202
179940,0migration/19297-21kthreadcore10:38:231
179940,0migration/18321-21fschecks_time11:53:191
179940,0migration/17862-21kthreadcore07:53:271
179940,0migration/14536-21timerwakeupswit10:33:291
179940,0migration/130031-21taskset11:45:501
179940,0migration/129116-21kthreadcore11:08:221
179940,0migration/12682-21kthreadcore10:53:251
179940,0migration/125152-21aten_r1power_vo09:13:161
179940,0migration/12466-21irqrtprio09:18:231
179940,0migration/122128-21runrttasks10:44:201
179940,0migration/121778-21kthreadcore07:18:311
179940,0migration/11764-21fschecks_time08:13:191
179940,0migration/111614-21irqcore12:13:211
179940,0migration/111482-21kthreadcore07:33:251
179940,0migration/110987-21kthreadcore10:58:171
169940,0watchdog/13235-21ssh09:57:561
169940,0watchdog/127467-21ssh11:26:121
169940,0watchdog/12285-21ntpq10:13:281
169940,0watchdog/122342-21cut09:28:241
169940,0watchdog/11803-21df_abs08:58:201
169940,0watchdog/117921-21perl11:03:201
169940,0watchdog/111973-21kthreadcore09:43:241
169940,0watchdog/110862-21ssh11:18:001
169940,0watchdog/10-21swapper/110:22:081
169940,0watchdog/10-21swapper/109:23:241
169940,0watchdog/10-21swapper/107:58:281
149940,0migration/04708-21kthreadcore11:13:240
149940,0migration/03345-21sh12:26:400
149940,0migration/032639-21kthreadcore09:53:250
149940,0migration/031555-21kthreadcore11:08:230
149940,0migration/02049-21fschecks_time08:58:210
149940,0migration/020088-21fschecks_time08:48:200
149940,0migration/018002-21timerwakeupswit07:13:310
149940,0migration/015562-21kthreadcore10:03:200
149940,0migration/010249-21kthreadcore10:58:170
8315030,0irq/26-0000:00:0-21swapper/308:33:043
339930,0migration/329433-21runrttasks08:29:293
329930,0watchdog/30-21swapper/308:23:243
329930,0watchdog/30-21swapper/308:18:203
29935030,0irq/33-eth0-tx-0-21swapper/212:12:572
29935030,0irq/33-eth0-tx-0-21swapper/212:08:082
29935030,0irq/33-eth0-tx-0-21swapper/211:28:022
29935030,0irq/33-eth0-tx-0-21swapper/211:08:112
29925030,0irq/32-eth0-rx-0-21swapper/308:38:193
29925030,0irq/32-eth0-rx-0-21swapper/308:08:213
29925030,0irq/32-eth0-rx-0-21swapper/307:48:183
29925030,0irq/32-eth0-rx-0-21swapper/307:43:163
293849932,0cyclictest0-21swapper/211:33:452
293849932,0cyclictest0-21swapper/209:17:592
293779932,0cyclictest9891-21awk09:03:221
293779932,0cyclictest3536-21snmpd08:19:351
293779932,0cyclictest32286-21awk11:48:171
293779932,0cyclictest29447-21grep07:23:321
293779932,0cyclictest25335-21grep10:48:251
293779932,0cyclictest21781-21sensors08:23:281
293779932,0cyclictest20442-21awk08:48:221
293779932,0cyclictest17961-21awk09:08:221
293779930,0cyclictest2631-21kthreadcore07:28:291
293779930,0cyclictest26208-21latency_hist08:07:561
293779930,0cyclictest24553-21kthreadcore08:28:211
293729933,0cyclictest31382-21kworker/0:209:49:300
293729933,0cyclictest31382-21kworker/0:209:39:280
293729933,0cyclictest21156-21kworker/0:012:15:260
293729933,0cyclictest17844-21kworker/0:107:58:210
293729932,0cyclictest17888-21awk09:08:210
293729932,0cyclictest15801-21awk08:23:170
293729932,0cyclictest14748-21awk11:38:170
249930,0watchdog/28473-21diskmemload10:43:122
179930,0migration/128069-21irqrtprio08:53:211
179930,0migration/117864-21irqrtprio11:58:211
169930,0watchdog/18473-21diskmemload09:33:241
169930,0watchdog/10-21swapper/112:08:241
169930,0watchdog/10-21swapper/107:43:241
169930,0watchdog/10-21swapper/107:13:211
169920,0watchdog/10-21swapper/107:47:561
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional