You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-24 - 05:59
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Wed Jul 24, 2024 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
100524435,0sleep10-21swapper/118:46:191
97923728,0sleep20-21swapper/218:46:042
104123727,0sleep30-21swapper/318:46:453
3228523625,0sleep00-21swapper/018:44:340
329970,0watchdog/30-21swapper/323:54:343
29925070,0irq/32-eth0-rx-0-21swapper/319:44:383
339960,0migration/320105-21fschecks_time21:49:293
329960,0watchdog/3299250irq/32-eth0-rx-21:29:143
329960,0watchdog/328792-21taskset20:56:223
329960,0watchdog/31547-21perl00:09:253
329960,0watchdog/313553-21ps23:36:183
329960,0watchdog/30-21swapper/323:59:583
329960,0watchdog/30-21swapper/323:49:263
329960,0watchdog/30-21swapper/323:44:143
329960,0watchdog/30-21swapper/323:22:063
329960,0watchdog/30-21swapper/323:15:103
329960,0watchdog/30-21swapper/322:59:263
329960,0watchdog/30-21swapper/322:52:503
329960,0watchdog/30-21swapper/322:44:263
329960,0watchdog/30-21swapper/322:43:463
329960,0watchdog/30-21swapper/322:33:543
329960,0watchdog/30-21swapper/322:29:023
329960,0watchdog/30-21swapper/321:55:183
329960,0watchdog/30-21swapper/321:46:023
329960,0watchdog/30-21swapper/321:37:503
329960,0watchdog/30-21swapper/321:19:303
329960,0watchdog/30-21swapper/321:18:103
329960,0watchdog/30-21swapper/321:10:503
329960,0watchdog/30-21swapper/320:44:463
329960,0watchdog/30-21swapper/320:34:343
329960,0watchdog/30-21swapper/320:14:303
329960,0watchdog/30-21swapper/319:39:343
329960,0watchdog/30-21swapper/319:29:423
329960,0watchdog/30-21swapper/300:14:333
29925060,0irq/32-eth0-rx-6399-21tr21:39:343
29925060,0irq/32-eth0-rx-25141-21diskstats00:04:283
29925060,0irq/32-eth0-rx-0-21swapper/323:41:063
29925060,0irq/32-eth0-rx-0-21swapper/323:24:233
29925060,0irq/32-eth0-rx-0-21swapper/322:35:413
29925060,0irq/32-eth0-rx-0-21swapper/322:14:403
149960,0migration/03864-21idleruntime-cro23:14:130
149960,0migration/025820-21munin-node23:04:410
149960,0migration/017355-21idleruntime19:44:290
13199965,0cyclictest1099-21grep22:34:320
8315050,0irq/26-0000:00:25588-21latency_hist22:49:142
8315050,0irq/26-0000:00:0-21swapper/319:24:473
339950,0migration/33716-21kthreadcore19:34:323
339950,0migration/327539-21ntp_offset22:09:323
339950,0migration/316194-21kthreadcore19:19:343
339950,0migration/313751-21gltestperf22:04:283
339950,0migration/31029-21kthreadcore20:39:383
329950,0watchdog/34158-21awk19:09:383
329950,0watchdog/319632-21munin-node23:04:263
329950,0watchdog/319432-21cut18:59:343
329950,0watchdog/30-21swapper/323:30:183
329950,0watchdog/30-21swapper/322:55:303
329950,0watchdog/30-21swapper/322:19:423
329950,0watchdog/30-21swapper/321:59:143
329950,0watchdog/30-21swapper/321:04:343
329950,0watchdog/30-21swapper/320:59:343
329950,0watchdog/30-21swapper/320:30:303
329950,0watchdog/30-21swapper/320:25:223
329950,0watchdog/30-21swapper/320:21:063
329950,0watchdog/30-21swapper/320:09:263
329950,0watchdog/30-21swapper/320:04:263
329950,0watchdog/30-21swapper/319:59:303
329950,0watchdog/30-21swapper/319:56:263
329950,0watchdog/30-21swapper/319:49:183
329950,0watchdog/30-21swapper/319:16:063
329950,0watchdog/30-21swapper/319:05:103
329950,0watchdog/30-21swapper/318:51:503
29935050,0irq/33-eth0-tx-7873-21munin-node23:54:422
29935050,0irq/33-eth0-tx-4234-21munin-node21:59:402
29935050,0irq/33-eth0-tx-28993-21grep00:04:352
29935050,0irq/33-eth0-tx-15251-21sed21:24:432
29935050,0irq/33-eth0-tx-10749-21munin-node21:44:272
29935050,0irq/33-eth0-tx-0-21swapper/223:36:352
29935050,0irq/33-eth0-tx-0-21swapper/223:24:292
29935050,0irq/33-eth0-tx-0-21swapper/223:06:312
29935050,0irq/33-eth0-tx-0-21swapper/222:41:222
29935050,0irq/33-eth0-tx-0-21swapper/222:32:582
29935050,0irq/33-eth0-tx-0-21swapper/222:21:082
29935050,0irq/33-eth0-tx-0-21swapper/221:51:022
29935050,0irq/33-eth0-tx-0-21swapper/221:35:462
29935050,0irq/33-eth0-tx-0-21swapper/221:21:432
29935050,0irq/33-eth0-tx-0-21swapper/221:13:182
29935050,0irq/33-eth0-tx-0-21swapper/200:14:262
29935050,0irq/33-eth0-tx-0-21swapper/020:49:140
29925050,0irq/32-eth0-rx-27718-21cat23:09:143
29925050,0irq/32-eth0-rx-14927-21port_sshpriv18:54:373
29925050,0irq/32-eth0-rx-14087-21kthreadcore21:24:333
29925050,0irq/32-eth0-rx-13943-21ssh20:49:333
259950,0migration/231154-21munin-node20:39:302
259950,0migration/223806-21kthreadcore22:09:292
259950,0migration/213798-21if_eth1.319:19:322
259950,0migration/212765-21kthreadcore22:59:312
249950,0watchdog/24790-21fschecks_time23:14:262
249950,0watchdog/227415-21df_abs19:29:262
249950,0watchdog/216661-21cpuspeed_turbos19:44:262
249950,0watchdog/215588-21kthreadcore20:49:422
249950,0watchdog/20-21swapper/222:18:382
19150,0ktimersoftd/10-21swapper/123:11:171
179950,0migration/18476-21munin-node21:39:401
179950,0migration/14085-21timerandwakeup23:29:431
179950,0migration/127994-21timerwakeupswit22:09:391
179950,0migration/117707-21kthreadcore20:04:341
179950,0migration/114845-21latency_hist21:09:131
179950,0migration/112408-21kthreadcore22:39:351
179950,0migration/112376-21runrttasks22:03:451
179950,0migration/110294-21kthreadcore21:24:321
169950,0watchdog/10-21swapper/119:29:261
149950,0migration/09364-21kthreadcore22:19:320
149950,0migration/07790-21kthreadcore19:14:360
149950,0migration/07282-21kthreadcore21:59:320
149950,0migration/06944-21sh21:20:220
149950,0migration/06939-21munin-node20:44:330
149950,0migration/05143-21kthreadcore22:54:300
149950,0migration/04977-21taskset23:31:330
149950,0migration/032658-21sh00:08:390
149950,0migration/030377-21taskset21:14:500
149950,0migration/03010-21kthreadcore20:39:390
149950,0migration/030061-21kthreadcore21:54:330
149950,0migration/029267-21kthreadcore19:29:330
149950,0migration/027458-21kthreadcore19:49:320
149950,0migration/026479-21kthreadcore23:44:340
149950,0migration/023736-21unixbench_singl22:44:450
149950,0migration/023618-21munin-node21:29:390
149950,0migration/022701-21sh22:27:470
149950,0migration/021683-21kthreadcore20:29:340
149950,0migration/021638-21sh23:39:580
149950,0migration/018496-21sh21:47:360
149950,0migration/01723-21kthreadcore19:09:340
149950,0migration/015426-21runrttasks00:14:360
149950,0migration/015222-21kthreadcore19:19:340
149950,0migration/014724-21missed_timers18:54:350
149950,0migration/014523-21kthreadcore20:04:330
149950,0migration/014396-21timerwakeupswit23:54:420
149950,0migration/013588-21kthreadcore21:24:330
13209955,0cyclictest0-21swapper/123:52:451
13209955,0cyclictest0-21swapper/123:39:211
13209955,0cyclictest0-21swapper/100:00:281
13209950,0cyclictest0-21swapper/123:25:001
13209950,0cyclictest0-21swapper/122:05:131
13209950,0cyclictest0-21swapper/121:57:111
13209950,0cyclictest0-21swapper/120:56:391
13209950,0cyclictest0-21swapper/100:10:151
13199955,0cyclictest0-21swapper/022:39:460
13199955,0cyclictest0-21swapper/021:04:330
29935040,0irq/33-eth0-tx-31832-21ssh23:49:192
29935040,0irq/33-eth0-tx-21473-21munin-node20:54:382
29935040,0irq/33-eth0-tx-12687-21diskmemload23:09:192
29935040,0irq/33-eth0-tx-0-21swapper/223:29:182
29935040,0irq/33-eth0-tx-0-21swapper/222:56:202
29935040,0irq/33-eth0-tx-0-21swapper/222:44:152
29935040,0irq/33-eth0-tx-0-21swapper/221:29:242
29935040,0irq/33-eth0-tx-0-21swapper/221:14:192
29935040,0irq/33-eth0-tx-0-21swapper/200:09:142
29935040,0irq/33-eth0-tx-0-21swapper/020:34:250
29935040,0irq/33-eth0-tx-0-21swapper/020:14:300
29935040,0irq/33-eth0-tx-0-21swapper/020:09:250
29935040,0irq/33-eth0-tx-0-21swapper/018:59:380
259940,0migration/29599-21munin-node20:24:242
259940,0migration/28646-21kthreadcore19:14:362
259940,0migration/28270-21kthreadcore19:59:322
259940,0migration/26846-21fschecks_count21:04:282
259940,0migration/23223-21kthreadcore18:49:332
259940,0migration/232107-21kthreadcore20:59:322
259940,0migration/22269-21cpuspeed_turbos20:19:242
259940,0migration/215117-21cpuspeed_turbos22:24:242
259940,0migration/214781-21kthreadcore23:19:342
259940,0migration/212558-21kthreadcore18:54:342
249940,0watchdog/26958-21kthreadcore21:39:342
249940,0watchdog/232020-21kthreadcore19:09:342
249940,0watchdog/231457-21kthreadcore21:54:342
249940,0watchdog/226307-21grep23:44:342
249940,0watchdog/221804-21cut19:24:302
249940,0watchdog/217423-21smartctl20:29:302
249940,0watchdog/216178-21sh23:59:142
249940,0watchdog/20-21swapper/223:39:342
249940,0watchdog/20-21swapper/220:14:422
249940,0watchdog/20-21swapper/219:54:342
179940,0migration/17700-21kthreadcore23:14:301
179940,0migration/15367-21kthreadcore22:34:341
179940,0migration/129210-21kthreadcore22:49:301
179940,0migration/126255-21timerwakeupswit19:24:391
179940,0migration/125974-21kthreadcore19:04:331
179940,0migration/125809-21kthreadcore22:29:311
179940,0migration/125400-21kthreadcore20:34:311
179940,0migration/12360-21kthreadcore20:59:331
179940,0migration/120652-21munin-node23:04:301
179940,0migration/120169-21kthreadcore22:44:351
179940,0migration/115275-21kthreadcore22:59:321
179940,0migration/114761-21kthreadcore00:14:321
179940,0migration/112011-21switchtime20:44:441
169940,0watchdog/18646-21yum20:19:421
169940,0watchdog/17277-21hwlatdetect23:34:301
169940,0watchdog/1235432sleep119:49:221
169940,0watchdog/120578-21kthreadcore20:29:341
169940,0watchdog/11236-21cut20:39:381
169940,0watchdog/10-21swapper/121:34:301
169940,0watchdog/10-21swapper/119:59:301
169940,0watchdog/10-21swapper/119:54:341
149940,0migration/028617-21sh22:11:040
149940,0migration/024531-21idleruntime22:29:280
149940,0migration/024351-21sh00:04:000
149940,0migration/020748-21runrttasks23:23:150
149940,0migration/013807-21kthreadcore20:24:310
149940,0migration/01179-21kthreadcore23:49:310
13319943,0cyclictest750-21awk22:34:302
13319943,0cyclictest6841-21awk20:44:322
13319943,0cyclictest3599-21lldpd20:10:442
13319943,0cyclictest3599-21lldpd20:07:142
13319943,0cyclictest3599-21lldpd19:34:142
13319943,0cyclictest24538-21head20:34:262
13319943,0cyclictest24377-21awk19:04:312
13319943,0cyclictest17177-21awk18:59:312
13319943,0cyclictest13992-21awk22:04:312
13209943,0cyclictest7417-21awk21:04:301
13209943,0cyclictest3599-21lldpd20:26:441
13209943,0cyclictest3262-21irqbalance19:36:081
13209943,0cyclictest31158-21awk22:14:291
13209943,0cyclictest2716-21awk18:49:311
13209943,0cyclictest25872-21awk00:04:321
13209943,0cyclictest24553-21awk23:44:321
13209943,0cyclictest23871-21snmpd23:22:321
13209943,0cyclictest23871-21snmpd22:55:321
13209943,0cyclictest23871-21snmpd21:47:311
13209943,0cyclictest23871-21snmpd21:31:311
13209943,0cyclictest23871-21snmpd19:04:001
13209943,0cyclictest20466-21awk21:49:311
13209943,0cyclictest17506-21awk19:44:311
13209943,0cyclictest15939-21awk22:24:281
13209943,0cyclictest13966-21cut20:49:331
13209942,0cyclictest1089-21irqcore21:19:321
13199944,0cyclictest0-21swapper/022:04:180
13199944,0cyclictest0-21swapper/021:52:380
13199944,0cyclictest0-21swapper/020:54:460
13199943,0cyclictest3599-21lldpd20:59:440
13199943,0cyclictest31899-21awk19:54:310
13199943,0cyclictest31091-21awk22:14:290
13199943,0cyclictest30203-21sadc19:09:130
13199943,0cyclictest26731-21awk22:49:280
13199943,0cyclictest23871-21snmpd23:00:470
13199943,0cyclictest23871-21snmpd19:39:300
13199943,0cyclictest1825-21awk00:09:280
13199940,0cyclictest0-21swapper/021:34:200
249930,0watchdog/20-21swapper/219:39:342
179930,0migration/113860-21irqcore19:19:331
169930,0watchdog/10-21swapper/121:14:301
169930,0watchdog/10-21swapper/120:14:421
169930,0watchdog/10-21swapper/119:09:341
13319930,0cyclictest23969-21diskstats19:49:262
13209933,0cyclictest3599-21lldpd19:39:441
13209932,0cyclictest8936-21awk23:54:291
13209932,0cyclictest7390-21awk22:19:281
13209932,0cyclictest5443-21aten2.4-expect19:14:271
13209930,0cyclictest0-21swapper/120:13:251
13199933,0cyclictest30996-21kworker/0:123:25:040
13199933,0cyclictest30996-21kworker/0:123:14:180
13199933,0cyclictest29469-21kworker/0:121:09:320
13199933,0cyclictest29469-21kworker/0:120:19:140
13199933,0cyclictest29469-21kworker/0:119:59:280
13199933,0cyclictest29469-21kworker/0:119:24:200
13199933,0cyclictest18284-21kworker/0:221:39:340
13199933,0cyclictest1517-21kworker/0:018:49:150
13199932,0cyclictest8494-21kthreadcore23:34:330
13199932,0cyclictest2054-21tr19:34:250
13209920,0cyclictest0-21swapper/118:54:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional