You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-29 - 10:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Thu Feb 29, 2024 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
763125242,0sleep30-21swapper/318:55:493
768324948,0sleep10-21swapper/118:56:231
74924241,0sleep00-21swapper/018:53:100
78323826,0sleep20-21swapper/218:53:242
79239960,0cyclictest0-21swapper/222:37:152
79119960,0cyclictest26824-21ssh22:36:540
339960,0migration/316418-21sh23:44:273
339960,0migration/314390-21sh21:50:203
329960,0watchdog/329651-21cut22:58:303
29935060,0irq/33-eth0-tx-20549-21hddtemp_smartct20:58:340
29925060,0irq/32-eth0-rx-9812-21grep21:08:323
29925060,0irq/32-eth0-rx-32531-21port_sshpriv20:18:373
149960,0migration/0469-21dump-pmu-power23:38:050
149960,0migration/01215-21kthreadcore20:43:370
79239955,0cyclictest0-21swapper/222:21:012
79239955,0cyclictest0-21swapper/200:10:062
79239954,0cyclictest24138-21cut19:53:352
79239950,0cyclictest0-21swapper/222:08:142
79199954,0cyclictest3262-21irqbalance19:18:311
79199954,0cyclictest18646-21sadc00:08:061
79119955,0cyclictest2792-21kworker/0:123:52:560
79119955,0cyclictest2792-21kworker/0:123:44:300
79119955,0cyclictest2792-21kworker/0:123:28:480
79119955,0cyclictest2792-21kworker/0:123:20:560
79119955,0cyclictest2792-21kworker/0:123:13:140
79119955,0cyclictest2792-21kworker/0:123:12:220
79119955,0cyclictest2792-21kworker/0:123:06:200
79119955,0cyclictest2792-21kworker/0:122:54:490
79119955,0cyclictest2792-21kworker/0:122:29:030
79119955,0cyclictest2792-21kworker/0:122:24:270
79119955,0cyclictest2792-21kworker/0:100:27:220
79119955,0cyclictest25091-21kworker/0:022:06:210
79119955,0cyclictest25091-21kworker/0:021:58:290
79119955,0cyclictest25091-21kworker/0:021:48:570
79119955,0cyclictest25091-21kworker/0:021:38:250
79119955,0cyclictest25091-21kworker/0:021:36:470
79119955,0cyclictest25091-21kworker/0:021:25:210
79119955,0cyclictest25091-21kworker/0:021:09:550
79119955,0cyclictest25091-21kworker/0:021:08:010
79119955,0cyclictest25091-21kworker/0:020:50:290
79119955,0cyclictest25091-21kworker/0:020:38:090
79119955,0cyclictest25091-21kworker/0:020:33:150
79119955,0cyclictest25091-21kworker/0:020:28:590
79119955,0cyclictest25091-21kworker/0:020:23:570
79119955,0cyclictest25091-21kworker/0:020:13:570
79119955,0cyclictest25091-21kworker/0:020:08:110
79119955,0cyclictest25091-21kworker/0:020:03:570
79119955,0cyclictest25091-21kworker/0:019:58:150
79119955,0cyclictest25091-21kworker/0:019:54:110
79119955,0cyclictest25091-21kworker/0:019:48:270
79119955,0cyclictest25091-21kworker/0:019:43:190
79119955,0cyclictest25091-21kworker/0:019:38:470
79119955,0cyclictest25091-21kworker/0:019:33:130
79119955,0cyclictest25091-21kworker/0:019:29:190
79119955,0cyclictest25091-21kworker/0:019:23:290
79119955,0cyclictest25091-21kworker/0:019:18:170
79119955,0cyclictest25091-21kworker/0:019:13:170
79119955,0cyclictest25091-21kworker/0:019:03:570
79119955,0cyclictest25091-21kworker/0:019:00:410
79119955,0cyclictest0-21swapper/000:10:420
339950,0migration/36097-21taskset23:19:163
339950,0migration/34601-21kthreadcore23:38:343
339950,0migration/33229-21fschecks_time23:58:343
339950,0migration/328915-21kthreadcore19:33:383
339950,0migration/328816-21munin-node21:38:383
339950,0migration/328656-21kthreadcore23:13:373
339950,0migration/327526-21kthreadcore23:33:353
339950,0migration/326510-21aten_r1power_po23:53:253
339950,0migration/326499-21kthreadcore20:38:373
339950,0migration/326263-21kthreadcore21:58:323
339950,0migration/323763-21kthreadcore23:48:433
339950,0migration/322083-21timerandwakeup20:33:423
339950,0migration/321641-21switchtime19:03:453
339950,0migration/320148-21kthreadcore23:08:353
339950,0migration/320042-21sh22:51:133
339950,0migration/31898-21fschecks_time20:23:283
339950,0migration/315626-21kthreadcore21:13:353
339950,0migration/314952-21timerwakeupswit20:28:433
339950,0migration/314311-21kthreadcore22:28:373
339950,0migration/314298-21munin-node19:43:373
339950,0migration/311006-21sendmail_mailqu20:03:373
339950,0migration/310912-21taskset00:23:163
339950,0migration/310716-21latency_hist00:23:063
329950,0watchdog/37244-21sh22:04:183
329950,0watchdog/3635-21kthreadcore21:03:353
329950,0watchdog/325124-21port_sshpriv19:28:393
329950,0watchdog/312714-21df_inode20:53:313
329950,0watchdog/30-21swapper/321:18:393
29935050,0irq/33-eth0-tx-8508-21munin-run22:08:052
29935050,0irq/33-eth0-tx-28650-21ssh21:21:462
29935050,0irq/33-eth0-tx-18525-21ssh00:27:402
29925050,0irq/32-eth0-rx-6386-21runrttasks22:35:093
29925050,0irq/32-eth0-rx-3883-21ssh21:24:013
29925050,0irq/32-eth0-rx-27531-21seq22:38:163
29925050,0irq/32-eth0-rx-25320-21python00:08:433
29925050,0irq/32-eth0-rx-18278-21python20:08:403
29925050,0irq/32-eth0-rx-0-21swapper/322:19:253
29925050,0irq/32-eth0-rx-0-21swapper/322:13:383
29925050,0irq/32-eth0-rx-0-21swapper/322:08:343
29925050,0irq/32-eth0-rx-0-21swapper/321:43:333
149950,0migration/08673-21latency_hist22:08:050
149950,0migration/08527-21fschecks_time23:23:280
149950,0migration/08164-21kthreadcore22:43:350
149950,0migration/07077-21munin-node23:38:370
149950,0migration/031674-21fschecks_time21:43:290
149950,0migration/031602-21kthreadcore22:58:380
149950,0migration/030075-21kthreadcore20:18:340
149950,0migration/02914-21munin-node23:58:330
149950,0migration/028775-21switchtime19:08:420
149950,0migration/02857-21taskset00:18:160
149950,0migration/02669-21latency_hist00:18:060
149950,0migration/026114-21kthreadcore21:18:390
149950,0migration/022526-21kthreadcore22:13:380
149950,0migration/022025-21smart_sdb21:53:430
149950,0migration/015715-21kthreadcore00:03:380
149950,0migration/015685-21kthreadcore20:53:360
79239943,0cyclictest9192-21awk18:58:322
79239943,0cyclictest3599-21lldpd22:28:362
79239943,0cyclictest3599-21lldpd22:15:022
79239943,0cyclictest3536-21snmpd23:43:242
79239943,0cyclictest3536-21snmpd23:13:232
79239943,0cyclictest3536-21snmpd21:23:072
79239943,0cyclictest3536-21snmpd20:18:222
79239943,0cyclictest3536-21snmpd20:15:222
79239943,0cyclictest3536-21snmpd19:37:212
79239943,0cyclictest3536-21snmpd19:23:512
79239943,0cyclictest3536-21snmpd19:11:212
79239943,0cyclictest3536-21snmpd00:15:242
79239943,0cyclictest3262-21irqbalance23:48:302
79239943,0cyclictest3262-21irqbalance23:40:302
79239943,0cyclictest3262-21irqbalance19:49:012
79239943,0cyclictest31358-21awk20:43:352
79239943,0cyclictest31011-21awk19:58:292
79239943,0cyclictest30897-21awk19:13:322
79239943,0cyclictest30676-21awk22:58:352
79239943,0cyclictest25332-21awk21:58:312
79239943,0cyclictest23634-21awk21:38:312
79239943,0cyclictest2113-21awk19:38:302
79239943,0cyclictest11918-21awk00:03:352
79239943,0cyclictest0-21swapper/223:08:112
79239941,0cyclictest0-21swapper/221:08:242
79199944,0cyclictest0-21swapper/120:58:521
79199943,0cyclictest9605-21awk20:28:321
79199943,0cyclictest8840-21awk23:23:301
79199943,0cyclictest3599-21lldpd23:38:121
79199943,0cyclictest3599-21lldpd23:33:421
79199943,0cyclictest3599-21lldpd23:17:401
79199943,0cyclictest3599-21lldpd22:23:041
79199943,0cyclictest3599-21lldpd21:58:571
79199943,0cyclictest3599-21lldpd21:55:261
79199943,0cyclictest3599-21lldpd21:24:401
79199943,0cyclictest3599-21lldpd00:18:441
79199943,0cyclictest3536-21snmpd21:48:071
79199943,0cyclictest3536-21snmpd20:13:371
79199943,0cyclictest3262-21irqbalance19:44:211
79199943,0cyclictest3262-21irqbalance19:39:011
79199943,0cyclictest3262-21irqbalance19:14:011
79199943,0cyclictest3262-21irqbalance19:03:111
79199943,0cyclictest3262-21irqbalance18:58:311
79199943,0cyclictest2866-21awk22:23:291
79199943,0cyclictest27612-21awk20:18:321
79199943,0cyclictest2458-21grep20:23:321
79199943,0cyclictest17536-21awk23:28:351
79199943,0cyclictest16826-21awk20:33:321
79199943,0cyclictest15116-21awk21:33:301
79199943,0cyclictest13597-21awk22:48:351
79199943,0cyclictest13348-21awk20:53:331
79199943,0cyclictest13104-21awk20:08:321
79199943,0cyclictest12911-21awk19:23:351
79119944,0cyclictest25091-21kworker/0:021:30:330
79119944,0cyclictest0-21swapper/022:40:220
79119944,0cyclictest0-21swapper/021:17:260
339940,0migration/38634-21kthreadcore19:18:343
339940,0migration/327749-21kthreadcore19:53:373
339940,0migration/312249-21df_inode19:23:323
339940,0migration/31075-21kthreadcore20:43:373
329940,0watchdog/36386-21runrttasks19:38:073
329940,0watchdog/326032-40yum-updatesd-he20:16:273
329940,0watchdog/30-21swapper/323:03:263
329940,0watchdog/30-21swapper/322:23:223
329940,0watchdog/30-21swapper/320:58:393
329940,0watchdog/30-21swapper/319:48:393
329940,0watchdog/30-21swapper/300:03:143
29935040,0irq/33-eth0-tx-7735-21munin-node23:23:362
29935040,0irq/33-eth0-tx-31414-21ssh23:18:062
29935040,0irq/33-eth0-tx-26357-21ssh23:53:182
29935040,0irq/33-eth0-tx-2224-21ssh22:23:272
29935040,0irq/33-eth0-tx-0-21swapper/223:58:302
29935040,0irq/33-eth0-tx-0-21swapper/223:33:352
29935040,0irq/33-eth0-tx-0-21swapper/223:28:312
29935040,0irq/33-eth0-tx-0-21swapper/222:53:212
29935040,0irq/33-eth0-tx-0-21swapper/222:48:082
29935040,0irq/33-eth0-tx-0-21swapper/222:43:342
29935040,0irq/33-eth0-tx-0-21swapper/221:55:502
29935040,0irq/33-eth0-tx-0-21swapper/221:30:302
29935040,0irq/33-eth0-tx-0-21swapper/221:13:112
29935040,0irq/33-eth0-tx-0-21swapper/200:18:102
29925040,0irq/32-eth0-rx-4164-40yum-updatesd-he19:16:263
29925040,0irq/32-eth0-rx-17419-21cut21:53:363
29925040,0irq/32-eth0-rx-0-21swapper/323:28:323
29925040,0irq/32-eth0-rx-0-21swapper/323:23:523
29925040,0irq/32-eth0-rx-0-21swapper/322:53:203
29925040,0irq/32-eth0-rx-0-21swapper/322:43:263
29925040,0irq/32-eth0-rx-0-21swapper/321:35:513
29925040,0irq/32-eth0-rx-0-21swapper/321:28:273
29925040,0irq/32-eth0-rx-0-21swapper/300:13:213
259940,0migration/28206-21runrttasks20:27:112
259940,0migration/28175-21idleruntime21:48:322
259940,0migration/27667-21kthreadcore20:48:382
259940,0migration/229957-21kthreadcore21:03:342
259940,0migration/229886-21runrttasks20:42:442
259940,0migration/219930-21latency21:33:342
259940,0migration/219586-21aten_r1power_cu20:58:292
259940,0migration/216214-21gltestperf19:03:322
259940,0migration/215648-21kthreadcore20:53:362
259940,0migration/215463-21kthreadcore20:08:342
259940,0migration/21384-21kthreadcore21:43:332
259940,0migration/210075-21munin-node20:28:342
249940,0watchdog/26316-21irqrtprio20:03:312
249940,0watchdog/26017-21cpu23:03:262
249940,0watchdog/2387-21kthreadcore22:38:382
249940,0watchdog/213827-21cut19:43:352
249940,0watchdog/20-21swapper/220:33:352
179940,0migration/19957-21sh00:01:121
179940,0migration/17465-21kthreadcore21:08:311
179940,0migration/14488-21taskset20:44:191
179940,0migration/13826-21kthreadcore22:03:361
179940,0migration/13746-21taskset22:43:101
179940,0migration/130527-21fschecks_count19:58:271
179940,0migration/125900-21kthreadcore21:38:331
179940,0migration/125035-21kthreadcore19:53:361
179940,0migration/124824-21kthreadcore19:08:351
179940,0migration/121800-21fschecks_time21:18:341
179940,0migration/116882-21switchtime22:28:441
179940,0migration/113481-21hwlatdetect21:13:301
169940,0watchdog/130491-21kthreadcore19:33:391
169940,0watchdog/13035-21ssh22:41:541
169940,0watchdog/121142-21ssh22:53:261
169940,0watchdog/118347-21df23:48:341
169940,0watchdog/113371-21ssh21:32:101
169940,0watchdog/10-21swapper/123:21:301
169940,0watchdog/10-21swapper/122:08:341
169940,0watchdog/10-21swapper/120:03:311
149940,0migration/026546-21fschecks_time22:18:320
8315030,0irq/26-0000:00:0-21swapper/219:28:322
8315030,0irq/26-0000:00:0-21swapper/219:18:322
79199933,0cyclictest3599-21lldpd22:14:001
79199932,0cyclictest5087-21cut20:48:301
79199932,0cyclictest3599-21lldpd23:43:151
79199932,0cyclictest31777-21grep21:43:291
79199932,0cyclictest20212-21awk22:33:311
79199930,0cyclictest24788-21cut19:28:361
79119932,0cyclictest13573-21awk22:48:350
79119930,0cyclictest302-21python23:53:390
329930,0watchdog/30-21swapper/319:08:353
29925030,0irq/32-eth0-rx-0-21swapper/320:48:303
29925030,0irq/32-eth0-rx-0-21swapper/319:01:093
179930,0migration/114126-21kthreadcore00:23:381
169930,0watchdog/16019-21cpu23:03:261
169930,0watchdog/11172-21smartctl00:13:461
169930,0watchdog/10-21swapper/123:53:261
169930,0watchdog/10-21swapper/123:08:221
169930,0watchdog/10-21swapper/122:58:301
169930,0watchdog/10-21swapper/121:03:351
169930,0watchdog/10-21swapper/120:38:431
169930,0watchdog/10-21swapper/119:48:391
169930,0watchdog/10-21swapper/100:08:101
329920,0watchdog/30-21swapper/319:58:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional