You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 03:56
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Sat Jul 27, 2024 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
271800,0ktimersoftd/220703-21perf18:48:532
2748223521,0sleep00-21swapper/018:44:550
2754323425,0sleep10-21swapper/118:45:401
2751622824,0sleep30-21swapper/318:45:223
329970,0watchdog/30-21swapper/323:48:233
329970,0watchdog/30-21swapper/321:39:233
29925070,0irq/32-eth0-rx-0-21swapper/323:32:363
29925070,0irq/32-eth0-rx-0-21swapper/323:17:383
29925070,0irq/32-eth0-rx-0-21swapper/323:10:373
29925070,0irq/32-eth0-rx-0-21swapper/321:32:473
29925070,0irq/32-eth0-rx-0-21swapper/320:29:243
29925070,0irq/32-eth0-rx-0-21swapper/300:08:503
279079976,0cyclictest6921-21diskmemload21:54:221
8315060,0irq/26-0000:00:0-21swapper/322:19:073
8315060,0irq/26-0000:00:0-21swapper/321:46:103
8315060,0irq/26-0000:00:0-21swapper/321:19:023
339960,0migration/39007-21sh23:55:063
339960,0migration/36670-21taskset20:03:483
339960,0migration/322960-21sh21:35:073
329960,0watchdog/30-21swapper/323:35:513
329960,0watchdog/30-21swapper/323:00:033
329960,0watchdog/30-21swapper/322:41:273
329960,0watchdog/30-21swapper/322:26:193
329960,0watchdog/30-21swapper/322:18:353
329960,0watchdog/30-21swapper/321:25:193
329960,0watchdog/30-21swapper/321:14:193
329960,0watchdog/30-21swapper/318:59:313
329960,0watchdog/30-21swapper/318:55:153
329960,0watchdog/30-21swapper/318:53:313
29925060,0irq/32-eth0-rx-9504-21cpuspeed_turbos22:44:143
29925060,0irq/32-eth0-rx-30339-21python22:34:323
29925060,0irq/32-eth0-rx-14782-21kthreadcore23:04:203
29925060,0irq/32-eth0-rx-0-21swapper/323:39:153
29925060,0irq/32-eth0-rx-0-21swapper/323:24:213
29925060,0irq/32-eth0-rx-0-21swapper/323:22:233
29925060,0irq/32-eth0-rx-0-21swapper/322:56:413
29925060,0irq/32-eth0-rx-0-21swapper/322:52:543
29925060,0irq/32-eth0-rx-0-21swapper/322:29:113
29925060,0irq/32-eth0-rx-0-21swapper/322:03:413
29925060,0irq/32-eth0-rx-0-21swapper/321:54:243
29925060,0irq/32-eth0-rx-0-21swapper/321:51:553
29925060,0irq/32-eth0-rx-0-21swapper/321:12:503
29925060,0irq/32-eth0-rx-0-21swapper/321:08:173
29925060,0irq/32-eth0-rx-0-21swapper/321:02:013
29925060,0irq/32-eth0-rx-0-21swapper/320:49:443
29925060,0irq/32-eth0-rx-0-21swapper/300:14:273
29925060,0irq/32-eth0-rx-0-21swapper/300:02:473
279089960,0cyclictest0-21swapper/223:02:402
279079966,0cyclictest191ktimersoftd/122:46:441
279079964,0cyclictest11018-21iostat21:09:291
279079963,0cyclictest6040-21cut18:54:201
279079963,0cyclictest27715-21if_err_eth122:54:171
279079963,0cyclictest22536-21interrupts20:14:211
279079963,0cyclictest18372-21grep23:24:201
279069960,0cyclictest0-21swapper/020:50:090
149960,0migration/027614-21kthreadcore20:39:240
149960,0migration/026720-21kthreadcore21:39:220
149960,0migration/013773-21sh22:25:570
8315050,0irq/26-0000:00:0-21swapper/320:24:223
339950,0migration/331393-21kthreadcore23:49:253
339950,0migration/328751-21kthreadcore19:54:233
339950,0migration/317224-21kthreadcore19:24:203
339950,0migration/316916-21munin-node20:54:203
339950,0migration/312788-21mdstat20:04:233
329950,0watchdog/30-21swapper/320:45:393
329950,0watchdog/30-21swapper/320:45:393
329950,0watchdog/30-21swapper/320:34:073
329950,0watchdog/30-21swapper/320:20:353
329950,0watchdog/30-21swapper/320:14:513
329950,0watchdog/30-21swapper/319:49:313
329950,0watchdog/30-21swapper/319:42:273
329950,0watchdog/30-21swapper/319:29:033
329950,0watchdog/30-21swapper/319:19:153
329950,0watchdog/30-21swapper/319:14:153
29935050,0irq/33-eth0-tx-29491-21ssh21:21:132
29935050,0irq/33-eth0-tx-16156-21grep22:09:212
29935050,0irq/33-eth0-tx-14148-21munin-node20:09:200
29935050,0irq/33-eth0-tx-0-21swapper/221:17:552
29935050,0irq/33-eth0-tx-0-21swapper/220:56:172
29935050,0irq/33-eth0-tx-0-21swapper/020:14:330
29935050,0irq/33-eth0-tx-0-21swapper/018:49:160
29925050,0irq/32-eth0-rx-0-21swapper/322:09:353
29925050,0irq/32-eth0-rx-0-21swapper/322:07:253
29925050,0irq/32-eth0-rx-0-21swapper/319:34:143
29925050,0irq/32-eth0-rx-0-21swapper/300:09:203
279089955,0cyclictest0-21swapper/222:16:002
279089955,0cyclictest0-21swapper/222:05:072
279089950,0cyclictest0-21swapper/223:53:452
279089950,0cyclictest0-21swapper/223:24:162
279089950,0cyclictest0-21swapper/220:52:072
279089950,0cyclictest0-21swapper/219:40:042
279089950,0cyclictest0-21swapper/200:14:522
279079953,0cyclictest921-21hostname21:04:131
279079953,0cyclictest8395-21rm23:39:121
279079953,0cyclictest8136-21gltestperf20:49:161
279079953,0cyclictest7735-21df_inode21:29:171
279079953,0cyclictest6937-21chrt20:04:121
279079953,0cyclictest6707-21sh19:19:141
279079953,0cyclictest6510-21aten2.4-expect22:24:121
279079953,0cyclictest5723-21munin-node23:19:121
279079953,0cyclictest4276-21ssh00:14:111
279079953,0cyclictest3891-21ssh22:04:161
279079953,0cyclictest3568-21diskstats19:39:161
279079953,0cyclictest3368-21aten_r1power_cu20:24:121
279079953,0cyclictest32713-21aten2.4-expect22:39:131
279079953,0cyclictest32669-21fschecks_time19:59:161
279079953,0cyclictest32309-21aten_r1power_en20:44:131
279079953,0cyclictest32309-21aten_r1power_en20:44:131
279079953,0cyclictest31894-21chrt23:34:051
279079953,0cyclictest31843-21sh19:14:141
279079953,0cyclictest31275-21munin-node21:24:141
279079953,0cyclictest30559-21grep22:19:151
279079953,0cyclictest3054-21munin-node22:59:151
279079953,0cyclictest29830-21grep23:14:131
279079953,0cyclictest29367-21perl20:19:181
279079953,0cyclictest28402-21cut19:34:141
279079953,0cyclictest28238-21aten2.4-expect18:49:141
279079953,0cyclictest27896-21cat00:09:021
279079953,0cyclictest27810-21sed21:59:151
279079953,0cyclictest26514-21grep23:49:181
279079953,0cyclictest25773-21perl20:39:181
279079953,0cyclictest25674-21perl19:54:171
279079953,0cyclictest25082-21grep21:39:141
279079953,0cyclictest24872-21expr19:09:141
279079953,0cyclictest24633-21fschecks_time22:34:191
279079953,0cyclictest24484-21cut20:59:111
279079953,0cyclictest23893-21perl23:29:141
279079953,0cyclictest22046-21ssh21:19:051
279079953,0cyclictest21499-21df_inode19:29:141
279079953,0cyclictest21242-21cat22:14:021
279079953,0cyclictest2016-21awk23:54:121
279079953,0cyclictest19906-21cut00:04:161
279079953,0cyclictest18415-21perl19:49:161
279079953,0cyclictest17927-21munin-node20:34:151
279079953,0cyclictest17816-21date22:49:021
279079953,0cyclictest17778-21fschecks_time23:44:151
279079953,0cyclictest17737-21df19:04:151
279079953,0cyclictest17644-21kthreadcore20:09:221
279079953,0cyclictest16468-21diskstats21:34:141
279079953,0cyclictest15992-21perl22:29:131
279079953,0cyclictest15783-21rm20:54:111
279079953,0cyclictest14481-21munin-node21:14:171
279079953,0cyclictest14005-21stty19:24:131
279079953,0cyclictest13468-21perl22:09:151
279079953,0cyclictest1244-21ssh21:44:171
279079953,0cyclictest11709-21chrt23:04:021
279079953,0cyclictest11492-21grep23:59:151
279079953,0cyclictest11347-21expr19:44:171
279079953,0cyclictest11044-21diskstats20:29:151
279079953,0cyclictest10984-21perl18:59:191
279079953,0cyclictest10341-21df_inode21:49:131
279079952,0cyclictest20422-21latency_hist23:09:021
279069955,0cyclictest0-21swapper/022:33:440
279069955,0cyclictest0-21swapper/022:06:370
279069950,0cyclictest0-21swapper/023:10:000
279069950,0cyclictest0-21swapper/021:59:410
279069950,0cyclictest0-21swapper/021:13:480
259950,0migration/26905-21timerandwakeup22:39:312
259950,0migration/26412-21kthreadcore18:54:212
259950,0migration/231038-21ntp_offset20:39:272
259950,0migration/220944-21taskset20:12:162
259950,0migration/218335-21kthreadcore22:29:182
249950,0watchdog/25354-21ssh21:24:472
149950,0migration/09838-21kthreadcore23:19:220
149950,0migration/09544-21kthreadcore00:14:210
149950,0migration/05581-21kthreadcore23:54:210
149950,0migration/05073-21sh23:18:160
149950,0migration/04174-21kthreadcore21:24:210
149950,0migration/03513-21kthreadcore22:19:220
149950,0migration/027962-21sh00:09:070
149950,0migration/027875-21taskset19:32:030
149950,0migration/025625-21idleruntime19:09:170
149950,0migration/024212-21sendmail_mailqu22:49:260
149950,0migration/024130-21kthreadcore21:19:200
149950,0migration/023821-21taskset21:37:220
149950,0migration/023626-21latency20:34:230
149950,0migration/023477-21cpuspeed_turbos23:29:130
149950,0migration/02068-21kthreadcore23:34:220
149950,0migration/019838-21if_eth1.221:54:170
149950,0migration/019651-21taskset22:10:200
149950,0migration/018947-21irqcore19:49:210
149950,0migration/016526-21proc_pri19:44:270
149950,0migration/01645-21irqcore22:39:200
149950,0migration/016226-21df_abs20:54:160
149950,0migration/014102-21kthreadcore22:44:210
149950,0migration/012476-21kthreadcore21:29:260
149950,0migration/011204-21hwlatdetect18:59:210
149950,0migration/01090-21cpuspeed_turbos21:04:150
149950,0migration/010172-21kthreadcore20:04:210
339940,0migration/315821-21kthreadcore20:09:223
329940,0watchdog/30-21swapper/320:43:153
329940,0watchdog/30-21swapper/319:44:233
329940,0watchdog/30-21swapper/319:09:113
29935040,0irq/33-eth0-tx-9139-21ssh22:44:052
29935040,0irq/33-eth0-tx-3761-21cat22:19:232
29935040,0irq/33-eth0-tx-29916-21ssh23:29:322
29935040,0irq/33-eth0-tx-2417-21ssh00:09:412
29935040,0irq/33-eth0-tx-12807-21hddtemp_smartct23:04:162
29935040,0irq/33-eth0-tx-0-21swapper/223:59:092
29935040,0irq/33-eth0-tx-0-21swapper/223:54:272
29935040,0irq/33-eth0-tx-0-21swapper/223:34:522
29935040,0irq/33-eth0-tx-0-21swapper/223:09:252
29935040,0irq/33-eth0-tx-0-21swapper/222:35:032
29935040,0irq/33-eth0-tx-0-21swapper/222:24:062
29935040,0irq/33-eth0-tx-0-21swapper/221:54:042
29935040,0irq/33-eth0-tx-0-21swapper/221:39:402
29935040,0irq/33-eth0-tx-0-21swapper/221:34:062
29935040,0irq/33-eth0-tx-0-21swapper/221:30:052
29935040,0irq/33-eth0-tx-0-21swapper/019:34:320
29935040,0irq/33-eth0-tx-0-21swapper/019:19:220
29925040,0irq/32-eth0-rx-0-21swapper/319:04:153
279089943,0cyclictest8029-21awk20:04:192
279089943,0cyclictest3599-21lldpd23:39:162
279089943,0cyclictest3599-21lldpd20:34:142
279089943,0cyclictest3599-21lldpd18:51:122
279089943,0cyclictest3262-21irqbalance21:49:072
279089943,0cyclictest3262-21irqbalance21:44:272
279089943,0cyclictest3262-21irqbalance21:04:072
279089943,0cyclictest14991-21awk19:24:182
279089943,0cyclictest11455-21awk19:44:192
279089941,0cyclictest0-21swapper/221:09:182
279069944,0cyclictest0-21swapper/021:18:130
279069943,0cyclictest3599-21lldpd23:07:160
279069943,0cyclictest3599-21lldpd00:01:170
279069943,0cyclictest29787-21sadc22:19:020
279069943,0cyclictest29629-21awk20:19:200
279069943,0cyclictest27310-21kworker/0:200:07:230
279069943,0cyclictest23871-21snmpd23:24:200
279069943,0cyclictest2314-21awk21:44:250
279069943,0cyclictest11690-21awk20:29:180
279069940,0cyclictest5337-21sed19:14:250
259940,0migration/25771-21aten_r1power_po23:19:132
259940,0migration/230078-21fschecks_time23:14:142
259940,0migration/228973-21fschecks_time19:34:172
259940,0migration/228194-21kthreadcore19:54:222
259940,0migration/227382-21aten_r1power_cu21:59:122
259940,0migration/226918-21cpuspeed_turbos22:54:132
259940,0migration/226292-21kthreadcore19:29:232
259940,0migration/221356-21kthreadcore23:44:202
259940,0migration/219854-21fschecks_count00:04:162
259940,0migration/219685-21kthreadcore22:49:212
249940,0watchdog/225402-21perl19:09:152
249940,0watchdog/224195-21crond20:59:032
249940,0watchdog/223783-21rt-features19:49:272
249940,0watchdog/22277-21kthreadcore19:14:232
249940,0watchdog/216216-21ntp_states18:59:272
249940,0watchdog/20-21swapper/220:19:232
149940,0migration/04731-21kthreadcore20:44:210
149940,0migration/04731-21kthreadcore20:44:210
149940,0migration/04489-21irqcore19:39:210
149940,0migration/03504-21forks22:59:170
149940,0migration/025457-21sh20:59:140
279089933,0cyclictest3599-21lldpd19:04:122
279089933,0cyclictest11729-21awk20:29:192
279089932,0cyclictest594-21awk19:59:182
279089932,0cyclictest22292-21grep20:14:162
279089930,0cyclictest611-21perl20:44:162
279089930,0cyclictest611-21perl20:44:162
279089930,0cyclictest3810-21diskstats20:24:142
279069933,0cyclictest27310-21kworker/0:223:49:150
279069933,0cyclictest27310-21kworker/0:223:44:150
279069933,0cyclictest27310-21kworker/0:223:39:030
279069933,0cyclictest27310-21kworker/0:222:34:130
279069933,0cyclictest27310-21kworker/0:221:49:130
279069933,0cyclictest27310-21kworker/0:220:24:090
279069933,0cyclictest19178-21kworker/0:119:04:220
279069932,0cyclictest4819-21grep18:54:200
279069932,0cyclictest25900-21awk19:54:200
279069932,0cyclictest23871-21snmpd22:54:050
279069932,0cyclictest14945-21awk19:24:170
279069930,0cyclictest0-21swapper/019:59:140
249930,0watchdog/29589-21kthreadcore19:19:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional