You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-03-02 - 17:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Sat Mar 02, 2024 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3003923829,0sleep20-21swapper/206:57:482
2980223526,0sleep10-21swapper/106:55:021
2976623329,0sleep30-21swapper/306:54:363
2970322925,0sleep00-21swapper/006:53:530
339960,0migration/331671-21sh11:51:593
339960,0migration/313117-21fschecks_time07:08:223
329960,0watchdog/315116-21sh10:40:133
301259960,0cyclictest13660-21kthreadcore10:58:331
29925060,0irq/32-eth0-rx-28762-21cpuspeed_turbos09:33:223
29925060,0irq/32-eth0-rx-21324-21iostat09:28:283
29925060,0irq/32-eth0-rx-17915-21rt-features09:03:323
29925060,0irq/32-eth0-rx-11579-21find09:58:343
29925060,0irq/32-eth0-rx-0-21swapper/312:18:243
149960,0migration/01880-21sh11:12:240
8315050,0irq/26-0000:00:32028-21munin-run11:52:562
8315050,0irq/26-0000:00:3011-21sed08:12:572
4150,0ktimersoftd/00-21swapper/009:08:400
4150,0ktimersoftd/00-21swapper/007:58:290
339950,0migration/39807-21munin-node08:58:463
339950,0migration/38054-21kthreadcore08:13:263
339950,0migration/37568-21fschecks_time08:38:233
339950,0migration/36409-21ksmtuned08:35:323
339950,0migration/34846-21taskset10:14:093
339950,0migration/34803-21taskset07:00:503
339950,0migration/328698-21aten2.4-expect12:13:203
339950,0migration/328101-21kthreadcore08:28:283
339950,0migration/326387-21kernelversion10:48:273
339950,0migration/326058-21timerwakeupswit11:23:393
339950,0migration/324532-21taskset12:04:453
339950,0migration/324213-21taskset08:23:563
339950,0migration/322041-21aten_r1power_fr10:08:193
339950,0migration/320688-21taskset08:43:533
339950,0migration/318759-21fschecks_time11:03:223
339950,0migration/316190-21aten_r1power_cu07:33:193
339950,0migration/312880-21munin-node10:18:483
339950,0migration/311425-21idleruntime08:18:243
329950,0watchdog/35601-21python10:33:333
329950,0watchdog/32434-21diskstats08:08:223
329950,0watchdog/32376-21sh11:13:133
329950,0watchdog/321167-21ssh10:06:213
329950,0watchdog/320760-21ssh11:42:333
329950,0watchdog/319015-21grep10:43:253
329950,0watchdog/314366-21irqstats09:43:253
329950,0watchdog/30-21swapper/311:58:253
329950,0watchdog/30-21swapper/309:53:213
329950,0watchdog/30-21swapper/309:08:303
29935050,0irq/33-eth0-tx-7940-21kthreadcore12:18:232
29935050,0irq/33-eth0-tx-5239-21munin-node07:03:310
29935050,0irq/33-eth0-tx-3536-21snmpd09:33:292
29935050,0irq/33-eth0-tx-3393-21munin-node08:58:460
29935050,0irq/33-eth0-tx-32350-21munin-node11:53:222
29935050,0irq/33-eth0-tx-23417-21munin-node07:38:210
29935050,0irq/33-eth0-tx-16173-21munin-node07:33:310
29935050,0irq/33-eth0-tx-0-21swapper/212:13:032
29935050,0irq/33-eth0-tx-0-21swapper/211:07:592
29935050,0irq/33-eth0-tx-0-21swapper/210:14:102
29935050,0irq/33-eth0-tx-0-21swapper/008:23:320
29925050,0irq/32-eth0-rx-8076-21ssh10:54:483
29925050,0irq/32-eth0-rx-6386-21runrttasks10:24:273
29925050,0irq/32-eth0-rx-31441-21kthreadcore11:28:253
29925050,0irq/32-eth0-rx-2298-21kthreadcore11:53:253
29925050,0irq/32-eth0-rx-19446-21ssh09:26:553
29925050,0irq/32-eth0-rx-0-21swapper/312:25:163
29925050,0irq/32-eth0-rx-0-21swapper/311:12:333
29925050,0irq/32-eth0-rx-0-21swapper/309:39:233
29925050,0irq/32-eth0-rx-0-21swapper/309:18:263
29925050,0irq/32-eth0-rx-0-21swapper/308:53:303
259950,0migration/28064-21switchtime07:23:352
259950,0migration/27861-21aten_r1power_en10:38:182
259950,0migration/25520-21kthreadcore11:13:262
259950,0migration/229059-21kthreadcore08:28:282
259950,0migration/214514-21kthreadcore11:18:232
249950,0watchdog/229936-21cut09:53:212
249950,0watchdog/222792-21grep09:48:292
249950,0watchdog/217699-21kthreadcore08:43:262
249950,0watchdog/217662-21aten2.4-expect08:23:222
179950,0migration/114822-21sh12:23:241
169950,0watchdog/120683-21kthreadcore08:23:301
149950,0migration/09951-21kthreadcore10:18:330
149950,0migration/09277-21latency_hist10:57:570
149950,0migration/08450-21mdstat11:13:270
149950,0migration/08350-21kthreadcore09:18:270
149950,0migration/08159-21kthreadcore09:38:250
149950,0migration/06199-21if_eth012:18:210
149950,0migration/032178-21kthreadcore11:28:250
149950,0migration/031586-21kthreadcore09:53:260
149950,0migration/030881-21kthreadcore07:18:330
149950,0migration/029986-21kthreadcore08:03:230
149950,0migration/027796-21kthreadcore08:28:280
149950,0migration/027609-21timerwakeupswit09:48:380
149950,0migration/02676-21taskset09:14:520
149950,0migration/023959-21kthreadcore07:13:260
149950,0migration/023712-21sh10:45:380
149950,0migration/01999-21cpuspeed_turbos07:23:210
149950,0migration/019960-21smart_sdb10:03:350
149950,0migration/019581-21taskset07:55:550
149950,0migration/019566-21sh11:23:200
149950,0migration/018847-21sh09:05:210
149950,0migration/017878-21taskset11:03:120
149950,0migration/016577-21sh12:01:480
149950,0migration/013265-21fschecks_time07:08:220
149950,0migration/01092-21if_eth111:53:230
8315040,0irq/26-0000:00:0-21swapper/208:58:282
8315040,0irq/26-0000:00:0-21swapper/208:17:582
4140,0ktimersoftd/00-21swapper/008:12:400
339940,0migration/332577-21kthreadcore09:13:283
339940,0migration/331517-21kthreadcore07:18:333
339940,0migration/323113-21kthreadcore07:13:263
329940,0watchdog/30-21swapper/311:42:573
329940,0watchdog/30-21swapper/311:33:253
329940,0watchdog/30-21swapper/308:48:223
329940,0watchdog/30-21swapper/307:53:303
329940,0watchdog/30-21swapper/307:23:223
301269943,0cyclictest8023-21awk08:38:252
301269943,0cyclictest3599-21lldpd11:38:292
301269943,0cyclictest3599-21lldpd11:27:592
301269943,0cyclictest31921-21kthreadcore06:58:232
301269943,0cyclictest21956-21munin-node10:08:212
301269943,0cyclictest21091-21awk07:58:232
301269943,0cyclictest17212-21awk07:33:242
301269943,0cyclictest10034-21awk07:28:222
301269940,0cyclictest0-21swapper/209:59:072
301269940,0cyclictest0-21swapper/208:16:232
301259943,0cyclictest8911-21awk10:38:251
301259943,0cyclictest6253-21awk12:18:211
301259943,0cyclictest5013-21awk09:18:241
301259943,0cyclictest3599-21lldpd12:12:291
301259943,0cyclictest3599-21lldpd11:53:291
301259943,0cyclictest3599-21lldpd11:50:591
301259943,0cyclictest3599-21lldpd11:22:591
301259943,0cyclictest3599-21lldpd09:58:271
301259943,0cyclictest3599-21lldpd09:49:041
301259943,0cyclictest3599-21lldpd09:42:591
301259943,0cyclictest3599-21lldpd08:51:191
301259943,0cyclictest3599-21lldpd08:33:151
301259943,0cyclictest3262-21irqbalance07:29:221
301259943,0cyclictest3262-21irqbalance07:26:321
301259943,0cyclictest3262-21irqbalance06:59:121
301259943,0cyclictest30579-21awk09:53:241
301259943,0cyclictest29741-21awk12:13:251
301259943,0cyclictest29221-21awk09:13:251
301259943,0cyclictest22382-21awk11:43:221
301259943,0cyclictest20959-21awk09:08:271
301259943,0cyclictest17160-21pmu-power11:18:271
301259943,0cyclictest15867-21awk10:23:301
301259943,0cyclictest13818-21awk11:38:221
301249943,0cyclictest663-21awk08:33:300
301249943,0cyclictest3599-21lldpd12:05:290
301249943,0cyclictest3536-21snmpd07:00:420
301249943,0cyclictest3262-21irqbalance11:35:310
301249943,0cyclictest29546-21awk09:33:250
301249943,0cyclictest29446-21kworker/0:210:52:570
301249943,0cyclictest29446-21kworker/0:210:33:160
301249943,0cyclictest25935-21awk10:48:250
301249943,0cyclictest23166-21basename10:28:210
301249943,0cyclictest20316-21kworker/0:209:58:000
301249943,0cyclictest10483-21kworker/0:011:42:030
301249940,0cyclictest10414-21munin-node08:18:220
301249940,0cyclictest0-21swapper/012:08:430
301249940,0cyclictest0-21swapper/009:28:510
29935040,0irq/33-eth0-tx-2565-21ssh09:14:252
29935040,0irq/33-eth0-tx-18954-21kthreadcore12:03:242
29935040,0irq/33-eth0-tx-11606-21munin-node09:03:222
29935040,0irq/33-eth0-tx-0-21swapper/212:23:052
29935040,0irq/33-eth0-tx-0-21swapper/212:08:212
29935040,0irq/33-eth0-tx-0-21swapper/211:43:182
29935040,0irq/33-eth0-tx-0-21swapper/211:33:262
29935040,0irq/33-eth0-tx-0-21swapper/211:23:242
29935040,0irq/33-eth0-tx-0-21swapper/211:03:052
29935040,0irq/33-eth0-tx-0-21swapper/210:34:112
29935040,0irq/33-eth0-tx-0-21swapper/210:28:152
29935040,0irq/33-eth0-tx-0-21swapper/210:22:582
29935040,0irq/33-eth0-tx-0-21swapper/209:28:012
29935040,0irq/33-eth0-tx-0-21swapper/209:17:572
29935040,0irq/33-eth0-tx-0-21swapper/008:13:270
29925040,0irq/32-eth0-rx-0-21swapper/312:08:183
29925040,0irq/32-eth0-rx-0-21swapper/311:18:253
29925040,0irq/32-eth0-rx-0-21swapper/310:59:513
29925040,0irq/32-eth0-rx-0-21swapper/310:28:003
29925040,0irq/32-eth0-rx-0-21swapper/309:48:303
29925040,0irq/32-eth0-rx-0-21swapper/307:48:193
259940,0migration/2868-21aten_r1power_en10:53:202
259940,0migration/28436-21kthreadcore07:03:282
259940,0migration/26475-21kthreadcore09:38:252
259940,0migration/231767-21kthreadcore07:18:332
259940,0migration/231340-21fschecks_time07:43:232
259940,0migration/228406-21latency_hist08:52:572
259940,0migration/224791-21taskset10:48:162
259940,0migration/222279-21hwlatdetect08:48:242
259940,0migration/216992-21kthreadcore09:43:272
259940,0migration/216222-21latency_hist10:42:572
259940,0migration/215047-21kthreadcore10:58:332
259940,0migration/213975-21kthreadcore09:23:282
259940,0migration/211975-21kthreadcore10:18:332
249940,0watchdog/223221-21tr07:13:262
249940,0watchdog/221485-21grep09:08:302
249940,0watchdog/20-21swapper/211:58:252
249940,0watchdog/20-21swapper/210:03:292
19140,0ktimersoftd/10-21swapper/108:44:331
179940,0migration/19585-21kthreadcore08:38:271
179940,0migration/18540-21runrttasks08:58:341
179940,0migration/18429-21latency11:13:271
179940,0migration/18221-21kthreadcore11:33:261
179940,0migration/18096-21kthreadcore10:18:321
179940,0migration/132576-21kthreadcore07:43:271
179940,0migration/130076-21kthreadcore07:18:331
179940,0migration/124712-21taskset10:47:591
179940,0migration/123704-21smart_sdb12:03:311
179940,0migration/123420-21munin-node10:28:221
179940,0migration/120450-21kthreadcore07:33:271
179940,0migration/119888-21cpuspeed_turbos07:13:201
179940,0migration/118838-21kthreadcore10:43:251
179940,0migration/11758-21munin-node11:28:281
179940,0migration/116449-21kthreadcore09:03:281
179940,0migration/114311-21kthreadcore09:23:281
179940,0migration/113777-21idleruntime07:53:261
179940,0migration/112952-21fschecks_count07:08:221
179940,0migration/111458-21sh09:39:291
169940,0watchdog/19750-21cut07:48:261
169940,0watchdog/19625-21grep11:58:211
169940,0watchdog/130290-21ssh10:13:131
169940,0watchdog/129762-21cut11:08:291
169940,0watchdog/129445-21ntp_states07:38:301
169940,0watchdog/122669-21kthreadcore09:28:291
169940,0watchdog/10-21swapper/111:03:251
169940,0watchdog/10-21swapper/108:53:301
149940,0migration/018234-21runrttasks11:20:150
149940,0migration/017671-21kthreadcore09:43:270
149940,0migration/015687-21kthreadcore08:43:250
149940,0migration/013883-21missed_timers10:38:300
149940,0migration/01068-21kthreadcore10:13:270
8315030,0irq/26-0000:00:0-21swapper/208:04:032
301269932,0cyclictest24457-21awk07:38:232
301269932,0cyclictest13871-21awk07:53:262
301269930,0cyclictest6425-21perl07:48:212
301269930,0cyclictest32730-21perl08:33:272
301269930,0cyclictest13325-21perl07:08:222
301259933,0cyclictest3599-21lldpd10:07:581
301259933,0cyclictest3599-21lldpd08:22:161
301259932,0cyclictest7314-21sensors10:53:331
301259932,0cyclictest3599-21lldpd08:13:141
301259932,0cyclictest3599-21lldpd08:03:121
301259932,0cyclictest3599-21lldpd07:02:581
301259932,0cyclictest3536-21snmpd09:33:121
301259932,0cyclictest21188-21awk07:58:241
301249933,0cyclictest7236-21kworker/0:110:23:120
301249933,0cyclictest20316-21kworker/0:210:07:580
301249933,0cyclictest20316-21kworker/0:209:23:000
301249933,0cyclictest20316-21kworker/0:208:53:000
301249933,0cyclictest20316-21kworker/0:208:48:060
301249933,0cyclictest1658-21kworker/0:008:37:580
301249933,0cyclictest1658-21kworker/0:007:48:180
301249933,0cyclictest1658-21kworker/0:007:43:220
301249933,0cyclictest1658-21kworker/0:007:28:220
301249933,0cyclictest10483-21kworker/0:012:23:190
301249933,0cyclictest10483-21kworker/0:012:12:570
301249933,0cyclictest10483-21kworker/0:011:47:570
301249933,0cyclictest10483-21kworker/0:011:42:590
29925030,0irq/32-eth0-rx-0-21swapper/308:03:183
29925030,0irq/32-eth0-rx-0-21swapper/308:00:153
29925030,0irq/32-eth0-rx-0-21swapper/307:43:283
29925030,0irq/32-eth0-rx-0-21swapper/307:40:173
29925030,0irq/32-eth0-rx-0-21swapper/307:03:193
169930,0watchdog/10-21swapper/110:33:331
169930,0watchdog/10-21swapper/110:03:291
169930,0watchdog/10-21swapper/108:28:221
169930,0watchdog/10-21swapper/108:08:221
329920,0watchdog/30-21swapper/307:28:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional