You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-15 - 18:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Mon Apr 15, 2024 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1367423930,0sleep30-21swapper/306:50:483
1383623726,0sleep00-21swapper/006:52:410
1383223625,0sleep10-21swapper/106:52:381
1382823526,0sleep20-21swapper/206:52:352
35160,0ktimersoftd/30-21swapper/311:58:453
35160,0ktimersoftd/30-21swapper/307:50:543
339960,0migration/39405-21kthreadcore09:20:323
339960,0migration/38186-21sh09:57:423
339960,0migration/35330-21sh09:36:533
339960,0migration/331810-21sh09:52:443
339960,0migration/320674-21sh10:41:433
339960,0migration/320383-21sh09:27:053
329960,0watchdog/313183-21ntp_states10:55:343
29935060,0irq/33-eth0-tx-0-21swapper/007:12:260
29925060,0irq/32-eth0-rx-29460-21perl10:30:263
29925060,0irq/32-eth0-rx-18333-21python11:35:353
29925060,0irq/32-eth0-rx-0-21swapper/311:27:493
141329965,0cyclictest0-21swapper/009:05:300
149960,0migration/08672-21kernelversion10:55:300
149960,0migration/04430-21kthreadcore09:55:300
149960,0migration/04041-21kthreadcore11:10:320
149960,0migration/03024-21kthreadcore12:05:300
149960,0migration/019573-21hddtemp_smartct08:50:270
339950,0migration/39355-21kthreadcore08:20:353
339950,0migration/39242-21missed_timers11:30:333
339950,0migration/38434-21kthreadcore07:55:383
339950,0migration/36182-21munin-node07:30:463
339950,0migration/36100-21timerwakeupswit12:05:393
339950,0migration/329689-21gltestperf07:05:303
339950,0migration/326461-21fschecks_time08:10:283
339950,0migration/325978-21kthreadcore11:05:293
339950,0migration/32486-21kthreadcore11:45:333
339950,0migration/324408-21kthreadcore09:10:313
339950,0migration/322147-21fschecks_count07:00:273
339950,0migration/322092-21munin-node09:30:233
339950,0migration/321106-21fschecks_count11:40:253
339950,0migration/320700-21fschecks_count10:25:263
339950,0migration/318036-21taskset08:03:583
339950,0migration/314151-21kthreadcore10:20:293
339950,0migration/313622-21taskset07:35:513
329950,0watchdog/39384-21sh11:15:223
329950,0watchdog/37817-21kthreadcore10:15:343
329950,0watchdog/328349-21python08:30:503
329950,0watchdog/324537-21timerandwakeup10:05:383
329950,0watchdog/310716-21unixbench_singl08:40:503
29935050,0irq/33-eth0-tx-6878-21munin-node09:40:292
29935050,0irq/33-eth0-tx-25174-21tune2fs09:50:272
29935050,0irq/33-eth0-tx-21758-21munin-node09:30:372
29935050,0irq/33-eth0-tx-19355-21date09:10:182
29935050,0irq/33-eth0-tx-0-21swapper/211:35:382
29935050,0irq/33-eth0-tx-0-21swapper/008:46:410
29935050,0irq/33-eth0-tx-0-21swapper/008:35:570
29935050,0irq/33-eth0-tx-0-21swapper/008:27:560
29935050,0irq/33-eth0-tx-0-21swapper/007:40:510
29935050,0irq/33-eth0-tx-0-21swapper/007:19:540
29925050,0irq/32-eth0-rx-8470-21cpu12:10:253
29925050,0irq/32-eth0-rx-467-21munin-node11:10:323
29925050,0irq/32-eth0-rx-2555-21ps09:15:553
29925050,0irq/32-eth0-rx-19365-21irqcore11:20:293
29925050,0irq/32-eth0-rx-17151-21rm09:45:293
29925050,0irq/32-eth0-rx-1232-21ssh08:58:533
29925050,0irq/32-eth0-rx-10744-21sed10:00:293
29925050,0irq/32-eth0-rx-0-21swapper/312:20:353
29925050,0irq/32-eth0-rx-0-21swapper/312:19:343
29925050,0irq/32-eth0-rx-0-21swapper/310:35:233
29925050,0irq/32-eth0-rx-0-21swapper/310:12:443
29925050,0irq/32-eth0-rx-0-21swapper/310:12:443
29925050,0irq/32-eth0-rx-0-21swapper/309:40:303
29925050,0irq/32-eth0-rx-0-21swapper/309:09:163
29925050,0irq/32-eth0-rx-0-21swapper/309:00:303
259950,0migration/226070-21kthreadcore07:45:422
249950,0watchdog/211502-21df_abs08:00:262
179950,0migration/130129-21kthreadcore11:25:331
179950,0migration/129578-21fschecks_time07:05:291
179950,0migration/129319-21munin-node09:15:291
179950,0migration/127278-21taskset10:26:541
179950,0migration/127175-21fschecks_count10:10:261
179950,0migration/127175-21fschecks_count10:10:261
179950,0migration/121679-21aten_r1power_po07:00:241
179950,0migration/120478-21missed_timers11:55:311
179950,0migration/114970-21kthreadcore09:25:311
169950,0watchdog/15862-21kthreadcore09:00:341
169950,0watchdog/120749-21processes07:40:501
141339955,0cyclictest0-21swapper/111:50:301
141339955,0cyclictest0-21swapper/110:52:381
141339950,0cyclictest0-21swapper/111:41:501
141339950,0cyclictest0-21swapper/111:16:171
141329950,0cyclictest0-21swapper/011:45:220
141329950,0cyclictest0-21swapper/011:27:240
141329950,0cyclictest0-21swapper/009:17:290
149950,0migration/09912-21kthreadcore09:40:300
149950,0migration/08631-21kthreadcore11:50:290
149950,0migration/0598-21kthreadcore07:05:340
149950,0migration/05862-21kthreadcore07:55:370
149950,0migration/04986-21kthreadcore10:15:330
149950,0migration/04615-21munin-node10:50:350
149950,0migration/032525-21kthreadcore10:30:320
149950,0migration/031688-21sh09:52:240
149950,0migration/030523-21kthreadcore12:20:310
149950,0migration/029659-21sh09:33:480
149950,0migration/028303-21kthreadcore12:00:310
149950,0migration/026278-21hwlatdetect07:25:350
149950,0migration/025974-21kthreadcore07:45:420
149950,0migration/023939-21kthreadcore08:05:340
149950,0migration/022089-21kthreadcore10:25:310
149950,0migration/020770-21kthreadcore09:45:300
149950,0migration/020375-21kthreadcore10:05:300
149950,0migration/019530-21munin-node10:40:330
149950,0migration/016607-21kthreadcore06:55:330
149950,0migration/013858-21kthreadcore12:10:320
149950,0migration/012021-21kthreadcore08:20:360
149950,0migration/010059-21sh09:04:000
8315040,0irq/26-0000:00:0-21swapper/207:40:552
35140,0ktimersoftd/32123-21jbd2/md0-812:00:173
339940,0migration/329514-21kthreadcore07:25:433
339940,0migration/328030-21taskset07:45:483
339940,0migration/323909-21kthreadcore08:50:353
339940,0migration/316376-21kthreadcore07:40:413
339940,0migration/314470-21kthreadcore07:15:353
339940,0migration/313045-21kthreadcore08:45:393
329940,0watchdog/30-21swapper/310:46:263
329940,0watchdog/30-21swapper/308:05:343
29935040,0irq/33-eth0-tx-4941-21munin-node10:35:252
29935040,0irq/33-eth0-tx-30783-21munin-node10:50:352
29935040,0irq/33-eth0-tx-30468-21munin-node09:35:292
29935040,0irq/33-eth0-tx-2844-21munin-node10:15:272
29935040,0irq/33-eth0-tx-28091-21munin-node09:15:282
29935040,0irq/33-eth0-tx-27721-21ssh11:41:342
29935040,0irq/33-eth0-tx-20121-21munin-node10:25:352
29935040,0irq/33-eth0-tx-10966-21ssh11:33:192
29935040,0irq/33-eth0-tx-0-21swapper/212:15:232
29935040,0irq/33-eth0-tx-0-21swapper/211:55:322
29935040,0irq/33-eth0-tx-0-21swapper/211:50:362
29935040,0irq/33-eth0-tx-0-21swapper/211:20:302
29935040,0irq/33-eth0-tx-0-21swapper/211:02:042
29935040,0irq/33-eth0-tx-0-21swapper/210:57:372
29935040,0irq/33-eth0-tx-0-21swapper/210:40:192
29935040,0irq/33-eth0-tx-0-21swapper/210:20:192
29935040,0irq/33-eth0-tx-0-21swapper/210:05:592
29935040,0irq/33-eth0-tx-0-21swapper/210:00:422
29935040,0irq/33-eth0-tx-0-21swapper/209:55:232
29935040,0irq/33-eth0-tx-0-21swapper/209:25:212
29935040,0irq/33-eth0-tx-0-21swapper/209:21:522
29935040,0irq/33-eth0-tx-0-21swapper/007:00:450
29925040,0irq/32-eth0-rx-31005-21munin-node10:50:253
29925040,0irq/32-eth0-rx-0-21swapper/311:51:563
29925040,0irq/32-eth0-rx-0-21swapper/311:00:233
29925040,0irq/32-eth0-rx-0-21swapper/308:25:263
29925040,0irq/32-eth0-rx-0-21swapper/308:15:293
259940,0migration/228662-21timerandwakeup10:45:392
259940,0migration/228199-21runrttasks07:01:122
259940,0migration/226506-21munin-node07:25:402
259940,0migration/226436-21kthreadcore08:30:402
259940,0migration/218643-21kthreadcore06:55:332
259940,0migration/217111-21munin-node07:15:412
259940,0migration/211003-21kernelversion11:15:322
259940,0migration/210088-21missed_timers08:40:392
249940,0watchdog/2718-21kthreadcore07:05:342
249940,0watchdog/26388-21tr07:10:382
249940,0watchdog/25863-21kthreadcore09:00:342
249940,0watchdog/229421-21fschecks_time10:30:262
249940,0watchdog/227375-21perl10:10:262
249940,0watchdog/227375-21perl10:10:262
249940,0watchdog/20-21swapper/211:05:302
179940,0migration/19787-21kthreadcore08:20:351
179940,0migration/19061-21kthreadcore07:55:381
179940,0migration/18368-21kthreadcore09:20:321
179940,0migration/17789-21kthreadcore07:10:391
179940,0migration/17660-21kthreadcore10:35:291
179940,0migration/15808-21kthreadcore08:15:371
179940,0migration/13426-21kthreadcore09:55:301
179940,0migration/132349-21kthreadcore10:30:321
179940,0migration/130857-21irqrtprio08:35:451
179940,0migration/129996-21runrttasks10:48:491
179940,0migration/128386-21kthreadcore08:10:371
179940,0migration/126972-21kthreadcore07:45:421
179940,0migration/125726-21kthreadcore08:30:401
179940,0migration/125243-21sh10:07:011
179940,0migration/117936-21taskset08:02:521
179940,0migration/117690-21kthreadcore08:25:471
169940,0watchdog/132073-21kthreadcore09:35:301
169940,0watchdog/127779-21kthreadcore11:05:301
169940,0watchdog/118241-21vmstat10:20:381
169940,0watchdog/117087-21ssh10:03:141
169940,0watchdog/113357-21timerwakeupswit09:40:381
169940,0watchdog/110717-21unixbench_singl08:40:501
169940,0watchdog/10-21swapper/111:35:301
169940,0watchdog/10-21swapper/110:15:341
169940,0watchdog/10-21swapper/107:50:261
169940,0watchdog/10-21swapper/106:55:341
141349943,0cyclictest9364-21awk12:10:292
141349943,0cyclictest8793-21awk08:20:322
141349943,0cyclictest4979-21awk07:55:312
141349943,0cyclictest3599-21lldpd12:06:282
141349943,0cyclictest3599-21lldpd08:53:412
141349943,0cyclictest3599-21lldpd08:45:402
141349943,0cyclictest3599-21lldpd08:37:112
141349943,0cyclictest3599-21lldpd08:08:402
141349943,0cyclictest3599-21lldpd07:53:092
141349943,0cyclictest26964-21awk08:55:302
141349943,0cyclictest26807-21awk12:20:272
141349943,0cyclictest24269-21awk12:00:292
141349943,0cyclictest23871-21snmpd11:45:232
141349943,0cyclictest23871-21snmpd11:25:532
141349943,0cyclictest23871-21snmpd09:45:512
141349943,0cyclictest1644-21awk11:10:292
141349943,0cyclictest11976-21awk09:05:282
141339944,0cyclictest0-21swapper/112:14:561
141339944,0cyclictest0-21swapper/111:22:171
141339944,0cyclictest0-21swapper/111:15:111
141339943,0cyclictest8550-21awk07:35:371
141339943,0cyclictest26792-21awk12:20:271
141339943,0cyclictest25404-21awk09:50:291
141339943,0cyclictest18146-21awk12:15:281
141339943,0cyclictest1233-21awk07:30:341
141339940,0cyclictest11710-21perf11:35:171
141329944,0cyclictest0-21swapper/008:58:050
141329943,0cyclictest30247-21awk07:50:370
141329943,0cyclictest23871-21snmpd08:43:360
141329943,0cyclictest23871-21snmpd08:34:360
141329943,0cyclictest23871-21snmpd07:23:350
141329943,0cyclictest20507-21awk09:10:270
141329943,0cyclictest12588-21awk10:20:260
141329943,0cyclictest10465-21awk10:00:280
149940,0migration/09528-21rt-features11:30:360
149940,0migration/04235-21kthreadcore08:15:370
149940,0migration/028100-21latency10:45:310
149940,0migration/011205-21munin-node09:20:410
329930,0watchdog/30-21swapper/308:38:143
329930,0watchdog/30-21swapper/307:20:463
329930,0watchdog/30-21swapper/306:55:343
29935030,0irq/33-eth0-tx-7446-21aten2.4-expect07:35:260
29935030,0irq/33-eth0-tx-32541-21aten2.4-expect07:30:240
29935030,0irq/33-eth0-tx-0-21swapper/008:10:240
29935030,0irq/33-eth0-tx-0-21swapper/008:00:250
29925030,0irq/32-eth0-rx-0-21swapper/307:10:243
249930,0watchdog/20-21swapper/207:35:222
249930,0watchdog/20-21swapper/207:30:382
179930,0migration/18312-21runrttasks10:55:291
179930,0migration/118199-21kthreadcore11:00:311
179930,0migration/111420-21fschecks_count07:15:261
169930,0watchdog/10-21swapper/109:45:301
169930,0watchdog/10-21swapper/108:05:341
141349933,0cyclictest0-21swapper/208:25:372
141349932,0cyclictest26771-21awk08:10:322
141349930,0cyclictest18995-21perl07:20:312
141349930,0cyclictest0-21swapper/208:15:292
141339933,0cyclictest438-21awk12:05:271
141339932,0cyclictest30450-21awk11:45:301
141339932,0cyclictest24344-21awk12:00:291
141339932,0cyclictest19397-21grep08:50:271
141339932,0cyclictest19169-21awk07:20:331
141339932,0cyclictest14635-21awk10:40:291
141339932,0cyclictest11951-21awk09:05:281
141339931,0cyclictest24288-21kthreadcore09:30:281
141339930,0cyclictest27112-21sh08:55:331
141339930,0cyclictest20321-21perl09:10:261
141329933,0cyclictest25303-21kworker/0:211:35:340
141329933,0cyclictest25303-21kworker/0:211:20:240
141329933,0cyclictest25303-21kworker/0:211:00:240
141329933,0cyclictest25303-21kworker/0:210:35:240
141329933,0cyclictest11665-21kworker/0:011:55:240
141329933,0cyclictest11665-21kworker/0:011:40:400
141329933,0cyclictest11143-21kworker/0:209:35:360
141329932,0cyclictest9642-21grep11:15:250
141329932,0cyclictest30652-21cut10:10:310
141329932,0cyclictest30652-21cut10:10:310
141329932,0cyclictest24486-21cpuspeed_turbos11:05:240
141329932,0cyclictest17528-21tune2fs12:15:250
141329932,0cyclictest14281-21awk09:25:280
169920,0watchdog/10-21swapper/107:25:181
141339920,0cyclictest11362-21cat08:45:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional