You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-21 - 05:48
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Sun Jul 21, 2024 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3057727267,0sleep10-21swapper/118:47:261
3071924132,0sleep30-21swapper/318:49:053
3048223627,0sleep20-21swapper/218:46:192
2475923625,0sleep00-21swapper/018:44:370
329980,0watchdog/328402-21kthreadcore23:09:413
329980,0watchdog/30-21swapper/321:34:413
329970,0watchdog/310843-21grep23:19:373
329970,0watchdog/30-21swapper/323:25:413
329970,0watchdog/30-21swapper/321:14:133
29925070,0irq/32-eth0-rx-18535-21aten_r1power_fr20:54:343
339960,0migration/31235-21kthreadcore20:19:423
329960,0watchdog/31759-21cat19:34:413
329960,0watchdog/30-21swapper/323:45:253
329960,0watchdog/30-21swapper/323:34:373
329960,0watchdog/30-21swapper/322:54:253
329960,0watchdog/30-21swapper/322:50:213
329960,0watchdog/30-21swapper/322:33:173
329960,0watchdog/30-21swapper/322:26:213
329960,0watchdog/30-21swapper/322:15:093
329960,0watchdog/30-21swapper/322:06:293
329960,0watchdog/30-21swapper/321:43:373
329960,0watchdog/30-21swapper/321:07:093
329960,0watchdog/30-21swapper/320:01:133
329960,0watchdog/30-21swapper/319:53:573
329960,0watchdog/30-21swapper/319:44:253
329960,0watchdog/30-21swapper/319:26:573
329960,0watchdog/30-21swapper/300:09:403
308189966,0cyclictest0-21swapper/122:28:551
29925060,0irq/32-eth0-rx-9414-21cat21:44:403
29925060,0irq/32-eth0-rx-21483-21munin-node00:04:353
29925060,0irq/32-eth0-rx-21270-21cut22:09:413
29925060,0irq/32-eth0-rx-18054-21cut23:04:413
29925060,0irq/32-eth0-rx-0-21swapper/323:39:323
29925060,0irq/32-eth0-rx-0-21swapper/323:01:413
29925060,0irq/32-eth0-rx-0-21swapper/322:40:013
29925060,0irq/32-eth0-rx-0-21swapper/322:22:503
29925060,0irq/32-eth0-rx-0-21swapper/321:59:523
29925060,0irq/32-eth0-rx-0-21swapper/321:57:503
29925060,0irq/32-eth0-rx-0-21swapper/321:32:023
29925060,0irq/32-eth0-rx-0-21swapper/321:26:263
29925060,0irq/32-eth0-rx-0-21swapper/321:20:293
29925060,0irq/32-eth0-rx-0-21swapper/321:14:593
29925060,0irq/32-eth0-rx-0-21swapper/321:01:463
29925060,0irq/32-eth0-rx-0-21swapper/320:50:123
29925060,0irq/32-eth0-rx-0-21swapper/300:18:053
29925060,0irq/32-eth0-rx-0-21swapper/300:01:153
149960,0migration/025622-21sh23:26:220
149960,0migration/010877-21kthreadcore00:14:450
8315050,0irq/26-0000:00:4921-21munin-run21:24:242
8315050,0irq/26-0000:00:0-21swapper/220:29:242
339950,0migration/39810-21latency_hist20:04:243
339950,0migration/38184-21kthreadcore18:54:413
339950,0migration/331247-21kthreadcore23:49:403
339950,0migration/32936-21kthreadcore18:49:413
339950,0migration/326010-21kthreadcore19:29:413
339950,0migration/319229-21munin-node20:29:423
339950,0migration/311881-21kthreadcore20:24:443
329950,0watchdog/30-21swapper/323:56:043
329950,0watchdog/30-21swapper/323:30:413
329950,0watchdog/30-21swapper/323:15:493
329950,0watchdog/30-21swapper/320:47:413
329950,0watchdog/30-21swapper/320:42:253
329950,0watchdog/30-21swapper/320:37:573
329950,0watchdog/30-21swapper/320:09:373
329950,0watchdog/30-21swapper/319:54:293
329950,0watchdog/30-21swapper/319:42:293
329950,0watchdog/30-21swapper/319:21:253
329950,0watchdog/30-21swapper/319:09:333
329950,0watchdog/30-21swapper/319:09:333
329950,0watchdog/30-21swapper/319:04:333
329950,0watchdog/30-21swapper/319:02:013
308189955,0cyclictest0-21swapper/120:49:351
308189955,0cyclictest0-21swapper/119:54:431
308189955,0cyclictest0-21swapper/119:34:421
308189950,0cyclictest0-21swapper/123:03:171
308189950,0cyclictest0-21swapper/122:44:221
308189950,0cyclictest0-21swapper/122:16:271
308189950,0cyclictest0-21swapper/121:54:041
308189950,0cyclictest0-21swapper/121:12:171
308189950,0cyclictest0-21swapper/121:05:211
308189950,0cyclictest0-21swapper/120:19:381
29935050,0irq/33-eth0-tx-5890-21df22:39:342
29935050,0irq/33-eth0-tx-29241-21awk22:49:442
29935050,0irq/33-eth0-tx-24660-21munin-node21:54:492
29935050,0irq/33-eth0-tx-22262-21ssh22:48:212
29935050,0irq/33-eth0-tx-20270-21aten2.4-expect23:44:352
29935050,0irq/33-eth0-tx-17615-21ssh22:08:152
29935050,0irq/33-eth0-tx-1313-21diskstats21:59:362
29935050,0irq/33-eth0-tx-12384-21kthreadcore20:49:400
29935050,0irq/33-eth0-tx-0-21swapper/223:51:172
29935050,0irq/33-eth0-tx-0-21swapper/221:44:442
29935050,0irq/33-eth0-tx-0-21swapper/221:37:272
29935050,0irq/33-eth0-tx-0-21swapper/221:26:232
29935050,0irq/33-eth0-tx-0-21swapper/220:59:152
29935050,0irq/33-eth0-tx-0-21swapper/020:04:490
29935050,0irq/33-eth0-tx-0-21swapper/019:14:460
29925050,0irq/32-eth0-rx-16426-21diskstats21:49:383
29925050,0irq/32-eth0-rx-0-21swapper/322:44:333
29925050,0irq/32-eth0-rx-0-21swapper/322:34:283
259950,0migration/227829-21fschecks_time20:59:382
259950,0migration/221959-21kthreadcore23:24:432
259950,0migration/220668-21aten_r1power_en20:34:342
259950,0migration/21969-21runrttasks19:13:302
259950,0migration/21969-21runrttasks19:13:302
259950,0migration/219570-21kthreadcore23:04:422
259950,0migration/21712-21timerwakeupswit19:54:492
259950,0migration/215848-21kthreadcore18:59:402
249950,0watchdog/231896-21cut18:49:372
19150,0ktimersoftd/10-21swapper/122:09:511
169950,0watchdog/127964-21kthreadcore22:49:411
169950,0watchdog/126701-21ntp_states22:29:451
169950,0watchdog/12347-21kthreadcore18:49:411
149950,0migration/0874-21aten_r1power_cu21:59:340
149950,0migration/0845-21latency19:09:440
149950,0migration/0845-21latency19:09:440
149950,0migration/05758-21taskset20:21:410
149950,0migration/04565-21missed_timers18:49:420
149950,0migration/031382-21kthreadcore22:34:420
149950,0migration/028263-21sh21:18:040
149950,0migration/025581-21munin-node20:14:390
149950,0migration/024434-21irqrtprio22:49:390
149950,0migration/022817-21munin-node21:34:360
149950,0migration/02236-21sh22:17:350
149950,0migration/021021-21kthreadcore20:54:400
149950,0migration/020882-21unixbench-2d22:44:510
149950,0migration/020798-21fschecks_count19:04:370
149950,0migration/020414-21kthreadcore23:04:430
149950,0migration/019448-21kthreadcore20:09:410
149950,0migration/019074-21munin-node19:44:420
149950,0migration/017436-21kthreadcore21:09:410
149950,0migration/016331-21kthreadcore18:59:400
149950,0migration/015456-21kthreadcore21:29:410
149950,0migration/013009-21sh21:27:580
149950,0migration/012725-21kthreadcore19:19:430
149950,0migration/011989-21sh23:56:530
149950,0migration/011912-21kthreadcore22:04:410
8315040,0irq/26-0000:00:0-21swapper/223:39:242
4140,0ktimersoftd/00-21swapper/023:35:550
329940,0watchdog/35453-21kthreadcore19:14:413
329940,0watchdog/30-21swapper/320:15:493
308199943,0cyclictest5836-21sadc19:39:252
308199943,0cyclictest4359-21awk21:04:412
308199943,0cyclictest3454-21awk19:14:392
308199943,0cyclictest32493-21awk20:19:402
308199943,0cyclictest23871-21snmpd00:10:262
308189944,0cyclictest0-21swapper/123:37:111
308189944,0cyclictest0-21swapper/121:25:131
308189943,0cyclictest6958-21awk18:54:391
308189943,0cyclictest3643-21awk19:59:391
308189943,0cyclictest3599-21lldpd23:48:441
308189943,0cyclictest3599-21lldpd23:21:061
308189943,0cyclictest3599-21lldpd23:05:351
308189943,0cyclictest3599-21lldpd22:23:341
308189943,0cyclictest3599-21lldpd21:58:331
308189943,0cyclictest3599-21lldpd20:41:191
308189943,0cyclictest3599-21lldpd20:35:191
308189943,0cyclictest3599-21lldpd20:04:491
308189943,0cyclictest3599-21lldpd19:50:481
308189943,0cyclictest3599-21lldpd19:16:191
308189943,0cyclictest3599-21lldpd19:06:191
308189943,0cyclictest32064-21awk21:39:381
308189943,0cyclictest31128-21sadc20:19:241
308189943,0cyclictest30894-21awk00:09:391
308189943,0cyclictest30122-21awk21:19:431
308189943,0cyclictest23871-21snmpd23:55:411
308189943,0cyclictest23871-21snmpd20:29:411
308189943,0cyclictest23871-21snmpd19:27:411
308189943,0cyclictest22521-21awk00:04:411
308189943,0cyclictest1975-21awk21:59:391
308189943,0cyclictest19517-21awk23:24:401
308189941,0cyclictest1962-40yum-updatesd-he19:13:291
308189941,0cyclictest1962-40yum-updatesd-he19:13:291
308179944,0cyclictest0-21swapper/022:41:320
308179943,0cyclictest9651-21sadc23:19:240
308179943,0cyclictest4385-21awk21:04:410
308179943,0cyclictest3709-21kworker/0:021:46:310
308179943,0cyclictest3709-21kworker/0:020:44:410
308179943,0cyclictest3599-21lldpd21:19:520
308179943,0cyclictest3262-21irqbalance23:44:380
308179943,0cyclictest3262-21irqbalance22:19:390
308179943,0cyclictest28165-21awk20:59:400
308179943,0cyclictest27827-21munin-node20:39:430
308179943,0cyclictest26180-21awk23:09:380
308179943,0cyclictest25727-21kworker/0:119:29:410
308179943,0cyclictest23871-21snmpd23:50:410
308179943,0cyclictest23871-21snmpd23:30:560
308179943,0cyclictest11052-21awk23:19:380
308179940,0cyclictest0-21swapper/022:26:520
308179940,0cyclictest0-21swapper/020:29:240
29935040,0irq/33-eth0-tx-6124-21ssh23:54:402
29935040,0irq/33-eth0-tx-22177-21ssh22:29:412
29935040,0irq/33-eth0-tx-13786-21ssh21:29:322
29935040,0irq/33-eth0-tx-13152-21ssh23:59:342
29935040,0irq/33-eth0-tx-12143-21ssh22:24:402
29935040,0irq/33-eth0-tx-10408-21cut22:59:452
29935040,0irq/33-eth0-tx-0-21swapper/223:31:002
29935040,0irq/33-eth0-tx-0-21swapper/223:14:372
29935040,0irq/33-eth0-tx-0-21swapper/223:09:352
29935040,0irq/33-eth0-tx-0-21swapper/222:54:252
29935040,0irq/33-eth0-tx-0-21swapper/222:34:272
29935040,0irq/33-eth0-tx-0-21swapper/222:14:312
29935040,0irq/33-eth0-tx-0-21swapper/222:11:242
29935040,0irq/33-eth0-tx-0-21swapper/221:50:522
29935040,0irq/33-eth0-tx-0-21swapper/221:14:262
29935040,0irq/33-eth0-tx-0-21swapper/221:09:332
29935040,0irq/33-eth0-tx-0-21swapper/200:14:382
29935040,0irq/33-eth0-tx-0-21swapper/200:04:412
29935040,0irq/33-eth0-tx-0-21swapper/020:34:380
29935040,0irq/33-eth0-tx-0-21swapper/020:29:450
29935040,0irq/33-eth0-tx-0-21swapper/019:39:400
259940,0migration/29161-21kthreadcore19:39:402
259940,0migration/24502-21kthreadcore21:39:412
259940,0migration/227547-21kthreadcore20:14:402
259940,0migration/227328-21latency_hist19:09:252
259940,0migration/221474-21hddtemp_smartct19:49:382
259940,0migration/217985-21hwlatdetect20:09:382
249940,0watchdog/29930-21ssh23:34:532
249940,0watchdog/225570-21kthreadcore19:29:412
249940,0watchdog/216027-21python20:04:492
249940,0watchdog/213854-21diskstats20:29:372
249940,0watchdog/210841-21grep23:19:372
249940,0watchdog/210764-21if_eth1.319:19:412
249940,0watchdog/20-21swapper/222:19:412
249940,0watchdog/20-21swapper/220:53:332
249940,0watchdog/20-21swapper/218:54:372
179940,0migration/131875-21kthreadcore20:59:421
179940,0migration/128672-21aten_r1power_en23:49:331
179940,0migration/119093-21ntp_offset18:59:431
179940,0migration/118523-21kthreadcore22:44:421
179940,0migration/115486-21missed_timers19:19:441
179940,0migration/110724-21kthreadcore20:24:441
169940,0watchdog/18805-21kthreadcore00:14:441
169940,0watchdog/18125-21ssh23:15:011
169940,0watchdog/16072-21munin-node19:39:411
169940,0watchdog/15364-21munin-node22:54:451
169940,0watchdog/128690-21sensors_fan21:34:451
169940,0watchdog/128401-21kthreadcore23:09:411
169940,0watchdog/124416-21diskstats19:29:371
169940,0watchdog/123754-21taskset20:12:171
169940,0watchdog/122818-21kthreadcore20:54:411
169940,0watchdog/117289-21kthreadcore23:39:451
169940,0watchdog/117052-21runrttasks22:06:411
169940,0watchdog/116967-21cut19:44:411
169940,0watchdog/113766-21perl23:59:361
169940,0watchdog/112046-21kthreadcore21:44:411
169940,0watchdog/10-21swapper/122:34:251
169940,0watchdog/10-21swapper/121:29:411
169940,0watchdog/10-21swapper/121:14:451
149940,0migration/07483-21kernelversion18:54:410
149940,0migration/028603-21fschecks_time19:54:380
149940,0migration/026004-21runrttasks22:12:320
308199932,0cyclictest4030-21sed19:59:402
308199932,0cyclictest17876-21awk19:24:402
308199930,0cyclictest0-21swapper/219:44:252
308189932,0cyclictest3599-21lldpd23:29:371
308189932,0cyclictest3599-21lldpd20:44:491
308179933,0cyclictest3709-21kworker/0:023:39:270
308179933,0cyclictest3709-21kworker/0:022:59:270
308179933,0cyclictest3709-21kworker/0:022:54:270
308179933,0cyclictest3709-21kworker/0:022:29:350
308179933,0cyclictest3709-21kworker/0:021:54:290
308179933,0cyclictest3709-21kworker/0:021:49:270
308179933,0cyclictest3709-21kworker/0:021:39:270
308179933,0cyclictest3709-21kworker/0:000:09:310
308179933,0cyclictest3709-21kworker/0:000:04:250
308179933,0cyclictest25727-21kworker/0:119:59:250
308179933,0cyclictest25727-21kworker/0:119:49:270
308179933,0cyclictest25727-21kworker/0:119:34:270
308179933,0cyclictest25727-21kworker/0:119:24:390
308179930,0cyclictest0-21swapper/023:59:250
308199922,0cyclictest27798-21cat20:39:242
308199921,0cyclictest3262-21irqbalance20:44:292
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional