You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-05-23 - 21:02
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Thu May 23, 2024 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2034524637,0sleep10-21swapper/106:52:461
1999923929,0sleep30-21swapper/306:48:443
2028123627,0sleep20-21swapper/206:52:032
2001123524,0sleep00-21swapper/006:48:530
20434991313,0cyclictest14906-21sensors09:18:282
204349999,0cyclictest9235-21sensors07:48:332
339960,0migration/38923-21sh09:56:173
339960,0migration/35883-21fschecks_time11:13:153
339960,0migration/34390-21sh11:11:153
339960,0migration/331975-21munin-node07:23:133
339960,0migration/328758-21sh11:24:163
339960,0migration/321637-21sh12:18:233
339960,0migration/320693-21sh11:20:113
329960,0watchdog/325599-21perl08:03:163
29925060,0irq/32-eth0-rx-11037-21diskstats10:18:143
29925060,0irq/32-eth0-rx-0-21swapper/309:50:453
204349965,0cyclictest21132-21kthreadcore10:03:222
204329966,0cyclictest9254-21kworker/0:009:34:240
204329966,0cyclictest0-21swapper/011:12:020
149960,0migration/031053-21munin-node08:48:360
8315050,0irq/26-0000:00:0-21swapper/208:48:302
339950,0migration/39321-21munin-node07:03:303
339950,0migration/34732-21diskmemload11:29:233
339950,0migration/330740-21kthreadcore06:58:333
339950,0migration/329126-21munin-node11:43:333
339950,0migration/328820-21kthreadcore09:08:363
339950,0migration/327558-21taskset07:39:253
339950,0migration/32705-21runrttasks08:08:213
339950,0migration/324871-21kthreadcore11:03:293
339950,0migration/322784-21kthreadcore08:23:323
339950,0migration/318589-21kthreadcore07:33:233
339950,0migration/318433-21munin-node07:58:173
339950,0migration/318361-21kthreadcore11:38:273
339950,0migration/318142-21fschecks_time09:43:183
339950,0migration/317411-21cpuspeed_turbos07:13:113
339950,0migration/315810-21sh12:14:293
339950,0migration/314538-21munin-node08:58:353
339950,0migration/312447-21fschecks_time10:38:183
339950,0migration/310469-21fschecks_time09:58:173
329950,0watchdog/36004-21nfs_client08:53:283
329950,0watchdog/311919-21sh10:56:083
29935050,0irq/33-eth0-tx-9975-21munin-node07:08:230
29935050,0irq/33-eth0-tx-8902-21munin-node09:38:292
29935050,0irq/33-eth0-tx-8796-21munin-node12:13:232
29935050,0irq/33-eth0-tx-7556-21ssh12:10:002
29935050,0irq/33-eth0-tx-3664-21ssh10:51:052
29935050,0irq/33-eth0-tx-32726-21munin-node09:13:122
29935050,0irq/33-eth0-tx-31202-21munin-node11:48:092
29935050,0irq/33-eth0-tx-2470-21cut09:33:222
29935050,0irq/33-eth0-tx-240562sleep212:03:022
29935050,0irq/33-eth0-tx-21512-21irqrtprio10:43:212
29935050,0irq/33-eth0-tx-19076-21cut12:18:232
29935050,0irq/33-eth0-tx-17019-21fschecks_count09:03:162
29935050,0irq/33-eth0-tx-14814-21ssh11:56:152
29935050,0irq/33-eth0-tx-13582-21ssh11:36:232
29935050,0irq/33-eth0-tx-10332-21munin-run10:17:592
29935050,0irq/33-eth0-tx-0-21swapper/211:22:542
29935050,0irq/33-eth0-tx-0-21swapper/211:11:382
29935050,0irq/33-eth0-tx-0-21swapper/210:58:262
29935050,0irq/33-eth0-tx-0-21swapper/209:23:582
29935050,0irq/33-eth0-tx-0-21swapper/007:28:270
29925050,0irq/32-eth0-rx-6386-21runrttasks12:12:043
29925050,0irq/32-eth0-rx-4878-21kthreadcore10:33:243
29925050,0irq/32-eth0-rx-26302-21port_sshpriv10:43:283
29925050,0irq/32-eth0-rx-18075-21irqcore09:23:213
29925050,0irq/32-eth0-rx-15729-21ssh09:20:093
29925050,0irq/32-eth0-rx-14017-21iostat_ios10:58:233
29925050,0irq/32-eth0-rx-13923-21ssh11:37:343
29925050,0irq/32-eth0-rx-0-21swapper/312:03:413
29925050,0irq/32-eth0-rx-0-21swapper/311:56:113
29925050,0irq/32-eth0-rx-0-21swapper/311:49:563
29925050,0irq/32-eth0-rx-0-21swapper/310:48:593
29925050,0irq/32-eth0-rx-0-21swapper/310:24:013
29925050,0irq/32-eth0-rx-0-21swapper/310:14:433
29925050,0irq/32-eth0-rx-0-21swapper/310:08:423
29925050,0irq/32-eth0-rx-0-21swapper/310:05:403
29925050,0irq/32-eth0-rx-0-21swapper/309:39:113
29925050,0irq/32-eth0-rx-0-21swapper/309:04:403
29925050,0irq/32-eth0-rx-0-21swapper/308:18:283
29925050,0irq/32-eth0-rx-0-21swapper/307:48:343
259950,0migration/232740-21latency_hist09:32:592
259950,0migration/231654-21sh09:10:062
259950,0migration/228979-21latency_hist11:07:592
259950,0migration/227683-21kthreadcore09:48:202
259950,0migration/223596-21unixbench-2d07:13:302
259950,0migration/220213-21runrttasks08:18:382
249950,0watchdog/25854-21fschecks_time11:13:152
249950,0watchdog/23812-21munin-node07:03:162
249950,0watchdog/230346-21kthreadcore08:03:242
249950,0watchdog/20-21swapper/208:13:202
204349955,0cyclictest0-21swapper/211:59:152
204349955,0cyclictest0-21swapper/211:23:092
204349950,0cyclictest32001-21diskmemload10:08:062
204349950,0cyclictest0-21swapper/210:55:522
204349950,0cyclictest0-21swapper/210:20:432
204329955,0cyclictest9254-21kworker/0:010:25:120
204329955,0cyclictest9254-21kworker/0:010:18:260
204329955,0cyclictest9254-21kworker/0:010:13:080
204329955,0cyclictest9254-21kworker/0:010:09:000
204329955,0cyclictest9254-21kworker/0:010:04:080
204329955,0cyclictest9254-21kworker/0:009:53:220
204329955,0cyclictest9254-21kworker/0:009:48:180
204329955,0cyclictest9254-21kworker/0:009:43:160
204329955,0cyclictest9254-21kworker/0:009:28:000
204329955,0cyclictest9254-21kworker/0:009:18:500
204329955,0cyclictest9254-21kworker/0:009:08:020
204329955,0cyclictest9254-21kworker/0:009:03:360
204329955,0cyclictest9254-21kworker/0:008:58:080
204329955,0cyclictest9254-21kworker/0:008:53:120
204329955,0cyclictest9254-21kworker/0:008:44:020
204329955,0cyclictest9254-21kworker/0:008:38:400
204329955,0cyclictest9254-21kworker/0:008:33:080
204329955,0cyclictest9254-21kworker/0:008:29:200
204329955,0cyclictest9254-21kworker/0:008:18:020
204329955,0cyclictest9254-21kworker/0:008:13:040
204329955,0cyclictest9254-21kworker/0:008:09:360
204329955,0cyclictest9254-21kworker/0:008:04:280
204329955,0cyclictest9254-21kworker/0:007:58:100
204329955,0cyclictest6969-21kworker/0:012:14:000
204329955,0cyclictest6969-21kworker/0:012:08:060
204329955,0cyclictest6969-21kworker/0:012:06:020
204329955,0cyclictest6969-21kworker/0:011:59:020
204329955,0cyclictest6969-21kworker/0:011:53:100
204329955,0cyclictest4256-21kworker/0:211:03:000
204329955,0cyclictest4256-21kworker/0:210:59:080
204329955,0cyclictest4256-21kworker/0:210:53:240
204329955,0cyclictest4256-21kworker/0:210:48:040
204329955,0cyclictest4256-21kworker/0:210:43:280
204329955,0cyclictest30420-21kworker/0:207:48:020
204329955,0cyclictest30420-21kworker/0:207:43:300
204329955,0cyclictest30420-21kworker/0:207:40:080
204329955,0cyclictest30420-21kworker/0:207:23:100
204329955,0cyclictest30420-21kworker/0:207:18:080
204329955,0cyclictest30420-21kworker/0:207:13:180
204329955,0cyclictest30420-21kworker/0:207:03:460
204329955,0cyclictest30420-21kworker/0:206:58:120
204329955,0cyclictest30420-21kworker/0:206:53:520
204329955,0cyclictest11570-21kworker/0:111:48:380
204329955,0cyclictest11570-21kworker/0:111:43:220
204329955,0cyclictest11570-21kworker/0:111:38:300
204329955,0cyclictest11570-21kworker/0:111:33:120
204329955,0cyclictest11570-21kworker/0:111:28:080
204329955,0cyclictest11570-21kworker/0:111:23:520
204329955,0cyclictest11570-21kworker/0:111:14:160
204329954,0cyclictest9254-21kworker/0:009:58:160
204329954,0cyclictest18899-21kthreadcore12:18:230
179950,0migration/1634-21munin-node08:08:181
179950,0migration/15785-21unixbench_singl07:23:351
179950,0migration/131393-21timerandwakeup08:48:421
179950,0migration/131307-21kthreadcore10:48:241
179950,0migration/129315-21munin-node07:43:181
179950,0migration/125910-21munin-node08:03:211
179950,0migration/121765-21fschecks_count11:03:221
179950,0migration/118921-21kthreadcore09:03:211
179950,0migration/118011-21kthreadcore07:33:231
179950,0migration/112419-21kthreadcore07:08:201
169950,0watchdog/130354-21expr11:28:111
169950,0watchdog/110962-21perl07:53:161
149950,0migration/0838-21fschecks_count09:13:120
149950,0migration/03197-21aten_r1power_cu10:33:090
149950,0migration/027743-21unixbench_multi08:23:460
149950,0migration/020105-21timerwakeupswit07:33:340
149950,0migration/019455-21missed_timers11:18:270
149950,0migration/017950-21irqcore09:23:210
149950,0migration/017173-21kthreadcore10:38:250
149950,0migration/015325-21sh09:38:320
149950,0migration/01522-21munin-node10:28:320
149950,0migration/010122-21latency_hist07:52:590
339940,0migration/38452-21kthreadcore07:28:223
329940,0watchdog/312959-21kthreadcore07:08:203
329940,0watchdog/30-21swapper/310:28:123
329940,0watchdog/30-21swapper/309:13:163
329940,0watchdog/30-21swapper/308:33:203
329940,0watchdog/30-21swapper/308:28:323
329940,0watchdog/30-21swapper/308:13:203
29935040,0irq/33-eth0-tx-1249-21ssh09:53:072
29935040,0irq/33-eth0-tx-0-21swapper/210:33:012
29935040,0irq/33-eth0-tx-0-21swapper/209:44:212
29935040,0irq/33-eth0-tx-0-21swapper/208:58:382
29925040,0irq/32-eth0-rx-0-21swapper/311:58:203
29925040,0irq/32-eth0-rx-0-21swapper/309:34:003
29925040,0irq/32-eth0-rx-0-21swapper/309:28:493
29925040,0irq/32-eth0-rx-0-21swapper/308:38:203
29925040,0irq/32-eth0-rx-0-21swapper/306:53:233
259940,0migration/25917-21kthreadcore08:33:192
259940,0migration/25344-21latency08:08:232
259940,0migration/25294-21runrttasks07:23:272
259940,0migration/230588-21kthreadcore07:43:202
259940,0migration/228586-21aten2.4-expect08:28:102
259940,0migration/226894-21kthreadcore08:23:342
259940,0migration/224545-21kthreadcore11:43:242
259940,0migration/221294-21fschecks_count06:53:162
259940,0migration/220222-21kthreadcore08:43:222
259940,0migration/214410-21aten_r1power_po11:38:152
249940,0watchdog/227529-21cpuspeed_turbos10:28:122
249940,0watchdog/210395-21diskstats07:08:122
249940,0watchdog/20-21swapper/211:28:112
249940,0watchdog/20-21swapper/209:58:402
204349944,0cyclictest0-21swapper/210:38:032
204349943,0cyclictest3599-21lldpd10:23:422
204349943,0cyclictest3599-21lldpd08:38:272
204349943,0cyclictest23871-21snmpd08:55:112
204349943,0cyclictest23871-21snmpd07:53:412
204349943,0cyclictest23871-21snmpd07:18:262
204349943,0cyclictest23871-21snmpd07:00:262
204349943,0cyclictest14759-21awk07:33:182
204339943,0cyclictest7183-21awk11:33:191
204339943,0cyclictest4401-21awk08:33:151
204339943,0cyclictest3599-21lldpd11:51:141
204339943,0cyclictest3599-21lldpd10:43:501
204339943,0cyclictest3599-21lldpd10:10:411
204339943,0cyclictest3599-21lldpd09:53:181
204339943,0cyclictest3599-21lldpd09:27:521
204339943,0cyclictest25772-21awk09:28:211
204339943,0cyclictest23871-21snmpd11:39:581
204339943,0cyclictest23871-21snmpd09:08:561
204339943,0cyclictest17610-21sadc10:02:591
204339943,0cyclictest16687-21awk11:58:161
204339943,0cyclictest11553-21awk08:38:251
204339940,0cyclictest0-21swapper/110:56:451
179940,0migration/18701-21latency_hist09:37:591
179940,0migration/17462-21munin-node07:28:181
179940,0migration/1744-21kthreadcore08:28:221
179940,0migration/13718-21fschecks_time07:48:181
179940,0migration/13101-21kthreadcore09:13:221
179940,0migration/12834-21latency11:08:311
179940,0migration/127507-21kthreadcore09:48:201
179940,0migration/124224-21kthreadcore08:23:331
179940,0migration/121444-21fschecks_count07:38:121
179940,0migration/121211-21fschecks_count06:53:161
179940,0migration/118014-21fschecks_time09:43:181
179940,0migration/117187-21kthreadcore10:38:251
179940,0migration/116244-21sh09:40:291
179940,0migration/115417-21kthreadcore11:18:241
179940,0migration/114970-21sh11:56:481
179940,0migration/110966-21kthreadcore12:13:241
179940,0migration/110737-21kthreadcore08:58:251
179940,0migration/110009-21sh10:16:561
169940,0watchdog/13576-21kthreadcore12:08:191
169940,0watchdog/130373-21pmu-power12:03:351
169940,0watchdog/130004-21ssh11:45:071
169940,0watchdog/129738-21ssh11:27:471
169940,0watchdog/119609-21ssh10:59:001
169940,0watchdog/119110-21ssh10:03:201
169940,0watchdog/116233-21ssh09:22:041
169940,0watchdog/112181-21irqstats10:18:201
169940,0watchdog/11194-21ssh10:28:281
169940,0watchdog/10-21swapper/112:18:231
169940,0watchdog/10-21swapper/111:13:151
169940,0watchdog/10-21swapper/107:03:201
29925030,0irq/32-eth0-rx-0-21swapper/308:48:183
29925030,0irq/32-eth0-rx-0-21swapper/308:43:143
29925030,0irq/32-eth0-rx-0-21swapper/307:53:223
29925030,0irq/32-eth0-rx-0-21swapper/307:43:103
29925030,0irq/32-eth0-rx-0-21swapper/307:18:363
249930,0watchdog/20-21swapper/207:38:202
204349932,0cyclictest23871-21snmpd07:29:412
204349932,0cyclictest19117-21cut07:58:232
204339933,0cyclictest15042-21awk08:18:221
204339932,0cyclictest3599-21lldpd10:33:191
204339932,0cyclictest3599-21lldpd08:43:071
204339932,0cyclictest3599-21lldpd08:13:101
204339932,0cyclictest32593-21sed08:53:181
204339932,0cyclictest20177-21awk10:23:201
204339930,0cyclictest26104-21cut07:18:271
169930,0watchdog/123262-21ps07:13:241
169930,0watchdog/10-21swapper/107:58:241
169920,0watchdog/10-21swapper/106:58:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional