You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-13 - 04:54
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Sat Sep 13, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
947523927,0sleep20-21swapper/218:20:122
965123725,0sleep10-21swapper/118:22:151
952123525,0sleep30-21swapper/318:20:413
970023026,0sleep00-21swapper/018:22:460
339970,0migration/311102-21sh22:28:193
329970,0watchdog/330258-21latency18:34:293
329970,0watchdog/322194-21sort19:39:253
329970,0watchdog/30-21swapper/322:34:243
329970,0watchdog/30-21swapper/322:18:283
29925070,0irq/32-eth0-rx-0-21swapper/321:59:253
29925070,0irq/32-eth0-rx-0-21swapper/320:43:223
8315060,0irq/26-0000:00:0-21swapper/318:59:243
4160,0ktimersoftd/00-21swapper/022:31:410
339960,0migration/330816-21sh22:59:113
339960,0migration/323874-21sh23:14:073
339960,0migration/318560-21sh21:15:023
339960,0migration/314724-21gltestperf18:49:243
329960,0watchdog/30-21swapper/323:31:203
329960,0watchdog/30-21swapper/323:24:363
329960,0watchdog/30-21swapper/323:19:203
329960,0watchdog/30-21swapper/323:06:243
329960,0watchdog/30-21swapper/322:49:283
329960,0watchdog/30-21swapper/322:23:243
329960,0watchdog/30-21swapper/321:28:213
329960,0watchdog/30-21swapper/321:10:253
329960,0watchdog/30-21swapper/321:02:453
329960,0watchdog/30-21swapper/320:54:133
329960,0watchdog/30-21swapper/320:45:053
329960,0watchdog/30-21swapper/320:38:093
329960,0watchdog/30-21swapper/319:48:013
329960,0watchdog/30-21swapper/319:35:053
329960,0watchdog/30-21swapper/319:20:533
329960,0watchdog/30-21swapper/318:39:253
29925060,0irq/32-eth0-rx-8320-21ssh20:51:483
29925060,0irq/32-eth0-rx-10377-21grep21:49:263
29925060,0irq/32-eth0-rx-0-21swapper/323:50:433
29925060,0irq/32-eth0-rx-0-21swapper/322:43:033
29925060,0irq/32-eth0-rx-0-21swapper/321:54:223
29925060,0irq/32-eth0-rx-0-21swapper/321:46:393
29925060,0irq/32-eth0-rx-0-21swapper/321:38:173
29925060,0irq/32-eth0-rx-0-21swapper/321:22:283
29925060,0irq/32-eth0-rx-0-21swapper/321:05:363
29925060,0irq/32-eth0-rx-0-21swapper/320:29:233
29925060,0irq/32-eth0-rx-0-21swapper/320:29:233
29925060,0irq/32-eth0-rx-0-21swapper/320:26:173
19160,0ktimersoftd/117209-21ssh22:12:121
98859955,0cyclictest0-21swapper/222:50:412
98859955,0cyclictest0-21swapper/218:49:372
98859950,0cyclictest0-21swapper/222:13:282
98859950,0cyclictest0-21swapper/220:44:242
98859950,0cyclictest0-21swapper/220:40:372
98849955,0cyclictest0-21swapper/123:43:191
98849955,0cyclictest0-21swapper/122:28:171
98849954,0cyclictest26511-21ssh23:54:031
98849950,0cyclictest0-21swapper/121:35:561
98839955,0cyclictest0-21swapper/023:48:370
98839955,0cyclictest0-21swapper/022:40:500
98839954,0cyclictest5879-21ssh23:01:190
98839950,0cyclictest0-21swapper/022:44:550
98839950,0cyclictest0-21swapper/022:20:200
98839950,0cyclictest0-21swapper/021:40:270
98839950,0cyclictest0-21swapper/021:21:030
98839950,0cyclictest0-21swapper/020:38:010
98839950,0cyclictest0-21swapper/019:41:150
4150,0ktimersoftd/00-21swapper/020:45:350
339950,0migration/37228-21munin-node18:44:213
339950,0migration/35881-21fschecks_time22:44:213
339950,0migration/34319-21munin-node19:49:253
339950,0migration/331378-21kthreadcore23:34:273
339950,0migration/320960-21taskset18:54:113
339950,0migration/317344-21munin-node18:29:203
339950,0migration/312505-21kthreadcore23:44:293
329950,0watchdog/31681-21grep22:04:203
329950,0watchdog/316174-21port_sshpriv18:24:293
329950,0watchdog/311177-21perl19:09:253
329950,0watchdog/30-21swapper/323:14:283
329950,0watchdog/30-21swapper/322:54:123
329950,0watchdog/30-21swapper/322:10:323
329950,0watchdog/30-21swapper/321:39:213
329950,0watchdog/30-21swapper/320:19:213
329950,0watchdog/30-21swapper/320:15:133
329950,0watchdog/30-21swapper/320:10:173
329950,0watchdog/30-21swapper/320:04:253
329950,0watchdog/30-21swapper/320:00:293
329950,0watchdog/30-21swapper/319:55:413
329950,0watchdog/30-21swapper/319:29:373
329950,0watchdog/30-21swapper/319:14:093
329950,0watchdog/30-21swapper/319:04:253
29935050,0irq/33-eth0-tx-6225-21ssh23:02:162
29935050,0irq/33-eth0-tx-15164-21munin-node23:24:322
29935050,0irq/33-eth0-tx-0-21swapper/018:44:330
29925050,0irq/32-eth0-rx-0-21swapper/323:39:243
29925050,0irq/32-eth0-rx-0-21swapper/322:29:593
29925050,0irq/32-eth0-rx-0-21swapper/321:29:543
29925050,0irq/32-eth0-rx-0-21swapper/319:26:583
27150,0ktimersoftd/20-21swapper/221:23:442
19150,0ktimersoftd/10-21swapper/121:23:441
179950,0migration/1677-21sendmail_mailqu22:19:321
179950,0migration/1525-21sh21:04:291
179950,0migration/114497-21kthreadcore19:09:291
149950,0migration/026638-21latency_hist23:54:080
149950,0migration/026528-21fschecks_time23:34:230
149950,0migration/016952-21unixbench_multi19:54:320
149950,0migration/015081-21kthreadcore20:54:270
98859944,0cyclictest0-21swapper/223:51:422
98859944,0cyclictest0-21swapper/223:29:372
98859944,0cyclictest0-21swapper/222:35:142
98859943,0cyclictest7753-21awk18:44:242
98859943,0cyclictest661-21awk20:09:252
98859943,0cyclictest4040-21tail21:09:222
98859943,0cyclictest3599-21lldpd22:24:082
98859943,0cyclictest3599-21lldpd19:04:572
98859943,0cyclictest3599-21lldpd18:41:582
98859943,0cyclictest31631-21sadc19:24:082
98859943,0cyclictest30041-21awk22:39:252
98859943,0cyclictest28007-21awk22:19:272
98859943,0cyclictest23871-21snmpd19:12:022
98859943,0cyclictest23871-21snmpd18:30:322
98859943,0cyclictest23871-21snmpd18:27:472
98859943,0cyclictest21141-21sadc20:24:082
98859943,0cyclictest15078-21awk19:34:242
98859943,0cyclictest11479-21awk19:54:222
98859943,0cyclictest10539-21awk20:54:242
98859941,0cyclictest0-21swapper/220:59:232
98849944,0cyclictest0-21swapper/121:24:541
98849944,0cyclictest0-21swapper/120:45:381
98849943,0cyclictest9923-21awk23:24:251
98849943,0cyclictest6326-21awk21:29:211
98849943,0cyclictest23871-21snmpd23:48:041
98849943,0cyclictest23871-21snmpd23:36:341
98849943,0cyclictest23871-21snmpd23:30:191
98849943,0cyclictest23871-21snmpd23:14:191
98849943,0cyclictest23871-21snmpd22:53:341
98849943,0cyclictest23871-21snmpd21:53:031
98849943,0cyclictest23871-21snmpd21:47:481
98849943,0cyclictest23871-21snmpd21:40:181
98849943,0cyclictest23871-21snmpd19:06:021
98849943,0cyclictest21466-21awk22:34:251
98849943,0cyclictest18333-21awk18:29:241
98849943,0cyclictest11433-21sadc21:14:071
98849943,0cyclictest10194-21sadc19:54:081
98839944,0cyclictest0-21swapper/018:59:100
98839943,0cyclictest4294-21awk19:49:250
98839943,0cyclictest382-21awk18:39:260
98839943,0cyclictest3599-21lldpd22:04:030
98839943,0cyclictest3599-21lldpd21:54:340
98839943,0cyclictest32279-21awk21:44:260
98839943,0cyclictest27996-21awk21:04:240
98839943,0cyclictest23871-21snmpd19:33:480
98839943,0cyclictest21619-21sort18:54:220
98839943,0cyclictest18563-21awk19:14:260
98839943,0cyclictest1759-21cut23:39:170
98839943,0cyclictest16728-21awk23:09:240
98839943,0cyclictest16411-21awk20:39:270
98839943,0cyclictest11333-21awk19:09:260
98839942,0cyclictest25978-21awk20:04:240
98839940,0cyclictest0-21swapper/020:02:320
8315040,0irq/26-0000:00:0-21swapper/222:44:082
29935040,0irq/33-eth0-tx-31542-21df21:44:242
29935040,0irq/33-eth0-tx-25713-21munin-node23:34:262
29935040,0irq/33-eth0-tx-21143-21diskmemload21:34:082
29935040,0irq/33-eth0-tx-0-21swapper/222:54:202
29935040,0irq/33-eth0-tx-0-21swapper/222:29:372
29935040,0irq/33-eth0-tx-0-21swapper/222:02:512
29935040,0irq/33-eth0-tx-0-21swapper/221:54:222
29935040,0irq/33-eth0-tx-0-21swapper/221:39:242
29935040,0irq/33-eth0-tx-0-21swapper/221:29:412
29935040,0irq/33-eth0-tx-0-21swapper/220:35:252
29935040,0irq/33-eth0-tx-0-21swapper/220:29:462
29935040,0irq/33-eth0-tx-0-21swapper/220:29:462
29935040,0irq/33-eth0-tx-0-21swapper/220:25:372
29935040,0irq/33-eth0-tx-0-21swapper/020:14:230
29935040,0irq/33-eth0-tx-0-21swapper/019:19:230
29935040,0irq/33-eth0-tx-0-21swapper/018:34:220
259940,0migration/27723-21latency23:39:262
259940,0migration/27464-21sh21:49:112
259940,0migration/24060-21kthreadcore20:49:312
259940,0migration/231286-21kthreadcore21:24:252
259940,0migration/229489-21irqcore18:59:242
259940,0migration/227810-21runrttasks19:40:222
259940,0migration/22626-21runrttasks19:45:512
259940,0migration/226254-21kthreadcore18:34:282
259940,0migration/223487-21kthreadcore22:14:272
259940,0migration/219874-21kthreadcore19:59:262
259940,0migration/219180-21kthreadcore23:09:262
259940,0migration/211421-21kthreadcore23:04:262
249940,0watchdog/232711-21cut23:19:202
249940,0watchdog/222245-21irqcore18:54:252
249940,0watchdog/21682-21fschecks_time22:04:202
249940,0watchdog/211923-21cpu21:14:212
249940,0watchdog/20-21swapper/220:14:292
179940,0migration/15741-21fschecks_time22:44:211
179940,0migration/132542-21fschecks_count20:09:231
179940,0migration/131805-21latency_hist19:24:091
179940,0migration/131361-21taskset18:38:211
179940,0migration/130907-21aten_r1power_en22:59:191
179940,0migration/130082-21kthreadcore20:04:261
179940,0migration/129963-21ksmtuned20:29:151
179940,0migration/129963-21ksmtuned20:29:151
179940,0migration/129131-21gltestperf18:59:221
179940,0migration/127863-21taskset18:58:061
179940,0migration/127003-21kthreadcore19:19:281
179940,0migration/126052-21kthreadcore20:24:261
179940,0migration/121933-21fschecks_time19:39:231
179940,0migration/120906-21kthreadcore21:54:261
179940,0migration/117081-21irqrtprio23:09:251
179940,0migration/115920-21kthreadcore19:34:261
179940,0migration/113805-21aten_r1power_cu18:49:211
179940,0migration/113800-21runrttasks20:35:581
179940,0migration/111563-21latency_hist22:29:081
179940,0migration/11112-21kthreadcore22:39:271
179940,0migration/111093-21kthreadcore20:14:291
169940,0watchdog/17146-21ssh23:04:161
169940,0watchdog/17011-21diskstats18:44:211
169940,0watchdog/122067-21grep20:59:291
169940,0watchdog/118509-21ssh21:14:531
169940,0watchdog/117854-21aten2.4-expect19:59:211
169940,0watchdog/11680-21tune2fs22:04:201
169940,0watchdog/115783-21ssh20:39:251
169940,0watchdog/10-21swapper/118:39:251
149940,0migration/08632-21runrttasks20:52:390
149940,0migration/08057-21sh22:05:250
149940,0migration/07374-21kthreadcore21:29:230
149940,0migration/06628-21taskset20:13:100
149940,0migration/05580-21kthreadcore19:04:250
149940,0migration/030255-21irqcore21:24:240
149940,0migration/027653-21runrttasks22:36:320
149940,0migration/022044-21kthreadcore23:29:280
149940,0migration/020879-21ksmtuned20:21:150
149940,0migration/017900-21kthreadcore22:49:270
149940,0migration/017302-21kthreadcore21:34:250
149940,0migration/017046-21taskset18:29:030
149940,0migration/015608-21sh23:24:470
149940,0migration/012970-21kthreadcore21:49:270
98859932,0cyclictest7570-21kthreadcore19:49:282
98859932,0cyclictest32745-21tune2fs19:24:232
98859932,0cyclictest26820-21munin-node21:04:252
98859932,0cyclictest25238-21awk23:14:212
98859932,0cyclictest12994-21sensors19:29:302
98849932,0cyclictest9790-21cut20:54:211
98849932,0cyclictest6808-21munin-node19:29:191
98849932,0cyclictest24721-21munin-node21:59:251
98849932,0cyclictest15995-21kthreadcore20:19:261
98849930,0cyclictest23250-21perl22:54:221
98839933,0cyclictest0-21swapper/020:59:270
98839932,0cyclictest7319-21cut22:24:270
98839932,0cyclictest6696-21sensors23:19:310
98839932,0cyclictest31102-21awk20:29:250
98839932,0cyclictest31102-21awk20:29:250
98839932,0cyclictest261402sleep022:18:020
98839932,0cyclictest25223-21awk23:14:210
98839932,0cyclictest23871-21snmpd21:13:480
98839932,0cyclictest23364-21ls22:54:240
98839932,0cyclictest22489-21awk20:24:240
98839932,0cyclictest13827-21basename18:49:210
98839932,0cyclictest10711-21awk22:09:250
98839930,0cyclictest1294-21cut19:24:270
98839930,0cyclictest0-21swapper/019:44:210
98839930,0cyclictest0-21swapper/019:34:220
98839930,0cyclictest0-21swapper/018:29:260
4130,0ktimersoftd/018675-21sh21:15:180
249930,0watchdog/224856-21runrttasks20:04:092
249930,0watchdog/210649-21diskstats23:44:242
249930,0watchdog/20-21swapper/219:14:092
179930,0migration/119643-21irqrtprio22:14:251
179930,0migration/113828-21kthreadcore18:24:261
169930,0watchdog/17763-21ssh20:50:211
169930,0watchdog/10-21swapper/123:19:201
169930,0watchdog/10-21swapper/119:14:251
98839921,0cyclictest0-21swapper/023:04:150
169920,0watchdog/10-21swapper/119:54:091
169920,0watchdog/10-21swapper/119:44:091
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional