You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 12:37
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Tue Dec 02, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1217623829,0sleep10-21swapper/118:15:291
1224623728,0sleep20-21swapper/218:16:222
1212623727,0sleep30-21swapper/318:14:553
1214923423,0sleep00-21swapper/018:15:120
329980,0watchdog/30-21swapper/323:29:423
29925080,0irq/32-eth0-rx-0-21swapper/323:23:193
329970,0watchdog/328118-21cut19:34:383
329970,0watchdog/30-21swapper/321:14:183
29925070,0irq/32-eth0-rx-0-21swapper/322:34:173
29925070,0irq/32-eth0-rx-0-21swapper/319:24:373
339960,0migration/38001-21munin-node23:34:353
339960,0migration/325918-21missed_timers23:04:423
339960,0migration/31964-21sh20:40:583
339960,0migration/313918-21munin-node22:04:333
339960,0migration/311745-21fschecks_count22:59:313
339960,0migration/311294-21sh22:00:033
329960,0watchdog/390272chrt19:21:343
329960,0watchdog/31030-21ssh22:50:503
329960,0watchdog/30-21swapper/323:28:023
329960,0watchdog/30-21swapper/322:14:343
329960,0watchdog/30-21swapper/322:14:343
329960,0watchdog/30-21swapper/322:12:263
329960,0watchdog/30-21swapper/321:58:343
329960,0watchdog/30-21swapper/321:44:503
329960,0watchdog/30-21swapper/321:40:263
329960,0watchdog/30-21swapper/321:33:583
329960,0watchdog/30-21swapper/321:24:303
329960,0watchdog/30-21swapper/321:19:263
329960,0watchdog/30-21swapper/320:54:383
329960,0watchdog/30-21swapper/320:48:423
329960,0watchdog/30-21swapper/320:36:023
329960,0watchdog/30-21swapper/320:29:383
329960,0watchdog/30-21swapper/320:25:103
329960,0watchdog/30-21swapper/320:19:423
329960,0watchdog/30-21swapper/319:18:543
29935060,0irq/33-eth0-tx-0-21swapper/019:04:410
29925060,0irq/32-eth0-rx-0-21swapper/323:45:513
29925060,0irq/32-eth0-rx-0-21swapper/322:54:173
29925060,0irq/32-eth0-rx-0-21swapper/322:45:533
29925060,0irq/32-eth0-rx-0-21swapper/321:49:373
29925060,0irq/32-eth0-rx-0-21swapper/321:37:183
29925060,0irq/32-eth0-rx-0-21swapper/319:54:313
169960,0watchdog/126666-21python19:54:421
149960,0migration/030112-21ntp_offset22:29:380
8315050,0irq/26-0000:00:30070-21crond22:14:182
8315050,0irq/26-0000:00:28403-21latency_hist23:09:172
339950,0migration/329528-21munin-node23:09:333
339950,0migration/322450-21kthreadcore20:14:373
339950,0migration/316125-21kthreadcore18:19:363
329950,0watchdog/37915-21iostat22:19:343
329950,0watchdog/323378-21cut19:09:393
329950,0watchdog/30-21swapper/323:39:343
329950,0watchdog/30-21swapper/323:14:503
329950,0watchdog/30-21swapper/322:39:263
329950,0watchdog/30-21swapper/322:24:183
329950,0watchdog/30-21swapper/321:09:343
329950,0watchdog/30-21swapper/320:59:383
329950,0watchdog/30-21swapper/320:04:223
329950,0watchdog/30-21swapper/319:59:223
329950,0watchdog/30-21swapper/319:49:303
329950,0watchdog/30-21swapper/319:46:423
329950,0watchdog/30-21swapper/319:39:583
329950,0watchdog/30-21swapper/319:29:303
329950,0watchdog/30-21swapper/319:04:393
329950,0watchdog/30-21swapper/318:59:313
329950,0watchdog/30-21swapper/318:54:393
329950,0watchdog/30-21swapper/318:49:193
329950,0watchdog/30-21swapper/318:45:033
329950,0watchdog/30-21swapper/318:39:273
329950,0watchdog/30-21swapper/318:38:313
329950,0watchdog/30-21swapper/318:33:073
329950,0watchdog/30-21swapper/318:28:073
29935050,0irq/33-eth0-tx-240722sleep019:34:320
29935050,0irq/33-eth0-tx-19428-21munin-node21:49:302
29935050,0irq/33-eth0-tx-0-21swapper/019:24:350
29925050,0irq/32-eth0-rx-12320-21ssh21:04:413
29925050,0irq/32-eth0-rx-0-21swapper/322:34:313
29925050,0irq/32-eth0-rx-0-21swapper/320:49:333
29925050,0irq/32-eth0-rx-0-21swapper/320:09:333
27150,0ktimersoftd/20-21swapper/219:24:182
259950,0migration/227416-21kthreadcore18:49:372
259950,0migration/226064-21kthreadcore19:54:382
259950,0migration/213349-21aten_r1power_po23:19:322
249950,0watchdog/214458-21kthreadcore21:44:382
179950,0migration/18507-21kthreadcore22:19:351
179950,0migration/18367-21pmu-power19:19:401
179950,0migration/18234-21sh20:27:091
179950,0migration/15762-21taskset18:59:211
179950,0migration/132518-21latency_hist22:34:181
179950,0migration/127991-21latency_hist21:54:181
179950,0migration/127600-21timerandwakeup20:54:451
179950,0migration/127405-21kthreadcore22:29:361
179950,0migration/120115-21kthreadcore22:44:371
179950,0migration/115324-21kthreadcore18:19:361
169950,0watchdog/120857-21grep20:34:381
169950,0watchdog/10-21swapper/120:49:381
149950,0migration/09620-21missed_timers20:44:370
149950,0migration/08557-21kthreadcore21:04:380
149950,0migration/0571-21kthreadcore19:39:380
149950,0migration/030416-21sh21:15:200
149950,0migration/029758-21kthreadcore18:29:360
149950,0migration/025769-21sendmail_mailqu21:49:410
149950,0migration/025760-21sh22:48:280
149950,0migration/024945-21kthreadcore22:09:380
149950,0migration/021999-21fschecks_time20:54:330
149950,0migration/020837-21kthreadcore22:24:370
149950,0migration/019439-21kthreadcore21:09:370
149950,0migration/016954-21fschecks_count18:44:320
149950,0migration/016333-21sh22:41:270
149950,0migration/01613-21taskset22:52:300
149950,0migration/013891-21if_eth018:19:340
149950,0migration/012915-21kthreadcore21:44:380
125799955,0cyclictest0-21swapper/221:55:112
125799955,0cyclictest0-21swapper/221:41:462
125799955,0cyclictest0-21swapper/221:30:352
125799955,0cyclictest0-21swapper/221:23:202
125799950,0cyclictest0-21swapper/223:27:242
125799950,0cyclictest0-21swapper/222:56:302
125799950,0cyclictest0-21swapper/222:19:372
125799950,0cyclictest0-21swapper/221:12:082
125719955,0cyclictest0-21swapper/122:43:451
125719954,0cyclictest19129-21sadc21:49:181
125719950,0cyclictest0-21swapper/123:11:301
125719950,0cyclictest0-21swapper/122:56:121
125719950,0cyclictest0-21swapper/121:38:191
125709955,0cyclictest0-21swapper/023:04:390
125709955,0cyclictest0-21swapper/020:23:410
125709954,0cyclictest23871-21snmpd20:17:300
125709950,0cyclictest0-21swapper/021:32:250
125709950,0cyclictest0-21swapper/021:26:160
29935040,0irq/33-eth0-tx-6508-21telnet23:34:302
29935040,0irq/33-eth0-tx-16578-21irqcore22:24:352
29935040,0irq/33-eth0-tx-0-21swapper/223:44:292
29935040,0irq/33-eth0-tx-0-21swapper/223:29:232
29935040,0irq/33-eth0-tx-0-21swapper/223:14:332
29935040,0irq/33-eth0-tx-0-21swapper/222:45:212
29935040,0irq/33-eth0-tx-0-21swapper/222:30:072
29935040,0irq/33-eth0-tx-0-21swapper/222:04:252
29935040,0irq/33-eth0-tx-0-21swapper/222:00:402
29935040,0irq/33-eth0-tx-0-21swapper/221:35:582
29935040,0irq/33-eth0-tx-0-21swapper/221:25:112
29935040,0irq/33-eth0-tx-0-21swapper/221:14:442
29935040,0irq/33-eth0-tx-0-21swapper/221:04:332
29935040,0irq/33-eth0-tx-0-21swapper/220:56:392
29935040,0irq/33-eth0-tx-0-21swapper/220:24:282
29935040,0irq/33-eth0-tx-0-21swapper/020:04:430
29935040,0irq/33-eth0-tx-0-21swapper/019:56:140
29935040,0irq/33-eth0-tx-0-21swapper/019:29:350
29935040,0irq/33-eth0-tx-0-21swapper/019:19:350
259940,0migration/29064-21unixbench_singl20:04:472
259940,0migration/29050-21kthreadcore20:44:362
259940,0migration/28459-21kthreadcore19:44:362
259940,0migration/231133-21kthreadcore19:14:412
259940,0migration/230946-21aten2.4-expect18:54:302
259940,0migration/22016-21kthreadcore19:39:382
259940,0migration/218155-21latency_hist20:34:182
259940,0migration/215024-21aten_r1power_fr23:39:272
259940,0migration/212935-21latency_hist19:04:182
259940,0migration/212786-21taskset18:19:272
259940,0migration/210663-21gltestperf20:09:342
249940,0watchdog/29696-21cpuspeed_turbos20:29:302
249940,0watchdog/227103-21ssh20:39:262
249940,0watchdog/220865-21perl19:09:352
249940,0watchdog/214605-21kthreadcore20:49:382
249940,0watchdog/20-21swapper/222:39:542
249940,0watchdog/20-21swapper/220:59:382
249940,0watchdog/20-21swapper/219:29:382
179940,0migration/15739-21kthreadcore20:44:351
179940,0migration/15130-21diskmemload23:31:021
179940,0migration/130254-21kthreadcore19:59:371
179940,0migration/127608-21fschecks_count20:39:311
179940,0migration/116647-21kthreadcore21:09:361
169940,0watchdog/14999-21sensors_fan19:39:421
169940,0watchdog/132348-21cut22:14:381
169940,0watchdog/132348-21cut22:14:381
169940,0watchdog/130457-21taskset23:45:421
169940,0watchdog/128119-21kthreadcore19:34:381
169940,0watchdog/127914-21ssh23:08:301
169940,0watchdog/123438-21ssh21:14:181
169940,0watchdog/120391-21ps22:06:261
169940,0watchdog/115122-21load18:39:391
169940,0watchdog/111507-21ssh22:00:381
169940,0watchdog/10-21swapper/123:23:221
169940,0watchdog/10-21swapper/123:01:501
169940,0watchdog/10-21swapper/121:04:381
169940,0watchdog/10-21swapper/119:09:231
149940,0migration/07843-21kthreadcore18:34:390
149940,0migration/02870-21kthreadcore20:24:350
149940,0migration/022025-21kthreadcore19:09:380
149940,0migration/019176-21fschecks_time20:34:350
149940,0migration/016559-21kthreadcore22:04:370
149940,0migration/014614-21taskset23:39:070
149940,0migration/011207-21kthreadcore19:44:370
149940,0migration/01100-21kthreadcore18:54:360
149940,0migration/010439-21taskset21:44:070
125799943,0cyclictest3599-21lldpd22:34:372
125799943,0cyclictest3599-21lldpd19:36:592
125799943,0cyclictest3599-21lldpd19:21:292
125799943,0cyclictest3599-21lldpd18:34:292
125799943,0cyclictest31557-21awk22:14:372
125799943,0cyclictest31557-21awk22:14:372
125799943,0cyclictest28761-21awk19:59:352
125799943,0cyclictest25279-21awk20:19:342
125799943,0cyclictest23871-21snmpd22:52:012
125799943,0cyclictest20996-21awk23:04:382
125799943,0cyclictest18055-21awk20:14:342
125799943,0cyclictest12493-21awk22:59:332
125719943,0cyclictest5811-21awk23:14:331
125719943,0cyclictest3599-21lldpd23:34:421
125719943,0cyclictest30916-21awk20:59:351
125719943,0cyclictest24727-21awk18:49:351
125719943,0cyclictest23871-21snmpd21:27:161
125719943,0cyclictest23871-21snmpd21:23:461
125719943,0cyclictest23871-21snmpd18:33:301
125719943,0cyclictest22925-21awk22:09:371
125719943,0cyclictest20601-21awk21:49:351
125719943,0cyclictest18338-21awk21:29:351
125719943,0cyclictest16126-21awk23:39:321
125709944,0cyclictest0-21swapper/021:54:480
125709943,0cyclictest6882-21awk18:59:340
125709943,0cyclictest3599-21lldpd23:40:450
125709943,0cyclictest3599-21lldpd23:03:380
125709943,0cyclictest3599-21lldpd21:23:030
125709943,0cyclictest3262-21irqbalance23:29:520
125709943,0cyclictest3262-21irqbalance23:24:420
125709943,0cyclictest3262-21irqbalance23:19:320
125709943,0cyclictest3262-21irqbalance23:14:320
125709943,0cyclictest3262-21irqbalance23:11:120
125709943,0cyclictest3262-21irqbalance22:54:420
125709943,0cyclictest3262-21irqbalance22:19:420
125709943,0cyclictest3262-21irqbalance22:14:320
125709943,0cyclictest3262-21irqbalance22:14:320
125709943,0cyclictest3262-21irqbalance22:00:020
125709943,0cyclictest30795-21awk20:59:340
125709943,0cyclictest28565-21awk19:14:390
125709943,0cyclictest24543-21awk23:44:340
125709943,0cyclictest13567-21awk20:49:360
125709943,0cyclictest10324-21awk18:39:350
29935030,0irq/33-eth0-tx-0-21swapper/019:59:310
29935030,0irq/33-eth0-tx-0-21swapper/018:24:390
259930,0migration/222862-21timerwakeupswit18:44:442
249930,0watchdog/20-21swapper/218:29:192
249930,0watchdog/20-21swapper/218:24:232
169930,0watchdog/18189-21expr21:39:381
169930,0watchdog/123965-21diskmemload22:24:301
169930,0watchdog/123965-21diskmemload20:22:141
169930,0watchdog/10-21swapper/120:14:341
169930,0watchdog/10-21swapper/119:29:381
169930,0watchdog/10-21swapper/118:34:391
149930,0migration/016459-21runrttasks20:30:320
125799933,0cyclictest3599-21lldpd18:39:292
125799932,0cyclictest7556-21cut18:59:362
125799931,0cyclictest17269-21kthreadcore19:49:372
125719933,0cyclictest3599-21lldpd19:48:311
125719933,0cyclictest14339-21awk19:49:341
125719933,0cyclictest10793-21awk20:09:351
125719933,0cyclictest0-21swapper/118:24:331
125719932,0cyclictest9229-21cat20:29:181
125719932,0cyclictest32425-21irqstats18:54:351
125719932,0cyclictest27465-21awk22:49:341
125719932,0cyclictest22922-21awk23:24:341
125719932,0cyclictest20508-21grep18:44:371
125719932,0cyclictest14146-21awk19:04:361
125719932,0cyclictest1199-21sensors19:14:451
125719932,0cyclictest10633-21awk19:24:351
125719930,0cyclictest3209-21perl20:04:351
125709932,0cyclictest5335-21kthreadcore22:34:380
125709932,0cyclictest32485-21grep21:34:450
125709932,0cyclictest10886-21awk20:09:360
125709932,0cyclictest0-21swapper/018:49:190
125709930,0cyclictest14069-21tune2fs19:49:320
125709920,0cyclictest0-21swapper/020:39:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional