You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-14 - 02:32
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Sun Sep 14, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2670424031,0sleep20-21swapper/218:24:022
2659423728,0sleep10-21swapper/118:22:451
2654823725,0sleep00-21swapper/018:22:160
2645122925,0sleep30-21swapper/318:21:073
329970,0watchdog/30-21swapper/321:40:493
29925070,0irq/32-eth0-rx-14560-21gltestperf21:44:213
29925070,0irq/32-eth0-rx-0-21swapper/320:43:213
8315060,0irq/26-0000:00:21780-21unixbench_multi23:39:332
329960,0watchdog/39400-21grep18:34:173
329960,0watchdog/3533-21mdstat19:09:253
329960,0watchdog/326321-21kthreadcore20:34:253
329960,0watchdog/30-21swapper/323:50:013
329960,0watchdog/30-21swapper/323:43:213
329960,0watchdog/30-21swapper/323:14:053
329960,0watchdog/30-21swapper/323:04:213
329960,0watchdog/30-21swapper/323:01:053
329960,0watchdog/30-21swapper/322:53:573
329960,0watchdog/30-21swapper/322:29:333
329960,0watchdog/30-21swapper/322:26:013
329960,0watchdog/30-21swapper/322:17:133
329960,0watchdog/30-21swapper/321:49:493
329960,0watchdog/30-21swapper/320:08:333
329960,0watchdog/30-21swapper/318:57:413
29925060,0irq/32-eth0-rx-9313-21kthreadcore21:59:223
29925060,0irq/32-eth0-rx-24922-21if_err_eth122:09:203
29925060,0irq/32-eth0-rx-13884-21aten_r1power_fr23:19:203
29925060,0irq/32-eth0-rx-0-21swapper/323:44:463
29925060,0irq/32-eth0-rx-0-21swapper/323:33:293
29925060,0irq/32-eth0-rx-0-21swapper/323:12:543
29925060,0irq/32-eth0-rx-0-21swapper/322:57:323
29925060,0irq/32-eth0-rx-0-21swapper/322:57:323
29925060,0irq/32-eth0-rx-0-21swapper/322:44:183
29925060,0irq/32-eth0-rx-0-21swapper/322:39:583
29925060,0irq/32-eth0-rx-0-21swapper/322:35:023
29925060,0irq/32-eth0-rx-0-21swapper/322:06:183
29925060,0irq/32-eth0-rx-0-21swapper/321:57:343
29925060,0irq/32-eth0-rx-0-21swapper/321:19:173
29925060,0irq/32-eth0-rx-0-21swapper/321:14:363
29925060,0irq/32-eth0-rx-0-21swapper/321:13:083
29925060,0irq/32-eth0-rx-0-21swapper/321:07:383
29925060,0irq/32-eth0-rx-0-21swapper/320:54:123
29925060,0irq/32-eth0-rx-0-21swapper/320:49:333
29925060,0irq/32-eth0-rx-0-21swapper/320:24:073
29925060,0irq/32-eth0-rx-0-21swapper/319:44:253
267789966,0cyclictest5784-21diskmemload20:28:482
267789960,0cyclictest0-21swapper/220:58:172
149960,0migration/016848-21sh22:24:040
149960,0migration/012930-21gltestperf22:59:220
4150,0ktimersoftd/00-21swapper/020:51:370
4150,0ktimersoftd/00-21swapper/019:29:270
339950,0migration/310951-21kthreadcore22:19:243
329950,0watchdog/32913-21if_eth018:29:213
329950,0watchdog/317543-21perl20:09:213
329950,0watchdog/30-21swapper/323:34:173
329950,0watchdog/30-21swapper/321:34:533
329950,0watchdog/30-21swapper/321:29:093
329950,0watchdog/30-21swapper/321:24:053
329950,0watchdog/30-21swapper/320:59:053
329950,0watchdog/30-21swapper/320:46:013
329950,0watchdog/30-21swapper/320:29:133
329950,0watchdog/30-21swapper/320:19:133
329950,0watchdog/30-21swapper/320:14:253
329950,0watchdog/30-21swapper/319:54:133
329950,0watchdog/30-21swapper/319:49:133
329950,0watchdog/30-21swapper/319:39:173
329950,0watchdog/30-21swapper/319:34:533
329950,0watchdog/30-21swapper/319:29:213
329950,0watchdog/30-21swapper/319:24:173
329950,0watchdog/30-21swapper/319:14:053
329950,0watchdog/30-21swapper/319:05:053
329950,0watchdog/30-21swapper/318:59:133
329950,0watchdog/30-21swapper/318:50:013
329950,0watchdog/30-21swapper/318:44:133
329950,0watchdog/30-21swapper/318:39:173
329950,0watchdog/30-21swapper/318:25:373
29935050,0irq/33-eth0-tx-8893-21ssh21:00:282
29935050,0irq/33-eth0-tx-5137-21munin-node21:39:362
29935050,0irq/33-eth0-tx-4385-21ssh21:37:472
29935050,0irq/33-eth0-tx-22922-21perl21:49:192
29935050,0irq/33-eth0-tx-13816-21munin-node23:19:232
29935050,0irq/33-eth0-tx-0-21swapper/223:00:012
29935050,0irq/33-eth0-tx-0-21swapper/222:42:152
29935050,0irq/33-eth0-tx-0-21swapper/222:38:062
29935050,0irq/33-eth0-tx-0-21swapper/222:10:202
29935050,0irq/33-eth0-tx-0-21swapper/221:45:052
29935050,0irq/33-eth0-tx-0-21swapper/221:24:062
29935050,0irq/33-eth0-tx-0-21swapper/221:16:192
29935050,0irq/33-eth0-tx-0-21swapper/220:51:422
29935050,0irq/33-eth0-tx-0-21swapper/220:29:162
29935050,0irq/33-eth0-tx-0-21swapper/020:19:260
29935050,0irq/33-eth0-tx-0-21swapper/019:54:180
29935050,0irq/33-eth0-tx-0-21swapper/018:54:060
29925050,0irq/32-eth0-rx-6044-21cut19:59:253
29925050,0irq/32-eth0-rx-0-21swapper/323:24:253
29925050,0irq/32-eth0-rx-0-21swapper/319:19:263
27150,0ktimersoftd/229597-21strings19:29:272
267789955,0cyclictest0-21swapper/223:09:562
267789955,0cyclictest0-21swapper/222:49:262
267789955,0cyclictest0-21swapper/220:45:402
267779955,0cyclictest5784-21diskmemload21:30:031
267779955,0cyclictest0-21swapper/123:46:531
267779955,0cyclictest0-21swapper/123:21:301
267779950,0cyclictest0-21swapper/122:34:411
267779950,0cyclictest0-21swapper/122:06:471
267779950,0cyclictest0-21swapper/121:36:491
267769955,0cyclictest2546-21nvme18:49:210
267769955,0cyclictest0-21swapper/023:11:370
267769950,0cyclictest0-21swapper/023:19:110
267769950,0cyclictest0-21swapper/021:03:460
259950,0migration/28178-21kthreadcore19:39:232
259950,0migration/28058-21kthreadcore23:14:242
259950,0migration/2356-21gltestperf23:49:212
259950,0migration/225836-21cpuspeed_turbos22:29:192
259950,0migration/217987-21fschecks_time22:24:192
259950,0migration/217677-21sh21:06:212
259950,0migration/213803-21idleruntime18:59:192
259950,0migration/210678-21sh22:56:002
259950,0migration/210678-21sh22:56:002
249950,0watchdog/27000-21ssh22:15:012
249950,0watchdog/23922-21diskstats21:19:212
249950,0watchdog/223173-21kthreadcore21:09:252
249950,0watchdog/213440-21grep22:19:252
179950,0migration/16290-21idleruntime23:14:211
179950,0migration/125757-21sh22:45:181
179950,0migration/124194-21munin-node20:14:221
179950,0migration/123788-21latency_hist20:14:041
179950,0migration/123268-21latency_hist20:34:041
179950,0migration/114775-21munin-node21:44:231
179950,0migration/11410-21timerwakeupswit21:14:331
169950,0watchdog/130823-21ssh22:10:171
169950,0watchdog/119321-21ls19:44:291
149950,0migration/05467-21latency23:49:260
149950,0migration/0496-21kthreadcore21:54:240
149950,0migration/04648-21kthreadcore22:14:270
149950,0migration/032205-21sh22:29:320
149950,0migration/032099-21latency_hist20:39:050
149950,0migration/032040-21kthreadcore21:34:240
149950,0migration/031596-21fschecks_time23:29:210
149950,0migration/030357-21sh23:44:490
149950,0migration/030350-21taskset23:28:360
149950,0migration/028529-21iostat22:49:210
149950,0migration/027174-21kthreadcore18:44:260
149950,0migration/024934-21kthreadcore19:04:230
149950,0migration/023644-21kthreadcore22:44:230
149950,0migration/022762-21fschecks_count21:49:180
149950,0migration/021680-21runrttasks22:04:300
149950,0migration/020743-21kthreadcore21:09:240
149950,0migration/020707-21cpuspeed_turbos23:04:200
149950,0migration/020666-21missed_timers20:29:240
149950,0migration/016287-21taskset20:06:280
149950,0migration/01617-21munin-node21:14:350
149950,0migration/015867-21fschecks_count23:39:200
149950,0migration/013491-21kthreadcore18:34:230
149950,0migration/012658-21fschecks_time21:24:190
149950,0migration/012563-21kthreadcore20:44:260
149950,0migration/011393-21idleruntime21:04:220
29935040,0irq/33-eth0-tx-20441-21ssh23:04:142
29935040,0irq/33-eth0-tx-0-21swapper/223:44:042
29935040,0irq/33-eth0-tx-0-21swapper/223:29:312
29935040,0irq/33-eth0-tx-0-21swapper/221:29:332
29935040,0irq/33-eth0-tx-0-21swapper/220:39:252
29935040,0irq/33-eth0-tx-0-21swapper/020:24:140
29935040,0irq/33-eth0-tx-0-21swapper/019:24:180
29935040,0irq/33-eth0-tx-0-21swapper/019:14:250
267789943,0cyclictest3599-21lldpd21:55:262
267789943,0cyclictest3599-21lldpd20:04:102
267789943,0cyclictest3599-21lldpd19:53:402
267789943,0cyclictest3599-21lldpd19:19:102
267789943,0cyclictest3599-21lldpd18:49:102
267789943,0cyclictest3599-21lldpd18:33:102
267789943,0cyclictest3599-21lldpd18:26:102
267789943,0cyclictest23871-21snmpd20:15:092
267789943,0cyclictest23871-21snmpd20:00:092
267789943,0cyclictest23871-21snmpd19:44:092
267789943,0cyclictest23871-21snmpd19:35:092
267789943,0cyclictest23871-21snmpd19:06:392
267789943,0cyclictest23871-21snmpd18:56:392
267789943,0cyclictest23871-21snmpd18:44:092
267789943,0cyclictest23871-21snmpd18:39:092
267789940,0cyclictest0-21swapper/221:59:122
267789940,0cyclictest0-21swapper/219:14:192
267779944,0cyclictest0-21swapper/120:39:171
267779944,0cyclictest0-21swapper/119:24:231
267779943,0cyclictest9808-21awk22:19:221
267779943,0cyclictest8000-21awk21:59:201
267779943,0cyclictest3599-21lldpd21:39:251
267779943,0cyclictest32053-21awk19:34:211
267779943,0cyclictest31838-21awk21:54:221
267779943,0cyclictest28363-21awk22:49:201
267779943,0cyclictest23871-21snmpd20:06:391
267779943,0cyclictest23871-21snmpd19:52:391
267779943,0cyclictest13862-21awk18:59:201
267779943,0cyclictest13049-21awk21:24:211
267769944,0cyclictest2546-21nvme23:36:500
267769944,0cyclictest2546-21nvme22:39:120
267769944,0cyclictest2546-21nvme21:45:180
267769944,0cyclictest2546-21nvme19:12:150
267769944,0cyclictest0-21swapper/023:14:090
267769944,0cyclictest0-21swapper/020:09:320
267769943,0cyclictest6301-21awk21:39:270
267769943,0cyclictest13976-21awk18:59:210
259940,0migration/24460-21kthreadcore20:19:242
259940,0migration/232129-21kthreadcore19:54:272
259940,0migration/224131-21fschecks_count20:34:212
259940,0migration/222393-21needreboot19:24:212
259940,0migration/216301-21idleruntime22:04:212
259940,0migration/211344-21kthreadcore23:34:252
249940,0watchdog/29403-21cpuspeed_turbos18:34:172
249940,0watchdog/20-21swapper/223:24:212
249940,0watchdog/20-21swapper/222:44:332
249940,0watchdog/20-21swapper/220:09:252
249940,0watchdog/20-21swapper/219:09:252
179940,0migration/1921-21sendmail_mailqu19:09:281
179940,0migration/18157-21taskset22:18:221
179940,0migration/16382-21kthreadcore18:29:241
179940,0migration/15927-21kthreadcore22:54:221
179940,0migration/15927-21kthreadcore22:54:221
179940,0migration/131809-21idleruntime23:29:231
179940,0migration/128884-21kthreadcore20:54:241
179940,0migration/12866-21kthreadcore20:19:231
179940,0migration/126909-21latency_hist18:24:041
179940,0migration/123956-21cpuspeed_turbos19:29:211
179940,0migration/122943-21fschecks_count23:24:211
179940,0migration/122623-21kthreadcore22:24:241
179940,0migration/115529-21cpuspeed_turbos23:39:191
179940,0migration/11127-21kthreadcore18:49:251
179940,0migration/111077-21fschecks_time21:04:201
169940,0watchdog/19401-21tr18:34:171
169940,0watchdog/19203-21ssh21:01:211
169940,0watchdog/18996-21grep20:44:211
169940,0watchdog/14425-21ssh23:12:331
169940,0watchdog/13923-21ssh21:19:211
169940,0watchdog/128822-21ps19:54:251
169940,0watchdog/125596-21munin-node22:29:331
169940,0watchdog/1231812chrt18:43:251
169940,0watchdog/118224-21ssh22:43:131
169940,0watchdog/110470-21iostat19:19:251
169940,0watchdog/110442-21kthreadcore23:34:251
169940,0watchdog/10-21swapper/121:10:051
169940,0watchdog/10-21swapper/120:34:251
169940,0watchdog/10-21swapper/120:24:171
169940,0watchdog/10-21swapper/119:59:251
169940,0watchdog/10-21swapper/118:44:251
149940,0migration/0981-21timerwakeupswit18:24:300
149940,0migration/03804-21irqrtprio19:59:240
149940,0migration/026213-21ntp_offset19:49:270
149940,0migration/018300-21if_eth1.222:24:210
267779932,0cyclictest6667-21awk18:54:211
267779932,0cyclictest3599-21lldpd19:39:091
267779932,0cyclictest23871-21snmpd19:17:391
267779932,0cyclictest21661-21awk23:04:231
267779932,0cyclictest21374-21sed19:04:221
267779932,0cyclictest18352-21awk20:49:211
267769933,0cyclictest26747-21kworker/0:118:39:100
267769933,0cyclictest26747-21kworker/0:118:29:100
267769933,0cyclictest26618-21kworker/0:121:29:250
267769933,0cyclictest26618-21kworker/0:121:19:090
267769933,0cyclictest24642-21kworker/0:020:34:250
267769933,0cyclictest24642-21kworker/0:020:14:150
267769933,0cyclictest24642-21kworker/0:019:45:250
267769933,0cyclictest24642-21kworker/0:019:35:010
267769932,0cyclictest4553-21awk22:54:190
267769932,0cyclictest4553-21awk22:54:190
267769932,0cyclictest29242-21cut20:54:240
267769932,0cyclictest29173-21cut22:09:230
267769932,0cyclictest15483-21sensors19:19:310
267769930,0cyclictest41ktimersoftd/021:59:250
267769930,0cyclictest2522-21perl22:34:210
267769930,0cyclictest0-21swapper/019:39:190
179930,0migration/115924-21kthreadcore22:59:261
169930,0watchdog/10-21swapper/123:49:291
169930,0watchdog/10-21swapper/121:49:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional