You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-21 - 10:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Mon Apr 21, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1847723829,0sleep10-21swapper/118:29:481
1865023727,0sleep20-21swapper/218:31:492
1854723625,0sleep00-21swapper/018:30:350
1841423228,0sleep30-21swapper/318:29:063
18818991414,0cyclictest22024-21sensors20:43:211
29925070,0irq/32-eth0-rx-0-21swapper/320:33:153
8315060,0irq/26-0000:00:25264-21munin-run23:07:572
4160,0ktimersoftd/00-21swapper/023:07:140
339960,0migration/39880-21sh20:58:363
339960,0migration/35209-21sh22:04:173
339960,0migration/3414-21sh23:58:483
339960,0migration/329080-21sh21:59:303
329960,0watchdog/332453-21netstat19:23:183
329960,0watchdog/3299250irq/32-eth0-rx-23:35:343
329960,0watchdog/315208-21sh20:42:263
329960,0watchdog/30-21swapper/323:48:213
329960,0watchdog/30-21swapper/323:47:303
329960,0watchdog/30-21swapper/323:24:383
329960,0watchdog/30-21swapper/323:13:263
329960,0watchdog/30-21swapper/323:08:503
329960,0watchdog/30-21swapper/322:58:383
329960,0watchdog/30-21swapper/322:54:183
329960,0watchdog/30-21swapper/322:45:103
329960,0watchdog/30-21swapper/322:38:103
329960,0watchdog/30-21swapper/322:29:503
329960,0watchdog/30-21swapper/322:23:343
329960,0watchdog/30-21swapper/322:19:143
329960,0watchdog/30-21swapper/322:13:543
329960,0watchdog/30-21swapper/321:35:183
329960,0watchdog/30-21swapper/321:29:103
329960,0watchdog/30-21swapper/321:13:103
329960,0watchdog/30-21swapper/321:07:583
329960,0watchdog/30-21swapper/320:42:583
329960,0watchdog/30-21swapper/319:32:583
29935060,0irq/33-eth0-tx-30271-21munin-node19:48:100
29925060,0irq/32-eth0-rx-17637-21tune2fs23:03:113
29925060,0irq/32-eth0-rx-11665-21wc21:02:573
29925060,0irq/32-eth0-rx-0-21swapper/323:55:513
29925060,0irq/32-eth0-rx-0-21swapper/323:38:473
29925060,0irq/32-eth0-rx-0-21swapper/323:30:273
29925060,0irq/32-eth0-rx-0-21swapper/321:54:053
29925060,0irq/32-eth0-rx-0-21swapper/320:48:163
29925060,0irq/32-eth0-rx-0-21swapper/319:43:123
188179960,0cyclictest0-21swapper/018:33:250
149960,0migration/07345-21sh22:56:040
149960,0migration/03185-21sh23:17:150
339950,0migration/36353-21kthreadcore21:43:163
339950,0migration/33797-21aten_r1power_cu23:18:093
339950,0migration/319358-21munin-node19:18:113
329950,0watchdog/331251-21expr21:18:143
329950,0watchdog/313784-21grep19:13:183
329950,0watchdog/30-21swapper/322:48:023
329950,0watchdog/30-21swapper/322:32:583
329950,0watchdog/30-21swapper/321:48:023
329950,0watchdog/30-21swapper/321:37:583
329950,0watchdog/30-21swapper/321:23:103
329950,0watchdog/30-21swapper/320:28:463
329950,0watchdog/30-21swapper/320:23:143
329950,0watchdog/30-21swapper/320:18:423
329950,0watchdog/30-21swapper/320:15:023
329950,0watchdog/30-21swapper/320:09:503
329950,0watchdog/30-21swapper/320:03:543
329950,0watchdog/30-21swapper/319:58:583
329950,0watchdog/30-21swapper/319:47:583
329950,0watchdog/30-21swapper/319:39:423
329950,0watchdog/30-21swapper/319:28:383
329950,0watchdog/30-21swapper/319:08:103
329950,0watchdog/30-21swapper/319:08:103
329950,0watchdog/30-21swapper/319:02:583
329950,0watchdog/30-21swapper/318:59:503
329950,0watchdog/30-21swapper/318:53:183
329950,0watchdog/30-21swapper/318:49:263
329950,0watchdog/30-21swapper/318:43:263
329950,0watchdog/30-21swapper/318:40:143
329950,0watchdog/30-21swapper/318:33:063
29935050,0irq/33-eth0-tx-6952-21munin-node20:38:272
29935050,0irq/33-eth0-tx-6508-40yum-updatesd-he22:07:572
29935050,0irq/33-eth0-tx-31501-21munin-node23:38:162
29935050,0irq/33-eth0-tx-24690-21ssh20:48:532
29935050,0irq/33-eth0-tx-14313-21ssh22:58:172
29935050,0irq/33-eth0-tx-12817-21munin-node22:38:172
29935050,0irq/33-eth0-tx-0-21swapper/223:51:462
29935050,0irq/33-eth0-tx-0-21swapper/223:23:002
29935050,0irq/33-eth0-tx-0-21swapper/200:01:252
29935050,0irq/33-eth0-tx-0-21swapper/020:33:070
29935050,0irq/33-eth0-tx-0-21swapper/020:08:170
29935050,0irq/33-eth0-tx-0-21swapper/020:03:200
29925050,0irq/32-eth0-rx-6508-40yum-updatesd-he22:07:593
29925050,0irq/32-eth0-rx-28396-21kthreadcore20:53:113
29925050,0irq/32-eth0-rx-0-21swapper/319:53:123
259950,0migration/28822-21aten_r1power_en22:13:042
259950,0migration/26799-21kthreadcore22:33:152
259950,0migration/25903-21sh21:20:052
259950,0migration/23740-21idleruntime21:43:132
259950,0migration/229915-21kthreadcore18:38:182
259950,0migration/228148-21kthreadcore19:23:162
259950,0migration/224234-21latency_hist20:47:572
259950,0migration/223967-21gltestperf19:43:142
259950,0migration/218842-21hddtemp_smartct23:53:072
259950,0migration/214674-21sh23:44:272
249950,0watchdog/29353-21kthreadcore21:23:142
249950,0watchdog/223945-21ntp_offset21:33:142
249950,0watchdog/215303-21kthreadcore19:58:182
19150,0ktimersoftd/10-21swapper/122:04:101
188199950,0cyclictest0-21swapper/221:52:592
188179955,0cyclictest0-21swapper/021:30:470
188179950,0cyclictest0-21swapper/022:50:430
188179950,0cyclictest0-21swapper/022:40:180
188179950,0cyclictest0-21swapper/019:07:590
188179950,0cyclictest0-21swapper/019:07:590
179950,0migration/19434-21fschecks_time19:33:201
179950,0migration/17496-21sh22:10:041
179950,0migration/1503-21kthreadcore19:03:211
179950,0migration/1468-21sendmail_mailqu20:08:211
179950,0migration/130345-21sh21:17:071
179950,0migration/121785-21needreboot19:38:161
179950,0migration/118802-21switchtime23:23:251
179950,0migration/115754-21kthreadcore22:38:171
179950,0migration/115167-21switchtime22:13:171
169950,0watchdog/18304-21munin-node22:58:141
169950,0watchdog/10-21swapper/120:03:181
149950,0migration/09921-21sh20:58:420
149950,0migration/07461-21kthreadcore21:43:170
149950,0migration/06777-21sh21:22:310
149950,0migration/05115-21irqcore23:18:150
149950,0migration/0359-21kthreadcore22:03:170
149950,0migration/032339-21latency21:38:160
149950,0migration/030444-21taskset23:35:490
149950,0migration/030273-21kthreadcore23:58:160
149950,0migration/029930-21aten_r1power_cu19:03:150
149950,0migration/025641-21kthreadcore20:28:150
149950,0migration/023259-21gltestperf21:13:070
149950,0migration/022988-21kthreadcore21:33:120
149950,0migration/022523-21taskset19:39:370
149950,0migration/022363-21df_abs21:58:130
149950,0migration/022100-21kthreadcore19:18:180
149950,0migration/021431-21kthreadcore20:23:140
149950,0migration/019881-21kthreadcore23:53:110
149950,0migration/019140-21taskset20:01:020
149950,0migration/016078-21taskset23:48:090
149950,0migration/015540-21aten_r1power_cu18:53:140
149950,0migration/015447-21sh21:27:300
149950,0migration/013796-21fschecks_count21:53:070
149950,0migration/011344-21kthreadcore20:18:160
149950,0migration/010333-21munin-node22:33:180
8315040,0irq/26-0000:00:6724-21latency_hist22:07:572
8315040,0irq/26-0000:00:0-21swapper/221:37:572
29935040,0irq/33-eth0-tx-21905-21ssh21:57:582
29935040,0irq/33-eth0-tx-15867-21ssh21:28:042
29935040,0irq/33-eth0-tx-0-21swapper/223:33:482
29935040,0irq/33-eth0-tx-0-21swapper/223:28:042
29935040,0irq/33-eth0-tx-0-21swapper/223:13:122
29935040,0irq/33-eth0-tx-0-21swapper/222:54:242
29935040,0irq/33-eth0-tx-0-21swapper/222:47:582
29935040,0irq/33-eth0-tx-0-21swapper/222:44:582
29935040,0irq/33-eth0-tx-0-21swapper/222:28:242
29935040,0irq/33-eth0-tx-0-21swapper/222:18:142
29935040,0irq/33-eth0-tx-0-21swapper/221:48:012
29935040,0irq/33-eth0-tx-0-21swapper/221:07:592
29935040,0irq/33-eth0-tx-0-21swapper/220:55:272
29935040,0irq/33-eth0-tx-0-21swapper/018:58:210
259940,0migration/28115-21kthreadcore19:08:152
259940,0migration/28115-21kthreadcore19:08:152
259940,0migration/24759-21kthreadcore20:58:172
259940,0migration/24730-21kthreadcore18:43:172
259940,0migration/23956-21kthreadcore20:13:202
259940,0migration/232670-21kthreadcore19:03:212
259940,0migration/232583-21kthreadcore19:48:142
259940,0migration/232041-21kthreadcore20:08:172
259940,0migration/211744-21sh21:03:042
249940,0watchdog/23752-21rm23:18:062
249940,0watchdog/231662-21sh20:33:142
249940,0watchdog/228710-21cut22:23:182
249940,0watchdog/20-21swapper/219:53:142
188199943,0cyclictest9999-21awk20:18:142
188199943,0cyclictest9415-21awk18:48:152
188199943,0cyclictest3599-21lldpd18:36:222
188199943,0cyclictest29688-21sadc19:02:582
188199943,0cyclictest23871-21snmpd23:08:262
188199943,0cyclictest23871-21snmpd20:29:252
188199943,0cyclictest23871-21snmpd19:22:402
188199943,0cyclictest23871-21snmpd18:53:102
188199943,0cyclictest23401-21awk21:13:092
188199943,0cyclictest17106-21awk20:23:122
188199943,0cyclictest16998-21awk19:38:132
188189944,0cyclictest0-21swapper/121:22:451
188189943,0cyclictest6291-21awk19:53:171
188189943,0cyclictest3810-21awk21:43:141
188189943,0cyclictest3599-21lldpd23:21:001
188189943,0cyclictest3599-21lldpd22:28:261
188189943,0cyclictest31321-21awk19:48:121
188189943,0cyclictest27548-21awk21:38:131
188189943,0cyclictest23871-21snmpd19:46:551
188189943,0cyclictest2182-21awk18:43:151
188189943,0cyclictest20326-21awk19:18:151
188189943,0cyclictest17965-21awk23:03:131
188189940,0cyclictest0-21swapper/122:49:461
188179943,0cyclictest3599-21lldpd19:56:210
188179943,0cyclictest28365-21kworker/0:123:11:060
188179943,0cyclictest28365-21kworker/0:122:14:120
188179943,0cyclictest23871-21snmpd23:31:110
188179941,0cyclictest0-21swapper/019:33:170
188179940,0cyclictest0-21swapper/020:43:290
179940,0migration/19182-21fschecks_time18:48:131
179940,0migration/18087-21latency_hist22:57:571
179940,0migration/17745-21timerandwakeup19:28:231
179940,0migration/16728-21kthreadcore22:33:151
179940,0migration/132693-21sensors_fan19:23:211
179940,0migration/132679-21ntp_kernel_err20:53:131
179940,0migration/128251-21kthreadcore22:23:181
179940,0migration/121116-21kthreadcore18:33:161
179940,0migration/120039-21runrttasks21:53:191
179940,0migration/11492-21latency_hist20:12:571
179940,0migration/112759-21sh21:52:081
169940,0watchdog/16194-21irqcore19:08:141
169940,0watchdog/16194-21irqcore19:08:141
169940,0watchdog/128061-21grep23:13:141
169940,0watchdog/123818-21cat22:18:261
169940,0watchdog/121125-21ssh21:10:221
169940,0watchdog/119981-21grep20:23:141
169940,0watchdog/119069-21chrt23:53:091
169940,0watchdog/1187192sleep119:14:101
169940,0watchdog/117855-21ssh21:03:181
169940,0watchdog/117317-21ssh23:52:011
169940,0watchdog/10-21swapper/123:38:181
169940,0watchdog/10-21swapper/121:33:141
169940,0watchdog/10-21swapper/120:58:181
169940,0watchdog/10-21swapper/119:58:141
149940,0migration/09602-21irqcore22:58:150
149940,0migration/06855-21latency_hist20:37:570
149940,0migration/028933-21missed_timers19:43:190
149940,0migration/020942-21runrttasks21:09:500
149940,0migration/015033-21sh23:45:360
149940,0migration/011535-21kthreadcore18:48:170
8315030,0irq/26-0000:00:0-21swapper/219:12:582
29935030,0irq/33-eth0-tx-0-21swapper/019:13:120
249930,0watchdog/28000-21runrttasks19:28:262
249930,0watchdog/20-21swapper/220:03:182
188199932,0cyclictest23871-21snmpd19:36:102
188189932,0cyclictest8147-21awk20:38:181
188189932,0cyclictest26250-21munin-node18:38:131
188189932,0cyclictest24500-21grep18:58:201
188189932,0cyclictest23993-21awk23:33:091
188189932,0cyclictest19626-21grep18:53:211
188189932,0cyclictest15046-21sensors20:18:201
188189930,0cyclictest7179-21sh21:23:021
188189930,0cyclictest29928-21runrttasks20:29:331
188189930,0cyclictest25711-21sh23:08:391
188189930,0cyclictest22552-21munin-node22:43:151
188179933,0cyclictest9471-21kworker/0:221:48:040
188179933,0cyclictest30750-21kworker/0:020:52:580
188179933,0cyclictest30750-21kworker/0:020:48:000
188179933,0cyclictest30750-21kworker/0:020:13:140
188179933,0cyclictest30750-21kworker/0:019:28:100
188179933,0cyclictest30750-21kworker/0:018:43:140
188179933,0cyclictest28365-21kworker/0:123:23:020
188179933,0cyclictest28365-21kworker/0:122:43:060
188179933,0cyclictest28365-21kworker/0:122:28:220
188179933,0cyclictest28365-21kworker/0:122:23:000
188179933,0cyclictest28365-21kworker/0:122:18:140
188179933,0cyclictest28365-21kworker/0:122:08:220
188179932,0cyclictest31767-21expr23:38:120
188179932,0cyclictest23871-21snmpd18:41:250
188179932,0cyclictest18077-21sensors21:03:200
188179930,0cyclictest0-21swapper/019:23:110
179930,0migration/127378-21irqcore23:58:151
169930,0watchdog/10-21swapper/123:43:141
169930,0watchdog/10-21swapper/123:29:261
169930,0watchdog/10-21swapper/121:58:141
169930,0watchdog/10-21swapper/121:28:061
169930,0watchdog/10-21swapper/120:48:101
169930,0watchdog/10-21swapper/120:33:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional