You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-15 - 08:35
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3.osadl.org (updated Wed Oct 15, 2025 00:46:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
122501160irq/42-ahci0-21swapper/219:08:234
22077211197sleep100-21swapper/1019:05:432
22087210894sleep50-21swapper/519:05:497
122501070irq/42-ahci0-21swapper/819:08:2810
22269210492sleep10-21swapper/119:08:101
22211210190sleep40-21swapper/419:07:256
12250920irq/42-ahci0-21swapper/719:06:269
3066450810irq/53-eth0-tx-0-21swapper/319:05:335
2224126955sleep60-21swapper/619:07:508
2226626657sleep110-21swapper/1119:08:073
2212526556sleep90-21swapper/919:06:1811
2209626157sleep00-21swapper/019:05:560
2255499320cyclictest0-21swapper/022:09:020
2255499320cyclictest0-21swapper/022:09:020
2259599192cyclictest304052sleep1023:05:582
2255499190cyclictest0-21swapper/021:13:070
2259599181cyclictest12250irq/42-ahci21:46:232
2259599181cyclictest12250irq/42-ahci20:10:012
2259599180cyclictest310332sleep1023:54:262
2259599180cyclictest12250irq/42-ahci19:50:272
2259499181cyclictest0-21swapper/923:50:1611
2259499181cyclictest0-21swapper/922:45:2211
22593991816cyclictest9757-21munin-run20:45:0210
22593991816cyclictest30302-21snmp_easybox.os21:35:2610
22593991816cyclictest2787-21snmp_rack3slot922:30:1310
22593991816cyclictest26682-21snmp_easybox.os19:15:0010
22593991816cyclictest26365-21snmp_easybox.os23:45:2810
22593991816cyclictest24832-21memory21:20:5510
22593991816cyclictest24782-21perl22:15:2210
22593991816cyclictest20563-21munin-run00:25:0010
22593991816cyclictest16211-21snmp_rack3slot923:30:4910
22593991816cyclictest10032-21perl21:55:2310
22595991716cyclictest1357-21python23:55:272
22595991716cyclictest0-21swapper/1023:10:172
2259599170cyclictest95352sleep1000:08:152
2259599170cyclictest95102sleep1020:41:402
2259599170cyclictest90032sleep1023:21:352
2259599170cyclictest85582sleep1020:36:272
2259599170cyclictest85492sleep1022:36:062
2259599170cyclictest62052sleep1020:31:262
2259599170cyclictest52262sleep1021:50:122
2259599170cyclictest51842sleep1022:34:542
2259599170cyclictest51752sleep1020:25:152
2259599170cyclictest48742sleep1000:00:262
2259599170cyclictest323232sleep1021:37:222
2259599170cyclictest3066350irq/52-eth0-rx-22:50:232
2259599170cyclictest3066350irq/52-eth0-rx-22:40:232
2259599170cyclictest3066350irq/52-eth0-rx-22:05:302
2259599170cyclictest3066350irq/52-eth0-rx-22:05:302
2259599170cyclictest3066350irq/52-eth0-rx-21:55:272
2259599170cyclictest3066350irq/52-eth0-rx-21:40:212
2259599170cyclictest3066350irq/52-eth0-rx-21:20:302
2259599170cyclictest3066350irq/52-eth0-rx-19:10:472
2259599170cyclictest3066350irq/52-eth0-rx-00:30:242
2259599170cyclictest299902sleep1019:15:302
2259599170cyclictest299672sleep1022:20:522
2259599170cyclictest299672sleep1022:20:512
2259599170cyclictest286052sleep1021:31:092
2259599170cyclictest283542sleep1000:35:072
2259599170cyclictest275742sleep1021:26:202
2259599170cyclictest27402sleep1020:18:012
2259599170cyclictest265662sleep1020:00:132
2259599170cyclictest261242sleep1023:00:322
2259599170cyclictest24632sleep1023:15:092
2259599170cyclictest243282sleep1023:45:122
2259599170cyclictest242552sleep1019:55:352
2259599170cyclictest239172sleep1000:26:062
2259599170cyclictest235532sleep1023:41:382
2259599170cyclictest232202chrt22:57:302
2259599170cyclictest204022chrt21:10:552
2259599170cyclictest19932sleep1019:20:542
2259599170cyclictest189702sleep1023:35:242
2259599170cyclictest174372sleep1000:20:042
2259599170cyclictest166292sleep1000:16:042
2259599170cyclictest162092sleep1023:30:472
2259599170cyclictest158052sleep1022:45:532
2259599170cyclictest149262sleep1019:43:342
2259599170cyclictest147372sleep1022:00:482
2259599170cyclictest146392sleep1021:00:062
2259599170cyclictest138452chrt20:56:292
2259599170cyclictest129972sleep1000:11:042
2259599170cyclictest126212chrt23:26:232
2259599170cyclictest124050irq/18-i801_smb22:25:362
2259599170cyclictest12250irq/42-ahci22:15:442
2259599170cyclictest12250irq/42-ahci22:10:012
2259599170cyclictest12250irq/42-ahci21:08:262
2259599170cyclictest12250irq/42-ahci20:10:072
2259599170cyclictest12250irq/42-ahci20:10:062
2259599170cyclictest12250irq/42-ahci19:45:062
2259599170cyclictest12250irq/42-ahci19:36:192
2259599170cyclictest12250irq/42-ahci19:33:342
2259599170cyclictest120412sleep1020:50:292
2259599170cyclictest103672sleep1020:46:292
22594991715cyclictest2045-21taskset19:21:3211
2259499170cyclictest0-21swapper/920:32:0511
22593991715cyclictest729-21iostat_ios23:55:1810
22593991715cyclictest6188-21snmp_easybox.os23:20:1210
22593991715cyclictest5243-21snmp_rack3slot920:25:2310
22593991715cyclictest4529-21snmp_easybox.os19:25:2610
22593991715cyclictest31999-21perl22:25:1810
22593991715cyclictest27512-21snmp_rack3slot921:25:2910
22593991715cyclictest24905-21ntp_states19:55:4910
22593991715cyclictest22689-21iostat_ios00:25:2210
22593991715cyclictest21138-21diskstats22:55:1710
22593991715cyclictest20299-21snmp_rack3slot921:10:1810
22593991715cyclictest19263-21snmp_rack3slot921:06:3210
22593991715cyclictest17498-21df_inode22:50:1710
22593991715cyclictest16433-21snmp_rack3slot919:45:2010
22593991715cyclictest15519-21diskstats21:00:2510
22593991715cyclictest13853-21diskstats22:45:1810
22593991715cyclictest10982-21irqstats22:40:2510
22566991715cyclictest2406-21snmpd22:26:444
22595991615cyclictest5134-21smartctl19:25:382
22595991615cyclictest19293-1kworker/10:1H21:18:522
22594991615cyclictest25582-21kworker/9:123:10:1811
22594991615cyclictest0-21swapper/921:20:1211
22593991614cyclictest9327-21users19:30:4210
22593991614cyclictest8058-21snmp_rack3slot921:51:1610
22593991614cyclictest7741-21perl00:05:1610
22593991614cyclictest387-21df_inode20:10:3310
22593991614cyclictest387-21df_inode20:10:3310
22593991614cyclictest29343-21users20:01:1210
22593991614cyclictest24202-21perl19:10:1610
22593991614cyclictest16228-21snmp_easybox.os22:05:1310
22593991614cyclictest16228-21snmp_easybox.os22:05:1210
22593991614cyclictest15451-21snmp_rack3slot922:02:1810
22566991614cyclictest2406-21snmpd21:16:444
22566991614cyclictest2406-21snmpd21:05:594
22566991614cyclictest2406-21snmpd20:31:134
22594991514cyclictest3066350irq/52-eth0-rx-22:15:4411
22594991514cyclictest0-21swapper/920:47:2911
22594991514cyclictest0-21swapper/900:35:1311
2259499151cyclictest0-21swapper/920:08:2811
22593991513cyclictest6035-21munin-run00:05:0010
22593991513cyclictest30962-21df20:05:2910
22593991513cyclictest29951-21snmp_easybox.os23:50:2710
22593991513cyclictest28382-21perl22:20:2110
22593991513cyclictest28382-21perl22:20:2110
22593991513cyclictest25594-21diskstats00:30:1510
22593991513cyclictest22114-21snmp_easybox.os21:15:2810
22593991513cyclictest2093-21snmp_easybox.os20:15:2210
22593991513cyclictest20772-21snmp_rack3slot922:10:2010
22593991513cyclictest1578-21yum23:10:4110
22593991513cyclictest1439-21snmp_rack3slot921:40:2210
22593991513cyclictest12034-21snmp_rack3slot920:50:2310
2259299156cyclictest0-21swapper/700:21:349
2259299155cyclictest0-21swapper/721:50:019
2259299154cyclictest0-21swapper/719:45:009
2257299150cyclictest0-21swapper/319:15:205
22566991513cyclictest2406-21snmpd23:31:024
22566991513cyclictest2406-21snmpd22:02:154
22566991513cyclictest2406-21snmpd21:57:594
22566991513cyclictest2406-21snmpd19:54:444
22555991514cyclictest7468-21python21:50:591
22555991514cyclictest3066350irq/52-eth0-rx-21:33:451
2255599150cyclictest0-21swapper/122:00:381
22596991412cyclictest58832chrt19:28:093
22596991412cyclictest0-21swapper/1121:25:163
2259499144cyclictest0-21swapper/921:30:0011
2259499144cyclictest0-21swapper/900:10:0011
2259499143cyclictest0-21swapper/920:50:0011
2259499143cyclictest0-21swapper/920:20:0211
22594991413cyclictest0-21swapper/922:35:2111
22594991413cyclictest0-21swapper/921:50:2811
22594991412cyclictest18578-21taskset19:48:3411
2259499140cyclictest0-21swapper/921:40:3911
2259399144cyclictest0-21swapper/819:45:0110
22593991412cyclictest6928-21perl22:35:2210
22593991412cyclictest5286-21users23:15:4510
22593991412cyclictest28560-21snmp_easybox.os21:30:3510
22593991412cyclictest25738-21memory23:00:2710
22593991412cyclictest20746-21snmp_easybox.os23:40:1510
22593991412cyclictest20068-21df_inode19:50:1610
22593991412cyclictest1973-21yum19:20:3810
22593991412cyclictest13897-21snmp_rack3slot920:57:0910
2259299144cyclictest0-21swapper/720:30:019
2259299144cyclictest0-21swapper/719:25:019
2259299144cyclictest0-21swapper/719:15:019
22579991412cyclictest15202-21sensors22:45:356
2257299141cyclictest7059-21snmp_rack3slot920:35:185
22566991412cyclictest2406-21snmpd20:18:444
22555991413cyclictest3066350irq/52-eth0-rx-00:05:251
5999130watchdog/70-21swapper/722:08:559
5999130watchdog/70-21swapper/722:08:559
5999130watchdog/70-21swapper/719:25:359
31130ksoftirqd/00-21swapper/021:50:530
22596991311cyclictest901ksoftirqd/1119:45:153
2259499135cyclictest0-21swapper/923:35:0911
2259499134cyclictest0-21swapper/921:20:0111
2259499133cyclictest0-21swapper/923:29:4911
2259499133cyclictest0-21swapper/921:00:0111
2259499133cyclictest0-21swapper/919:35:5711
2259499133cyclictest0-21swapper/919:30:4611
2259499133cyclictest0-21swapper/900:16:1611
2259499133cyclictest0-21swapper/900:11:3911
2259399133cyclictest0-21swapper/820:34:1310
22593991311cyclictest31473-21munin-run00:40:0010
22593991311cyclictest10990-21perl23:25:1510
2259299134cyclictest0-21swapper/723:20:019
2259299134cyclictest0-21swapper/720:15:019
2259299134cyclictest0-21swapper/720:15:009
2259299133cyclictest0-21swapper/723:04:089
2259299133cyclictest0-21swapper/722:30:009
2259299133cyclictest0-21swapper/721:50:019
2259299133cyclictest0-21swapper/719:45:019
2259299133cyclictest0-21swapper/719:10:009
22579991311cyclictest227382sleep422:11:326
2256699134cyclictest0-21swapper/223:55:004
22566991311cyclictest2406-21snmpd23:39:004
22566991311cyclictest2406-21snmpd23:02:464
22566991311cyclictest2406-21snmpd22:24:004
22566991311cyclictest2406-21snmpd22:23:594
13450130irq/18-ehci_hcd0-21swapper/023:35:280
901120ksoftirqd/110-21swapper/1119:20:133
551120ksoftirqd/622318-21qemu-kvm19:20:138
481120ksoftirqd/522317-21qemu-kvm19:45:157
481120ksoftirqd/522317-21qemu-kvm19:20:137
341120ksoftirqd/30-21swapper/319:20:145
31120ksoftirqd/00-21swapper/019:20:130
2259699125cyclictest0-21swapper/1119:15:253
2259499123cyclictest0-21swapper/922:20:0211
2259499123cyclictest0-21swapper/922:20:0111
2259499123cyclictest0-21swapper/920:05:0211
2259499122cyclictest0-21swapper/922:53:3711
2259499122cyclictest0-21swapper/922:26:5311
2259499122cyclictest0-21swapper/919:11:2211
22594991211cyclictest9236-1kworker/9:0H22:05:2211
22594991211cyclictest9236-1kworker/9:0H22:05:2211
22594991211cyclictest0-21swapper/923:20:2111
22594991210cyclictest19242-21memory00:20:2311
22593991211cyclictest27608-21snmp_easybox.os23:05:1610
22593991210cyclictest8275-21perl20:35:3710
22593991210cyclictest11048-21diskstats19:35:1610
2259299122cyclictest0-21swapper/722:36:049
22585991210cyclictest31084-21awk23:55:008
2257299128cyclictest0-21swapper/322:50:395
2257299128cyclictest0-21swapper/319:42:025
2257299125cyclictest0-21swapper/323:25:295
2257299125cyclictest0-21swapper/319:56:275
2257299122cyclictest27224-21snmp_rack3slot923:47:055
2257299120cyclictest0-21swapper/321:20:245
2256699124cyclictest0-21swapper/221:20:444
2256699122cyclictest0-21swapper/222:10:264
22566991210cyclictest2406-21snmpd23:46:014
22566991210cyclictest2406-21snmpd22:16:454
2255499120cyclictest0-21swapper/022:10:410
2255499120cyclictest0-21swapper/019:15:340
2255499120cyclictest0-21swapper/000:20:280
551110ksoftirqd/60-21swapper/619:45:148
411110ksoftirqd/40-21swapper/419:20:146
341110ksoftirqd/30-21swapper/319:45:145
31110ksoftirqd/00-21swapper/019:25:340
2499110watchdog/20-21swapper/221:43:034
2259699113cyclictest0-21swapper/1123:21:303
2259699113cyclictest0-21swapper/1100:28:013
2259699113cyclictest0-21swapper/1100:23:103
2259699113cyclictest0-21swapper/1100:12:233
2259699112cyclictest0-21swapper/1122:39:363
2259699110cyclictest0-21swapper/1121:13:063
2259499114cyclictest0-21swapper/923:00:2311
2259499113cyclictest0-21swapper/922:30:2911
2259499113cyclictest0-21swapper/921:46:0311
2259499113cyclictest0-21swapper/921:37:3911
2259499113cyclictest0-21swapper/921:10:1111
2259499113cyclictest0-21swapper/920:40:0111
2259499113cyclictest0-21swapper/919:26:5811
2259499113cyclictest0-21swapper/919:16:1211
2259499112cyclictest0-21swapper/921:32:1411
22594991110cyclictest0-21swapper/921:55:1711
2259499110cyclictest0-21swapper/922:55:2611
2259399119cyclictest14223-21snmp_rack3slot900:15:1310
2259399114cyclictest0-21swapper/820:25:0110
2259299114cyclictest0-21swapper/722:12:089
2259299113cyclictest0-21swapper/723:48:099
2259299113cyclictest0-21swapper/723:38:019
2259299113cyclictest0-21swapper/723:33:329
2259299113cyclictest0-21swapper/723:20:009
2259299113cyclictest0-21swapper/722:55:019
2259299113cyclictest0-21swapper/722:15:589
2259299113cyclictest0-21swapper/721:57:329
2259299113cyclictest0-21swapper/721:40:019
2259299113cyclictest0-21swapper/720:16:379
2259299113cyclictest0-21swapper/700:19:309
2259299111cyclictest0-21swapper/722:50:019
2259299111cyclictest0-21swapper/700:14:309
2258599119cyclictest31299-21qemu-kvm20:25:228
2258599119cyclictest31299-21qemu-kvm20:20:158
2258599119cyclictest31299-21qemu-kvm20:10:168
2258599119cyclictest31299-21qemu-kvm20:10:168
2258599119cyclictest31298-21qemu-kvm20:40:258
2258599119cyclictest31298-21qemu-kvm20:15:188
2258199110cyclictest0-21swapper/500:40:017
2257999113cyclictest0-21swapper/422:25:236
2257999113cyclictest0-21swapper/422:15:266
2257999113cyclictest0-21swapper/421:40:406
2257999113cyclictest0-21swapper/421:20:416
2257999112cyclictest0-21swapper/419:55:366
2257999112cyclictest0-21swapper/400:25:216
2257299117cyclictest0-21swapper/323:23:445
2257299117cyclictest0-21swapper/323:07:495
2257299115cyclictest0-21swapper/323:50:165
2257299115cyclictest0-21swapper/323:35:165
2257299115cyclictest0-21swapper/323:00:335
2257299115cyclictest0-21swapper/322:30:275
2257299115cyclictest0-21swapper/321:55:275
2257299114cyclictest0-21swapper/322:56:335
2257299113cyclictest0-21swapper/322:40:225
2257299113cyclictest0-21swapper/319:30:005
2257299111cyclictest0-21swapper/323:13:575
2257299110cyclictest0-21swapper/323:40:295
2257299110cyclictest0-21swapper/319:50:465
2257299110cyclictest0-21swapper/300:15:275
2256699119cyclictest2406-21snmpd23:12:164
2256699113cyclictest0-21swapper/220:00:404
2256699112cyclictest0-21swapper/222:40:214
2256699112cyclictest0-21swapper/200:10:194
2256699110cyclictest0-21swapper/220:10:454
2256699110cyclictest0-21swapper/220:10:444
2255599113cyclictest0-21swapper/122:10:251
22555991110cyclictest3066350irq/52-eth0-rx-19:15:401
2255499119cyclictest0-21swapper/022:30:340
1699110watchdog/00-21swapper/021:04:340
1699110watchdog/00-21swapper/020:39:190
14850110irq/18-uhci_hcd0-21swapper/022:25:350
901100ksoftirqd/110-21swapper/1122:10:243
901100ksoftirqd/110-21swapper/1120:00:163
8999100migration/117518-21switchtime21:50:593
761100ksoftirqd/90-21swapper/923:40:0711
621100ksoftirqd/70-21swapper/723:40:069
621100ksoftirqd/70-21swapper/721:11:069
6199100migration/729452-21if_err_tap023:50:229
6199100migration/72202-21cpuspeed_turbos22:30:129
551100ksoftirqd/619193-21qemu-kvm22:10:258
551100ksoftirqd/60-21swapper/623:10:318
551100ksoftirqd/60-21swapper/622:40:058
5499100migration/626957-21cpuspeed_turbos20:00:128
5499100migration/624313-21latency_hist00:30:008
5499100migration/620685-21runrttasks21:14:508
5299100watchdog/60-21swapper/622:04:568
5299100watchdog/60-21swapper/621:47:388
481100ksoftirqd/50-21swapper/522:40:057
481100ksoftirqd/50-21swapper/521:46:437
481100ksoftirqd/50-21swapper/519:37:387
461100rcuc/50-21swapper/520:05:167
4599100watchdog/50-21swapper/522:51:117
3899100watchdog/40-21swapper/421:03:236
3899100watchdog/40-21swapper/420:52:466
3899100watchdog/40-21swapper/420:34:586
3899100watchdog/40-21swapper/420:12:196
3899100watchdog/40-21swapper/420:12:186
3899100watchdog/40-21swapper/420:01:556
3899100watchdog/40-21swapper/419:26:366
3899100watchdog/40-21swapper/419:11:426
3066450100irq/53-eth0-tx-23057-21munin-node19:10:191
31100ksoftirqd/00-21swapper/022:40:040
31100ksoftirqd/00-21swapper/022:35:540
31100ksoftirqd/00-21swapper/021:35:260
271100ksoftirqd/20-21swapper/222:35:534
271100ksoftirqd/20-21swapper/221:11:064
2259699109cyclictest901ksoftirqd/1122:40:203
2259699108cyclictest0-21swapper/1120:08:193
2259699104cyclictest0-21swapper/1123:34:083
2259699103cyclictest0-21swapper/1123:58:473
2259699103cyclictest0-21swapper/1123:51:003
2259699103cyclictest0-21swapper/1123:38:153
2259699103cyclictest0-21swapper/1122:49:153
2259699103cyclictest0-21swapper/1121:47:593
2259699103cyclictest0-21swapper/1100:19:163
2259699102cyclictest0-21swapper/1120:36:283
2259699102cyclictest0-21swapper/1100:09:463
2259699100cyclictest0-21swapper/1123:46:053
2259499109cyclictest0-21swapper/923:05:1811
2259499108cyclictest4320-21sshd20:20:3311
2259499104cyclictest0-21swapper/923:46:1811
2259499104cyclictest0-21swapper/919:50:2711
2259499103cyclictest0-21swapper/923:57:4011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional