You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 09:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3.osadl.org (updated Tue Nov 25, 2025 00:46:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
122501140irq/42-ahci0-21swapper/219:08:264
122501120irq/42-ahci0-21swapper/119:09:131
18459210599sleep80-21swapper/819:09:4610
18279210291sleep110-21swapper/1119:08:013
18203210189sleep70-21swapper/719:06:569
1794529685sleep100-21swapper/1019:05:362
12250900irq/42-ahci0-21swapper/419:06:086
12250850irq/42-ahci0-21swapper/919:08:2311
3066350840irq/52-eth0-rx-0-21swapper/319:09:305
1827326553sleep50-21swapper/519:07:557
1815926553sleep00-21swapper/019:06:250
1810525955sleep60-21swapper/619:05:438
1858499290cyclictest0-21swapper/022:11:060
18632992710cyclictest831ksoftirqd/1022:50:282
18632992710cyclictest831ksoftirqd/1022:50:282
1863299247cyclictest831ksoftirqd/1021:16:342
1863299225cyclictest831ksoftirqd/1019:49:512
1863299225cyclictest831ksoftirqd/1019:49:502
1858499220cyclictest0-21swapper/021:10:340
1863299192cyclictest831ksoftirqd/1020:45:322
18632991817cyclictest831ksoftirqd/1023:00:202
18632991817cyclictest831ksoftirqd/1023:00:192
18632991817cyclictest831ksoftirqd/1020:10:152
18632991817cyclictest0-21swapper/1022:55:152
18632991817cyclictest0-21swapper/1022:55:152
18632991817cyclictest0-21swapper/1019:27:482
18632991816cyclictest32599-21python23:50:272
1863299181cyclictest831ksoftirqd/1021:55:132
1863299181cyclictest831ksoftirqd/1000:35:272
1863299181cyclictest124050irq/18-i801_smb23:35:302
1863299180cyclictest831ksoftirqd/1023:30:172
1863199181cyclictest0-21swapper/922:13:5611
1863199181cyclictest0-21swapper/919:40:1911
18628991816cyclictest8598-21snmp_rack3slot922:35:1310
18628991816cyclictest8598-21snmp_rack3slot922:35:1210
18628991816cyclictest812-21snmp_rack3slot923:51:5010
18628991816cyclictest8078-21snmp_rack3slot920:20:1810
18628991816cyclictest568-21perl20:10:1510
18628991816cyclictest31095-21snmp_easybox.os21:35:2510
18628991816cyclictest28949-21snmp_rack3slot919:20:1610
18628991816cyclictest26000-21snmp_rack3slot923:41:4810
18628991816cyclictest24305-21iostat22:55:2010
18628991816cyclictest24305-21iostat22:55:1910
18628991816cyclictest24093-21snmp_rack3slot919:15:1610
18628991816cyclictest22015-21snmp_rack3slot920:40:1310
18628991816cyclictest21466-21snmp_rack3slot922:06:3610
18628991816cyclictest18556-21munin-run22:50:0110
18628991816cyclictest18556-21munin-run22:50:0110
18628991816cyclictest16617-21df23:30:1510
18628991816cyclictest1640-21yum19:25:3410
18628991816cyclictest14567-21perl19:45:1610
18628991816cyclictest14567-21perl19:45:1610
18628991816cyclictest10577-21snmp_rack3slot919:40:1510
18628991816cyclictest10528-21snmp_rack3slot921:51:4710
1860799183cyclictest0-21swapper/219:35:214
18607991816cyclictest2406-21snmpd00:09:094
18632991716cyclictest831ksoftirqd/1023:40:132
18632991716cyclictest831ksoftirqd/1022:35:162
18632991716cyclictest831ksoftirqd/1022:35:152
18632991716cyclictest831ksoftirqd/1022:25:292
18632991716cyclictest831ksoftirqd/1021:00:112
18632991716cyclictest831ksoftirqd/1019:30:162
18632991716cyclictest7495-1kworker/10:0H00:32:022
18632991716cyclictest22152-21diskstats20:40:142
18632991716cyclictest0-21swapper/1023:15:172
18632991716cyclictest0-21swapper/1022:45:012
18632991716cyclictest0-21swapper/1022:45:002
1863299171cyclictest831ksoftirqd/1000:25:002
1863299170cyclictest95262sleep1020:20:442
1863299170cyclictest831ksoftirqd/1022:20:012
1863299170cyclictest831ksoftirqd/1022:05:162
1863299170cyclictest831ksoftirqd/1021:50:292
1863299170cyclictest831ksoftirqd/1021:45:142
1863299170cyclictest831ksoftirqd/1021:35:002
1863299170cyclictest831ksoftirqd/1021:05:292
1863299170cyclictest831ksoftirqd/1020:55:312
1863299170cyclictest831ksoftirqd/1020:08:102
1863299170cyclictest831ksoftirqd/1019:10:142
1863299170cyclictest831ksoftirqd/1000:25:162
1863299170cyclictest81032sleep1000:00:432
1863299170cyclictest4862chrt23:07:262
1863299170cyclictest43452chrt20:15:172
1863299170cyclictest31932sleep1021:41:052
1863299170cyclictest314362sleep1020:51:342
1863299170cyclictest3066350irq/52-eth0-rx-23:55:372
1863299170cyclictest3066350irq/52-eth0-rx-22:30:192
1863299170cyclictest3066350irq/52-eth0-rx-22:10:152
1863299170cyclictest3066350irq/52-eth0-rx-21:25:292
1863299170cyclictest3066350irq/52-eth0-rx-19:40:162
1863299170cyclictest3066350irq/52-eth0-rx-19:36:012
1863299170cyclictest305132sleep1019:21:362
1863299170cyclictest297522sleep1023:48:292
1863299170cyclictest282572sleep1021:30:512
1863299170cyclictest268142sleep1019:15:472
1863299170cyclictest259362sleep1022:15:082
1863299170cyclictest244302sleep1020:00:052
1863299170cyclictest244302sleep1020:00:042
1863299170cyclictest235972sleep1019:55:532
1863299170cyclictest211232sleep1021:22:502
1863299170cyclictest200062chrt19:51:272
1863299170cyclictest190522sleep1000:15:452
1863299170cyclictest156972sleep1000:14:452
1863299170cyclictest150542sleep1023:26:272
1863299170cyclictest149202chrt22:00:062
1863299170cyclictest148892sleep1022:44:592
1863299170cyclictest134122sleep1020:29:082
1863299170cyclictest12250irq/42-ahci20:35:302
1863299170cyclictest12250irq/42-ahci20:30:472
1863299170cyclictest119822sleep1000:08:332
1863299170cyclictest108692sleep1021:10:112
1863199170cyclictest0-21swapper/922:55:1811
1863199170cyclictest0-21swapper/922:55:1811
1863199170cyclictest0-21swapper/920:40:1211
1863199170cyclictest0-21swapper/919:19:5611
18628991715cyclictest6687-21snmp_rack3slot919:35:1610
18628991715cyclictest6561-21perl00:00:1510
18628991715cyclictest6379-21snmp_rack3slot921:01:2410
18628991715cyclictest344-21snmp_rack3slot923:05:3510
18628991715cyclictest30439-21snmp_rack3slot922:20:1210
18628991715cyclictest27731-21snmp_rack3slot900:30:1510
18628991715cyclictest26804-21snmp_rack3slot922:15:1510
18628991715cyclictest26308-21snmp_rack3slot921:30:1510
18628991715cyclictest26121-21perl20:45:1710
18628991715cyclictest23080-21snmp_rack3slot922:10:1410
18628991715cyclictest22809-21ntp_states19:55:2510
18628991715cyclictest21003-21perl00:20:1810
18628991715cyclictest19461-21memory20:35:2310
18628991715cyclictest19403-21perl21:20:1610
18628991715cyclictest19090-21snmp_easybox.os19:10:1210
18628991715cyclictest19016-21users00:15:3010
18628991715cyclictest14137-21snmp_rack3slot921:56:3710
18628991715cyclictest14011-21snmp_easybox.os20:30:1110
18628991715cyclictest12470-21iostat_ios21:10:2110
18628991715cyclictest1230-21snmp_rack3slot922:25:1310
18628991715cyclictest11876-21snmp_rack3slot900:07:0310
18628991715cyclictest10589-21snmp_rack3slot920:25:1210
18607991715cyclictest2406-21snmpd22:12:244
18607991715cyclictest2406-21snmpd19:12:234
18631991615cyclictest0-21swapper/923:45:1311
18631991615cyclictest0-21swapper/923:20:3111
18631991615cyclictest0-21swapper/922:40:2711
18628991615cyclictest2579-21users20:55:3310
18628991614cyclictest6317-21snmp_easybox.os22:30:2210
18628991614cyclictest4459-21yum23:55:3310
18628991614cyclictest31649-21diskstats00:35:1210
18628991614cyclictest27222-21diskstats23:00:1610
18628991614cyclictest27222-21diskstats23:00:1610
18628991614cyclictest21890-21snmp_rack3slot922:50:4810
18628991614cyclictest21890-21snmp_rack3slot922:50:4710
18628991614cyclictest20598-21perl23:35:1810
18628991614cyclictest19184-21snmp_rack3slot919:50:2710
18628991614cyclictest15443-21snmp_rack3slot900:11:1410
18628991614cyclictest1238-21snmp_rack3slot921:40:1710
1862299163cyclictest0-21swapper/722:33:029
18584991615cyclictest18496-21turbostat19:14:550
18631991514cyclictest0-21swapper/923:50:2111
18631991514cyclictest0-21swapper/919:30:3211
18628991513cyclictest6530-21iostat23:15:2010
18628991513cyclictest27327-21snmp_rack3slot920:01:3810
18628991513cyclictest27327-21snmp_rack3slot920:01:3710
18628991513cyclictest17298-21yum21:15:3710
18628991513cyclictest10182-21irqstats23:20:2010
1862299154cyclictest0-21swapper/719:40:219
1862299154cyclictest0-21swapper/700:00:029
18607991513cyclictest2406-21snmpd20:39:534
18584991514cyclictest12250irq/42-ahci20:10:030
3066350140irq/52-eth0-rx-0-21swapper/322:11:055
31140ksoftirqd/00-21swapper/021:30:290
18632991413cyclictest12-21rcu_preempt23:20:262
18632991413cyclictest0-21swapper/1023:10:022
18631991413cyclictest0-21swapper/922:35:1611
18631991413cyclictest0-21swapper/922:35:1511
18628991413cyclictest31440-21snmp_rack3slot920:51:3710
18628991412cyclictest27527-21df23:45:1310
18628991412cyclictest26385-21snmp_rack3slot900:26:3810
18628991412cyclictest15050-21snmp_rack3slot923:26:2510
18628991412cyclictest10013-21snmp_rack3slot921:06:2510
1862299144cyclictest0-21swapper/720:20:029
1862299144cyclictest0-21swapper/700:15:019
1862299143cyclictest0-21swapper/719:55:029
18619991412cyclictest17632-21sensors22:45:286
18619991412cyclictest17632-21sensors22:45:286
182050140irq/55-eth1-rx-0-21swapper/722:11:059
12250140irq/42-ahci0-21swapper/122:11:141
5299130watchdog/60-21swapper/620:09:028
1863199133cyclictest0-21swapper/919:45:0111
1863199133cyclictest0-21swapper/919:45:0011
18631991312cyclictest3066350irq/52-eth0-rx-19:58:1511
1863199130cyclictest0-21swapper/922:15:1311
1862899133cyclictest0-21swapper/819:30:0110
18628991311cyclictest3109-21snmp_easybox.os20:15:1210
18628991311cyclictest22583-21df21:25:1510
1862299134cyclictest0-21swapper/723:15:029
1862299134cyclictest0-21swapper/722:25:009
1862299134cyclictest0-21swapper/700:35:009
1862299133cyclictest0-21swapper/723:45:019
1862299133cyclictest0-21swapper/723:15:019
1862299133cyclictest0-21swapper/723:10:009
1862299133cyclictest0-21swapper/722:25:389
1862299133cyclictest0-21swapper/722:09:089
1862299133cyclictest0-21swapper/720:45:019
1862299133cyclictest0-21swapper/720:39:239
1862299133cyclictest0-21swapper/720:23:239
1862299133cyclictest0-21swapper/719:59:279
1862299133cyclictest0-21swapper/700:07:399
18619991311cyclictest14396-21sensors23:25:256
1860799133cyclictest0-21swapper/222:35:204
1860799133cyclictest0-21swapper/222:35:194
1860799133cyclictest0-21swapper/222:30:164
1860799133cyclictest0-21swapper/222:10:014
1860799133cyclictest0-21swapper/200:25:174
18607991311cyclictest2406-21snmpd00:37:544
1859699133cyclictest0-21swapper/120:25:181
18584991312cyclictest2393-21nscd20:15:220
18584991312cyclictest12250irq/42-ahci20:09:530
14850130irq/18-uhci_hcd0-21swapper/019:15:010
8799120watchdog/110-21swapper/1100:35:353
7399120watchdog/90-21swapper/920:08:1111
31120ksoftirqd/00-21swapper/023:10:240
31120ksoftirqd/00-21swapper/022:20:240
1863199122cyclictest0-21swapper/921:30:0011
18631991211cyclictest0-21swapper/921:50:1511
1862299123cyclictest0-21swapper/723:53:329
1862299123cyclictest0-21swapper/722:00:019
1862299122cyclictest0-21swapper/721:09:599
1862299122cyclictest0-21swapper/719:48:529
1862299122cyclictest0-21swapper/719:48:519
1862299122cyclictest0-21swapper/700:22:099
18621991210cyclictest32200-21awk21:40:018
1861399129cyclictest0-21swapper/322:20:115
1861399124cyclictest0-21swapper/323:50:175
1861399122cyclictest0-21swapper/320:40:205
1861399121cyclictest0-21swapper/323:15:195
1860799122cyclictest0-21swapper/220:45:204
14850120irq/18-uhci_hcd0-21swapper/000:20:330
13450120irq/18-ehci_hcd0-21swapper/023:50:250
13450120irq/18-ehci_hcd0-21swapper/021:20:260
8999110migration/1131632-21munin-node22:20:233
341110ksoftirqd/30-21swapper/323:10:145
31110ksoftirqd/00-21swapper/022:05:250
31110ksoftirqd/00-21swapper/000:00:260
201110ksoftirqd/12340-40kipmi022:25:211
1863199113cyclictest0-21swapper/923:35:0211
1863199113cyclictest0-21swapper/923:27:0211
1863199113cyclictest0-21swapper/920:37:2811
1863199112cyclictest0-21swapper/921:10:0211
1863199112cyclictest0-21swapper/900:30:0111
1863199110cyclictest0-21swapper/922:20:2611
1862899119cyclictest30905-21snmp_rack3slot920:05:5010
1862899110cyclictest0-21swapper/823:15:0110
1862299119cyclictest0-21swapper/721:35:169
1862299113cyclictest0-21swapper/722:40:039
1862299113cyclictest0-21swapper/722:05:019
1862299113cyclictest0-21swapper/721:32:089
1862299113cyclictest0-21swapper/721:20:089
1862299113cyclictest0-21swapper/721:10:399
1862299113cyclictest0-21swapper/720:33:089
1862299113cyclictest0-21swapper/719:26:229
1862299111cyclictest0-21swapper/721:27:079
1862299111cyclictest0-21swapper/719:21:269
1862299111cyclictest0-21swapper/719:15:019
1862199119cyclictest10194-21qemu-kvm21:08:368
1862199113cyclictest0-21swapper/622:55:338
1862199113cyclictest0-21swapper/622:55:328
1862199113cyclictest0-21swapper/621:52:018
1862199113cyclictest0-21swapper/621:16:018
1862199113cyclictest0-21swapper/620:38:258
1862199113cyclictest0-21swapper/620:02:018
1862199113cyclictest0-21swapper/620:02:018
1862199113cyclictest0-21swapper/600:31:038
1862199113cyclictest0-21swapper/600:15:178
1862199111cyclictest0-21swapper/619:50:318
1862199110cyclictest31130-21qemu-kvm20:30:018
1862099119cyclictest0-21swapper/520:50:147
1861999113cyclictest0-21swapper/423:10:166
1861999113cyclictest0-21swapper/422:40:016
1861999113cyclictest0-21swapper/422:40:006
1861999113cyclictest0-21swapper/422:05:186
1861999113cyclictest0-21swapper/421:05:176
1861999113cyclictest0-21swapper/419:50:026
1861999113cyclictest0-21swapper/419:50:016
1861999111cyclictest0-21swapper/400:30:396
1861399119cyclictest0-21swapper/321:05:125
1861399119cyclictest0-21swapper/300:20:005
1861399114cyclictest2340-40kipmi022:07:215
1861399113cyclictest0-21swapper/323:55:195
1861399113cyclictest0-21swapper/323:05:175
1861399113cyclictest0-21swapper/321:40:205
1861399113cyclictest0-21swapper/321:10:205
1861399113cyclictest0-21swapper/320:50:185
1861399113cyclictest0-21swapper/320:30:185
1861399113cyclictest0-21swapper/319:20:185
1861399112cyclictest0-21swapper/323:40:175
1861399110cyclictest0-21swapper/320:35:325
1860799113cyclictest0-21swapper/222:20:004
1860799113cyclictest0-21swapper/220:30:014
1860799113cyclictest0-21swapper/219:15:234
1859699113cyclictest0-21swapper/122:30:181
18596991110cyclictest3066350irq/52-eth0-rx-23:35:281
1699110watchdog/00-21swapper/000:36:110
14850110irq/18-uhci_hcd0-21swapper/023:55:260
13450110irq/18-ehci_hcd0-21swapper/020:55:270
901100ksoftirqd/110-21swapper/1122:50:133
901100ksoftirqd/110-21swapper/1122:50:123
901100ksoftirqd/110-21swapper/1122:05:173
901100ksoftirqd/110-21swapper/1121:46:063
901100ksoftirqd/110-21swapper/1120:20:033
8999100migration/1128171-21cstates19:20:123
8999100migration/1116992-21if_tap022:45:203
8999100migration/1116992-21if_tap022:45:193
8999100migration/1115862-21munin-node19:45:333
8999100migration/1115862-21munin-node19:45:333
8999100migration/1115134-21latency_hist22:45:013
8999100migration/1111009-21latency_hist21:55:013
881100rcuc/110-21swapper/1120:12:473
761100ksoftirqd/90-21swapper/923:10:1311
761100ksoftirqd/90-21swapper/920:10:2011
7599100migration/9894-21cstates21:40:1511
621100ksoftirqd/70-21swapper/722:50:139
621100ksoftirqd/70-21swapper/722:50:139
551100ksoftirqd/60-21swapper/623:10:148
551100ksoftirqd/60-21swapper/623:01:158
551100ksoftirqd/60-21swapper/623:01:148
551100ksoftirqd/60-21swapper/622:05:178
551100ksoftirqd/60-21swapper/620:20:038
5499100migration/69410-21gltestperf22:35:168
5499100migration/69410-21gltestperf22:35:158
5499100migration/623648-21runrttasks19:56:358
5499100migration/620523-21idleruntime19:10:198
5499100migration/614807-21munin-node00:10:258
5499100migration/614353-21taskset21:59:228
5499100migration/611725-21munin-node00:05:318
481100ksoftirqd/531130-21qemu-kvm20:20:037
481100ksoftirqd/50-21swapper/522:50:137
481100ksoftirqd/50-21swapper/522:50:127
461100rcuc/50-21swapper/519:26:377
4599100watchdog/518342-21qemu-kvm19:47:237
4599100watchdog/518342-21qemu-kvm19:47:237
4599100watchdog/518341-21qemu-kvm19:13:477
4599100watchdog/50-21swapper/519:30:437
4599100watchdog/50-21swapper/519:17:107
411100ksoftirqd/40-21swapper/420:10:206
3899100watchdog/40-21swapper/420:39:436
341100ksoftirqd/30-21swapper/322:50:135
341100ksoftirqd/30-21swapper/322:50:125
3199100watchdog/30-21swapper/319:48:025
3199100watchdog/30-21swapper/319:48:025
3066350100irq/52-eth0-rx-0-21swapper/023:45:170
3066350100irq/52-eth0-rx-0-21swapper/000:05:230
31100ksoftirqd/00-21swapper/023:35:290
31100ksoftirqd/00-21swapper/023:01:150
31100ksoftirqd/00-21swapper/023:01:140
31100ksoftirqd/00-21swapper/019:50:230
271100ksoftirqd/20-21swapper/221:08:144
271100ksoftirqd/20-21swapper/219:30:184
271100ksoftirqd/20-21swapper/200:10:224
201100ksoftirqd/10-21swapper/123:10:141
201100ksoftirqd/10-21swapper/123:01:151
201100ksoftirqd/10-21swapper/123:01:151
201100ksoftirqd/10-21swapper/122:35:311
201100ksoftirqd/10-21swapper/122:35:301
201100ksoftirqd/10-21swapper/119:11:041
1863199103cyclictest0-21swapper/922:30:0011
1863199103cyclictest0-21swapper/921:15:0011
1863199102cyclictest0-21swapper/921:50:0111
1863199102cyclictest0-21swapper/921:05:0111
1863199102cyclictest0-21swapper/920:20:1411
1863199102cyclictest0-21swapper/919:55:0111
1863199102cyclictest0-21swapper/919:29:5611
1863199102cyclictest0-21swapper/919:23:2611
1863199102cyclictest0-21swapper/900:39:1211
1863199100cyclictest0-21swapper/923:30:0111
1863199100cyclictest0-21swapper/920:25:2111
1862899108cyclictest17709-21snmp_rack3slot922:00:3610
1862899102cyclictest0-21swapper/821:49:3010
1862299103cyclictest0-21swapper/721:47:089
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional