You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-13 - 00:19
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3.osadl.org (updated Wed Nov 12, 2025 12:46:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22941210492sleep60-21swapper/607:06:258
23134210087sleep80-21swapper/807:08:1610
12250970irq/42-ahci0-21swapper/107:09:041
3066350950irq/52-eth0-rx-0-21swapper/307:07:015
3066450920irq/53-eth0-tx-0-21swapper/707:09:119
3066350880irq/52-eth0-rx-0-21swapper/507:05:527
12250870irq/42-ahci0-21swapper/207:08:524
2321028378sleep100-21swapper/1007:09:172
2131527265sleep90-21swapper/907:05:1611
3066350640irq/52-eth0-rx-0-21swapper/407:05:236
2288926152sleep00-21swapper/007:05:430
2245425955sleep110-21swapper/1107:05:303
2338999280cyclictest0-21swapper/407:20:276
23402992624cyclictest31468-21snmp_rack3slot911:00:1310
23363992625cyclictest0-21swapper/008:18:440
23405992119cyclictest0-21swapper/1108:21:223
2340399192cyclictest0-21swapper/911:25:1111
2340399192cyclictest0-21swapper/907:30:0111
23404991817cyclictest19558-1kworker/10:2H07:40:492
2340499181cyclictest831ksoftirqd/1012:15:012
2340499181cyclictest831ksoftirqd/1010:55:112
2340499181cyclictest831ksoftirqd/1010:30:142
2340499181cyclictest3066450irq/53-eth0-tx-10:40:142
2340499181cyclictest3066350irq/52-eth0-rx-10:22:182
2340499181cyclictest271882chrt11:37:592
2340499181cyclictest12250irq/42-ahci10:37:472
2340499180cyclictest12250irq/42-ahci09:10:372
2340399182cyclictest0-21swapper/911:50:2311
23403991816cyclictest7699-21snmp_easybox.os10:25:2511
2340399181cyclictest0-21swapper/910:53:1711
2340399181cyclictest0-21swapper/910:10:2611
2340299187cyclictest0-21swapper/809:15:0110
23402991816cyclictest9618-21df_inode09:45:1210
23402991816cyclictest7914-21snmp_easybox.os08:15:0910
23402991816cyclictest7418-21snmp_easybox.os07:30:1110
23402991816cyclictest7339-21munin-run08:15:0010
23402991816cyclictest7339-21munin-run08:15:0010
23402991816cyclictest4717-21snmp_rack3slot907:25:1710
23402991816cyclictest341-21snmp_rack3slot907:20:1310
23402991816cyclictest30705-21snmp_rack3slot911:41:2410
23402991816cyclictest30556-21diskstats08:45:1310
23402991816cyclictest30315-21snmp_rack3slot910:56:3510
23402991816cyclictest28513-21perl07:15:1510
23402991816cyclictest26922-21snmp_rack3slot908:40:1210
23402991816cyclictest25061-21snmp_rack3slot907:10:1610
23402991816cyclictest24045-21irqstats08:35:2010
23402991816cyclictest2182-21munin-run11:50:0110
23402991816cyclictest21438-21diskstats11:30:1410
23402991816cyclictest19681-21snmp_easybox.os10:00:1310
23402991816cyclictest19373-21snmp_rack3slot907:45:1510
23402991816cyclictest18838-21snmp_easybox.os08:30:1210
23402991816cyclictest16903-21snmp_rack3slot909:10:1610
23402991816cyclictest16614-21snmp_rack3slot912:07:1310
23402991816cyclictest14503-21perl11:20:1510
23402991816cyclictest14503-21perl11:20:1510
23402991816cyclictest11231-21perl12:00:1610
23405991715cyclictest72582sleep1108:14:003
23405991715cyclictest72582sleep1108:14:003
23404991716cyclictest831ksoftirqd/1007:25:282
23404991716cyclictest23272-21smartctl07:50:172
23404991716cyclictest19558-1kworker/10:2H08:16:492
23404991716cyclictest0-21swapper/1012:32:462
23404991716cyclictest0-21swapper/1012:20:402
23404991716cyclictest0-21swapper/1009:40:192
2340499170cyclictest93442chrt11:57:372
2340499170cyclictest89042sleep1011:11:332
2340499170cyclictest89042sleep1011:11:322
2340499170cyclictest831ksoftirqd/1010:07:052
2340499170cyclictest811rcuc/1009:55:002
2340499170cyclictest74402sleep1008:55:522
2340499170cyclictest64692sleep1010:25:142
2340499170cyclictest6372chrt09:31:562
2340499170cyclictest62472sleep1012:39:432
2340499170cyclictest57812chrt11:53:372
2340499170cyclictest54052sleep1011:08:192
2340499170cyclictest42912chrt09:35:442
2340499170cyclictest38222sleep1008:51:042
2340499170cyclictest34292sleep1008:07:112
2340499170cyclictest320822sleep1008:01:232
2340499170cyclictest316902sleep1007:17:052
2340499170cyclictest31382sleep1007:24:192
2340499170cyclictest307032sleep1011:41:232
2340499170cyclictest3066350irq/52-eth0-rx-11:00:122
2340499170cyclictest3066350irq/52-eth0-rx-09:55:412
2340499170cyclictest3066350irq/52-eth0-rx-09:00:312
2340499170cyclictest295702sleep1009:28:212
2340499170cyclictest288882sleep1008:41:022
2340499170cyclictest283572chrt07:55:462
2340499170cyclictest260822chrt07:10:282
2340499170cyclictest255992chrt09:20:312
2340499170cyclictest253082sleep1010:50:172
2340499170cyclictest239382sleep1012:17:512
2340499170cyclictest233622sleep1011:30:342
2340499170cyclictest229842chrt10:46:052
2340499170cyclictest224692sleep1010:00:472
2340499170cyclictest221992sleep1009:17:192
2340499170cyclictest20162chrt11:47:592
2340499170cyclictest198222chrt11:26:462
2340499170cyclictest166152sleep1012:07:152
2340499170cyclictest161562sleep1011:21:202
2340499170cyclictest161562sleep1011:21:202
2340499170cyclictest142882sleep1008:20:372
2340499170cyclictest133112sleep1009:05:182
2340499170cyclictest128872sleep1012:01:022
2340499170cyclictest125892sleep1007:35:192
2340499170cyclictest12250irq/42-ahci11:15:502
2340499170cyclictest12250irq/42-ahci11:15:502
2340499170cyclictest12250irq/42-ahci08:45:402
2340499170cyclictest12250irq/42-ahci08:32:182
2340499170cyclictest12250irq/42-ahci08:12:542
2340499170cyclictest12250irq/42-ahci08:12:532
2340499170cyclictest12250irq/42-ahci07:46:372
2340499170cyclictest12250irq/42-ahci07:30:152
2340499170cyclictest115432sleep1009:45:342
2340399170cyclictest11122-21missed_timers10:30:2211
23402991715cyclictest7185-21df11:55:1710
23402991715cyclictest5944-21snmp_rack3slot912:35:3610
23402991715cyclictest3357-21snmp_rack3slot908:06:1510
23402991715cyclictest32356-21snmp_easybox.os09:30:2410
23402991715cyclictest30066-21diskstats08:00:1410
23402991715cyclictest28350-21snmp_rack3slot907:55:3910
23402991715cyclictest2818-21snmp_rack3slot910:20:1710
23402991715cyclictest26653-21snmp_rack3slot910:51:1310
23402991715cyclictest2652-21perl09:35:1510
23402991715cyclictest25140-21memory10:05:2110
23402991715cyclictest25025-21snmp_easybox.os09:20:2610
23402991715cyclictest2240-21snmp_rack3slot912:30:3610
23402991715cyclictest18443-21perl12:10:1510
23402991715cyclictest1768-21diskstats08:50:1610
23402991715cyclictest16468-21snmp_rack3slot909:55:1310
23402991715cyclictest15484-21snmp_rack3slot907:40:1310
23402991715cyclictest15156-21users09:50:3110
23402991715cyclictest12797-21memory07:35:2210
23402991715cyclictest12747-21fw_packets08:20:1610
23397991715cyclictest3549-21qemu-kvm08:47:308
23397991715cyclictest3549-21qemu-kvm08:09:388
2339799170cyclictest27149-21snmp_rack3slot908:40:138
2338999171cyclictest0-21swapper/408:10:276
2338999171cyclictest0-21swapper/408:10:276
2338399171cyclictest0-21swapper/308:15:205
23377991715cyclictest2406-21snmpd09:57:274
2337799171cyclictest0-21swapper/208:45:294
2337799171cyclictest0-21swapper/208:15:264
2336799171cyclictest0-21swapper/108:05:171
2340599165cyclictest0-21swapper/1108:19:093
2340499160cyclictest3066350irq/52-eth0-rx-12:28:582
2340399165cyclictest0-21swapper/907:20:0111
23403991615cyclictest0-21swapper/911:57:4811
23403991615cyclictest0-21swapper/910:40:1511
23403991615cyclictest0-21swapper/909:16:1811
23403991615cyclictest0-21swapper/907:50:1311
23403991614cyclictest31133-21chrt12:26:5111
2340399161cyclictest0-21swapper/907:35:2211
23402991615cyclictest22327-21snmp_easybox.os10:45:2410
23402991614cyclictest7639-21iostat_ios11:10:1810
23402991614cyclictest7639-21iostat_ios11:10:1810
23402991614cyclictest7214-21iostat_ios10:25:1910
23402991614cyclictest31947-21snmp_rack3slot910:15:1710
23402991614cyclictest31095-21snmp_rack3slot912:26:2510
23402991614cyclictest23828-21snmp_rack3slot912:16:2410
23402991614cyclictest17803-21diskstats11:25:1210
23402991614cyclictest13737-21df_inode10:35:1410
23402991614cyclictest12483-21snmp_rack3slot911:15:5110
23402991614cyclictest12483-21snmp_rack3slot911:15:5010
23402991614cyclictest11280-21ntp_states10:30:2310
23402991614cyclictest11193-21snmp_rack3slot909:01:5010
23397991615cyclictest3549-21qemu-kvm08:30:418
2339799161cyclictest3549-21qemu-kvm08:27:198
2339799160cyclictest5184-21latency_hist10:25:018
2339799160cyclictest19676-21head11:25:288
23390991615cyclictest5195-21cpuspeed09:40:127
23390991615cyclictest3550-21qemu-kvm08:27:187
23390991615cyclictest27339-21latency_hist11:40:017
23390991614cyclictest3549-21qemu-kvm08:15:077
2339099160cyclictest0-21swapper/507:10:317
23383991614cyclictest4284-21cpuspeed08:10:155
23383991614cyclictest4284-21cpuspeed08:10:155
2338399160cyclictest0-21swapper/307:55:265
2338399160cyclictest0-21swapper/307:10:245
23377991614cyclictest2406-21snmpd10:52:284
23377991614cyclictest2406-21snmpd08:10:574
23377991614cyclictest2406-21snmpd08:10:574
2340399153cyclictest0-21swapper/907:35:0111
23403991514cyclictest28267-21sensors_temp08:40:2411
23403991514cyclictest0-21swapper/908:35:2011
23403991514cyclictest0-21swapper/907:49:3811
23403991513cyclictest28714-21diskstats11:40:1311
23403991513cyclictest18839-21chrt08:30:1211
2340299154cyclictest0-21swapper/811:35:0110
23402991514cyclictest24686-21users07:50:3210
23402991513cyclictest7533-21snmp_rack3slot908:57:0410
23402991513cyclictest4359-21iostat11:50:1810
23402991513cyclictest25465-21df_inode12:20:1410
23402991513cyclictest2361-21snmp_easybox.os11:05:1410
23402991513cyclictest14804-21snmp_rack3slot909:06:3810
2339799151cyclictest0-21swapper/609:31:008
23390991514cyclictest12797-21latency_hist11:20:017
2338999150cyclictest0-21swapper/408:45:136
2338999150cyclictest0-21swapper/408:40:016
23377991513cyclictest2406-21snmpd12:11:294
2337799150cyclictest0-21swapper/207:20:114
23367991514cyclictest3066350irq/52-eth0-rx-09:55:161
2336799150cyclictest0-21swapper/108:20:321
2336799150cyclictest0-21swapper/107:35:281
2336799150cyclictest0-21swapper/107:10:171
2336399154cyclictest0-21swapper/009:15:010
23363991514cyclictest18764-21turbostat09:15:000
13450150irq/18-ehci_hcd0-21swapper/011:40:260
13450150irq/18-ehci_hcd0-21swapper/007:15:000
23403991413cyclictest0-21swapper/911:30:1511
23403991413cyclictest0-21swapper/911:05:4211
23403991413cyclictest0-21swapper/910:55:0011
23402991412cyclictest8025-21snmp_rack3slot909:42:0310
2339799146cyclictest0-21swapper/607:15:008
2339799140cyclictest23123-21qemu-kvm07:25:168
23390991413cyclictest3550-21qemu-kvm08:23:157
23383991412cyclictest2691-21runrttasks07:21:585
23377991412cyclictest2406-21snmpd10:03:124
2336399141cyclictest0-21swapper/010:24:360
8799130watchdog/110-21swapper/1111:56:513
2340599133cyclictest0-21swapper/1108:45:453
2340599133cyclictest0-21swapper/1107:54:073
2340599130cyclictest0-21swapper/1108:08:103
2340499130cyclictest0-21swapper/1010:10:152
2340399135cyclictest0-21swapper/910:20:0111
2340399134cyclictest0-21swapper/909:35:0011
2340399134cyclictest0-21swapper/908:05:0111
2340399133cyclictest0-21swapper/912:21:1711
23403991312cyclictest20203-21sshd12:11:3711
23403991312cyclictest0-21swapper/908:20:0111
23403991312cyclictest0-21swapper/908:00:2811
2340299134cyclictest0-21swapper/810:45:0110
2340199134cyclictest0-21swapper/710:20:009
2339799133cyclictest0-21swapper/610:51:198
2339799133cyclictest0-21swapper/610:43:138
2339799133cyclictest0-21swapper/610:28:118
2339799133cyclictest0-21swapper/609:46:058
2339799133cyclictest0-21swapper/609:02:248
23397991310cyclictest0-21swapper/607:45:168
2339799131cyclictest14970-21qemu-kvm09:17:078
2339799131cyclictest14969-21qemu-kvm09:21:208
2339099133cyclictest0-21swapper/508:50:187
23390991312cyclictest6878-21cstates11:55:177
23390991312cyclictest17801-21cstates12:10:137
23383991311cyclictest3901-21cpuspeed_turbos07:25:145
23363991312cyclictest31ksoftirqd/008:25:260
23363991312cyclictest12250irq/42-ahci08:09:550
23363991312cyclictest0-21swapper/010:13:110
2336399131cyclictest0-21swapper/010:25:160
13450130irq/18-ehci_hcd0-21swapper/012:00:250
13450130irq/18-ehci_hcd0-21swapper/009:45:230
5999120watchdog/70-21swapper/711:15:199
5999120watchdog/70-21swapper/711:15:189
481120ksoftirqd/50-21swapper/510:40:017
3199120watchdog/30-21swapper/309:53:545
31120ksoftirqd/00-21swapper/008:40:280
2340599124cyclictest0-21swapper/1109:45:013
2340599122cyclictest0-21swapper/1112:36:173
2340599122cyclictest0-21swapper/1109:00:023
2340499123cyclictest0-21swapper/1008:25:012
23404991211cyclictest25739-21sed08:40:012
2340399124cyclictest0-21swapper/911:05:0111
2340399124cyclictest0-21swapper/907:15:0011
23403991211cyclictest0-21swapper/908:50:2011
23403991211cyclictest0-21swapper/907:40:1311
23403991210cyclictest29640-21chrt09:29:1911
2340399121cyclictest0-21swapper/912:35:0011
2340299120cyclictest0-21swapper/810:10:1610
2340199126cyclictest0-21swapper/707:22:299
2340199124cyclictest0-21swapper/709:59:599
2340199123cyclictest0-21swapper/707:55:019
2339799124cyclictest0-21swapper/609:50:008
2339799123cyclictest0-21swapper/612:20:018
2339799122cyclictest0-21swapper/612:00:458
2339799122cyclictest0-21swapper/611:41:008
2339799122cyclictest0-21swapper/609:57:448
2339799122cyclictest0-21swapper/607:33:348
2339799121cyclictest3550-21qemu-kvm08:15:088
2339799121cyclictest3549-21qemu-kvm08:21:148
2339799121cyclictest26373-21qemu-kvm10:35:138
2339799121cyclictest23123-21qemu-kvm07:24:178
2339799121cyclictest14969-21qemu-kvm09:09:148
2339799121cyclictest0-21swapper/611:17:538
2339799121cyclictest0-21swapper/611:17:538
2339799121cyclictest0-21swapper/608:52:138
2339799120cyclictest26374-21qemu-kvm10:10:148
2339799120cyclictest0-21swapper/611:25:008
2339799120cyclictest0-21swapper/611:24:598
2339799120cyclictest0-21swapper/610:17:118
2339799120cyclictest0-21swapper/609:40:578
2339799120cyclictest0-21swapper/609:11:508
23390991211cyclictest21121-21fschecks_count10:45:157
2338999129cyclictest0-21swapper/409:00:166
2338999123cyclictest0-21swapper/412:10:186
2338399122cyclictest0-21swapper/311:10:175
2338399122cyclictest0-21swapper/311:10:175
2338399122cyclictest0-21swapper/307:50:185
2337799122cyclictest0-21swapper/209:45:164
23377991210cyclictest2406-21snmpd08:54:124
2336799123cyclictest0-21swapper/108:15:151
2336799122cyclictest0-21swapper/107:55:211
23363991211cyclictest31ksoftirqd/009:26:440
23363991211cyclictest2390-21nscd10:15:250
13450120irq/18-ehci_hcd0-21swapper/012:05:250
901110ksoftirqd/110-21swapper/1110:40:013
901110ksoftirqd/110-21swapper/1108:40:293
761110ksoftirqd/90-21swapper/911:15:1611
761110ksoftirqd/90-21swapper/911:15:1611
481110ksoftirqd/50-21swapper/508:40:297
411110ksoftirqd/40-21swapper/411:15:166
411110ksoftirqd/40-21swapper/411:15:166
411110ksoftirqd/40-21swapper/408:40:286
3899110watchdog/40-21swapper/412:06:586
341110ksoftirqd/30-21swapper/308:40:135
31110ksoftirqd/00-21swapper/011:15:170
31110ksoftirqd/00-21swapper/011:15:160
31110ksoftirqd/00-21swapper/008:35:240
2340599117cyclictest0-21swapper/1110:49:143
2340599113cyclictest0-21swapper/1111:03:443
2340599113cyclictest0-21swapper/1110:56:423
2340599113cyclictest0-21swapper/1110:22:133
2340599113cyclictest0-21swapper/1109:27:293
2340599113cyclictest0-21swapper/1109:12:403
2340399119cyclictest4279-21snmp_rack3slot909:35:3611
2340399119cyclictest18520-21chrt09:11:4311
2340399114cyclictest0-21swapper/909:25:0011
23403991110cyclictest8525-1kworker/9:2H08:55:1411
2340399110cyclictest0-21swapper/907:10:2211
2340199115cyclictest0-21swapper/712:24:309
2340199114cyclictest0-21swapper/711:25:029
2340199113cyclictest0-21swapper/710:40:019
2340199113cyclictest0-21swapper/710:37:599
2340199113cyclictest0-21swapper/710:32:149
2340199113cyclictest0-21swapper/710:23:439
2340199113cyclictest0-21swapper/709:51:429
2340199113cyclictest0-21swapper/709:49:579
2340199113cyclictest0-21swapper/709:45:019
2340199113cyclictest0-21swapper/709:16:409
2340199113cyclictest0-21swapper/708:55:399
2340199113cyclictest0-21swapper/708:20:149
23401991110cyclictest4408-21perl12:35:159
2340199111cyclictest0-21swapper/711:56:459
2340199111cyclictest0-21swapper/710:07:069
2340199111cyclictest0-21swapper/708:14:089
2340199111cyclictest0-21swapper/708:14:089
2339799113cyclictest0-21swapper/607:15:008
2339799111cyclictest23123-21qemu-kvm07:35:138
2339799111cyclictest23122-21qemu-kvm07:40:038
2339799111cyclictest14969-21qemu-kvm09:25:078
2339799111cyclictest0-21swapper/612:20:248
2339799111cyclictest0-21swapper/612:10:318
2339799111cyclictest0-21swapper/611:55:248
2339799111cyclictest0-21swapper/611:45:158
2339799111cyclictest0-21swapper/611:35:368
2339799111cyclictest0-21swapper/611:30:148
2339799111cyclictest0-21swapper/611:10:318
2339799111cyclictest0-21swapper/611:10:308
2339799111cyclictest0-21swapper/611:00:138
2339799111cyclictest0-21swapper/610:45:168
2339799111cyclictest0-21swapper/610:00:168
2339799110cyclictest9582-21cut10:30:128
2339799110cyclictest5960-21tail08:55:188
2339799110cyclictest5015-21missed_timers12:35:218
2339799110cyclictest3550-21qemu-kvm08:10:258
2339799110cyclictest3550-21qemu-kvm08:10:248
2339799110cyclictest30727-21munin-node08:00:198
2339799110cyclictest27724-21grep10:55:118
2339799110cyclictest2715-21tr11:05:158
2339799110cyclictest25932-21tr07:55:158
2339799110cyclictest2536-21latency_hist11:50:018
2339799110cyclictest23629-21cpuspeed_turbos10:05:138
2339799110cyclictest0-21swapper/612:30:158
2339799110cyclictest0-21swapper/612:25:088
2339799110cyclictest0-21swapper/612:06:458
2339799110cyclictest0-21swapper/609:35:128
2339799110cyclictest0-21swapper/608:35:028
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional