You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 06:12
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3.osadl.org (updated Fri Dec 05, 2025 00:46:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12132210894sleep90-21swapper/919:07:2811
12089210793sleep80-21swapper/819:06:5110
123702106101sleep10-21swapper/119:09:571
12013210591sleep100-21swapper/1019:05:532
12028210491sleep110-21swapper/1119:06:073
12101210391sleep60-21swapper/619:07:028
122501020irq/42-ahci0-21swapper/219:08:574
3066350920irq/52-eth0-rx-0-21swapper/419:05:116
3066450850irq/53-eth0-tx-0-21swapper/319:05:285
3066350690irq/52-eth0-rx-0-21swapper/519:06:427
1200226554sleep00-21swapper/019:05:440
1201026453sleep70-21swapper/719:05:509
12517993013cyclictest811rcuc/1021:50:012
1251799258cyclictest831ksoftirqd/1020:31:162
1251799258cyclictest831ksoftirqd/1019:57:002
1247699230cyclictest0-21swapper/019:10:270
12476992120cyclictest31ksoftirqd/022:20:000
12517991817cyclictest831ksoftirqd/1022:25:162
12517991817cyclictest831ksoftirqd/1020:15:132
12517991817cyclictest0-21swapper/1022:02:242
12517991817cyclictest0-21swapper/1019:25:322
1251799181cyclictest831ksoftirqd/1023:35:012
1251799181cyclictest831ksoftirqd/1023:15:182
1251799181cyclictest831ksoftirqd/1022:30:242
1251799181cyclictest831ksoftirqd/1022:20:162
1251799181cyclictest831ksoftirqd/1020:35:162
1251799181cyclictest831ksoftirqd/1019:35:322
1251799181cyclictest831ksoftirqd/1019:20:152
1251799181cyclictest831ksoftirqd/1000:25:162
1251799181cyclictest831ksoftirqd/1000:15:012
1251799181cyclictest831ksoftirqd/1000:15:012
1251799181cyclictest831ksoftirqd/1000:10:152
1251799181cyclictest811rcuc/1023:40:302
1251799181cyclictest3066350irq/52-eth0-rx-22:10:162
1251799181cyclictest12250irq/42-ahci22:55:282
1251799181cyclictest12250irq/42-ahci21:09:422
1251799181cyclictest12250irq/42-ahci20:49:262
1251799180cyclictest129672sleep1019:10:052
1251699181cyclictest0-21swapper/921:50:1511
1251699181cyclictest0-21swapper/921:30:3911
1251699181cyclictest0-21swapper/921:30:3911
1251699181cyclictest0-21swapper/920:43:4611
12515991816cyclictest9099-21perl20:30:2210
12515991816cyclictest557-21snmp_easybox.os20:20:1210
12515991816cyclictest32636-21snmp_rack3slot919:35:1310
12515991816cyclictest30175-21df_inode20:15:1310
12515991816cyclictest27992-21snmp_rack3slot922:25:1610
12515991816cyclictest25299-21snmp_rack3slot920:51:1610
12515991816cyclictest22401-21df_inode19:20:1210
12515991816cyclictest21508-21diskstats23:45:1510
12515991816cyclictest16152-21munin-node21:25:1510
12515991816cyclictest15834-21snmp_rack3slot922:51:2510
12515991816cyclictest11603-21snmp_rack3slot922:00:3610
12515991816cyclictest11324-21iostat_ios23:30:2110
12514991816cyclictest842650kworker/u24:120:20:209
1247699180cyclictest0-21swapper/019:28:030
5999170watchdog/70-21swapper/720:45:039
1251899170cyclictest0-21swapper/1119:10:333
12517991716cyclictest831ksoftirqd/1000:00:322
12517991716cyclictest22867-21kworker/10:121:50:182
12517991716cyclictest20230-21snmp_rack3slot921:30:202
12517991716cyclictest20230-21snmp_rack3slot921:30:202
12517991716cyclictest0-21swapper/1022:15:152
12517991716cyclictest0-21swapper/1021:40:272
12517991716cyclictest0-21swapper/1000:05:122
1251799170cyclictest88812sleep1023:25:342
1251799170cyclictest83912sleep1021:15:082
1251799170cyclictest83912sleep1021:15:082
1251799170cyclictest831ksoftirqd/1023:45:002
1251799170cyclictest831ksoftirqd/1023:20:132
1251799170cyclictest831ksoftirqd/1021:35:172
1251799170cyclictest831ksoftirqd/1021:10:112
1251799170cyclictest831ksoftirqd/1021:00:232
1251799170cyclictest831ksoftirqd/1020:25:142
1251799170cyclictest831ksoftirqd/1020:05:242
1251799170cyclictest831ksoftirqd/1019:15:152
1251799170cyclictest831ksoftirqd/1000:30:022
1251799170cyclictest831ksoftirqd/1000:30:012
1251799170cyclictest811rcuc/1020:10:142
1251799170cyclictest79852sleep1021:55:592
1251799170cyclictest78872chrt22:40:282
1251799170cyclictest53632sleep1019:40:202
1251799170cyclictest42552sleep1022:35:302
1251799170cyclictest35682sleep1020:22:492
1251799170cyclictest309692sleep1023:59:142
1251799170cyclictest3066350irq/52-eth0-rx-22:50:082
1251799170cyclictest3066350irq/52-eth0-rx-21:20:152
1251799170cyclictest3066350irq/52-eth0-rx-19:30:242
1251799170cyclictest3066350irq/52-eth0-rx-00:35:342
1251799170cyclictest306202sleep1023:14:322
1251799170cyclictest271592sleep1023:51:492
1251799170cyclictest260922sleep1020:55:052
1251799170cyclictest254792sleep1020:53:402
1251799170cyclictest239212sleep1023:05:072
1251799170cyclictest231662chrt23:01:442
1251799170cyclictest213262sleep1020:02:592
1251799170cyclictest156962sleep1021:25:082
1251799170cyclictest153212sleep1022:06:022
1251799170cyclictest146702sleep1000:20:172
1251799170cyclictest140012sleep1019:52:462
1251799170cyclictest127042sleep1023:32:472
1251799170cyclictest12250irq/42-ahci22:45:012
1251799170cyclictest12250irq/42-ahci19:45:072
1251699171cyclictest0-21swapper/920:55:1311
1251699170cyclictest761ksoftirqd/919:55:1911
1251699170cyclictest20825-21runrttasks19:17:2111
1251699170cyclictest0-21swapper/920:50:2111
1251699170cyclictest0-21swapper/919:47:3011
12515991715cyclictest6383-21perl21:55:1810
12515991715cyclictest5776-21ntp_states19:40:2310
12515991715cyclictest32253-21snmp_easybox.os21:45:2810
12515991715cyclictest31848-21snmp_rack3slot922:30:1910
12515991715cyclictest28762-21perl23:10:2010
12515991715cyclictest27818-21snmp_rack3slot921:40:1610
12515991715cyclictest26901-21perl20:10:1710
12515991715cyclictest26365-21snmp_easybox.os23:50:2410
12515991715cyclictest22768-21df20:05:1410
12515991715cyclictest20428-21perl21:30:2110
12515991715cyclictest20428-21perl21:30:2010
12515991715cyclictest17252-21iostat_ios19:15:1510
12515991715cyclictest11892-21diskstats19:50:1610
12515991715cyclictest10170-21snmp_rack3slot922:45:1610
12507991715cyclictest4082-21qemu-kvm21:08:188
12504991715cyclictest5945-21sensors19:40:256
1249299173cyclictest0-21swapper/223:20:324
1249299173cyclictest0-21swapper/221:51:584
1249299173cyclictest0-21swapper/221:34:584
1249299173cyclictest0-21swapper/221:34:584
1249299173cyclictest0-21swapper/200:31:034
1249299173cyclictest0-21swapper/200:31:024
12492991715cyclictest2406-21snmpd22:05:124
12492991715cyclictest2406-21snmpd00:37:284
1249299171cyclictest0-21swapper/220:52:554
1249299171cyclictest0-21swapper/200:17:064
1249299171cyclictest0-21swapper/200:17:064
12476991716cyclictest3066350irq/52-eth0-rx-22:20:060
12476991716cyclictest0-21swapper/021:18:440
12476991716cyclictest0-21swapper/021:18:440
1251899164cyclictest0-21swapper/1122:40:223
12518991614cyclictest3452-21chrt20:21:143
1251899160cyclictest0-21swapper/1122:25:203
1251899160cyclictest0-21swapper/1121:20:143
1251899160cyclictest0-21swapper/1119:47:003
12516991615cyclictest20090-21taskset23:44:3411
12516991615cyclictest1181-21chrt22:31:5211
12516991615cyclictest0-21swapper/922:37:0411
12516991615cyclictest0-21swapper/922:25:4611
12516991615cyclictest0-21swapper/900:00:1911
12516991614cyclictest6763-21iostat_ios21:55:2211
12515991614cyclictest8904-21snmp_rack3slot923:25:4910
12515991614cyclictest6924-21perl22:40:1810
12515991614cyclictest682-21iostat_ios00:00:2110
12515991614cyclictest30119-21sensors_volt23:55:2910
12515991614cyclictest29963-21perl19:30:1510
12515991614cyclictest25823-21perl00:35:2110
12515991614cyclictest20894-21snmp_rack3slot922:15:1410
12515991614cyclictest16637-21snmp_rack3slot900:21:2510
12515991614cyclictest12922-21snmp_rack3slot921:20:1410
1251599160cyclictest0-21swapper/821:10:1410
12514991614cyclictest0-21swapper/722:29:559
1250799165cyclictest0-21swapper/620:55:018
12505991614cyclictest15534-21qemu-kvm22:09:007
12504991614cyclictest18809-21sensors22:55:286
1249799161cyclictest0-21swapper/320:46:595
1249799161cyclictest0-21swapper/300:39:575
1249799160cyclictest0-21swapper/320:05:595
1249799160cyclictest0-21swapper/319:40:215
1249299162cyclictest0-21swapper/219:36:144
1249299162cyclictest0-21swapper/219:22:514
1249299162cyclictest0-21swapper/200:09:194
12492991614cyclictest2406-21snmpd23:32:134
12492991614cyclictest2406-21snmpd20:58:274
12492991614cyclictest2406-21snmpd20:07:424
1249299161cyclictest0-21swapper/220:11:544
1249299160cyclictest0-21swapper/222:20:384
1249299160cyclictest0-21swapper/200:10:224
12476991614cyclictest64492sleep020:25:280
1251899158cyclictest7220-21chrt20:28:033
1251899156cyclictest0-21swapper/1119:33:363
1251899152cyclictest0-21swapper/1123:32:523
1251899150cyclictest0-21swapper/1123:36:553
1251899150cyclictest0-21swapper/1122:13:193
1251899150cyclictest0-21swapper/1121:36:493
1251899150cyclictest0-21swapper/1121:09:423
1251899150cyclictest0-21swapper/1119:37:563
1251899150cyclictest0-21swapper/1119:15:133
1251899150cyclictest0-21swapper/1100:23:423
12516991514cyclictest16411-21df19:15:0111
12516991514cyclictest0-21swapper/921:45:2411
12516991514cyclictest0-21swapper/920:30:3011
12516991514cyclictest0-21swapper/919:25:2011
12516991513cyclictest21851-21iostat_ios23:00:2011
12515991513cyclictest9515-21perl21:15:1710
12515991513cyclictest9515-21perl21:15:1610
12515991513cyclictest2145-21perl21:05:1710
12515991513cyclictest17395-21df22:55:1510
12515991513cyclictest13838-21snmp_rack3slot919:10:1510
12515991513cyclictest13791-21snmp_rack3slot923:35:1410
12515991510cyclictest0-21swapper/822:20:3310
1251599150cyclictest0-21swapper/821:38:3510
1251599150cyclictest0-21swapper/820:47:3910
1251599150cyclictest0-21swapper/819:45:4710
1251599150cyclictest0-21swapper/800:08:2110
12505991514cyclictest5672-21idleruntime20:25:187
12505991514cyclictest2922-21cstates23:20:137
12505991514cyclictest28299-21python20:55:297
12504991513cyclictest19207-21sensors23:40:266
1250499150cyclictest0-21swapper/419:10:206
1249799150cyclictest0-21swapper/321:05:265
1249299153cyclictest0-21swapper/223:11:494
1249299153cyclictest0-21swapper/220:43:524
1249299152cyclictest0-21swapper/219:46:054
1249299151cyclictest0-21swapper/219:34:164
1249299150cyclictest0-21swapper/223:27:344
1249299150cyclictest0-21swapper/221:02:004
1248499150cyclictest0-21swapper/120:05:251
1247699150cyclictest0-21swapper/019:15:200
481140ksoftirqd/50-21swapper/522:15:047
271140ksoftirqd/20-21swapper/221:13:544
271140ksoftirqd/20-21swapper/220:48:084
1251899148cyclictest0-21swapper/1121:25:263
1251899147cyclictest0-21swapper/1119:50:143
1251899146cyclictest0-21swapper/1120:47:213
1251899146cyclictest0-21swapper/1119:20:223
1251899145cyclictest0-21swapper/1121:31:423
1251899145cyclictest0-21swapper/1121:31:413
1251899144cyclictest0-21swapper/1120:00:013
1251899143cyclictest0-21swapper/1121:48:203
1251899142cyclictest0-21swapper/1122:50:213
1251899142cyclictest0-21swapper/1122:02:113
1251899142cyclictest0-21swapper/1120:42:483
1251899142cyclictest0-21swapper/1120:07:563
1251899142cyclictest0-21swapper/1100:19:543
1251899142cyclictest0-21swapper/1100:19:543
1251899140cyclictest0-21swapper/1123:15:573
1251899140cyclictest0-21swapper/1123:05:243
1251899140cyclictest0-21swapper/1122:45:193
1251899140cyclictest0-21swapper/1122:32:423
1251899140cyclictest0-21swapper/1122:18:143
1251899140cyclictest0-21swapper/1122:05:383
1251899140cyclictest0-21swapper/1121:12:573
1251899140cyclictest0-21swapper/1120:51:453
1251899140cyclictest0-21swapper/1120:10:333
1251899140cyclictest0-21swapper/1100:34:343
1251899140cyclictest0-21swapper/1100:34:343
1251899140cyclictest0-21swapper/1100:12:263
12517991413cyclictest831ksoftirqd/1020:40:182
1251699143cyclictest0-21swapper/921:00:0111
12516991413cyclictest2192-1kworker/9:2H23:26:0411
12516991413cyclictest0-21swapper/922:00:3111
12516991413cyclictest0-21swapper/920:35:1511
12516991413cyclictest0-21swapper/920:00:2511
1251599148cyclictest0-21swapper/820:35:3310
1251599148cyclictest0-21swapper/800:31:2910
1251599148cyclictest0-21swapper/800:31:2810
1251599148cyclictest0-21swapper/800:12:0110
1251599147cyclictest0-21swapper/823:07:4310
1251599147cyclictest0-21swapper/822:05:4410
1251599147cyclictest0-21swapper/821:00:2210
1251599144cyclictest0-21swapper/823:20:0110
1251599144cyclictest0-21swapper/823:00:0210
1251599141cyclictest0-21swapper/823:40:2010
1251599140cyclictest0-21swapper/823:24:5110
1251599140cyclictest0-21swapper/822:11:2910
1251599140cyclictest0-21swapper/821:51:0110
1251599140cyclictest0-21swapper/820:42:0010
1251599140cyclictest0-21swapper/820:03:0010
1250799145cyclictest0-21swapper/622:05:228
12505991413cyclictest6971-21cstates00:10:147
12505991413cyclictest14870-21cpuspeed_turbos19:55:147
1249799141cyclictest0-21swapper/321:10:325
1249799140cyclictest0-21swapper/322:10:575
1249299145cyclictest0-21swapper/222:50:214
1249299143cyclictest0-21swapper/222:34:194
1249299143cyclictest0-21swapper/222:03:264
1249299143cyclictest0-21swapper/221:46:044
1249299143cyclictest0-21swapper/221:21:014
1249299143cyclictest0-21swapper/220:21:324
1249299143cyclictest0-21swapper/200:27:004
1249299143cyclictest0-21swapper/200:20:184
1249299142cyclictest0-21swapper/221:40:384
12492991412cyclictest2406-21snmpd21:59:574
1249299141cyclictest0-21swapper/221:08:144
1249299141cyclictest0-21swapper/219:17:244
1248499140cyclictest0-21swapper/121:15:181
1248499140cyclictest0-21swapper/121:15:171
1247699143cyclictest0-21swapper/020:20:340
761130ksoftirqd/90-21swapper/923:20:2811
621130ksoftirqd/70-21swapper/723:35:299
621130ksoftirqd/70-21swapper/723:22:449
621130ksoftirqd/70-21swapper/722:40:029
621130ksoftirqd/70-21swapper/722:23:099
621130ksoftirqd/70-21swapper/722:15:059
621130ksoftirqd/70-21swapper/721:13:549
621130ksoftirqd/70-21swapper/720:56:449
551130ksoftirqd/64082-21qemu-kvm21:13:548
551130ksoftirqd/615533-21qemu-kvm22:15:048
551130ksoftirqd/615526-21qemu-kvm22:40:018
551130ksoftirqd/60-21swapper/621:54:528
551130ksoftirqd/60-21swapper/621:25:518
481130ksoftirqd/54082-21qemu-kvm21:25:517
481130ksoftirqd/50-21swapper/522:40:017
481130ksoftirqd/50-21swapper/521:57:427
481130ksoftirqd/50-21swapper/520:36:047
481130ksoftirqd/50-21swapper/500:17:427
481130ksoftirqd/50-21swapper/500:17:427
4599130watchdog/50-21swapper/521:51:277
411130ksoftirqd/40-21swapper/423:13:106
411130ksoftirqd/40-21swapper/422:24:436
411130ksoftirqd/40-21swapper/421:54:516
411130ksoftirqd/40-21swapper/421:13:536
411130ksoftirqd/40-21swapper/420:52:206
341130ksoftirqd/30-21swapper/321:54:515
341130ksoftirqd/30-21swapper/320:53:385
341130ksoftirqd/30-21swapper/320:36:035
31130ksoftirqd/00-21swapper/021:57:430
31130ksoftirqd/00-21swapper/021:54:510
31130ksoftirqd/00-21swapper/021:31:500
31130ksoftirqd/00-21swapper/021:31:500
31130ksoftirqd/00-21swapper/021:25:520
31130ksoftirqd/00-21swapper/020:56:440
31130ksoftirqd/00-21swapper/020:53:380
271130ksoftirqd/20-21swapper/223:41:094
271130ksoftirqd/20-21swapper/220:30:404
201130ksoftirqd/10-21swapper/123:35:291
201130ksoftirqd/10-21swapper/122:40:021
201130ksoftirqd/10-21swapper/122:15:051
201130ksoftirqd/10-21swapper/121:54:511
201130ksoftirqd/10-21swapper/121:13:531
1251899136cyclictest0-21swapper/1122:36:543
1251899136cyclictest0-21swapper/1119:57:053
1251899133cyclictest0-21swapper/1123:46:283
1251899133cyclictest0-21swapper/1123:25:273
1251899133cyclictest0-21swapper/1121:56:263
1251899133cyclictest0-21swapper/1121:02:183
1251899132cyclictest0-21swapper/1123:12:123
1251899132cyclictest0-21swapper/1122:57:263
1251899132cyclictest0-21swapper/1100:25:423
1251899131cyclictest0-21swapper/1123:20:573
1251899131cyclictest0-21swapper/1120:30:423
1251899130cyclictest0-21swapper/1120:58:133
1251699133cyclictest0-21swapper/923:45:0011
12516991312cyclictest741rcuc/922:20:0211
12516991312cyclictest32219-21taskset20:17:0511
12516991312cyclictest0-21swapper/923:50:2011
12516991312cyclictest0-21swapper/922:40:2211
12516991312cyclictest0-21swapper/922:15:1311
12516991312cyclictest0-21swapper/921:40:2511
1251699130cyclictest0-21swapper/923:10:3311
1251599139cyclictest0-21swapper/800:26:3310
1251599137cyclictest0-21swapper/822:36:0710
1251599136cyclictest25182-21snmp_easybox.os19:25:1110
1251599131cyclictest0-21swapper/819:55:2110
1251599130cyclictest0-21swapper/820:55:1110
1250799134cyclictest0-21swapper/623:35:198
1250799133cyclictest0-21swapper/620:26:268
12505991312cyclictest9287-21cstates22:00:127
1249799133cyclictest0-21swapper/321:15:205
1249799133cyclictest0-21swapper/321:15:195
1249799133cyclictest0-21swapper/300:33:245
1249799133cyclictest0-21swapper/300:33:245
1249799133cyclictest0-21swapper/300:13:115
1249799132cyclictest0-21swapper/323:38:095
1249799132cyclictest0-21swapper/320:00:545
1249799132cyclictest0-21swapper/319:38:585
1249799132cyclictest0-21swapper/300:25:565
1249799132cyclictest0-21swapper/300:21:085
1249799132cyclictest0-21swapper/300:15:105
1249799132cyclictest0-21swapper/300:15:105
1249799130cyclictest0-21swapper/323:44:185
1249799130cyclictest0-21swapper/323:04:055
1249799130cyclictest0-21swapper/319:55:325
1249299134cyclictest0-21swapper/219:10:214
1249299133cyclictest0-21swapper/223:55:034
1249299133cyclictest0-21swapper/223:50:294
1249299133cyclictest0-21swapper/223:15:594
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional