You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 22:20
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3.osadl.org (updated Wed Nov 05, 2025 12:46:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
122501120irq/42-ahci0-21swapper/207:09:094
7551211195sleep30-21swapper/307:06:415
7647211096sleep80-21swapper/807:07:5810
122501090irq/42-ahci0-21swapper/107:08:591
7573210593sleep70-21swapper/707:06:579
7466210190sleep110-21swapper/1107:05:363
784829792sleep100-21swapper/1007:10:002
12250940irq/42-ahci0-21swapper/507:05:197
780028069sleep40-21swapper/407:09:186
755427264sleep60-21swapper/607:06:448
754226351sleep90-21swapper/907:06:3411
748525955sleep00-21swapper/007:05:500
795499330cyclictest0-21swapper/007:20:410
7954993114cyclictest0-21swapper/009:17:410
7954993114cyclictest0-21swapper/009:17:400
800399225cyclictest12250irq/42-ahci07:20:352
7954992119cyclictest0-21swapper/007:16:330
800299203cyclictest761ksoftirqd/912:32:5611
800299203cyclictest761ksoftirqd/912:25:2211
800299203cyclictest761ksoftirqd/912:25:2211
800299203cyclictest761ksoftirqd/912:20:3211
800299203cyclictest761ksoftirqd/912:11:1011
800299203cyclictest761ksoftirqd/912:03:1111
800299203cyclictest761ksoftirqd/911:56:0911
800299203cyclictest761ksoftirqd/911:52:2711
800299203cyclictest761ksoftirqd/911:46:1711
800299203cyclictest761ksoftirqd/911:41:5611
800299203cyclictest761ksoftirqd/911:36:3411
800299203cyclictest761ksoftirqd/911:32:5311
800299203cyclictest761ksoftirqd/911:25:3811
800299203cyclictest761ksoftirqd/911:22:1511
800299203cyclictest761ksoftirqd/911:07:3711
800299203cyclictest761ksoftirqd/910:46:2411
800299203cyclictest761ksoftirqd/910:44:3811
800299203cyclictest761ksoftirqd/910:30:4211
800299203cyclictest761ksoftirqd/910:26:1811
800299203cyclictest761ksoftirqd/910:21:2611
800299203cyclictest761ksoftirqd/910:15:1811
800299203cyclictest761ksoftirqd/910:10:2911
800299203cyclictest761ksoftirqd/909:54:0411
800299203cyclictest761ksoftirqd/909:40:1811
800299203cyclictest761ksoftirqd/909:37:1911
800299203cyclictest761ksoftirqd/909:26:0011
800299203cyclictest761ksoftirqd/909:23:4111
800299203cyclictest761ksoftirqd/909:18:1111
800299203cyclictest761ksoftirqd/909:18:1011
800299203cyclictest761ksoftirqd/909:10:0911
800299203cyclictest761ksoftirqd/909:05:4011
800299203cyclictest761ksoftirqd/909:01:5811
800299203cyclictest761ksoftirqd/908:55:5611
800299203cyclictest761ksoftirqd/908:45:2411
800299203cyclictest761ksoftirqd/908:41:1011
800299203cyclictest761ksoftirqd/908:30:3911
800299203cyclictest761ksoftirqd/908:28:1511
800299203cyclictest761ksoftirqd/908:10:2511
800299203cyclictest761ksoftirqd/908:03:1211
800299203cyclictest761ksoftirqd/907:54:1911
800299203cyclictest761ksoftirqd/907:41:2911
800299203cyclictest761ksoftirqd/907:39:1211
800299203cyclictest761ksoftirqd/907:34:3811
800299203cyclictest761ksoftirqd/907:26:2611
800299203cyclictest761ksoftirqd/907:21:3411
800299203cyclictest761ksoftirqd/907:17:1011
800299203cyclictest761ksoftirqd/907:10:1611
800299192cyclictest761ksoftirqd/912:35:4011
800299192cyclictest761ksoftirqd/912:15:4811
800299192cyclictest761ksoftirqd/912:07:2511
800299192cyclictest761ksoftirqd/911:16:4911
800299192cyclictest761ksoftirqd/911:13:1211
800299192cyclictest761ksoftirqd/911:02:5911
800299192cyclictest761ksoftirqd/910:55:0211
800299192cyclictest761ksoftirqd/910:50:3511
800299192cyclictest761ksoftirqd/910:35:0211
800299192cyclictest761ksoftirqd/910:05:2411
800299192cyclictest761ksoftirqd/910:01:5611
800299192cyclictest761ksoftirqd/909:58:3211
800299192cyclictest761ksoftirqd/909:46:0111
800299192cyclictest761ksoftirqd/909:31:2111
800299192cyclictest761ksoftirqd/909:31:2111
800299192cyclictest761ksoftirqd/908:50:5911
800299192cyclictest761ksoftirqd/908:35:4911
800299192cyclictest761ksoftirqd/908:24:0711
800299192cyclictest761ksoftirqd/908:16:1311
800299192cyclictest761ksoftirqd/908:07:3611
800299192cyclictest761ksoftirqd/907:56:2311
800299192cyclictest761ksoftirqd/907:45:0311
8003991817cyclictest811rcuc/1012:29:112
8003991817cyclictest811rcuc/1012:29:112
8003991817cyclictest0-21swapper/1007:31:042
800399181cyclictest3066350irq/52-eth0-rx-08:15:122
800399181cyclictest12250irq/42-ahci08:23:002
8001991816cyclictest9675-21df_inode11:35:1310
8001991816cyclictest8367-21df_inode09:20:1310
8001991816cyclictest8068-21snmp_rack3slot911:31:5210
8001991816cyclictest701-21iostat07:40:1910
8001991816cyclictest6770-21snmp_easybox.os10:45:2810
8001991816cyclictest3549-21ntp_states11:25:2210
8001991816cyclictest3412-21snmp_rack3slot907:45:1210
8001991816cyclictest32626-21snmp_easybox.os09:10:1210
8001991816cyclictest31859-21irqstats11:20:1910
8001991816cyclictest29897-21snmp_rack3slot909:05:1410
8001991816cyclictest29174-21sensors_temp12:00:2810
8001991816cyclictest24862-21snmp_easybox.os08:15:1310
8001991816cyclictest2412-21hddtemp_smartct10:40:1610
8001991816cyclictest23416-21snmp_rack3slot910:25:1510
8001991816cyclictest21518-21snmp_rack3slot907:25:1510
8001991816cyclictest20516-21iostat_ios10:20:2010
8001991816cyclictest18657-21perl08:05:1710
8001991816cyclictest17691-21snmp_rack3slot907:20:1310
8001991816cyclictest15818-21ntp_states08:00:2110
8001991816cyclictest15723-21snmp_rack3slot912:26:5210
8001991816cyclictest15723-21snmp_rack3slot912:26:5110
8001991816cyclictest15679-21snmp_rack3slot909:30:1810
8001991816cyclictest15679-21snmp_rack3slot909:30:1710
8001991816cyclictest15302-21snmp_rack3slot911:41:2710
8001991816cyclictest13978-21snmp_rack3slot907:15:1710
8001991816cyclictest12785-21snmp_easybox.os08:40:2610
795499180cyclictest0-21swapper/007:29:230
8003991716cyclictest4812-21kworker/10:112:20:022
8003991716cyclictest3090-1kworker/10:2H12:20:022
800399170cyclictest97892sleep1008:35:342
800399170cyclictest95772sleep1007:54:172
800399170cyclictest75692sleep1010:46:002
800399170cyclictest60682sleep1009:15:262
800399170cyclictest60682sleep1009:15:252
800399170cyclictest58572sleep1010:00:192
800399170cyclictest47692sleep1012:11:202
800399170cyclictest45232sleep1011:28:172
800399170cyclictest4122sleep1010:39:112
800399170cyclictest40632chrt10:42:482
800399170cyclictest321962sleep1009:51:292
800399170cyclictest317872sleep1009:07:372
800399170cyclictest308322sleep1007:37:022
800399170cyclictest3066350irq/52-eth0-rx-11:45:272
800399170cyclictest3066350irq/52-eth0-rx-08:55:172
800399170cyclictest299572sleep1012:02:452
800399170cyclictest288222sleep1011:15:262
800399170cyclictest285742sleep1009:46:542
800399170cyclictest282362sleep1009:03:482
800399170cyclictest253622sleep1010:25:462
800399170cyclictest249962sleep1009:42:412
800399170cyclictest229852sleep1011:10:032
800399170cyclictest225412sleep1008:10:172
800399170cyclictest225092sleep1011:50:422
800399170cyclictest219392chrt10:23:442
800399170cyclictest214342sleep1009:38:402
800399170cyclictest207792sleep1008:51:352
800399170cyclictest205362sleep1008:08:422
800399170cyclictest19572chrt07:41:022
800399170cyclictest193472sleep1011:05:012
800399170cyclictest193272sleep1012:31:362
800399170cyclictest173162sleep1008:49:112
800399170cyclictest165762sleep1008:00:302
800399170cyclictest156892chrt11:00:012
800399170cyclictest146982sleep1010:14:202
800399170cyclictest135432sleep1008:42:092
800399170cyclictest131792sleep1007:58:402
800399170cyclictest12250irq/42-ahci11:31:362
800399170cyclictest12250irq/42-ahci10:18:432
800399170cyclictest12250irq/42-ahci10:05:302
800399170cyclictest12250irq/42-ahci09:25:122
800399170cyclictest12250irq/42-ahci09:11:132
800399170cyclictest12250irq/42-ahci08:30:122
800399170cyclictest12250irq/42-ahci08:25:352
800399170cyclictest12250irq/42-ahci07:45:352
800399170cyclictest12250irq/42-ahci07:27:072
800399170cyclictest121862sleep1007:15:122
800399170cyclictest118452sleep1011:38:532
800399170cyclictest112932sleep1010:51:482
800399170cyclictest10722sleep1012:06:572
800399170cyclictest104972sleep1007:10:252
8001991715cyclictest9531-21iostat_ios10:05:2010
8001991715cyclictest8133-21snmp_rack3slot908:35:1510
8001991715cyclictest6417-21df_inode12:15:1410
8001991715cyclictest5950-21snmp_easybox.os09:15:2410
8001991715cyclictest5950-21snmp_easybox.os09:15:2410
8001991715cyclictest30929-21iostat_ios09:50:1610
8001991715cyclictest27057-21diskstats10:30:1610
8001991715cyclictest26785-21iostat_ios09:00:1710
8001991715cyclictest26769-21perl09:45:1610
8001991715cyclictest26191-21snmp_rack3slot911:56:2510
8001991715cyclictest25552-21snmp_rack3slot907:30:1410
8001991715cyclictest23419-21snmp_rack3slot909:40:1610
8001991715cyclictest22564-21snmp_rack3slot911:51:2410
8001991715cyclictest21555-21snmp_easybox.os08:55:1310
8001991715cyclictest21446-21snmp_easybox.os11:05:3010
8001991715cyclictest21211-21perl12:35:1510
8001991715cyclictest21165-21snmp_easybox.os08:10:1310
8001991715cyclictest19248-21yum12:30:3510
8001991715cyclictest17962-21snmp_easybox.os08:50:1110
8001991715cyclictest17830-21snmp_easybox.os11:00:2410
8001991715cyclictest16945-21diskstats11:45:1610
8001991715cyclictest15732-21snmp_rack3slot910:15:1310
8001991715cyclictest13637-21iostat10:55:1810
8001991715cyclictest13006-21snmp_rack3slot907:56:2910
8001991715cyclictest12087-21snmp_rack3slot912:21:4810
8001991715cyclictest11642-21snmp_easybox.os10:10:1210
8001991715cyclictest10487-21snmp_easybox.os10:50:2510
7990991715cyclictest12840-21sensors08:40:266
8001991614cyclictest8663-21snmp_rack3slot907:50:2510
8001991614cyclictest4729-21snmp_rack3slot912:10:4810
8001991614cyclictest3401-21yum09:55:3110
8001991614cyclictest32433-21memory12:05:1810
8001991614cyclictest29893-21snmp_easybox.os10:35:1410
8001991614cyclictest29550-21irqstats07:35:2310
8001991614cyclictest28209-21iostat_ios11:15:1910
8001991614cyclictest20506-21snmp_easybox.os09:35:2410
8001991614cyclictest1226-21irqstats08:25:2210
7990991615cyclictest12250irq/42-ahci07:20:356
7976991614cyclictest2406-21snmpd10:11:104
7965991615cyclictest5570-21python08:30:261
7965991615cyclictest3066350irq/52-eth0-rx-08:01:551
8001991513cyclictest5423-21snmp_easybox.os08:30:2410
8001991513cyclictest29540-21perl08:20:1610
7996991514cyclictest10968-21qemu-kvm10:09:008
7990991513cyclictest26494-21sensors07:30:256
7983991513cyclictest4648-21latency_hist11:30:015
8003991413cyclictest32455-21python11:20:232
800199144cyclictest0-21swapper/809:30:0110
800099144cyclictest0-21swapper/710:30:019
800099144cyclictest0-21swapper/709:35:019
800099143cyclictest0-21swapper/709:15:029
7983991412cyclictest737-21timerwakeupswit12:05:275
7983991412cyclictest27908-21latency_hist08:20:005
7983991412cyclictest2691-21runrttasks12:19:525
7983991412cyclictest22527-21cstates09:40:135
7983991412cyclictest21629-21cstates08:10:145
7983991412cyclictest21290-21timerandwakeup10:20:305
7983991412cyclictest18132-21cpuspeed_turbos08:50:125
7983991412cyclictest13382-21switchtime09:25:285
7983991412cyclictest13282-21latency_hist08:00:005
7983991412cyclictest13074-21cpuspeed_turbos12:25:165
7983991412cyclictest13074-21cpuspeed_turbos12:25:165
7983991412cyclictest1202-21iostat08:25:205
797699143cyclictest0-21swapper/209:05:164
796599143cyclictest0-21swapper/107:20:181
7965991413cyclictest5066-21python07:45:261
7965991413cyclictest3066350irq/52-eth0-rx-08:45:221
4599140watchdog/50-21swapper/509:12:477
31140ksoftirqd/00-21swapper/008:35:260
8003991312cyclictest24696-21smartctl11:55:152
8003991312cyclictest152892sleep1011:41:192
8001991311cyclictest9232-21df07:10:1410
8001991311cyclictest25766-21yum11:10:3210
800099134cyclictest0-21swapper/709:40:199
800099133cyclictest0-21swapper/712:39:119
800099133cyclictest0-21swapper/712:19:259
800099133cyclictest0-21swapper/711:52:419
800099133cyclictest0-21swapper/711:28:269
800099133cyclictest0-21swapper/711:08:269
800099133cyclictest0-21swapper/711:04:419
800099133cyclictest0-21swapper/710:53:249
800099133cyclictest0-21swapper/710:14:069
799699134cyclictest0-21swapper/609:05:008
799699134cyclictest0-21swapper/608:39:058
799699133cyclictest0-21swapper/609:08:348
7990991311cyclictest1830-21sensors08:25:266
7983991311cyclictest5643-21cstates11:30:155
7983991311cyclictest10623-21cstates07:55:175
797699134cyclictest0-21swapper/211:10:174
797699133cyclictest0-21swapper/211:25:164
797699133cyclictest0-21swapper/207:25:174
796599134cyclictest0-21swapper/110:40:011
7965991311cyclictest23374-21python08:10:281
795499133cyclictest0-21swapper/010:30:200
795499133cyclictest0-21swapper/008:50:180
7954991311cyclictest21854-21chrt10:22:350
31130ksoftirqd/00-21swapper/009:50:240
31130ksoftirqd/00-21swapper/008:30:240
1699130watchdog/00-21swapper/007:38:510
8003991211cyclictest3090-1kworker/10:2H09:30:202
8003991211cyclictest3090-1kworker/10:2H09:30:192
800399120cyclictest831ksoftirqd/1009:20:142
8001991210cyclictest5856-21iostat_ios10:00:1810
8001991210cyclictest15413-21perl08:45:1610
800099124cyclictest0-21swapper/708:23:349
800099122cyclictest0-21swapper/712:13:109
800099122cyclictest0-21swapper/710:03:399
799699125cyclictest0-21swapper/607:57:028
799699124cyclictest0-21swapper/611:25:008
799699124cyclictest0-21swapper/610:39:598
799699124cyclictest0-21swapper/609:55:018
799699123cyclictest0-21swapper/608:10:008
799699123cyclictest0-21swapper/607:50:208
7996991211cyclictest20493-21qemu-kvm08:08:248
799099124cyclictest0-21swapper/409:35:176
7990991210cyclictest0-21swapper/410:05:136
798399122cyclictest0-21swapper/307:45:175
797699122cyclictest0-21swapper/210:30:204
7976991210cyclictest2406-21snmpd09:40:244
796599122cyclictest0-21swapper/110:05:191
796599122cyclictest0-21swapper/108:35:181
7965991211cyclictest3066350irq/52-eth0-rx-10:45:211
7965991211cyclictest28824-21python11:15:261
795499123cyclictest0-21swapper/010:10:250
795499123cyclictest0-21swapper/008:00:150
795499122cyclictest0-21swapper/008:15:180
795499120cyclictest0-21swapper/011:20:240
795499120cyclictest0-21swapper/007:30:240
271120ksoftirqd/20-21swapper/209:15:284
271120ksoftirqd/20-21swapper/209:15:284
201120ksoftirqd/10-21swapper/109:15:291
201120ksoftirqd/10-21swapper/109:15:281
901110ksoftirqd/110-21swapper/1111:45:213
901110ksoftirqd/110-21swapper/1109:15:293
901110ksoftirqd/110-21swapper/1109:15:283
8003991110cyclictest21459-21smartctl12:35:162
800399110cyclictest34482chrt09:56:052
800399110cyclictest289832sleep1010:30:342
800399110cyclictest150582sleep1010:58:252
800099114cyclictest0-21swapper/711:25:009
800099114cyclictest0-21swapper/709:47:549
800099114cyclictest0-21swapper/709:05:009
800099113cyclictest0-21swapper/712:28:169
800099113cyclictest0-21swapper/712:28:169
800099113cyclictest0-21swapper/712:24:259
800099113cyclictest0-21swapper/712:03:499
800099113cyclictest0-21swapper/711:35:419
800099113cyclictest0-21swapper/709:59:259
800099113cyclictest0-21swapper/709:52:259
800099113cyclictest0-21swapper/707:52:109
800099113cyclictest0-21swapper/707:42:339
800099110cyclictest0-21swapper/710:58:119
799699114cyclictest0-21swapper/612:30:108
799699114cyclictest0-21swapper/607:41:028
799699113cyclictest0-21swapper/610:02:058
799699113cyclictest0-21swapper/609:45:028
799699113cyclictest0-21swapper/607:12:018
799699112cyclictest0-21swapper/611:35:158
799699111cyclictest0-21swapper/611:30:238
799699111cyclictest0-21swapper/610:26:058
799199119cyclictest7715-21qemu-kvm07:17:297
799199119cyclictest20493-21qemu-kvm08:27:037
799199119cyclictest19667-21python08:05:297
799199119cyclictest10968-21qemu-kvm10:17:507
799099113cyclictest0-21swapper/409:20:186
799099113cyclictest0-21swapper/407:35:206
799099112cyclictest0-21swapper/409:50:166
799099111cyclictest0-21swapper/410:50:266
799099110cyclictest0-21swapper/407:55:166
798399113cyclictest0-21swapper/307:15:225
797699115cyclictest0-21swapper/211:37:304
797699113cyclictest0-21swapper/212:10:184
797699113cyclictest0-21swapper/211:50:204
797699113cyclictest0-21swapper/210:50:174
797699113cyclictest0-21swapper/210:45:244
797699113cyclictest0-21swapper/209:45:194
797699112cyclictest0-21swapper/212:20:544
797699111cyclictest0-21swapper/211:23:384
797699111cyclictest0-21swapper/208:09:444
797699111cyclictest0-21swapper/207:58:464
797699110cyclictest12-21rcu_preempt09:10:314
796599119cyclictest0-21swapper/107:40:131
795499119cyclictest2691-21runrttasks09:21:270
795499118cyclictest0-21swapper/010:25:250
795499113cyclictest0-21swapper/012:30:190
795499113cyclictest0-21swapper/010:55:160
795499110cyclictest0-21swapper/010:05:270
5499110migration/66124-21munin-node10:45:208
5499110migration/618288-21munin-node11:45:288
411110ksoftirqd/40-21swapper/411:45:216
3199110watchdog/30-21swapper/307:32:505
31110ksoftirqd/00-21swapper/011:45:200
31110ksoftirqd/00-21swapper/009:35:250
31110ksoftirqd/00-21swapper/007:10:250
1699110watchdog/00-21swapper/012:01:420
1699110watchdog/00-21swapper/011:56:390
1699110watchdog/00-21swapper/011:11:020
14850110irq/18-uhci_hcd0-21swapper/010:45:300
901100ksoftirqd/110-21swapper/1108:08:443
8799100watchdog/110-21swapper/1112:36:393
8799100watchdog/110-21swapper/1112:03:463
8799100watchdog/110-21swapper/1111:51:473
800499108cyclictest0-21swapper/1110:25:183
800499108cyclictest0-21swapper/1107:45:123
800499102cyclictest0-21swapper/1107:29:323
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional