You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 15:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3.osadl.org (updated Wed Aug 27, 2025 12:46:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
122501280irq/42-ahci0-21swapper/307:08:555
29710211197sleep100-21swapper/1007:08:112
29539210793sleep80-21swapper/807:05:5710
29687210693sleep40-21swapper/407:07:536
29596210693sleep70-21swapper/707:06:449
122501030irq/42-ahci0-21swapper/107:08:481
122501010irq/42-ahci0-21swapper/207:09:094
3066350890irq/52-eth0-rx-0-21swapper/1107:05:233
2951728575sleep50-21swapper/507:05:427
2987428078sleep90-21swapper/907:09:4611
2964525853sleep60-21swapper/607:07:208
2951125753sleep00-21swapper/007:05:370
3004899340cyclictest0-21swapper/1110:23:153
3004899340cyclictest0-21swapper/1109:28:043
3004799214cyclictest831ksoftirqd/1010:03:122
3001999204cyclictest2340-40kipmi009:10:505
30048991918cyclictest0-21swapper/1108:15:053
3004799192cyclictest831ksoftirqd/1010:15:142
30047991918cyclictest0-21swapper/1008:55:162
30047991817cyclictest831ksoftirqd/1012:10:302
30047991817cyclictest811rcuc/1007:31:252
30047991817cyclictest0-21swapper/1012:35:192
3004799181cyclictest831ksoftirqd/1012:15:262
3004799181cyclictest831ksoftirqd/1008:50:132
3004699181cyclictest761ksoftirqd/911:45:1411
3004699181cyclictest0-21swapper/910:08:5511
3004699181cyclictest0-21swapper/909:15:1711
3004699181cyclictest0-21swapper/909:15:1711
3001999181cyclictest0-21swapper/310:45:205
2999999180cyclictest0-21swapper/010:20:480
3004899170cyclictest0-21swapper/1109:08:383
30047991716cyclictest54902sleep1012:27:362
30047991716cyclictest54902sleep1012:27:352
30047991716cyclictest20773-21kworker/10:111:15:002
30047991716cyclictest0-21swapper/1010:25:012
3004799170cyclictest95352sleep1007:21:012
3004799170cyclictest90812sleep1012:31:582
3004799170cyclictest86302sleep1010:20:082
3004799170cyclictest831ksoftirqd/1011:45:112
3004799170cyclictest831ksoftirqd/1011:35:222
3004799170cyclictest831ksoftirqd/1011:20:142
3004799170cyclictest831ksoftirqd/1010:40:012
3004799170cyclictest831ksoftirqd/1009:45:132
3004799170cyclictest831ksoftirqd/1009:10:162
3004799170cyclictest831ksoftirqd/1007:50:142
3004799170cyclictest81992sleep1011:00:372
3004799170cyclictest49942sleep1011:41:052
3004799170cyclictest41712sleep1010:10:562
3004799170cyclictest39762sleep1008:45:102
3004799170cyclictest32882sleep1008:42:462
3004799170cyclictest323592sleep1009:20:382
3004799170cyclictest319052sleep1008:36:202
3004799170cyclictest3066350irq/52-eth0-rx-12:20:272
3004799170cyclictest3066350irq/52-eth0-rx-09:30:332
3004799170cyclictest3066350irq/52-eth0-rx-09:05:142
3004799170cyclictest3066350irq/52-eth0-rx-08:05:302
3004799170cyclictest3066350irq/52-eth0-rx-07:15:152
3004799170cyclictest304952sleep1007:10:102
3004799170cyclictest304832chrt10:50:122
3004799170cyclictest300602sleep1011:30:522
3004799170cyclictest288062sleep1009:17:012
3004799170cyclictest288062sleep1009:17:012
3004799170cyclictest276952sleep1007:45:382
3004799170cyclictest270182sleep1008:30:192
3004799170cyclictest268162sleep1010:45:112
3004799170cyclictest255202sleep1009:56:182
3004799170cyclictest247762sleep1008:28:322
3004799170cyclictest243642sleep1007:44:502
3004799170cyclictest236052sleep1011:25:022
3004799170cyclictest225402sleep1010:38:222
3004799170cyclictest219032chrt09:51:282
3004799170cyclictest203612sleep1012:05:082
3004799170cyclictest186972sleep1010:30:332
3004799170cyclictest177932sleep1007:35:142
3004799170cyclictest176872chrt09:00:352
3004799170cyclictest175742sleep1008:19:432
3004799170cyclictest16812sleep1010:55:112
3004799170cyclictest166962sleep1012:00:072
3004799170cyclictest158892sleep1011:56:092
3004799170cyclictest156482chrt11:12:382
3004799170cyclictest146742sleep1009:42:172
3004799170cyclictest136902sleep1008:11:182
3004799170cyclictest124102chrt07:25:252
3004799170cyclictest122692sleep1011:51:182
3004799170cyclictest12250irq/42-ahci10:08:012
3004799170cyclictest12250irq/42-ahci09:35:062
3004799170cyclictest12250irq/42-ahci09:25:342
3004799170cyclictest12250irq/42-ahci08:20:482
3004799170cyclictest12250irq/42-ahci07:55:132
3004799170cyclictest120812sleep1011:08:382
30046991716cyclictest31675-21processes09:20:2511
30046991716cyclictest16269-1kworker/9:0H12:07:2911
30046991715cyclictest30471-21snmp_rack3slot912:16:1311
3004699170cyclictest0-21swapper/909:25:2311
3004699170cyclictest0-21swapper/907:55:2311
3004699170cyclictest0-21swapper/907:40:1311
30034991716cyclictest21600-21qemu-kvm09:18:578
30034991716cyclictest21600-21qemu-kvm09:18:568
30028991716cyclictest566-21qemu-kvm10:25:157
30028991715cyclictest10180-21qemu-kvm08:08:337
30025991715cyclictest7991-21sensors11:45:246
2999999170cyclictest0-21swapper/007:23:550
30048991615cyclictest0-21swapper/1108:09:303
30048991614cyclictest10715-21tr07:25:143
3004699165cyclictest0-21swapper/907:40:0111
3004699164cyclictest0-21swapper/908:40:0011
30046991615cyclictest741rcuc/908:12:4111
30046991615cyclictest16269-1kworker/9:0H11:26:0611
30046991615cyclictest10148-21latency_hist07:25:0111
30046991615cyclictest0-21swapper/911:30:2711
30046991615cyclictest0-21swapper/909:10:1611
30046991615cyclictest0-21swapper/907:35:0011
30034991614cyclictest29757-21qemu-kvm07:15:458
30028991615cyclictest10545-21cstates_1112:35:157
30025991614cyclictest28121-21sensors09:15:286
30025991614cyclictest28121-21sensors09:15:286
30025991614cyclictest27610-21sensors08:30:256
3001999160cyclictest0-21swapper/308:45:225
3001999160cyclictest0-21swapper/307:40:155
29999991614cyclictest0-21swapper/008:19:590
30048991514cyclictest49902sleep1110:15:103
30048991514cyclictest12250irq/42-ahci10:09:103
30048991514cyclictest0-21swapper/1108:25:223
30047991514cyclictest831ksoftirqd/1008:05:002
3004699154cyclictest0-21swapper/909:35:0111
30046991514cyclictest0-21swapper/911:40:0111
30046991514cyclictest0-21swapper/907:20:0211
30046991513cyclictest32099-21iostat_ios07:10:1911
30034991513cyclictest29756-21qemu-kvm07:21:098
3003499150cyclictest0-21swapper/610:58:288
31150ksoftirqd/00-21swapper/010:10:260
29999991514cyclictest0-21swapper/008:30:000
30048991413cyclictest0-21swapper/1108:23:203
3004699145cyclictest0-21swapper/909:05:0011
3004699144cyclictest0-21swapper/908:55:0011
3004699144cyclictest0-21swapper/908:20:0111
3004699143cyclictest0-21swapper/909:35:0011
3004699143cyclictest0-21swapper/908:35:0011
30046991413cyclictest0-21swapper/910:15:1411
30046991413cyclictest0-21swapper/910:00:4911
30046991413cyclictest0-21swapper/909:55:0111
30046991413cyclictest0-21swapper/908:05:2211
3004699140cyclictest0-21swapper/911:05:1411
30045991411cyclictest25218-21irqstats11:25:2310
30034991413cyclictest566-21qemu-kvm10:08:338
30028991413cyclictest566-21qemu-kvm10:13:477
30028991413cyclictest11997-21cstates_1108:55:147
30028991413cyclictest10181-21qemu-kvm08:37:397
3001999141cyclictest10340-21python09:35:265
3001599140cyclictest0-21swapper/210:45:004
29999991413cyclictest2388-21nscd09:14:300
29999991413cyclictest132762sleep007:27:290
29999991413cyclictest0-21swapper/008:09:530
8799130watchdog/110-21swapper/1111:13:023
3004899134cyclictest0-21swapper/1107:55:023
3004899133cyclictest0-21swapper/1112:17:283
3004899133cyclictest0-21swapper/1111:44:113
3004899133cyclictest0-21swapper/1111:35:003
3004899133cyclictest0-21swapper/1110:56:433
3004899133cyclictest0-21swapper/1110:12:303
3004899133cyclictest0-21swapper/1109:00:003
3004899133cyclictest0-21swapper/1107:21:003
3004899133cyclictest0-21swapper/1107:12:043
30048991311cyclictest279752chrt09:15:273
30048991311cyclictest279752chrt09:15:263
3004699135cyclictest0-21swapper/910:33:4011
3004699134cyclictest0-21swapper/908:45:0211
3004699134cyclictest0-21swapper/907:15:0111
3004699133cyclictest0-21swapper/911:11:2711
3004699133cyclictest0-21swapper/909:45:3311
3004699133cyclictest0-21swapper/909:00:3011
30046991312cyclictest3066350irq/52-eth0-rx-12:00:0511
30046991312cyclictest0-21swapper/910:50:1911
30046991312cyclictest0-21swapper/908:25:2511
30045991311cyclictest9949-21snmp_rack3slot908:05:4910
30045991311cyclictest9889-21snmp_rack3slot910:20:1710
30045991311cyclictest9886-21snmp_easybox.os12:35:1310
30045991311cyclictest9449-21snmp_easybox.os11:50:1210
30045991311cyclictest9300-21perl09:35:1610
30045991311cyclictest8597-21snmp_rack3slot911:45:4910
30045991311cyclictest841-21perl07:55:1510
30045991311cyclictest830-21snmp_rack3slot910:50:5010
30045991311cyclictest8264-21snmp_easybox.os12:30:2510
30045991311cyclictest8211-21snmp_rack3slot911:00:4910
30045991311cyclictest6776-21snmp_easybox.os07:20:1310
30045991311cyclictest5522-21snmp_rack3slot910:15:1410
30045991311cyclictest4841-21snmp_rack3slot908:45:1510
30045991311cyclictest436-21ntp_states11:35:2410
30045991311cyclictest3764-21iostat_ios11:40:2110
30045991311cyclictest3559-21snmp_easybox.os08:00:1210
30045991311cyclictest3420-21munin-node12:25:1410
30045991311cyclictest3420-21munin-node12:25:1410
30045991311cyclictest31458-21snmp_rack3slot912:20:1310
30045991311cyclictest31277-21snmp_rack3slot907:10:1510
30045991311cyclictest31114-21perl10:05:1810
30045991311cyclictest30059-21snmp_rack3slot911:30:5210
30045991311cyclictest29610-21snmp_easybox.os09:20:1310
30045991311cyclictest29510-21snmp_rack3slot908:35:1410
30045991311cyclictest2825-21perl10:55:1510
30045991311cyclictest28177-21snmp_rack3slot910:45:1710
30045991311cyclictest27692-21snmp_easybox.os12:15:1210
30045991311cyclictest27577-21fw_packets10:00:1510
30045991311cyclictest27171-21snmp_rack3slot909:15:1810
30045991311cyclictest27171-21snmp_rack3slot909:15:1710
30045991311cyclictest26273-21snmp_rack3slot908:30:1410
30045991311cyclictest25760-21df_inode07:45:1610
30045991311cyclictest25544-21snmp_rack3slot909:56:3710
30045991311cyclictest24008-21snmp_easybox.os12:10:1210
30045991311cyclictest23535-21snmp_rack3slot910:40:1310
30045991311cyclictest22499-21snmp_rack3slot909:10:1210
30045991311cyclictest22029-21snmp_easybox.os11:20:2410
30045991311cyclictest21952-21snmp_rack3slot908:25:1310
30045991311cyclictest21615-21perl12:05:1810
30045991311cyclictest21447-21snmp_rack3slot907:40:1210
30045991311cyclictest20690-21perl10:35:1910
30045991311cyclictest20494-21munin-node09:50:1910
30045991311cyclictest19609-21snmp_rack3slot912:01:5010
30045991311cyclictest18815-21snmp_rack3slot909:05:1510
30045991311cyclictest18580-21snmp_rack3slot908:20:1510
30045991311cyclictest18507-21snmp_rack3slot907:35:1610
30045991311cyclictest1833-21snmp_rack3slot910:10:1410
30045991311cyclictest1732-21snmp_rack3slot909:25:1710
30045991311cyclictest17118-21df11:15:1710
30045991311cyclictest17083-21snmp_rack3slot910:30:1410
30045991311cyclictest1637-21df07:15:0210
30045991311cyclictest1615-21snmp_rack3slot908:40:1610
30045991311cyclictest15913-21snmp_rack3slot911:56:2610
30045991311cyclictest15767-21diskstats09:00:1310
30045991311cyclictest14771-21snmp_rack3slot907:30:1610
30045991311cyclictest14492-21snmp_easybox.os08:15:1110
30045991311cyclictest14314-21irqstats11:10:2010
30045991311cyclictest13870-21iostat_ios10:25:1910
30045991311cyclictest13122-21snmp_rack3slot909:40:1710
30045991311cyclictest12115-21df_inode08:55:1410
30045991311cyclictest10910-21snmp_easybox.os08:10:1210
30045991311cyclictest10541-21snmp_rack3slot908:52:1510
30045991311cyclictest10287-21perl11:05:1610
3003499133cyclictest0-21swapper/612:18:088
3003499133cyclictest0-21swapper/611:42:078
3003499133cyclictest0-21swapper/608:06:548
30028991312cyclictest8409-21df_abs08:50:147
30028991312cyclictest7621-21python11:00:267
30028991311cyclictest566-21qemu-kvm10:16:297
3002599133cyclictest0-21swapper/409:10:196
30025991311cyclictest2025-21sensors07:15:156
31130ksoftirqd/00-21swapper/007:15:300
2999999133cyclictest0-21swapper/010:50:190
29999991312cyclictest0-21swapper/008:30:090
2999999130cyclictest0-21swapper/011:10:260
5499120migration/617912-21munin-node10:30:238
3004899123cyclictest0-21swapper/1111:50:003
3004899123cyclictest0-21swapper/1111:30:023
3004899122cyclictest0-21swapper/1112:25:453
3004899122cyclictest0-21swapper/1112:25:453
3004899122cyclictest0-21swapper/1111:59:563
3004899122cyclictest0-21swapper/1109:22:253
3004899122cyclictest0-21swapper/1108:11:253
3004899122cyclictest0-21swapper/1107:50:413
3004699123cyclictest0-21swapper/911:20:0011
3004699122cyclictest0-21swapper/912:11:1411
30046991211cyclictest741rcuc/907:49:5011
30046991211cyclictest0-21swapper/911:40:2311
30045991210cyclictest5550-21snmp_rack3slot909:30:1910
30045991210cyclictest31427-21snmp_rack3slot907:51:3810
3004199122cyclictest0-21swapper/711:19:009
3003499125cyclictest0-21swapper/612:15:018
3003499125cyclictest0-21swapper/608:35:018
3003499124cyclictest0-21swapper/608:45:028
3003499123cyclictest0-21swapper/609:20:108
3003499122cyclictest0-21swapper/610:36:308
3003499122cyclictest0-21swapper/607:40:508
3003499121cyclictest0-21swapper/608:45:018
3003499120cyclictest0-21swapper/607:50:008
3003499120cyclictest0-21swapper/607:40:008
30025991210cyclictest4365-21sensors11:40:286
30025991210cyclictest32675-21sensors10:50:276
30025991210cyclictest12549-21sensors07:25:266
3001999128cyclictest18122-21python10:30:255
3001999125cyclictest20428-21snmp_easybox.os12:05:145
3001999122cyclictest2340-40kipmi009:18:595
3001999122cyclictest2340-40kipmi009:18:585
3001999122cyclictest2340-40kipmi007:50:205
3001999121cyclictest0-21swapper/312:30:205
30007991210cyclictest4377-21python11:40:291
30007991210cyclictest22571-21python12:05:281
31120ksoftirqd/00-21swapper/008:55:250
2999999129cyclictest0-21swapper/011:50:250
3004899115cyclictest0-21swapper/1110:34:513
3004899113cyclictest8736-21sed11:05:003
3004899113cyclictest0-21swapper/1112:31:453
3004899113cyclictest0-21swapper/1112:10:293
3004899113cyclictest0-21swapper/1112:02:453
3004899113cyclictest0-21swapper/1111:37:103
3004899113cyclictest0-21swapper/1110:49:033
3004899113cyclictest0-21swapper/1110:43:363
3004899113cyclictest0-21swapper/1109:41:043
3004899113cyclictest0-21swapper/1108:44:323
3004899113cyclictest0-21swapper/1108:35:143
3004899113cyclictest0-21swapper/1108:32:343
3004899113cyclictest0-21swapper/1108:03:493
3004899113cyclictest0-21swapper/1107:42:103
3004899113cyclictest0-21swapper/1107:34:593
3004899112cyclictest0-21swapper/1112:38:143
3004899112cyclictest0-21swapper/1107:15:243
30048991110cyclictest3066350irq/52-eth0-rx-09:10:013
3004899111cyclictest0-21swapper/1110:53:053
3004699119cyclictest14369-21sshd08:59:5111
3004699118cyclictest0-21swapper/910:24:5711
3004699113cyclictest0-21swapper/912:22:4611
3004699113cyclictest0-21swapper/910:56:0211
3004699113cyclictest0-21swapper/910:11:3411
3004699113cyclictest0-21swapper/909:41:5111
30046991110cyclictest16269-1kworker/9:0H10:45:5111
3004699111cyclictest0-21swapper/912:30:1311
3004699110cyclictest0-21swapper/910:35:2011
3004599119cyclictest11210-21diskstats07:25:1510
3004199110cyclictest0-21swapper/707:55:199
3003499114cyclictest0-21swapper/610:50:008
3003499113cyclictest0-21swapper/612:25:328
3003499113cyclictest0-21swapper/612:25:318
3003499113cyclictest0-21swapper/611:52:208
3003499113cyclictest0-21swapper/611:35:008
3003499113cyclictest0-21swapper/611:30:458
3003499113cyclictest0-21swapper/611:10:018
3003499113cyclictest0-21swapper/609:52:578
3003499113cyclictest0-21swapper/609:37:508
3003499113cyclictest0-21swapper/609:00:258
3003499113cyclictest0-21swapper/608:56:448
3003499113cyclictest0-21swapper/608:26:448
3003499113cyclictest0-21swapper/607:25:428
3003499112cyclictest0-21swapper/611:10:008
3003499112cyclictest0-21swapper/610:13:428
3003499110cyclictest0-21swapper/608:55:008
3003499110cyclictest0-21swapper/608:00:008
3002599113cyclictest0-21swapper/412:00:196
3001999119cyclictest0-21swapper/310:15:145
3001999115cyclictest0-21swapper/310:10:145
3001999111cyclictest0-21swapper/311:50:195
3001999110cyclictest0-21swapper/312:25:145
3001999110cyclictest0-21swapper/312:25:145
30007991110cyclictest201ksoftirqd/112:10:161
31110ksoftirqd/00-21swapper/010:40:290
31110ksoftirqd/00-21swapper/010:05:280
2999999113cyclictest0-21swapper/011:20:160
2999999113cyclictest0-21swapper/009:05:210
2999999112cyclictest0-21swapper/009:30:000
29999991110cyclictest917-21chrt07:11:490
29999991110cyclictest0-21swapper/008:10:310
1699110watchdog/00-21swapper/008:51:510
13450110irq/18-ehci_hcd0-21swapper/012:15:270
8799100watchdog/110-21swapper/1107:36:143
7599100migration/921439-21munin-node11:20:1711
7599100migration/912796-21cstates10:25:1211
7399100watchdog/90-21swapper/910:41:4211
621100ksoftirqd/70-21swapper/712:10:169
621100ksoftirqd/70-21swapper/708:25:039
6199100migration/76214-21munin-node08:45:309
6199100migration/73748-21taskset09:26:569
6199100migration/731757-21munin-node09:20:259
6199100migration/723617-21cpuspeed_turbos10:40:139
6199100migration/720884-21gltestperf10:35:209
5999100watchdog/70-21swapper/709:46:409
5499100migration/66473-21runrttasks08:02:488
5499100migration/618904-21munin-node12:00:298
5499100migration/614063-21mdstat10:25:208
481100ksoftirqd/50-21swapper/509:51:387
481100ksoftirqd/50-21swapper/508:25:047
4599100watchdog/50-21swapper/507:38:517
4599100watchdog/50-21swapper/507:11:267
3899100watchdog/40-21swapper/412:11:306
3899100watchdog/40-21swapper/411:57:556
3899100watchdog/40-21swapper/411:02:366
3899100watchdog/40-21swapper/409:52:306
341100ksoftirqd/30-21swapper/310:20:345
341100ksoftirqd/30-21swapper/308:25:035
3199100watchdog/30-21swapper/311:10:185
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional