You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 20:48
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3.osadl.org (updated Fri Nov 21, 2025 12:46:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
122501090irq/42-ahci0-21swapper/107:08:421
23562210391sleep50-21swapper/507:07:517
3066350940irq/52-eth0-rx-0-21swapper/207:08:134
3066350930irq/52-eth0-rx-0-21swapper/307:05:205
2371629287sleep40-21swapper/407:09:146
2369527572sleep90-21swapper/907:08:5511
3066350650irq/52-eth0-rx-0-21swapper/1107:05:173
2341726553sleep00-21swapper/007:05:590
2376426462sleep80-21swapper/807:09:5510
2347326454sleep60-21swapper/607:06:418
2364626260sleep70-21swapper/707:08:189
2349325955sleep100-21swapper/1007:06:562
23875993617cyclictest0-21swapper/010:18:070
23916993215cyclictest831ksoftirqd/1008:08:342
23875993015cyclictest0-21swapper/009:29:240
2391799290cyclictest0-21swapper/1108:14:283
2391799260cyclictest0-21swapper/1108:21:283
23875992614cyclictest0-21swapper/011:00:010
2391799250cyclictest0-21swapper/1110:16:383
2391699247cyclictest831ksoftirqd/1007:30:122
23916992423cyclictest0-21swapper/1009:04:442
2391699203cyclictest831ksoftirqd/1010:31:192
2391699192cyclictest831ksoftirqd/1012:36:492
2391699192cyclictest831ksoftirqd/1012:30:302
2391699192cyclictest831ksoftirqd/1012:20:262
2391699192cyclictest831ksoftirqd/1012:20:262
2391699192cyclictest831ksoftirqd/1012:02:102
2391699192cyclictest831ksoftirqd/1011:54:402
2391699192cyclictest831ksoftirqd/1011:35:132
2391699192cyclictest831ksoftirqd/1011:30:162
2391699192cyclictest831ksoftirqd/1011:15:172
2391699192cyclictest831ksoftirqd/1011:11:122
2391699192cyclictest831ksoftirqd/1010:50:192
2391699192cyclictest831ksoftirqd/1010:14:532
2391699192cyclictest831ksoftirqd/1010:06:422
2391699192cyclictest831ksoftirqd/1009:50:182
2391699192cyclictest831ksoftirqd/1009:35:232
2391699192cyclictest831ksoftirqd/1009:22:392
2391699192cyclictest831ksoftirqd/1008:49:002
2391699192cyclictest831ksoftirqd/1008:20:442
2391699192cyclictest831ksoftirqd/1007:47:522
2391699192cyclictest831ksoftirqd/1007:21:342
2387599190cyclictest0-21swapper/009:08:380
2391799180cyclictest0-21swapper/1109:08:383
23916991817cyclictest30058-21kworker/10:109:05:162
23916991817cyclictest0-21swapper/1012:25:052
23916991817cyclictest0-21swapper/1012:15:052
23916991817cyclictest0-21swapper/1012:05:042
23916991817cyclictest0-21swapper/1011:55:092
23916991817cyclictest0-21swapper/1011:45:012
23916991817cyclictest0-21swapper/1011:40:032
23916991817cyclictest0-21swapper/1011:25:052
23916991817cyclictest0-21swapper/1011:20:072
23916991817cyclictest0-21swapper/1011:00:072
23916991817cyclictest0-21swapper/1010:55:002
23916991817cyclictest0-21swapper/1010:35:012
23916991817cyclictest0-21swapper/1010:20:042
23916991817cyclictest0-21swapper/1010:15:162
23916991817cyclictest0-21swapper/1009:55:032
23916991817cyclictest0-21swapper/1009:45:052
23916991817cyclictest0-21swapper/1009:40:062
23916991817cyclictest0-21swapper/1009:40:062
23916991817cyclictest0-21swapper/1009:30:362
23916991817cyclictest0-21swapper/1009:30:362
23916991817cyclictest0-21swapper/1009:25:012
23916991817cyclictest0-21swapper/1009:10:272
23916991817cyclictest0-21swapper/1009:10:272
23916991817cyclictest0-21swapper/1008:55:082
23916991817cyclictest0-21swapper/1008:50:102
23916991817cyclictest0-21swapper/1008:40:092
23916991817cyclictest0-21swapper/1008:35:012
23916991817cyclictest0-21swapper/1008:25:112
23916991817cyclictest0-21swapper/1008:00:022
23916991817cyclictest0-21swapper/1007:55:032
23916991817cyclictest0-21swapper/1007:35:072
23916991817cyclictest0-21swapper/1007:25:232
23916991817cyclictest0-21swapper/1007:15:062
2391699181cyclictest831ksoftirqd/1012:10:152
2391699181cyclictest831ksoftirqd/1011:05:162
2391699181cyclictest831ksoftirqd/1010:45:042
2391699181cyclictest831ksoftirqd/1010:40:202
2391699181cyclictest831ksoftirqd/1010:25:292
2391699181cyclictest831ksoftirqd/1010:00:102
2391699181cyclictest831ksoftirqd/1009:15:122
2391699181cyclictest831ksoftirqd/1008:30:162
2391699181cyclictest831ksoftirqd/1008:15:002
2391699181cyclictest831ksoftirqd/1008:10:112
2391699181cyclictest831ksoftirqd/1007:50:132
2391699181cyclictest831ksoftirqd/1007:40:492
2391699181cyclictest831ksoftirqd/1007:10:022
2391699181cyclictest831ksoftirqd/1007:10:012
23915991816cyclictest29478-21chrt08:42:0511
23915991816cyclictest1861-21taskset10:19:1911
2391599181cyclictest18839-1kworker/9:0H10:55:3211
23914991816cyclictest9769-21snmp_rack3slot911:56:0010
23914991816cyclictest9227-21snmp_rack3slot907:30:1510
23914991816cyclictest8032-21snmp_rack3slot908:56:5010
23914991816cyclictest5379-21snmp_easybox.os11:50:2310
23914991816cyclictest30929-21irqstats07:15:2410
23914991816cyclictest29987-21iostat11:40:1910
23914991816cyclictest2924-21snmp_rack3slot912:31:2610
23914991816cyclictest28806-21snmp_easybox.os12:25:1010
23914991816cyclictest27649-21snmp_rack3slot909:25:1110
23914991816cyclictest27257-21snmp_rack3slot910:51:5010
23914991816cyclictest20585-21memory07:45:2010
23914991816cyclictest19288-21memory11:25:1910
23914991816cyclictest18780-21munin-run08:30:0110
23914991816cyclictest16291-21perl07:40:1710
23914991816cyclictest15264-21snmp_rack3slot909:05:5310
23914991816cyclictest14723-21diskstats11:20:1510
23914991816cyclictest1266-21df07:20:1110
23912991816cyclictest15487-21qemu-kvm09:08:188
2391799170cyclictest5429-21chrt10:22:073
2391599170cyclictest18839-1kworker/9:0H09:14:1411
2391599170cyclictest18839-1kworker/9:0H09:14:1311
2391599170cyclictest0-21swapper/909:35:2211
23914991715cyclictest9492-21snmp_rack3slot911:12:0010
23914991715cyclictest9487-21snmp_rack3slot908:15:1510
23914991715cyclictest6492-21yum12:35:3010
23914991715cyclictest3922-21snmp_rack3slot908:06:4710
23914991715cyclictest32121-21snmp_rack3slot910:15:1510
23914991715cyclictest28384-21memory08:40:2110
23914991715cyclictest27574-21snmp_rack3slot911:35:4810
23914991715cyclictest23787-21df_inode08:35:1310
23914991715cyclictest23502-21perl07:50:1510
23914991715cyclictest22715-21iostat_ios11:30:1810
23914991715cyclictest22063-21users08:30:3110
23914991715cyclictest2174-21snmp_rack3slot911:02:1610
23914991715cyclictest20661-21snmp_rack3slot909:15:1410
23914991715cyclictest1613-21snmp_easybox.os11:45:2410
23914991715cyclictest13247-21perl08:20:1510
23914991715cyclictest12423-21munin-run09:50:0110
23914991715cyclictest10929-21snmp_easybox.os09:00:2410
2391299176cyclictest0-21swapper/607:55:018
23904991715cyclictest3707-21sensors08:50:246
2387599170cyclictest0-21swapper/010:08:280
2391599165cyclictest0-21swapper/907:37:3411
23915991615cyclictest0-21swapper/912:25:1711
23915991614cyclictest7654-21taskset08:12:1811
23915991614cyclictest29961-21taskset09:26:3411
23915991614cyclictest27051-21taskset10:09:5211
2391599160cyclictest0-21swapper/910:45:1411
2391599160cyclictest0-21swapper/907:20:2611
23914991614cyclictest5378-21perl07:25:1210
23914991614cyclictest4987-21ntp_states11:05:2310
23914991614cyclictest32093-21snmp_rack3slot909:30:1510
23914991614cyclictest32093-21snmp_rack3slot909:30:1410
23914991614cyclictest31181-21snmp_rack3slot908:45:1710
23914991614cyclictest26018-21df_inode12:20:1610
23914991614cyclictest26018-21df_inode12:20:1510
23914991614cyclictest25230-21df_inode07:10:1510
23914991614cyclictest25230-21df_inode07:10:1510
23914991614cyclictest24750-21snmp_rack3slot909:20:1510
23914991614cyclictest24310-21yum12:15:3110
23914991614cyclictest2312-21diskstats08:50:1310
23914991614cyclictest2123-21snmp_rack3slot909:35:1110
23914991614cyclictest13001-21snmp_easybox.os09:50:1110
23914991614cyclictest11082-21df_inode11:15:1210
23911991615cyclictest23634-21qemu-kvm07:47:207
23911991614cyclictest26920-21qemu-kvm10:08:257
23904991614cyclictest7349-21sensors08:55:256
23904991614cyclictest6436-21sensors07:25:226
2389999161cyclictest0-21swapper/311:10:285
2389999161cyclictest0-21swapper/310:20:215
2389299161cyclictest0-21swapper/211:10:294
2389299160cyclictest0-21swapper/207:15:254
2388699161cyclictest0-21swapper/107:45:271
2388699160cyclictest0-21swapper/107:10:231
2388699160cyclictest0-21swapper/107:10:221
13450160irq/18-ehci_hcd0-21swapper/007:15:140
23917991514cyclictest3066350irq/52-eth0-rx-09:28:513
23917991513cyclictest85122sleep1107:30:113
2391599154cyclictest0-21swapper/910:45:0011
2391599154cyclictest0-21swapper/908:15:0111
23915991514cyclictest18542-21taskset08:26:5111
23915991514cyclictest0-21swapper/910:51:5611
23915991513cyclictest26067-21idleruntime-cro08:40:0011
23915991513cyclictest16896-21taskset11:23:5011
23915991513cyclictest10069-21idleruntime-cro12:00:0011
23914991513cyclictest32596-21snmp_rack3slot908:01:1310
23914991513cyclictest17019-21df_inode09:10:1010
23914991513cyclictest17019-21df_inode09:10:1010
23914991513cyclictest14552-21snmp_rack3slot910:35:1510
23911991513cyclictest7007-21sh09:40:177
23911991513cyclictest7007-21sh09:40:167
2391199150cyclictest0-21swapper/508:40:157
23904991514cyclictest16058-21sensors11:20:266
2389299150cyclictest0-21swapper/208:10:124
2387599150cyclictest0-21swapper/008:25:120
2387599150cyclictest0-21swapper/007:45:130
13450150irq/18-ehci_hcd0-21swapper/008:50:240
8799140watchdog/110-21swapper/1111:46:273
2391599144cyclictest0-21swapper/909:40:0111
2391599144cyclictest0-21swapper/909:40:0111
2391599144cyclictest0-21swapper/908:35:0111
2391599143cyclictest0-21swapper/912:00:2011
23915991413cyclictest0-21swapper/911:50:1311
23915991413cyclictest0-21swapper/909:30:1711
23915991413cyclictest0-21swapper/909:30:1711
2391599141cyclictest0-21swapper/911:45:1911
23914991412cyclictest5648-21munin-run10:24:5910
23914991412cyclictest17139-21snmp_rack3slot912:06:5110
2391399145cyclictest0-21swapper/711:05:199
2391399145cyclictest0-21swapper/708:35:199
2391399145cyclictest0-21swapper/707:15:019
2389999141cyclictest0-21swapper/307:20:185
2389999140cyclictest0-21swapper/311:04:445
2389999140cyclictest0-21swapper/310:35:415
2389999140cyclictest0-21swapper/309:30:385
2389999140cyclictest0-21swapper/309:30:375
2389999140cyclictest0-21swapper/308:32:145
23875991413cyclictest0-21swapper/009:15:250
4599130watchdog/50-21swapper/508:31:437
31130ksoftirqd/00-21swapper/010:40:280
2391799134cyclictest0-21swapper/1112:25:023
2391799134cyclictest0-21swapper/1112:25:013
2391799134cyclictest0-21swapper/1111:55:013
2391799133cyclictest0-21swapper/1111:38:243
2391799133cyclictest0-21swapper/1111:30:023
2391599134cyclictest0-21swapper/912:15:1911
2391599133cyclictest0-21swapper/912:21:4611
2391599133cyclictest0-21swapper/912:21:4511
2391599133cyclictest0-21swapper/911:40:0011
2391599133cyclictest0-21swapper/909:56:3811
2391599133cyclictest0-21swapper/909:07:0811
2391599133cyclictest0-21swapper/909:00:2311
23915991312cyclictest0-21swapper/912:30:1311
2391599131cyclictest0-21swapper/907:15:1411
2391599130cyclictest0-21swapper/911:30:2711
23914991311cyclictest5952-21perl08:10:1310
23914991311cyclictest30145-21snmp_rack3slot910:55:2310
23914991311cyclictest23018-21snmp_rack3slot910:00:4710
23914991311cyclictest21844-21perl10:45:1610
23914991311cyclictest13739-21munin-run12:05:0110
2391399135cyclictest0-21swapper/709:27:459
2391399134cyclictest0-21swapper/710:25:019
2391399133cyclictest0-21swapper/708:20:259
2391299134cyclictest0-21swapper/610:50:008
2391299133cyclictest0-21swapper/609:37:158
2391199130cyclictest0-21swapper/511:10:017
2389999136cyclictest0-21swapper/309:39:485
2389999135cyclictest0-21swapper/311:36:125
2389999135cyclictest0-21swapper/311:07:005
2389999135cyclictest0-21swapper/309:12:405
2389999135cyclictest0-21swapper/309:12:405
2389999135cyclictest0-21swapper/308:03:565
2389999135cyclictest0-21swapper/307:59:375
2389999134cyclictest0-21swapper/309:48:125
2389999133cyclictest794-21ntp_offset10:15:235
2389999131cyclictest0-21swapper/308:10:015
2389999130cyclictest0-21swapper/312:26:415
2389999130cyclictest0-21swapper/312:15:315
2389999130cyclictest0-21swapper/312:07:325
2389999130cyclictest0-21swapper/311:55:355
2389999130cyclictest0-21swapper/311:44:065
2389999130cyclictest0-21swapper/311:31:305
2389999130cyclictest0-21swapper/311:18:035
2389999130cyclictest0-21swapper/310:59:425
2389999130cyclictest0-21swapper/310:11:505
2389999130cyclictest0-21swapper/309:17:115
2389999130cyclictest0-21swapper/309:03:065
2389999130cyclictest0-21swapper/308:52:185
2389999130cyclictest0-21swapper/308:35:355
2389999130cyclictest0-21swapper/308:19:515
2389299133cyclictest0-21swapper/207:10:184
2389299133cyclictest0-21swapper/207:10:184
2388699133cyclictest0-21swapper/110:00:001
2388699133cyclictest0-21swapper/107:20:151
23875991312cyclictest304232chrt10:11:060
23875991312cyclictest2419-21sshd10:25:200
23875991311cyclictest0-21swapper/008:10:220
2387599130cyclictest0-21swapper/007:35:250
2391799124cyclictest0-21swapper/1110:05:013
2391799124cyclictest0-21swapper/1108:55:013
2391799123cyclictest0-21swapper/1111:05:013
2391799123cyclictest0-21swapper/1111:00:013
2391799123cyclictest0-21swapper/1110:50:013
2391799123cyclictest0-21swapper/1109:11:483
2391799123cyclictest0-21swapper/1109:11:473
2391799122cyclictest0-21swapper/1110:12:483
2391799122cyclictest0-21swapper/1107:13:113
2391799122cyclictest0-21swapper/1107:13:113
2391599123cyclictest0-21swapper/912:05:0211
2391599123cyclictest0-21swapper/911:10:0111
2391599123cyclictest0-21swapper/910:25:0011
2391599123cyclictest0-21swapper/910:20:0111
2391599123cyclictest0-21swapper/908:00:0011
23915991211cyclictest741rcuc/907:47:2211
23915991211cyclictest26482-21taskset09:23:5111
23915991211cyclictest0-21swapper/910:00:1811
23914991210cyclictest18212-21perl10:40:1610
2391399122cyclictest0-21swapper/711:41:099
2391399122cyclictest0-21swapper/711:33:399
2391399122cyclictest0-21swapper/708:05:379
2391299125cyclictest0-21swapper/609:53:468
2391299124cyclictest0-21swapper/612:16:508
2391299123cyclictest0-21swapper/609:22:048
23912991211cyclictest23633-21qemu-kvm07:47:268
2391199120cyclictest0-21swapper/508:50:177
2390499122cyclictest0-21swapper/410:15:186
23904991210cyclictest19684-21sensors11:25:246
2389999126cyclictest0-21swapper/312:24:075
2389999126cyclictest0-21swapper/312:24:065
2389999126cyclictest0-21swapper/312:00:455
2389999126cyclictest0-21swapper/310:46:395
2389999126cyclictest0-21swapper/309:59:245
2389999126cyclictest0-21swapper/309:41:405
2389999126cyclictest0-21swapper/309:41:395
2389999125cyclictest0-21swapper/310:00:135
2389999125cyclictest0-21swapper/307:45:465
2389999124cyclictest0-21swapper/308:57:285
2389999123cyclictest13810-21switchtime07:35:295
2389999121cyclictest0-21swapper/311:25:235
2389999121cyclictest0-21swapper/309:25:105
2389999121cyclictest0-21swapper/308:40:155
2389999121cyclictest0-21swapper/308:25:135
2389999121cyclictest0-21swapper/308:20:215
2389999121cyclictest0-21swapper/308:10:135
2389999121cyclictest0-21swapper/307:40:195
2389999121cyclictest0-21swapper/307:30:115
2389999121cyclictest0-21swapper/307:25:145
2389999120cyclictest0-21swapper/311:50:045
2389999120cyclictest0-21swapper/309:50:315
2389999120cyclictest0-21swapper/308:45:135
2389999120cyclictest0-21swapper/307:50:125
2389299124cyclictest0-21swapper/207:35:184
2389299122cyclictest0-21swapper/210:45:174
2387599124cyclictest0-21swapper/007:25:150
2387599122cyclictest0-21swapper/010:20:170
2387599122cyclictest0-21swapper/008:30:180
2387599120cyclictest0-21swapper/009:10:230
2387599120cyclictest0-21swapper/009:10:220
901110ksoftirqd/1129049-21grep12:25:123
621110ksoftirqd/70-21swapper/712:00:239
551110ksoftirqd/69107-21cpuspeed_turbos09:00:118
5499110migration/617955-21munin-node09:10:198
5499110migration/617955-21munin-node09:10:188
411110ksoftirqd/40-21swapper/412:25:116
31110ksoftirqd/00-21swapper/012:25:110
31110ksoftirqd/00-21swapper/007:10:260
31110ksoftirqd/00-21swapper/007:10:260
2391799119cyclictest19289-21taskset08:30:063
2391799119cyclictest14631-21munin-run07:40:013
2391799115cyclictest0-21swapper/1109:40:473
2391799115cyclictest0-21swapper/1109:40:463
2391799114cyclictest0-21swapper/1109:55:013
2391799113cyclictest0-21swapper/1112:35:013
2391799113cyclictest0-21swapper/1111:25:003
2391799113cyclictest0-21swapper/1110:43:183
2391799113cyclictest0-21swapper/1109:59:243
2391799113cyclictest0-21swapper/1108:04:223
2391799113cyclictest0-21swapper/1107:41:523
2391799112cyclictest0-21swapper/1111:05:003
2391799112cyclictest0-21swapper/1108:07:223
2391599115cyclictest0-21swapper/910:39:4611
2391599113cyclictest0-21swapper/910:15:0111
2391599113cyclictest0-21swapper/907:40:1811
23915991110cyclictest0-21swapper/909:15:1411
23915991110cyclictest0-21swapper/907:30:2911
2391599111cyclictest0-21swapper/911:40:2411
2391599110cyclictest31965-21cpuspeed_turbos11:00:1411
2391499119cyclictest25482-21iostat10:05:2110
2391399113cyclictest0-21swapper/712:35:399
2391399113cyclictest0-21swapper/712:23:009
2391399113cyclictest0-21swapper/712:23:009
2391399113cyclictest0-21swapper/712:15:109
2391399113cyclictest0-21swapper/711:56:119
2391399113cyclictest0-21swapper/711:46:099
2391399113cyclictest0-21swapper/711:20:089
2391399113cyclictest0-21swapper/711:01:169
2391399113cyclictest0-21swapper/709:50:389
2391399113cyclictest0-21swapper/709:05:139
2391399113cyclictest0-21swapper/708:30:009
2391399113cyclictest0-21swapper/708:28:299
2391399113cyclictest0-21swapper/707:23:419
2391399112cyclictest0-21swapper/711:53:409
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional