You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 08:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Tue Jul 01, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
949801020,96rtkit-daemon0-21swapper/1019:06:162
275991950,85ptp4l0-21swapper/119:06:291
1569926655,8sleep50-21swapper/519:07:457
41532620,0sleep100-21swapper/1020:55:242
1574826149,10sleep00-21swapper/019:08:290
212322600,0sleep90-21swapper/922:00:1911
1579525951,6sleep60-21swapper/619:09:118
263842580,0sleep10-21swapper/122:50:171
1582225746,9sleep70-21swapper/719:09:369
1255425746,9sleep110-21swapper/1119:05:023
1557025139,10sleep40-21swapper/419:05:566
1623799211,15cyclictest0-21swapper/419:15:016
1623799211,15cyclictest0-21swapper/400:15:006
1626999202,11cyclictest0-21swapper/1121:25:003
1623799202,14cyclictest0-21swapper/419:45:006
292702180,0sleep00-21swapper/000:20:260
289212180,0sleep80-21swapper/822:55:0110
1621799183,8cyclictest0-21swapper/000:25:240
1624699172,8cyclictest0-21swapper/600:10:018
1623799172,9cyclictest0-21swapper/420:20:006
1623399171,10cyclictest0-21swapper/300:35:015
157972178,7sleep80-21swapper/819:09:1310
275991160,1ptp4l841-21missed_timers23:40:201
275991160,1ptp4l841-21missed_timers23:40:191
275991160,1ptp4l21-21ksoftirqd/121:10:191
1626099162,9cyclictest0-21swapper/920:30:0011
1623399163,8cyclictest0-21swapper/322:30:015
1621799163,9cyclictest0-21swapper/000:20:010
1621799163,11cyclictest0-21swapper/000:40:010
1621799162,9cyclictest0-21swapper/019:55:250
275991150,1ptp4l21-21ksoftirqd/123:20:201
275991150,1ptp4l21-21ksoftirqd/123:20:191
275991150,1ptp4l21-21ksoftirqd/122:27:181
275991150,1ptp4l21-21ksoftirqd/119:25:231
275991150,0ptp4l0-21swapper/123:25:161
275991150,0ptp4l0-21swapper/123:25:161
275991150,0ptp4l0-21swapper/100:35:251
275991150,0ptp4l0-21swapper/100:25:201
1626999153,7cyclictest0-21swapper/1100:10:013
1626999152,9cyclictest0-21swapper/1122:05:003
1626999152,7cyclictest0-21swapper/1122:55:013
1626099153,7cyclictest0-21swapper/919:25:0111
1626099152,8cyclictest0-21swapper/922:15:0011
16237991512,2cyclictest11662-21turbostat23:15:006
1623399153,7cyclictest0-21swapper/320:10:005
1623399150,9cyclictest0-21swapper/300:20:005
1621799154,7cyclictest0-21swapper/023:45:010
1621799154,7cyclictest0-21swapper/023:45:000
1621799154,7cyclictest0-21swapper/020:30:010
1621799153,7cyclictest0-21swapper/023:25:010
1621799153,7cyclictest0-21swapper/023:25:000
1621799152,8cyclictest0-21swapper/019:20:000
141802150,0sleep1201rcuc/100:00:011
296712140,0sleep90-21swapper/919:25:1511
275991140,1ptp4l21-21ksoftirqd/121:20:261
275991140,1ptp4l21-21ksoftirqd/119:55:001
1626099143,7cyclictest0-21swapper/919:20:0011
1626099142,7cyclictest0-21swapper/921:10:0011
1624699142,7cyclictest0-21swapper/600:30:018
1624699141,7cyclictest0-21swapper/623:40:018
1624699141,7cyclictest0-21swapper/623:40:008
1623799142,7cyclictest0-21swapper/422:05:276
1623399143,7cyclictest0-21swapper/320:55:005
1623399142,8cyclictest0-21swapper/323:45:015
1623399142,8cyclictest0-21swapper/323:45:015
1623399142,7cyclictest0-21swapper/322:35:015
1623399142,7cyclictest0-21swapper/322:05:015
1623399141,8cyclictest0-21swapper/321:10:015
1621799142,7cyclictest0-21swapper/021:40:010
32362130,0sleep60-21swapper/620:15:178
279022130,0sleep50-21swapper/521:30:007
275991130,3ptp4l27493-21awk20:45:181
275991130,1ptp4l21-21ksoftirqd/100:31:171
275991130,0ptp4l0-21swapper/123:02:341
275991130,0ptp4l0-21swapper/121:42:241
275991130,0ptp4l0-21swapper/119:45:151
275991130,0ptp4l0-21swapper/119:20:221
202762130,0sleep60-21swapper/620:35:198
1626099132,7cyclictest0-21swapper/923:30:0111
1626099132,7cyclictest0-21swapper/923:30:0111
1626099132,7cyclictest0-21swapper/919:40:0011
1624299133,7cyclictest0-21swapper/520:25:017
1624299132,7cyclictest0-21swapper/523:45:257
1624299131,8cyclictest28631-21kworker/u24:021:45:017
1623399133,7cyclictest0-21swapper/320:05:005
1623399132,7cyclictest0-21swapper/300:30:015
1623399132,6cyclictest0-21swapper/321:50:005
1621799132,7cyclictest0-21swapper/021:10:260
1621799131,6cyclictest0-21swapper/020:20:000
158372130,0sleep40-21swapper/419:49:596
275991120,3ptp4l29668-21expr22:55:161
275991120,1ptp4l21-21ksoftirqd/121:29:211
275991120,1ptp4l21-21ksoftirqd/119:15:011
275991120,1ptp4l21-21ksoftirqd/100:20:501
275991120,0ptp4l0-21swapper/123:30:591
275991120,0ptp4l0-21swapper/123:30:581
275991120,0ptp4l0-21swapper/123:15:151
275991120,0ptp4l0-21swapper/123:13:311
275991120,0ptp4l0-21swapper/120:14:011
275991120,0ptp4l0-21swapper/100:08:151
1626999122,6cyclictest0-21swapper/1119:10:243
1626999122,6cyclictest0-21swapper/1100:20:263
1626499129,2cyclictest10085-21awk23:55:002
1626499120,11cyclictest0-21swapper/1020:50:002
1626099123,6cyclictest0-21swapper/923:50:0111
1626099121,6cyclictest0-21swapper/921:40:0011
1626099120,7cyclictest0-21swapper/920:55:0011
1626099120,11cyclictest0-21swapper/923:04:5911
1624699122,7cyclictest0-21swapper/620:04:598
1624699122,7cyclictest0-21swapper/600:20:018
1624299122,6cyclictest0-21swapper/523:27:297
1624299122,6cyclictest0-21swapper/523:27:287
1624299120,11cyclictest32484-21date19:30:007
156392121,4sleep30-21swapper/319:06:545
275991110,2ptp4l18007-21latency_hist00:05:001
275991110,2ptp4l17258-21cpuspeed_turbos21:15:141
275991110,2ptp4l11180-21hddtemp_smartct22:30:191
275991110,2ptp4l10333-21latency_hist23:55:011
275991110,1ptp4l21-21ksoftirqd/123:35:261
275991110,1ptp4l21-21ksoftirqd/123:35:261
275991110,1ptp4l21-21ksoftirqd/121:35:291
275991110,1ptp4l21-21ksoftirqd/121:00:051
275991110,0ptp4l0-21swapper/122:35:171
275991110,0ptp4l0-21swapper/122:20:161
275991110,0ptp4l0-21swapper/122:05:271
275991110,0ptp4l0-21swapper/121:55:231
275991110,0ptp4l0-21swapper/121:50:191
275991110,0ptp4l0-21swapper/121:45:191
275991110,0ptp4l0-21swapper/120:50:211
275991110,0ptp4l0-21swapper/120:35:141
275991110,0ptp4l0-21swapper/120:30:011
275991110,0ptp4l0-21swapper/119:55:141
275991110,0ptp4l0-21swapper/119:30:201
275991110,0ptp4l0-21swapper/100:10:091
191591110,0getstats0-21swapper/220:25:264
1626999112,5cyclictest0-21swapper/1120:00:003
1626999110,10cyclictest0-21swapper/1123:00:253
1626499111,6cyclictest0-21swapper/1021:40:002
1626099111,5cyclictest0-21swapper/919:55:0011
1624299110,7cyclictest0-21swapper/500:30:017
1623799118,2cyclictest0-21swapper/421:40:166
1623399111,6cyclictest0-21swapper/322:50:265
1622899110,10cyclictest7396-21expr23:50:154
275991100,2ptp4l24783-21fschecks_time20:00:151
275991100,2ptp4l21548-21cat22:00:231
275991100,1ptp4l9376-21grep19:40:201
275991100,1ptp4l31398-21expr22:15:201
275991100,1ptp4l22483-21fschecks_time22:45:181
275991100,1ptp4l19279-21cat19:10:261
275991100,1ptp4l0-21swapper/120:25:211
275991100,1ptp4l0-21swapper/100:15:141
275991100,0ptp4l0-21swapper/123:45:161
1626999108,1cyclictest0-21swapper/1121:05:163
1626999107,2cyclictest0-21swapper/1122:05:173
1626999100,9cyclictest31375-21irqrtprio20:50:183
1626999100,9cyclictest1898-21users23:40:263
1626999100,9cyclictest1898-21users23:40:253
1626999100,9cyclictest0-21swapper/1121:45:313
1626999100,9cyclictest0-21swapper/1121:40:193
1626099100,9cyclictest1020-21runrttasks21:01:3611
1625599100,9cyclictest0-21swapper/820:00:1710
1624699102,5cyclictest0-21swapper/622:35:288
1624699102,5cyclictest0-21swapper/620:40:018
1624699101,6cyclictest0-21swapper/621:30:008
1624699100,9cyclictest0-21swapper/622:50:258
1623799100,9cyclictest8356-21users21:00:276
1623399100,6cyclictest0-21swapper/322:40:275
1623399100,2cyclictest22825-21latency_hist00:15:015
1622899100,3cyclictest9102-21cat19:40:184
1622499107,2cyclictest17592-21/usr/sbin/munin22:40:161
1621799100,9cyclictest4677-21diskstats21:40:150
1621799100,9cyclictest0-21swapper/022:35:220
1621799100,2cyclictest15910-21irqrtprio00:00:180
125652101,6sleep90-21swapper/919:05:1311
27609190,1phc2sys0-21swapper/219:05:264
27599190,1ptp4l21-21ksoftirqd/123:05:171
27599190,1ptp4l21-21ksoftirqd/120:55:091
27599190,1ptp4l21-21ksoftirqd/120:05:151
27599190,0ptp4l0-21swapper/122:10:201
27599190,0ptp4l0-21swapper/121:30:151
27599190,0ptp4l0-21swapper/121:05:161
27599190,0ptp4l0-21swapper/120:40:131
27599190,0ptp4l0-21swapper/120:20:181
27599190,0ptp4l0-21swapper/120:15:181
27599190,0ptp4l0-21swapper/119:35:191
162699991,5cyclictest0-21swapper/1123:00:013
162699991,5cyclictest0-21swapper/1120:59:593
162649992,5cyclictest0-21swapper/1022:30:002
162649990,9cyclictest0-21swapper/1022:20:192
162649990,8cyclictest0-21swapper/1021:40:242
162649990,4cyclictest0-21swapper/1022:59:552
162649990,1cyclictest0-21swapper/1000:15:162
162609998,1cyclictest0-21swapper/900:35:1611
162609990,8cyclictest0-21swapper/922:35:1811
162609990,8cyclictest0-21swapper/921:58:2111
162519991,2cyclictest0-21swapper/723:45:189
162519990,3cyclictest15486-21munin-run23:20:009
162469997,1cyclictest26383-21latency_hist00:20:018
162469992,4cyclictest0-21swapper/619:35:018
162469991,5cyclictest0-21swapper/600:10:208
162469990,1cyclictest0-21swapper/621:40:258
162469990,1cyclictest0-21swapper/620:55:268
162429991,5cyclictest0-21swapper/522:35:007
162429991,5cyclictest0-21swapper/519:15:017
162429990,9cyclictest0-21swapper/520:30:157
162429990,2cyclictest3620-21tail20:55:227
162429990,1cyclictest0-21swapper/520:05:237
162429990,1cyclictest0-21swapper/519:45:007
162429990,1cyclictest0-21swapper/500:10:097
162379996,2cyclictest0-21swapper/423:25:136
162379996,2cyclictest0-21swapper/423:25:126
162379990,8cyclictest4247-21date20:55:266
162339997,1cyclictest0-21swapper/319:25:145
162339990,8cyclictest15643-21expr21:55:165
162339990,8cyclictest14802-21latency_hist21:55:005
162289998,1cyclictest0-21swapper/223:05:004
162289998,1cyclictest0-21swapper/222:15:194
162289992,7cyclictest2380-21sshd20:14:434
162289990,9cyclictest0-21swapper/223:10:274
162289990,8cyclictest27930-21latency20:45:194
162289990,8cyclictest23210-21cpuspeed22:05:154
162289990,8cyclictest16823-21irqrtprio21:55:204
162289990,8cyclictest10125-21expr21:05:184
162289990,8cyclictest0-21swapper/219:15:234
162289990,1cyclictest16527-21sed22:35:274
162289990,1cyclictest0-21swapper/223:27:294
162289990,1cyclictest0-21swapper/223:27:294
162179992,6cyclictest0-21swapper/020:15:000
162179990,8cyclictest0-21swapper/021:45:170
162179990,2cyclictest0-21swapper/019:10:170
30106280,1sleep030101-21turbostat22:15:000
20331280,0sleep50-21swapper/519:55:147
162699987,1cyclictest0-21swapper/1123:05:203
162699987,1cyclictest0-21swapper/1122:40:263
162699985,2cyclictest0-21swapper/1122:12:113
162699982,4cyclictest0-21swapper/1100:30:173
162699980,8cyclictest0-21swapper/1123:25:203
162699980,8cyclictest0-21swapper/1123:25:203
162699980,1cyclictest0-21swapper/1121:35:243
162699980,1cyclictest0-21swapper/1121:10:203
162699980,1cyclictest0-21swapper/1121:00:193
162699980,1cyclictest0-21swapper/1120:15:203
162699980,1cyclictest0-21swapper/1119:50:153
162649987,1cyclictest0-21swapper/1021:55:152
162649987,1cyclictest0-21swapper/1020:30:262
162649987,1cyclictest0-21swapper/1019:15:112
162649987,1cyclictest0-21swapper/1019:10:142
162649984,3cyclictest3683-21gdbus19:55:122
162649980,8cyclictest0-21swapper/1020:35:182
162649980,8cyclictest0-21swapper/1020:20:222
162649980,1cyclictest0-21swapper/1020:50:212
162649980,1cyclictest0-21swapper/1019:20:262
162609987,1cyclictest0-21swapper/923:05:2611
162609987,1cyclictest0-21swapper/922:25:1711
162609987,1cyclictest0-21swapper/921:20:1611
162609987,1cyclictest0-21swapper/900:10:1811
162609981,6cyclictest0-21swapper/923:20:0111
162609980,8cyclictest0-21swapper/923:00:0111
162609980,8cyclictest0-21swapper/920:05:2111
162609980,8cyclictest0-21swapper/919:55:2211
162609980,8cyclictest0-21swapper/900:25:2311
162609980,1cyclictest0-21swapper/919:10:1511
162559987,1cyclictest0-21swapper/823:50:1610
162559987,1cyclictest0-21swapper/823:40:1910
162559987,1cyclictest0-21swapper/823:40:1810
162559987,1cyclictest0-21swapper/822:05:2610
162559980,8cyclictest0-21swapper/821:25:2110
162559980,8cyclictest0-21swapper/821:10:1610
162559980,8cyclictest0-21swapper/820:25:0110
162559980,7cyclictest0-21swapper/823:45:5210
162559980,7cyclictest0-21swapper/823:05:1010
162559980,7cyclictest0-21swapper/821:58:1010
162559980,1cyclictest0-21swapper/820:20:2210
162559980,1cyclictest0-21swapper/800:10:5010
162519987,1cyclictest0-21swapper/700:25:019
162519987,1cyclictest0-21swapper/700:00:159
162519985,2cyclictest0-21swapper/723:15:009
162519984,3cyclictest3837-21pcscd21:15:169
162519984,3cyclictest3591-21gdbus19:30:539
162519980,1cyclictest0-21swapper/722:50:269
162519980,1cyclictest0-21swapper/722:00:479
162469980,8cyclictest113490irq/43-enp1s0-T23:50:218
162469980,8cyclictest0-21swapper/622:25:228
162469980,8cyclictest0-21swapper/622:05:288
162469980,8cyclictest0-21swapper/619:10:158
162469980,1cyclictest0-21swapper/621:30:168
162469980,1cyclictest0-21swapper/621:00:148
162469980,1cyclictest0-21swapper/619:45:138
162429987,1cyclictest0-21swapper/521:50:167
162429985,2cyclictest25789-21cat22:50:147
162429981,4cyclictest0-21swapper/521:05:017
162429980,1cyclictest0-21swapper/523:40:257
162429980,1cyclictest0-21swapper/523:40:257
162429980,1cyclictest0-21swapper/520:45:177
162429980,1cyclictest0-21swapper/520:10:167
162429980,1cyclictest0-21swapper/519:50:177
162429980,1cyclictest0-21swapper/500:35:157
162379987,1cyclictest0-21swapper/420:25:206
162379980,8cyclictest0-21swapper/421:05:246
162379980,7cyclictest0-21swapper/400:35:186
162379980,3cyclictest24778-21latency_hist19:20:016
162379980,3cyclictest19776-21cat23:25:016
162379980,3cyclictest19776-21cat23:25:016
162379980,1cyclictest0-21swapper/423:16:336
162379980,1cyclictest0-21swapper/422:35:186
162379980,1cyclictest0-21swapper/422:15:256
162379980,1cyclictest0-21swapper/419:55:266
162339982,5cyclictest0-21swapper/320:45:015
162339980,8cyclictest0-21swapper/323:35:145
162339980,8cyclictest0-21swapper/323:35:135
162339980,7cyclictest1020-21runrttasks20:24:205
162339980,7cyclictest0-21swapper/323:01:485
162289987,1cyclictest0-21swapper/222:45:264
162289987,1cyclictest0-21swapper/221:50:204
162289987,1cyclictest0-21swapper/221:25:184
162289987,1cyclictest0-21swapper/220:50:154
162289984,3cyclictest907-21dbus-daemon21:20:144
162289980,8cyclictest0-21swapper/223:30:254
162289980,8cyclictest0-21swapper/223:30:244
162289980,8cyclictest0-21swapper/222:55:214
162289980,7cyclictest4006-21sed20:55:244
162289980,7cyclictest3666-21cut00:30:194
162289980,7cyclictest17068-21cpuspeed21:15:134
162289980,7cyclictest0-21swapper/200:10:144
162289980,3cyclictest10672-21expr22:30:164
162289980,2cyclictest12991-21/usr/sbin/munin23:15:144
162289980,1cyclictest7359-21missed_timers21:00:214
162289980,1cyclictest30152-21awk00:25:014
162289980,1cyclictest0-21swapper/223:35:204
162289980,1cyclictest0-21swapper/223:35:204
162289980,1cyclictest0-21swapper/222:10:314
162289980,1cyclictest0-21swapper/221:45:184
162289980,1cyclictest0-21swapper/220:15:294
162289980,1cyclictest0-21swapper/220:05:224
162289980,1cyclictest0-21swapper/219:45:144
162289980,1cyclictest0-21swapper/200:35:124
162179980,1cyclictest0-21swapper/023:45:210
162179980,1cyclictest0-21swapper/021:30:150
6939270,0sleep50-21swapper/522:25:187
28928270,0sleep6501rcuc/620:05:168
162699976,1cyclictest0-21swapper/1123:50:133
162699976,1cyclictest0-21swapper/1123:23:003
162699976,1cyclictest0-21swapper/1123:22:593
162699976,1cyclictest0-21swapper/1122:45:443
162699976,1cyclictest0-21swapper/1122:30:213
162699976,1cyclictest0-21swapper/1122:22:483
162699976,1cyclictest0-21swapper/1121:50:263
162699976,1cyclictest0-21swapper/1120:41:063
162699976,1cyclictest0-21swapper/1120:35:183
162699976,1cyclictest0-21swapper/1119:45:233
162699976,1cyclictest0-21swapper/1119:30:013
162699976,1cyclictest0-21swapper/1119:27:313
162699976,1cyclictest0-21swapper/1100:26:583
162699976,1cyclictest0-21swapper/1100:15:243
162699974,2cyclictest15393-21aten2_r3power_p21:55:153
162699974,2cyclictest1020-21runrttasks19:21:093
162699973,3cyclictest24906-21hddtemp_smartct23:30:173
162699973,3cyclictest24906-21hddtemp_smartct23:30:163
162699973,3cyclictest11798-21head23:55:163
162699970,7cyclictest0-21swapper/1123:35:163
162699970,7cyclictest0-21swapper/1123:35:163
162699970,7cyclictest0-21swapper/1123:14:213
162699970,7cyclictest0-21swapper/1122:36:263
162699970,7cyclictest0-21swapper/1121:25:013
162699970,7cyclictest0-21swapper/1120:30:263
162699970,7cyclictest0-21swapper/1120:10:593
162699970,7cyclictest0-21swapper/1120:05:213
162699970,7cyclictest0-21swapper/1119:40:223
162699970,7cyclictest0-21swapper/1100:35:263
162699970,1cyclictest0-21swapper/1123:48:183
162699970,1cyclictest0-21swapper/1123:15:543
162699970,1cyclictest0-21swapper/1122:29:033
162699970,1cyclictest0-21swapper/1122:15:173
162699970,1cyclictest0-21swapper/1121:15:393
162699970,1cyclictest0-21swapper/1120:48:233
162699970,1cyclictest0-21swapper/1120:26:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional