You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 14:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Sun Nov 23, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911130,57ptp4l0-21swapper/107:05:071
182732710,0sleep10-21swapper/109:55:131
1832527055,7sleep50-21swapper/507:07:357
1845326553,10sleep00-21swapper/007:09:200
1839526453,9sleep40-21swapper/407:08:346
1840226351,10sleep110-21swapper/1107:08:423
1838826352,9sleep100-21swapper/1007:08:282
1836726344,9sleep60-21swapper/607:08:128
1765326350,10sleep70-21swapper/707:05:259
91832610,0sleep110-21swapper/1108:20:013
312512470,0sleep50-21swapper/510:55:007
1887599222,16cyclictest0-21swapper/409:00:016
18902992019,1cyclictest0-21swapper/1010:25:232
1888499203,10cyclictest0-21swapper/607:15:008
1887599181,13cyclictest0-21swapper/409:45:026
1890299170,8cyclictest0-21swapper/1012:30:242
1888899176,8cyclictest9937-21kworker/u24:407:45:009
1888899171,12cyclictest5175-21kworker/u24:110:50:009
275991160,1ptp4l21-21ksoftirqd/109:25:231
275991160,1ptp4l21-21ksoftirqd/107:15:201
275991160,1ptp4l21-21ksoftirqd/107:15:201
275991160,0ptp4l0-21swapper/111:20:241
275991160,0ptp4l0-21swapper/108:55:371
1885999162,10cyclictest5175-21kworker/u24:109:15:011
183432167,7sleep80-21swapper/807:07:5110
17572160,0sleep30-21swapper/307:25:235
17572160,0sleep30-21swapper/307:25:225
275991150,1ptp4l21-21ksoftirqd/109:20:231
275991150,0ptp4l0-21swapper/111:55:231
275991150,0ptp4l0-21swapper/110:29:391
275991150,0ptp4l0-21swapper/107:55:191
1889799153,7cyclictest0-21swapper/909:15:0111
1889799150,2cyclictest0-21swapper/912:20:1711
1889799150,10cyclictest18758-21df_inode09:55:1411
1889399152,8cyclictest0-21swapper/809:30:0110
1889399152,8cyclictest0-21swapper/808:05:0010
1888899153,7cyclictest0-21swapper/709:05:249
1888899152,8cyclictest0-21swapper/711:05:009
1888899150,3cyclictest0-21swapper/708:35:179
1888499153,7cyclictest0-21swapper/610:50:008
1887999153,7cyclictest0-21swapper/510:20:017
1887599152,8cyclictest0-21swapper/412:10:266
1887599150,2cyclictest0-21swapper/410:35:186
1886999153,7cyclictest0-21swapper/312:40:005
1886999152,8cyclictest0-21swapper/309:35:255
1886599152,9cyclictest0-21swapper/211:05:004
1885499153,7cyclictest0-21swapper/012:00:000
1885499153,7cyclictest0-21swapper/011:20:000
275991140,1ptp4l21-21ksoftirqd/107:50:181
275991140,0ptp4l0-21swapper/110:31:091
275991140,0ptp4l0-21swapper/108:30:001
275991140,0ptp4l0-21swapper/108:10:001
1889399143,7cyclictest0-21swapper/807:25:0010
1889399143,7cyclictest0-21swapper/807:25:0010
1889399143,6cyclictest0-21swapper/812:15:0010
1889399142,7cyclictest0-21swapper/809:40:2310
1889399142,7cyclictest0-21swapper/808:35:0110
1888899143,7cyclictest0-21swapper/710:55:269
1888899142,7cyclictest0-21swapper/711:35:009
1888899142,7cyclictest0-21swapper/708:55:019
1888899142,7cyclictest0-21swapper/707:58:279
1888499142,7cyclictest0-21swapper/611:00:008
1887999142,8cyclictest0-21swapper/508:55:017
275991130,1ptp4l21-21ksoftirqd/111:50:031
275991130,1ptp4l21-21ksoftirqd/111:04:101
275991130,1ptp4l21-21ksoftirqd/110:17:531
275991130,1ptp4l21-21ksoftirqd/108:10:191
275991130,1ptp4l21-21ksoftirqd/107:10:241
275991130,0ptp4l0-21swapper/110:55:221
275991130,0ptp4l0-21swapper/109:44:091
275991130,0ptp4l0-21swapper/108:46:071
275991130,0ptp4l0-21swapper/107:46:581
1890699130,12cyclictest2699-21munin-run11:00:003
1889399132,7cyclictest0-21swapper/810:20:0110
1889399132,6cyclictest0-21swapper/811:45:0010
1888899132,7cyclictest0-21swapper/709:25:019
1888899132,6cyclictest0-21swapper/708:25:019
1888499132,7cyclictest0-21swapper/608:00:238
1888499132,6cyclictest0-21swapper/607:55:018
1888499130,8cyclictest0-21swapper/610:40:008
1887999133,9cyclictest0-21swapper/507:15:017
1887999133,6cyclictest0-21swapper/508:20:027
1887999132,7cyclictest0-21swapper/511:55:007
1887999132,7cyclictest0-21swapper/510:40:007
1887999132,7cyclictest0-21swapper/507:35:007
1887999132,7cyclictest0-21swapper/507:35:007
1887999132,6cyclictest0-21swapper/512:35:007
1886999133,6cyclictest0-21swapper/310:45:005
1886999133,6cyclictest0-21swapper/308:35:005
1886599132,8cyclictest0-21swapper/210:45:004
1886599132,7cyclictest0-21swapper/211:00:004
275991120,4ptp4l21-21ksoftirqd/110:10:171
275991120,1ptp4l21-21ksoftirqd/111:39:431
275991120,0ptp4l0-21swapper/112:35:141
275991120,0ptp4l0-21swapper/112:05:161
275991120,0ptp4l0-21swapper/111:06:201
275991120,0ptp4l0-21swapper/110:40:261
275991120,0ptp4l0-21swapper/107:40:201
275991120,0ptp4l0-21swapper/107:30:221
275991120,0ptp4l0-21swapper/107:30:211
202982120,0chrt20297-21smart_sda09:55:222
1890299122,5cyclictest0-21swapper/1008:10:012
1890299120,8cyclictest0-21swapper/1010:10:012
1889799121,7cyclictest0-21swapper/912:10:0011
1889799121,7cyclictest0-21swapper/910:00:2311
1889399122,6cyclictest0-21swapper/808:20:0110
1888499122,6cyclictest0-21swapper/608:10:258
1887999122,6cyclictest0-21swapper/508:25:017
1887999121,6cyclictest0-21swapper/512:05:007
1887599121,7cyclictest0-21swapper/409:30:246
1886999121,6cyclictest0-21swapper/310:30:245
1886599122,6cyclictest0-21swapper/212:20:004
1886599122,6cyclictest0-21swapper/209:35:014
1885999121,7cyclictest0-21swapper/110:25:001
1885499122,6cyclictest0-21swapper/010:25:250
1885499122,6cyclictest0-21swapper/008:10:010
1885499120,8cyclictest0-21swapper/012:30:000
276091110,3phc2sys20204-21sed11:20:174
275991110,3ptp4l28862-21switchtime08:00:221
275991110,3ptp4l23297-21ptp4l-jitter08:35:201
275991110,3ptp4l12892-21basename08:25:001
275991110,3ptp4l0-21swapper/110:00:011
275991110,1ptp4l21-21ksoftirqd/112:20:211
275991110,1ptp4l21-21ksoftirqd/109:30:211
275991110,1ptp4l21-21ksoftirqd/108:30:161
275991110,1ptp4l0-21swapper/112:30:181
275991110,0ptp4l0-21swapper/111:30:251
275991110,0ptp4l0-21swapper/110:48:561
275991110,0ptp4l0-21swapper/109:45:011
275991110,0ptp4l0-21swapper/108:41:541
275991110,0ptp4l0-21swapper/107:39:111
275991110,0ptp4l0-21swapper/107:39:111
1890699112,5cyclictest0-21swapper/1112:30:263
1889799111,5cyclictest0-21swapper/911:50:0011
1889799110,10cyclictest20435-21irqrtprio11:20:1811
1889399110,7cyclictest0-21swapper/809:55:0110
1888899113,3cyclictest8061-21aten2_r3power_p12:30:149
1888899113,2cyclictest11019-21latency_hist11:10:009
1888899111,3cyclictest0-21swapper/709:25:219
1888899111,2cyclictest0-21swapper/709:14:009
1888899110,4cyclictest9017-21wc11:50:009
1888899110,4cyclictest6176-21latency_hist09:40:019
1888899110,4cyclictest25049-21cat08:40:129
1888899110,4cyclictest23299-21ptp4l12:05:219
1888899110,4cyclictest18112-21egrep09:55:139
1888899110,4cyclictest12182-21timerandwakeup111:50:259
1888899110,4cyclictest10265-21ptp4l-jitter09:00:219
1888899110,3cyclictest8204-21sshd10:24:389
1888899110,3cyclictest31124-21chrt10:54:029
1888899110,3cyclictest18053-21chrt11:18:049
1888899110,3cyclictest14654-21idleruntime11:55:169
1888899110,11cyclictest0-21swapper/710:44:299
1886999118,2cyclictest0-21swapper/309:15:015
1886599118,2cyclictest25859-21turbostat11:30:004
1886599118,2cyclictest0-21swapper/208:45:144
1886599112,6cyclictest0-21swapper/212:25:004
1886599112,6cyclictest0-21swapper/210:05:244
1886599112,6cyclictest0-21swapper/209:15:014
1886599112,6cyclictest0-21swapper/208:05:024
1886599110,3cyclictest12442-21latency_hist10:30:004
275991100,6ptp4l21-21ksoftirqd/111:49:251
275991100,1ptp4l8966-21sort09:00:141
275991100,1ptp4l8521-21cat08:15:251
275991100,1ptp4l4975-21smartctl09:35:221
275991100,1ptp4l29642-21irqrtprio07:20:161
275991100,1ptp4l29642-21irqrtprio07:20:161
275991100,1ptp4l26447-21/usr/sbin/munin10:05:131
275991100,1ptp4l21-21ksoftirqd/112:12:261
275991100,0ptp4l0-21swapper/111:25:141
1890699108,1cyclictest6837-21sed08:55:433
1890699102,5cyclictest0-21swapper/1111:10:003
1890699102,5cyclictest0-21swapper/1110:15:013
1890699100,5cyclictest0-21swapper/1109:14:553
1890699100,3cyclictest19592-21runrttasks10:38:073
1890699100,2cyclictest3115-21users08:50:243
1890299107,2cyclictest20472-21irqstats11:20:182
1890299100,9cyclictest0-21swapper/1009:40:012
1889799100,9cyclictest0-21swapper/911:00:1311
1889399102,5cyclictest0-21swapper/808:15:0110
1888899106,4cyclictest0-21swapper/712:22:579
1888899103,2cyclictest9902-21cpuspeed_turbos08:20:149
1888899102,3cyclictest20409-21irqrtprio11:20:179
1888899102,3cyclictest1804-21cpu11:40:129
1888899102,3cyclictest12571-21ptp4l-jitter09:45:229
1888899101,2cyclictest1676-21taskset09:31:459
1888899101,2cyclictest0-21swapper/707:10:239
1888899100,9cyclictest0-21swapper/711:25:029
1888899100,4cyclictest848-21aten2_r3power_c10:15:159
1888899100,4cyclictest5523-21awk12:25:149
1888899100,4cyclictest27851-21irqrtprio08:00:179
1888899100,4cyclictest2270-21expr08:10:159
1888899100,4cyclictest22433-21unixbench_singl09:15:249
1888899100,4cyclictest20000-21timerwakeupswit08:30:259
1888899100,4cyclictest16740-21cpuspeed_turbos10:35:149
1888899100,3cyclictest8198-21irqstats09:40:189
1888899100,3cyclictest6498-21chrt07:32:039
1888899100,3cyclictest6498-21chrt07:32:039
1888899100,3cyclictest585-21idleruntime07:25:159
1888899100,3cyclictest585-21idleruntime07:25:159
1888899100,3cyclictest4350-21/usr/sbin/munin08:55:329
1888899100,3cyclictest31277-21perl11:35:159
1888899100,3cyclictest30211-21/usr/sbin/munin08:05:129
1888899100,3cyclictest29396-21irqrtprio07:20:169
1888899100,3cyclictest29396-21irqrtprio07:20:169
1888899100,3cyclictest29139-21cpuspeed_turbos12:15:139
1888899100,3cyclictest27012-21if_err_enp1s0.110:05:169
1888899100,3cyclictest25055-21cpuspeed12:10:149
1888899100,3cyclictest23017-21idleruntime07:15:019
1888899100,3cyclictest23017-21idleruntime07:15:019
1888899100,3cyclictest22311-21cstates10:00:129
1888899100,3cyclictest21203-21sed07:50:249
1888899100,3cyclictest15855-21cut09:50:239
1888899100,3cyclictest12978-21fw_conntrack12:35:159
1888899100,3cyclictest10525-21turbostat.cron07:39:569
1888899100,3cyclictest10525-21turbostat.cron07:39:559
1888899100,1cyclictest0-21swapper/710:30:229
1888899100,10cyclictest0-21swapper/711:14:069
1887999107,2cyclictest0-21swapper/511:10:157
1887999102,5cyclictest0-21swapper/507:40:027
1887999102,5cyclictest0-21swapper/507:40:017
1887999100,9cyclictest0-21swapper/509:10:207
1886999107,2cyclictest0-21swapper/311:55:125
1886999100,9cyclictest0-21swapper/308:50:155
1886599102,5cyclictest0-21swapper/211:45:004
1886599101,6cyclictest0-21swapper/208:10:014
1886599100,9cyclictest0-21swapper/209:25:224
1886599100,9cyclictest0-21swapper/208:10:164
1885499102,5cyclictest0-21swapper/007:25:020
1885499102,5cyclictest0-21swapper/007:25:010
1885499100,2cyclictest13402-21diskstats10:30:140
27599190,1ptp4l21-21ksoftirqd/109:15:131
27599190,1ptp4l21-21ksoftirqd/109:05:151
27599190,1ptp4l21-21ksoftirqd/107:25:111
27599190,1ptp4l21-21ksoftirqd/107:25:111
27599190,1ptp4l1778-21cpu11:40:121
27599190,0ptp4l0-21swapper/112:25:121
27599190,0ptp4l0-21swapper/112:15:141
27599190,0ptp4l0-21swapper/112:00:001
27599190,0ptp4l0-21swapper/111:15:141
27599190,0ptp4l0-21swapper/111:10:001
27599190,0ptp4l0-21swapper/110:50:141
27599190,0ptp4l0-21swapper/110:35:141
27599190,0ptp4l0-21swapper/109:50:191
27599190,0ptp4l0-21swapper/108:50:121
189069991,6cyclictest0-21swapper/1108:30:233
189069990,9cyclictest0-21swapper/1109:05:203
189029998,1cyclictest0-21swapper/1010:30:152
189029998,1cyclictest0-21swapper/1009:15:012
189029990,9cyclictest11597-21strings11:50:202
189029990,9cyclictest0-21swapper/1011:40:152
189029990,1cyclictest0-21swapper/1009:25:202
189029990,1cyclictest0-21swapper/1008:20:172
188979998,1cyclictest0-21swapper/909:00:2411
188979992,4cyclictest0-21swapper/909:35:0211
188979991,5cyclictest0-21swapper/907:35:0211
188979991,5cyclictest0-21swapper/907:35:0111
188979991,4cyclictest0-21swapper/909:10:0111
188979990,9cyclictest0-21swapper/911:50:2511
188979990,9cyclictest0-21swapper/910:35:0011
188979990,4cyclictest0-21swapper/907:54:5511
188979990,1cyclictest0-21swapper/910:10:2411
188939998,1cyclictest0-21swapper/812:25:1210
188939998,1cyclictest0-21swapper/808:35:1310
188939998,1cyclictest0-21swapper/807:40:0110
188939996,2cyclictest0-21swapper/809:10:1710
188939991,4cyclictest0-21swapper/810:35:0010
188939990,9cyclictest0-21swapper/812:30:2510
188939990,9cyclictest0-21swapper/810:50:1910
188939990,9cyclictest0-21swapper/807:50:2310
188939990,9cyclictest0-21swapper/807:45:2010
188939990,8cyclictest0-21swapper/812:20:1410
188939990,1cyclictest0-21swapper/812:15:1510
188939990,1cyclictest0-21swapper/811:50:1410
188939990,1cyclictest0-21swapper/808:55:3310
188889990,3cyclictest9538-21expr10:25:159
188889990,3cyclictest5819-21cut08:15:139
188889990,3cyclictest30399-21grep10:10:179
188889990,3cyclictest29307-21cstates08:45:149
188889990,3cyclictest18040-21grep12:00:169
188889990,3cyclictest14999-21aten2_r3power_v07:45:139
188849996,2cyclictest0-21swapper/609:25:268
188849996,2cyclictest0-21swapper/607:35:168
188849996,2cyclictest0-21swapper/607:35:168
188849992,4cyclictest0-21swapper/612:30:178
188849990,8cyclictest0-21swapper/609:10:198
188849990,1cyclictest0-21swapper/609:30:138
188849990,1cyclictest0-21swapper/608:20:258
188799997,1cyclictest0-21swapper/509:25:227
188799992,4cyclictest0-21swapper/508:35:017
188799990,9cyclictest0-21swapper/511:30:007
188799990,1cyclictest0-21swapper/510:20:207
188799990,1cyclictest0-21swapper/507:50:187
188759992,5cyclictest0-21swapper/411:20:246
188759990,8cyclictest0-21swapper/412:30:176
188699992,4cyclictest0-21swapper/311:25:145
188699990,9cyclictest0-21swapper/308:55:435
188699990,9cyclictest0-21swapper/307:50:185
188659997,2cyclictest0-21swapper/212:00:004
188659996,2cyclictest21503-21sed09:15:184
188659990,8cyclictest418-21perl07:25:144
188659990,8cyclictest418-21perl07:25:144
188659990,8cyclictest10669-21fw_conntrack11:50:164
188659990,8cyclictest0-21swapper/207:55:164
188549998,1cyclictest0-21swapper/011:10:180
188549992,5cyclictest0-21swapper/009:15:010
188549991,5cyclictest0-21swapper/012:00:010
188549990,9cyclictest0-21swapper/008:55:390
29087280,0sleep110-21swapper/1110:10:013
189069987,1cyclictest0-21swapper/1112:35:183
189069987,1cyclictest0-21swapper/1109:50:263
189069986,1cyclictest28387-21cat08:45:013
189069982,4cyclictest0-21swapper/1112:30:003
189069982,4cyclictest0-21swapper/1108:30:013
189069981,5cyclictest0-21swapper/1109:50:013
189069980,8cyclictest0-21swapper/1107:20:203
189069980,8cyclictest0-21swapper/1107:20:203
189069980,1cyclictest0-21swapper/1112:10:253
189029987,1cyclictest0-21swapper/1012:00:182
189029986,1cyclictest12631-21df09:05:132
189029985,2cyclictest31775-21diskstats09:30:142
189029980,8cyclictest0-21swapper/1011:35:162
189029980,8cyclictest0-21swapper/1008:55:372
189029980,8cyclictest0-21swapper/1008:35:152
189029980,8cyclictest0-21swapper/1007:59:072
189029980,1cyclictest0-21swapper/1012:25:092
189029980,1cyclictest0-21swapper/1010:35:222
188979987,1cyclictest0-21swapper/909:35:2111
188979982,5cyclictest0-21swapper/907:15:0111
188979982,5cyclictest0-21swapper/907:15:0111
188979981,5cyclictest0-21swapper/907:25:0111
188979981,5cyclictest0-21swapper/907:25:0111
188979980,8cyclictest0-21swapper/911:05:4511
188979980,8cyclictest0-21swapper/910:45:1511
188979980,8cyclictest0-21swapper/910:20:2011
188979980,8cyclictest0-21swapper/909:20:1811
188979980,8cyclictest0-21swapper/908:40:1311
188979980,8cyclictest0-21swapper/908:35:2311
188979980,8cyclictest0-21swapper/908:30:1611
188979980,8cyclictest0-21swapper/907:45:1511
188979980,1cyclictest0-21swapper/911:35:2611
188979980,1cyclictest0-21swapper/911:30:1911
188939987,1cyclictest0-21swapper/811:55:0010
188939987,1cyclictest0-21swapper/811:10:1910
188939987,1cyclictest0-21swapper/809:55:2010
188939987,1cyclictest0-21swapper/809:35:1710
188939987,1cyclictest0-21swapper/807:15:2010
188939987,1cyclictest0-21swapper/807:15:2010
188939980,7cyclictest1020-21runrttasks10:29:4610
188939980,1cyclictest0-21swapper/811:05:0010
188939980,1cyclictest0-21swapper/808:52:2010
188849980,8cyclictest0-21swapper/612:35:138
188849980,8cyclictest0-21swapper/608:15:238
188849980,7cyclictest0-21swapper/609:35:458
188849980,1cyclictest0-21swapper/612:15:248
188849980,1cyclictest0-21swapper/611:54:078
188849980,1cyclictest0-21swapper/611:00:228
188849980,1cyclictest0-21swapper/609:48:448
188849980,1cyclictest0-21swapper/608:40:218
188799987,1cyclictest0-21swapper/512:05:147
188799980,8cyclictest0-21swapper/512:20:137
188799980,8cyclictest0-21swapper/511:10:007
188799980,1cyclictest0-21swapper/507:20:167
188799980,1cyclictest0-21swapper/507:20:167
188759987,1cyclictest0-21swapper/412:00:206
188759987,1cyclictest0-21swapper/410:49:116
188759987,1cyclictest0-21swapper/409:55:016
188759986,1cyclictest0-21swapper/412:15:166
188759986,1cyclictest0-21swapper/411:57:266
188759980,8cyclictest0-21swapper/411:55:006
188759980,7cyclictest0-21swapper/409:47:206
188759980,3cyclictest15527-21cat12:40:006
188759980,3cyclictest14564-21cat11:15:006
188759980,3cyclictest10538-21latency_hist11:10:006
188759980,3cyclictest0-21swapper/411:00:006
188759980,1cyclictest0-21swapper/408:40:146
188699987,1cyclictest0-21swapper/310:20:255
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional