You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 16:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Fri Nov 28, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911240,112ptp4l0-21swapper/107:08:361
272692980,0sleep100-21swapper/1009:55:162
38602830,0sleep20-21swapper/208:40:234
2370627362,9sleep50-21swapper/507:05:047
2488127050,9sleep70-21swapper/707:05:189
2691626655,9sleep110-21swapper/1107:08:473
2693126553,10sleep00-21swapper/007:09:000
2699426453,9sleep100-21swapper/1007:09:462
2692426049,9sleep60-21swapper/607:08:548
2677525948,9sleep40-21swapper/407:06:416
273142560,0sleep100-21swapper/1012:00:172
27434992011,7cyclictest0-21swapper/711:15:009
2741199203,10cyclictest0-21swapper/211:35:014
2741199203,10cyclictest0-21swapper/209:15:004
2744999183,9cyclictest0-21swapper/912:05:0111
27430991817,1cyclictest0-21swapper/608:50:168
2740699182,9cyclictest0-21swapper/108:40:001
2741699173,9cyclictest0-21swapper/307:40:005
2741699173,8cyclictest0-21swapper/310:45:005
2741699173,8cyclictest0-21swapper/310:45:005
2741699173,8cyclictest0-21swapper/307:10:015
27416991713,2cyclictest0-21swapper/308:30:185
2741199173,9cyclictest0-21swapper/208:55:014
2741199173,8cyclictest0-21swapper/208:05:004
275991160,1ptp4l21-21ksoftirqd/111:00:231
275991160,1ptp4l21-21ksoftirqd/111:00:221
275991160,0ptp4l0-21swapper/112:10:171
275991160,0ptp4l0-21swapper/111:10:151
275991160,0ptp4l0-21swapper/109:00:311
275991160,0ptp4l0-21swapper/107:45:141
2743499163,8cyclictest0-21swapper/712:15:019
2743499163,8cyclictest0-21swapper/709:45:009
2743499163,8cyclictest0-21swapper/709:00:009
2741699163,8cyclictest0-21swapper/312:35:005
2741699160,9cyclictest0-21swapper/310:35:005
2741199163,9cyclictest0-21swapper/208:20:004
237092168,6sleep80-21swapper/807:05:0710
201542160,0sleep110-21swapper/1111:10:133
275991150,6ptp4l0-21swapper/112:20:261
275991150,1ptp4l21-21ksoftirqd/108:25:101
275991150,0ptp4l0-21swapper/112:00:131
275991150,0ptp4l0-21swapper/111:55:141
275991150,0ptp4l0-21swapper/108:05:371
2744999153,7cyclictest0-21swapper/912:30:0011
2744999152,9cyclictest0-21swapper/909:15:0011
2744999151,2cyclictest0-21swapper/908:25:1611
2741699153,7cyclictest0-21swapper/310:25:005
2741199152,8cyclictest0-21swapper/209:30:004
2741199151,3cyclictest0-21swapper/209:55:204
2740699153,7cyclictest0-21swapper/111:25:001
2740699153,7cyclictest0-21swapper/109:20:001
2740699152,7cyclictest0-21swapper/109:10:001
2740299153,9cyclictest0-21swapper/010:55:000
2740299153,9cyclictest0-21swapper/010:55:000
191491150,0getstats0-21swapper/108:50:281
191491150,0getstats0-21swapper/108:45:281
49922140,0sleep20-21swapper/210:50:144
49922140,0sleep20-21swapper/210:50:144
275991140,1ptp4l21-21ksoftirqd/112:05:201
2745399142,8cyclictest0-21swapper/1007:40:252
2745399140,8cyclictest389-21munin-run08:40:012
2744999143,7cyclictest0-21swapper/910:50:0011
2744999143,7cyclictest0-21swapper/910:50:0011
2743499143,7cyclictest0-21swapper/708:35:009
2743499142,7cyclictest0-21swapper/710:10:009
2743499142,7cyclictest0-21swapper/707:45:249
2742199143,7cyclictest0-21swapper/412:25:226
2742199143,6cyclictest0-21swapper/411:25:266
2742199140,2cyclictest20209-21if_err_enp1s0.109:45:176
2742199140,13cyclictest14209-21cat08:15:006
2741699142,7cyclictest0-21swapper/310:15:235
2741699142,7cyclictest0-21swapper/308:50:245
2741699142,7cyclictest0-21swapper/307:30:005
2741699142,6cyclictest0-21swapper/311:55:235
2741199140,3cyclictest0-21swapper/209:15:194
2740699141,8cyclictest0-21swapper/109:15:001
93662130,1sleep3321rcuc/312:20:135
275991130,1ptp4l21-21ksoftirqd/110:12:451
275991130,1ptp4l21-21ksoftirqd/109:20:171
275991130,0ptp4l0-21swapper/111:35:551
275991130,0ptp4l0-21swapper/110:06:461
275991130,0ptp4l0-21swapper/108:37:231
275991130,0ptp4l0-21swapper/107:37:381
2744499132,7cyclictest0-21swapper/811:20:0010
2744499132,7cyclictest0-21swapper/809:00:3410
2743499133,6cyclictest0-21swapper/710:10:019
2743499132,7cyclictest0-21swapper/710:55:239
2743499132,7cyclictest0-21swapper/710:55:239
2743099133,7cyclictest0-21swapper/611:45:008
2743099133,7cyclictest0-21swapper/608:15:008
2743099131,8cyclictest30830-21kworker/u24:010:30:008
2742199136,7cyclictest0-21swapper/411:15:106
2742199130,3cyclictest4175-21cat07:20:006
2742199130,2cyclictest25466-21cat08:30:016
2741699132,7cyclictest0-21swapper/309:40:005
2741699132,7cyclictest0-21swapper/308:20:245
2741699132,6cyclictest0-21swapper/307:35:005
2741699131,7cyclictest0-21swapper/312:10:245
2740299132,6cyclictest0-21swapper/011:35:000
198852130,0sleep00-21swapper/008:20:190
53812120,0sleep20-21swapper/208:45:124
30762120,0sleep110-21swapper/1107:15:233
275991120,3ptp4l20063-21hddtemp_smartct09:45:161
275991120,0ptp4l0-21swapper/112:30:361
275991120,0ptp4l0-21swapper/112:18:471
275991120,0ptp4l0-21swapper/109:42:311
275991120,0ptp4l0-21swapper/108:21:091
2745399122,6cyclictest0-21swapper/1010:20:232
2744999122,6cyclictest0-21swapper/911:30:0011
2744999122,6cyclictest0-21swapper/910:20:0011
2744499123,6cyclictest0-21swapper/812:20:0010
2744499120,7cyclictest0-21swapper/810:10:0110
2743099122,6cyclictest0-21swapper/608:25:238
2742199123,2cyclictest818-21unixbench_singl10:00:256
2742199123,2cyclictest27163-21irqrtprio12:00:176
2742199122,6cyclictest0-21swapper/411:40:256
2742199122,3cyclictest31329-21cat12:05:206
2742199120,3cyclictest11160-21chrt07:26:306
2742199120,2cyclictest7593-21idleruntime-cro07:25:006
2742199120,1cyclictest0-21swapper/411:10:016
2741699122,7cyclictest0-21swapper/307:50:005
2740699122,6cyclictest0-21swapper/112:30:001
2740299122,6cyclictest0-21swapper/007:20:000
275991110,2ptp4l29308-21cstates10:40:131
275991110,2ptp4l29308-21cstates10:40:121
275991110,2ptp4l23443-21/usr/sbin/munin09:50:151
275991110,2ptp4l15683-21latency_hist07:35:001
275991110,1ptp4l0-21swapper/111:30:151
275991110,1ptp4l0-21swapper/109:35:161
275991110,0ptp4l0-21swapper/110:16:391
2745399117,3cyclictest2113-21cpuspeed_turbos10:05:122
2745399112,6cyclictest0-21swapper/1011:00:272
2745399112,6cyclictest0-21swapper/1011:00:262
2745399110,5cyclictest8346-21awk12:20:002
2744499112,6cyclictest0-21swapper/810:55:0010
2744499112,6cyclictest0-21swapper/810:55:0010
2743499112,6cyclictest0-21swapper/709:20:239
2743099111,9cyclictest0-21swapper/611:30:178
2743099111,9cyclictest0-21swapper/607:25:168
2742199111,6cyclictest0-21swapper/408:00:236
2742199110,3cyclictest8900-21chrt10:11:596
2742199110,3cyclictest7366-21chrt10:52:046
2742199110,3cyclictest7366-21chrt10:52:046
2742199110,3cyclictest7314-21ptp4l-jitter12:15:196
2742199110,3cyclictest4569-21idleruntime-cro12:15:016
2742199110,3cyclictest30929-21cat07:14:556
2742199110,3cyclictest30554-21latency10:40:186
2742199110,3cyclictest30554-21latency10:40:186
2742199110,3cyclictest30228-21chrt11:20:316
2742199110,3cyclictest28403-21fw_forwarded_lo07:50:166
2742199110,3cyclictest27216-21chrt09:11:186
2742199110,3cyclictest23730-21latency11:55:176
2742199110,3cyclictest11009-21cat10:15:176
2742199110,1cyclictest0-21swapper/412:30:166
2742199110,1cyclictest0-21swapper/410:09:216
2742199110,11cyclictest0-21swapper/410:45:246
2742199110,11cyclictest0-21swapper/410:45:236
2742199110,10cyclictest23654-21latency_hist09:10:006
2742199110,10cyclictest11941-21/usr/sbin/munin07:30:166
2742199110,10cyclictest0-21swapper/409:24:236
2741699118,2cyclictest25194-21turbostat08:30:005
2741199118,2cyclictest0-21swapper/207:25:234
2741199110,10cyclictest0-21swapper/210:35:234
2741199110,10cyclictest0-21swapper/210:35:224
275991100,1ptp4l5079-21diskstats10:50:151
275991100,1ptp4l5079-21diskstats10:50:141
275991100,1ptp4l3634-21awk09:25:111
275991100,1ptp4l31111-21df_inode10:00:141
275991100,1ptp4l23125-21timerwakeupswit12:35:261
275991100,1ptp4l21324-21irqrtprio07:40:191
275991100,1ptp4l12471-21strings11:40:201
275991100,1ptp4l10947-21cstates08:10:181
275991100,0ptp4l0-21swapper/111:45:171
275991100,0ptp4l0-21swapper/110:35:141
275991100,0ptp4l0-21swapper/110:35:141
275991100,0ptp4l0-21swapper/109:00:001
275991100,0ptp4l0-21swapper/108:04:191
275991100,0ptp4l0-21swapper/107:55:141
275991100,0ptp4l0-21swapper/107:15:001
2745899100,9cyclictest9660-21df_inode12:20:133
2745899100,9cyclictest24411-21latency_hist10:35:003
2745399102,5cyclictest0-21swapper/1009:05:002
2745399101,5cyclictest0-21swapper/1010:55:012
2745399101,5cyclictest0-21swapper/1010:55:002
2745399100,3cyclictest21892-21tr11:55:102
2744999102,5cyclictest0-21swapper/908:25:0011
2744999100,9cyclictest0-21swapper/907:35:1711
2744499102,5cyclictest0-21swapper/811:30:0010
2744499102,5cyclictest0-21swapper/809:50:0010
2744499102,5cyclictest0-21swapper/808:50:0010
2744499101,5cyclictest0-21swapper/810:05:0010
2743499100,9cyclictest23656-21diskmemload11:00:159
2743499100,9cyclictest23656-21diskmemload11:00:149
2743099100,1cyclictest0-21swapper/612:10:178
2742699100,9cyclictest2885-21timerandwakeup10:45:267
2742699100,9cyclictest2885-21timerandwakeup10:45:257
2742699100,9cyclictest23656-21diskmemload11:01:247
2742699100,9cyclictest23656-21diskmemload11:01:247
2742199102,3cyclictest3893-21grep09:25:126
2742199102,3cyclictest2077-21cut08:40:146
2742199102,2cyclictest20689-21expr08:20:246
2742199102,1cyclictest30342-21awk09:15:236
2742199101,3cyclictest13569-21cpu10:20:126
2742199101,3cyclictest0-21swapper/411:00:186
2742199101,3cyclictest0-21swapper/411:00:186
2742199100,9cyclictest0-21swapper/410:30:206
2742199100,9cyclictest0-21swapper/408:00:006
2742199100,7cyclictest0-21swapper/410:57:486
2742199100,7cyclictest0-21swapper/410:57:476
2742199100,4cyclictest29653-21chrt09:58:226
2742199100,3cyclictest9775-21switchtime09:30:246
2742199100,3cyclictest7920-21if_enp1s0.108:05:176
2742199100,3cyclictest6818-21awk11:35:136
2742199100,3cyclictest6047-21latency_hist11:35:006
2742199100,3cyclictest5303-21cpuspeed_turbos08:45:126
2742199100,3cyclictest28079-21sshd10:37:406
2742199100,3cyclictest28079-21sshd10:37:396
2742199100,3cyclictest26791-21fschecks_time08:30:166
2742199100,3cyclictest23836-21cpuspeed_turbos07:45:146
2742199100,3cyclictest23780-21cut09:50:166
2742199100,3cyclictest22422-21unixbench-2d07:40:266
2742199100,3cyclictest20955-21if_err_enp1s011:10:176
2742199100,3cyclictest20487-21cpu12:35:146
2742199100,3cyclictest19490-21users09:00:346
2742199100,3cyclictest19107-21sed10:25:196
2742199100,3cyclictest18322-21timerandwakeup07:35:236
2742199100,3cyclictest15773-21expr09:40:136
2742199100,3cyclictest15053-21grep11:45:176
2742199100,3cyclictest11986-21entropy09:35:156
2742199100,3cyclictest11232-21cat12:20:236
2740299102,5cyclictest0-21swapper/011:45:010
2740299102,5cyclictest0-21swapper/010:25:250
2740299102,5cyclictest0-21swapper/009:40:000
2740299102,5cyclictest0-21swapper/007:30:250
2740299102,4cyclictest0-21swapper/011:20:000
27599190,0ptp4l0-21swapper/111:50:121
27599190,0ptp4l0-21swapper/111:25:161
27599190,0ptp4l0-21swapper/111:15:111
27599190,0ptp4l0-21swapper/111:05:121
27599190,0ptp4l0-21swapper/110:55:121
27599190,0ptp4l0-21swapper/110:55:121
27599190,0ptp4l0-21swapper/110:45:131
27599190,0ptp4l0-21swapper/110:45:131
27599190,0ptp4l0-21swapper/110:30:131
27599190,0ptp4l0-21swapper/110:25:131
27599190,0ptp4l0-21swapper/110:20:091
27599190,0ptp4l0-21swapper/109:55:151
27599190,0ptp4l0-21swapper/109:30:121
27599190,0ptp4l0-21swapper/108:30:141
27599190,0ptp4l0-21swapper/108:15:131
27599190,0ptp4l0-21swapper/107:50:161
27599190,0ptp4l0-21swapper/107:25:141
27599190,0ptp4l0-21swapper/107:20:141
27599190,0ptp4l0-21swapper/107:15:011
274589990,9cyclictest0-21swapper/1107:25:123
274449998,1cyclictest0-21swapper/811:45:1410
274449991,5cyclictest0-21swapper/810:45:0010
274449991,5cyclictest0-21swapper/810:45:0010
274449991,5cyclictest0-21swapper/810:30:0010
274449990,9cyclictest0-21swapper/809:20:0010
274449990,8cyclictest18565-21sed12:30:2210
274349990,9cyclictest0-21swapper/710:40:159
274349990,9cyclictest0-21swapper/710:40:149
274349990,8cyclictest0-21swapper/708:45:139
274309998,1cyclictest0-21swapper/611:25:238
274309992,4cyclictest0-21swapper/611:55:138
274309990,8cyclictest30294-21sshd10:00:078
274269990,9cyclictest0-21swapper/511:45:197
274269990,9cyclictest0-21swapper/510:00:147
274269990,8cyclictest25736-21diskstats10:35:127
274269990,8cyclictest25736-21diskstats10:35:117
274269990,1cyclictest13414-21ps07:30:197
274269990,1cyclictest0-21swapper/512:15:017
274269990,1cyclictest0-21swapper/508:05:167
274219990,3cyclictest30821-21idleruntime08:35:136
274219990,3cyclictest18837-21df11:50:136
274219990,3cyclictest14370-21cpu08:15:116
274219990,3cyclictest12916-21cpuspeed08:55:126
274169990,8cyclictest0-21swapper/310:55:115
274169990,8cyclictest0-21swapper/310:55:115
274169990,8cyclictest0-21swapper/310:45:255
274169990,8cyclictest0-21swapper/310:45:255
274169990,2cyclictest14313-21aten2_r3power_v11:45:155
274119998,1cyclictest0-21swapper/212:30:254
274119990,9cyclictest0-21swapper/208:35:094
274119990,9cyclictest0-21swapper/207:20:244
274119990,8cyclictest21863-21chrt11:55:094
274119990,8cyclictest0-21swapper/211:50:244
274119990,8cyclictest0-21swapper/210:20:154
274119990,8cyclictest0-21swapper/209:05:254
274119990,3cyclictest21327-21/usr/sbin/munin07:40:194
274029998,1cyclictest0-21swapper/010:45:190
274029998,1cyclictest0-21swapper/010:45:190
274029997,1cyclictest0-21swapper/012:05:170
274029992,5cyclictest0-21swapper/009:25:230
274029992,4cyclictest0-21swapper/010:00:250
274029991,5cyclictest0-21swapper/010:10:010
274589987,1cyclictest0-21swapper/1110:40:183
274589987,1cyclictest0-21swapper/1110:40:183
274589987,1cyclictest0-21swapper/1107:35:203
274589986,1cyclictest0-21swapper/1109:42:093
274589984,3cyclictest29023-21idleruntime-cro08:35:003
274589980,8cyclictest0-21swapper/1107:50:133
274589980,7cyclictest0-21swapper/1111:45:293
274589980,1cyclictest0-21swapper/1111:03:003
274589980,1cyclictest0-21swapper/1111:03:003
274589980,1cyclictest0-21swapper/1110:05:193
274589980,1cyclictest0-21swapper/1109:00:333
274589980,1cyclictest0-21swapper/1108:05:133
274589980,1cyclictest0-21swapper/1107:10:163
274539980,1cyclictest0-21swapper/1011:35:172
274499987,1cyclictest0-21swapper/909:45:1311
274499987,1cyclictest0-21swapper/908:15:2411
274499980,8cyclictest0-21swapper/912:30:1511
274499980,8cyclictest0-21swapper/908:55:1211
274499980,8cyclictest0-21swapper/908:39:5611
274499980,8cyclictest0-21swapper/907:20:1511
274499980,4cyclictest0-21swapper/909:15:2611
274499980,1cyclictest0-21swapper/912:20:0011
274499980,1cyclictest0-21swapper/909:57:5311
274499980,1cyclictest0-21swapper/909:40:2511
274499980,1cyclictest0-21swapper/907:50:2111
274499980,1cyclictest0-21swapper/907:45:2411
274449987,1cyclictest0-21swapper/809:30:1510
274449987,1cyclictest0-21swapper/809:25:1710
274449987,1cyclictest0-21swapper/809:08:3810
274449985,2cyclictest25210-21sed07:45:1910
274449982,4cyclictest0-21swapper/812:20:2510
274449981,4cyclictest0-21swapper/811:35:0010
274449980,8cyclictest0-21swapper/811:20:1310
274449980,8cyclictest0-21swapper/811:05:2510
274449980,8cyclictest0-21swapper/810:00:0010
274449980,8cyclictest0-21swapper/808:10:3010
274449980,8cyclictest0-21swapper/808:00:0010
274449980,1cyclictest0-21swapper/812:35:2010
274449980,1cyclictest0-21swapper/811:50:2510
274449980,1cyclictest0-21swapper/811:35:2210
274349987,1cyclictest0-21swapper/712:35:189
274349987,1cyclictest0-21swapper/711:35:159
274349987,1cyclictest0-21swapper/710:25:179
274349987,1cyclictest0-21swapper/709:10:169
274349982,6cyclictest0-21swapper/708:05:019
274349980,1cyclictest0-21swapper/711:40:259
274309987,1cyclictest0-21swapper/609:35:238
274309987,1cyclictest0-21swapper/609:10:168
274309980,8cyclictest0-21swapper/612:05:008
274309980,8cyclictest0-21swapper/609:30:128
274309980,1cyclictest0-21swapper/607:30:258
274269987,1cyclictest0-21swapper/512:30:387
274269987,1cyclictest0-21swapper/509:55:267
274269987,1cyclictest0-21swapper/509:05:237
274269987,1cyclictest0-21swapper/509:00:297
274269987,1cyclictest0-21swapper/508:30:167
274269987,1cyclictest0-21swapper/508:00:187
274269987,1cyclictest0-21swapper/507:15:127
274269980,8cyclictest0-21swapper/510:25:197
274269980,8cyclictest0-21swapper/509:50:177
274269980,8cyclictest0-21swapper/509:25:147
274269980,8cyclictest0-21swapper/508:15:137
274269980,8cyclictest0-21swapper/507:25:247
274269980,1cyclictest0-21swapper/510:13:397
274269980,1cyclictest0-21swapper/508:50:017
274269980,1cyclictest0-21swapper/507:40:187
274219980,3cyclictest9396-21cstates08:50:136
274169983,4cyclictest0-21swapper/307:55:015
274169980,7cyclictest0-21swapper/311:21:225
274169980,1cyclictest0-21swapper/312:25:185
274169980,1cyclictest0-21swapper/312:00:155
274169980,1cyclictest0-21swapper/311:40:055
274169980,1cyclictest0-21swapper/310:00:135
274119987,1cyclictest0-21swapper/210:45:004
274119987,1cyclictest0-21swapper/210:45:004
274119987,1cyclictest0-21swapper/210:00:254
274119987,1cyclictest0-21swapper/208:05:184
274119980,8cyclictest0-21swapper/210:10:154
274119980,7cyclictest0-21swapper/211:45:174
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional