You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 05:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Wed Dec 10, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
275991980,86ptp4l0-21swapper/119:05:191
3075826655,9sleep110-21swapper/1119:07:323
2996926653,10sleep50-21swapper/519:05:267
3089426453,9sleep00-21swapper/019:09:330
3087526351,10sleep70-21swapper/719:09:169
179362620,0sleep20-21swapper/221:35:264
3071226151,8sleep40-21swapper/419:06:496
3063526153,6sleep60-21swapper/619:05:508
3065226049,9sleep100-21swapper/1019:06:062
48512560,0sleep70-21swapper/700:10:009
48512560,0sleep70-21swapper/700:10:009
31135992016,3cyclictest0-21swapper/823:14:5610
275991190,1ptp4l21-21ksoftirqd/120:40:211
3110799183,9cyclictest0-21swapper/323:15:005
314972170,0sleep20-21swapper/221:15:004
3115099170,16cyclictest0-21swapper/1121:00:193
3112599172,8cyclictest0-21swapper/623:25:008
3110799173,9cyclictest0-21swapper/322:10:005
29652170,0sleep110-21swapper/1119:55:123
3115099161,10cyclictest0-21swapper/1120:10:003
3114199162,9cyclictest0-21swapper/923:25:0011
3114199162,8cyclictest0-21swapper/922:40:0011
3114199161,2cyclictest0-21swapper/922:00:1811
3113599164,7cyclictest0-21swapper/822:20:0010
3113599163,8cyclictest0-21swapper/821:00:0010
3113599162,8cyclictest0-21swapper/821:50:0010
3112599163,8cyclictest0-21swapper/622:55:008
3112599163,8cyclictest0-21swapper/620:50:008
3110199162,9cyclictest0-21swapper/200:30:004
275991160,0ptp4l0-21swapper/123:40:191
275991160,0ptp4l0-21swapper/122:55:161
3115099153,8cyclictest0-21swapper/1122:15:003
3114199153,7cyclictest0-21swapper/922:00:0011
3114199151,7cyclictest0-21swapper/921:35:0011
3113599152,8cyclictest0-21swapper/800:22:3310
3113599152,8cyclictest0-21swapper/800:00:0110
3112599153,8cyclictest19885-21kworker/u24:223:15:008
3112599153,8cyclictest0-21swapper/620:55:008
3112599151,8cyclictest0-21swapper/619:40:018
3111599150,10cyclictest6665-21cut20:40:186
3110799153,7cyclictest0-21swapper/322:25:005
3110799152,8cyclictest0-21swapper/321:05:005
3108799153,7cyclictest0-21swapper/021:55:260
3108799153,7cyclictest0-21swapper/020:00:010
3108799152,8cyclictest0-21swapper/019:40:010
306502157,6sleep80-21swapper/819:06:0410
275991150,1ptp4l9461-21grep20:45:141
275991150,1ptp4l21-21ksoftirqd/122:20:231
275991150,0ptp4l0-21swapper/120:15:281
275991150,0ptp4l0-21swapper/119:30:251
275991150,0ptp4l0-21swapper/119:20:251
162082150,0sleep20-21swapper/219:30:124
3115099141,7cyclictest0-21swapper/1123:40:003
3115099141,7cyclictest0-21swapper/1123:40:003
3114199143,7cyclictest0-21swapper/921:55:0011
3114199142,7cyclictest0-21swapper/923:05:2411
3114199142,7cyclictest0-21swapper/921:30:0011
3114199142,7cyclictest0-21swapper/900:30:0111
3114199142,7cyclictest0-21swapper/900:10:0111
3114199142,7cyclictest0-21swapper/900:10:0011
3113599142,7cyclictest0-21swapper/821:20:0010
3112599142,7cyclictest0-21swapper/621:30:008
3112599142,7cyclictest0-21swapper/621:10:278
3112599141,8cyclictest19885-21kworker/u24:219:25:018
3112199144,5cyclictest0-21swapper/522:05:007
3112199142,8cyclictest0-21swapper/520:05:017
3110799142,8cyclictest0-21swapper/323:40:015
3110799142,8cyclictest0-21swapper/323:40:005
3110799142,8cyclictest0-21swapper/321:25:005
275991140,1ptp4l21-21ksoftirqd/123:20:141
275991140,1ptp4l21-21ksoftirqd/122:25:261
275991140,0ptp4l0-21swapper/123:37:291
275991140,0ptp4l0-21swapper/123:37:291
275991140,0ptp4l0-21swapper/120:50:191
3114199133,6cyclictest0-21swapper/922:15:0011
3114199131,7cyclictest0-21swapper/922:35:0011
3113599131,11cyclictest0-21swapper/800:25:1910
3113599130,1cyclictest0-21swapper/823:20:2110
3112199132,7cyclictest0-21swapper/523:20:007
3111599130,10cyclictest0-21swapper/420:35:006
3110799132,7cyclictest0-21swapper/320:50:005
3110799132,7cyclictest0-21swapper/300:05:015
3110799132,7cyclictest0-21swapper/300:05:005
3110199132,7cyclictest0-21swapper/220:40:004
3108799132,7cyclictest0-21swapper/022:35:000
3108799132,7cyclictest0-21swapper/020:30:000
3108799132,7cyclictest0-21swapper/000:25:010
275991130,1ptp4l21-21ksoftirqd/122:54:401
275991130,1ptp4l21-21ksoftirqd/100:10:161
275991130,0ptp4l0-21swapper/120:55:211
275991130,0ptp4l0-21swapper/119:52:031
275991130,0ptp4l0-21swapper/100:35:141
3115099122,6cyclictest0-21swapper/1100:10:013
3115099122,6cyclictest0-21swapper/1100:10:003
3115099122,5cyclictest0-21swapper/1122:35:003
3114199122,6cyclictest0-21swapper/920:20:0011
3114199121,7cyclictest0-21swapper/920:00:0111
3114199120,4cyclictest13780-21kernelversion19:25:1811
3113599124,2cyclictest17627-21latency_hist20:15:0010
3113599123,3cyclictest25400-21perl21:05:1610
3113599121,7cyclictest27361-21kworker/u24:021:30:0010
3112599123,7cyclictest0-21swapper/623:50:008
3112599123,7cyclictest0-21swapper/623:50:008
3112599123,6cyclictest0-21swapper/622:50:008
3112599120,11cyclictest9208-21munin-run22:10:008
3112199122,6cyclictest0-21swapper/522:25:257
3112199122,6cyclictest0-21swapper/521:40:007
3112199120,8cyclictest0-21swapper/521:15:007
3110799122,6cyclictest0-21swapper/321:21:585
275991120,3ptp4l27893-21unixbench-2d22:30:271
275991120,3ptp4l11603-21perl20:05:161
275991120,1ptp4l21-21ksoftirqd/120:23:421
275991120,0ptp4l0-21swapper/123:59:421
275991120,0ptp4l0-21swapper/123:31:511
275991120,0ptp4l0-21swapper/122:05:161
275991120,0ptp4l0-21swapper/121:15:121
275991120,0ptp4l0-21swapper/120:00:161
275991120,0ptp4l0-21swapper/119:36:011
3115099113,5cyclictest0-21swapper/1100:35:003
3114199110,10cyclictest0-21swapper/900:00:1511
3114199110,10cyclictest0-21swapper/900:00:1511
3113599112,7cyclictest0-21swapper/821:05:0010
3112199111,6cyclictest27361-21kworker/u24:021:15:007
3108799112,6cyclictest0-21swapper/000:35:010
3108799112,5cyclictest0-21swapper/021:05:000
275991110,6ptp4l21-21ksoftirqd/122:42:561
275991110,3ptp4l28862-21/usr/sbin/munin22:35:191
275991110,2ptp4l5968-21irqrtprio22:45:171
275991110,2ptp4l32364-21diskstats19:10:141
275991110,2ptp4l30053-21cpuspeed_turbos20:30:121
275991110,2ptp4l29243-21memory23:15:191
275991110,2ptp4l28878-21sed19:45:181
275991110,2ptp4l20908-21tr21:00:141
275991110,2ptp4l16193-21ls22:15:231
275991110,2ptp4l14561-21latency_hist21:35:001
275991110,2ptp4l10525-21df_inode22:10:131
275991110,2ptp4l0-21swapper/120:10:181
275991110,1ptp4l0-21swapper/119:55:131
275991110,0ptp4l0-21swapper/123:47:171
275991110,0ptp4l0-21swapper/123:47:171
275991110,0ptp4l0-21swapper/123:05:201
275991110,0ptp4l0-21swapper/123:01:581
275991110,0ptp4l0-21swapper/120:28:171
275991110,0ptp4l0-21swapper/100:22:341
275991110,0ptp4l0-21swapper/100:00:161
275991110,0ptp4l0-21swapper/100:00:161
94980100,6rtkit-daemon0-21swapper/919:05:1711
94980100,4rtkit-daemon0-21swapper/320:05:265
3115099101,5cyclictest0-21swapper/1123:10:003
3114199101,6cyclictest19885-21kworker/u24:223:15:0011
3114199100,1cyclictest0-21swapper/900:30:2311
3113599100,2cyclictest0-21swapper/822:45:1910
3113099100,1cyclictest0-21swapper/721:50:239
3112199108,1cyclictest0-21swapper/523:45:007
3111599108,1cyclictest22979-21cut22:25:206
3111599100,9cyclictest22329-21cpuspeed_turbos20:20:136
3110799102,5cyclictest0-21swapper/323:05:005
3110199102,5cyclictest0-21swapper/223:40:004
3110199102,5cyclictest0-21swapper/223:40:004
3110199102,5cyclictest0-21swapper/220:25:264
3110199100,9cyclictest25585-21/usr/sbin/munin21:05:174
3110199100,9cyclictest15599-21missed_timers22:15:184
3109399100,9cyclictest3863-21if_p34p119:15:021
3108799100,9cyclictest0-21swapper/023:55:220
3108799100,2cyclictest23334-21cstates21:45:130
275991100,2ptp4l3013-21cpuspeed_turbos23:25:131
275991100,2ptp4l24311-21df_inode19:40:131
275991100,2ptp4l15386-21cpuspeed_turbos21:35:141
275991100,1ptp4l31075-21cyclictest21:40:111
275991100,1ptp4l3040-21fw_conntrack22:00:161
275991100,1ptp4l27804-21hddtemp_smartct21:50:161
275991100,1ptp4l23891-21fw_forwarded_lo21:45:151
275991100,1ptp4l2296-21fw_conntrack00:05:201
275991100,1ptp4l2296-21fw_conntrack00:05:191
275991100,1ptp4l0-21swapper/121:25:201
275991100,0ptp4l0-21swapper/121:21:581
275991100,0ptp4l0-21swapper/121:10:271
311509992,4cyclictest0-21swapper/1120:45:253
311419998,1cyclictest0-21swapper/919:50:0111
311419997,1cyclictest0-21swapper/920:20:1311
311419990,1cyclictest0-21swapper/920:35:2211
311359994,4cyclictest30894-21sort00:00:1610
311359994,4cyclictest30894-21sort00:00:1610
311359992,6cyclictest0-21swapper/823:35:0010
311359992,6cyclictest0-21swapper/823:35:0010
311359992,6cyclictest0-21swapper/819:25:2410
311359990,8cyclictest0-21swapper/822:55:0010
311259998,1cyclictest0-21swapper/621:21:588
311259990,9cyclictest0-21swapper/621:45:218
311259990,9cyclictest0-21swapper/620:00:158
311259990,9cyclictest0-21swapper/600:05:298
311259990,9cyclictest0-21swapper/600:05:298
311259990,1cyclictest0-21swapper/622:15:178
311259990,1cyclictest0-21swapper/600:10:168
311219998,1cyclictest0-21swapper/519:10:157
311219997,1cyclictest0-21swapper/522:45:157
311159990,9cyclictest0-21swapper/400:05:176
311159990,9cyclictest0-21swapper/400:05:176
311159990,1cyclictest0-21swapper/422:20:196
311079997,1cyclictest0-21swapper/320:00:285
311079990,1cyclictest0-21swapper/322:35:185
311019990,8cyclictest30441-21fschecks_count00:00:144
311019990,8cyclictest30441-21fschecks_count00:00:144
311019990,8cyclictest30077-21timerwakeupswit19:45:244
311019990,8cyclictest27-21ksoftirqd/222:45:124
311019990,8cyclictest22763-21aten2_r3power_c21:45:114
311019990,8cyclictest22437-21sed19:35:244
311019990,8cyclictest18367-21cat20:15:174
311019990,8cyclictest11556-21cpuspeed_turbos21:30:134
311019990,8cyclictest0-21swapper/223:30:254
311019990,8cyclictest0-21swapper/222:30:274
311019990,8cyclictest0-21swapper/222:25:164
311019990,8cyclictest0-21swapper/200:30:244
310879992,4cyclictest0-21swapper/020:40:000
310879990,5cyclictest0-21swapper/022:10:000
27599190,1ptp4l21-21ksoftirqd/123:50:131
27599190,1ptp4l21-21ksoftirqd/123:50:131
27599190,1ptp4l21-21ksoftirqd/123:10:001
27599190,1ptp4l21-21ksoftirqd/119:25:141
27599190,0ptp4l1536-21grep20:35:121
27599190,0ptp4l0-21swapper/121:55:141
27599190,0ptp4l0-21swapper/121:05:131
27599190,0ptp4l0-21swapper/100:30:151
27599190,0ptp4l0-21swapper/100:25:151
27599190,0ptp4l0-21swapper/100:15:131
4414280,0sleep60-21swapper/623:25:188
311509986,1cyclictest0-21swapper/1100:13:353
311509985,2cyclictest30630-21perl20:30:143
311509982,3cyclictest0-21swapper/1123:50:003
311509982,3cyclictest0-21swapper/1123:50:003
311509980,7cyclictest27574-21diskmemload22:00:523
311509980,1cyclictest0-21swapper/1123:15:253
311459987,1cyclictest0-21swapper/1023:45:132
311459987,1cyclictest0-21swapper/1023:45:132
311459987,1cyclictest0-21swapper/1023:35:002
311459987,1cyclictest0-21swapper/1023:35:002
311459986,1cyclictest0-21swapper/1019:25:012
311459985,2cyclictest0-21swapper/1020:50:282
311459980,8cyclictest0-21swapper/1022:25:222
311459980,8cyclictest0-21swapper/1022:00:142
311459980,8cyclictest0-21swapper/1021:55:212
311459980,8cyclictest0-21swapper/1020:08:402
311459980,1cyclictest0-21swapper/1023:40:262
311459980,1cyclictest0-21swapper/1021:20:192
311419987,1cyclictest0-21swapper/923:40:1311
311419987,1cyclictest0-21swapper/923:15:1811
311419987,1cyclictest0-21swapper/922:50:2111
311419987,1cyclictest0-21swapper/922:25:2211
311419987,1cyclictest0-21swapper/921:45:1911
311419987,1cyclictest0-21swapper/921:20:1811
311419987,1cyclictest0-21swapper/921:00:2311
311419987,1cyclictest0-21swapper/920:50:2711
311419987,1cyclictest0-21swapper/919:35:1611
311419980,8cyclictest0-21swapper/920:00:2611
311359987,1cyclictest0-21swapper/822:30:1710
311359983,4cyclictest0-21swapper/822:00:2510
311359980,1cyclictest0-21swapper/823:05:1910
311359980,1cyclictest0-21swapper/822:05:1610
311359980,1cyclictest0-21swapper/820:40:1810
311309987,1cyclictest0-21swapper/721:30:149
311309987,1cyclictest0-21swapper/720:40:279
311309987,1cyclictest0-21swapper/720:10:259
311309980,8cyclictest0-21swapper/723:35:159
311309980,8cyclictest0-21swapper/723:35:159
311309980,7cyclictest0-21swapper/723:55:229
311309980,1cyclictest0-21swapper/723:55:009
311309980,1cyclictest0-21swapper/723:55:009
311309980,1cyclictest0-21swapper/723:23:029
311309980,1cyclictest0-21swapper/723:09:599
311309980,1cyclictest0-21swapper/722:15:299
311309980,1cyclictest0-21swapper/721:39:409
311309980,1cyclictest0-21swapper/721:18:309
311309980,1cyclictest0-21swapper/720:51:429
311309980,1cyclictest0-21swapper/720:25:219
311309980,1cyclictest0-21swapper/720:15:179
311309980,1cyclictest0-21swapper/719:30:169
311309980,1cyclictest0-21swapper/719:25:129
311309980,1cyclictest0-21swapper/700:32:379
311309980,1cyclictest0-21swapper/700:15:239
311259987,1cyclictest0-21swapper/623:35:278
311259987,1cyclictest0-21swapper/623:35:278
311259987,1cyclictest0-21swapper/621:55:258
311259987,1cyclictest0-21swapper/621:15:178
311259987,1cyclictest0-21swapper/620:30:238
311259987,1cyclictest0-21swapper/620:25:228
311259987,1cyclictest0-21swapper/620:15:258
311259987,1cyclictest0-21swapper/619:50:198
311259987,1cyclictest0-21swapper/619:15:198
311259987,1cyclictest0-21swapper/600:15:188
311259984,3cyclictest27038-21timerwakeupswit21:05:258
311259980,1cyclictest0-21swapper/619:10:258
311219987,1cyclictest0-21swapper/522:40:177
311219987,1cyclictest0-21swapper/521:25:277
311219987,1cyclictest0-21swapper/520:20:137
311219987,1cyclictest0-21swapper/500:10:197
311219985,2cyclictest4617-21turbostat20:40:007
311219980,7cyclictest27574-21diskmemload21:31:457
311219980,1cyclictest0-21swapper/522:30:207
311219980,1cyclictest0-21swapper/500:30:127
311159987,1cyclictest0-21swapper/420:15:196
311159982,5cyclictest21442-21idleruntime-cro23:50:006
311159982,5cyclictest21442-21idleruntime-cro23:50:006
311159980,8cyclictest0-21swapper/422:15:116
311159980,8cyclictest0-21swapper/400:30:206
311159980,8cyclictest0-21swapper/400:10:256
311159980,7cyclictest25955-21cpuspeed23:55:156
311159980,7cyclictest12245-21cat20:49:556
311159980,1cyclictest0-21swapper/423:25:226
311159980,1cyclictest0-21swapper/419:10:246
311079987,1cyclictest0-21swapper/323:50:005
311079987,1cyclictest0-21swapper/323:50:005
311079985,2cyclictest16655-21df19:30:135
311079982,5cyclictest0-21swapper/323:00:005
311079980,1cyclictest0-21swapper/321:40:145
311079980,1cyclictest0-21swapper/320:35:245
311079980,1cyclictest0-21swapper/300:05:195
311079980,1cyclictest0-21swapper/300:05:195
311019987,1cyclictest0-21swapper/219:15:194
311019980,8cyclictest0-21swapper/223:20:264
311019980,8cyclictest0-21swapper/222:05:144
311019980,8cyclictest0-21swapper/221:15:164
311019980,8cyclictest0-21swapper/220:50:174
311019980,8cyclictest0-21swapper/220:20:244
311019980,7cyclictest8199-21/usr/sbin/munin19:20:144
311019980,7cyclictest5043-21ps19:55:204
311019980,7cyclictest4272-21ps22:00:224
311019980,7cyclictest24995-21expr00:35:144
311019980,7cyclictest0-21swapper/223:05:154
311019980,7cyclictest0-21swapper/220:35:224
311019980,2cyclictest26474-21vmstat19:40:244
311019980,1cyclictest15847-21sed23:40:184
311019980,1cyclictest0-21swapper/223:55:164
311019980,1cyclictest0-21swapper/223:45:144
311019980,1cyclictest0-21swapper/223:45:144
311019980,1cyclictest0-21swapper/223:10:174
311019980,1cyclictest0-21swapper/222:55:264
311019980,1cyclictest0-21swapper/221:20:204
311019980,1cyclictest0-21swapper/220:45:144
311019980,1cyclictest0-21swapper/200:15:234
311019980,1cyclictest0-21swapper/200:10:144
310879987,1cyclictest0-21swapper/020:53:400
310879987,1cyclictest0-21swapper/020:20:240
310879986,1cyclictest0-21swapper/019:25:010
310879980,8cyclictest0-21swapper/020:55:190
310879980,8cyclictest0-21swapper/000:30:160
310879980,7cyclictest0-21swapper/020:10:160
310879980,1cyclictest0-21swapper/019:10:130
27633281,5sleep30-21swapper/319:05:115
311509976,1cyclictest27574-21diskmemload21:16:303
311509976,1cyclictest0-21swapper/1123:50:083
311509976,1cyclictest0-21swapper/1123:50:083
311509976,1cyclictest0-21swapper/1123:25:093
311509976,1cyclictest0-21swapper/1122:45:043
311509976,1cyclictest0-21swapper/1122:40:123
311509976,1cyclictest0-21swapper/1122:35:223
311509976,1cyclictest0-21swapper/1122:25:533
311509976,1cyclictest0-21swapper/1122:16:083
311509976,1cyclictest0-21swapper/1122:09:153
311509976,1cyclictest0-21swapper/1121:35:303
311509976,1cyclictest0-21swapper/1121:25:173
311509976,1cyclictest0-21swapper/1121:20:093
311509976,1cyclictest0-21swapper/1121:10:133
311509976,1cyclictest0-21swapper/1120:55:423
311509976,1cyclictest0-21swapper/1120:40:483
311509976,1cyclictest0-21swapper/1120:16:353
311509976,1cyclictest0-21swapper/1119:51:043
311509976,1cyclictest0-21swapper/1119:45:223
311509976,1cyclictest0-21swapper/1119:41:073
311509976,1cyclictest0-21swapper/1119:37:083
311509976,1cyclictest0-21swapper/1119:15:283
311509976,1cyclictest0-21swapper/1100:36:033
311509976,1cyclictest0-21swapper/1100:25:333
311509976,1cyclictest0-21swapper/1100:00:083
311509976,1cyclictest0-21swapper/1100:00:083
311509970,7cyclictest0-21swapper/1123:55:043
311509970,7cyclictest0-21swapper/1123:40:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional