You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 14:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Fri Nov 21, 2025 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911000,91ptp4l0-21swapper/107:07:141
2809926756,9sleep50-21swapper/507:06:387
2832026655,9sleep00-21swapper/007:09:450
2817826553,10sleep40-21swapper/407:07:496
2805126554,9sleep110-21swapper/1107:05:563
2833026456,6sleep100-21swapper/1007:09:552
2807326451,10sleep70-21swapper/707:06:169
215342640,0sleep50-21swapper/508:20:217
2803326351,10sleep60-21swapper/607:05:408
41642620,0sleep00-21swapper/008:40:180
69902560,0sleep70-21swapper/712:15:139
218892190,0sleep20-21swapper/211:10:134
2873699181,3cyclictest0-21swapper/1009:10:152
2872299174,8cyclictest0-21swapper/708:50:019
2871799172,9cyclictest0-21swapper/610:55:008
2869599173,8cyclictest0-21swapper/112:40:001
2869599173,8cyclictest0-21swapper/112:40:001
2869599170,3cyclictest21-21ksoftirqd/108:50:171
275991170,1ptp4l0-21swapper/108:05:111
275991170,0ptp4l0-21swapper/109:52:291
275991170,0ptp4l0-21swapper/107:55:121
301712160,0sleep40-21swapper/407:50:296
2873299163,7cyclictest0-21swapper/912:10:0011
2873299162,8cyclictest0-21swapper/907:25:2111
28717991614,1cyclictest0-21swapper/611:34:558
2871799160,4cyclictest1828-21awk08:40:018
2871799160,11cyclictest0-21swapper/609:10:008
2868899162,8cyclictest0-21swapper/008:50:010
275991160,1ptp4l21-21ksoftirqd/110:40:141
275991160,0ptp4l0-21swapper/109:15:171
275991160,0ptp4l0-21swapper/109:10:201
275991160,0ptp4l0-21swapper/107:35:221
2874199151,8cyclictest0-21swapper/1107:40:013
2873299150,3cyclictest0-21swapper/911:50:1611
28726991512,2cyclictest0-21swapper/807:25:1510
2871799153,7cyclictest0-21swapper/610:50:008
28717991512,2cyclictest0-21swapper/610:10:168
2870499152,8cyclictest0-21swapper/308:20:015
2869599154,6cyclictest0-21swapper/110:50:001
2869599153,7cyclictest0-21swapper/112:25:001
2869599153,7cyclictest0-21swapper/107:30:011
2869599153,6cyclictest0-21swapper/108:15:011
2868899153,8cyclictest0-21swapper/010:15:000
2868899153,7cyclictest0-21swapper/011:25:000
2868899153,7cyclictest0-21swapper/007:48:490
2868899153,6cyclictest0-21swapper/009:35:010
275991150,1ptp4l21-21ksoftirqd/109:25:171
2874199143,7cyclictest0-21swapper/1112:35:003
2874199143,7cyclictest0-21swapper/1112:35:003
2874199142,8cyclictest0-21swapper/1112:20:003
2874199142,8cyclictest0-21swapper/1110:25:003
2873699141,6cyclictest0-21swapper/1010:05:002
2873299142,7cyclictest0-21swapper/911:05:2511
2873299140,9cyclictest0-21swapper/910:00:0011
2872699143,7cyclictest0-21swapper/811:05:0010
2872699142,7cyclictest0-21swapper/807:40:0110
2872299142,8cyclictest0-21swapper/708:00:019
2872299142,7cyclictest0-21swapper/711:20:009
2872299142,7cyclictest0-21swapper/710:15:009
2872299141,8cyclictest0-21swapper/710:05:009
2871799142,6cyclictest0-21swapper/607:20:018
2871399142,7cyclictest0-21swapper/512:20:007
2871399142,7cyclictest0-21swapper/509:50:007
2871399140,14cyclictest0-21swapper/511:30:237
28704991411,3cyclictest0-21swapper/307:25:145
2869599143,7cyclictest0-21swapper/108:50:021
2869599142,8cyclictest0-21swapper/108:30:241
2869599142,6cyclictest0-21swapper/110:10:001
2868899142,8cyclictest0-21swapper/011:15:000
2868899142,7cyclictest0-21swapper/008:35:240
275991140,0ptp4l0-21swapper/110:22:201
275991140,0ptp4l0-21swapper/108:39:211
275991140,0ptp4l0-21swapper/107:50:351
2873299132,7cyclictest0-21swapper/910:20:0011
2873299132,6cyclictest0-21swapper/909:30:3711
2873299130,12cyclictest0-21swapper/909:39:5511
2872699132,7cyclictest0-21swapper/811:25:2410
2872699132,6cyclictest0-21swapper/809:35:0010
2872299132,7cyclictest0-21swapper/710:35:239
2871799132,7cyclictest0-21swapper/609:15:008
2871399133,6cyclictest0-21swapper/511:05:007
2871399130,13cyclictest0-21swapper/508:15:217
2871399130,13cyclictest0-21swapper/508:13:187
2869999132,8cyclictest0-21swapper/207:10:014
2869999130,7cyclictest0-21swapper/209:30:004
2869599132,7cyclictest0-21swapper/112:00:001
275991130,1ptp4l21-21ksoftirqd/111:50:171
275991130,0ptp4l0-21swapper/111:29:391
275991130,0ptp4l0-21swapper/111:03:201
275991130,0ptp4l0-21swapper/109:34:031
275991130,0ptp4l0-21swapper/107:40:271
318782120,0sleep50-21swapper/507:10:257
303512120,0sleep80-21swapper/811:20:2010
2874199122,6cyclictest0-21swapper/1108:45:013
2874199120,11cyclictest0-21swapper/1111:09:553
2873299123,8cyclictest0-21swapper/911:30:0011
2873299122,6cyclictest0-21swapper/907:35:2511
2873299121,6cyclictest0-21swapper/910:25:2311
2872299122,6cyclictest0-21swapper/709:50:009
2872299120,1cyclictest0-21swapper/708:20:169
2871799122,6cyclictest0-21swapper/607:15:018
2871799121,7cyclictest9560-21kworker/u24:111:10:008
2871799121,7cyclictest0-21swapper/608:25:018
2871399122,7cyclictest0-21swapper/511:00:007
2871399122,6cyclictest0-21swapper/512:40:017
2871399122,6cyclictest0-21swapper/512:40:007
2871399122,6cyclictest0-21swapper/508:00:017
2871399121,8cyclictest9560-21kworker/u24:112:00:007
2871399120,12cyclictest0-21swapper/512:01:007
2871399120,12cyclictest0-21swapper/510:31:457
2871399120,11cyclictest16018-21cut11:45:147
2871399120,11cyclictest0-21swapper/507:45:127
2870499121,6cyclictest0-21swapper/307:45:015
2868899122,7cyclictest9560-21kworker/u24:112:20:000
282382121,8sleep80-21swapper/807:08:4110
275991120,4ptp4l21-21ksoftirqd/109:55:121
275991120,3ptp4l0-21swapper/110:15:201
275991120,2ptp4l32419-21df_inode10:00:121
275991120,0ptp4l0-21swapper/112:03:311
275991120,0ptp4l0-21swapper/111:37:051
275991120,0ptp4l0-21swapper/110:30:151
275991120,0ptp4l0-21swapper/108:55:251
275991120,0ptp4l0-21swapper/108:40:151
275991120,0ptp4l0-21swapper/108:25:141
275991120,0ptp4l0-21swapper/108:15:191
2874199112,5cyclictest0-21swapper/1109:20:003
2874199110,10cyclictest0-21swapper/1108:35:013
2873699112,6cyclictest0-21swapper/1012:10:002
2873699112,6cyclictest0-21swapper/1008:45:012
2873699110,1cyclictest0-21swapper/1009:25:232
2872299119,1cyclictest0-21swapper/707:29:559
2872299112,6cyclictest0-21swapper/707:40:019
2871799110,1cyclictest0-21swapper/612:30:008
2871799110,1cyclictest0-21swapper/607:25:108
2871399114,6cyclictest11272-21tr10:15:127
2871399112,6cyclictest0-21swapper/508:30:017
2871399112,5cyclictest0-21swapper/510:49:277
2871399111,6cyclictest0-21swapper/511:55:007
2871399110,7cyclictest220331cpuspeed_turbos09:05:177
2871399110,5cyclictest0-21swapper/507:52:277
2871399110,11cyclictest0-21swapper/512:30:247
2871399110,11cyclictest0-21swapper/512:30:247
2871399110,11cyclictest0-21swapper/509:55:507
2871399110,11cyclictest0-21swapper/509:50:037
2871399110,11cyclictest0-21swapper/508:50:127
2870499110,6cyclictest0-21swapper/307:15:015
2870499110,10cyclictest0-21swapper/311:30:005
2869999112,6cyclictest0-21swapper/212:05:004
2869999112,6cyclictest0-21swapper/209:20:004
2869999112,6cyclictest0-21swapper/208:35:014
2869999112,6cyclictest0-21swapper/208:15:254
2869999111,7cyclictest0-21swapper/210:35:004
2869999111,7cyclictest0-21swapper/209:00:014
2868899111,6cyclictest9560-21kworker/u24:110:50:000
275991110,4ptp4l21-21ksoftirqd/110:55:231
275991110,2ptp4l28248-21latency_hist11:20:001
275991110,2ptp4l18531-21expr09:00:121
275991110,2ptp4l1559-21ptp4l07:15:021
275991110,1ptp4l0-21swapper/111:40:191
275991110,1ptp4l0-21swapper/111:05:141
275991110,0ptp4l0-21swapper/109:39:201
275991110,0ptp4l0-21swapper/108:00:131
262952111,7sleep90-21swapper/907:05:1811
2874199102,6cyclictest0-21swapper/1110:05:003
2874199100,9cyclictest0-21swapper/1109:10:123
2874199100,9cyclictest0-21swapper/1107:10:253
2873699100,9cyclictest0-21swapper/1011:30:122
2873299101,8cyclictest9560-21kworker/u24:108:15:0111
2873299100,2cyclictest0-21swapper/912:20:1811
2872699106,3cyclictest12486-21/usr/sbin/munin09:35:2410
2872299107,2cyclictest5231-21memory10:05:209
2872299100,9cyclictest5575-21cut07:20:019
2871799102,5cyclictest9560-21kworker/u24:110:20:008
2871399104,5cyclictest15500-21processes07:30:207
2871399103,6cyclictest6993-21cat08:45:147
2871399101,8cyclictest1490-21tr11:25:187
2871399100,9cyclictest4672-21grep07:15:247
2871399100,9cyclictest19806-21expr10:25:157
2871399100,6cyclictest286722cpu11:20:147
2871399100,6cyclictest148751fschecks_count12:25:147
2871399100,10cyclictest0-21swapper/512:21:587
2871399100,10cyclictest0-21swapper/509:35:197
2871399100,10cyclictest0-21swapper/508:00:157
2871399100,10cyclictest0-21swapper/507:40:157
2871399100,10cyclictest0-21swapper/507:20:127
2870899103,5cyclictest0-21swapper/412:00:256
2870899102,5cyclictest0-21swapper/412:20:006
2870899100,9cyclictest3531-21fw_conntrack12:10:156
2870499102,5cyclictest0-21swapper/307:35:015
2870499101,5cyclictest0-21swapper/312:30:005
2870499101,5cyclictest0-21swapper/312:30:005
2870499100,6cyclictest0-21swapper/311:40:005
2869999102,5cyclictest0-21swapper/211:55:004
2869999102,5cyclictest0-21swapper/210:25:004
2869999102,5cyclictest0-21swapper/207:55:014
2869999100,9cyclictest0-21swapper/210:20:144
2868899108,1cyclictest0-21swapper/008:30:010
275991100,2ptp4l23924-21latency_hist09:50:001
275991100,2ptp4l21-21ksoftirqd/112:30:171
275991100,2ptp4l21-21ksoftirqd/112:30:171
275991100,1ptp4l8690-21smart_sda12:15:221
275991100,1ptp4l7945-21/usr/sbin/munin10:10:141
275991100,1ptp4l6870-21hddtemp_smartct07:20:141
275991100,1ptp4l5157-21idleruntime-cro10:50:001
275991100,1ptp4l30221-21smartctl07:10:171
275991100,1ptp4l19089-21timerwakeupswit09:40:261
275991100,1ptp4l14391-21perl07:30:131
275991100,1ptp4l0-21swapper/107:45:161
275991100,0ptp4l0-21swapper/111:20:191
287419996,2cyclictest22241-21if_enp1s0.111:10:153
287419990,8cyclictest1020-21runrttasks11:32:323
287369997,1cyclictest0-21swapper/1010:35:132
287369997,1cyclictest0-21swapper/1007:15:022
287369991,5cyclictest0-21swapper/1009:10:012
287369990,1cyclictest0-21swapper/1011:15:152
287269990,1cyclictest0-21swapper/807:45:1810
287229992,6cyclictest0-21swapper/707:45:019
287229990,8cyclictest13250-21cut11:00:139
287229990,8cyclictest0-21swapper/711:30:239
287229990,1cyclictest0-21swapper/707:10:209
287179990,9cyclictest0-21swapper/612:05:218
287179990,9cyclictest0-21swapper/611:55:168
287179990,8cyclictest0-21swapper/610:55:168
287139994,4cyclictest9048-21missed_timers10:10:187
287139994,4cyclictest32483-21switchtime10:40:267
287139991,7cyclictest3805-21expr10:05:147
287139990,9cyclictest0-21swapper/512:11:307
287139990,9cyclictest0-21swapper/511:40:017
287139990,9cyclictest0-21swapper/511:18:197
287139990,9cyclictest0-21swapper/510:35:117
287139990,9cyclictest0-21swapper/510:20:117
287139990,9cyclictest0-21swapper/510:01:367
287139990,9cyclictest0-21swapper/509:30:137
287139990,9cyclictest0-21swapper/509:25:077
287139990,9cyclictest0-21swapper/509:21:177
287139990,9cyclictest0-21swapper/509:15:057
287139990,9cyclictest0-21swapper/508:31:537
287139990,8cyclictest9469-21irqrtprio08:05:157
287139990,8cyclictest9458-21cut07:25:097
287139990,8cyclictest5835-21sort10:50:147
287139990,8cyclictest317651expr08:35:157
287139990,8cyclictest31654-21sh12:05:157
287139990,8cyclictest28177-21timerandwakeup109:10:227
287139990,8cyclictest2650-21cpuspeed_turbos08:40:137
287139990,8cyclictest23627-21timerwakeupswit11:10:227
287139990,8cyclictest20788-21unixbench_singl09:00:247
287139990,8cyclictest18566-21ps07:35:197
287139990,8cyclictest17501-21cpuspeed_turbos11:05:137
287139990,8cyclictest16753-21cpuspeed_turbos09:40:147
287139990,8cyclictest14331-21cut08:55:137
287139990,8cyclictest0-21swapper/511:35:147
287089998,1cyclictest0-21swapper/407:48:496
287089992,4cyclictest14850irq/20-ata_piix08:55:026
287049996,2cyclictest9059-21perl08:05:135
287049996,2cyclictest30384-21awk08:35:015
286999996,2cyclictest28957-21tr11:20:154
286999991,5cyclictest0-21swapper/210:00:004
286999990,8cyclictest8698-21sshd10:52:204
286999990,8cyclictest29014-21sed10:35:234
286999990,8cyclictest19198-21/usr/sbin/munin07:35:214
286999990,8cyclictest15420-21if_err_p34p112:25:164
286999990,2cyclictest32516-21expr10:00:134
286889997,1cyclictest0-21swapper/012:10:150
286889997,1cyclictest0-21swapper/009:35:240
286889994,4cyclictest32155-21cstates10:00:110
286889992,4cyclictest0-21swapper/011:10:000
286889990,9cyclictest0-21swapper/010:05:160
286889990,9cyclictest0-21swapper/008:05:210
286889990,8cyclictest6947-21open_files09:25:180
286889990,1cyclictest0-21swapper/010:55:170
27599190,1ptp4l21-21ksoftirqd/112:25:121
27599190,1ptp4l21-21ksoftirqd/110:25:121
27599190,0ptp4l0-21swapper/112:10:011
27599190,0ptp4l0-21swapper/112:05:171
27599190,0ptp4l0-21swapper/111:45:141
27599190,0ptp4l0-21swapper/111:30:001
27599190,0ptp4l0-21swapper/111:10:001
27599190,0ptp4l0-21swapper/110:35:121
27599190,0ptp4l0-21swapper/109:20:121
27599190,0ptp4l0-21swapper/109:05:171
27599190,0ptp4l0-21swapper/108:20:011
287419987,1cyclictest0-21swapper/1110:50:183
287419987,1cyclictest0-21swapper/1110:25:233
287419987,1cyclictest0-21swapper/1108:15:253
287419987,1cyclictest0-21swapper/1108:05:223
287419986,1cyclictest24976-21diskmemload11:56:533
287419986,1cyclictest0-21swapper/1107:40:283
287419980,1cyclictest0-21swapper/1109:30:383
287419980,1cyclictest0-21swapper/1107:45:123
287419980,1cyclictest0-21swapper/1107:30:143
287369987,1cyclictest0-21swapper/1010:15:132
287369980,8cyclictest0-21swapper/1007:20:142
287369980,1cyclictest0-21swapper/1012:24:422
287369980,1cyclictest0-21swapper/1010:45:232
287369980,1cyclictest0-21swapper/1009:39:192
287329986,1cyclictest0-21swapper/909:00:1211
287329982,4cyclictest0-21swapper/910:10:1311
287329980,8cyclictest0-21swapper/908:15:1611
287329980,1cyclictest0-21swapper/911:10:0111
287329980,1cyclictest0-21swapper/909:50:1911
287269986,1cyclictest0-21swapper/810:00:0010
287269981,4cyclictest0-21swapper/807:20:0110
287269980,8cyclictest0-21swapper/811:15:1810
287269980,8cyclictest0-21swapper/810:30:0010
287269980,1cyclictest0-21swapper/811:40:1910
287269980,1cyclictest0-21swapper/808:25:2310
287229987,1cyclictest0-21swapper/711:50:429
287229985,2cyclictest25732-21munin-run10:35:009
287229985,2cyclictest0-21swapper/707:40:299
287229980,7cyclictest0-21swapper/711:26:149
287229980,1cyclictest0-21swapper/711:10:169
287229980,1cyclictest0-21swapper/709:05:289
287179987,1cyclictest0-21swapper/612:14:208
287179987,1cyclictest0-21swapper/611:20:208
287179987,1cyclictest0-21swapper/610:25:198
287179987,1cyclictest0-21swapper/607:45:138
287179987,1cyclictest0-21swapper/607:30:208
287179985,2cyclictest23230-21dnf10:30:168
287179982,4cyclictest0-21swapper/607:20:128
287179980,8cyclictest0-21swapper/609:00:208
287179980,8cyclictest0-21swapper/608:30:138
287179980,1cyclictest0-21swapper/612:20:158
287179980,1cyclictest0-21swapper/611:00:178
287179980,1cyclictest0-21swapper/609:54:268
287179980,1cyclictest0-21swapper/608:55:198
287089987,1cyclictest0-21swapper/412:30:286
287089987,1cyclictest0-21swapper/412:30:286
287089987,1cyclictest0-21swapper/409:40:226
287089987,1cyclictest0-21swapper/407:20:226
287089985,2cyclictest11811-21taskset09:33:036
287089980,8cyclictest0-21swapper/411:00:276
287089980,8cyclictest0-21swapper/410:15:186
287089980,7cyclictest0-21swapper/411:36:246
287089980,7cyclictest0-21swapper/411:30:206
287089980,1cyclictest0-21swapper/408:15:276
287049987,1cyclictest0-21swapper/312:10:195
287049987,1cyclictest0-21swapper/307:35:275
287049981,6cyclictest0-21swapper/309:35:005
287049981,4cyclictest0-21swapper/310:10:005
287049980,8cyclictest0-21swapper/308:35:165
287049980,7cyclictest0-21swapper/309:05:005
287049980,1cyclictest0-21swapper/312:25:165
287049980,1cyclictest0-21swapper/312:05:205
287049980,1cyclictest0-21swapper/310:45:185
287049980,1cyclictest0-21swapper/310:05:265
287049980,1cyclictest0-21swapper/308:15:165
286999987,1cyclictest0-21swapper/207:20:224
286999982,5cyclictest112750irq/18-p34p109:45:134
286999980,7cyclictest14443-21sed11:40:244
286999980,3cyclictest31471-21/usr/sbin/munin07:10:254
286999980,2cyclictest32663-21sed10:40:264
286999980,1cyclictest27665-21unixbench_singl11:15:244
286999980,1cyclictest1341-21cat07:15:034
286999980,1cyclictest0-21swapper/211:35:004
286999980,1cyclictest0-21swapper/211:10:014
286999980,1cyclictest0-21swapper/208:50:194
286889987,1cyclictest0-21swapper/010:40:190
286889985,2cyclictest30539-21latency_hist08:35:010
286889980,1cyclictest0-21swapper/011:15:200
28123281,5sleep30-21swapper/307:07:005
18329280,1sleep60-21swapper/611:05:188
12036280,0sleep110-21swapper/1108:50:193
287419976,1cyclictest0-21swapper/1112:35:253
287419976,1cyclictest0-21swapper/1112:35:253
287419976,1cyclictest0-21swapper/1112:07:333
287419976,1cyclictest0-21swapper/1111:50:363
287419976,1cyclictest0-21swapper/1111:40:413
287419976,1cyclictest0-21swapper/1111:35:263
287419976,1cyclictest0-21swapper/1110:40:223
287419976,1cyclictest0-21swapper/1109:40:273
287419976,1cyclictest0-21swapper/1109:35:063
287419976,1cyclictest0-21swapper/1109:25:243
287419976,1cyclictest0-21swapper/1109:06:363
287419976,1cyclictest0-21swapper/1108:57:213
287419976,1cyclictest0-21swapper/1108:12:413
287419976,1cyclictest0-21swapper/1108:01:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional