You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 01:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Tue Dec 02, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
275991950,86ptp4l0-21swapper/107:05:151
2807526957,10sleep50-21swapper/507:07:357
2815026654,10sleep00-21swapper/007:08:430
2808126452,10sleep110-21swapper/1107:07:413
2795926453,9sleep100-21swapper/1007:05:532
182062640,0sleep90-21swapper/911:05:1711
2495926250,10sleep70-21swapper/707:05:139
219222620,0sleep90-21swapper/912:35:1511
2807626150,9sleep60-21swapper/607:07:368
2795326151,8sleep40-21swapper/407:05:466
267012580,0sleep110-21swapper/1111:15:203
326792520,0sleep110-21swapper/1108:35:263
102362470,0sleep70-21swapper/707:25:159
94980240,6rtkit-daemon0-21swapper/807:05:5310
2858399233,14cyclictest0-21swapper/212:35:004
2859299221,8cyclictest0-21swapper/408:15:006
2858399193,9cyclictest0-21swapper/210:50:024
2858899184,8cyclictest0-21swapper/310:20:005
2858399183,10cyclictest0-21swapper/211:15:014
2858399183,10cyclictest0-21swapper/211:10:014
2857399183,7cyclictest0-21swapper/011:50:000
46882170,0sleep60-21swapper/610:05:198
2858899172,9cyclictest0-21swapper/310:35:005
2858399172,7cyclictest0-21swapper/208:25:014
2857399173,8cyclictest0-21swapper/008:05:000
2857399173,8cyclictest0-21swapper/008:05:000
2857399172,7cyclictest0-21swapper/009:35:020
2857399170,3cyclictest0-21swapper/007:45:170
275991170,0ptp4l0-21swapper/110:50:191
2861599164,7cyclictest0-21swapper/908:40:0011
2861599163,8cyclictest0-21swapper/910:10:0111
2860999163,8cyclictest0-21swapper/808:44:2010
2860999161,10cyclictest12966-21kworker/u24:008:00:0110
2860999161,10cyclictest12966-21kworker/u24:008:00:0010
28606991612,3cyclictest0-21swapper/707:40:169
2858899163,8cyclictest0-21swapper/311:40:275
2858899162,9cyclictest0-21swapper/308:05:015
2858899162,9cyclictest0-21swapper/308:05:005
2858899161,9cyclictest0-21swapper/312:00:005
2858399161,9cyclictest0-21swapper/210:35:004
2857899162,9cyclictest0-21swapper/107:30:011
2857399164,7cyclictest0-21swapper/012:35:270
2857399163,7cyclictest0-21swapper/011:44:560
275991160,1ptp4l21-21ksoftirqd/109:15:161
208962160,0sleep60-21swapper/611:10:148
191591160,0getstats0-21swapper/209:05:304
2861599153,7cyclictest0-21swapper/907:40:0011
2860999152,8cyclictest0-21swapper/811:15:0010
2860999152,8cyclictest0-21swapper/807:45:0010
2860999150,2cyclictest0-21swapper/807:25:1710
2860699153,7cyclictest0-21swapper/710:10:009
2860699152,8cyclictest0-21swapper/712:10:009
2860699152,8cyclictest0-21swapper/709:55:009
2860699152,7cyclictest0-21swapper/708:00:249
2860699152,7cyclictest0-21swapper/708:00:239
2859299150,3cyclictest0-21swapper/411:55:176
2858899152,9cyclictest12966-21kworker/u24:012:30:015
2858899152,8cyclictest0-21swapper/309:30:015
2858399153,8cyclictest0-21swapper/210:00:014
2857399152,8cyclictest0-21swapper/007:20:000
2857399152,8cyclictest0-21swapper/007:20:000
275991150,1ptp4l21-21ksoftirqd/110:05:221
275991150,1ptp4l21-21ksoftirqd/108:25:271
275991150,0ptp4l0-21swapper/110:15:011
2861599142,7cyclictest0-21swapper/907:20:0011
2861599142,7cyclictest0-21swapper/907:20:0011
2861599140,13cyclictest21073-21awk09:05:0011
2860999143,7cyclictest0-21swapper/810:10:0110
2860999142,7cyclictest0-21swapper/810:50:0110
2860999142,7cyclictest0-21swapper/810:20:0210
2860699143,7cyclictest0-21swapper/710:50:019
2860199142,8cyclictest5375-21kworker/u24:207:25:008
2860199141,8cyclictest0-21swapper/608:00:008
2860199141,8cyclictest0-21swapper/608:00:008
2858899143,7cyclictest0-21swapper/310:10:015
2858899143,6cyclictest0-21swapper/309:55:015
2858899142,8cyclictest0-21swapper/310:30:015
2858899142,8cyclictest0-21swapper/309:20:245
2858899142,8cyclictest0-21swapper/308:55:015
2858399143,7cyclictest0-21swapper/208:15:004
2857899142,7cyclictest0-21swapper/111:55:001
2857899142,7cyclictest0-21swapper/108:05:271
2857899142,7cyclictest0-21swapper/108:05:261
2857899140,3cyclictest0-21swapper/112:05:191
2857399143,7cyclictest0-21swapper/007:30:310
2857399143,6cyclictest0-21swapper/010:15:010
275991140,1ptp4l21-21ksoftirqd/111:25:111
275991140,0ptp4l0-21swapper/109:00:151
275991140,0ptp4l0-21swapper/108:20:151
191591140,0getstats0-21swapper/207:40:294
191591140,0getstats0-21swapper/207:25:294
29602130,0sleep110-21swapper/1108:40:163
2861899139,3cyclictest0-21swapper/1011:45:162
2861899132,7cyclictest0-21swapper/1009:50:262
2861899132,7cyclictest0-21swapper/1008:15:002
2861599131,8cyclictest0-21swapper/909:50:0011
2860999132,7cyclictest0-21swapper/809:00:0010
2860999132,6cyclictest0-21swapper/811:05:0010
2860699133,6cyclictest0-21swapper/709:10:019
2860199132,6cyclictest0-21swapper/612:15:008
2859299132,6cyclictest0-21swapper/407:50:226
2859299132,6cyclictest0-21swapper/407:50:226
2858899133,6cyclictest0-21swapper/312:05:005
2858899133,6cyclictest0-21swapper/310:05:015
2858899132,7cyclictest0-21swapper/307:10:235
2857899131,8cyclictest12966-21kworker/u24:011:00:021
2857399133,7cyclictest0-21swapper/009:00:010
275991130,1ptp4l21-21ksoftirqd/111:17:121
275991130,1ptp4l21-21ksoftirqd/109:58:021
275991130,1ptp4l21-21ksoftirqd/107:23:341
223592130,0sleep70-21swapper/710:30:119
2861899120,2cyclictest0-21swapper/1012:35:252
2861599120,7cyclictest0-21swapper/909:35:0111
2861599120,7cyclictest0-21swapper/908:55:2711
2861599120,7cyclictest0-21swapper/908:10:2411
2860999122,6cyclictest0-21swapper/812:30:0010
2860999122,6cyclictest0-21swapper/807:20:2410
28609991210,1cyclictest0-21swapper/808:35:0110
2860699120,11cyclictest32305-21/usr/sbin/munin11:25:139
2860699120,11cyclictest0-21swapper/710:05:009
2860199121,7cyclictest0-21swapper/610:45:248
2858899122,6cyclictest0-21swapper/308:40:005
2858899122,5cyclictest0-21swapper/312:15:005
2857899122,7cyclictest0-21swapper/107:40:001
2857899122,6cyclictest0-21swapper/107:55:001
2857899122,6cyclictest0-21swapper/107:55:001
2857399122,6cyclictest0-21swapper/009:00:000
2857399122,6cyclictest0-21swapper/008:30:240
275991120,3ptp4l16030-21grep07:30:321
275991120,3ptp4l11133-21fschecks_time08:50:161
275991120,2ptp4l16479-21munin-run11:04:591
275991120,1ptp4l21-21ksoftirqd/108:44:551
275991120,0ptp4l0-21swapper/112:01:161
275991120,0ptp4l0-21swapper/111:45:171
275991120,0ptp4l0-21swapper/111:33:021
275991120,0ptp4l0-21swapper/110:04:271
275991120,0ptp4l0-21swapper/109:50:451
2861899113,5cyclictest0-21swapper/1009:50:002
2861899112,6cyclictest0-21swapper/1009:55:232
2861899110,2cyclictest0-21swapper/1007:40:172
2861599111,6cyclictest0-21swapper/911:15:0011
2860999117,3cyclictest20507-21grep09:45:1310
2860699118,2cyclictest27948-21turbostat07:50:019
2860699110,10cyclictest23078-21sed09:05:189
2859299110,2cyclictest16143-21latency_hist09:40:016
275991110,2ptp4l9650-21perl09:30:221
275991110,2ptp4l7156-21cpu10:10:141
275991110,2ptp4l24091-21fschecks_time11:55:161
275991110,2ptp4l21670-21fschecks_count07:40:151
275991110,2ptp4l1919-21cpuspeed_turbos07:15:121
275991110,2ptp4l1919-21cpuspeed_turbos07:15:121
275991110,2ptp4l14970-21perl12:25:151
275991110,1ptp4l21-21ksoftirqd/111:20:111
275991110,0ptp4l0-21swapper/110:28:281
2862499108,2cyclictest21826-21turbostat10:30:003
2862499100,9cyclictest0-21swapper/1109:45:133
2862499100,1cyclictest0-21swapper/1111:00:173
2861899108,1cyclictest0-21swapper/1011:40:152
2861899108,1cyclictest0-21swapper/1009:05:162
2861899102,4cyclictest0-21swapper/1008:10:012
2861899102,4cyclictest0-21swapper/1008:10:012
2861599100,1cyclictest0-21swapper/907:15:0011
2860999100,9cyclictest25306-21users10:30:2410
2860999100,5cyclictest0-21swapper/810:04:5510
2860699109,1cyclictest0-21swapper/710:45:149
2860699100,1cyclictest0-21swapper/712:24:549
2858399107,2cyclictest0-21swapper/210:25:214
2858399100,9cyclictest24893-21diskmemload12:25:154
2858399100,9cyclictest0-21swapper/211:55:134
2858399100,9cyclictest0-21swapper/210:05:184
2858399100,3cyclictest1020-21runrttasks12:35:154
2857399100,9cyclictest0-21swapper/010:50:150
275991100,2ptp4l15059-21perl08:55:181
275991100,1ptp4l7067-21expr08:45:191
275991100,1ptp4l3407-21timerandwakeup109:20:231
275991100,1ptp4l32225-21processes10:40:211
275991100,1ptp4l3082-21diskstats12:10:151
275991100,1ptp4l23511-21sed11:10:271
275991100,1ptp4l22650-21fschecks_time09:05:151
275991100,1ptp4l22173-21cstates12:35:161
275991100,1ptp4l2211-21expr10:45:131
275991100,1ptp4l21-21ksoftirqd/111:44:561
275991100,1ptp4l21-21ksoftirqd/109:37:191
275991100,1ptp4l18142-21perl11:05:161
251182101,6sleep90-21swapper/907:05:1611
286249996,2cyclictest24893-21diskmemload10:59:103
286249990,9cyclictest0-21swapper/1111:40:153
286249990,2cyclictest18154-21fw_forwarded_lo07:35:173
286249990,1cyclictest0-21swapper/1110:30:173
286249990,1cyclictest0-21swapper/1107:55:273
286249990,1cyclictest0-21swapper/1107:55:263
286189993,3cyclictest0-21swapper/1011:55:012
286189991,5cyclictest0-21swapper/1008:45:002
286189990,8cyclictest0-21swapper/1012:15:012
286189990,1cyclictest0-21swapper/1010:20:122
286099998,1cyclictest0-21swapper/811:30:2310
286099997,1cyclictest0-21swapper/807:50:2210
286099997,1cyclictest0-21swapper/807:50:2210
286099996,2cyclictest29275-21df11:20:1310
286099990,9cyclictest0-21swapper/809:05:1810
286099990,8cyclictest0-21swapper/809:30:2810
286069998,1cyclictest0-21swapper/711:35:249
286069998,1cyclictest0-21swapper/711:20:019
286069990,9cyclictest0-21swapper/708:45:239
286069990,1cyclictest0-21swapper/708:20:119
286019991,5cyclictest0-21swapper/612:10:008
286019990,8cyclictest0-21swapper/607:20:018
286019990,8cyclictest0-21swapper/607:20:008
285979990,9cyclictest0-21swapper/511:25:177
285979990,8cyclictest0-21swapper/508:40:167
285979990,1cyclictest0-21swapper/510:20:267
285979990,1cyclictest0-21swapper/509:05:247
285929996,2cyclictest27457-21cstates12:00:116
285929990,9cyclictest0-21swapper/410:25:206
285889992,6cyclictest0-21swapper/307:55:005
285889992,6cyclictest0-21swapper/307:55:005
285889990,9cyclictest0-21swapper/310:50:245
285889990,9cyclictest0-21swapper/307:45:005
285889990,1cyclictest0-21swapper/311:10:215
285839990,8cyclictest0-21swapper/212:05:154
285839990,3cyclictest24879-21/usr/sbin/munin11:15:134
285839990,2cyclictest0-21swapper/208:55:164
285739990,1cyclictest0-21swapper/011:14:590
27599190,1ptp4l21-21ksoftirqd/110:20:121
27599190,0ptp4l0-21swapper/112:30:001
27599190,0ptp4l0-21swapper/112:20:001
27599190,0ptp4l0-21swapper/112:15:011
27599190,0ptp4l0-21swapper/111:35:151
27599190,0ptp4l0-21swapper/110:35:161
27599190,0ptp4l0-21swapper/110:30:021
27599190,0ptp4l0-21swapper/109:45:141
27599190,0ptp4l0-21swapper/109:40:141
27599190,0ptp4l0-21swapper/109:25:121
27599190,0ptp4l0-21swapper/109:10:161
27599190,0ptp4l0-21swapper/108:35:211
27599190,0ptp4l0-21swapper/108:30:131
27599190,0ptp4l0-21swapper/108:15:011
27599190,0ptp4l0-21swapper/108:10:011
27599190,0ptp4l0-21swapper/108:00:131
27599190,0ptp4l0-21swapper/108:00:131
27599190,0ptp4l0-21swapper/107:55:201
27599190,0ptp4l0-21swapper/107:55:191
27599190,0ptp4l0-21swapper/107:45:021
27599190,0ptp4l0-21swapper/107:10:021
9498080,0rtkit-daemon0-21swapper/210:38:234
286249987,1cyclictest0-21swapper/1112:12:103
286249987,1cyclictest0-21swapper/1111:55:193
286249987,1cyclictest0-21swapper/1110:20:133
286249987,1cyclictest0-21swapper/1110:10:173
286249980,7cyclictest0-21swapper/1107:22:143
286249980,1cyclictest0-21swapper/1112:38:153
286249980,1cyclictest0-21swapper/1111:39:533
286249980,1cyclictest0-21swapper/1107:45:203
286189987,1cyclictest0-21swapper/1012:15:152
286189987,1cyclictest0-21swapper/1010:44:452
286189980,8cyclictest0-21swapper/1009:35:222
286189980,8cyclictest0-21swapper/1008:15:192
286189980,8cyclictest0-21swapper/1007:15:002
286189980,7cyclictest0-21swapper/1010:14:562
286189980,1cyclictest0-21swapper/1009:30:262
286159987,1cyclictest0-21swapper/908:30:2411
286159981,5cyclictest7018-21taskset11:31:1911
286159981,5cyclictest0-21swapper/908:40:0111
286159980,8cyclictest0-21swapper/909:20:0111
286159980,1cyclictest0-21swapper/909:50:1811
286159980,1cyclictest0-21swapper/908:15:1411
286099980,8cyclictest0-21swapper/812:10:2010
286099980,8cyclictest0-21swapper/811:28:2410
286099980,8cyclictest0-21swapper/810:55:0210
286099980,8cyclictest0-21swapper/810:50:1910
286099980,8cyclictest0-21swapper/809:20:2310
286099980,1cyclictest0-21swapper/812:15:2510
286099980,1cyclictest0-21swapper/807:15:1610
286099980,1cyclictest0-21swapper/807:15:1610
286069982,5cyclictest0-21swapper/710:30:019
286069980,8cyclictest0-21swapper/708:15:149
286069980,8cyclictest0-21swapper/707:15:149
286069980,8cyclictest0-21swapper/707:15:149
286069980,1cyclictest0-21swapper/709:25:149
286069980,1cyclictest0-21swapper/709:15:209
286019983,5cyclictest0-21swapper/612:15:008
286019980,8cyclictest0-21swapper/608:50:218
286019980,8cyclictest0-21swapper/607:30:328
286019980,1cyclictest0-21swapper/609:45:148
286019980,1cyclictest0-21swapper/607:35:158
285979987,1cyclictest0-21swapper/511:05:167
285979987,1cyclictest0-21swapper/508:01:007
285979987,1cyclictest0-21swapper/508:01:007
285979986,1cyclictest0-21swapper/512:30:147
285979980,8cyclictest1009-21snmpd07:48:117
285979980,8cyclictest0-21swapper/512:25:127
285979980,8cyclictest0-21swapper/511:55:207
285979980,8cyclictest0-21swapper/510:10:157
285979980,8cyclictest0-21swapper/508:59:277
285979980,8cyclictest0-21swapper/508:30:237
285979980,1cyclictest0-21swapper/508:05:197
285979980,1cyclictest0-21swapper/508:05:197
285979980,1cyclictest0-21swapper/507:55:227
285979980,1cyclictest0-21swapper/507:55:227
285929987,1cyclictest0-21swapper/408:20:196
285929986,1cyclictest0-21swapper/411:00:146
285929985,2cyclictest9731-21fw_forwarded_lo09:30:226
285929984,3cyclictest25625-21grep09:50:186
285929982,5cyclictest7581-21cat11:35:006
285929981,5cyclictest0-21swapper/408:55:006
285929980,8cyclictest0-21swapper/411:45:176
285929980,8cyclictest0-21swapper/411:05:196
285929980,2cyclictest11343-21expr10:15:166
285889987,1cyclictest0-21swapper/312:15:125
285889987,1cyclictest0-21swapper/311:05:165
285889987,1cyclictest0-21swapper/309:10:165
285889987,1cyclictest0-21swapper/308:20:255
285889987,1cyclictest0-21swapper/308:15:215
285889987,1cyclictest0-21swapper/307:25:125
285889986,1cyclictest24893-21diskmemload09:38:275
285889980,8cyclictest0-21swapper/312:20:155
285889980,8cyclictest0-21swapper/307:35:125
285889980,1cyclictest0-21swapper/310:40:205
285839980,8cyclictest0-21swapper/211:50:174
285839980,8cyclictest0-21swapper/211:40:154
285839980,8cyclictest0-21swapper/211:35:164
285839980,8cyclictest0-21swapper/207:30:274
285839980,7cyclictest6220-21/usr/sbin/munin11:30:204
285839980,7cyclictest19398-21sshd09:43:194
285839980,1cyclictest0-21swapper/209:35:164
285839980,1cyclictest0-21swapper/209:15:194
285839980,1cyclictest0-21swapper/207:35:154
285739987,1cyclictest0-21swapper/009:50:260
285739987,1cyclictest0-21swapper/009:45:360
285739987,1cyclictest0-21swapper/008:45:290
285739980,8cyclictest0-21swapper/011:55:000
285739980,1cyclictest0-21swapper/007:35:150
24953281,5sleep30-21swapper/307:05:095
286249976,1cyclictest0-21swapper/1112:25:523
286249976,1cyclictest0-21swapper/1112:15:173
286249976,1cyclictest0-21swapper/1112:00:073
286249976,1cyclictest0-21swapper/1111:50:093
286249976,1cyclictest0-21swapper/1111:45:493
286249976,1cyclictest0-21swapper/1111:25:113
286249976,1cyclictest0-21swapper/1111:10:053
286249976,1cyclictest0-21swapper/1111:06:283
286249976,1cyclictest0-21swapper/1110:50:173
286249976,1cyclictest0-21swapper/1110:15:193
286249976,1cyclictest0-21swapper/1110:05:143
286249976,1cyclictest0-21swapper/1110:00:183
286249976,1cyclictest0-21swapper/1109:55:133
286249976,1cyclictest0-21swapper/1109:28:053
286249976,1cyclictest0-21swapper/1108:25:153
286249976,1cyclictest0-21swapper/1108:20:403
286249976,1cyclictest0-21swapper/1108:10:053
286249976,1cyclictest0-21swapper/1108:05:253
286249976,1cyclictest0-21swapper/1108:05:243
286249976,1cyclictest0-21swapper/1107:10:243
286249975,1cyclictest0-21swapper/1109:15:163
286249974,2cyclictest18048-21cpuspeed_turbos09:00:133
286249970,7cyclictest0-21swapper/1112:30:553
286249970,7cyclictest0-21swapper/1112:21:183
286249970,7cyclictest0-21swapper/1112:05:113
286249970,7cyclictest0-21swapper/1110:46:563
286249970,7cyclictest0-21swapper/1110:41:283
286249970,7cyclictest0-21swapper/1110:35:073
286249970,7cyclictest0-21swapper/1109:50:233
286249970,7cyclictest0-21swapper/1109:43:163
286249970,7cyclictest0-21swapper/1109:30:433
286249970,7cyclictest0-21swapper/1109:24:113
286249970,7cyclictest0-21swapper/1108:55:013
286249970,7cyclictest0-21swapper/1108:47:343
286249970,7cyclictest0-21swapper/1108:15:143
286249970,7cyclictest0-21swapper/1107:40:163
286249970,7cyclictest0-21swapper/1107:26:523
286249970,1cyclictest0-21swapper/1111:31:383
286249970,1cyclictest0-21swapper/1111:21:063
286249970,1cyclictest0-21swapper/1109:36:023
286249970,1cyclictest0-21swapper/1109:11:393
286249970,1cyclictest0-21swapper/1109:07:063
286249970,1cyclictest0-21swapper/1108:50:383
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional