You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 00:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Thu Nov 20, 2025 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911040,94ptp4l0-21swapper/107:07:141
94980880,82rtkit-daemon0-21swapper/1007:05:052
605826654,10sleep00-21swapper/007:07:340
617126554,9sleep50-21swapper/507:09:167
611926452,10sleep70-21swapper/707:08:309
616426351,10sleep110-21swapper/1107:09:103
604926251,9sleep40-21swapper/407:07:266
603826150,9sleep60-21swapper/607:07:178
282022540,0sleep110-21swapper/1111:45:273
657699202,11cyclictest11-21ksoftirqd/007:15:010
229282200,0sleep110-21swapper/1108:10:163
657699184,8cyclictest0-21swapper/010:20:000
661999173,9cyclictest0-21swapper/908:20:0111
661999172,10cyclictest0-21swapper/910:35:0011
43712177,7sleep80-21swapper/807:05:1810
275991170,1ptp4l21-21ksoftirqd/110:00:141
662899162,8cyclictest0-21swapper/1109:40:003
661999164,8cyclictest310-21kworker/u24:109:10:0011
661099162,8cyclictest0-21swapper/707:15:019
658999164,7cyclictest0-21swapper/307:15:015
658999163,8cyclictest0-21swapper/311:35:005
658999162,8cyclictest0-21swapper/309:25:015
657699163,8cyclictest0-21swapper/011:35:000
657699162,8cyclictest0-21swapper/009:20:000
657699162,8cyclictest0-21swapper/008:35:020
42002160,0sleep50-21swapper/507:45:197
27772160,0sleep00-21swapper/007:45:140
275991160,0ptp4l0-21swapper/109:25:191
275991160,0ptp4l0-21swapper/108:45:231
176612160,0sleep90-21swapper/911:35:0011
7272150,0sleep70-21swapper/711:55:009
662899152,8cyclictest0-21swapper/1110:40:003
662899152,7cyclictest0-21swapper/1107:35:013
661999155,6cyclictest310-21kworker/u24:112:40:0011
661999155,6cyclictest310-21kworker/u24:112:40:0011
661499150,9cyclictest0-21swapper/812:20:0010
661099152,7cyclictest0-21swapper/708:40:019
660499153,8cyclictest0-21swapper/611:40:008
660499151,8cyclictest0-21swapper/611:50:008
659999154,6cyclictest0-21swapper/509:00:007
658999153,7cyclictest0-21swapper/312:05:005
658999153,7cyclictest0-21swapper/310:20:005
658999152,8cyclictest0-21swapper/309:55:245
658599152,9cyclictest0-21swapper/207:35:014
657699152,8cyclictest0-21swapper/008:10:010
275991150,1ptp4l21-21ksoftirqd/110:05:151
275991150,1ptp4l13240-21users11:25:291
275991150,0ptp4l0-21swapper/111:00:211
275991150,0ptp4l0-21swapper/108:40:141
275991150,0ptp4l0-21swapper/108:35:161
662899142,8cyclictest0-21swapper/1111:45:003
662899142,8cyclictest0-21swapper/1111:45:003
661999142,7cyclictest0-21swapper/911:55:2211
661499142,8cyclictest0-21swapper/807:10:2510
661499142,7cyclictest0-21swapper/811:00:0010
661099142,7cyclictest0-21swapper/711:15:259
660499143,7cyclictest0-21swapper/612:30:008
659999141,8cyclictest0-21swapper/510:30:007
658599142,9cyclictest0-21swapper/209:25:004
658599140,8cyclictest0-21swapper/210:30:004
658099142,8cyclictest0-21swapper/107:40:011
658099142,7cyclictest0-21swapper/109:55:011
657699142,7cyclictest0-21swapper/012:40:000
657699142,7cyclictest0-21swapper/012:40:000
657699142,7cyclictest0-21swapper/011:25:000
319902140,0sleep80-21swapper/808:20:2610
275991140,1ptp4l21-21ksoftirqd/107:40:201
275991140,0ptp4l0-21swapper/109:35:221
275991140,0ptp4l0-21swapper/108:20:151
275991140,0ptp4l0-21swapper/107:55:141
198622140,1sleep10-21swapper/107:25:131
662899132,7cyclictest0-21swapper/1111:30:283
662899132,7cyclictest0-21swapper/1111:20:263
661999132,6cyclictest0-21swapper/908:45:2211
661999132,6cyclictest0-21swapper/908:00:2311
661499132,7cyclictest0-21swapper/811:40:0010
661499132,6cyclictest0-21swapper/811:05:0010
661499132,10cyclictest310-21kworker/u24:111:45:0010
661499132,10cyclictest310-21kworker/u24:111:45:0010
661099133,8cyclictest0-21swapper/707:50:019
661099132,8cyclictest0-21swapper/709:05:009
661099132,7cyclictest0-21swapper/710:15:009
661099132,7cyclictest0-21swapper/709:50:009
660499132,7cyclictest0-21swapper/609:15:248
660499132,6cyclictest0-21swapper/609:40:018
659999133,6cyclictest0-21swapper/507:25:017
659999132,7cyclictest0-21swapper/509:45:007
659999132,6cyclictest0-21swapper/510:43:407
658999131,8cyclictest0-21swapper/310:00:235
657699132,7cyclictest0-21swapper/012:25:000
275991130,5ptp4l21-21ksoftirqd/111:55:221
275991130,5ptp4l21-21ksoftirqd/108:30:191
275991130,1ptp4l21-21ksoftirqd/110:11:201
275991130,1ptp4l21-21ksoftirqd/108:05:111
275991130,0ptp4l0-21swapper/111:32:411
275991130,0ptp4l0-21swapper/111:15:141
275991130,0ptp4l0-21swapper/108:50:181
113932130,0sleep50-21swapper/512:05:247
662899123,5cyclictest0-21swapper/1111:00:003
662899122,7cyclictest0-21swapper/1110:35:003
662899122,7cyclictest0-21swapper/1109:40:273
662899121,7cyclictest0-21swapper/1108:20:263
662899120,11cyclictest0-21swapper/1112:04:553
661499121,7cyclictest0-21swapper/812:00:0010
661099120,11cyclictest15291-21expr09:25:129
660499122,6cyclictest0-21swapper/612:20:008
659999120,11cyclictest0-21swapper/508:35:017
658999122,6cyclictest0-21swapper/311:50:005
658599121,7cyclictest0-21swapper/211:30:004
658099122,7cyclictest0-21swapper/112:35:001
275991120,3ptp4l15787-21sed08:00:181
275991120,2ptp4l22089-21perl10:15:151
275991120,1ptp4l21-21ksoftirqd/112:10:191
275991120,1ptp4l21-21ksoftirqd/110:40:161
275991120,1ptp4l21-21ksoftirqd/107:24:351
275991120,0ptp4l0-21swapper/111:50:251
275991120,0ptp4l0-21swapper/111:36:091
275991120,0ptp4l0-21swapper/111:10:161
275991120,0ptp4l0-21swapper/110:31:401
275991120,0ptp4l0-21swapper/109:43:051
275991120,0ptp4l0-21swapper/109:10:151
662899112,6cyclictest0-21swapper/1108:30:243
662899110,10cyclictest0-21swapper/1108:44:563
661999110,10cyclictest18765-21diskstats08:05:1311
661499117,3cyclictest21284-21cpuspeed_turbos10:15:1310
661099111,2cyclictest0-21swapper/708:20:099
660499112,6cyclictest0-21swapper/611:10:008
660499112,6cyclictest0-21swapper/608:30:018
659999117,3cyclictest10134-21cat11:25:007
659999111,6cyclictest0-21swapper/512:35:007
658999110,6cyclictest0-21swapper/312:25:015
658599112,6cyclictest0-21swapper/212:40:004
658599112,6cyclictest0-21swapper/212:40:004
658599112,6cyclictest0-21swapper/212:25:004
658599112,6cyclictest0-21swapper/208:45:014
657699112,5cyclictest0-21swapper/011:50:000
277382110,0sleep20-21swapper/209:00:004
275991110,4ptp4l21-21ksoftirqd/109:05:141
275991110,4ptp4l21-21ksoftirqd/107:15:021
275991110,2ptp4l6266-21cut07:50:011
275991110,2ptp4l24460-21fw_forwarded_lo07:30:151
275991110,2ptp4l22837-21cut08:10:151
275991110,2ptp4l14387-21latency_hist09:25:001
275991110,1ptp4l0-21swapper/110:50:181
275991110,0ptp4l0-21swapper/111:40:251
275991110,0ptp4l0-21swapper/111:40:251
275991110,0ptp4l0-21swapper/110:45:171
275991110,0ptp4l0-21swapper/109:45:211
275991110,0ptp4l0-21swapper/107:50:211
662899107,2cyclictest6703-21munin-run09:15:003
662899100,9cyclictest3979-21phc2sys09:50:363
662899100,9cyclictest11512-21ps07:15:023
662399100,1cyclictest0-21swapper/1010:30:142
661999101,8cyclictest31499-21sshd09:05:0011
661999101,6cyclictest0-21swapper/912:15:0011
661499107,2cyclictest7067-21if_err_enp1s009:55:1810
661499104,3cyclictest0-21swapper/810:40:2410
661499102,6cyclictest0-21swapper/810:40:0010
661099100,9cyclictest0-21swapper/710:20:149
660499102,5cyclictest0-21swapper/611:25:008
660499102,5cyclictest0-21swapper/611:20:008
660499102,5cyclictest0-21swapper/608:50:258
659999101,5cyclictest0-21swapper/508:45:017
659999100,9cyclictest22879-21fw_conntrack11:40:147
659999100,9cyclictest22879-21fw_conntrack11:40:147
659999100,1cyclictest0-21swapper/508:15:267
659499109,1cyclictest0-21swapper/407:30:236
658999100,9cyclictest11107-21diskstats07:55:125
658999100,6cyclictest0-21swapper/307:35:015
658599102,5cyclictest0-21swapper/207:30:234
658599100,3cyclictest8859-21sed11:20:254
658599100,3cyclictest21213-21latency_hist11:40:004
658099102,5cyclictest0-21swapper/112:25:211
657699100,10cyclictest0-21swapper/009:10:000
275991100,5ptp4l21-21ksoftirqd/111:07:021
275991100,1ptp4l32740-21tune2fs12:35:151
275991100,1ptp4l32740-21tune2fs12:35:151
275991100,1ptp4l25850-21cpuspeed_turbos08:15:161
275991100,1ptp4l24879-21df08:55:371
275991100,1ptp4l21-21ksoftirqd/109:55:041
275991100,1ptp4l21-21ksoftirqd/108:27:361
275991100,1ptp4l0-21swapper/107:15:011
66289996,2cyclictest4367-21turbostat12:00:003
66239998,1cyclictest0-21swapper/1012:25:182
66239990,9cyclictest0-21swapper/1010:36:252
66239990,1cyclictest0-21swapper/1011:30:162
66239990,1cyclictest0-21swapper/1010:00:142
66239990,1cyclictest0-21swapper/1008:35:282
66239990,1cyclictest0-21swapper/1007:15:022
66199997,1cyclictest0-21swapper/908:55:4711
66199995,3cyclictest10243-21runrttasks09:15:5411
66149998,1cyclictest0-21swapper/810:10:1610
66149990,5cyclictest0-21swapper/808:15:2710
66109998,1cyclictest0-21swapper/711:20:209
66109992,6cyclictest0-21swapper/707:55:019
66109990,8cyclictest20070-21/usr/sbin/munin09:30:179
66109990,1cyclictest0-21swapper/707:20:189
66049998,1cyclictest0-21swapper/612:10:188
66049992,4cyclictest0-21swapper/609:50:408
66049990,9cyclictest0-21swapper/609:20:118
66049990,8cyclictest126592sleep607:15:178
66049990,3cyclictest32691-21munin-run11:55:008
65999996,2cyclictest29173-21head11:05:187
65949990,9cyclictest0-21swapper/409:10:186
65949990,1cyclictest0-21swapper/411:50:266
65949990,1cyclictest0-21swapper/408:25:016
65899992,3cyclictest0-21swapper/309:00:015
65899991,5cyclictest0-21swapper/311:05:005
65859997,1cyclictest0-21swapper/207:25:234
65859991,6cyclictest0-21swapper/210:20:004
65859990,2cyclictest27823-21expr11:05:124
65859990,1cyclictest0-21swapper/212:10:204
65769990,8cyclictest0-21swapper/010:05:000
65769990,8cyclictest0-21swapper/008:55:470
4707291,5sleep90-21swapper/907:05:1911
27609190,7phc2sys0-21swapper/207:05:114
27599190,1ptp4l21-21ksoftirqd/112:05:131
27599190,1ptp4l21-21ksoftirqd/112:00:001
27599190,1ptp4l21-21ksoftirqd/111:20:001
27599190,1ptp4l21-21ksoftirqd/110:55:141
27599190,1ptp4l21-21ksoftirqd/110:25:001
27599190,1ptp4l21-21ksoftirqd/110:20:001
27599190,1ptp4l21-21ksoftirqd/109:15:001
27599190,0ptp4l0-21swapper/112:20:121
27599190,0ptp4l0-21swapper/112:15:141
27599190,0ptp4l0-21swapper/111:45:171
27599190,0ptp4l0-21swapper/110:35:141
27599190,0ptp4l0-21swapper/109:30:131
27599190,0ptp4l0-21swapper/109:00:131
66289987,1cyclictest0-21swapper/1110:10:193
66289987,1cyclictest0-21swapper/1107:25:243
66289980,8cyclictest0-21swapper/1110:05:233
66289980,8cyclictest0-21swapper/1107:10:183
66289980,1cyclictest0-21swapper/1111:15:203
66289980,1cyclictest0-21swapper/1110:25:133
66239987,1cyclictest0-21swapper/1011:25:572
66239987,1cyclictest0-21swapper/1008:55:362
66239987,1cyclictest0-21swapper/1008:25:222
66239987,1cyclictest0-21swapper/1007:10:182
66239984,3cyclictest28111-21cut11:05:132
66239980,8cyclictest0-21swapper/1012:10:152
66239980,8cyclictest0-21swapper/1009:30:122
66239980,8cyclictest0-21swapper/1008:50:132
66239980,1cyclictest0-21swapper/1010:20:222
66239980,1cyclictest0-21swapper/1007:55:202
66199987,1cyclictest0-21swapper/909:40:1411
66199987,1cyclictest0-21swapper/908:35:1611
66199980,8cyclictest403-1kworker/9:1H12:24:0911
66199980,8cyclictest0-21swapper/910:50:1911
66199980,8cyclictest0-21swapper/908:40:2011
66199980,7cyclictest2671-21diskmemload10:55:3811
66199980,7cyclictest13605-21munin-run08:00:0111
66199980,7cyclictest0-21swapper/912:25:1511
66199980,1cyclictest0-21swapper/909:50:4111
66199980,1cyclictest0-21swapper/908:50:1611
66199980,1cyclictest0-21swapper/907:40:2311
66149987,1cyclictest0-21swapper/812:25:1010
66149987,1cyclictest0-21swapper/807:42:3710
66149981,6cyclictest0-21swapper/807:25:1610
66149980,8cyclictest0-21swapper/812:20:1710
66149980,8cyclictest0-21swapper/809:25:1210
66149980,8cyclictest0-21swapper/807:35:1910
66149980,7cyclictest0-21swapper/810:26:1410
66149980,7cyclictest0-21swapper/808:34:4010
66149980,1cyclictest0-21swapper/809:10:0010
66109987,1cyclictest0-21swapper/712:00:199
66109987,1cyclictest0-21swapper/711:55:179
66109987,1cyclictest0-21swapper/707:15:019
66109985,2cyclictest18962-21timerandwakeup08:45:229
66109980,8cyclictest0-21swapper/712:15:189
66109980,8cyclictest0-21swapper/710:00:129
66109980,8cyclictest0-21swapper/709:40:169
66109980,8cyclictest0-21swapper/708:55:389
66109980,1cyclictest0-21swapper/712:30:129
66109980,1cyclictest0-21swapper/709:55:169
66109980,1cyclictest0-21swapper/709:35:139
66109980,1cyclictest0-21swapper/709:05:099
66109980,1cyclictest0-21swapper/707:35:149
66049987,1cyclictest0-21swapper/610:30:168
66049987,1cyclictest0-21swapper/610:20:248
66049987,1cyclictest0-21swapper/610:15:138
66049985,2cyclictest8603-21irqstats07:10:188
66049981,4cyclictest0-21swapper/609:55:008
66049980,8cyclictest0-21swapper/610:55:258
66049980,8cyclictest0-21swapper/610:10:248
66049980,8cyclictest0-21swapper/607:45:168
66049980,5cyclictest0-21swapper/611:25:008
66049980,1cyclictest0-21swapper/612:00:218
66049980,1cyclictest0-21swapper/611:30:298
66049980,1cyclictest0-21swapper/609:30:198
65999980,8cyclictest0-21swapper/509:00:207
65999980,1cyclictest0-21swapper/508:10:227
65999980,1cyclictest0-21swapper/508:00:167
65949987,1cyclictest0-21swapper/411:55:136
65949987,1cyclictest0-21swapper/409:30:146
65949987,1cyclictest0-21swapper/409:15:256
65949987,1cyclictest0-21swapper/408:35:216
65949987,1cyclictest0-21swapper/407:55:026
65949987,1cyclictest0-21swapper/407:10:176
65949980,8cyclictest0-21swapper/411:36:066
65949980,8cyclictest0-21swapper/411:00:176
65949980,8cyclictest0-21swapper/409:45:216
65949980,8cyclictest0-21swapper/408:40:176
65949980,8cyclictest0-21swapper/408:10:126
65949980,8cyclictest0-21swapper/407:49:446
65949980,1cyclictest0-21swapper/411:15:196
65949980,1cyclictest0-21swapper/410:16:146
65949980,1cyclictest0-21swapper/410:05:246
65949980,1cyclictest0-21swapper/408:45:136
65949980,1cyclictest0-21swapper/407:15:156
65899987,1cyclictest0-21swapper/311:25:175
65899987,1cyclictest0-21swapper/308:15:165
65899986,1cyclictest17199-21diskstats12:15:145
65899985,2cyclictest16652-21turbostat10:10:005
65899985,2cyclictest0-21swapper/310:45:125
65899980,8cyclictest0-21swapper/311:40:125
65899980,8cyclictest0-21swapper/311:40:125
65899980,1cyclictest0-21swapper/311:15:245
65899980,1cyclictest0-21swapper/308:05:195
65859987,1cyclictest0-21swapper/210:10:164
65859987,1cyclictest0-21swapper/209:25:144
65859985,2cyclictest28553-21users08:15:274
65859980,8cyclictest0-21swapper/212:05:184
65859980,8cyclictest0-21swapper/211:50:264
65859980,8cyclictest0-21swapper/210:40:144
65859980,7cyclictest30269-21fschecks_count08:20:174
65859980,7cyclictest2671-21diskmemload09:17:034
65859980,7cyclictest19115-21/usr/sbin/munin08:05:144
65859980,7cyclictest0-21swapper/212:15:214
65859980,7cyclictest0-21swapper/208:40:014
65859980,3cyclictest435-21/usr/sbin/munin11:10:174
65859980,2cyclictest28666-21df_inode09:00:134
65859980,2cyclictest1607-21ptp4l-jitter09:05:184
65859980,2cyclictest10317-21/usr/sbin/munin10:00:134
65859980,1cyclictest17499-21latency07:20:184
65859980,1cyclictest0-21swapper/211:45:164
65859980,1cyclictest0-21swapper/211:30:004
65859980,1cyclictest0-21swapper/210:20:234
65859980,1cyclictest0-21swapper/210:09:394
65859980,1cyclictest0-21swapper/209:55:164
65769985,2cyclictest2674-21turbostat09:14:560
65769980,4cyclictest0-21swapper/007:25:010
65769980,1cyclictest0-21swapper/012:15:180
65769980,1cyclictest0-21swapper/010:50:180
65769980,1cyclictest0-21swapper/009:55:240
65769980,1cyclictest0-21swapper/007:40:190
66289976,1cyclictest0-21swapper/1112:16:213
66289976,1cyclictest0-21swapper/1112:06:483
66289976,1cyclictest0-21swapper/1111:26:333
66289976,1cyclictest0-21swapper/1111:09:103
66289976,1cyclictest0-21swapper/1110:51:093
66289976,1cyclictest0-21swapper/1110:46:063
66289976,1cyclictest0-21swapper/1110:40:543
66289976,1cyclictest0-21swapper/1110:15:443
66289976,1cyclictest0-21swapper/1109:25:093
66289976,1cyclictest0-21swapper/1109:17:563
66289976,1cyclictest0-21swapper/1108:55:463
66289976,1cyclictest0-21swapper/1108:35:033
66289976,1cyclictest0-21swapper/1108:15:343
66289976,1cyclictest0-21swapper/1107:50:143
66289976,1cyclictest0-21swapper/1107:35:413
66289970,7cyclictest0-21swapper/1112:35:143
66289970,7cyclictest0-21swapper/1112:35:143
66289970,7cyclictest0-21swapper/1112:30:143
66289970,7cyclictest0-21swapper/1112:20:083
66289970,7cyclictest0-21swapper/1111:50:153
66289970,7cyclictest0-21swapper/1111:35:313
66289970,7cyclictest0-21swapper/1111:10:093
66289970,7cyclictest0-21swapper/1111:01:393
66289970,7cyclictest0-21swapper/1110:20:183
66289970,7cyclictest0-21swapper/1109:55:233
66289970,7cyclictest0-21swapper/1109:20:223
66289970,7cyclictest0-21swapper/1109:05:163
66289970,7cyclictest0-21swapper/1107:57:193
66289970,1cyclictest0-21swapper/1112:26:423
66289970,1cyclictest0-21swapper/1112:11:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional