You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 16:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Sun Nov 30, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911050,95ptp4l0-21swapper/107:08:571
2759911050,95ptp4l0-21swapper/107:08:561
1446226757,6sleep70-21swapper/707:05:169
1446226757,6sleep70-21swapper/707:05:159
1676926658,6sleep60-21swapper/607:07:278
1676926658,6sleep60-21swapper/607:07:278
1670126653,10sleep50-21swapper/507:06:267
1670126653,10sleep50-21swapper/507:06:267
1672026554,9sleep110-21swapper/1107:06:443
1672026554,9sleep110-21swapper/1107:06:443
1669626554,9sleep00-21swapper/007:06:220
1669626554,9sleep00-21swapper/007:06:220
1669326352,9sleep100-21swapper/1007:06:192
1669326352,9sleep100-21swapper/1007:06:192
1365826048,9sleep40-21swapper/407:05:126
1365826048,9sleep40-21swapper/407:05:126
46572570,0sleep100-21swapper/1011:45:132
144132540,0sleep100-21swapper/1011:55:262
1731599270,1cyclictest0-21swapper/1010:30:252
1731599220,1cyclictest0-21swapper/1007:35:232
275991210,1ptp4l21-21ksoftirqd/108:50:131
1728199212,15cyclictest0-21swapper/410:15:016
1728199211,16cyclictest0-21swapper/407:30:006
1728199211,16cyclictest0-21swapper/407:30:006
1732099200,18cyclictest0-21swapper/1107:45:173
275991180,0ptp4l0-21swapper/110:40:141
1727299183,9cyclictest0-21swapper/210:55:004
275991170,1ptp4l21-21ksoftirqd/108:40:001
1728999171,13cyclictest21287-21munin-node07:15:017
1728999171,13cyclictest21287-21munin-node07:15:017
1727799172,9cyclictest0-21swapper/309:45:025
275991160,1ptp4l21-21ksoftirqd/112:35:211
275991160,1ptp4l21-21ksoftirqd/109:06:141
275991160,1ptp4l21-21ksoftirqd/108:25:221
275991160,1ptp4l21-21ksoftirqd/107:10:191
275991160,1ptp4l21-21ksoftirqd/107:10:191
275991160,0ptp4l0-21swapper/107:55:141
1727799163,7cyclictest0-21swapper/309:25:015
1726899162,8cyclictest0-21swapper/109:50:021
186662150,0sleep70-21swapper/709:15:179
1730999153,7cyclictest0-21swapper/908:05:0011
1730999152,8cyclictest0-21swapper/908:55:0111
1730999152,8cyclictest0-21swapper/908:30:0111
1730999152,7cyclictest0-21swapper/910:05:0111
1730999151,9cyclictest0-21swapper/910:10:0211
1728199150,3cyclictest0-21swapper/407:10:166
1728199150,3cyclictest0-21swapper/407:10:166
1727799153,7cyclictest0-21swapper/312:30:015
1727799152,8cyclictest0-21swapper/307:15:255
1727799152,8cyclictest0-21swapper/307:15:255
1727299152,8cyclictest0-21swapper/211:05:284
1726899155,6cyclictest0-21swapper/108:45:011
1726899152,8cyclictest0-21swapper/107:25:001
1726899152,8cyclictest0-21swapper/107:25:001
35172140,0sleep90-21swapper/911:45:0011
275991140,1ptp4l21-21ksoftirqd/111:27:141
275991140,0ptp4l0-21swapper/108:17:301
1730999143,6cyclictest0-21swapper/912:35:0011
1730999142,8cyclictest0-21swapper/909:15:2811
1730999142,7cyclictest0-21swapper/911:55:0011
1730999142,7cyclictest0-21swapper/911:25:2511
1730999142,7cyclictest0-21swapper/910:35:2711
1730999141,8cyclictest9471-21kworker/u24:210:00:0111
1730999141,7cyclictest0-21swapper/909:45:0011
1727799141,7cyclictest0-21swapper/307:55:005
1727299142,8cyclictest0-21swapper/212:10:004
1726899142,7cyclictest0-21swapper/112:20:011
1726899142,7cyclictest0-21swapper/111:55:001
1726899142,7cyclictest0-21swapper/109:25:011
275991130,5ptp4l21-21ksoftirqd/111:00:141
275991130,1ptp4l21-21ksoftirqd/111:48:531
275991130,1ptp4l21-21ksoftirqd/108:00:281
275991130,1ptp4l21-21ksoftirqd/107:35:251
275991130,0ptp4l0-21swapper/111:18:261
1732099132,7cyclictest0-21swapper/1110:45:003
1731599131,1cyclictest18268-21cat10:40:002
1731599130,12cyclictest20888-21awk07:15:002
1731599130,12cyclictest20888-21awk07:15:002
1728999132,6cyclictest0-21swapper/508:00:017
1728199132,7cyclictest0-21swapper/412:05:266
1728199132,7cyclictest0-21swapper/411:55:266
1727799134,5cyclictest0-21swapper/311:00:005
1727799133,7cyclictest0-21swapper/310:10:245
1727799132,8cyclictest0-21swapper/309:00:015
1727799130,12cyclictest0-21swapper/310:39:595
1727299132,7cyclictest0-21swapper/212:35:014
1726899132,7cyclictest0-21swapper/109:25:241
94980120,7rtkit-daemon0-21swapper/807:07:0110
94980120,7rtkit-daemon0-21swapper/807:07:0110
275991120,4ptp4l21-21ksoftirqd/108:30:121
275991120,3ptp4l12670-21fschecks_count11:55:171
275991120,1ptp4l21-21ksoftirqd/110:55:001
275991120,1ptp4l21-21ksoftirqd/109:34:461
275991120,1ptp4l21-21ksoftirqd/107:15:151
275991120,1ptp4l21-21ksoftirqd/107:15:141
275991120,0ptp4l0-21swapper/112:30:151
275991120,0ptp4l0-21swapper/111:35:221
275991120,0ptp4l0-21swapper/111:10:301
275991120,0ptp4l0-21swapper/110:08:141
275991120,0ptp4l0-21swapper/109:47:281
1732099122,7cyclictest0-21swapper/1109:40:223
1732099122,6cyclictest0-21swapper/1108:40:013
1731599122,6cyclictest0-21swapper/1009:45:242
1731599120,8cyclictest0-21swapper/1010:00:012
1730999122,6cyclictest0-21swapper/912:15:2611
1730999122,6cyclictest0-21swapper/910:55:0111
1730999120,7cyclictest0-21swapper/910:25:0011
1730999120,7cyclictest0-21swapper/909:10:2411
1730099126,2cyclictest1020-21runrttasks09:45:299
1730099120,12cyclictest0-21swapper/707:25:149
1730099120,12cyclictest0-21swapper/707:25:149
1728999121,7cyclictest9471-21kworker/u24:209:20:027
1728999121,6cyclictest0-21swapper/508:55:257
1728199122,6cyclictest0-21swapper/408:00:246
1727799122,6cyclictest0-21swapper/307:20:025
1727799122,6cyclictest0-21swapper/307:20:015
1727799121,7cyclictest9471-21kworker/u24:212:15:005
1727299128,3cyclictest32482-21irqrtprio08:50:184
1727299122,7cyclictest0-21swapper/208:45:024
1726399129,2cyclictest30425-21turbostat08:50:000
1726399120,8cyclictest9471-21kworker/u24:210:35:000
275991110,6ptp4l21-21ksoftirqd/108:05:321
275991110,4ptp4l21-21ksoftirqd/109:55:241
275991110,2ptp4l17574-21memory12:00:201
275991110,2ptp4l15856-21fschecks_time10:35:161
275991110,1ptp4l0-21swapper/111:30:181
275991110,1ptp4l0-21swapper/110:20:171
275991110,1ptp4l0-21swapper/110:15:151
275991110,1ptp4l0-21swapper/109:15:171
275991110,1ptp4l0-21swapper/109:10:171
275991110,1ptp4l0-21swapper/108:45:171
275991110,1ptp4l0-21swapper/107:50:281
275991110,0ptp4l0-21swapper/109:02:111
1732099112,6cyclictest0-21swapper/1110:45:273
1730499110,3cyclictest0-21swapper/808:40:1810
1730099112,6cyclictest0-21swapper/707:45:009
1730099112,5cyclictest0-21swapper/712:25:019
1730099110,1cyclictest0-21swapper/711:37:479
1730099110,1cyclictest0-21swapper/711:26:209
1730099110,1cyclictest0-21swapper/711:06:139
1730099110,10cyclictest0-21swapper/707:35:149
1728999112,6cyclictest0-21swapper/512:00:267
1728999112,5cyclictest0-21swapper/508:35:007
1728999111,6cyclictest0-21swapper/511:30:007
1728999111,6cyclictest0-21swapper/510:20:277
1728199111,6cyclictest0-21swapper/412:10:266
1728199111,6cyclictest0-21swapper/407:20:306
1728199111,6cyclictest0-21swapper/407:20:296
1728199110,9cyclictest28599-21irqrtprio08:45:196
1727299112,6cyclictest0-21swapper/212:05:004
1727299112,6cyclictest0-21swapper/210:00:274
1727299112,5cyclictest0-21swapper/211:00:004
275991100,3ptp4l21-21ksoftirqd/111:05:241
275991100,1ptp4l753-21ps12:20:201
275991100,1ptp4l21-21ksoftirqd/110:00:211
275991100,1ptp4l0-21swapper/107:45:251
191591100,0getstats27-21ksoftirqd/210:35:224
1732099101,5cyclictest0-21swapper/1107:25:263
1732099101,5cyclictest0-21swapper/1107:25:253
1731599102,5cyclictest0-21swapper/1009:05:012
1731599102,5cyclictest0-21swapper/1007:35:002
1731599101,5cyclictest0-21swapper/1008:10:242
1731599100,9cyclictest0-21swapper/1009:15:002
1731599100,5cyclictest0-21swapper/1010:59:592
1730999108,1cyclictest0-21swapper/912:20:1211
1730999101,6cyclictest0-21swapper/910:40:2311
1730999100,9cyclictest0-21swapper/910:25:2311
1730499107,2cyclictest0-21swapper/809:45:0010
1730499100,10cyclictest0-21swapper/811:55:2010
1730099102,7cyclictest0-21swapper/710:50:009
1730099101,6cyclictest0-21swapper/709:15:019
1730099100,9cyclictest0-21swapper/709:25:189
1730099100,7cyclictest0-21swapper/712:08:549
1730099100,7cyclictest0-21swapper/709:36:389
1730099100,7cyclictest0-21swapper/707:31:189
1730099100,6cyclictest0-21swapper/712:13:429
1730099100,6cyclictest0-21swapper/711:12:139
1730099100,6cyclictest0-21swapper/710:11:189
1730099100,6cyclictest0-21swapper/708:57:409
1730099100,1cyclictest0-21swapper/712:18:539
1730099100,1cyclictest0-21swapper/711:55:599
1730099100,1cyclictest0-21swapper/711:53:229
1730099100,1cyclictest0-21swapper/711:31:119
1730099100,1cyclictest0-21swapper/707:20:309
1730099100,1cyclictest0-21swapper/707:20:299
1730099100,1cyclictest0-21swapper/707:12:169
1730099100,1cyclictest0-21swapper/707:12:169
1729599107,2cyclictest0-21swapper/607:45:198
1728999101,5cyclictest0-21swapper/511:34:587
1728199100,2cyclictest24355-21basename08:00:016
1727799102,5cyclictest0-21swapper/311:40:005
1727799102,5cyclictest0-21swapper/308:25:005
1727799102,5cyclictest0-21swapper/308:15:275
1727799100,9cyclictest27439-21sed11:30:245
1727799100,9cyclictest0-21swapper/312:30:275
1727299106,3cyclictest24210-21/usr/sbin/munin12:10:174
1727299102,5cyclictest0-21swapper/210:30:004
1727299100,3cyclictest28847-21switchtime07:20:294
1727299100,3cyclictest28847-21switchtime07:20:294
1727299100,1cyclictest0-21swapper/211:20:204
1726899107,2cyclictest1117-21turbostat07:30:011
1726899107,2cyclictest1117-21turbostat07:30:001
1726899107,2cyclictest1109-21fschecks_time11:40:211
27609190,3phc2sys0-21swapper/207:08:534
27609190,3phc2sys0-21swapper/207:08:534
27599190,1ptp4l21-21ksoftirqd/112:25:131
27599190,1ptp4l21-21ksoftirqd/111:20:141
27599190,1ptp4l21-21ksoftirqd/110:10:141
27599190,1ptp4l21-21ksoftirqd/109:40:121
27599190,1ptp4l21-21ksoftirqd/109:35:111
27599190,1ptp4l21-21ksoftirqd/108:20:131
27599190,1ptp4l21-21ksoftirqd/107:40:171
27599190,0ptp4l0-21swapper/112:10:151
27599190,0ptp4l0-21swapper/112:05:141
27599190,0ptp4l0-21swapper/110:50:181
27599190,0ptp4l0-21swapper/110:45:151
27599190,0ptp4l0-21swapper/110:30:151
27599190,0ptp4l0-21swapper/110:25:141
27599190,0ptp4l0-21swapper/108:55:161
27599190,0ptp4l0-21swapper/108:10:011
27599190,0ptp4l0-21swapper/107:30:141
173209998,1cyclictest0-21swapper/1107:35:003
173209990,8cyclictest13596-21diskmemload12:22:333
173209990,8cyclictest0-21swapper/1107:55:003
173209990,1cyclictest0-21swapper/1108:25:013
173159998,1cyclictest0-21swapper/1012:10:262
173159998,1cyclictest0-21swapper/1009:35:012
173159998,1cyclictest0-21swapper/1008:20:242
173159997,1cyclictest0-21swapper/1011:00:192
173159996,2cyclictest20964-21irqrtprio08:35:182
173159996,2cyclictest11439-21fw_forwarded_lo09:05:172
173159990,9cyclictest0-21swapper/1011:25:152
173159990,9cyclictest0-21swapper/1007:45:162
173159990,8cyclictest0-21swapper/1008:45:002
173159990,4cyclictest0-21swapper/1008:30:012
173159990,1cyclictest0-21swapper/1010:05:252
173099990,8cyclictest0-21swapper/907:25:1911
173099990,8cyclictest0-21swapper/907:25:1811
173099990,2cyclictest0-21swapper/908:47:0311
173049991,7cyclictest9471-21kworker/u24:207:35:0010
173049990,9cyclictest0-21swapper/810:20:0010
173049990,9cyclictest0-21swapper/809:25:1810
173049990,1cyclictest0-21swapper/808:15:2610
173009998,1cyclictest0-21swapper/712:35:159
173009992,6cyclictest0-21swapper/710:20:019
173009992,4cyclictest0-21swapper/708:45:259
173009991,5cyclictest0-21swapper/708:35:249
173009991,5cyclictest0-21swapper/708:05:019
173009991,2cyclictest0-21swapper/710:40:019
173009990,6cyclictest0-21swapper/708:52:329
173009990,6cyclictest0-21swapper/708:32:129
173009990,6cyclictest0-21swapper/708:22:079
173009990,6cyclictest0-21swapper/707:54:249
173009990,1cyclictest0-21swapper/710:25:259
172959998,1cyclictest0-21swapper/610:34:598
172959990,4cyclictest0-21swapper/609:25:008
172959990,2cyclictest26750-21fw_conntrack09:25:158
172959990,1cyclictest0-21swapper/608:50:138
172899990,2cyclictest31012-21/usr/sbin/munin11:35:247
172819990,8cyclictest0-21swapper/411:10:176
172779996,2cyclictest13596-21diskmemload09:21:305
172779992,4cyclictest0-21swapper/311:20:005
172779991,5cyclictest0-21swapper/311:10:005
172779990,7cyclictest23693-21fw_conntrack10:45:175
172729991,6cyclictest0-21swapper/211:05:004
172729991,5cyclictest0-21swapper/212:20:004
172729991,5cyclictest0-21swapper/209:05:014
172729990,9cyclictest0-21swapper/209:25:024
172729990,8cyclictest4822-21latency_hist09:40:014
172729990,8cyclictest16089-21grep08:30:134
172729990,8cyclictest0-21swapper/208:15:264
172729990,1cyclictest0-21swapper/210:25:244
172639995,3cyclictest30436-21timerandwakeup10:10:250
172639991,5cyclictest0-21swapper/012:25:000
16679291,6sleep90-21swapper/907:06:0611
16679291,6sleep90-21swapper/907:06:0511
28124280,0sleep100-21swapper/1011:35:012
173209987,1cyclictest0-21swapper/1109:00:013
173209987,1cyclictest0-21swapper/1107:20:293
173209987,1cyclictest0-21swapper/1107:20:293
173209980,1cyclictest0-21swapper/1112:10:173
173209980,1cyclictest0-21swapper/1109:15:293
173159987,1cyclictest0-21swapper/1007:20:232
173159987,1cyclictest0-21swapper/1007:20:232
173159980,8cyclictest0-21swapper/1011:15:252
173159980,8cyclictest0-21swapper/1010:20:262
173159980,8cyclictest0-21swapper/1009:40:132
173159980,8cyclictest0-21swapper/1009:27:152
173159980,1cyclictest0-21swapper/1012:20:232
173159980,1cyclictest0-21swapper/1012:15:062
173159980,1cyclictest0-21swapper/1009:20:112
173099987,1cyclictest0-21swapper/912:10:1411
173099987,1cyclictest0-21swapper/911:45:1311
173099980,8cyclictest0-21swapper/908:30:1411
173099980,1cyclictest0-21swapper/911:55:2211
173099980,1cyclictest0-21swapper/908:05:1411
173099980,1cyclictest0-21swapper/907:20:2611
173099980,1cyclictest0-21swapper/907:20:2611
173049987,1cyclictest0-21swapper/811:25:1710
173049987,1cyclictest0-21swapper/810:50:2210
173049987,1cyclictest0-21swapper/808:35:2210
173049987,1cyclictest0-21swapper/808:10:1310
173049986,1cyclictest0-21swapper/809:20:1010
173049983,4cyclictest0-21swapper/808:34:2710
173049980,1cyclictest0-21swapper/810:20:1510
173049980,1cyclictest0-21swapper/809:55:1910
173049980,1cyclictest0-21swapper/808:00:1910
173049980,0cyclictest0-21swapper/811:05:1610
173009987,1cyclictest0-21swapper/710:40:209
173009987,1cyclictest0-21swapper/710:20:209
173009987,1cyclictest0-21swapper/710:00:279
173009982,5cyclictest0-21swapper/708:20:019
173009980,1cyclictest0-21swapper/709:20:109
172959987,1cyclictest0-21swapper/609:50:158
172959987,1cyclictest0-21swapper/607:55:188
172959980,8cyclictest0-21swapper/612:00:158
172959980,8cyclictest0-21swapper/607:10:178
172959980,8cyclictest0-21swapper/607:10:168
172959980,7cyclictest0-21swapper/608:36:318
172959980,1cyclictest0-21swapper/612:35:158
172959980,1cyclictest0-21swapper/612:20:168
172959980,1cyclictest0-21swapper/611:55:188
172959980,1cyclictest0-21swapper/611:08:158
172959980,1cyclictest0-21swapper/609:30:228
172899986,1cyclictest58262sleep512:28:497
172899980,8cyclictest0-21swapper/510:45:167
172899980,8cyclictest0-21swapper/509:20:177
172899980,8cyclictest0-21swapper/509:10:137
172899980,1cyclictest0-21swapper/509:55:217
172899980,1cyclictest0-21swapper/507:45:147
172819987,1cyclictest0-21swapper/407:45:136
172819980,8cyclictest0-21swapper/409:10:246
172819980,8cyclictest0-21swapper/409:05:256
172819980,1cyclictest0-21swapper/410:45:226
172819980,1cyclictest0-21swapper/410:30:226
172819980,1cyclictest0-21swapper/409:35:126
172779987,1cyclictest0-21swapper/310:25:225
172779987,1cyclictest0-21swapper/308:30:325
172779986,1cyclictest15773-21switchtime07:45:235
172779980,8cyclictest0-21swapper/309:35:205
172779980,1cyclictest0-21swapper/311:25:205
172779980,1cyclictest0-21swapper/311:15:225
172779980,1cyclictest0-21swapper/309:10:125
172729987,1cyclictest0-21swapper/210:10:184
172729987,1cyclictest0-21swapper/210:05:154
172729987,1cyclictest0-21swapper/209:50:194
172729987,1cyclictest0-21swapper/207:45:234
172729987,1cyclictest0-21swapper/207:15:244
172729987,1cyclictest0-21swapper/207:15:244
172729981,6cyclictest0-21swapper/212:30:004
172729980,8cyclictest0-21swapper/211:50:244
172729980,8cyclictest0-21swapper/210:15:134
172729980,8cyclictest0-21swapper/209:40:234
172729980,7cyclictest15600-21/usr/sbin/munin11:15:224
172729980,1cyclictest0-21swapper/212:35:154
172729980,1cyclictest0-21swapper/212:20:154
172729980,1cyclictest0-21swapper/209:55:244
172729980,1cyclictest0-21swapper/209:30:234
172729980,1cyclictest0-21swapper/209:10:244
172729980,1cyclictest0-21swapper/207:55:154
172729980,1cyclictest0-21swapper/207:35:004
172639987,1cyclictest0-21swapper/011:25:130
172639986,1cyclictest5013-21taskset11:02:580
172639985,2cyclictest28441-21if_enp1s012:15:180
172639980,4cyclictest0-21swapper/007:50:020
172639980,1cyclictest0-21swapper/011:05:270
15998281,5sleep30-21swapper/307:05:235
15998281,5sleep30-21swapper/307:05:235
173209976,1cyclictest0-21swapper/1112:35:043
173209976,1cyclictest0-21swapper/1112:25:233
173209976,1cyclictest0-21swapper/1112:15:333
173209976,1cyclictest0-21swapper/1112:05:363
173209976,1cyclictest0-21swapper/1111:55:013
173209976,1cyclictest0-21swapper/1111:45:033
173209976,1cyclictest0-21swapper/1111:20:163
173209976,1cyclictest0-21swapper/1111:05:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional