You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-13 - 18:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Thu Nov 13, 2025 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911140,57ptp4l0-21swapper/107:06:041
94980790,65rtkit-daemon0-21swapper/007:06:020
1483126949,10sleep40-21swapper/407:05:116
1810826654,10sleep50-21swapper/507:09:487
1459726653,10sleep70-21swapper/707:05:019
1782026554,9sleep60-21swapper/607:05:368
1809826452,9sleep100-21swapper/1007:09:422
1811626352,9sleep110-21swapper/1107:09:553
127982540,0sleep100-21swapper/1008:25:002
1850799212,14cyclictest0-21swapper/208:25:014
211412190,0sleep70-21swapper/707:10:269
1850299192,10cyclictest0-21swapper/107:45:001
275991170,0ptp4l0-21swapper/107:25:221
1853699173,8cyclictest0-21swapper/811:10:0010
1852799170,3cyclictest0-21swapper/608:00:188
1850299172,9cyclictest0-21swapper/111:39:591
18497991715,1cyclictest25478-21turbostat08:00:010
275991160,1ptp4l21-21ksoftirqd/110:20:221
275991160,0ptp4l0-21swapper/110:30:171
275991160,0ptp4l0-21swapper/109:00:161
275991160,0ptp4l0-21swapper/108:42:421
192072160,0sleep40-21swapper/409:15:126
1853699163,7cyclictest0-21swapper/809:45:0010
1853299162,8cyclictest0-21swapper/707:25:019
1852799160,3cyclictest0-21swapper/612:00:178
1852199165,7cyclictest0-21swapper/509:25:017
1852199164,7cyclictest0-21swapper/511:15:007
1852199163,8cyclictest0-21swapper/507:40:017
1852199162,9cyclictest0-21swapper/507:50:017
1850799163,8cyclictest0-21swapper/212:34:594
1850799163,8cyclictest0-21swapper/212:30:014
1850799163,8cyclictest0-21swapper/208:55:244
1850799163,8cyclictest0-21swapper/208:15:014
1850799162,8cyclictest0-21swapper/209:05:004
1849799162,8cyclictest0-21swapper/010:50:000
275991150,1ptp4l21-21ksoftirqd/110:05:151
275991150,0ptp4l0-21swapper/110:55:181
1853299152,9cyclictest0-21swapper/711:00:009
1853299152,7cyclictest0-21swapper/708:10:009
1853299151,8cyclictest0-21swapper/711:55:009
1852799152,8cyclictest0-21swapper/612:15:018
1852799152,8cyclictest0-21swapper/611:35:008
18527991510,4cyclictest26238-21turbostat09:25:018
1850799153,8cyclictest0-21swapper/211:20:254
1850799153,8cyclictest0-21swapper/208:15:004
1850799152,8cyclictest0-21swapper/210:00:244
1850299153,7cyclictest0-21swapper/107:50:251
181112157,6sleep80-21swapper/807:09:5210
275991140,0ptp4l0-21swapper/109:26:411
1853699143,6cyclictest0-21swapper/809:55:2410
1853299143,7cyclictest0-21swapper/708:40:019
1853299142,7cyclictest0-21swapper/712:15:019
1852799143,7cyclictest0-21swapper/609:40:008
1852799143,7cyclictest0-21swapper/607:45:018
1852799142,7cyclictest0-21swapper/609:45:008
1852799142,7cyclictest0-21swapper/607:30:008
1852799140,3cyclictest1907-21if_err_enp1s010:15:208
1852799140,3cyclictest11851-21sed11:50:248
1852799140,1cyclictest0-21swapper/609:25:168
1852199143,7cyclictest0-21swapper/509:35:017
1852199142,8cyclictest0-21swapper/510:04:597
1852199141,8cyclictest0-21swapper/512:20:007
1850799142,8cyclictest0-21swapper/208:00:254
1850799142,7cyclictest0-21swapper/209:25:274
1850299142,7cyclictest0-21swapper/107:30:231
1850299141,7cyclictest0-21swapper/109:20:011
155682140,0sleep30-21swapper/311:15:135
275991130,5ptp4l21-21ksoftirqd/107:55:151
275991130,1ptp4l21-21ksoftirqd/111:22:011
275991130,1ptp4l21-21ksoftirqd/108:50:201
275991130,0ptp4l0-21swapper/111:05:231
275991130,0ptp4l0-21swapper/110:48:461
275991130,0ptp4l0-21swapper/108:45:141
275991130,0ptp4l0-21swapper/108:30:401
275991130,0ptp4l0-21swapper/107:45:211
18541991311,1cyclictest4809-21turbostat08:15:0111
1853699131,7cyclictest0-21swapper/810:50:0010
1853299132,6cyclictest0-21swapper/711:25:279
1853299131,11cyclictest30735-21cat08:45:259
1852799132,7cyclictest0-21swapper/609:45:258
1852799131,8cyclictest0-21swapper/609:59:598
1852799130,4cyclictest249991cstates08:40:168
1852799130,3cyclictest0-21swapper/607:15:168
1852199132,6cyclictest0-21swapper/508:25:017
1852199131,8cyclictest0-21swapper/511:20:007
1850799132,7cyclictest0-21swapper/211:30:004
1849799132,7cyclictest0-21swapper/008:15:010
1849799131,8cyclictest0-21swapper/009:00:010
1849799130,8cyclictest0-21swapper/007:45:010
180052131,5sleep30-21swapper/307:08:235
275991120,1ptp4l0-21swapper/109:45:151
275991120,0ptp4l0-21swapper/112:29:571
275991120,0ptp4l0-21swapper/110:44:451
275991120,0ptp4l0-21swapper/107:35:171
1855099122,7cyclictest0-21swapper/1108:30:013
1853699122,6cyclictest0-21swapper/808:30:0010
1852799120,7cyclictest0-21swapper/610:44:598
1852799120,3cyclictest22906-21cpuspeed_turbos11:25:168
1852199122,6cyclictest0-21swapper/512:00:007
1849799122,6cyclictest0-21swapper/011:30:000
1849799121,7cyclictest0-21swapper/011:55:260
1849799121,6cyclictest0-21swapper/008:10:000
275991110,4ptp4l21-21ksoftirqd/109:40:201
275991110,2ptp4l2281-21cat11:40:131
275991110,2ptp4l12061-21fw_conntrack11:10:171
275991110,1ptp4l30013-21if_err_enp1s0.112:15:211
275991110,1ptp4l21-21ksoftirqd/108:15:141
275991110,1ptp4l21-21ksoftirqd/108:05:161
275991110,1ptp4l0-21swapper/109:20:171
275991110,1ptp4l0-21swapper/109:05:201
275991110,0ptp4l0-21swapper/112:31:331
1855099112,5cyclictest0-21swapper/1108:45:003
1855099110,10cyclictest25946-21fw_conntrack12:10:163
1854599110,2cyclictest0-21swapper/1010:05:182
1854199110,7cyclictest0-21swapper/911:39:5911
1854199110,7cyclictest0-21swapper/910:10:0011
1853699110,6cyclictest0-21swapper/809:10:0010
1853299112,6cyclictest0-21swapper/711:20:009
1852799112,8cyclictest8034-21gltestperf07:35:178
1852799112,8cyclictest19967-21grep07:10:188
1852799110,8cyclictest0-21swapper/612:34:348
1852799110,5cyclictest6150-21forks08:15:158
1852799110,5cyclictest4154-21cpuspeed_turbos08:55:138
1852799110,5cyclictest3461-21grep11:00:138
1852799110,5cyclictest32460-21grep08:50:118
1852799110,5cyclictest31934-21munin-run12:20:008
1852799110,5cyclictest20595-21irqrtprio09:15:188
1852799110,5cyclictest16052-21gltestperf11:15:158
1852799110,5cyclictest14653-21latency_hist11:15:008
1852799110,5cyclictest14052-21cpuspeed_turbos09:50:138
1852799110,5cyclictest11816-21date08:20:238
1852799110,2cyclictest192522sleep610:38:488
1852799110,2cyclictest0-21swapper/610:10:008
1852799110,2cyclictest0-21swapper/607:45:238
1852199110,10cyclictest0-21swapper/512:10:257
1850299112,8cyclictest0-21swapper/112:40:001
1850299110,10cyclictest27194-21gltestperf08:00:161
1849799112,5cyclictest0-21swapper/010:30:260
1849799110,10cyclictest26273-21fschecks_time10:05:170
179982115,5sleep90-21swapper/907:08:1711
275991100,2ptp4l29686-21proc_pri10:50:221
275991100,2ptp4l21-21ksoftirqd/110:00:151
275991100,2ptp4l11215-21kernelversion11:50:191
275991100,1ptp4l31104-21cpuspeed_turbos09:30:181
275991100,1ptp4l25603-21diskstats12:10:151
275991100,1ptp4l24886-21fw_forwarded_lo07:15:161
275991100,1ptp4l24428-21memory11:25:221
275991100,1ptp4l21-21ksoftirqd/112:20:111
275991100,1ptp4l15257-21cpuspeed_turbos11:15:121
275991100,1ptp4l14900-21missed_timers08:25:201
275991100,1ptp4l14081-21cpuspeed_turbos09:50:121
275991100,1ptp4l0-21swapper/108:25:011
275991100,0ptp4l0-21swapper/111:55:151
275991100,0ptp4l0-21swapper/111:45:131
275991100,0ptp4l0-21swapper/110:10:271
1855099107,2cyclictest30902-21perl08:05:153
1855099107,2cyclictest21028-21memory09:15:193
1855099101,6cyclictest0-21swapper/1109:30:003
1854599107,2cyclictest0-21swapper/1009:40:202
1854599100,9cyclictest0-21swapper/1012:10:142
1854599100,1cyclictest0-21swapper/1007:10:252
1854199100,6cyclictest0-21swapper/907:55:0111
1854199100,1cyclictest0-21swapper/907:30:1711
1853699107,2cyclictest0-21swapper/810:40:3010
1852799102,7cyclictest28355-21cut08:45:148
1852799101,8cyclictest2606-21fschecks_count11:40:138
1852799101,8cyclictest0-21swapper/607:30:138
1852799100,5cyclictest9369-21kernelversion09:00:218
1852799100,5cyclictest4533-21grep12:25:148
1852799100,5cyclictest30225-21expr10:10:168
1852799100,5cyclictest21690-21cpuspeed_turbos10:00:138
1852799100,5cyclictest21379-21cpuspeed_turbos12:05:138
1852799100,4cyclictest8076-21sort10:24:598
1852799100,4cyclictest6977-21chrt11:05:038
1852799100,4cyclictest5994-21cut11:45:138
1852799100,4cyclictest4155-21ls08:10:258
1852799100,4cyclictest32672-21gltestperf10:55:188
1852799100,4cyclictest32448-21/usr/sbin/munin12:20:118
1852799100,4cyclictest31769-21gltestperf09:30:208
1852799100,4cyclictest30183-21cpuspeed_turbos08:05:138
1852799100,4cyclictest27989-21cpuspeed_turbos07:20:148
1852799100,4cyclictest27931-21tr10:50:148
1852799100,4cyclictest23670-21chrt10:45:018
1852799100,4cyclictest22473-21cpuspeed_turbos07:55:138
1852799100,4cyclictest21096-21cpuspeed_turbos08:35:148
1852799100,4cyclictest20679-21which07:50:218
1852799100,4cyclictest19200-21cpuspeed_turbos11:20:128
1852799100,4cyclictest18136-21grep09:10:248
1852799100,4cyclictest17154-21cpuspeed_turbos08:30:148
1852799100,4cyclictest13937-21phc2sys-jitter12:35:228
1852799100,4cyclictest13834-21cstates11:55:158
1852799100,4cyclictest13463-21cpuspeed_turbos08:25:148
1852799100,4cyclictest13369-21grep10:30:188
1852799100,4cyclictest11507-21aten2_r3power_p09:05:148
1852799100,3cyclictest8801-21tr10:25:138
1852799100,3cyclictest30512-21grep11:35:138
1852199107,2cyclictest0-21swapper/512:30:167
1852199101,6cyclictest0-21swapper/509:05:277
1852199100,2cyclictest0-21swapper/512:25:197
1852199100,2cyclictest0-21swapper/510:45:237
1851699100,9cyclictest8691-21chrt11:46:166
1851199104,4cyclictest11526-21gdbus11:12:375
1849799100,9cyclictest8171-21basename07:35:180
1849799100,9cyclictest0-21swapper/008:20:140
8473290,0chrt0-21swapper/810:25:1210
27599190,1ptp4l21-21ksoftirqd/109:10:151
27599190,0ptp4l21-21ksoftirqd/110:25:141
27599190,0ptp4l0-21swapper/112:05:011
27599190,0ptp4l0-21swapper/112:00:151
27599190,0ptp4l0-21swapper/111:30:131
27599190,0ptp4l0-21swapper/111:00:151
27599190,0ptp4l0-21swapper/110:35:001
27599190,0ptp4l0-21swapper/110:15:011
27599190,0ptp4l0-21swapper/109:55:141
27599190,0ptp4l0-21swapper/109:35:171
27599190,0ptp4l0-21swapper/108:55:131
27599190,0ptp4l0-21swapper/108:35:021
27599190,0ptp4l0-21swapper/108:10:141
27599190,0ptp4l0-21swapper/107:20:131
27599190,0ptp4l0-21swapper/107:10:161
185509998,1cyclictest0-21swapper/1108:30:143
185509995,3cyclictest9563-21cut08:20:113
185509992,5cyclictest0-21swapper/1108:15:013
185509991,5cyclictest0-21swapper/1110:50:003
185509990,1cyclictest7109-21irqrtprio11:45:193
185509990,1cyclictest0-21swapper/1109:50:243
185509990,1cyclictest0-21swapper/1109:35:143
185459990,2cyclictest22598-21/usr/sbin/munin11:25:152
185419998,1cyclictest0-21swapper/912:25:2711
185419998,1cyclictest0-21swapper/909:35:1711
185419991,5cyclictest0-21swapper/909:30:2911
185419991,5cyclictest0-21swapper/907:40:0111
185419990,8cyclictest13266-21/usr/sbin/munin10:30:1711
185419990,8cyclictest0-21swapper/910:35:1311
185419990,6cyclictest0-21swapper/912:35:0111
185419990,6cyclictest0-21swapper/910:00:0011
185419990,6cyclictest0-21swapper/909:05:0111
185419990,2cyclictest10103-21sshd11:05:5711
185369997,1cyclictest6470-21munin-run11:04:5910
185369992,3cyclictest0-21swapper/810:15:0010
185329997,1cyclictest14754-21diskmemload12:20:139
185329992,4cyclictest0-21swapper/712:00:019
185329990,9cyclictest0-21swapper/708:30:169
185329990,9cyclictest0-21swapper/708:25:019
185219997,1cyclictest0-21swapper/511:40:147
185219996,2cyclictest32354-21turbostat10:19:557
185219990,8cyclictest0-21swapper/510:20:137
185169992,4cyclictest0-21swapper/407:10:006
185169991,5cyclictest0-21swapper/411:15:006
185169991,5cyclictest0-21swapper/410:55:006
185169991,5cyclictest0-21swapper/410:15:006
185169990,6cyclictest0-21swapper/411:05:006
185119996,2cyclictest0-21swapper/307:30:015
185119990,8cyclictest0-21swapper/310:50:205
185119990,1cyclictest0-21swapper/309:35:175
185079996,2cyclictest20459-21turbostat12:05:004
185079990,8cyclictest0-21swapper/211:15:144
185079990,8cyclictest0-21swapper/211:10:194
185079990,8cyclictest0-21swapper/209:50:144
185079990,8cyclictest0-21swapper/209:10:014
185079990,3cyclictest19785-21/usr/sbin/munin08:30:274
185079990,3cyclictest16359-21/usr/sbin/munin07:45:194
184979997,1cyclictest0-21swapper/012:20:110
184979996,2cyclictest1020-21runrttasks10:21:160
184979991,5cyclictest0-21swapper/009:15:000
184979990,2cyclictest27765-21munin-run08:45:010
184979990,1cyclictest0-21swapper/008:15:130
9498080,1rtkit-daemon0-21swapper/007:22:420
31253280,0sleep30-21swapper/311:35:155
27609181,4phc2sys0-21swapper/207:05:234
185509987,1cyclictest0-21swapper/1111:30:153
185509987,1cyclictest0-21swapper/1110:20:203
185509987,1cyclictest0-21swapper/1109:55:183
185509987,1cyclictest0-21swapper/1107:55:113
185509987,1cyclictest0-21swapper/1107:40:143
185509982,3cyclictest0-21swapper/1112:35:253
185509981,4cyclictest0-21swapper/1110:20:003
185509980,8cyclictest0-21swapper/1110:05:203
185509980,1cyclictest0-21swapper/1107:50:053
185509980,1cyclictest0-21swapper/1107:30:113
185459987,1cyclictest0-21swapper/1011:45:002
185459987,1cyclictest0-21swapper/1011:05:202
185459987,1cyclictest0-21swapper/1010:20:202
185459986,1cyclictest0-21swapper/1012:15:252
185459982,5cyclictest0-21swapper/1009:05:002
185459980,8cyclictest0-21swapper/1009:50:242
185459980,1cyclictest0-21swapper/1007:35:142
185419987,1cyclictest0-21swapper/911:40:2311
185419987,1cyclictest0-21swapper/911:20:1311
185419987,1cyclictest0-21swapper/910:20:2311
185419987,1cyclictest0-21swapper/909:15:1911
185419980,8cyclictest0-21swapper/912:05:1811
185419980,8cyclictest0-21swapper/911:10:2511
185419980,8cyclictest0-21swapper/910:15:1811
185419980,6cyclictest0-21swapper/910:45:0011
185419980,5cyclictest0-21swapper/909:00:0111
185419980,1cyclictest0-21swapper/912:35:2511
185419980,1cyclictest0-21swapper/909:05:1811
185419980,1cyclictest0-21swapper/908:50:1811
185369986,1cyclictest23227-21cat09:20:1410
185369986,1cyclictest14621-21fschecks_count09:50:1410
185369980,2cyclictest14754-21diskmemload10:53:2910
185369980,1cyclictest0-21swapper/809:30:1910
185369980,1cyclictest0-21swapper/809:00:1710
185329980,8cyclictest0-21swapper/710:35:159
185329980,8cyclictest0-21swapper/708:55:249
185329980,8cyclictest0-21swapper/708:25:149
185329980,1cyclictest0-21swapper/712:31:069
185329980,1cyclictest0-21swapper/709:30:499
185219987,1cyclictest0-21swapper/509:15:247
185219987,1cyclictest0-21swapper/509:10:247
185219980,8cyclictest0-21swapper/509:50:167
185219980,8cyclictest0-21swapper/509:40:497
185169987,1cyclictest0-21swapper/407:50:016
185169980,8cyclictest0-21swapper/408:15:196
185169980,5cyclictest0-21swapper/410:40:006
185169980,5cyclictest0-21swapper/407:20:016
185169980,3cyclictest30215-21date07:20:256
185169980,1cyclictest0-21swapper/410:05:166
185169980,1cyclictest0-21swapper/409:00:176
185119987,1cyclictest0-21swapper/311:25:195
185119982,4cyclictest0-21swapper/308:55:145
185119982,4cyclictest0-21swapper/308:40:185
185119980,8cyclictest0-21swapper/309:10:185
185119980,8cyclictest0-21swapper/308:20:185
185119980,8cyclictest0-21swapper/307:10:175
185079986,1cyclictest16335-21sshd11:56:034
185079982,5cyclictest0-21swapper/208:40:004
185079981,5cyclictest0-21swapper/211:00:164
185079980,8cyclictest0-21swapper/207:35:154
185079980,8cyclictest0-21swapper/207:15:004
185079980,7cyclictest699-21expr12:20:114
185079980,7cyclictest12998-21diskstats10:30:154
185079980,7cyclictest11768-21/usr/sbin/munin12:35:294
185079980,7cyclictest0-21swapper/212:15:164
185079980,7cyclictest0-21swapper/212:05:234
185079980,7cyclictest0-21swapper/211:25:274
185079980,7cyclictest0-21swapper/211:05:204
185079980,7cyclictest0-21swapper/210:10:264
185079980,7cyclictest0-21swapper/209:15:124
185079980,7cyclictest0-21swapper/207:40:144
185079980,7cyclictest0-21swapper/207:20:174
185079980,2cyclictest1170-21users09:30:304
185079980,2cyclictest10367-21/usr/sbin/munin11:50:144
185079980,2cyclictest0-21swapper/207:50:224
185079980,2cyclictest0-21swapper/207:30:124
185079980,1cyclictest0-21swapper/212:10:244
185079980,1cyclictest0-21swapper/210:05:214
185079980,1cyclictest0-21swapper/208:35:144
185079980,1cyclictest0-21swapper/208:05:154
184979986,1cyclictest0-21swapper/010:55:200
184979980,8cyclictest0-21swapper/012:35:210
184979980,8cyclictest0-21swapper/011:20:190
185509976,1cyclictest0-21swapper/1112:20:153
185509976,1cyclictest0-21swapper/1111:55:493
185509976,1cyclictest0-21swapper/1111:26:043
185509976,1cyclictest0-21swapper/1111:05:023
185509976,1cyclictest0-21swapper/1110:40:123
185509976,1cyclictest0-21swapper/1110:37:373
185509976,1cyclictest0-21swapper/1110:30:113
185509976,1cyclictest0-21swapper/1109:40:283
185509976,1cyclictest0-21swapper/1109:13:003
185509976,1cyclictest0-21swapper/1109:05:073
185509976,1cyclictest0-21swapper/1109:01:163
185509976,1cyclictest0-21swapper/1108:53:303
185509976,1cyclictest0-21swapper/1108:15:193
185509976,1cyclictest0-21swapper/1108:02:233
185509976,1cyclictest0-21swapper/1107:39:043
185509976,1cyclictest0-21swapper/1107:15:253
185509970,7cyclictest0-21swapper/1112:15:183
185509970,7cyclictest0-21swapper/1112:05:283
185509970,7cyclictest0-21swapper/1112:00:393
185509970,7cyclictest0-21swapper/1110:25:463
185509970,7cyclictest0-21swapper/1110:01:043
185509970,7cyclictest0-21swapper/1109:31:153
185509970,7cyclictest0-21swapper/1109:21:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional