You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 01:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Wed Nov 19, 2025 00:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
94980990,95rtkit-daemon0-21swapper/519:06:027
275991980,86ptp4l0-21swapper/119:05:271
564226857,9sleep60-21swapper/619:09:068
564726756,9sleep110-21swapper/1119:09:123
561026654,10sleep00-21swapper/019:08:370
140292660,0sleep10-21swapper/100:10:221
569726452,10sleep70-21swapper/719:09:559
560526351,9sleep100-21swapper/1019:08:352
483926254,5sleep40-21swapper/419:05:286
100302610,1sleep0121rcuc/022:00:190
608299283,19cyclictest0-21swapper/419:15:016
608299283,19cyclictest0-21swapper/419:15:006
608299222,15cyclictest0-21swapper/420:40:016
275991200,1ptp4l21-21ksoftirqd/122:30:161
610099183,10cyclictest0-21swapper/821:35:0010
607399183,10cyclictest0-21swapper/200:00:014
610599172,10cyclictest0-21swapper/922:24:5911
610599170,3cyclictest0-21swapper/919:25:1811
609199173,8cyclictest0-21swapper/623:45:018
607399174,8cyclictest0-21swapper/222:55:014
607399172,10cyclictest0-21swapper/221:00:004
607399170,16cyclictest0-21swapper/223:15:284
275991170,0ptp4l0-21swapper/120:20:211
7822160,0sleep80-21swapper/823:55:1610
611599163,7cyclictest0-21swapper/1122:10:273
611599161,7cyclictest0-21swapper/1122:00:003
610599163,7cyclictest0-21swapper/923:09:5911
610099163,8cyclictest0-21swapper/820:25:0110
609699163,8cyclictest0-21swapper/723:54:599
608799161,14cyclictest0-21swapper/523:30:187
608799161,10cyclictest32653-21kworker/u24:223:50:017
608799160,15cyclictest0-21swapper/519:15:207
608799160,15cyclictest0-21swapper/519:15:197
607399163,8cyclictest0-21swapper/220:30:014
607399163,8cyclictest0-21swapper/219:10:004
607399162,9cyclictest0-21swapper/223:09:594
606399164,7cyclictest0-21swapper/021:05:010
606399163,8cyclictest0-21swapper/022:54:590
606399163,7cyclictest0-21swapper/019:50:000
606399162,8cyclictest0-21swapper/019:15:010
606399162,8cyclictest0-21swapper/019:15:010
275991160,1ptp4l21-21ksoftirqd/123:55:221
275991160,1ptp4l21-21ksoftirqd/121:35:191
275991160,0ptp4l0-21swapper/120:55:201
611599152,8cyclictest0-21swapper/1120:45:003
611599151,2cyclictest0-21swapper/1122:15:183
610599152,8cyclictest0-21swapper/919:35:0111
610599152,8cyclictest0-21swapper/900:25:2611
610099153,7cyclictest0-21swapper/800:15:0010
610099152,8cyclictest0-21swapper/822:20:0010
609199153,9cyclictest0-21swapper/619:55:008
609199152,8cyclictest0-21swapper/622:20:008
609199152,8cyclictest0-21swapper/621:40:008
609199152,8cyclictest0-21swapper/620:15:008
607899153,7cyclictest0-21swapper/323:10:015
607899152,8cyclictest0-21swapper/319:15:275
607899152,8cyclictest0-21swapper/319:15:275
607399153,8cyclictest0-21swapper/222:40:004
607399150,14cyclictest0-21swapper/221:35:194
606399153,7cyclictest0-21swapper/020:45:010
606399152,8cyclictest0-21swapper/021:55:000
275991150,1ptp4l21-21ksoftirqd/120:05:141
275991150,1ptp4l21-21ksoftirqd/119:55:141
275991150,0ptp4l0-21swapper/119:20:231
275991150,0ptp4l0-21swapper/119:20:221
275991150,0ptp4l0-21swapper/100:00:151
611599142,7cyclictest0-21swapper/1123:40:013
611099142,7cyclictest0-21swapper/1019:35:282
610599142,7cyclictest0-21swapper/922:29:5911
610599142,7cyclictest0-21swapper/921:40:2511
610599140,13cyclictest0-21swapper/921:00:0111
610099143,6cyclictest0-21swapper/800:20:0110
610099142,6cyclictest0-21swapper/821:15:0010
610099141,8cyclictest0-21swapper/823:40:0110
609699143,8cyclictest0-21swapper/723:45:019
609699142,7cyclictest0-21swapper/721:50:009
609699142,7cyclictest0-21swapper/719:55:019
609699142,7cyclictest0-21swapper/700:25:019
6096991412,2cyclictest0-21swapper/719:25:189
609199143,6cyclictest0-21swapper/622:09:598
6091991412,1cyclictest0-21swapper/620:10:008
608799142,8cyclictest0-21swapper/519:45:017
608299143,6cyclictest0-21swapper/400:00:246
607899143,7cyclictest0-21swapper/323:35:285
607899143,7cyclictest0-21swapper/320:30:015
607899142,7cyclictest0-21swapper/322:50:015
607899142,7cyclictest0-21swapper/320:45:005
607899142,7cyclictest0-21swapper/320:10:285
607399141,9cyclictest32653-21kworker/u24:223:35:004
607399140,2cyclictest4634-21awk20:30:144
607399140,2cyclictest0-21swapper/219:50:214
606399142,7cyclictest0-21swapper/019:45:010
54842141,11sleep80-21swapper/819:06:4510
275991140,1ptp4l21-21ksoftirqd/123:20:151
275991140,1ptp4l21-21ksoftirqd/121:25:171
275991140,0ptp4l0-21swapper/122:16:361
275991140,0ptp4l0-21swapper/122:11:361
275991140,0ptp4l0-21swapper/100:35:181
611599131,7cyclictest0-21swapper/1121:35:013
611099132,6cyclictest0-21swapper/1020:40:282
611099130,7cyclictest27587-21awk22:24:592
610599132,7cyclictest0-21swapper/920:10:0011
610099132,6cyclictest0-21swapper/800:30:0010
609199131,6cyclictest0-21swapper/620:50:008
608799130,8cyclictest16889-21kworker/u24:021:30:017
607899132,6cyclictest0-21swapper/322:20:005
607399133,7cyclictest0-21swapper/200:35:014
607399132,7cyclictest0-21swapper/220:45:014
607399132,6cyclictest0-21swapper/222:25:004
607399130,7cyclictest0-21swapper/200:40:014
607399130,2cyclictest0-21swapper/221:16:344
607399130,1cyclictest0-21swapper/220:40:354
607399130,13cyclictest0-21swapper/222:05:184
607399130,12cyclictest0-21swapper/222:15:264
606399133,5cyclictest0-21swapper/021:15:020
606399132,6cyclictest0-21swapper/022:45:010
275991130,5ptp4l21-21ksoftirqd/119:10:161
275991130,5ptp4l21-21ksoftirqd/119:10:151
275991130,1ptp4l27117-21needreboot23:45:201
275991130,1ptp4l21-21ksoftirqd/123:05:141
275991130,1ptp4l21-21ksoftirqd/122:56:521
275991130,1ptp4l21-21ksoftirqd/120:15:101
275991130,0ptp4l0-21swapper/121:10:161
275991130,0ptp4l0-21swapper/119:15:271
275991130,0ptp4l0-21swapper/119:15:271
275991130,0ptp4l0-21swapper/100:05:201
83982120,0sleep90-21swapper/919:50:2611
611599120,3cyclictest0-21swapper/1119:45:203
611599120,11cyclictest0-21swapper/1120:30:003
610599122,6cyclictest0-21swapper/922:49:5911
610599122,6cyclictest0-21swapper/920:00:0111
610599121,6cyclictest0-21swapper/922:55:2611
610599120,7cyclictest0-21swapper/920:39:5611
610599120,7cyclictest0-21swapper/900:34:5911
610599120,11cyclictest21743-21cat20:50:2211
610099120,11cyclictest0-21swapper/800:29:5410
608299126,5cyclictest20749-21latency_hist23:39:596
608299120,12cyclictest0-21swapper/400:39:286
607899120,11cyclictest28779-21latency_hist23:50:005
607399120,3cyclictest20174-21/usr/sbin/munin22:55:164
607399120,2cyclictest30378-21tail22:25:234
607399120,2cyclictest24026-21hddtemp_smartct19:30:204
607399120,2cyclictest18434-21perl20:05:164
607399120,1cyclictest0-21swapper/222:33:314
607399120,1cyclictest0-21swapper/219:45:214
607399120,12cyclictest0-21swapper/200:10:284
607399120,11cyclictest0-21swapper/220:15:244
606999120,11cyclictest0-21swapper/119:50:001
606399129,2cyclictest0-21swapper/021:55:160
275991120,5ptp4l21-21ksoftirqd/122:45:231
275991120,4ptp4l21-21ksoftirqd/119:30:181
275991120,3ptp4l5309-21irqrtprio22:35:211
275991120,3ptp4l30899-21cpuspeed_turbos19:40:131
275991120,1ptp4l21-21ksoftirqd/123:35:011
275991120,1ptp4l21-21ksoftirqd/100:20:221
275991120,0ptp4l0-21swapper/123:30:151
275991120,0ptp4l0-21swapper/123:19:411
275991120,0ptp4l0-21swapper/121:15:051
275991120,0ptp4l0-21swapper/120:30:121
252142120,0sleep70-21swapper/723:45:149
94980110,0rtkit-daemon0-21swapper/323:51:525
611599118,2cyclictest0-21swapper/1100:00:003
611599113,5cyclictest0-21swapper/1100:30:293
611599111,7cyclictest0-21swapper/1119:20:013
611599111,7cyclictest0-21swapper/1119:20:003
611599111,6cyclictest0-21swapper/1120:15:013
611599111,6cyclictest0-21swapper/1119:30:003
611599111,5cyclictest0-21swapper/1120:05:253
611599110,1cyclictest0-21swapper/1123:31:533
610599111,6cyclictest0-21swapper/921:35:0111
610599110,10cyclictest0-21swapper/919:40:1911
610099110,10cyclictest0-21swapper/821:35:2410
610099110,10cyclictest0-21swapper/821:00:0110
609199112,5cyclictest0-21swapper/622:45:018
608799119,1cyclictest0-21swapper/523:24:597
608799111,7cyclictest0-21swapper/500:35:017
608799110,1cyclictest0-21swapper/500:25:177
608799110,1cyclictest0-21swapper/500:15:197
608299118,3cyclictest0-21swapper/422:35:146
608299118,2cyclictest19422-21hddtemp_smartct21:30:186
608299113,2cyclictest2550-21phc2sys-jitter20:25:216
608299112,3cyclictest14411-21forks20:00:196
608299112,2cyclictest23930-21gltestperf23:00:186
608299111,3cyclictest6305-21cut23:20:146
608299111,2cyclictest7314-21if_enp4s019:50:186
608299110,7cyclictest14701-21cat19:20:016
608299110,7cyclictest14701-21cat19:20:006
608299110,3cyclictest5614-21/usr/sbin/munin21:10:246
608299110,3cyclictest5605-21tr21:55:136
608299110,3cyclictest32086-21cpu22:30:146
608299110,3cyclictest27354-21sshd22:21:496
608299110,3cyclictest12964-21runrttasks23:28:536
608299110,11cyclictest0-21swapper/420:45:236
608299110,10cyclictest0-21swapper/419:35:006
607899111,6cyclictest16889-21kworker/u24:023:00:005
607899111,6cyclictest11322-21pool22:00:285
607899110,2cyclictest15471-21cat00:15:005
607399113,5cyclictest0-21swapper/221:00:004
607399112,3cyclictest9283-21cpuspeed_turbos22:00:164
607399112,2cyclictest5884-21chrt21:11:044
607399112,2cyclictest19298-21cat00:20:004
607399112,2cyclictest18756-21expr21:30:164
607399111,8cyclictest0-21swapper/219:36:154
607399110,4cyclictest32224-21latency_hist20:25:004
607399110,4cyclictest22631-21unixbench_singl20:50:274
607399110,4cyclictest1795-21wc19:45:004
607399110,4cyclictest10565-21aten2_r3power_e21:20:124
607399110,3cyclictest8980-21timerandwakeup119:10:234
607399110,3cyclictest8980-21timerandwakeup119:10:234
607399110,3cyclictest6163-21fschecks_time21:55:154
607399110,3cyclictest29751-21/usr/sbin/munin23:50:144
607399110,3cyclictest22723-21tr20:10:224
607399110,3cyclictest21523-21cpuspeed_turbos23:40:154
607399110,3cyclictest16617-21tr19:20:194
607399110,3cyclictest16617-21tr19:20:194
607399110,2cyclictest0-21swapper/219:55:134
607399110,1cyclictest0-21swapper/221:45:184
607399110,1cyclictest0-21swapper/221:25:534
607399110,1cyclictest0-21swapper/220:00:154
607399110,11cyclictest0-21swapper/223:20:224
607399110,11cyclictest0-21swapper/223:00:444
607399110,10cyclictest31724-21/usr/sbin/munin23:10:274
606999112,5cyclictest0-21swapper/100:20:011
275991110,4ptp4l21-21ksoftirqd/123:50:111
275991110,2ptp4l27747-21fw_forwarded_lo19:35:171
275991110,2ptp4l27232-21/usr/sbin/munin21:40:181
275991110,1ptp4l7677-21irqstats19:50:201
275991110,1ptp4l21-21ksoftirqd/122:00:311
275991110,1ptp4l21-21ksoftirqd/121:57:581
275991110,1ptp4l0-21swapper/121:30:181
275991110,0ptp4l0-21swapper/121:00:231
275991110,0ptp4l0-21swapper/120:25:191
118752110,0sleep30-21swapper/319:55:205
611599109,1cyclictest0-21swapper/1123:54:013
611599109,1cyclictest0-21swapper/1123:40:243
611599109,1cyclictest0-21swapper/1123:23:423
611599109,1cyclictest0-21swapper/1100:16:013
611599109,1cyclictest0-21swapper/1100:06:033
611599109,1cyclictest0-21swapper/1100:02:283
611599100,9cyclictest2148-21diskmemload21:20:223
611599100,1cyclictest0-21swapper/1123:26:323
611599100,1cyclictest0-21swapper/1100:10:433
611599100,10cyclictest0-21swapper/1123:45:103
611599100,10cyclictest0-21swapper/1100:36:433
611599100,10cyclictest0-21swapper/1100:27:323
611599100,10cyclictest0-21swapper/1100:22:223
611099109,1cyclictest0-21swapper/1000:37:572
611099108,1cyclictest0-21swapper/1023:20:212
611099100,9cyclictest0-21swapper/1023:09:542
610599108,1cyclictest0-21swapper/922:00:2811
610599100,9cyclictest0-21swapper/921:45:1511
610099107,2cyclictest4394-21/usr/sbin/munin20:30:1510
610099100,9cyclictest0-21swapper/820:25:2410
609699100,9cyclictest0-21swapper/722:50:199
609699100,5cyclictest0-21swapper/721:24:569
609199109,1cyclictest0-21swapper/622:36:068
609199104,5cyclictest0-21swapper/623:59:548
609199104,5cyclictest0-21swapper/600:30:018
609199100,1cyclictest0-21swapper/622:30:178
608799109,1cyclictest0-21swapper/523:57:537
608799109,1cyclictest0-21swapper/523:41:557
608799109,1cyclictest0-21swapper/523:26:457
608799109,1cyclictest0-21swapper/500:36:247
608799109,1cyclictest0-21swapper/500:11:087
608799109,1cyclictest0-21swapper/500:01:347
608799107,2cyclictest9357-21perl00:05:167
608799100,10cyclictest0-21swapper/523:52:087
608799100,10cyclictest0-21swapper/523:36:307
608799100,10cyclictest0-21swapper/500:22:427
608299106,3cyclictest26614-21df_inode21:40:156
608299105,5cyclictest0-21swapper/422:07:506
608299103,2cyclictest28364-21turbostat20:20:006
608299103,1cyclictest10579-21entropy19:55:146
608299102,3cyclictest18117-21irqrtprio22:10:196
608299102,2cyclictest32275-21cut23:55:006
608299102,2cyclictest21221-21ptp4l-jitter19:25:236
608299102,2cyclictest15130-21irqrtprio23:30:196
608299102,1cyclictest11533-21ls22:45:166
608299101,3cyclictest32556-21cat21:05:206
608299100,7cyclictest0-21swapper/419:43:216
608299100,3cyclictest8520-21cat00:05:146
608299100,3cyclictest7960-21basename21:15:186
608299100,3cyclictest3614-21irqrtprio19:45:216
608299100,3cyclictest32122-21irqrtprio23:10:286
608299100,3cyclictest29767-21grep20:20:166
608299100,3cyclictest29648-21sshd23:06:486
608299100,3cyclictest28866-21expr22:25:156
608299100,3cyclictest28329-21irqstats19:35:196
608299100,3cyclictest26193-21cut20:55:256
608299100,3cyclictest23261-21cat20:10:226
608299100,3cyclictest22964-21uname23:40:216
608299100,3cyclictest21265-21diskstats22:15:156
608299100,3cyclictest20286-21cat00:20:146
608299100,3cyclictest17962-21cat20:05:156
608299100,3cyclictest17198-21sed19:20:226
608299100,3cyclictest17198-21sed19:20:216
608299100,3cyclictest16464-21nfs4_client21:25:236
608299100,3cyclictest12739-21expr00:10:176
608299100,3cyclictest10347-21open_files20:35:226
608299100,3cyclictest0-21swapper/421:35:186
608299100,1cyclictest21256-21smartctl22:55:226
608299100,1cyclictest0-21swapper/422:53:236
608299100,1cyclictest0-21swapper/421:00:166
607899107,2cyclictest26481-21users20:55:265
607899102,5cyclictest0-21swapper/321:45:005
607899100,1cyclictest0-21swapper/320:35:265
607399103,2cyclictest8344-21cpu20:35:144
607399102,3cyclictest17049-21cpuspeed_turbos22:10:154
607399100,3cyclictest8242-21aten2_r3power_c00:05:124
607399100,3cyclictest4761-21tr00:00:134
607399100,3cyclictest19943-21awk00:20:134
607399100,3cyclictest11210-21open_files19:15:034
607399100,3cyclictest11210-21open_files19:15:034
607399100,2cyclictest32157-21sort21:05:184
607399100,2cyclictest27140-21if_enp1s021:40:174
607399100,2cyclictest23677-21cpuspeed00:25:144
607399100,2cyclictest18993-21hostname19:25:144
607399100,2cyclictest17390-21chrt23:35:084
607399100,2cyclictest10852chrt21:50:084
607399100,2cyclictest0-21swapper/223:45:144
607399100,2cyclictest0-21swapper/223:25:004
607399100,2cyclictest0-21swapper/222:40:154
607399100,10cyclictest0-21swapper/222:45:164
606399107,2cyclictest0-21swapper/022:55:200
606399100,9cyclictest23903-21timerandwakeup120:10:270
275991100,6ptp4l25441-21gltestperf22:20:171
275991100,2ptp4l20069-21cpuspeed_turbos20:50:161
275991100,1ptp4l598-21sed21:05:211
275991100,1ptp4l4113-21sed21:50:281
275991100,1ptp4l29988-21tr21:45:151
275991100,1ptp4l28589-21hddtemp_smartct00:30:181
275991100,1ptp4l23897-21fw_forwarded_lo23:00:191
275991100,1ptp4l21-21ksoftirqd/122:26:351
275991100,1ptp4l19794-21fschecks_count19:25:151
275991100,1ptp4l16255-21fw_conntrack22:50:161
275991100,1ptp4l13968-21fw_conntrack22:05:171
275991100,1ptp4l12068-21awk23:25:231
61159998,1cyclictest0-21swapper/1119:35:143
61159990,1cyclictest0-21swapper/1120:00:173
61109998,1cyclictest0-21swapper/1022:45:212
61109992,4cyclictest0-21swapper/1021:25:172
61109990,1cyclictest0-21swapper/1019:40:192
61009998,1cyclictest0-21swapper/823:20:2110
61009991,6cyclictest18399-21crond22:54:5910
61009990,9cyclictest0-21swapper/819:55:2610
61009990,4cyclictest0-21swapper/822:34:5910
60969998,1cyclictest0-21swapper/723:36:169
60969997,1cyclictest0-21swapper/721:50:209
60969996,2cyclictest1009-21snmpd20:23:099
60919997,1cyclictest0-21swapper/621:10:168
60919990,9cyclictest0-21swapper/621:20:008
60919990,8cyclictest2148-21diskmemload23:16:358
60919990,8cyclictest0-21swapper/622:40:158
60879996,2cyclictest6247-21fschecks_time21:55:157
60879992,6cyclictest0-21swapper/520:20:007
60879990,8cyclictest2148-21diskmemload22:50:287
60829990,9cyclictest0-21swapper/420:50:276
60829990,3cyclictest9659-21expr22:00:186
60829990,3cyclictest7965-21cpuspeed_turbos22:40:156
60829990,3cyclictest6671-21ptp4l20:30:216
60829990,3cyclictest29740-21cat00:30:286
60829990,3cyclictest25457-21expr23:45:146
60829990,3cyclictest1895-21aten2_r3power_c23:15:146
60829990,3cyclictest1691-21if_enp4s023:55:196
60829990,3cyclictest16510-21cat00:15:156
60829990,3cyclictest1495-21expr21:50:156
60829990,3cyclictest10885-21cpuspeed_turbos21:20:146
60829990,2cyclictest31561-21memory21:45:226
60789998,1cyclictest0-21swapper/321:45:165
60789991,5cyclictest0-21swapper/319:15:005
60789991,5cyclictest0-21swapper/319:15:005
60789990,9cyclictest0-21swapper/300:00:135
60639998,1cyclictest0-21swapper/022:15:190
60639997,1cyclictest0-21swapper/019:15:180
60639997,1cyclictest0-21swapper/019:15:180
60639996,2cyclictest26822-21turbostat00:29:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional