You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-24 - 10:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Wed Dec 24, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4042932212151,65sleep30-21swapper/319:06:433
403827928655,17sleep00-21swapper/019:05:110
404283428569,9sleep20-21swapper/219:05:292
404285127963,10sleep10-21swapper/119:05:411
1912032310,3sleep00-21swapper/000:30:160
4043318992421,1cyclictest0-21swapper/323:55:003
4043318992320,1cyclictest0-21swapper/322:35:193
4043318992318,2cyclictest0-21swapper/319:20:003
4043318992110,7cyclictest104553-21taskset23:07:273
404331899208,9cyclictest56519-21chrt22:23:333
4043318992017,1cyclictest0-21swapper/320:45:133
4043318991916,1cyclictest0-21swapper/322:30:003
4043318991915,2cyclictest0-21swapper/320:55:153
4043318991915,2cyclictest0-21swapper/319:45:003
404331899191,14cyclictest94210-21chrt23:00:023
404331899191,14cyclictest131002-21turbostat23:35:003
404331899191,14cyclictest131002-21turbostat23:35:003
404331899187,8cyclictest173530-21taskset00:15:123
404331899182,10cyclictest4166726-21taskset21:05:033
4043318991815,1cyclictest0-21swapper/319:10:243
4043318991815,1cyclictest0-21swapper/300:30:253
4043318991815,1cyclictest0-21swapper/300:20:183
4043318991815,1cyclictest0-21swapper/300:05:003
404331899181,14cyclictest0-21swapper/321:00:163
404331899181,13cyclictest30195-21turbostat22:00:003
404331899181,10cyclictest4124108-21chrt20:21:573
1967232180,7sleep2421rcuc/200:35:152
40890252171,6sleep24090709-21kthreadcore19:50:202
404331899179,4cyclictest40664052sleep319:30:113
404331899172,10cyclictest4093203-21taskset19:55:123
4043318991715,1cyclictest0-21swapper/319:50:143
4043318991714,1cyclictest0-21swapper/321:20:193
4043318991713,2cyclictest4060929-21turbostat19:25:013
4043318991713,2cyclictest4060929-21turbostat19:25:003
4043318991713,2cyclictest0-21swapper/300:10:173
404331899171,12cyclictest110271-21chrt23:15:123
404331899171,12cyclictest0-21swapper/300:05:133
404331899171,10cyclictest83472-21taskset22:50:113
404331899171,10cyclictest4255-21taskset21:35:123
404331899170,13cyclictest0-21swapper/322:40:113
1037632171,7sleep0103813-21ntpq23:05:200
404331899161,9cyclictest9478-21taskset21:40:013
404331899161,9cyclictest4192967-21taskset21:30:073
404331899161,9cyclictest4139658-21taskset20:35:323
404331899161,9cyclictest141699-21taskset23:45:073
404331899161,9cyclictest110038-21taskset23:15:003
4043318991613,1cyclictest0-21swapper/322:00:153
4043318991613,1cyclictest0-21swapper/321:25:203
4043318991613,1cyclictest0-21swapper/320:30:173
4043318991612,2cyclictest0-21swapper/322:10:253
4043318991612,2cyclictest0-21swapper/322:05:143
4043318991612,2cyclictest0-21swapper/320:25:193
404331899161,12cyclictest0-21swapper/320:10:173
404331899161,11cyclictest4076961-21sleep19:40:003
404331899160,2cyclictest0-21swapper/320:55:003
404331899160,13cyclictest0-21swapper/323:40:223
404331899160,13cyclictest0-21swapper/322:30:233
404331899160,13cyclictest0-21swapper/322:15:163
404331899160,13cyclictest0-21swapper/321:15:203
404331899160,13cyclictest0-21swapper/320:05:203
404331899160,10cyclictest120263-21chrt23:21:573
939972154,2sleep2171rcu_preempt23:00:022
404331899152,1cyclictest0-21swapper/323:35:223
404331899151,9cyclictest4144940-21taskset20:40:333
4043318991513,1cyclictest0-21swapper/323:00:213
4043318991512,2cyclictest0-21swapper/319:25:263
4043318991512,2cyclictest0-21swapper/300:25:153
4043318991512,1cyclictest0-21swapper/319:45:223
4043318991512,1cyclictest0-21swapper/300:35:143
404331899151,12cyclictest0-21swapper/323:55:193
404331899150,13cyclictest0-21swapper/321:10:163
404331899150,13cyclictest0-21swapper/320:15:143
404331899150,12cyclictest0-21swapper/323:25:163
404331899150,12cyclictest0-21swapper/322:45:143
404331899150,12cyclictest0-21swapper/321:45:203
404331899150,12cyclictest0-21swapper/320:00:173
952782141,8sleep196919-21kthreadcore23:00:201
1737712141,8sleep1174033-21grep00:15:151
1625492141,4sleep1162785-21cut00:05:011
1193482141,9sleep0119488-21/usr/sbin/munin23:20:200
689192131,7sleep070612-21kthreadcore22:35:180
566312131,6sleep056654-21munin-run22:25:000
40927142131,8sleep14092717-21sshd19:52:261
40822092131,2sleep10-21swapper/119:45:011
40724862131,8sleep24073615-21cut19:35:162
4043318991310,1cyclictest0-21swapper/321:50:143
659672121,7sleep266352-21tail22:30:192
641992121,7sleep165966-21kthreadcore22:30:181
42332121,5sleep24252-21apt-get21:35:122
41036722121,7sleep14103828-21cstates20:05:131
41033882121,7sleep04103427-21perf20:04:590
40987112121,3sleep14098925-21smartctl20:00:161
353752121,5sleep135424-21munin-run22:05:011
1207642120,2sleep20-21swapper/223:25:102
781942111,4sleep078212-21apt-get22:45:120
535292114,3sleep255326-21cut22:20:172
41929742110,5sleep04192993-21apt-get21:30:140
41865602111,5sleep24186613-21ntp_states21:20:212
416802110,7sleep243301-21kthreadcore22:10:192
41560322111,5sleep04156054-21cpu20:55:100
408492112,4sleep140868-21apt-get22:10:141
40827512112,4sleep14082770-21apt-get19:45:171
40704412111,5sleep14070494-21ntp_states19:30:221
40547362111,4sleep14054789-21ntp_states19:15:231
342662111,5sleep034367-21ntpq22:00:190
1450632111,6sleep0145586-21needreboot23:45:200
834502101,4sleep283469-21apt-get22:50:112
614802102,1sleep0171rcu_preempt22:25:250
597892101,4sleep261043-21memory22:25:182
513472101,4sleep151368-21munin-run22:20:001
463242100,1sleep00-21swapper/022:15:030
41548082101,4sleep14154864-21ntpq20:50:221
41443082101,5sleep04144354-21ntpq20:40:230
41337592101,4sleep04133807-21ntpq20:30:210
41325282101,4sleep24133617-21memory20:30:192
41246832101,4sleep24124896-21df_inode20:25:142
40936902101,5sleep24093832-21grep19:55:162
252142101,4sleep225233-21apt-get21:55:122
1835112101,4sleep0183560-21munin-run00:25:000
1824972101,4sleep1182543-21ntpq00:20:211
1680672101,4sleep0168323-21df00:10:140
1561902101,4sleep0156238-21ntpq23:55:220
1351432101,4sleep2135191-21ntpq23:35:212
1314052101,4sleep0131528-21diskstats23:35:140
1260272101,5sleep00-21swapper/023:30:130
1260272101,5sleep00-21swapper/023:30:130
1090052101,4sleep1109051-21ntpq23:10:221
1050182101,3sleep0105037-21apt-get23:10:140
74358290,5sleep276065-21kthreadcore22:40:172
58110291,4sleep159788-21grep22:25:181
4182513291,3sleep04182532-21apt-get21:20:120
4176991291,4sleep24177012-21munin-run21:15:002
4166739291,2sleep00-21swapper/021:05:120
4146122291,4sleep04147581-21cat20:45:160
4135938291,4sleep04137607-21grep20:35:180
4135316291,4sleep14135362-21fschecks_count20:35:161
4130849291,4sleep14132526-21grep20:30:181
4128556292,2sleep14128602-21ntpq20:25:211
4109440291,4sleep24109517-21kthreadcore20:10:192
4107620291,4sleep04107666-21ntpq20:05:200
4091767291,3sleep04091908-21ntpq19:50:210
4083650291,4sleep24085280-21grep19:45:222
4075832291,4sleep14075926-21modprobe19:35:191
4070762291,4sleep24071009-21users19:30:272
4070393291,3sleep04070439-21ntpq19:30:220
189006291,3sleep2189025-21apt-get00:30:122
18768291,3sleep118824-21ntpq21:45:211
187649291,4sleep0187744-21modprobe00:25:200
147458291,4sleep2147691-21tr23:50:152
13794291,4sleep213867-21awk21:40:252
136474291,3sleep1136493-21apt-get23:40:121
110531291,4sleep0110634-21perl23:15:140
109053291,4sleep2109106-21ntp_states23:10:232
9097282,1sleep10-21swapper/121:38:351
71988280,1sleep10-21swapper/122:35:281
52202281,3sleep153528-21cut22:20:161
4193578281,3sleep11187-21cut21:30:191
4182155280,1sleep20-21swapper/221:19:042
4171998280,3sleep00-21swapper/021:10:130
4161962280,4sleep24163601-21cut21:00:212
4157536281,3sleep14159278-21kthreadcore20:55:151
4154078280,1sleep20-21swapper/220:50:192
4109211281,3sleep14109438-21basename20:10:181
4077228281,3sleep04077235-21cpu19:40:110
4071712282,2sleep04043313-21cyclictest19:35:010
4065130281,3sleep24065177-21ntpq19:25:222
4060216282,3sleep24060290-21xargs19:20:252
4060216282,3sleep24060290-21xargs19:20:252
4051296281,3sleep04052727-21kthreadcore19:15:170
4044051281,3sleep04044864-21cut19:10:160
24693280,1sleep20-21swapper/221:51:442
188461280,1sleep20-21swapper/200:26:262
179659281,3sleep2181345-21cut00:20:182
174034281,3sleep2175661-21kthreadcore00:15:162
168324281,3sleep1168397-21fschecks_time00:10:151
163383281,3sleep20-21swapper/200:05:182
158040281,3sleep2158110-21kthreadcore00:00:172
156515281,4sleep2156766-21tail23:55:282
15316281,3sleep016397-21cut21:45:160
14487281,3sleep114732-21latency_hist21:45:021
137078281,3sleep2138546-21cut23:40:172
130286281,4sleep1761-21sshd23:30:271
130286281,4sleep1761-21sshd23:30:261
110804281,3sleep20-21swapper/223:15:162
99115270,1sleep20-21swapper/223:03:332
93984270,1sleep10-21swapper/122:59:521
88862271,2sleep088454-21apt-get22:55:020
88427271,2sleep00-21swapper/022:54:460
87285270,2sleep187395-21strings22:50:181
82189271,3sleep10-21swapper/122:45:211
76820271,2sleep00-21swapper/022:40:190
63038271,3sleep064197-21kthreadcore22:30:170
4183086271,2sleep10-21swapper/121:20:161
4171466270,1sleep20-21swapper/221:06:422
4166202271,3sleep00-21swapper/021:05:000
4161258270,2sleep10-21swapper/121:00:161
4151355271,3sleep04152338-21kthreadcore20:50:170
4150417270,1sleep20-21swapper/220:48:382
4145291271,3sleep10-21swapper/120:45:011
4120423271,2sleep00-21swapper/020:20:190
4118418270,1sleep10-21swapper/120:15:271
4108384270,4sleep20-21swapper/220:06:422
4098417270,2sleep20-21swapper/220:00:142
4097441271,3sleep04097544-21timerandwakeup19:55:260
4087398270,1sleep00-21swapper/019:46:040
4082173270,1sleep20-21swapper/219:44:332
4059917271,2sleep14059968-21ntpq19:20:201
4059917271,2sleep14059968-21ntpq19:20:201
4055324270,1sleep20-21swapper/219:15:362
36171271,3sleep20-21swapper/222:05:182
20257271,3sleep120280-21forks21:50:151
194237270,2sleep0194256-21apt-get00:35:100
193888270,6sleep10-21swapper/100:33:511
161829270,1sleep00-21swapper/000:00:280
134964270,2sleep1135137-21strings23:35:191
130978271,2sleep2131003-21munin-run23:35:012
130978271,2sleep2131003-21munin-run23:35:002
121058271,2sleep1121272-21smartctl23:25:131
103517270,2sleep2103701-21strings23:05:182
94283260,1sleep00-21swapper/023:00:150
8951260,4sleep00-21swapper/021:36:460
79821261,2sleep20-21swapper/222:45:192
73525260,2sleep10-21swapper/122:40:161
71518260,2sleep20-21swapper/222:35:202
50247261,1sleep20-21swapper/222:15:202
4192503261,1sleep20-21swapper/221:27:262
4187766260,3sleep00-21swapper/021:25:120
4177251260,2sleep10-21swapper/121:15:111
4167931260,2sleep10-21swapper/121:05:171
4124082260,1sleep20-21swapper/220:21:372
4124033260,1sleep10-21swapper/120:21:001
4118943260,1sleep00-21swapper/020:17:300
41150261,2sleep00-21swapper/022:10:170
4097856260,1sleep10-21swapper/119:56:311
4055958260,1sleep00-21swapper/019:20:110
4055958260,1sleep00-21swapper/019:20:100
4044865261,2sleep10-21swapper/119:10:171
4043774260,2sleep24043875-21fschecks_time19:10:142
39900260,3sleep00-21swapper/022:05:280
3705260,1sleep20-21swapper/221:31:392
31788261,2sleep20-21swapper/222:00:182
195074260,2sleep10-21swapper/100:35:151
19430260,1sleep20-21swapper/221:46:222
184342261,2sleep10-21swapper/100:25:171
172993261,2sleep20-21swapper/200:15:002
153613261,2sleep10-21swapper/123:55:191
151725260,1sleep00-21swapper/023:52:210
146388260,1sleep10-21swapper/123:46:211
146384260,1sleep20-21swapper/223:46:192
13716260,1sleep00-21swapper/021:40:230
116094261,2sleep10-21swapper/123:20:171
114980260,1sleep10-21swapper/123:16:391
104518260,1sleep10-21swapper/123:07:011
4187782250,1sleep10-21swapper/121:25:131
4172020250,1sleep10-21swapper/121:10:141
4140588250,1sleep20-21swapper/220:40:172
4139957250,1sleep20-21swapper/220:39:202
4129260250,3sleep00-21swapper/020:26:130
4064988250,1sleep10-21swapper/119:25:211
40518250,3sleep10-21swapper/122:09:091
29003250,1sleep10-21swapper/121:55:191
20737250,1sleep00-21swapper/021:50:170
166431250,1sleep00-21swapper/000:05:200
140251250,1sleep00-21swapper/023:40:190
125595250,3sleep00-21swapper/023:28:120
115698250,1sleep20-21swapper/223:20:132
4177547240,1sleep00-21swapper/021:15:140
4156301240,1sleep20-21swapper/220:55:132
4145715240,1sleep10-21swapper/120:45:141
4114141240,2sleep20-21swapper/220:15:122
4108648240,1sleep00-21swapper/020:10:000
4061451240,1sleep00-21swapper/019:25:150
25107240,1sleep00-21swapper/021:55:020
173555240,1sleep00-21swapper/000:15:140
162811240,1sleep10-21swapper/100:05:131
147242240,1sleep10-21swapper/123:50:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional