You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 00:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Mon Dec 29, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6005212359,59sleep30-21swapper/307:07:273
595328352,10sleep20-21swapper/207:06:452
604527866,6sleep00-21swapper/007:07:560
591427552,17sleep10-21swapper/107:06:161
6340992724,1cyclictest0-21swapper/309:10:143
6340992623,1cyclictest0-21swapper/311:35:003
6340992623,1cyclictest0-21swapper/311:35:003
76102230,2sleep10-21swapper/107:10:161
634099231,9cyclictest255977-21sleep11:05:013
6340992313,7cyclictest266457-21turbostat11:15:003
6340992118,1cyclictest0-21swapper/312:19:593
6340992118,1cyclictest0-21swapper/308:05:013
6340992118,1cyclictest0-21swapper/307:35:003
634099203,10cyclictest28899-21taskset07:25:373
634099203,10cyclictest28899-21taskset07:25:363
6340992017,1cyclictest0-21swapper/312:05:213
6340992017,1cyclictest0-21swapper/309:20:003
6340992016,2cyclictest0-21swapper/307:36:593
6340992014,2cyclictest0-21swapper/310:30:283
6340992014,2cyclictest0-21swapper/310:30:283
634099197,8cyclictest69388-21taskset08:05:173
6340991917,1cyclictest0-21swapper/310:50:003
6340991917,1cyclictest0-21swapper/307:15:013
6340991916,1cyclictest0-21swapper/308:40:183
6340991916,1cyclictest0-21swapper/307:50:163
634099191,4cyclictest113563-21sleep08:50:003
634099191,13cyclictest0-21swapper/310:15:123
634099188,8cyclictest71648-21turbostat08:14:543
634099185,9cyclictest81853-21chrt08:16:523
634099185,1cyclictest0-21swapper/308:25:203
6340991815,2cyclictest0-21swapper/310:40:003
6340991815,1cyclictest0-21swapper/311:55:133
6340991815,1cyclictest0-21swapper/310:10:193
6340991814,2cyclictest0-21swapper/307:24:593
634099181,14cyclictest0-21swapper/311:05:103
6340991811,4cyclictest0-21swapper/309:40:133
634099181,12cyclictest10666-21taskset07:10:213
634099181,10cyclictest274663-21taskset11:20:173
634099181,10cyclictest182467-21taskset09:55:083
634099178,5cyclictest50657-21taskset07:49:143
634099171,9cyclictest346472-21taskset12:30:113
634099171,9cyclictest346472-21taskset12:30:113
6340991715,1cyclictest0-21swapper/311:15:163
6340991714,1cyclictest0-21swapper/312:25:133
6340991713,2cyclictest0-21swapper/310:05:133
6340991713,2cyclictest0-21swapper/307:55:153
634099171,13cyclictest0-21swapper/312:10:373
634099171,13cyclictest0-21swapper/310:00:263
634099170,13cyclictest0-21swapper/309:00:123
634099165,8cyclictest242575-21chrt10:50:153
634099161,9cyclictest213943-21taskset10:25:063
634099161,9cyclictest208682-21taskset10:20:013
6340991613,1cyclictest0-21swapper/311:40:153
6340991613,1cyclictest0-21swapper/309:05:113
6340991612,3cyclictest0-21swapper/309:49:593
6340991612,2cyclictest0-21swapper/311:25:133
634099161,13cyclictest0-21swapper/308:35:133
634099161,12cyclictest0-21swapper/308:20:243
634099161,10cyclictest246051-21taskset10:55:153
634099160,13cyclictest0-21swapper/310:45:003
634099160,13cyclictest0-21swapper/309:35:203
634099160,13cyclictest0-21swapper/309:25:253
634099160,12cyclictest0-21swapper/309:30:213
634099160,10cyclictest177239-21taskset09:50:113
634099151,9cyclictest308432-21taskset11:50:353
6340991513,1cyclictest0-21swapper/308:55:163
6340991512,1cyclictest0-21swapper/307:40:193
6340991511,2cyclictest0-21swapper/312:38:373
634099151,11cyclictest0-21swapper/309:20:123
6340991510,3cyclictest0-21swapper/312:20:133
634099150,12cyclictest0-21swapper/311:45:233
634099150,11cyclictest0-21swapper/308:50:203
634099150,10cyclictest97800-21chrt08:34:543
634099150,10cyclictest314611-21taskset12:00:133
3345882151,9sleep0330580-21/usr/sbin/munin12:15:220
3195802151,9sleep1319969-21latency_hist12:05:011
3093392151,10sleep0309555-21/usr/sbin/munin11:55:140
634099141,8cyclictest292590-21taskset11:38:053
407482141,9sleep040815-21tune2fs07:40:130
2339572141,2sleep00-21swapper/010:40:230
1230062141,8sleep1123060-21ntpq08:55:221
471222131,7sleep248811-21kthreadcore07:45:172
3556912130,8sleep2355724-21ntpq12:35:202
3505762131,8sleep1350690-21awk12:30:221
3505762131,8sleep1350690-21awk12:30:221
1879732131,7sleep0187685-21/usr/sbin/munin10:00:140
895012120,8sleep291275-21cut08:25:172
601652124,3sleep060212-21ntpq07:55:210
338142121,7sleep233889-21/usr/sbin/munin07:30:262
3078432121,6sleep2307896-21ntp_states11:50:232
2877982121,6sleep2287817-21apt-get11:35:112
247892125,3sleep124980-21kthreadcore07:25:161
247892125,3sleep124980-21kthreadcore07:25:161
2144852121,4sleep1214826-21cat10:25:171
2087152121,7sleep0208915-21cat10:20:110
1346442120,1sleep20-21swapper/209:09:292
931422111,5sleep093245-21perl08:30:160
811022111,5sleep181151-21ntpq08:15:221
677432111,6sleep268705-21kthreadcore08:05:172
334852111,6sleep033526-21ntpq07:30:190
3077362111,5sleep0307793-21ntpq11:50:210
2868542111,6sleep2287107-21unixbench_multi11:30:272
2868542111,6sleep2287107-21unixbench_multi11:30:272
1926582111,5sleep1192677-21munin-run10:05:011
1830212111,6sleep2183517-21kthreadcore09:55:162
1787122110,6sleep1180427-21kthreadcore09:50:181
1670242111,5sleep1167234-21hddtemp_smartct09:40:161
1599182111,5sleep1159968-21ntpq09:30:221
1575382111,7sleep2159274-21tr09:30:192
92722101,4sleep210374-21memory07:10:172
876212101,4sleep087640-21apt-get08:25:110
779432104,3sleep079561-21kthreadcore08:15:170
718972101,4sleep171916-21apt-get08:10:131
654282101,6sleep265476-21ntpq08:00:192
3465362101,4sleep0346739-21diskstats12:30:130
3465362101,4sleep0346739-21diskstats12:30:120
3402112105,2sleep00-21swapper/012:20:260
3358762101,5sleep2336129-21df12:20:132
3358542101,3sleep10-21swapper/112:20:121
3308782100,7sleep2331088-21/usr/sbin/munin12:15:162
3262412101,4sleep0327879-21cut12:10:160
2777452101,4sleep0277843-21irqstats11:25:160
2770142101,5sleep0277032-21munin-run11:24:590
2707132101,4sleep1270762-21ntpq11:15:191
2404782101,5sleep2240634-21cstates10:50:112
2034102101,4sleep2203429-21apt-get10:15:122
1851862101,4sleep0186243-21memory09:55:180
1760082101,4sleep1176057-21ntpq09:45:211
1598242101,5sleep0159912-21ntpq09:30:210
1389892101,4sleep1139045-21ntp_states09:10:221
1339952102,4sleep1134002-21sensors_temp09:05:241
1085822101,4sleep0108601-21apt-get08:45:100
105162101,4sleep010564-21ntpq07:10:200
86387290,3sleep286430-21ntpq08:20:192
72676291,4sleep274300-21cut08:10:172
49845291,4sleep049969-21grep07:45:230
330021290,1sleep10-21swapper/112:11:141
299315291,4sleep0300928-21cut11:45:160
293271291,3sleep0292751-21apt-get11:40:030
266443291,4sleep21-21systemd11:15:002
261541291,3sleep0261744-21diskstats11:10:120
260035291,4sleep2260176-21ntpq11:05:182
250031291,4sleep2250166-21timerwakeupswit10:55:282
234912291,4sleep1235111-21cat10:45:011
229388290,5sleep2229797-21apt-config10:40:002
219848291,4sleep2221454-21cut10:30:192
219848291,4sleep2221454-21cut10:30:182
192941291,4sleep1193117-21cat10:05:131
191655291,4sleep2191702-21ntpq10:00:212
180429291,4sleep2181098-21cat09:50:192
172554291,4sleep0173028-21cut09:45:160
170801291,4sleep0170855-21ntpq09:40:230
145496291,3sleep2145752-21df09:20:132
122955291,4sleep0123004-21ntpq08:55:210
114398290,2sleep0171rcu_preempt08:50:160
109186291,4sleep1110730-21cut08:45:161
108794291,5sleep20-21swapper/208:45:132
93415281,3sleep294122-21cut08:30:182
82388280,2sleep10-21swapper/108:20:111
67010281,3sleep067209-21cat08:05:150
61426280,2sleep10-21swapper/108:00:101
56487281,3sleep256700-21smartctl07:55:152
39819280,1sleep10-21swapper/107:36:511
38273281,3sleep038962-21sed07:35:170
354838281,3sleep0355583-21missed_timers12:35:180
351463281,3sleep2553-21dbus-daemon12:35:002
351463281,3sleep2553-21dbus-daemon12:35:002
345145281,3sleep2345196-21ntpq12:25:182
34235281,4sleep134236-21sshd07:31:371
321146281,3sleep0322796-21kthreadcore12:05:160
319970281,3sleep2319593-21apt-get12:05:022
314029281,3sleep2314542-21gpgv12:00:002
309038281,2sleep1309057-21apt-get11:55:111
277346281,3sleep2277562-21diskstats11:25:142
271021281,3sleep2271082-21sed11:15:242
252864281,3sleep1254632-21grep11:00:201
251026281,4sleep0251154-21tr11:00:140
245732280,2sleep00-21swapper/010:55:140
239258281,3sleep1239311-21ntp_states10:45:211
233785281,2sleep20-21swapper/210:40:212
23056281,3sleep023107-21ld07:20:220
22967281,3sleep222997-21ntpq07:20:192
224389281,2sleep10-21swapper/110:35:111
214827281,3sleep2216455-21kthreadcore10:25:182
198727281,3sleep0200122-21kthreadcore10:10:160
161804280,4sleep20-21swapper/209:35:152
150706281,2sleep00-21swapper/009:25:120
146024281,3sleep1146078-21kthreadcore09:20:161
141420281,3sleep0143108-21kthreadcore09:15:180
124238281,3sleep1124257-21apt-get09:00:111
11992281,3sleep10-21swapper/107:15:021
92234270,1sleep10-21swapper/108:25:281
66660270,2sleep10-21swapper/108:05:121
66277270,1sleep00-21swapper/008:03:310
60961270,1sleep10-21swapper/107:57:431
45772271,2sleep145790-21apt-get07:45:121
40469270,2sleep20-21swapper/207:40:122
356156271,3sleep1356343-21vmstat12:35:261
35036270,1sleep20-21swapper/207:35:072
343266270,4sleep0344960-21/usr/sbin/munin12:25:160
334876271,3sleep10-21swapper/112:15:271
330211270,1sleep20-21swapper/212:13:402
314589270,2sleep20-21swapper/212:00:132
291812270,4sleep00-21swapper/011:35:220
28897271,3sleep228898-21sshd07:25:372
28897271,3sleep228898-21sshd07:25:372
282550270,1sleep00-21swapper/011:30:130
282550270,1sleep00-21swapper/011:30:120
277844271,3sleep10-21swapper/111:25:171
272004270,2sleep10-21swapper/111:20:121
256486271,3sleep1256560-21fschecks_time11:05:131
245421270,1sleep00-21swapper/010:54:260
212760271,3sleep10-21swapper/110:20:191
212524271,3sleep2212649-21netstat10:20:172
204911271,2sleep00-21swapper/010:15:180
200126271,2sleep10-21swapper/110:10:171
197882270,1sleep00-21swapper/010:09:530
195049271,3sleep2196708-21latency10:05:182
181993270,1sleep00-21swapper/009:52:120
166727270,2sleep20-21swapper/209:40:132
161735271,3sleep1161801-21tune2fs09:35:151
144864270,1sleep10-21swapper/109:15:541
139841270,1sleep20-21swapper/209:13:552
139615270,1sleep00-21swapper/009:11:010
13778270,3sleep00-21swapper/007:15:130
130326271,3sleep00-21swapper/009:05:160
128943270,1sleep00-21swapper/009:01:550
117815271,3sleep10-21swapper/108:50:211
103402270,1sleep10-21swapper/108:40:151
98690261,2sleep00-21swapper/008:35:160
87174260,1sleep00-21swapper/008:22:390
81410261,2sleep20-21swapper/208:15:272
76668260,1sleep00-21swapper/008:12:130
55822260,1sleep20-21swapper/207:53:392
55788260,1sleep10-21swapper/107:53:131
51587261,2sleep00-21swapper/007:50:150
325023260,1sleep10-21swapper/112:08:501
316370260,2sleep00-21swapper/012:00:190
307795260,2sleep10-21swapper/111:50:221
297926261,2sleep20-21swapper/211:40:332
295120261,2sleep10-21swapper/111:40:201
28981260,1sleep00-21swapper/007:26:390
28981260,1sleep00-21swapper/007:26:390
286847260,3sleep10-21swapper/111:30:261
286847260,3sleep10-21swapper/111:30:251
271782260,3sleep20-21swapper/211:20:012
271395260,1sleep00-21swapper/011:15:540
265558261,2sleep10-21swapper/111:10:221
250555260,4sleep10-21swapper/110:57:551
240456260,2sleep10-21swapper/110:50:101
240066260,1sleep00-21swapper/010:48:210
239847260,4sleep20-21swapper/210:45:342
222967260,2sleep0223079-21strings10:30:210
222967260,2sleep0223079-21strings10:30:210
202267261,2sleep20-21swapper/210:10:232
176921260,1sleep20-21swapper/209:49:182
17594260,1sleep20-21swapper/207:15:192
162060261,2sleep00-21swapper/009:35:180
150992261,2sleep20-21swapper/209:25:132
149825260,3sleep0150070-21uptime09:20:270
145188260,2sleep20-21swapper/209:20:002
124258260,2sleep20-21swapper/209:00:122
118488262,2sleep2431ktimers/208:51:472
108156261,2sleep0800-21snmpd08:42:550
108027260,4sleep20-21swapper/208:41:172
102466260,2sleep20-21swapper/208:35:272
102351261,2sleep10-21swapper/108:35:261
96670250,1sleep10-21swapper/108:30:211
303396250,1sleep10-21swapper/111:48:381
292495250,3sleep10-21swapper/111:36:501
257352250,2sleep00-21swapper/011:05:160
251294250,1sleep20-21swapper/211:00:172
228161250,1sleep00-21swapper/010:35:180
22788250,1sleep10-21swapper/107:20:191
219157250,1sleep10-21swapper/110:30:141
219157250,1sleep10-21swapper/110:30:131
187339250,1sleep10-21swapper/109:59:161
150731250,2sleep10-21swapper/109:25:131
119291250,2sleep20-21swapper/208:55:152
40463240,1sleep10-21swapper/107:40:061
340964240,1sleep10-21swapper/112:25:011
298838240,1sleep20-21swapper/211:45:132
214277240,1sleep00-21swapper/010:25:160
203700240,1sleep10-21swapper/110:15:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional