You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 09:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Tue Dec 09, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
112660211961,53sleep30-21swapper/319:05:013
112909210387,10sleep10-21swapper/119:05:031
11785628960,19sleep00-21swapper/019:09:510
11759028655,10sleep20-21swapper/219:06:272
118014993531,2cyclictest0-21swapper/323:25:263
118014992724,2cyclictest0-21swapper/300:10:013
3769642260,3sleep134-21ksoftirqd/122:27:481
11801499262,17cyclictest272240-21taskset21:25:123
118014992611,10cyclictest451655-21turbostat23:19:563
5743922250,2sleep30-21swapper/300:30:183
11801499252,15cyclictest397950-21taskset22:40:173
118014992521,2cyclictest0-21swapper/322:50:283
118014992419,3cyclictest0-21swapper/322:55:153
118014992414,5cyclictest3696762sleep322:24:103
3041362231,10sleep2247079-21diskmemload21:44:122
11801499232,15cyclictest549551-21taskset00:15:143
5646632221,12sleep2564668-21ssh00:24:282
4995092221,2sleep20-21swapper/223:43:112
11801499222,15cyclictest572661-21sleep00:30:003
11801499222,13cyclictest500647-21taskset23:44:563
11801499222,13cyclictest321623-21taskset21:55:113
118014992219,1cyclictest0-21swapper/321:10:203
118014992218,2cyclictest0-21swapper/321:10:003
118014992216,2cyclictest0-21swapper/323:00:163
11801499221,16cyclictest333994-21chrt22:00:203
4020492210,1sleep20-21swapper/222:43:582
11801499212,13cyclictest294775-21taskset21:37:273
11801499212,12cyclictest522502-21taskset23:56:263
11801499212,12cyclictest448906-21taskset23:10:403
118014992118,1cyclictest0-21swapper/321:15:143
118014992117,2cyclictest0-21swapper/321:40:263
118014992117,2cyclictest0-21swapper/319:15:163
11801499202,12cyclictest546298-21taskset00:10:373
118014992017,1cyclictest0-21swapper/322:35:143
118014992017,1cyclictest0-21swapper/321:20:213
118014992016,2cyclictest0-21swapper/323:05:183
11801499201,2cyclictest0-21swapper/322:07:273
118014992012,4cyclictest3592382sleep322:15:363
11801499201,14cyclictest383663-21chrt22:30:353
11801499201,14cyclictest0-21swapper/319:20:123
11801499201,13cyclictest285977-21taskset21:31:343
4349972191,10sleep20-21swapper/223:04:222
2953132191,2sleep00-21swapper/021:38:170
11801499192,13cyclictest311318-21taskset21:47:413
11801499192,12cyclictest351472-21taskset22:11:063
118014991916,1cyclictest0-21swapper/320:50:263
118014991916,1cyclictest0-21swapper/320:45:183
118014991915,2cyclictest376600-21taskset22:27:133
11801499191,14cyclictest0-21swapper/322:46:523
5866062181,2sleep10-21swapper/100:37:341
2927852181,6sleep2247079-21diskmemload21:35:222
1561042181,11sleep2156161-21cut19:40:232
1361092181,11sleep2136361-21df19:25:122
11801499182,12cyclictest586653-21taskset00:37:393
11801499182,12cyclictest241776-21turbostat21:05:003
118014991815,1cyclictest0-21swapper/319:10:243
118014991815,1cyclictest0-21swapper/300:00:143
118014991814,2cyclictest0-21swapper/323:45:043
118014991814,2cyclictest0-21swapper/323:30:163
118014991814,2cyclictest0-21swapper/320:04:173
11801499181,13cyclictest485080-21taskset23:35:133
11801499181,13cyclictest167497-21turbostat19:55:003
11801499181,13cyclictest0-21swapper/323:20:103
11801499181,11cyclictest316467-21taskset21:50:183
11801499181,11cyclictest213730-21taskset20:35:183
11801499180,14cyclictest0-21swapper/323:52:373
5790732171,3sleep10-21swapper/100:32:491
5550452171,2sleep00-21swapper/000:16:290
5468772171,2sleep00-21swapper/000:11:310
5005692171,3sleep00-21swapper/023:44:500
4438752171,3sleep00-21swapper/023:10:100
4259602171,2sleep10-21swapper/122:58:001
4010492171,3sleep00-21swapper/022:42:190
3937462171,3sleep00-21swapper/022:38:390
3928112171,4sleep1247079-21diskmemload22:37:111
3503282171,5sleep1350379-21ntpq22:10:211
3445472171,4sleep1247079-21diskmemload22:08:051
3266102171,4sleep1247079-21diskmemload21:55:371
3265992171,3sleep00-21swapper/021:55:370
3114092171,4sleep1247079-21diskmemload21:47:511
2549202171,12sleep2254930-21ssh21:14:422
11801499172,11cyclictest220682-21sleep20:45:003
118014991714,1cyclictest0-21swapper/320:30:213
118014991714,1cyclictest0-21swapper/319:39:593
118014991714,1cyclictest0-21swapper/319:25:153
11801499171,13cyclictest564174-21chrt00:23:363
11801499171,13cyclictest0-21swapper/320:15:263
5695752161,11sleep2569703-21rm00:25:252
5640722161,3sleep1247079-21diskmemload00:23:271
5547092160,3sleep20-21swapper/200:15:582
5050642161,10sleep2505212-21ntpq23:45:202
4763522161,2sleep00-21swapper/023:30:020
4641452163,8sleep0464278-21ntpq23:20:200
4247162161,4sleep0247079-21diskmemload22:56:030
4102012161,3sleep20-21swapper/222:49:032
3524262161,11sleep2352436-21ssh22:12:352
2877622161,3sleep20-21swapper/221:34:182
2797892161,4sleep2247079-21diskmemload21:29:412
2630952161,2sleep00-21swapper/021:19:410
118014991614,1cyclictest0-21swapper/320:55:203
118014991614,1cyclictest0-21swapper/320:55:203
11801499161,13cyclictest0-21swapper/319:55:173
11801499161,10cyclictest156965-21chrt19:44:243
11801499160,13cyclictest0-21swapper/319:45:203
5778122151,2sleep20-21swapper/200:30:342
5234142151,2sleep10-21swapper/123:57:531
5163582151,8sleep2247079-21diskmemload23:54:262
5140822150,1sleep10-21swapper/123:50:481
5135492151,2sleep00-21swapper/023:50:270
4922802151,2sleep10-21swapper/123:39:371
4912202150,1sleep20-21swapper/223:37:582
4562542151,9sleep1247079-21diskmemload23:15:241
4499762151,3sleep1247079-21diskmemload23:12:231
4428252151,2sleep20-21swapper/223:08:552
3782472151,10sleep2378259-21cat22:29:552
3688902151,2sleep10-21swapper/122:22:561
3204882150,1sleep20-21swapper/221:54:192
2480392151,9sleep0248491-21grep21:10:190
11801499159,3cyclictest199504-21turbostat20:25:013
11801499151,9cyclictest183089-21taskset20:06:483
11801499151,9cyclictest145750-21taskset19:30:253
11801499151,11cyclictest204802-21turbostat20:30:013
4424852141,7sleep1247079-21diskmemload23:08:221
4328872141,9sleep1432895-21ssh23:01:051
3203422141,8sleep1320355-21ssh21:54:061
2693612141,8sleep2247079-21diskmemload21:21:392
1983752141,4sleep2198456-21ntpq20:20:192
11801499141,9cyclictest188641-21turbostat20:15:003
5222392131,8sleep2522250-21sh23:56:022
5077302131,7sleep1507740-21ssh23:48:231
5069562131,6sleep00-21swapper/023:47:060
4829392131,8sleep2482949-21ssh23:32:492
4817312131,6sleep10-21swapper/123:31:031
4727112131,7sleep2247079-21diskmemload23:25:272
4349752131,8sleep0247079-21diskmemload23:04:200
3580362131,7sleep2358337-21ssh22:15:192
3534922130,1sleep00-21swapper/022:14:170
3362162131,3sleep0336228-21ssh22:02:470
3339362131,5sleep2333992-21ntpq22:00:202
2596492131,3sleep1259704-21ssh21:15:211
1293082131,8sleep1125432-21/usr/sbin/munin19:15:211
1187422131,7sleep0120141-21kthreadcore19:10:190
5791272121,4sleep0247079-21diskmemload00:32:560
4973782121,5sleep10-21swapper/123:40:261
4659572121,5sleep2247079-21diskmemload23:21:582
4655312121,4sleep1247079-21diskmemload23:21:171
4577142121,4sleep0247079-21diskmemload23:16:510
4485062121,5sleep20-21swapper/223:10:282
4271422121,7sleep2427148-21sh22:59:492
4182632121,4sleep2247079-21diskmemload22:53:362
4001162121,5sleep10-21swapper/122:40:521
3855302121,5sleep00-21swapper/022:33:290
3855082121,4sleep2247079-21diskmemload22:33:272
3787002122,5sleep1378715-21apt-get22:30:141
3774572121,4sleep0247079-21diskmemload22:28:360
3695422121,4sleep2247079-21diskmemload22:23:572
3596362121,4sleep1247079-21diskmemload22:16:151
3445342121,5sleep00-21swapper/022:08:040
2947262120,5sleep10-21swapper/121:37:221
2721882121,7sleep0247079-21diskmemload21:25:100
2689452121,4sleep1247079-21diskmemload21:21:021
2166092121,7sleep0218324-21cut20:40:170
1625162125,3sleep1162584-21cpuspeed_turbos19:50:121
1259502121,7sleep2126212-21kthreadcore19:15:172
5869202111,3sleep00-21swapper/000:38:090
5859812111,4sleep2247079-21diskmemload00:36:272
5705622111,3sleep1247079-21diskmemload00:26:201
5653962111,5sleep0565621-21df_inode00:25:150
5634792111,4sleep0247079-21diskmemload00:22:300
5559262111,4sleep1247079-21diskmemload00:17:501
5462792111,3sleep10-21swapper/100:10:351
5397152111,4sleep2247079-21diskmemload00:08:012
5391092111,4sleep1247079-21diskmemload00:07:031
5322502111,3sleep1247079-21diskmemload00:03:591
5318802111,4sleep2247079-21diskmemload00:03:242
5293222111,3sleep00-21swapper/000:00:210
4726682111,3sleep10-21swapper/123:25:261
4583292111,3sleep20-21swapper/223:17:492
4413732111,4sleep0247079-21diskmemload23:06:370
4167482111,4sleep0247079-21diskmemload22:51:090
4031172111,3sleep1247079-21diskmemload22:45:131
4030832111,5sleep0403107-21apt-get22:45:120
3921152111,4sleep2247079-21diskmemload22:36:042
3687922111,6sleep0368802-21ssh22:22:470
3587572111,3sleep00-21swapper/022:15:250
3381042111,4sleep2338333-21df_inode22:05:122
3185182111,3sleep00-21swapper/021:51:160
3107032114,1sleep2247079-21diskmemload21:46:452
3019442111,4sleep1247079-21diskmemload21:40:511
2983242111,6sleep0299984-21kthreadcore21:40:180
2880152111,3sleep00-21swapper/021:34:400
2856302111,4sleep10-21swapper/121:31:041
2636322111,3sleep00-21swapper/021:20:050
2547292111,4sleep1247079-21diskmemload21:14:251
2439622111,6sleep1245793-21kthreadcore21:05:191
2206532111,4sleep2220671-21munin-run20:45:002
1525632114,3sleep0153471-21cat19:40:170
1344022111,2sleep10-21swapper/119:20:181
5462872101,3sleep20-21swapper/200:10:372
5332172101,3sleep0247079-21diskmemload00:05:060
5220542100,3sleep0247079-21diskmemload23:55:440
4888332101,5sleep0488897-21ld23:35:190
4681592101,3sleep0247079-21diskmemload23:25:050
4170782101,3sleep10-21swapper/122:51:431
3353892101,2sleep10-21swapper/122:01:311
3221222101,4sleep2322199-21irqstats21:55:152
3104282101,3sleep0247079-21diskmemload21:46:190
2777872101,3sleep1247079-21diskmemload21:26:301
2605802100,1sleep20-21swapper/221:15:462
2423122101,4sleep2242521-21hddtemp_smartct21:05:162
2354632101,4sleep1235509-21ntpq20:55:251
2354632101,4sleep1235509-21ntpq20:55:251
2300852101,4sleep1230136-21ntpq20:50:201
1783132101,4sleep0178332-21apt-get20:05:140
1770122100,1sleep00-21swapper/020:00:200
1625872101,4sleep2162781-21diskstats19:50:132
1624942101,4sleep0162513-21apt-get19:50:110
1577692104,2sleep10-21swapper/119:45:151
1454282101,4sleep2145474-21ntpq19:30:212
1360872101,3sleep10-21swapper/119:25:111
242045291,4sleep0242174-21cut21:05:140
237347291,4sleep2239008-21cut21:00:192
235410291,3sleep0235461-21ntpq20:55:240
235410291,3sleep0235461-21ntpq20:55:240
218326291,4sleep1219494-21memory20:40:171
195927291,4sleep0197696-21grep20:20:170
189934291,4sleep1191667-21kthreadcore20:15:171
169715291,4sleep2171520-21grep19:55:182
153473291,4sleep1155238-21grep19:40:181
145758291,4sleep0145980-21unixbench_multi19:30:270
130832291,3sleep0131035-21diskstats19:20:130
206023281,3sleep2207699-21kthreadcore20:30:172
204070281,3sleep0802-21snmpd20:25:260
201273281,3sleep1203060-21tr20:25:191
193888281,4sleep0194301-21latency_hist20:20:010
183603280,3sleep1183622-21apt-get20:10:121
181405281,3sleep2182219-21/usr/sbin/munin20:05:202
150394281,3sleep20-21swapper/219:35:182
134964280,1sleep20-21swapper/219:20:232
236728270,2sleep10-21swapper/121:00:141
231570271,3sleep2231394-21/usr/sbin/munin20:55:182
231570271,3sleep2231394-21/usr/sbin/munin20:55:172
230928270,1sleep20-21swapper/220:52:292
229960270,1sleep00-21swapper/020:50:190
210917271,2sleep1210350-21/usr/sbin/munin20:35:171
205052270,2sleep10-21swapper/120:30:111
194304271,2sleep1193900-21apt-get20:20:011
188579270,1sleep20-21swapper/220:14:342
183597270,1sleep00-21swapper/020:10:070
168316270,3sleep1169714-21cut19:55:171
161238271,3sleep2765-21ntpd19:45:202
140789270,1sleep00-21swapper/019:25:300
129776271,2sleep0125432-21/usr/sbin/munin19:15:280
211937260,2sleep20-21swapper/220:35:182
204574260,4sleep20-21swapper/220:27:162
177733260,1sleep20-21swapper/220:01:032
177722260,1sleep10-21swapper/120:00:541
148582261,2sleep10-21swapper/119:35:171
147273261,2sleep00-21swapper/019:35:160
224994250,2sleep20-21swapper/220:45:242
220937251,1sleep00-21swapper/020:45:140
215124250,3sleep00-21swapper/020:37:040
189152250,2sleep20-21swapper/220:15:142
182387250,2sleep10-21swapper/120:05:231
167744250,1sleep00-21swapper/019:55:120
161920250,1sleep00-21swapper/019:46:020
145286250,1sleep10-21swapper/119:30:201
118456250,1sleep20-21swapper/219:10:172
118302250,1sleep10-21swapper/119:10:161
236444240,1sleep00-21swapper/021:00:010
220912240,2sleep10-21swapper/120:45:111
205377240,1sleep00-21swapper/020:30:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional