You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-12 - 14:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sat Sep 12, 2026 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
115372154815,1515sleep111538-21ntp_states19:24:161
110422127116,1243sleep018277-1kworker/0:0H19:23:390
7201295610,934sleep27238-21ntpq20:53:522
993628218,795sleep010003-21df21:03:230
17724266018,624sleep017794-21/usr/sbin/munin21:28:220
17724266018,624sleep017794-21/usr/sbin/munin21:28:220
250992491131,219sleep225179-21fschecks_time20:08:262
1404824060,13sleep30-21swapper/322:58:273
21536238719,354sleep321537-21sshd19:57:253
9967236995,254sleep310053-21users22:43:383
9967236995,254sleep310053-21users22:43:383
157792311139,21sleep20-21swapper/219:38:262
7979230717,275sleep18023-21ntpq22:38:321
7979230717,275sleep18023-21ntpq22:38:321
16381228411,259sleep0367-21snmpd23:04:470
240272281126,136sleep224117-21cut21:48:272
240272281126,136sleep224117-21cut21:48:272
2877227951,206sleep22880-21cron00:08:062
29815227737,25sleep20-21swapper/220:23:282
29815227737,25sleep20-21swapper/220:23:282
1440822719,201sleep214555-21latency_hist21:18:112
13769223324,187sleep313914-21grep21:13:393
19163220230,151sleep219327-21grep23:13:412
31461219553,31sleep20-21swapper/223:53:372
31461219553,31sleep20-21swapper/223:53:372
1257821930,73sleep25503-21cyclictest22:53:292
18337217910,28sleep218451-21timerandwakeup21:28:362
18337217910,28sleep218451-21timerandwakeup21:28:362
27265217213,140sleep026765-21apt-get21:58:140
27265217213,140sleep026765-21apt-get21:58:140
24148216921,123sleep124295-21grep20:03:411
24148216921,123sleep124295-21grep20:03:411
26471216814,135sleep20-21swapper/223:38:312
1931521637,18sleep219343-21df21:33:252
1883216114,121sleep20-21swapper/220:38:182
14391216019,28sleep10-21swapper/121:18:101
21880215912,133sleep221898-21uname23:23:302
47162158114,32sleep24725-21modprobe20:48:072
14825215418,108sleep314974-21unixbench_singl19:33:393
13137215428,112sleep213215-21switchtime19:28:372
13137215428,112sleep213215-21switchtime19:28:372
345421539,130sleep23395-21apt-get00:08:552
345421539,130sleep23395-21apt-get00:08:552
2591521535,12sleep20-21swapper/223:37:262
121221536,13sleep20-21swapper/222:18:102
2945214721,101sleep03059-21latency_hist00:08:080
1101621439,83sleep211161-21latency_hist00:33:092
1101621439,83sleep211161-21latency_hist00:33:092
1789821389,14sleep20-21swapper/223:09:252
8915213359,59sleep20-21swapper/219:18:082
14196213212,103sleep20-21swapper/222:58:312
1664421319,67sleep21-21systemd23:08:082
891721249,98sleep20-21swapper/220:58:372
15363212426,74sleep015505-21unixbench_singl21:18:370
15363212426,74sleep015505-21unixbench_singl21:18:370
2337521207,64sleep20-21swapper/221:44:352
2652121196,67sleep20-21swapper/220:13:112
2652121196,67sleep20-21swapper/220:13:112
2783321189,54sleep20-21swapper/223:43:242
32416211744,16sleep20-21swapper/222:13:342
23702211714,78sleep120536-21kworker/u8:221:48:091
1724121166,67sleep20-21swapper/219:43:232
987621159,53sleep20-21swapper/219:18:402
5814211548,13sleep20-21swapper/200:13:402
5814211548,13sleep20-21swapper/200:13:402
26744211513,84sleep126801-21sendmail-msp21:58:071
2113821155,63sleep20-21swapper/223:20:432
1557521156,63sleep20-21swapper/221:19:082
1557521156,63sleep20-21swapper/221:19:082
3254221148,57sleep232544-21munin-run20:33:092
903021137,95sleep20-21swapper/200:23:492
3053221137,60sleep20-21swapper/223:51:432
2213421137,50sleep20-21swapper/219:58:292
2133821127,63sleep20-21swapper/219:54:502
522921118,46sleep20-21swapper/222:28:402
17240211126,58sleep117234-21apt-get19:43:221
2736021107,14sleep10-21swapper/120:13:381
2736021107,14sleep10-21swapper/120:13:381
5669210817,74sleep15811-21unixbench_multi00:13:381
5669210817,74sleep15811-21unixbench_multi00:13:381
3486210820,73sleep03590-21timerandwakeup22:23:370
1893210812,77sleep11985-21/usr/sbin/munin00:03:291
6771210715,73sleep06293-21apt-get20:53:170
32515210745,45sleep20-21swapper/223:58:212
8332210622,64sleep08477-21unixbench_singl22:38:390
8332210622,64sleep08477-21unixbench_singl22:38:390
26980210615,75sleep027183-21latency_hist21:58:100
14188210615,75sleep114194-21df_inode19:33:251
30691210514,74sleep330661-21latency_hist23:53:093
775210442,13sleep20-21swapper/220:33:332
1085321046,50sleep20-21swapper/200:32:212
6358210353,27sleep20-21swapper/222:33:322
4668210316,70sleep14670-21cron00:13:061
4668210316,70sleep14670-21cron00:13:061
29090210318,69sleep129121-21munin-plugin-st23:48:071
2783421038,12sleep30-21swapper/323:43:253
23549210210,23sleep00-21swapper/020:03:280
23549210210,23sleep00-21swapper/020:03:280
31197210114,72sleep331228-21latency23:53:303
31197210114,72sleep331228-21latency23:53:303
15648210021,59sleep315357-21/usr/sbin/munin23:03:283
11544210011,77sleep211606-21uname19:24:172
818129912,73sleep08253-21sensors_temp19:13:380
557329913,71sleep05668-21sed00:13:370
557329913,71sleep05668-21sed00:13:370
3218629822,52sleep032223-21sendmail23:58:070
3218629822,52sleep032223-21sendmail23:58:070
294942980,13sleep10-21swapper/122:03:391
1981829813,72sleep119851-21latency_hist23:18:081
970629722,56sleep39857-21latency_hist21:03:103
736729712,58sleep30-21swapper/320:53:573
324492978,77sleep018277-1kworker/0:0H20:32:300
243712975,12sleep20-21swapper/223:32:302
219392975,20sleep20-21swapper/221:41:002
2161129728,40sleep321603-21apt-get23:23:233
1262329720,56sleep112767-21apt-key00:38:091
1262329720,56sleep112767-21apt-key00:38:091
2604629536,38sleep00-21swapper/023:38:090
1149429528,42sleep111491-21apt-get21:08:211
2941629465,15sleep20-21swapper/220:23:082
2941629465,15sleep20-21swapper/220:23:082
1924529424,31sleep10-21swapper/121:33:201
67912935,11sleep20-21swapper/219:10:492
67912935,11sleep20-21swapper/219:10:492
341429310,70sleep33395-21apt-get00:08:283
341429310,70sleep33395-21apt-get00:08:283
2137329316,57sleep121422-21latency_hist23:23:081
234372926,73sleep20-21swapper/220:03:172
234372926,73sleep20-21swapper/220:03:172
1716629211,66sleep0367-21snmpd23:08:300
1061629222,41sleep310766-21tail21:03:383
1169529164,16sleep2367-21snmpd22:49:432
912728952,19sleep20-21swapper/222:43:172
912728952,19sleep20-21swapper/222:43:172
2239228911,61sleep322436-21cpu21:43:233
662928831,42sleep20-21swapper/200:18:242
2680528851,22sleep226838-21munin-run21:58:082
2211728815,59sleep122205-21basename23:23:371
1259528831,11sleep20-21swapper/221:11:042
101128819,49sleep31142-21/usr/sbin/munin20:33:383
3166628729,42sleep331683-21modprobe22:13:063
2984628630,44sleep20-21swapper/222:04:382
2727328633,41sleep20-21swapper/221:58:202
2727328633,41sleep20-21swapper/221:58:202
1528728631,10sleep20-21swapper/219:37:012
663628519,32sleep30-21swapper/300:18:253
42572856,11sleep30-21swapper/319:03:283
1740028531,12sleep20-21swapper/221:27:392
1740028531,12sleep20-21swapper/221:27:392
624928432,41sleep20-21swapper/220:52:472
408128433,40sleep20-21swapper/222:27:232
3166028433,40sleep20-21swapper/222:13:012
187852849,16sleep20-21swapper/219:48:202
2546728311,17sleep10-21swapper/120:08:331
190828316,34sleep10-21swapper/120:38:211
853828229,42sleep20-21swapper/222:38:532
853828229,42sleep20-21swapper/222:38:532
819228227,31sleep00-21swapper/000:23:270
779428212,55sleep37861-21sort19:13:283
145928217,25sleep30-21swapper/322:18:273
1106028229,41sleep20-21swapper/221:06:242
510228115,51sleep15228-21sed22:28:391
1064428117,47sleep010799-21sed19:23:100
592328010,53sleep15981-21/usr/sbin/munin19:08:301
592328010,53sleep15981-21/usr/sbin/munin19:08:301
1362828016,47sleep113782-21latency_hist22:58:101
606027920,43sleep36068-21diskstats22:33:253
3222927918,42sleep132234-40mandb23:58:081
3222927918,42sleep132234-40mandb23:58:081
1845227940,23sleep318592-21sed21:28:373
1845227940,23sleep318592-21sed21:28:373
1055027921,41sleep310630-21munin-run19:23:093
745627819,29sleep10-21swapper/100:18:391
476927827,27sleep00-21swapper/020:48:090
469027817,29sleep00-21swapper/019:03:370
288327822,35sleep31-21systemd00:08:073
2188527819,45sleep021904-21apt-get19:58:190
2102727810,54sleep321064-21idleruntime21:38:263
476827622,33sleep14911-21timerwakeupswit19:03:381
2049427516,41sleep120310-21/usr/sbin/munin19:53:271
1444327514,42sleep013802-21/usr/sbin/munin22:58:370
1294627415,43sleep112990-21ntpq19:28:321
1294627415,43sleep112990-21ntpq19:28:321
421227221,23sleep20-21swapper/219:03:262
3088627226,28sleep00-21swapper/022:08:350
675527118,29sleep10-21swapper/122:33:391
2685127117,36sleep326973-21cat21:58:093
1584927118,28sleep10-21swapper/123:03:351
641127013,41sleep06605-21apt-config20:53:090
3266627019,22sleep00-21swapper/022:13:400
254212700,6sleep10-21swapper/121:53:091
254212700,6sleep10-21swapper/121:53:091
1616127014,41sleep316225-21ps19:38:363
1078627027,24sleep10-21swapper/122:48:251
1275226923,23sleep30-21swapper/321:13:073
617526819,27sleep00-21swapper/019:08:370
617526819,27sleep00-21swapper/019:08:370
2507426825,24sleep10-21swapper/123:33:301
2030926811,42sleep020328-21apt19:53:180
1524026815,24sleep30-21swapper/321:18:363
1524026815,24sleep30-21swapper/321:18:363
1428826815,25sleep10-21swapper/122:58:341
1279226812,17sleep00-21swapper/022:53:370
2139126721,24sleep00-21swapper/021:38:360
2009726719,20sleep00-21swapper/019:53:100
1900626712,37sleep019085-21/usr/sbin/munin23:13:390
1550726718,33sleep115530-21users21:18:381
1550726718,33sleep115530-21users21:18:381
910226623,21sleep00-21swapper/020:58:390
655426612,22sleep00-21swapper/000:18:220
2944926615,29sleep30-21swapper/320:23:093
2944926615,29sleep30-21swapper/320:23:093
2286126618,20sleep00-21swapper/021:43:360
141426610,12sleep10-21swapper/122:18:251
3220026514,36sleep132202-21sshd22:13:251
2939926520,24sleep10-21swapper/120:23:071
2939926520,24sleep10-21swapper/120:23:071
12452654,48sleep00-21swapper/000:01:590
923026418,31sleep19128-21/usr/sbin/munin22:43:241
923026418,31sleep19128-21/usr/sbin/munin22:43:241
806426415,33sleep18093-21awk20:58:071
3218626417,31sleep332183-21apt-get22:13:193
3164126418,27sleep10-21swapper/120:28:321
2523826418,25sleep30-21swapper/321:53:083
2523826418,25sleep30-21swapper/321:53:083
2161326411,21sleep00-21swapper/023:23:240
1743426433,16sleep10-21swapper/121:28:071
1743426433,16sleep10-21swapper/121:28:071
1117626421,29sleep311195-21apt-get00:33:193
1117626421,29sleep311195-21apt-get00:33:193
632226318,27sleep36350-21munin-run20:53:083
308926313,35sleep33145-21interrupts22:23:283
2640726320,20sleep00-21swapper/023:38:290
48482625,17sleep10-21swapper/120:48:101
3038726214,20sleep10-21swapper/120:23:391
3038726214,20sleep10-21swapper/120:23:391
3014226212,23sleep00-21swapper/020:23:340
3014226212,23sleep00-21swapper/020:23:340
2941626222,18sleep00-21swapper/022:03:390
450426119,26sleep34510-21diskstats22:28:253
2790526118,25sleep00-21swapper/020:18:090
2790526118,25sleep00-21swapper/020:18:090
2448726111,37sleep024557-21wc21:48:370
2448726111,37sleep024557-21wc21:48:370
129362616,11sleep00-21swapper/019:28:310
129362616,11sleep00-21swapper/019:28:310
2975926014,25sleep30-21swapper/320:23:253
2975926014,25sleep30-21swapper/320:23:253
296526013,29sleep10-21swapper/122:23:261
2874926017,25sleep028835-21ls20:18:380
2874926017,25sleep028835-21ls20:18:380
2856926013,32sleep128617-21systemd-run22:03:071
2856926013,32sleep128617-21systemd-run22:03:071
1987926013,28sleep019977-21sed21:33:390
140626017,25sleep01413-21sshd22:18:240
1033826021,23sleep10-21swapper/121:03:321
838125914,28sleep10-21swapper/120:58:231
3177525920,22sleep10-21swapper/122:13:091
2365825918,25sleep323737-21awk23:28:353
2290625910,37sleep10-21swapper/119:59:371
1935925914,32sleep119428-21switchtime19:48:351
1748625918,20sleep30-21swapper/321:28:093
1748625918,20sleep30-21swapper/321:28:093
1629925912,33sleep116391-21cat19:38:381
1609625917,25sleep016143-21ntpq19:38:330
138225919,16sleep10-21swapper/100:03:091
1070025916,26sleep010694-21apt-get22:48:200
2758625815,25sleep30-21swapper/323:43:093
201362589,22sleep00-21swapper/023:18:230
1709925817,18sleep30-21swapper/323:08:293
1312325822,20sleep30-21swapper/322:53:403
1016525816,24sleep310255-21sshd00:28:343
973725723,23sleep19628-21/usr/sbin/munin00:28:241
601425714,24sleep36080-21memory19:08:323
601425714,24sleep36080-21memory19:08:323
374925717,21sleep00-21swapper/000:09:210
374925717,21sleep00-21swapper/000:09:210
2937725717,23sleep129371-21apt-get23:48:191
2631825721,19sleep00-21swapper/020:13:080
2145925717,17sleep10-21swapper/121:38:371
1845125718,15sleep30-21swapper/323:13:173
141512575,11sleep00-21swapper/021:16:030
1232925714,17sleep30-21swapper/319:28:103
923125615,24sleep19226-21apt-get19:18:231
764625614,25sleep17640-21apt-get19:13:221
451325612,29sleep04526-21/usr/sbin/munin22:28:260
965225515,22sleep00-21swapper/000:28:190
598425513,25sleep05978-21apt-get22:33:220
472825517,24sleep34767-21munin-run20:48:083
44925514,22sleep0540-21perl20:33:270
345625515,24sleep13451-21apt-get20:43:211
2218025516,18sleep30-21swapper/319:58:303
1855025513,19sleep00-21swapper/019:48:090
856025414,24sleep38566-21ntp_states00:23:343
1955625411,19sleep30-21swapper/319:48:373
1738725415,23sleep117393-21ntp_states23:08:351
1615325415,17sleep00-21swapper/021:23:210
1615325415,17sleep00-21swapper/021:23:210
1448525413,25sleep014518-21/usr/sbin/munin19:33:320
1196025414,25sleep30-21swapper/322:53:073
103932545,37sleep00-21swapper/022:47:510
103932545,37sleep00-21swapper/022:47:510
181382539,17sleep10-21swapper/121:28:311
181382539,17sleep10-21swapper/121:28:311
3251625210,25sleep332583-21df23:58:223
2644825212,17sleep10-21swapper/123:38:301
244625214,17sleep00-21swapper/020:38:360
2346425213,22sleep323460-21apt-get20:03:223
2346425213,22sleep323460-21apt-get20:03:223
1369325217,17sleep30-21swapper/319:31:253
1369325217,17sleep30-21swapper/319:31:253
39725110,25sleep1403-21df_inode20:33:241
316825115,21sleep33198-21munin-run20:43:073
2937625115,19sleep00-21swapper/023:48:180
2730325112,16sleep30-21swapper/321:58:253
2730325112,16sleep30-21swapper/321:58:253
2478425110,17sleep30-21swapper/323:33:243
2315925111,23sleep123153-21apt-get23:28:211
1221325111,23sleep012359-21sed21:08:400
2752925014,17sleep30-21swapper/320:14:253
2752925014,17sleep30-21swapper/320:14:253
213212509,16sleep30-21swapper/323:23:063
87812495,31sleep30-21swapper/322:42:013
87812495,31sleep30-21swapper/322:42:013
2783124911,21sleep027827-21apt-get23:43:220
2590824916,16sleep30-21swapper/320:08:393
2359424913,20sleep023613-21ntp_states23:28:320
1968124913,21sleep319723-21ntp_states21:33:343
631624710,15sleep30-21swapper/300:18:093
631624710,15sleep30-21swapper/300:18:093
1761924714,15sleep30-21swapper/319:43:323
828324610,16sleep10-21swapper/100:23:281
3123824613,21sleep30-21swapper/320:28:223
3017324615,15sleep30-21swapper/322:08:103
1120424511,16sleep00-21swapper/000:33:240
1120424511,16sleep00-21swapper/000:33:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional