You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 01:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Apr 15, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6193225585,2538sleep024338-21kworker/u8:009:26:330
147832131513,1285sleep214776-21apt-get08:11:332
5420287320,823sleep226028-21kworker/u8:207:41:292
17263265119,616sleep217276-21/usr/sbin/munin08:16:502
31324257436,445sleep00-21swapper/012:28:270
13693243016,398sleep113697-21ntpq08:06:441
20191242126,381sleep020309-21latency_hist10:11:190
3337235620,282sleep03401-21sendmail_mailst09:16:440
3337235620,282sleep03401-21sendmail_mailst09:16:440
2102235215,287sleep02155-21fschecks_count10:56:370
2102235215,287sleep02155-21fschecks_count10:56:370
9770233331,264sleep09777-21ntpq09:36:430
2977423126,11sleep00-21swapper/012:23:270
17417230345,178sleep00-21swapper/010:01:340
27780227410,246sleep227896-21munin-node07:11:212
29974225316,151sleep00-21swapper/010:41:390
21519223251,124sleep00-21swapper/008:31:380
643221219,149sleep00-21swapper/010:51:370
22247220447,141sleep022269-21latency_hist12:01:180
14978219244,127sleep014172-21/usr/sbin/munin09:51:490
666821899,140sleep06779-21latency_hist11:11:190
16630218728,122sleep016655-21awk11:41:400
11313218428,99sleep00-21swapper/009:42:490
29501217016,138sleep229551-21unin-run10:41:172
29501217016,138sleep229551-21unin-run10:41:172
7002216910,122sleep07044-21grep07:46:320
73321669,117sleep0881-21latency_hist12:36:180
2797721646,119sleep028054-21sensors_temp07:11:230
25051216428,101sleep025176-21cat08:41:480
25051216428,101sleep025176-21cat08:41:480
17921216411,93sleep00-21swapper/011:46:360
19755216116,12sleep00-21swapper/011:51:410
1945215613,131sleep12064-21cat09:11:491
13231215257,65sleep00-21swapper/008:06:310
32702215035,101sleep032706-21modprobe07:26:170
31012214712,101sleep031049-21ntp_kernel_pll_09:01:420
31012214712,101sleep031049-21ntp_kernel_pll_09:01:420
27136214632,74sleep013441-21snmpd10:31:460
27136214632,74sleep013441-21snmpd10:31:460
1319721467,99sleep013268-21wc09:46:480
27916214527,80sleep028064-21sed12:16:490
2641621439,97sleep026441-21latency_hist10:31:180
25814214220,13sleep00-21swapper/012:11:400
20165214210,118sleep00-21swapper/008:26:460
980821398,46sleep09874-21modprobe07:56:170
28243213520,70sleep00-21swapper/010:36:310
28243213520,70sleep00-21swapper/010:36:310
4987213418,59sleep04933-21latency_hist07:41:190
17834213414,103sleep017997-21grep08:21:200
17041213331,86sleep317065-21modprobe10:01:173
29669213216,99sleep329769-21timerandwakeup08:56:493
907213016,96sleep3940-21/usr/sbin/munin07:26:403
3803213012,69sleep00-21swapper/011:01:300
3803213012,69sleep00-21swapper/011:01:300
16163212717,68sleep00-21swapper/008:16:200
2268212617,89sleep30-21swapper/307:31:373
1241212511,74sleep00-21swapper/009:11:270
28843212321,82sleep228984-21latency_hist08:56:192
4342212224,71sleep14481-21grep07:36:481
31013212120,82sleep131160-21sed12:26:471
15527212113,87sleep10-21swapper/109:56:181
27126211914,88sleep30-21swapper/310:31:453
27126211914,88sleep30-21swapper/310:31:453
26090211615,85sleep113347-1kworker/1:1H07:06:191
10219211644,23sleep00-21swapper/007:56:370
32113211534,36sleep00-21swapper/010:46:480
18120211411,87sleep217638-21apt-get08:21:262
970421135,73sleep00-21swapper/011:20:400
708211314,83sleep3728-21latency_hist12:36:173
5604211314,57sleep00-21swapper/009:25:320
27569211326,47sleep00-21swapper/008:51:370
2620211246,50sleep00-21swapper/007:31:460
28925211019,75sleep129027-21timerandwakeup07:11:481
53021097,85sleep00-21swapper/007:26:290
5593210817,62sleep35623-21fschecks_time07:41:553
8310210743,17sleep00-21swapper/009:31:460
1206221077,63sleep012103-21ntpq08:01:420
2719121067,85sleep027224-21awk08:51:160
24610210417,75sleep00-21swapper/010:23:350
24610210417,75sleep00-21swapper/010:23:350
5936210326,43sleep05970-21ps07:42:040
23795210330,39sleep023805-21munin-run12:06:190
23335210340,46sleep00-21swapper/008:36:470
21212210129,28sleep10-21swapper/111:56:391
8479210014,13sleep20-21swapper/209:31:482
2307621005,83sleep00-21swapper/010:18:460
22014210015,71sleep322083-21df10:16:313
1495329915,69sleep315012-21basename08:11:383
2644229826,36sleep10-21swapper/110:31:191
2577729815,69sleep325807-21idleruntime12:11:393
181252988,73sleep00-21swapper/008:21:280
1354829829,24sleep00-21swapper/011:31:420
1912229729,44sleep219247-21cat11:51:192
1077329726,48sleep110976-21sed09:41:201
988629622,21sleep20-21swapper/207:56:192
2614929617,62sleep226298-21latency_hist07:06:202
1973029513,67sleep319753-21/usr/sbin/munin11:51:403
3035829422,34sleep00-21swapper/007:16:480
214262946,77sleep00-21swapper/010:12:570
1685829417,43sleep00-21swapper/009:58:560
314152939,16sleep30-21swapper/307:21:273
238329321,26sleep10-21swapper/110:56:421
238329321,26sleep10-21swapper/110:56:421
2895029220,57sleep229001-21interrupts12:21:392
273922926,54sleep00-21swapper/007:08:320
1158729218,34sleep00-21swapper/011:26:290
159592905,53sleep00-21swapper/011:41:070
670928917,55sleep16716-21modprobe07:46:171
1485928935,27sleep30-21swapper/309:51:483
109792898,45sleep013441-21snmpd11:22:090
2979628823,47sleep329770-21/usr/sbin/munin10:41:343
2661528819,50sleep326754-21tail08:46:473
1274428813,52sleep212625-21/usr/sbin/munin09:46:382
3229128713,56sleep232297-21ntp_states12:31:442
2424328719,46sleep324383-21unixbench_multi10:21:483
2424328719,46sleep324383-21unixbench_multi10:21:483
235628728,34sleep10-21swapper/107:31:391
2291528711,62sleep222946-21tune2fs08:36:372
1167728727,38sleep20-21swapper/211:26:352
727428628,47sleep07310-21ntpq11:11:390
727428628,47sleep07310-21ntpq11:11:390
2693928619,44sleep226960-21latency_hist12:16:182
1012528612,35sleep10-21swapper/107:56:301
661428524,40sleep16631-21apt11:11:161
1709328527,30sleep10-21swapper/110:01:191
2977428421,19sleep00-21swapper/008:56:500
2464128324,43sleep30-21swapper/308:41:383
2464128324,43sleep30-21swapper/308:41:383
2100728318,52sleep121043-21cut08:31:191
2093928320,16sleep20-21swapper/211:56:322
1396028324,32sleep30-21swapper/309:51:203
240442826,35sleep00-21swapper/012:06:310
147928218,33sleep30-21swapper/309:11:383
3207328116,48sleep232191-21wc07:21:472
315462818,47sleep30-21swapper/312:31:173
2171028016,44sleep121223-21/usr/sbin/munin08:31:431
1061528013,50sleep210657-21switchtime11:21:472
893127818,27sleep30-21swapper/307:51:443
51527816,46sleep2538-21cut10:51:352
491827822,41sleep2222-21systemd-journal07:41:172
3088827812,48sleep230982-21memory09:01:402
3088827812,48sleep230982-21memory09:01:402
1781627720,44sleep217891-21mailstats10:01:442
16622779,59sleep113347-1kworker/1:1H10:55:501
1108927721,34sleep110584-21apt-get09:41:281
106012770,8sleep30-21swapper/309:41:173
3153727626,30sleep10-21swapper/110:46:371
266682767,54sleep20-21swapper/210:31:322
266682767,54sleep20-21swapper/210:31:322
1597127614,48sleep2281-21dbus-daemon11:41:172
1439527621,36sleep214430-21munin-run11:36:172
1299327625,24sleep10-21swapper/108:06:191
837227515,26sleep10-21swapper/109:31:471
2681227512,44sleep326844-21/usr/sbin/munin07:06:463
1336127515,27sleep20-21swapper/208:06:372
1219627516,44sleep112298-21sed11:26:471
1053127521,30sleep110613-21grep11:21:461
2165627417,27sleep30-21swapper/308:31:413
1645527420,29sleep116511-21if_eth011:41:371
857827318,31sleep30-21swapper/311:16:353
582327320,31sleep26036-21ls09:26:192
3034927310,46sleep330356-21ntp_states07:16:473
184527319,25sleep10-21swapper/107:31:191
825227222,29sleep30-21swapper/311:16:193
825227222,29sleep30-21swapper/311:16:193
2449827212,25sleep10-21swapper/108:41:361
2449827212,25sleep10-21swapper/108:41:361
434327017,23sleep10-21swapper/111:02:441
434327017,23sleep10-21swapper/111:02:441
3217527021,33sleep00-21swapper/009:06:310
3200127018,34sleep332111-21/usr/sbin/munin10:46:473
2780327023,20sleep30-21swapper/312:16:473
1508527022,26sleep30-21swapper/311:36:423
876126922,26sleep20-21swapper/211:16:382
385326923,29sleep33921-21/usr/sbin/munin11:02:053
385326923,29sleep33921-21/usr/sbin/munin11:02:053
3193826919,25sleep10-21swapper/109:06:191
3193826919,25sleep10-21swapper/109:06:191
2382726822,23sleep20-21swapper/212:06:212
1992326816,37sleep119997-21cat11:51:461
1012926826,22sleep30-21swapper/311:21:363
750926719,34sleep27600-21sed11:11:462
750926719,34sleep27600-21sed11:11:462
3217726712,40sleep232245-21df09:06:332
2886326720,22sleep10-21swapper/112:21:381
1554826718,21sleep20-21swapper/209:56:192
1340926724,26sleep10-21swapper/111:31:391
89326617,20sleep20-21swapper/207:26:392
615826610,23sleep20-21swapper/209:26:272
1944726621,30sleep219455-21/usr/sbin/munin11:51:352
173432665,10sleep30-21swapper/311:44:013
141626619,24sleep20-21swapper/209:11:372
2400926520,28sleep20-21swapper/210:21:432
2400926520,28sleep20-21swapper/210:21:432
2198426522,20sleep20-21swapper/210:16:262
2104426516,27sleep20-21swapper/208:31:202
1350926528,21sleep30-21swapper/311:31:413
1109426523,25sleep311087-21apt-get09:41:303
80326422,22sleep30-21swapper/310:51:403
285426423,27sleep32899-21/usr/sbin/munin09:16:313
285426423,27sleep32899-21/usr/sbin/munin09:16:313
2380626415,35sleep123775-21latency_hist12:06:201
2177526418,16sleep30-21swapper/310:16:193
1847726412,34sleep318608-21sed11:46:473
1498526314,21sleep10-21swapper/111:36:401
554226221,25sleep30-21swapper/311:06:383
349626218,26sleep20-21swapper/211:01:202
349626218,26sleep20-21swapper/211:01:202
2290726223,19sleep10-21swapper/108:36:361
188242626,10sleep20-21swapper/210:06:202
971026119,20sleep20-21swapper/209:36:412
744726111,33sleep37449-21cron09:31:173
3265126117,28sleep20-21swapper/210:51:182
3027126112,19sleep10-21swapper/107:16:451
2775226120,21sleep30-21swapper/308:51:403
2404126114,30sleep30-21swapper/312:06:303
1970126119,17sleep30-21swapper/308:26:323
1682926121,26sleep116851-21/usr/sbin/munin08:16:451
572226020,18sleep10-21swapper/109:26:181
2913726015,24sleep10-21swapper/108:56:341
2849926021,17sleep20-21swapper/212:21:192
2174226021,24sleep321840-21users11:56:493
209626019,24sleep32100-21diskstats10:56:363
209626019,24sleep32100-21diskstats10:56:363
1586126022,21sleep20-21swapper/209:56:352
3263025913,31sleep132636-21ntp_states09:06:441
2508325916,25sleep325103-21apt-get10:26:283
1799925917,24sleep118113-21modprobe08:21:211
80432587,9sleep10-21swapper/111:14:351
80432587,9sleep10-21swapper/111:14:351
2464725820,20sleep20-21swapper/212:06:452
2161825822,22sleep221655-21sed08:31:402
1505425815,16sleep10-21swapper/108:11:401
1292325814,27sleep113012-21/usr/sbin/munin09:46:411
606825717,20sleep20-21swapper/211:06:492
3037125721,20sleep20-21swapper/210:41:502
3030325720,19sleep20-21swapper/207:16:462
2748525715,24sleep127555-21df08:51:341
2700825715,20sleep20-21swapper/207:06:492
2521425718,19sleep00-21swapper/007:01:420
2475625715,25sleep324750-21apt-get07:01:293
1896925715,24sleep119059-21fschecks_time10:06:361
119562579,13sleep20-21swapper/208:01:402
2916825615,24sleep329214-21users07:11:503
2873825615,24sleep328733-21apt-get12:21:323
2800425618,19sleep10-21swapper/110:36:191
2800425618,19sleep10-21swapper/110:36:191
274442567,11sleep10-21swapper/107:09:141
2621125616,18sleep10-21swapper/112:11:491
2100625614,21sleep20-21swapper/210:11:502
209272566,11sleep10-21swapper/110:11:491
870325515,23sleep18760-21irqstats11:16:371
855325515,23sleep18545-21apt-get07:51:341
377725518,23sleep23294-21apt-get11:01:282
377725518,23sleep23294-21apt-get11:01:282
285125515,23sleep12845-21apt-get09:16:291
285125515,23sleep12845-21apt-get09:16:291
2572325514,23sleep20-21swapper/210:26:482
234792556,37sleep30-21swapper/312:02:463
1692025514,25sleep316997-21switchtime08:16:473
1284725516,23sleep312869-21munin-run11:31:173
1142925511,17sleep20-21swapper/209:42:512
699825419,18sleep10-21swapper/107:46:291
440225421,16sleep30-21swapper/309:21:323
3271025415,24sleep1281-21dbus-daemon07:26:181
2824125414,24sleep328238-21apt-get10:36:303
2824125414,24sleep328238-21apt-get10:36:303
2244425414,24sleep122467-21ntp_states10:16:411
1634125418,22sleep216345-21diskstats11:41:342
1168525415,23sleep30-21swapper/308:01:333
102325419,18sleep30-21swapper/309:11:183
1012725412,24sleep310122-21apt-get07:56:323
749825316,21sleep10-21swapper/109:31:191
41672530,10sleep30-21swapper/307:36:453
312525313,22sleep23219-21memory09:16:382
312525313,22sleep23219-21memory09:16:382
2425325314,22sleep124310-21irqstats12:06:361
1789525314,17sleep30-21swapper/310:01:453
700025212,23sleep36995-21apt-get07:46:313
3098325216,21sleep331008-21awk09:01:413
3098325216,21sleep331008-21awk09:01:413
300082527,16sleep10-21swapper/110:41:401
2599725215,20sleep126004-21df_inode08:46:331
2363325210,24sleep123640-21df_inode10:21:341
2363325210,24sleep123640-21df_inode10:21:341
756025115,21sleep27638-21sed07:46:462
1884925113,23sleep118850-21sshd11:48:271
1731225113,22sleep117324-21apt-get10:01:271
1020325113,22sleep210212-21diskstats07:56:352
3046925017,21sleep230509-21basename12:26:362
2488025014,20sleep224952-21mailstats08:41:462
2488025014,20sleep224952-21mailstats08:41:462
1811625012,22sleep317638-21apt-get08:21:233
2850624919,15sleep10-21swapper/110:36:401
2850624919,15sleep10-21swapper/110:36:401
2319624918,15sleep30-21swapper/308:36:423
143842496,30sleep30-21swapper/308:09:573
899624812,22sleep39020-21munin-run09:36:173
3229024811,21sleep132289-21ntp_states12:31:431
2653724714,17sleep20-21swapper/208:46:462
200722475,30sleep30-21swapper/310:10:253
2905924511,19sleep20-21swapper/208:56:312
272592456,14sleep10-21swapper/112:16:331
20602458,15sleep20-21swapper/207:31:282
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional