You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 21:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Wed Mar 25, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
75932171510,1672sleep28301-1kworker/2:1H10:33:042
17513256420,525sleep317640-21/usr/sbin/munin11:03:013
17513256420,525sleep317640-21/usr/sbin/munin11:03:013
3071925167,498sleep013441-21snmpd08:20:020
375223813,207sleep2468-21tune2fs08:27:512
25925223443,179sleep025943-21latency_hist11:32:320
2606222039,104sleep02728-21timerwakeupswit11:58:000
15369222013,122sleep015402-21fschecks_count10:57:490
15369222013,122sleep015402-21fschecks_count10:57:490
13697221721,184sleep213800-21grep07:27:512
11715221632,110sleep011810-21memory09:02:540
17256221336,163sleep017287-21latency07:38:070
12653220615,118sleep012786-21latency_hist12:32:320
23660220016,170sleep323741-21find11:22:593
6219219911,177sleep00-21swapper/008:44:210
10323219713,167sleep031601-1kworker/0:2H10:42:310
13115218972,84sleep00-21swapper/012:32:490
2045721888,169sleep213441-21snmpd09:32:032
111321880,12sleep06523-21cyclictest10:12:500
23391217410,99sleep023400-21acpi07:57:410
25930217333,116sleep025994-21sendmail_mailqu09:48:000
13088217123,126sleep313101-21fschecks_count09:07:503
1299521685,15sleep20-21swapper/212:32:342
11041216615,134sleep111111-21sensors07:18:011
10366216614,135sleep310424-21sendmail_mailqu08:58:003
10366216614,135sleep310424-21sendmail_mailqu08:58:003
23049216332,35sleep00-21swapper/007:57:330
2736121625,11sleep00-21swapper/009:52:520
2736121625,11sleep00-21swapper/009:52:520
30081215717,122sleep230166-21switchtime11:43:002
831221568,55sleep00-21swapper/008:52:440
3929215545,92sleep04011-21mailstats08:38:030
4813215417,116sleep24975-21sort08:42:332
18673215112,65sleep00-21swapper/007:42:510
12603214322,96sleep312650-21unin-run12:32:313
5685214044,20sleep05681-21apt-get10:27:410
1616021407,85sleep013023-21kworker/0:107:34:190
21596213764,56sleep00-21swapper/011:17:470
1350521370,11sleep10-21swapper/109:07:591
897921347,16sleep00-21swapper/007:13:200
22951213317,96sleep323046-21touch07:57:323
24265213247,71sleep024278-21needreboot09:42:550
5060213035,12sleep00-21swapper/010:23:090
30451213016,90sleep330481-21fschecks_count10:02:493
28782213042,66sleep00-21swapper/009:57:410
11803213016,83sleep00-21swapper/007:22:320
31070212718,16sleep30-21swapper/311:47:343
2275221279,103sleep022790-21ntpq09:37:540
11099212615,96sleep311170-21mailstats10:42:583
29989212510,29sleep00-21swapper/011:42:580
5898212315,88sleep25956-21/usr/sbin/munin10:27:512
27981212318,86sleep227966-21apt-get11:37:472
719212256,52sleep00-21swapper/010:12:310
719212256,52sleep00-21swapper/010:12:310
1695021209,51sleep00-21swapper/009:18:040
12580212031,69sleep212602-21latency_hist12:32:302
2463211918,80sleep32589-21wc08:33:013
19014211920,76sleep119161-21cat09:27:341
26884211714,81sleep00-21swapper/008:07:550
1974521157,94sleep00-21swapper/009:28:010
10434211438,27sleep09864-21/usr/sbin/munin08:58:010
10434211438,27sleep09864-21/usr/sbin/munin08:58:010
8069211320,74sleep28212-21dpkg07:12:332
15038211211,86sleep015063-21latency_hist10:57:320
15038211211,86sleep015063-21latency_hist10:57:320
16408211118,79sleep216421-21uname09:17:542
8712210970,19sleep00-21swapper/012:17:590
7905210817,73sleep07934-21idleruntime-cro12:17:300
853210614,25sleep00-21swapper/008:28:010
7033210515,77sleep37164-21latency_hist10:32:323
23404210315,70sleep122887-21apt-get07:57:421
7458210211,34sleep07477-21apt-get10:32:420
6637210212,78sleep20-21swapper/212:12:442
14952210213,72sleep014996-21ntpq09:12:550
16988210111,75sleep10-21swapper/107:38:001
28279210011,16sleep00-21swapper/008:12:510
12767210058,23sleep20-21swapper/209:07:322
798629911,73sleep07999-21apt-get07:12:310
3251229910,74sleep032601-21latency_hist08:27:330
1350929911,11sleep30-21swapper/312:33:163
1420629811,73sleep214288-21systemctl10:52:582
1420629811,73sleep214288-21systemctl10:52:582
958129717,63sleep19659-21ls07:14:091
523829719,55sleep15196-21apt-get08:43:091
25982978,13sleep00-21swapper/010:17:480
1695929734,37sleep316952-21apt-get07:37:463
1202329710,74sleep012033-21find07:22:400
3200929612,70sleep332098-21fschecks_time10:07:493
3200929612,70sleep332098-21fschecks_time10:07:493
2222429611,64sleep122356-21timerwakeupswit07:53:021
2101629634,13sleep00-21swapper/007:48:370
2101629634,13sleep00-21swapper/007:48:370
1119729629,39sleep311352-21grep07:18:033
1094429610,67sleep00-21swapper/012:26:230
607229537,10sleep00-21swapper/012:09:030
1684029412,19sleep00-21swapper/011:02:450
1684029412,19sleep00-21swapper/011:02:450
36329334,10sleep00-21swapper/011:52:350
1956829335,10sleep00-21swapper/011:11:330
787429234,12sleep00-21swapper/008:50:390
2211629219,57sleep222165-21grep09:37:322
1034429212,14sleep10-21swapper/108:57:581
1034429212,14sleep10-21swapper/108:57:581
3135729134,11sleep00-21swapper/010:04:030
2033429114,58sleep220424-21/usr/sbin/munin07:47:542
2033429114,58sleep220424-21/usr/sbin/munin07:47:542
1475329134,10sleep00-21swapper/007:31:080
449729034,10sleep00-21swapper/012:03:420
285629021,51sleep22874-21uname10:17:542
2419329034,11sleep00-21swapper/011:25:500
1883429035,10sleep00-21swapper/009:26:070
1125329015,60sleep111379-21sed10:43:001
274222895,12sleep10-21swapper/108:08:041
18162897,28sleep00-21swapper/008:32:420
134682897,71sleep013502-21ps09:07:590
131212895,73sleep00-21swapper/010:48:400
3008128823,43sleep130095-21apt10:02:301
1455028822,34sleep20-21swapper/209:12:452
534728724,33sleep20-21swapper/208:43:182
2919528758,19sleep00-21swapper/011:41:570
271232875,71sleep00-21swapper/011:33:420
271232875,71sleep00-21swapper/011:33:420
946928619,53sleep39493-21proc_pri07:14:073
2193428635,40sleep00-21swapper/009:36:130
2895828514,54sleep229044-21/usr/sbin/munin09:58:072
2767628521,44sleep327840-21tail09:52:593
2767628521,44sleep327840-21tail09:52:593
2495228563,10sleep00-21swapper/008:02:420
2169328518,53sleep221771-21grep07:52:512
2109028433,10sleep00-21swapper/011:16:050
1111228322,48sleep211196-21expr07:18:022
1506528217,29sleep10-21swapper/110:57:331
1506528217,29sleep10-21swapper/110:57:331
523028120,45sleep35296-21fschecks_time07:02:493
3260228012,53sleep06523-21cyclictest11:52:310
3083028016,48sleep130925-21apt-key11:47:311
174528014,44sleep11876-21latency_hist11:57:331
1575328017,20sleep30-21swapper/307:33:013
1061228023,27sleep20-21swapper/208:58:032
1061228023,27sleep20-21swapper/208:58:032
3128027914,52sleep130798-21apt-get11:47:401
2869427930,25sleep328847-21awk11:38:213
2757927914,52sleep327658-21rm11:37:323
2757927914,52sleep327658-21rm11:37:323
697727710,14sleep30-21swapper/308:47:553
2355027723,29sleep20-21swapper/207:58:202
3128827622,31sleep30-21swapper/308:22:523
2413727614,44sleep224171-21irqstats09:42:532
1243027620,23sleep10-21swapper/110:47:531
1154227654,12sleep00-21swapper/010:43:150
957227522,35sleep29591-21apt08:57:312
737527521,26sleep30-21swapper/312:13:013
2825727521,35sleep228270-21modprobe09:57:312
2825727521,35sleep228270-21modprobe09:57:312
234312759,14sleep10-21swapper/111:22:531
1501427524,34sleep315051-21ps09:12:583
1224727524,29sleep20-21swapper/210:47:502
517527429,22sleep30-21swapper/308:42:423
2498127417,42sleep125012-21nfs4_client11:27:531
19822749,13sleep10-21swapper/111:57:451
957227320,27sleep20-21swapper/212:22:332
695627318,37sleep16982-21apt-get10:32:301
401427311,48sleep14037-21ps12:02:581
401327319,28sleep10-21swapper/108:38:041
1894527324,25sleep20-21swapper/209:27:312
1061527321,29sleep20-21swapper/210:42:452
1434827220,23sleep20-21swapper/209:12:332
1232127221,21sleep30-21swapper/310:47:513
1660027112,44sleep316570-21latency_hist11:02:323
1660027112,44sleep316570-21latency_hist11:02:323
138762719,15sleep10-21swapper/107:27:541
1340127118,32sleep10-21swapper/112:33:141
120182710,7sleep10-21swapper/112:28:001
1022927119,36sleep210235-21ntp_states12:22:582
2833827015,20sleep10-21swapper/108:12:521
1655027024,23sleep10-21swapper/111:02:301
1655027024,23sleep10-21swapper/111:02:301
645626927,27sleep26474-21munin-run07:07:322
640426923,28sleep16436-21munin-run07:07:311
2748126920,23sleep20-21swapper/211:37:312
2748126920,23sleep20-21swapper/211:37:312
73362689,44sleep37428-21cat07:08:023
2783426817,31sleep327975-21cut08:12:343
2282926822,25sleep10-21swapper/111:22:321
2191926823,28sleep30-21swapper/311:17:543
202326816,23sleep10-21swapper/108:32:511
2618826713,24sleep30-21swapper/309:48:033
2090826722,26sleep20-21swapper/211:13:532
1969426721,27sleep20-21swapper/211:12:322
1199026718,19sleep30-21swapper/309:03:013
3246026615,37sleep232487-21basename08:27:312
2947826615,26sleep30-21swapper/308:17:353
2706426624,19sleep10-21swapper/109:52:451
2706426624,19sleep10-21swapper/109:52:451
2474026624,22sleep324823-21wc08:02:333
1555226619,21sleep20-21swapper/207:32:562
959626514,32sleep3281-21dbus-daemon08:57:323
484526514,24sleep30-21swapper/312:07:323
2544226520,29sleep225480-21grep08:03:002
2127026514,34sleep121282-21munin-run11:17:321
1622526517,29sleep30-21swapper/309:17:513
966126423,27sleep20-21swapper/207:14:112
802326419,27sleep18053-21munin-run07:12:321
786826417,25sleep10-21swapper/112:17:031
2933826418,30sleep329341-21cron08:17:313
2742126417,29sleep127480-21basename11:37:301
2742126417,29sleep127480-21basename11:37:301
2552926429,20sleep10-21swapper/109:47:491
2366426416,33sleep223731-21latency_hist09:42:332
562926224,20sleep00-21swapper/007:02:590
423726213,20sleep20-21swapper/210:22:502
313582626,43sleep10-21swapper/110:04:041
2565626222,21sleep10-21swapper/108:03:031
2518826223,25sleep325276-21switchtime11:27:593
1969926220,21sleep30-21swapper/309:28:003
1845726217,19sleep10-21swapper/111:07:471
1596926220,21sleep10-21swapper/107:33:031
3141126124,20sleep10-21swapper/108:22:541
253526120,26sleep30-21swapper/311:57:593
2131926116,20sleep30-21swapper/307:52:313
2131926116,20sleep30-21swapper/307:52:313
1977326110,26sleep30-21swapper/307:47:313
159326117,21sleep20-21swapper/210:13:012
130622617,11sleep30-21swapper/307:24:213
88726016,18sleep30-21swapper/311:52:593
822326012,35sleep38258-21grep12:17:463
569626017,25sleep30-21swapper/312:08:013
3016226019,23sleep10-21swapper/108:18:011
301026020,21sleep20-21swapper/208:36:232
1380126021,21sleep30-21swapper/307:27:523
915125922,19sleep30-21swapper/310:37:493
77622598,11sleep10-21swapper/108:49:101
48525912,23sleep20-21swapper/211:52:502
3138325915,28sleep231392-21df_inode11:47:492
310225912,16sleep10-21swapper/110:18:011
2941925914,29sleep229477-21latency_hist08:17:342
2880625920,20sleep30-21swapper/309:57:443
2357025919,20sleep20-21swapper/211:22:582
76725818,17sleep10-21swapper/110:12:321
76725818,17sleep10-21swapper/110:12:321
762425812,20sleep20-21swapper/208:48:062
355225818,24sleep33630-21fw_packets08:37:543
3016925820,15sleep30-21swapper/311:43:013
268025814,29sleep32692-21/usr/sbin/munin10:17:513
2482025815,27sleep224870-21/usr/sbin/munin11:27:502
2450825818,19sleep30-21swapper/309:43:023
2198625817,29sleep222030-21processes11:17:572
2128725820,17sleep30-21swapper/309:33:013
2139725720,17sleep10-21swapper/109:33:031
20422574,11sleep30-21swapper/310:14:323
1912325714,18sleep20-21swapper/207:43:022
160325715,28sleep31659-21sed08:32:333
1285325715,22sleep112951-21users07:23:021
1108725719,18sleep10-21swapper/112:27:331
544025619,17sleep30-21swapper/310:27:323
2973325615,26sleep229631-21/usr/sbin/munin08:17:502
2538025614,29sleep325417-21ntp_kernel_pll_08:02:573
1819125618,22sleep318287-21mailstats09:23:003
822125520,18sleep10-21swapper/112:17:441
480725521,17sleep10-21swapper/110:23:021
3272325516,24sleep132741-21apt08:27:421
2049225517,24sleep120531-21modprobe09:32:301
1866825518,22sleep318754-21memory11:07:533
1321425515,25sleep20-21swapper/207:26:182
107925514,17sleep10-21swapper/110:12:481
1037325516,17sleep20-21swapper/210:42:332
529325416,23sleep25350-21irqstats12:07:522
2800525413,23sleep127966-21apt-get11:38:031
508525313,23sleep15152-21df12:07:461
2572325313,20sleep20-21swapper/209:47:542
246625314,21sleep22534-21systemctl11:57:582
1695825314,23sleep216427-21apt-get07:37:452
81125216,15sleep20-21swapper/210:12:332
2006225213,23sleep120057-21apt-get07:47:451
2006225213,23sleep120057-21apt-get07:47:451
1454925214,22sleep114544-21apt-get09:12:441
3096225119,18sleep20-21swapper/210:03:002
2245625110,16sleep30-21swapper/309:37:493
2020425116,16sleep30-21swapper/311:12:543
1728125114,22sleep217292-21ntp_states11:02:552
1728125114,22sleep217292-21ntp_states11:02:552
1549125114,24sleep215541-21if_eth010:57:512
1549125114,24sleep215541-21if_eth010:57:512
1023625121,14sleep30-21swapper/312:22:593
394425015,21sleep23989-21ntp_states12:02:552
3252125015,20sleep20-21swapper/210:08:012
2652425014,18sleep20-21swapper/208:07:452
1982625014,15sleep10-21swapper/107:47:331
1621225016,19sleep116219-21df_inode09:17:491
745524913,20sleep36973-21apt-get10:32:413
2235324914,18sleep10-21swapper/109:37:431
1274424912,20sleep112766-21munin-run09:07:311
913924812,18sleep10-21swapper/110:37:471
829224811,21sleep18305-21apt-get08:52:411
1733824811,22sleep117419-21mailstats11:02:581
1733824811,22sleep117419-21mailstats11:02:581
73792468,13sleep10-21swapper/110:32:341
148962456,29sleep30-21swapper/310:56:393
148962456,29sleep30-21swapper/310:56:393
2408424010,17sleep123878-21/usr/sbin/munin09:42:521
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional