You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-11 - 14:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Apr 11, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
175172632427,6285sleep112708-21kworker/1:022:34:021
1994928959,870sleep119988-21ntpq22:41:581
1994928959,870sleep119988-21ntpq22:41:581
1289724609,439sleep112933-21ntpq00:01:591
23092238719,211sleep123114-21apt00:36:321
12265238212,224sleep112414-21latency_hist00:01:341
6358231811,223sleep16376-21fschecks_count23:41:521
270672317141,28sleep127089-21munin-run23:06:341
153052284100,162sleep115335-21awk00:11:321
10970228293,22sleep10-21swapper/122:12:051
578922440,8sleep30-21swapper/320:12:033
25291220915,180sleep225297-21ntpq19:32:012
22417220949,144sleep122427-21find21:06:431
30510220716,168sleep130653-21latency_hist23:16:351
30510220716,168sleep130653-21latency_hist23:16:351
28291220310,174sleep10-21swapper/119:41:531
1035222036,161sleep113441-21snmpd23:53:061
1035222036,161sleep113441-21snmpd23:53:061
24342219615,26sleep10-21swapper/122:56:521
5329218934,94sleep15469-21latency_hist21:56:351
5329218934,94sleep15469-21latency_hist21:56:351
23748218212,158sleep123791-21processes19:26:591
23748218212,158sleep123791-21processes19:26:591
4825216712,54sleep34966-21latency_hist20:11:363
3600216614,134sleep03682-21fschecks_time20:06:520
3085021650,23sleep330854-21fschecks_count23:16:533
15565216515,134sleep10-21swapper/122:27:041
12628216518,130sleep312567-21latency_hist20:36:363
1614216414,90sleep10-21swapper/121:42:051
30227216310,15sleep10-21swapper/121:31:471
22672216011,136sleep022675-21grep22:51:460
9836215921,113sleep10-21swapper/120:26:531
18521215916,128sleep118606-21cut22:37:011
21979215810,96sleep10-21swapper/121:06:351
27401215510,16sleep00-21swapper/023:06:370
16330215531,100sleep216354-21munin-plugin-st19:06:332
13109215514,84sleep113147-21latency_hist22:21:341
8818215313,125sleep18919-21cat20:22:041
245121517,53sleep10-21swapper/121:46:491
245121517,53sleep10-21swapper/121:46:491
6067215012,116sleep16215-21latency_hist23:41:351
872021447,126sleep113441-21snmpd23:47:051
14694214311,94sleep114838-21cat22:26:351
288021384,11sleep30-21swapper/320:02:073
32672213716,106sleep1356-21date19:56:361
27143213212,100sleep227250-21cat23:06:352
3226213010,54sleep13241-21idleruntime-cro20:06:331
26945213012,104sleep027004-21latency19:37:160
16631213010,60sleep10-21swapper/119:06:441
18135212982,25sleep10-21swapper/119:11:361
29991212814,98sleep230082-21sort21:31:362
19249212721,18sleep10-21swapper/100:22:021
3417212333,78sleep13457-21latency23:31:541
17219212318,87sleep317295-21sed19:07:023
29539212214,87sleep329581-21/usr/sbin/munin23:11:573
29539212214,87sleep329581-21/usr/sbin/munin23:11:573
2775521217,16sleep10-21swapper/123:07:231
22952212115,89sleep022988-21wc21:07:310
490621206,17sleep20-21swapper/223:36:522
10043211922,74sleep310181-21unixbench_multi23:52:033
10043211922,74sleep310181-21unixbench_multi23:52:033
2528211514,83sleep32531-21/usr/sbin/munin21:46:513
2528211514,83sleep32531-21/usr/sbin/munin21:46:513
2974121148,67sleep10-21swapper/119:46:461
8433211312,88sleep38474-21date22:06:343
721521138,24sleep17311-21grep22:01:521
21808211310,17sleep10-21swapper/119:21:501
20449211310,43sleep10-21swapper/119:16:531
10619211317,75sleep010667-21install23:56:330
10619211317,75sleep010667-21install23:56:330
24680211262,36sleep124827-21grep21:12:041
21202211211,22sleep00-21swapper/022:46:500
19491211214,84sleep119530-21grep20:56:581
1470621128,70sleep10-21swapper/120:41:551
18807211110,87sleep018936-21awk22:37:040
2543521107,17sleep10-21swapper/119:32:041
28769210910,65sleep128863-21fschecks_time21:26:541
2665321099,68sleep126637-21apt-get19:36:531
26263210937,57sleep126389-21cat21:17:051
2911210812,79sleep10-21swapper/123:31:341
1147121089,61sleep10-21swapper/120:31:541
31421077,18sleep132484-21/usr/sbin/munin23:21:561
31421077,18sleep132484-21/usr/sbin/munin23:21:561
2141921078,17sleep10-21swapper/122:46:551
9072210626,65sleep19119-21ntpq22:06:571
32024210516,72sleep232104-21cmp23:21:332
2267621048,38sleep10-21swapper/122:51:481
1751121047,13sleep10-21swapper/120:51:441
22017210316,69sleep222070-21if_err_eth000:31:522
17680210313,32sleep10-21swapper/100:17:011
505921027,36sleep10-21swapper/120:11:461
12549210219,66sleep012578-21awk20:36:330
14532210118,67sleep314597-21awk00:07:023
1719229913,28sleep30-21swapper/300:16:473
533129812,55sleep05418-21sort23:37:020
235662980,8sleep00-21swapper/019:26:530
235662980,8sleep00-21swapper/019:26:530
1643229812,30sleep10-21swapper/120:47:001
2756429720,58sleep327080-21apt-get23:06:443
1499729718,57sleep215105-21fschecks_time22:26:532
1283929611,71sleep312858-21apt-get20:36:443
1581029522,50sleep115043-21/usr/sbin/munin19:02:041
59352949,70sleep00-21swapper/020:12:050
388329412,66sleep13490-21/usr/sbin/munin20:06:581
1572329416,25sleep00-21swapper/020:46:350
368129313,60sleep03752-21/usr/sbin/munin23:32:010
3219429315,60sleep032001-21apt-get23:21:350
2242929323,53sleep221929-21apt-get21:06:442
614129122,55sleep16227-21cat21:57:031
2217929111,65sleep222191-21sh19:21:592
1380829134,26sleep00-21swapper/000:06:340
847729015,60sleep08625-21date22:06:350
78042907,27sleep10-21swapper/120:20:141
2160029031,44sleep121727-21latency_hist00:31:351
214529011,62sleep02216-21sensors23:27:040
360728915,54sleep23614-21/usr/sbin/munin23:31:592
16612895,28sleep10-21swapper/120:01:201
280742886,29sleep10-21swapper/121:22:301
1059828817,53sleep210642-21ntpq22:11:582
5142876,28sleep10-21swapper/121:40:231
317622876,28sleep10-21swapper/123:18:411
2019028714,21sleep30-21swapper/319:16:473
1317128719,34sleep313289-21timerwakeupswit00:02:053
1577328611,32sleep10-21swapper/100:11:521
1577328611,32sleep10-21swapper/100:11:521
2090028517,51sleep321026-21sed00:27:013
206928316,43sleep21583-21/usr/sbin/munin23:27:022
138782835,27sleep10-21swapper/120:38:241
3100228223,37sleep30-21swapper/319:51:333
556828018,37sleep35543-21/usr/sbin/munin21:56:493
328128016,44sleep33357-21/usr/sbin/munin23:31:523
1916428028,28sleep00-21swapper/019:12:040
2689427921,33sleep30-21swapper/321:21:353
2626527922,26sleep00-21swapper/019:36:350
1077527917,45sleep310846-21expr22:12:033
3138827820,39sleep231494-21fschecks_time19:51:522
1103627819,42sleep011101-21fschecks_time23:56:530
1909727715,40sleep019150-21ls20:56:480
1800727721,35sleep018020-21munin-node19:11:350
1779827719,29sleep30-21swapper/322:36:353
1189027714,48sleep311985-21perl22:16:523
2710627613,47sleep027127-21apt-get21:21:440
26872757,15sleep20-21swapper/221:46:542
26872757,15sleep20-21swapper/221:46:542
2207927519,31sleep20-21swapper/221:06:362
2027327521,35sleep220288-21cut19:16:502
1665127411,49sleep216707-21if_eth022:31:532
1532927424,30sleep00-21swapper/019:01:550
1487727419,37sleep214942-21awk20:42:002
2950727319,25sleep30-21swapper/319:46:363
2653127311,52sleep00-21swapper/021:17:320
2026227315,38sleep020281-21apt-get00:26:410
1302827317,37sleep013085-21ls20:36:530
2868927222,27sleep20-21swapper/219:42:032
2795827214,16sleep30-21swapper/319:41:353
2725127214,41sleep327274-21apt-key23:06:363
2481927218,31sleep30-21swapper/322:57:033
229522729,13sleep00-21swapper/000:34:450
1899927225,30sleep319037-21/usr/sbin/munin19:11:593
1820827221,29sleep018363-21grep20:52:040
59427110,19sleep00-21swapper/019:56:550
3207527112,44sleep032138-21latency21:36:550
1687427120,34sleep016910-21apt00:16:320
1687427120,34sleep016910-21apt00:16:320
893927019,27sleep20-21swapper/222:06:532
31652700,9sleep00-21swapper/021:47:040
31652700,9sleep00-21swapper/021:47:040
2618427019,36sleep326195-21munin-run19:36:343
2446427010,46sleep224528-21ps21:12:012
2321427016,39sleep223277-21/usr/sbin/munin22:52:012
2154527016,38sleep221580-21awk00:31:322
1728227010,43sleep017422-21cat20:51:350
1635727032,21sleep30-21swapper/319:06:343
56582696,10sleep30-21swapper/323:37:213
3059526914,25sleep20-21swapper/221:31:562
266822690,7sleep20-21swapper/219:37:092
2279926912,42sleep322857-21memory21:07:263
2152526917,29sleep30-21swapper/322:46:573
2144926915,22sleep00-21swapper/019:21:330
1148326912,42sleep211554-21sensors23:57:032
2092026819,24sleep20-21swapper/222:46:362
2092026819,24sleep20-21swapper/222:46:362
913726612,40sleep09197-21processes22:07:000
874626617,21sleep00-21swapper/020:22:030
665026617,28sleep30-21swapper/323:41:583
3067426615,27sleep00-21swapper/021:31:580
301972666,48sleep00-21swapper/023:13:520
301972666,48sleep00-21swapper/023:13:520
1791826616,34sleep217492-21/usr/sbin/munin20:51:582
1479826615,24sleep20-21swapper/200:07:052
299052658,11sleep00-21swapper/021:31:050
2945526522,26sleep029460-21modprobe19:46:330
2844326526,19sleep20-21swapper/221:26:352
1228826511,36sleep012349-21sendmail_mailst22:17:010
1070626522,22sleep20-21swapper/223:56:352
1070626522,22sleep20-21swapper/223:56:352
314962647,12sleep30-21swapper/319:51:533
2950726420,27sleep229538-21/usr/sbin/munin23:11:562
2950726420,27sleep229538-21/usr/sbin/munin23:11:562
1894926425,19sleep20-21swapper/222:37:062
2354626314,34sleep323562-21/usr/sbin/munin19:26:533
2354626314,34sleep323562-21/usr/sbin/munin19:26:533
2018726314,32sleep320279-21cat22:42:043
2018726314,32sleep320279-21cat22:42:043
2005426316,21sleep20-21swapper/200:26:352
1013126316,24sleep20-21swapper/220:26:582
1636526219,24sleep20-21swapper/220:46:572
161402627,11sleep30-21swapper/322:31:183
998326114,32sleep310024-21latency_hist22:11:353
40926119,20sleep30-21swapper/319:56:503
405826112,33sleep24126-21swap20:07:032
3233926111,31sleep30-21swapper/321:37:023
2159026117,19sleep20-21swapper/222:47:002
1465826123,21sleep00-21swapper/022:26:340
912826017,23sleep00-21swapper/023:51:360
567326018,23sleep20-21swapper/220:12:032
3260026018,21sleep20-21swapper/223:21:532
3260026018,21sleep20-21swapper/223:21:532
225126011,25sleep00-21swapper/020:01:560
220826021,22sleep30-21swapper/321:46:353
1917926014,27sleep30-21swapper/320:56:513
1778426015,23sleep30-21swapper/320:51:553
1657326017,20sleep30-21swapper/320:47:023
1433326012,26sleep30-21swapper/322:23:023
715125916,24sleep07222-21sendmail_mailtr20:17:020
2312425816,26sleep323130-21ntp_states22:51:583
151225816,19sleep00-21swapper/021:42:040
1105025819,16sleep30-21swapper/320:31:353
699525715,23sleep27132-21grep23:42:052
2453025715,16sleep30-21swapper/321:12:023
1803625715,26sleep318029-21apt-get22:36:473
1335125715,25sleep213347-21apt-get22:21:482
131425718,26sleep01343-21munin-run23:26:320
131425718,26sleep01343-21munin-run23:26:320
1136525712,19sleep20-21swapper/220:31:502
51925618,25sleep3586-21ps23:22:033
51925618,25sleep3586-21ps23:22:033
2988925616,22sleep00-21swapper/019:46:530
2173825614,25sleep321733-21apt-get19:21:483
1621125616,19sleep00-21swapper/022:31:350
134142565,35sleep20-21swapper/200:02:082
974225514,25sleep09736-21apt-get20:26:480
892025515,22sleep20-21swapper/220:22:052
758425516,23sleep00-21swapper/023:46:350
710725510,31sleep37147-21ps20:17:013
3029625513,25sleep330302-21df_inode21:31:493
2424825514,24sleep024242-21apt-get22:56:470
166182554,15sleep20-21swapper/200:13:232
166182554,15sleep20-21swapper/200:13:232
1428425516,21sleep00-21swapper/000:06:550
1085525517,22sleep00-21swapper/022:12:040
565125415,21sleep20-21swapper/221:56:522
476325415,17sleep00-21swapper/021:52:050
476325415,17sleep00-21swapper/021:52:050
399025412,24sleep23984-21apt-get21:51:472
399025412,24sleep23984-21apt-get21:51:472
3262925416,16sleep00-21swapper/019:56:350
3217425418,19sleep20-21swapper/221:36:572
2942625415,19sleep30-21swapper/321:27:083
222022545,12sleep00-21swapper/019:22:010
207862548,39sleep00-21swapper/022:45:530
207862548,39sleep00-21swapper/022:45:530
183742545,38sleep00-21swapper/000:20:320
1447625413,25sleep314487-21df_inode20:41:493
120752545,28sleep20-21swapper/222:16:552
2225525314,23sleep322287-21ntp_states00:31:583
1907025314,26sleep218584-21/usr/sbin/munin19:12:022
16112538,27sleep30-21swapper/323:26:513
1559425312,25sleep315589-21apt-get00:11:463
1559425312,25sleep315589-21apt-get00:11:463
749425213,22sleep20-21swapper/222:01:572
3001125212,20sleep20-21swapper/219:46:552
2769825212,25sleep227704-21df_inode23:07:152
2444925214,24sleep024451-21ntp_states21:11:590
1919825216,20sleep30-21swapper/300:22:003
281692514,14sleep30-21swapper/321:23:453
2791325114,16sleep00-21swapper/023:08:180
2190925112,24sleep00-21swapper/021:06:340
2062425111,23sleep320643-21apt-get21:01:443
203625114,21sleep22045-21df_inode20:01:502
1703425118,18sleep20-21swapper/219:06:572
1294525113,22sleep212949-21diskstats20:36:512
87522505,10sleep20-21swapper/223:47:222
692825013,23sleep27012-21memory20:16:562
688425010,18sleep00-21swapper/022:01:340
444525012,22sleep34451-21ntp_states21:52:003
444525012,22sleep34451-21ntp_states21:52:003
2379225013,15sleep20-21swapper/219:27:002
2379225013,15sleep20-21swapper/219:27:002
1874425014,21sleep018811-21df00:21:490
263902498,14sleep20-21swapper/221:17:062
1966924916,17sleep20-21swapper/220:57:032
1121824916,21sleep311277-21latency23:56:563
2429824818,16sleep20-21swapper/222:56:492
171252479,15sleep00-21swapper/022:32:040
867624613,18sleep3281-21dbus-daemon20:22:023
1746324612,18sleep20-21swapper/200:16:542
14412464,32sleep20-21swapper/219:58:302
91124511,20sleep2970-21df21:41:492
78182429,17sleep30-21swapper/323:46:463
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional