You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-02 - 19:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Aug 02, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
283629937,972sleep02874-21ntpq09:18:020
8745275021,714sleep18844-21/usr/sbin/munin07:53:091
8745275021,714sleep18844-21/usr/sbin/munin07:53:091
1911273617,702sleep31979-21/usr/sbin/munin07:32:523
1911273617,702sleep31979-21/usr/sbin/munin07:32:523
1634825279,503sleep016395-21ntpq08:18:030
31762250525,464sleep0364-21snmpd09:07:530
3158423016,283sleep031609-21ntpq12:32:580
1234022778,257sleep012376-21expr11:32:550
1660227354,204sleep01680-21latency_hist07:32:360
1660227354,204sleep01680-21latency_hist07:32:360
20588226114,189sleep00-21swapper/008:32:520
23490225553,181sleep023503-21fschecks_count12:07:550
23948225337,200sleep00-21swapper/008:42:490
14932223445,110sleep015006-21cat08:13:060
14932223445,110sleep015006-21cat08:13:060
3246223313,198sleep00-21swapper/007:37:370
3246223313,198sleep00-21swapper/007:37:370
26944222654,102sleep00-21swapper/012:18:010
5300222175,100sleep05425-21latency_hist09:27:390
12038221913,20sleep00-21swapper/009:48:020
13917221743,19sleep013962-21fschecks_count11:37:560
16693219211,165sleep116669-21apt-get10:03:001
16693219211,165sleep116669-21apt-get10:03:001
598821895,26sleep00-21swapper/007:43:430
598821895,26sleep00-21swapper/007:43:430
29124218614,153sleep114575-21kworker/u8:310:42:461
21872218616,97sleep021944-21mailstats10:18:050
8989218013,16sleep30-21swapper/307:53:113
8989218013,16sleep30-21swapper/307:53:113
15623217423,35sleep115653-21munin-run08:17:351
15623217423,35sleep115653-21munin-run08:17:351
11433217326,90sleep00-21swapper/009:47:090
28224217239,41sleep0364-21snmpd08:56:220
29724216222,42sleep00-21swapper/012:27:540
3233021548,129sleep00-21swapper/010:52:540
4094215312,123sleep04203-21sort07:38:060
4094215312,123sleep04203-21sort07:38:060
2964421505,29sleep00-21swapper/008:59:130
25929214935,98sleep026096-21latency_hist07:07:380
30909214844,90sleep00-21swapper/007:21:470
10705214846,90sleep010743-21grep11:27:500
30659214542,57sleep00-21swapper/009:03:040
26181214215,115sleep00-21swapper/012:17:000
17628213920,98sleep117778-21unixbench_multi11:48:101
14156213945,70sleep00-21swapper/008:12:370
2473821378,112sleep00-21swapper/010:27:590
2473821378,112sleep00-21swapper/010:27:590
31729213513,107sleep031771-21proc_pri07:23:040
31729213513,107sleep031771-21proc_pri07:23:040
21587213523,94sleep00-21swapper/012:02:470
27509213312,57sleep00-21swapper/007:12:380
27509213312,57sleep00-21swapper/007:12:380
27444212716,31sleep127473-21munin-run07:12:351
27444212716,31sleep127473-21munin-run07:12:351
29518212612,99sleep229549-21nfs_client10:42:592
17734212632,47sleep0364-21snmpd10:04:570
17734212632,47sleep0364-21snmpd10:04:570
25038212515,96sleep325105-21switchtime10:28:063
25038212515,96sleep325105-21switchtime10:28:063
13394212222,85sleep113399-21entropy09:52:551
13394212222,85sleep113399-21entropy09:52:551
1091521226,13sleep00-21swapper/008:01:520
818121219,52sleep00-21swapper/011:18:050
416921205,68sleep00-21swapper/011:07:230
824211821,81sleep00-21swapper/009:12:450
26476211720,52sleep00-21swapper/010:33:030
27147211614,80sleep20-21swapper/208:52:572
15156211627,77sleep015199-21cat09:57:590
13308211519,40sleep00-21swapper/009:52:500
13308211519,40sleep00-21swapper/009:52:500
31287211319,77sleep331396-21perl12:32:533
2026921138,39sleep00-21swapper/011:57:580
3991211113,82sleep24059-21df09:22:522
32441211118,79sleep332568-21unixbench_multi09:08:083
28898211121,60sleep00-21swapper/010:42:360
2635211011,85sleep02667-21munin-run11:02:350
19957210810,40sleep00-21swapper/008:28:120
19957210810,40sleep00-21swapper/008:28:120
18800210817,76sleep218834-21cat08:27:372
27361210713,76sleep227496-21cat10:37:382
2318521076,12sleep00-21swapper/008:39:340
230321070,6sleep10-21swapper/109:17:381
19013210610,57sleep018454-21/usr/sbin/munin11:53:060
1634421065,10sleep00-21swapper/011:43:490
7128210515,65sleep00-21swapper/007:48:070
7128210515,65sleep00-21swapper/007:48:070
24782210525,62sleep124836-21if_err_eth007:02:591
976221046,83sleep09872-21timerandwakeup11:23:060
5623210411,43sleep00-21swapper/009:27:550
28255210435,21sleep00-21swapper/012:22:570
25280210420,70sleep325407-21unixbench_multi07:03:093
20596210415,30sleep320679-21users10:13:083
16650210320,63sleep316163-21apt-get10:02:463
16650210320,63sleep316163-21apt-get10:02:463
9712210211,76sleep29780-21sh07:57:542
3205421027,80sleep032205-21latency_hist10:52:380
3007521005,36sleep00-21swapper/010:43:170
28280210020,54sleep228437-21unixbench_multi10:38:062
1245121005,10sleep00-21swapper/008:06:440
83052985,78sleep00-21swapper/009:37:080
1603129814,62sleep215918-21/usr/sbin/munin08:17:562
1252129815,59sleep30-21swapper/311:32:583
182292969,42sleep00-21swapper/010:07:480
68702958,39sleep00-21swapper/011:13:090
107882956,12sleep30-21swapper/311:27:533
2308529411,67sleep323241-21latency_hist12:07:383
511929323,54sleep15186-21fschecks_time07:42:551
511929323,54sleep15186-21fschecks_time07:42:551
456029318,59sleep34628-21fschecks_time11:07:553
2881529317,52sleep228960-21unixbench_multi07:13:092
2838729314,56sleep10-21swapper/108:57:361
2204329317,60sleep221343-21/usr/sbin/munin10:18:072
884729210,69sleep08894-21basename09:37:560
2340629218,58sleep123472-21ls12:07:521
2664129121,44sleep326732-21fschecks_time12:17:553
2622929124,48sleep326269-21sendmail12:17:353
2453429114,59sleep224439-21/usr/sbin/munin10:27:522
2453429114,59sleep224439-21/usr/sbin/munin10:27:522
207442915,27sleep00-21swapper/010:13:490
1831629118,53sleep218337-21fschecks_count10:07:542
787829013,55sleep17580-21/usr/sbin/munin11:17:581
578228930,32sleep10-21swapper/111:12:371
2300128924,45sleep123016-21fschecks_count10:22:551
3070128713,62sleep330719-21cpu10:47:523
23522875,11sleep10-21swapper/110:58:571
1493628712,59sleep214960-21sh09:57:532
474228621,47sleep14748-21modprobe07:42:351
474228621,47sleep14748-21modprobe07:42:351
301128613,58sleep33023-21fschecks_count11:02:543
2955028511,60sleep329593-21ntp_offset10:43:003
1847528516,39sleep30-21swapper/311:52:483
1795628531,32sleep30-21swapper/308:23:063
1535328511,31sleep20-21swapper/211:42:462
109682857,12sleep30-21swapper/309:43:083
2272928411,19sleep00-21swapper/010:22:380
1749128418,49sleep00-21swapper/008:22:510
1262328417,47sleep112230-21/usr/sbin/munin11:33:001
3090028316,49sleep1225-21systemd-journal12:32:351
479228215,43sleep34773-21latency_hist07:42:373
479228215,43sleep34773-21latency_hist07:42:373
2792428219,47sleep00-21swapper/007:12:420
2586628216,52sleep125891-21netstat08:47:591
1076328217,49sleep110837-21cat09:43:061
2728128120,45sleep327321-21awk10:37:353
1464728120,31sleep30-21swapper/309:57:373
1464728120,31sleep30-21swapper/309:57:373
932128016,54sleep00-21swapper/007:56:470
932128016,54sleep00-21swapper/007:56:470
3011028022,38sleep230174-21ps12:28:042
2821928014,50sleep128250-21fschecks_count07:12:561
2353628015,16sleep20-21swapper/212:08:052
1874828025,37sleep118812-21cat11:52:581
2654527910,34sleep10-21swapper/112:17:491
2549927923,38sleep00-21swapper/008:47:500
866427816,24sleep30-21swapper/309:37:513
565627819,11sleep00-21swapper/011:11:390
385327822,26sleep20-21swapper/207:38:002
385327822,26sleep20-21swapper/207:38:002
2224727818,39sleep222287-21fschecks_count08:37:552
104027815,44sleep21179-21unixbench_multi07:28:062
104027815,44sleep21179-21unixbench_multi07:28:062
1614927717,26sleep30-21swapper/311:43:073
580527617,43sleep35841-21latency09:27:583
3239827612,49sleep232458-21fw_packets10:52:562
305532767,15sleep10-21swapper/107:18:331
3152827522,30sleep20-21swapper/207:22:582
3152827522,30sleep20-21swapper/207:22:582
1795727518,10sleep00-21swapper/011:49:380
143072750,7sleep10-21swapper/108:12:371
822027418,24sleep20-21swapper/207:52:572
822027418,24sleep20-21swapper/207:52:572
2998527419,39sleep230137-21cut09:02:382
2818327418,41sleep228073-21/usr/sbin/munin12:22:552
2611027429,28sleep10-21swapper/110:32:561
809927311,13sleep30-21swapper/311:18:043
2192427313,17sleep10-21swapper/108:37:381
267827229,26sleep20-21swapper/211:02:372
1514027217,30sleep20-21swapper/211:42:382
1443427213,25sleep30-21swapper/311:38:073
1246327220,20sleep20-21swapper/211:32:572
2872227112,12sleep30-21swapper/308:57:553
2243227117,36sleep122468-21cat08:37:581
2137027124,31sleep221422-21ls12:02:362
3019926912,27sleep10-21swapper/109:02:531
240472696,14sleep10-21swapper/108:43:111
2195626911,41sleep122042-21threads10:18:061
2146226922,21sleep30-21swapper/310:17:563
1741426917,34sleep217476-21/usr/sbin/munin11:48:072
113482690,7sleep20-21swapper/208:02:582
1048226912,35sleep210576-21/usr/sbin/munin09:42:592
987426815,26sleep10-21swapper/111:23:071
830826819,25sleep10-21swapper/109:37:091
290026822,28sleep30-21swapper/309:18:053
2446626815,38sleep324503-21ps08:43:453
2080326825,21sleep10-21swapper/110:14:351
316626715,33sleep23229-21sendmail-msp07:37:352
316626715,33sleep23229-21sendmail-msp07:37:352
2896326715,25sleep30-21swapper/307:13:103
2291726720,30sleep222985-21/usr/sbin/munin10:22:522
1940526713,36sleep119418-21/usr/sbin/munin08:28:011
1940526713,36sleep119418-21/usr/sbin/munin08:28:011
915226619,29sleep39148-21apt-get11:22:493
75342666,51sleep10-21swapper/107:48:411
75342666,51sleep10-21swapper/107:48:411
638026616,33sleep36526-21latency_hist07:47:383
602226622,28sleep36029-21grep11:12:513
2891226613,28sleep10-21swapper/110:42:371
2568226614,36sleep225740-21if_err_eth008:47:562
1986626619,29sleep219909-21fschecks_count10:12:552
958426511,42sleep20-21swapper/209:38:152
684526517,29sleep16907-21latency_hist09:32:361
311326524,18sleep20-21swapper/209:18:082
2664926524,18sleep30-21swapper/307:08:053
2664926524,18sleep30-21swapper/307:08:053
155426523,19sleep30-21swapper/310:57:583
658526414,23sleep20-21swapper/207:47:532
658526414,23sleep20-21swapper/207:47:532
3229326414,34sleep132362-21/usr/sbin/munin09:08:061
3095126423,20sleep30-21swapper/312:32:373
3072626417,26sleep20-21swapper/209:03:062
3046626428,20sleep30-21swapper/309:02:593
2681126422,20sleep30-21swapper/308:52:373
2562626425,19sleep10-21swapper/112:13:091
169702646,11sleep30-21swapper/308:19:393
998126318,29sleep10-21swapper/107:58:011
886426322,27sleep21-21systemd11:22:352
55002639,41sleep25527-21proc_pri07:43:042
55002639,41sleep25527-21proc_pri07:43:042
459126315,24sleep10-21swapper/109:23:201
347226317,22sleep20-21swapper/211:03:052
31162637,13sleep10-21swapper/111:02:561
1172426318,28sleep311730-21sshd08:03:083
3210326219,23sleep20-21swapper/212:33:082
3102926212,22sleep30-21swapper/307:22:383
2685126216,33sleep226977-21timerwakeupswit07:08:082
2685126216,33sleep226977-21timerwakeupswit07:08:082
2373626215,25sleep223436-21apt-get08:42:392
2119226222,24sleep321278-21timerandwakeup08:33:073
1954126221,22sleep10-21swapper/110:12:371
1541926217,29sleep315490-21/usr/sbin/munin09:58:063
2869026113,26sleep10-21swapper/108:57:521
2020326111,22sleep30-21swapper/311:57:573
201302615,12sleep20-21swapper/208:30:282
201302615,12sleep20-21swapper/208:30:282
1534626114,29sleep115353-21/usr/sbin/munin09:58:041
1301926120,26sleep213051-21/usr/sbin/munin09:52:362
117472618,12sleep10-21swapper/108:03:101
2886026020,17sleep30-21swapper/310:42:353
1429826015,25sleep10-21swapper/111:38:051
113226024,18sleep30-21swapper/310:57:373
988025914,31sleep39936-21basename07:57:593
3270825920,24sleep132709-21ntp_states10:53:031
238825912,28sleep10-21swapper/107:33:061
238825912,28sleep10-21swapper/107:33:061
1654925915,29sleep116626-21/usr/sbin/munin08:18:081
735425819,27sleep27432-21sed09:32:582
629125817,27sleep26361-21latency11:12:592
34725814,26sleep10-21swapper/107:27:491
34725814,26sleep10-21swapper/107:27:491
152425816,27sleep31650-21/usr/sbin/munin09:13:043
1513425819,19sleep20-21swapper/208:13:082
1513425819,19sleep20-21swapper/208:13:082
1484325819,22sleep30-21swapper/308:13:053
1484325819,22sleep30-21swapper/308:13:053
754525715,23sleep30-21swapper/309:33:033
3135225713,29sleep331375-21fschecks_count07:22:553
3135225713,29sleep331375-21fschecks_count07:22:553
2422525718,19sleep30-21swapper/310:27:373
2216325717,22sleep322230-21df08:37:523
1746625719,17sleep10-21swapper/108:22:481
1545725712,29sleep115464-21diskstats11:42:531
1411225716,20sleep30-21swapper/308:12:363
3126725612,18sleep10-21swapper/112:32:511
2865025619,23sleep10-21swapper/112:23:061
2706325616,25sleep327130-21df08:52:543
1331025614,18sleep20-21swapper/209:52:522
1331025614,18sleep20-21swapper/209:52:522
492625516,19sleep20-21swapper/211:08:022
480425510,17sleep30-21swapper/309:23:223
3195925515,26sleep231982-21/usr/sbin/munin10:52:362
2654625514,24sleep226540-21apt-get12:17:502
2526325514,22sleep125363-21users10:28:081
2526325514,22sleep125363-21users10:28:081
2505125516,21sleep20-21swapper/212:12:582
1176625514,24sleep10-21swapper/109:47:541
1045525518,15sleep20-21swapper/211:27:362
3133825414,17sleep10-21swapper/107:22:531
3133825414,17sleep10-21swapper/107:22:531
2459825413,23sleep224594-21apt-get07:02:522
2402325414,25sleep324097-21find12:08:193
2300025413,18sleep20-21swapper/212:07:372
2132225412,28sleep121369-21systemd-run12:02:351
3175825313,24sleep231752-21apt-get09:07:512
3069925313,24sleep130694-21apt-get10:47:501
2058525314,22sleep220581-21apt-get08:32:502
143125320,17sleep20-21swapper/209:13:032
1089325316,19sleep10-21swapper/111:27:551
866225213,23sleep18657-21apt-get09:37:491
33725211,21sleep3341-21apt-get07:27:473
33725211,21sleep3341-21apt-get07:27:473
252025215,21sleep12524-21diskstats09:17:551
71802517,33sleep10-21swapper/111:16:131
574625111,22sleep25804-21irqstats09:27:572
2969325112,23sleep229712-21apt-get07:17:492
1231125118,16sleep30-21swapper/309:48:093
659525010,16sleep30-21swapper/307:47:543
659525010,16sleep30-21swapper/307:47:543
2818925016,19sleep30-21swapper/312:22:563
1847925012,17sleep20-21swapper/211:52:512
2181924913,18sleep10-21swapper/112:02:561
1834024912,21sleep318437-21perl10:07:553
1317824912,21sleep20-21swapper/208:07:592
1903724610,18sleep30-21swapper/308:27:513
1903724610,18sleep30-21swapper/308:27:513
1678924414,16sleep20-21swapper/210:03:102
1678924414,16sleep20-21swapper/210:03:102
107782447,14sleep20-21swapper/211:27:522
502924310,16sleep30-21swapper/307:42:493
171802408,16sleep30-21swapper/311:48:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional