You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 18:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sun Nov 23, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
182952101725,980sleep018334-21ntpq12:24:010
21925277825,733sleep321977-21/usr/sbin/munin09:08:553
19559246870,380sleep019573-21fschecks_count12:28:530
19559246870,380sleep019573-21fschecks_count12:28:530
10152237634,327sleep110221-21/usr/sbin/munin10:14:071
10152237634,327sleep110221-21/usr/sbin/munin10:14:071
5479236157,127sleep05541-21sshd11:43:550
19285235317,320sleep219329-21ntpq10:44:002
22553226569,152sleep022590-21latency07:28:570
8862226073,160sleep00-21swapper/008:28:540
226622606,203sleep00-21swapper/011:33:450
18019225853,187sleep018023-21users08:54:420
18019225853,187sleep018023-21users08:54:420
5384225713,229sleep25413-21awk08:18:342
5384225713,229sleep25413-21awk08:18:342
62622556,233sleep00-21swapper/009:44:050
62622556,233sleep00-21swapper/009:44:050
23371225219,113sleep023433-21munin-run10:58:360
29916224759,178sleep0362-21snmpd07:52:230
15564223730,31sleep015694-21sort10:33:360
7086222752,119sleep00-21swapper/010:04:080
21633222522,28sleep00-21swapper/007:24:120
21633222522,28sleep00-21swapper/007:24:120
29527221917,189sleep229541-21cat09:33:452
2339219917,164sleep32461-21wc08:08:373
2339219917,164sleep32461-21wc08:08:373
12115219628,109sleep00-21swapper/008:38:470
297642188102,23sleep00-21swapper/009:33:550
885218617,152sleep21005-21sed09:44:112
885218617,152sleep21005-21sed09:44:112
15012218244,80sleep00-21swapper/008:48:360
1839217629,134sleep01877-21grep09:48:540
30600216842,114sleep030674-21cat07:53:560
2537821639,111sleep00-21swapper/011:03:490
10356216364,24sleep00-21swapper/008:33:500
26401216182,54sleep00-21swapper/009:23:460
16410216172,12sleep00-21swapper/007:09:300
23596215912,74sleep023673-21latency09:13:560
18906215910,99sleep00-21swapper/010:43:500
5468215826,93sleep05540-21wc09:59:070
25639215769,74sleep00-21swapper/009:19:090
9979214846,84sleep010023-21ntpq10:14:010
9979214846,84sleep010023-21ntpq10:14:010
750214632,96sleep0872-21sshd08:03:380
2235421438,12sleep30-21swapper/307:28:513
18901214356,72sleep00-21swapper/008:58:580
16681214143,37sleep00-21swapper/008:53:370
16681214143,37sleep00-21swapper/008:53:370
14081213421,98sleep00-21swapper/007:05:060
9251213319,95sleep1313-21rs:main1
7206213113,71sleep00-21swapper/011:48:560
27612212943,43sleep00-21swapper/011:09:100
14912212818,87sleep315034-21/usr/sbin/munin10:29:073
4205212742,38sleep00-21swapper/011:38:580
25124212752,60sleep025141-21idleruntime-cro07:38:340
25124212752,60sleep025141-21idleruntime-cro07:38:340
12817212731,27sleep012887-21idleruntime10:23:550
32620212415,88sleep232756-21grep07:59:072
17103212131,34sleep017249-21head12:19:040
13477212135,43sleep00-21swapper/007:03:530
13477212135,43sleep00-21swapper/007:03:530
8454211924,76sleep00-21swapper/010:09:050
7359211813,86sleep37421-21/usr/sbin/munin11:48:593
18138211811,26sleep00-21swapper/010:39:050
13101211832,62sleep313198-21dpkg07:03:353
13101211832,62sleep313198-21dpkg07:03:353
9311211724,75sleep29452-21grep11:54:062
224721178,22sleep02273-21munin-run08:08:340
224721178,22sleep02273-21munin-run08:08:340
3477211462,14sleep00-21swapper/009:53:540
3477211462,14sleep00-21swapper/009:53:540
21202211017,69sleep320711-21/usr/sbin/munin07:24:073
21202211017,69sleep320711-21/usr/sbin/munin07:24:073
17062211028,63sleep117083-21awk10:38:341
2541421097,62sleep00-21swapper/007:38:500
23793210921,73sleep123318-21apt-get10:58:451
20543210817,67sleep120609-21fschecks_time10:48:561
13492210823,63sleep113428-21apt-get07:04:021
13492210823,63sleep113428-21apt-get07:04:021
24546210623,65sleep224674-21unixbench_multi07:34:082
24546210623,65sleep224674-21unixbench_multi07:34:082
1406021060,9sleep10-21swapper/112:09:071
12641210540,16sleep00-21swapper/012:04:470
22247210432,17sleep00-21swapper/009:09:040
1489210418,68sleep30-21swapper/311:29:083
1489210418,68sleep30-21swapper/311:29:083
7453210318,70sleep27485-21proc_pri11:49:022
13230210314,13sleep00-21swapper/012:08:500
2791021025,13sleep00-21swapper/007:44:350
2791021025,13sleep00-21swapper/007:44:350
19718210211,75sleep319800-21mailstats07:19:393
19718210211,75sleep319800-21mailstats07:19:393
12375210216,61sleep312402-21munin-run10:23:333
4215210126,25sleep04277-21grep08:13:550
4215210126,25sleep04277-21grep08:13:550
31862210121,17sleep00-21swapper/009:39:080
25874210013,72sleep125895-21awk07:39:031
25087210012,74sleep125089-21sshd07:38:101
25087210012,74sleep125089-21sshd07:38:101
12133210014,66sleep112135-21apt-get08:38:481
183562985,13sleep00-21swapper/007:14:470
1328329815,69sleep313376-21cat10:24:063
2021129725,51sleep120352-21grep12:29:071
2021129725,51sleep120352-21grep12:29:071
1886229718,62sleep218990-21sort07:18:372
152002977,52sleep015248-21ntpq12:13:590
152002977,52sleep015248-21ntpq12:13:590
588629623,16sleep00-21swapper/008:18:560
41852968,14sleep10-21swapper/109:54:081
41852968,14sleep10-21swapper/109:54:081
1218329521,57sleep212227-21ntp_offset06:59:072
1218329521,57sleep212227-21ntp_offset06:59:072
1204529539,21sleep0360-21ntpd10:19:140
139129417,57sleep11433-21ntpq08:04:031
139129417,57sleep11433-21ntpq08:04:031
1175229416,54sleep211952-21latency_hist08:38:372
1651129318,59sleep21-21systemd08:53:352
1651129318,59sleep21-21systemd08:53:352
1214029315,54sleep211653-21/usr/sbin/munin12:04:042
2517129230,16sleep00-21swapper/011:03:370
47629122,14sleep00-21swapper/011:28:360
47629122,14sleep00-21swapper/011:28:360
3051329115,59sleep230552-21sort07:53:542
2872029111,60sleep20-21swapper/207:48:452
459129028,42sleep34616-21munin-run09:58:343
459129028,42sleep34616-21munin-run09:58:343
2899129045,11sleep00-21swapper/011:14:020
2899129045,11sleep00-21swapper/011:14:020
635628919,49sleep26510-21grep08:19:062
207262899,17sleep00-21swapper/010:48:590
1706228917,55sleep217058-21apt-get08:53:472
1706228917,55sleep217058-21apt-get08:53:472
280982889,43sleep028183-21perl09:28:550
2264728814,58sleep322753-21ls10:54:063
933128712,59sleep39372-21latency_hist10:13:363
2963428717,51sleep129639-21/usr/sbin/munin09:33:521
2274528516,50sleep222260-21/usr/sbin/munin07:29:032
1909128518,53sleep119104-21open_files08:59:031
1398728525,31sleep10-21swapper/110:28:361
112128533,14sleep00-21swapper/011:29:010
112128533,14sleep00-21swapper/011:29:010
1995828417,32sleep20-21swapper/212:29:042
1995828417,32sleep20-21swapper/212:29:042
1758128410,62sleep317611-21proc_pri08:54:373
1758128410,62sleep317611-21proc_pri08:54:373
1630328418,49sleep216326-21/usr/sbin/munin07:09:282
1766528313,51sleep317759-21memory07:13:573
1012828310,60sleep110131-21grep11:58:501
594428213,55sleep15983-21expr08:18:571
1947328110,17sleep30-21swapper/312:28:483
1947328110,17sleep30-21swapper/312:28:483
1450228123,37sleep314526-21apt12:13:343
412728021,44sleep34122-21apt-get08:13:503
412728021,44sleep34122-21apt-get08:13:503
2354428018,41sleep123583-21awk07:33:341
2097628013,51sleep320170-21/usr/sbin/munin09:04:103
72232797,55sleep00-21swapper/008:23:490
287027923,25sleep30-21swapper/311:34:043
322342785,60sleep10-21swapper/109:41:561
1923127846,16sleep00-21swapper/007:19:240
1923127846,16sleep00-21swapper/007:19:240
905327715,49sleep39120-21run-parts11:54:033
201932778,56sleep020189-21apt-get09:03:470
1496127723,30sleep20-21swapper/208:48:342
1166527722,31sleep10-21swapper/108:38:361
1087127722,36sleep211004-21grep11:59:072
69727625,27sleep10-21swapper/108:03:341
50922767,56sleep10-21swapper/108:14:491
50922767,56sleep10-21swapper/108:14:491
2923127619,38sleep329259-21modprobe09:33:343
2872527616,28sleep10-21swapper/107:48:481
1915127623,30sleep219170-21apt-get07:18:452
1915127623,30sleep219170-21apt-get07:18:452
176032766,54sleep10-21swapper/107:13:551
765227517,41sleep10-21swapper/111:49:051
562727520,29sleep30-21swapper/309:59:103
2494327511,49sleep224944-21sshd11:01:452
1789927513,46sleep317920-21apt-get12:23:483
146027528,31sleep11461-21cron09:48:341
146027528,31sleep11461-21cron09:48:341
1228227517,42sleep112356-21find12:04:071
2401227416,17sleep30-21swapper/310:58:553
2174327424,21sleep20-21swapper/209:08:492
1433127418,40sleep114427-21timerandwakeup08:44:171
671127328,22sleep30-21swapper/310:03:593
388827318,26sleep20-21swapper/208:13:372
3033227324,25sleep10-21swapper/107:53:491
1492627312,43sleep315043-21apt-key07:08:363
1491827324,29sleep30-21swapper/312:13:543
1491827324,29sleep30-21swapper/312:13:543
1245927218,24sleep30-21swapper/308:39:353
85672718,46sleep00-21swapper/011:53:480
300612719,17sleep00-21swapper/011:18:470
2837727119,27sleep20-21swapper/209:29:012
1762227118,36sleep217653-21/usr/sbin/munin10:38:552
1743927120,24sleep30-21swapper/310:38:523
1555127112,30sleep10-21swapper/107:08:421
1487127132,27sleep214917-21apt-key07:08:352
517527018,27sleep20-21swapper/211:43:362
504027013,35sleep25081-21cut09:58:572
2601026918,24sleep10-21swapper/111:04:061
2557026917,29sleep30-21swapper/311:03:563
241826917,26sleep10-21swapper/111:33:541
2341326912,41sleep123492-21fschecks_time09:13:531
1996826916,25sleep20-21swapper/209:03:362
877526816,32sleep38770-21apt-get08:28:483
421926826,24sleep10-21swapper/111:38:591
215126816,38sleep22187-21ntpq09:49:002
1768126816,37sleep317658-21latency_hist12:23:353
1004026813,41sleep310064-21proc_pri10:14:053
1004026813,41sleep310064-21proc_pri10:14:053
3111026712,26sleep30-21swapper/309:38:513
2977326713,36sleep329797-21munin-run11:18:343
2977326713,36sleep329797-21munin-run11:18:343
2836726720,30sleep20-21swapper/207:48:372
2836726720,30sleep20-21swapper/207:48:372
2679526712,23sleep10-21swapper/109:23:591
2618426711,40sleep326149-21latency_hist09:23:363
2020026711,41sleep10-21swapper/110:48:351
1846226714,21sleep10-21swapper/112:24:061
1545826717,32sleep314887-21apt-get07:08:403
853426619,24sleep30-21swapper/308:28:363
3237126615,35sleep331866-21/usr/sbin/munin07:59:043
2846126622,19sleep30-21swapper/307:48:383
2846126622,19sleep30-21swapper/307:48:383
2071026619,28sleep220729-21apt-get07:23:462
2071026619,28sleep220729-21apt-get07:23:462
1402526616,22sleep20-21swapper/210:28:372
1159326614,36sleep211622-21ps10:19:042
566426522,22sleep30-21swapper/311:43:583
326126520,24sleep10-21swapper/108:09:081
299426538,13sleep00-21swapper/008:09:030
2564926514,22sleep30-21swapper/307:38:573
2077426517,21sleep10-21swapper/109:04:081
1687426521,34sleep00-21swapper/010:36:060
2201326417,24sleep10-21swapper/109:08:571
2056526421,27sleep220588-21netstat09:04:012
113522646,12sleep00-21swapper/012:03:090
380526219,29sleep33835-21munin-run08:13:343
3040426217,26sleep30-21swapper/307:53:513
2358226214,22sleep20-21swapper/210:58:382
1180026214,23sleep10-21swapper/110:19:071
585726116,30sleep15889-21proc_pri11:44:041
2019926111,17sleep10-21swapper/107:20:261
2019926111,17sleep10-21swapper/107:20:261
1678626114,29sleep216821-21ntp_states12:18:582
589026015,21sleep20-21swapper/211:44:052
3064426014,28sleep10-21swapper/111:19:041
3006526018,28sleep230038-21/usr/sbin/munin11:18:492
2773326019,23sleep20-21swapper/209:28:362
2750126017,23sleep10-21swapper/107:44:081
2750126017,23sleep10-21swapper/107:44:081
2198526010,11sleep10-21swapper/107:28:251
2198526010,11sleep10-21swapper/107:28:251
2170826014,19sleep30-21swapper/310:53:343
1711426022,17sleep30-21swapper/310:38:363
3081325912,30sleep330518-21kworker/3:009:38:353
2508325914,30sleep125137-21if_eth009:18:581
230625924,17sleep30-21swapper/309:49:053
185342596,38sleep20-21swapper/212:24:072
1659325916,25sleep316678-21apt-config08:53:363
1659325916,25sleep316678-21apt-config08:53:363
1543325923,15sleep20-21swapper/212:14:052
1543325923,15sleep20-21swapper/212:14:052
1095925917,19sleep30-21swapper/308:34:053
488725815,27sleep14881-21apt-get09:58:481
1346425815,28sleep213604-21latency_hist08:43:372
914425718,18sleep10-21swapper/108:28:591
767725717,23sleep17683-21ntp_states08:24:021
651425714,23sleep30-21swapper/308:19:073
3153825715,23sleep231562-21ntp_states09:39:022
205025719,23sleep22089-21sed11:33:352
205025719,23sleep22089-21sed11:33:352
1756425716,23sleep117621-21irqstats10:38:541
2398625614,18sleep30-21swapper/307:33:563
2398625614,18sleep30-21swapper/307:33:563
107752568,10sleep20-21swapper/210:17:512
107752568,10sleep20-21swapper/210:17:512
734125518,17sleep20-21swapper/210:04:102
731825517,20sleep20-21swapper/208:23:552
644525515,23sleep16515-21df10:03:531
3161525514,25sleep131605-21apt-get11:23:491
3161525514,25sleep131605-21apt-get11:23:491
2545125519,16sleep20-21swapper/209:19:072
1580825512,26sleep215801-21apt-get10:33:502
360025419,15sleep20-21swapper/211:38:362
2851325413,24sleep228505-21apt-get11:13:482
2851325413,24sleep228505-21apt-get11:13:482
2738225415,25sleep30-21swapper/311:09:053
1444925414,22sleep114501-21irqstats10:28:571
1011525415,16sleep20-21swapper/208:33:362
341025316,21sleep23414-21diskstats09:53:522
341025316,21sleep23414-21diskstats09:53:522
301892539,19sleep30-21swapper/311:18:543
1412025313,24sleep114143-21ntp_states07:05:071
2960225214,20sleep329702-21users07:49:433
2843925216,23sleep128441-21sh09:29:041
2549425216,24sleep225569-21grep11:03:552
2374225213,15sleep30-21swapper/309:13:593
212025212,19sleep10-21swapper/109:48:591
190452528,19sleep30-21swapper/308:59:013
1769325214,17sleep10-21swapper/108:54:391
1769325214,17sleep10-21swapper/108:54:391
1762925215,22sleep117664-21munin-run12:23:331
1552425215,18sleep10-21swapper/108:48:581
1552425215,18sleep10-21swapper/108:48:581
1431025213,23sleep214314-21diskstats10:28:542
296982517,33sleep10-21swapper/111:17:351
296982517,33sleep10-21swapper/111:17:351
2332425113,23sleep223321-21apt-get09:13:462
1499125114,21sleep315010-21munin-run08:48:353
145012515,34sleep20-21swapper/212:13:332
2347825012,23sleep123581-21gpgv10:58:371
193312509,15sleep30-21swapper/310:44:013
3188924915,16sleep20-21swapper/211:23:582
3188924915,16sleep20-21swapper/211:23:582
3161724913,20sleep331684-21df11:23:513
3161724913,20sleep331684-21df11:23:513
2827424915,21sleep20-21swapper/211:13:362
2820824916,16sleep10-21swapper/111:13:351
151024917,17sleep30-21swapper/309:48:363
151024917,17sleep30-21swapper/309:48:363
1185924918,17sleep30-21swapper/312:03:573
1012424911,21sleep310118-21apt-get11:58:483
143524712,20sleep21446-21ntp_states08:04:042
143524712,20sleep21446-21ntp_states08:04:042
2686124612,18sleep30-21swapper/309:24:013
71524511,17sleep10-21swapper/111:28:491
71524511,17sleep10-21swapper/111:28:491
148982457,14sleep10-21swapper/112:13:521
148982457,14sleep10-21swapper/112:13:521
82402449,17sleep20-21swapper/210:08:592
856824311,16sleep10-21swapper/111:53:491
42972398,14sleep30-21swapper/311:39:013
2139923911,15sleep20-21swapper/210:49:332
262872377,14sleep20-21swapper/207:39:082
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional