You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-20 - 15:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.15.26-xenomai (Profile)

Latency plot of shadow in rack #b, slot #7
Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackbslot7s (updated Mon Mar 20, 2023 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
345029647,943sleep13461-21ntpq10:34:431
1709228167,793sleep117131-21ntpq09:34:451
650125596,485sleep16537-21ntpq10:44:441
24558246217,428sleep224503-21latency_hist11:44:142
21354228755,212sleep121385-21munin-plugin-st11:34:121
21354228755,212sleep121385-21munin-plugin-st11:34:121
19808225965,170sleep119833-21kworker/u8:011:29:121
1510022288,209sleep031289-1kworker/0:1H09:29:310
1916221821,181sleep11987-21fschecks_time07:04:341
21403221211,187sleep021440-21fschecks_count09:49:330
11876221113,182sleep011933-21awk07:34:360
11876221113,182sleep011933-21awk07:34:360
26151220121,104sleep126167-21latency_hist11:49:141
87522009,178sleep0369-21snmpd10:26:020
540219543,111sleep1645-21timerwakeupswit10:24:501
30152219510,23sleep10-21swapper/111:59:521
9335219112,162sleep39431-21fschecks_time10:54:433
11675218523,103sleep10-21swapper/109:19:151
13213217688,70sleep10-21swapper/107:39:291
2759321739,150sleep127629-21ntpq08:24:421
8286217111,84sleep10-21swapper/110:49:491
2033821689,123sleep120414-21sed09:44:541
19641216313,134sleep019416-21/usr/sbin/munin07:59:340
21646215969,65sleep10-21swapper/111:34:251
1272721579,120sleep112759-21awk11:04:461
2099621568,133sleep021007-21vmstat11:29:470
2099621568,133sleep021007-21vmstat11:29:470
29033215563,67sleep229071-21wc08:29:352
29304214816,26sleep30-21swapper/310:14:353
25882214319,105sleep126037-21latency_hist10:04:161
5142214018,88sleep05172-21/usr/sbin/munin12:24:410
1174214011,113sleep21246-21latency_hist10:29:152
6915213812,112sleep06946-21grep07:19:300
4642213854,14sleep14680-21grep10:39:391
32626213817,90sleep0302-21grep08:39:510
24294213733,87sleep124401-21latency_hist09:59:151
6148213320,98sleep36187-21ps08:59:473
6148213320,98sleep36187-21ps08:59:473
1652621339,24sleep10-21swapper/107:49:381
1652621339,24sleep10-21swapper/107:49:381
32338213210,32sleep132355-21apt12:09:121
28379213218,15sleep10-21swapper/111:54:431
487321319,19sleep10-21swapper/112:24:341
21996213114,53sleep122106-21cat09:49:471
31023212912,95sleep130740-21/usr/sbin/munin10:19:381
28117212844,69sleep10-21swapper/110:09:451
1844212714,22sleep10-21swapper/112:14:331
2167212518,89sleep32294-21grep10:29:493
30019212316,86sleep030083-21modprobe08:34:130
10479212362,26sleep1369-21snmpd10:59:091
23910212228,26sleep123829-21latency_hist08:14:161
24527212116,46sleep224566-21/usr/sbin/munin08:14:422
22737212117,38sleep222600-21apt-get09:54:162
11918212119,86sleep011985-21/usr/sbin/munin09:19:300
13464211863,18sleep113466-21apt-get09:24:321
11247211811,88sleep211285-21/usr/sbin/munin10:59:472
341821178,93sleep13527-21munin-node07:09:181
6675211616,82sleep16681-21latency_hist07:19:151
1008621166,86sleep110083-21apt-get07:29:281
7181211412,88sleep37198-21kernelversion07:19:373
7358211359,18sleep17444-21perl09:04:341
6360211214,22sleep20-21swapper/208:59:512
6360211214,22sleep20-21swapper/208:59:512
26834211130,45sleep10-21swapper/108:23:041
16765211116,77sleep016841-21latency_hist11:19:140
16765211116,77sleep016841-21latency_hist11:19:140
26494211018,65sleep126583-21fschecks_time11:49:341
79421088,86sleep1946-21modprobe08:44:151
1895021075,11sleep10-21swapper/107:56:411
30135210621,63sleep330288-21latency_hist08:34:163
18315210515,73sleep018470-21date11:24:140
18315210515,73sleep018470-21date11:24:140
13956210519,60sleep10-21swapper/111:09:341
13956210519,60sleep10-21swapper/111:09:341
814021045,55sleep10-21swapper/107:23:551
2759210418,58sleep10-21swapper/110:34:151
14262210415,75sleep314293-21ntp_kernel_pll_11:09:403
14262210415,75sleep314293-21ntp_kernel_pll_11:09:403
23837210212,21sleep123935-21timerandwakeup11:39:541
2203721025,85sleep10-21swapper/108:06:341
18560210214,70sleep00-21swapper/011:24:350
23056210112,73sleep323066-21/usr/sbin/munin09:54:373
18726210135,20sleep10-21swapper/109:39:461
9555210010,71sleep10-21swapper/107:26:091
6102210028,44sleep26098-21apt-get10:44:332
27657210015,68sleep327696-21basename11:54:123
1671321005,10sleep10-21swapper/111:19:111
1671321005,10sleep10-21swapper/111:19:111
134721005,58sleep10-21swapper/112:12:591
4392995,10sleep10-21swapper/108:41:051
302802996,58sleep10-21swapper/110:16:431
251232995,10sleep10-21swapper/108:15:581
1365929919,62sleep313713-21/usr/sbin/munin09:24:423
405029716,24sleep10-21swapper/112:19:541
3155129612,72sleep331572-21awk12:04:503
261362967,15sleep10-21swapper/110:04:321
2447629626,53sleep024508-21munin-plugin-st11:44:120
2001329613,65sleep10-21swapper/107:59:431
75392955,56sleep10-21swapper/112:31:241
2616829517,54sleep226323-21grep11:49:152
141492957,74sleep014150-21sshd07:40:090
1849029415,65sleep218518-21kernelversion09:39:382
1249729411,67sleep312588-21cat09:19:463
965629311,70sleep0343-21lldpd07:27:250
600929313,16sleep00-21swapper/007:14:460
585629310,70sleep05914-21latency_hist10:44:150
3090129315,61sleep230982-21sed08:34:442
2658629317,60sleep226644-21/usr/sbin/munin11:49:352
178532938,17sleep10-21swapper/111:19:481
178532938,17sleep10-21swapper/111:19:481
120332939,16sleep112082-21systemd-run11:04:131
732329259,17sleep00-21swapper/009:04:330
25422928,15sleep10-21swapper/108:49:281
223629213,63sleep02259-21pluginstate12:14:440
2145529211,66sleep021491-21grep08:04:430
1198629212,12sleep10-21swapper/109:19:311
1461329124,44sleep214755-21grep11:09:482
1461329124,44sleep214755-21grep11:09:482
69342908,69sleep16975-21munin-run09:04:131
69342908,69sleep16975-21munin-run09:04:131
775528816,53sleep37816-21modprobe12:34:123
940828721,45sleep29511-21grep09:09:522
1522828712,11sleep10-21swapper/107:44:421
1522828712,11sleep10-21swapper/107:44:421
43952868,58sleep10-21swapper/108:54:371
763028519,30sleep20-21swapper/209:04:392
342628513,19sleep00-21swapper/012:19:400
303382855,10sleep00-21swapper/010:17:270
123332850,7sleep20-21swapper/209:19:392
150328428,34sleep11536-21munin-run07:04:131
2648228312,56sleep026493-21awk11:49:330
1640328319,44sleep316489-21grep07:49:363
1640328319,44sleep316489-21grep07:49:363
1506228317,47sleep314737-21apt-get09:29:223
119342837,14sleep10-21swapper/107:34:371
119342837,14sleep10-21swapper/107:34:371
308828222,30sleep20-21swapper/210:34:362
3078928223,38sleep230798-21modprobe12:04:122
2790428217,31sleep00-21swapper/008:24:490
2330028211,55sleep323307-21sh11:39:393
97302815,10sleep10-21swapper/109:10:111
830428025,24sleep20-21swapper/207:24:162
78002800,6sleep30-21swapper/309:04:433
314582805,41sleep10-21swapper/108:37:251
3042628011,57sleep330450-21fschecks_count08:34:333
1867627835,22sleep30-21swapper/309:39:443
1798627816,41sleep318023-21fschecks_count07:54:453
162832786,11sleep10-21swapper/109:33:201
291602775,11sleep10-21swapper/108:29:381
186027719,24sleep30-21swapper/312:14:353
648327517,44sleep26488-21sort12:29:312
421827516,17sleep20-21swapper/208:54:342
219252755,11sleep20-21swapper/208:05:092
315127423,24sleep30-21swapper/307:09:163
2897327421,24sleep30-21swapper/310:14:153
1635227413,41sleep216382-21munin-run09:34:142
1600927413,49sleep0232-21systemd-journal07:49:090
1600927413,49sleep0232-21systemd-journal07:49:090
1203427411,47sleep312065-21/usr/sbin/munin07:34:393
1203427411,47sleep312065-21/usr/sbin/munin07:34:393
460627328,24sleep20-21swapper/210:39:362
2009827315,32sleep20-21swapper/211:29:292
2009827315,32sleep20-21swapper/211:29:292
3272327213,15sleep30-21swapper/310:24:443
3170427214,29sleep20-21swapper/212:04:532
418027114,41sleep24217-21ntp_kernel_pll_07:09:432
346627110,46sleep33496-21/usr/sbin/munin10:34:453
2953927114,29sleep00-21swapper/011:59:340
2583927126,20sleep00-21swapper/010:04:150
3271526914,31sleep3307-21/usr/sbin/munin12:09:343
169502697,15sleep20-21swapper/209:34:422
1452126922,28sleep30-21swapper/307:44:153
3142426818,31sleep231546-21timerwakeupswit10:19:472
2739326823,19sleep00-21swapper/010:09:150
2359426820,37sleep20-21swapper/208:11:352
1094926811,42sleep011016-21switchtime09:14:460
376026714,19sleep00-21swapper/007:09:290
359726717,28sleep30-21swapper/312:19:433
31426727,18sleep00-21swapper/006:59:330
2864126717,21sleep30-21swapper/311:54:493
979826621,30sleep29799-21cron07:29:132
928026619,20sleep00-21swapper/009:09:490
248182667,11sleep30-21swapper/311:44:243
2107826620,20sleep20-21swapper/209:49:152
1868826624,26sleep00-21swapper/009:39:450
1516326620,28sleep20-21swapper/207:44:392
1516326620,28sleep20-21swapper/207:44:392
1481726617,23sleep00-21swapper/009:29:150
1363026618,23sleep20-21swapper/211:09:152
1304926618,20sleep00-21swapper/011:04:530
8742657,15sleep00-21swapper/012:09:480
2856626515,26sleep30-21swapper/308:29:163
740426412,38sleep07407-21systemd10:49:150
589526418,20sleep00-21swapper/008:59:400
589526418,20sleep00-21swapper/008:59:400
576026415,20sleep20-21swapper/207:14:402
2293126416,31sleep00-21swapper/011:39:120
85926320,20sleep30-21swapper/306:59:443
290012639,12sleep00-21swapper/011:56:070
2298126320,27sleep20-21swapper/211:39:142
185322636,13sleep30-21swapper/311:24:303
1735626310,11sleep30-21swapper/309:34:513
1046026313,29sleep20-21swapper/207:29:412
321426217,21sleep00-21swapper/010:34:380
305872626,12sleep00-21swapper/010:19:160
2977726212,34sleep329807-21irqstats11:59:413
2298026216,27sleep222977-21apt-get09:54:322
209926217,24sleep00-21swapper/007:04:370
1417226215,19sleep20-21swapper/209:24:532
543726112,24sleep30-21swapper/308:59:153
2068026119,19sleep00-21swapper/009:44:570
1649226014,20sleep00-21swapper/007:49:370
1649226014,20sleep00-21swapper/007:49:370
1127426011,22sleep30-21swapper/309:14:493
241262595,11sleep00-21swapper/008:14:320
205262599,11sleep30-21swapper/309:44:563
1944125914,27sleep219435-21apt-get07:59:282
1595925912,20sleep20-21swapper/211:14:442
1595925912,20sleep20-21swapper/211:14:442
148925917,25sleep30-21swapper/308:44:423
1053925920,22sleep00-21swapper/007:29:430
854625819,21sleep30-21swapper/310:49:513
79025820,18sleep20-21swapper/206:59:432
662325813,29sleep30-21swapper/307:19:133
3051625817,28sleep2322-21dbus-daemon10:19:142
2763425814,25sleep027701-21df10:09:310
2658725813,30sleep20-21swapper/210:04:452
2636625816,19sleep20-21swapper/208:19:462
1990125817,21sleep30-21swapper/307:59:413
181752589,12sleep20-21swapper/211:23:072
181752589,12sleep20-21swapper/211:23:072
1602125815,23sleep30-21swapper/311:14:453
1602125815,23sleep30-21swapper/311:14:453
513825713,25sleep35294-21latency_hist07:14:163
3250625721,18sleep30-21swapper/308:39:503
2412425715,25sleep324118-21apt-get08:14:313
2299125714,27sleep323148-21latency_hist11:39:153
1503025715,20sleep30-21swapper/307:44:363
1503025715,20sleep30-21swapper/307:44:363
997225618,17sleep00-21swapper/010:54:560
454025612,19sleep00-21swapper/008:54:400
3034425616,16sleep00-21swapper/008:34:300
256502567,10sleep30-21swapper/310:02:243
2295725611,23sleep00-21swapper/009:54:300
66302557,14sleep00-21swapper/012:29:410
2142425512,22sleep30-21swapper/311:34:143
2142425512,22sleep30-21swapper/311:34:143
1667525514,25sleep016670-21apt-get09:34:320
1128625517,24sleep311371-21mailstats10:59:483
951325419,18sleep30-21swapper/309:09:533
769425415,23sleep27720-21df10:49:332
2855525417,21sleep20-21swapper/211:54:482
2567125413,24sleep025665-21apt-get08:19:280
2157625414,24sleep321588-21irqstats09:49:363
139025412,27sleep21408-21apt-get10:29:272
1232225413,24sleep312325-21apt-get11:04:323
1166725412,25sleep211660-21apt-get07:34:302
1166725412,25sleep211660-21apt-get07:34:302
1094625415,24sleep310993-21users07:29:513
950225316,23sleep29544-21sort10:54:462
2098925314,23sleep321056-21df08:04:303
1919125313,23sleep019200-21/usr/sbin/munin07:59:140
914725214,23sleep39245-21timerandwakeup07:24:463
791725214,20sleep00-21swapper/010:49:390
71325211,23sleep2784-21sensors_temp12:09:452
2486125212,20sleep00-21swapper/011:44:370
1761025219,17sleep30-21swapper/311:19:453
1761025219,17sleep30-21swapper/311:19:453
1413225213,19sleep00-21swapper/011:09:370
1413225213,19sleep00-21swapper/011:09:370
1083525216,19sleep20-21swapper/209:14:442
494725114,22sleep34954-21diskstats12:24:363
332725113,19sleep20-21swapper/212:19:342
3197125112,24sleep231978-21diskstats08:39:362
2852525115,19sleep20-21swapper/208:29:152
2818125118,18sleep20-21swapper/210:09:462
179825111,22sleep21896-21users08:44:452
1676025115,19sleep20-21swapper/207:49:432
1676025115,19sleep20-21swapper/207:49:432
106662515,11sleep00-21swapper/010:59:160
3134725013,23sleep031359-21irqstats12:04:430
312825011,20sleep20-21swapper/207:09:152
2261525013,22sleep322510-21/usr/sbin/munin08:09:343
1882925014,20sleep218837-21modprobe07:55:122
628124911,22sleep26454-21date12:29:152
230424910,23sleep02426-21latency_hist08:49:150
1853624910,26sleep218555-21apt-get11:24:332
2606124815,18sleep30-21swapper/308:19:393
254124811,21sleep02537-21apt-get08:49:270
53682479,15sleep30-21swapper/310:39:543
14912479,16sleep00-21swapper/010:29:330
618824614,16sleep30-21swapper/310:44:383
305124615,17sleep20-21swapper/208:49:452
2653824611,14sleep30-21swapper/310:04:423
228112467,15sleep00-21swapper/008:09:390
300124410,16sleep30-21swapper/308:49:423
250182447,13sleep20-21swapper/209:59:452
132072446,13sleep20-21swapper/207:39:262
308622409,14sleep30-21swapper/310:19:353
30792409,18sleep20-21swapper/212:19:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional