You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-10-05 - 03:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s (updated Thu Oct 05, 2023 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20269280316,8013sleep114356-21kworker/1:022:39:411
1140225248,501sleep315945-21kworker/3:123:57:103
1140225248,501sleep315945-21kworker/3:123:57:103
22817247333,401sleep122892-21mailstats22:47:531
22817247333,401sleep122892-21mailstats22:47:531
210742320102,195sleep121220-21unixbench_multi20:57:481
12081230811,280sleep11-21systemd20:32:091
1294622888,268sleep0367-21snmpd00:00:220
31500224136,83sleep131549-21latency_hist23:17:111
1644222544,122sleep10-21swapper/119:57:371
3145122239,197sleep131522-21switchtime19:47:481
2890222312,149sleep13000-21timerandwakeup23:27:571
2890222312,149sleep13000-21timerandwakeup23:27:571
14289221143,115sleep10-21swapper/122:22:091
588222107,187sleep15944-21mailstats23:37:541
588222107,187sleep15944-21mailstats23:37:541
30184220218,167sleep330266-21grep21:27:453
9064218456,91sleep19195-21unixbench_singl22:02:481
9064218456,91sleep19195-21unixbench_singl22:02:481
1312721818,160sleep113171-21date00:02:091
1896921808,95sleep10-21swapper/122:37:111
20275217850,112sleep120349-21mailstats19:12:441
12746217861,106sleep112772-21cat22:17:091
6250217752,102sleep10-21swapper/120:12:351
7757216918,133sleep10-21swapper/120:17:391
7412216989,62sleep17473-21/usr/sbin/munin23:42:471
26368216811,39sleep10-21swapper/121:17:101
2921821665,29sleep10-21swapper/121:24:451
1094821595,28sleep10-21swapper/123:53:081
1094821595,28sleep10-21swapper/123:53:081
6792215837,66sleep117586-21cyclictest21:57:311
1419215627,119sleep1367-21snmpd21:39:461
28347215229,107sleep128413-21latency_hist23:07:101
16221215112,125sleep116335-21latency_hist00:12:101
30339215030,104sleep130466-21cat21:27:481
12556215093,42sleep10-21swapper/120:32:411
9652214811,124sleep19677-21awk20:22:401
16954214832,101sleep117015-21sendmail_mailqu00:12:471
21755214310,118sleep121799-21awk19:17:451
21755214310,118sleep121799-21awk19:17:451
28991214232,60sleep10-21swapper/119:42:111
22913214120,97sleep00-21swapper/000:32:360
19305214120,103sleep319362-21grep22:37:373
15099214147,60sleep115165-21cat22:22:461
9077214011,112sleep09151-21find23:47:510
9077214011,112sleep09151-21find23:47:510
1496321379,111sleep014155-21/usr/sbin/munin20:37:570
1496321379,111sleep014155-21/usr/sbin/munin20:37:570
17494213541,60sleep217506-21modprobe19:07:082
8311213116,101sleep38313-21acpi22:02:253
8311213116,101sleep38313-21acpi22:02:253
29950213012,27sleep130061-21sort19:42:561
29950213012,27sleep130061-21sort19:42:561
27289212916,18sleep10-21swapper/121:17:501
11376212910,98sleep00-21swapper/020:27:480
4369212651,63sleep14407-21latency_hist20:07:101
2211321269,80sleep10-21swapper/100:28:291
9154212213,22sleep10-21swapper/123:47:521
9154212213,22sleep10-21swapper/123:47:521
2170721220,10sleep20-21swapper/200:27:482
26578212010,21sleep10-21swapper/119:32:461
1812321165,93sleep00-21swapper/022:32:460
1868221137,32sleep10-21swapper/119:07:471
1868221137,32sleep10-21swapper/119:07:471
5830211250,17sleep10-21swapper/121:52:501
26379211211,29sleep00-21swapper/019:32:410
2256521120,9sleep317598-21cobalt_printf00:32:093
2080211216,44sleep12125-21cat21:42:371
901211150,24sleep1959-21if_err_eth023:22:361
3710211118,79sleep03746-21cat21:47:410
26721211149,20sleep10-21swapper/123:02:101
26721211149,20sleep10-21swapper/123:02:101
24338211116,15sleep10-21swapper/122:52:491
24338211116,15sleep10-21swapper/122:52:491
613321105,93sleep00-21swapper/020:12:210
14686211012,57sleep10-21swapper/120:37:541
14686211012,57sleep10-21swapper/120:37:541
9839210922,68sleep39910-21ls20:22:463
22497210914,75sleep222559-21/usr/sbin/munin21:02:502
985721077,53sleep10-21swapper/122:07:281
985721077,53sleep10-21swapper/122:07:281
16858210718,39sleep10-21swapper/120:45:411
16220210617,69sleep015699-21/usr/sbin/munin20:42:450
20024210424,18sleep10-21swapper/100:22:401
457210312,54sleep10-21swapper/123:22:081
16354210329,55sleep10-21swapper/122:27:381
2337321028,54sleep10-21swapper/119:22:451
22029210119,26sleep30-21swapper/321:02:393
15808210115,26sleep20-21swapper/220:42:352
5813210018,38sleep10-21swapper/120:11:261
5813210018,38sleep10-21swapper/120:11:261
80229921,57sleep3815-21fschecks_count23:22:343
3068229912,72sleep230803-21cat23:12:572
2478929930,16sleep10-21swapper/121:12:091
1780629727,18sleep117883-21mailstats20:47:471
1655629719,53sleep316648-21tune2fs00:12:363
368029623,52sleep33758-21find20:02:483
2573229631,47sleep10-21swapper/119:32:101
2006229630,54sleep120093-21munin-run20:57:071
654329516,63sleep06638-21sed21:57:090
353729516,25sleep10-21swapper/121:47:381
2267529520,55sleep022816-21sed21:02:520
1843729426,22sleep30-21swapper/319:07:453
1843729426,22sleep30-21swapper/319:07:453
2909429317,26sleep129172-21ls23:07:511
1645829214,60sleep116515-21irqstats19:02:351
2568029119,50sleep325698-21awk19:32:083
2404629123,14sleep10-21swapper/100:37:091
757829014,59sleep37693-21users21:57:493
2360729015,58sleep022871-21/usr/sbin/munin19:22:480
1930729012,41sleep119345-21modprobe00:22:071
2952528810,62sleep30-21swapper/319:42:453
2952528810,62sleep30-21swapper/319:42:453
2201228818,49sleep322137-21ls19:17:513
2201228818,49sleep322137-21ls19:17:513
1250128819,51sleep312575-21threads23:58:363
120328811,38sleep10-21swapper/119:57:091
995528712,60sleep39964-21diskstats22:07:343
995528712,60sleep39964-21diskstats22:07:343
3080528721,44sleep330937-21grep23:12:583
2113228622,44sleep021157-21awk22:42:430
1067428417,52sleep010794-21grep23:52:510
1067428417,52sleep010794-21grep23:52:510
821928326,31sleep30-21swapper/323:47:093
3016028313,18sleep20-21swapper/221:27:442
2047428320,47sleep220510-21munin-run22:42:082
1579028320,42sleep315817-21munin-run22:27:083
2886528224,24sleep00-21swapper/019:42:100
2273628212,52sleep322780-21/usr/sbin/munin00:32:313
873928111,51sleep28798-21latency23:47:402
873928111,51sleep28798-21latency23:47:402
1516828117,47sleep215297-21timerwakeupswit22:22:472
1184328024,35sleep211817-21apt-get23:57:502
505127914,50sleep25055-21cron23:37:072
216822796,24sleep20-21swapper/221:02:092
489127820,41sleep24890-21gpgv21:52:112
2239427820,38sleep322449-21fschecks_count22:47:383
2239427820,38sleep322449-21fschecks_count22:47:383
2082027811,53sleep220821-21sshd00:26:442
1277427819,23sleep20-21swapper/222:17:102
809227727,32sleep00-21swapper/022:02:100
3176127711,27sleep131816-21awk21:32:441
2343327710,13sleep20-21swapper/219:22:462
222752777,13sleep00-21swapper/022:47:230
222752777,13sleep00-21swapper/022:47:230
2115827748,18sleep121188-21sshd22:42:441
1297927717,44sleep313056-21grep20:32:513
3241227611,32sleep20-21swapper/219:52:292
3119127620,27sleep20-21swapper/219:47:412
73912759,51sleep07411-21/usr/sbin/munin23:42:460
2833827512,47sleep328367-21/usr/sbin/munin21:22:373
2670527518,39sleep026768-21cut21:17:370
2549927525,31sleep025538-21proc_pri21:12:450
784127418,26sleep00-21swapper/020:17:420
470827424,26sleep20-21swapper/220:07:352
470827424,26sleep20-21swapper/220:07:352
43027418,40sleep0489-21sendmail_mailqu19:52:440
1155727418,35sleep211704-21sed20:27:502
1133227415,42sleep211377-21unin-run23:57:092
966827336,22sleep00-21swapper/022:07:110
966827336,22sleep00-21swapper/022:07:110
534527325,25sleep20-21swapper/221:52:392
2975527215,25sleep20-21swapper/223:12:082
1853327216,29sleep00-21swapper/000:17:420
1428327226,27sleep30-21swapper/320:37:443
1428327226,27sleep30-21swapper/320:37:443
1277927221,37sleep212824-21ntp_offset20:32:462
491427133,28sleep10-21swapper/123:35:201
351127112,42sleep03552-21systemd-run23:32:080
351127112,42sleep03552-21systemd-run23:32:080
274327116,35sleep22243-21/usr/sbin/munin23:27:542
274327116,35sleep22243-21/usr/sbin/munin23:27:542
2169327118,36sleep321743-21latency_hist21:02:103
133772717,49sleep10-21swapper/100:02:291
133772717,49sleep10-21swapper/100:02:291
815627019,24sleep20-21swapper/220:17:482
603427015,40sleep36153-21cat23:37:563
603427015,40sleep36153-21cat23:37:563
225182707,12sleep00-21swapper/019:21:070
225182707,12sleep00-21swapper/019:21:070
219432705,12sleep10-21swapper/121:02:321
2039227021,29sleep00-21swapper/020:57:310
1908327010,28sleep30-21swapper/320:52:373
1860327020,25sleep00-21swapper/020:52:100
1764427022,33sleep217712-21/usr/sbin/munin22:32:322
1750927022,28sleep31-21systemd19:07:093
313926922,31sleep33152-21apt21:47:073
2512226916,34sleep225145-21fschecks_count21:12:352
2473726918,35sleep324751-21apt21:12:073
2243426920,21sleep00-21swapper/000:32:070
2705726817,24sleep00-21swapper/023:02:370
1946026821,28sleep00-21swapper/020:52:470
621526717,29sleep26239-21df_inode20:12:322
46982675,11sleep00-21swapper/020:07:330
46982675,11sleep00-21swapper/020:07:330
3204426713,37sleep232086-21/usr/sbin/munin23:17:492
2094226715,26sleep30-21swapper/300:27:103
1644426719,30sleep216463-21apt-get00:12:272
1300426715,27sleep30-21swapper/322:17:313
2289426617,26sleep20-21swapper/222:47:542
2289426617,26sleep20-21swapper/222:47:542
180302667,15sleep20-21swapper/200:17:242
3164726515,30sleep331775-21uptime19:47:503
3123826515,29sleep20-21swapper/221:32:292
2508826522,28sleep325150-21sendmail-msp22:57:073
2508826522,28sleep325150-21sendmail-msp22:57:073
2013326517,34sleep30-21swapper/319:12:383
1737926510,38sleep017361-21latency_hist22:32:090
478526413,23sleep30-21swapper/320:07:363
478526413,23sleep30-21swapper/320:07:363
2879226419,23sleep20-21swapper/219:42:082
275926415,32sleep32820-21wc23:27:553
275926415,32sleep32820-21wc23:27:553
2717926415,32sleep227229-21/usr/sbin/munin23:02:392
1529526413,25sleep30-21swapper/300:07:443
3191926318,28sleep332047-21/usr/sbin/munin21:32:463
206026321,20sleep30-21swapper/319:57:473
184292638,14sleep30-21swapper/320:50:563
1756726316,18sleep00-21swapper/019:07:100
161526317,29sleep01642-21fschecks_count19:57:360
2898226225,21sleep30-21swapper/323:07:493
1912826214,29sleep019251-21date19:12:100
1912826214,29sleep019251-21date19:12:100
14332625,17sleep00-21swapper/021:39:580
1048826221,22sleep00-21swapper/022:07:470
1048826221,22sleep00-21swapper/022:07:470
585826113,19sleep00-21swapper/023:37:530
585826113,19sleep00-21swapper/023:37:530
376226121,18sleep00-21swapper/020:02:490
2891726116,19sleep00-21swapper/023:07:460
263472615,12sleep30-21swapper/319:32:403
2570226122,18sleep30-21swapper/321:12:483
2007126112,26sleep00-21swapper/000:22:430
1520126113,35sleep215294-21cut00:07:432
1368226113,24sleep00-21swapper/022:17:480
820226018,19sleep30-21swapper/320:17:493
2997026017,27sleep00-21swapper/021:27:380
2579826019,25sleep325842-21ntpq22:57:443
2579826019,25sleep325842-21ntpq22:57:443
2468726010,38sleep0367-21snmpd21:11:280
2385026026,19sleep20-21swapper/222:52:342
2385026026,19sleep20-21swapper/222:52:342
2193226011,27sleep20-21swapper/219:17:502
2193226011,27sleep20-21swapper/219:17:502
1802026018,21sleep30-21swapper/322:32:413
1683726013,33sleep016848-21netstat00:12:410
1466426020,20sleep00-21swapper/000:07:080
1466426020,20sleep00-21swapper/000:07:080
522325911,25sleep30-21swapper/321:52:323
3020825919,21sleep00-21swapper/023:12:420
2137125914,22sleep30-21swapper/322:42:503
203182596,12sleep00-21swapper/022:40:170
201825914,21sleep20-21swapper/223:27:102
18992597,12sleep00-21swapper/021:42:320
184192597,37sleep30-21swapper/300:17:373
1431425917,16sleep20-21swapper/222:22:102
1198325918,18sleep20-21swapper/223:58:182
565725816,20sleep20-21swapper/223:37:472
565725816,20sleep20-21swapper/223:37:472
41512584,17sleep20-21swapper/220:04:532
311982585,11sleep00-21swapper/021:32:150
2881925817,21sleep30-21swapper/319:42:093
14232576,12sleep00-21swapper/023:22:480
135562575,41sleep00-21swapper/020:36:230
926025614,24sleep09279-21apt-get20:22:270
420325616,21sleep00-21swapper/023:32:450
2616425617,18sleep20-21swapper/222:57:512
2616425617,18sleep20-21swapper/222:57:512
2119525611,29sleep221219-21ntp_states22:42:452
1988925610,34sleep20-21swapper/220:54:502
132025617,23sleep31223-21latency_hist19:57:113
1142325614,26sleep011417-21apt-get22:12:310
71642557,36sleep30-21swapper/320:13:583
701225517,18sleep20-21swapper/223:42:362
2854825513,24sleep028584-21ntpq21:22:420
2627925515,20sleep20-21swapper/219:32:392
2462325513,22sleep00-21swapper/022:52:520
2462325513,22sleep00-21swapper/022:52:520
2057725514,24sleep020675-21users19:12:470
1939625514,26sleep20-21swapper/219:12:122
190025510,19sleep20-21swapper/219:57:422
1700925515,24sleep317018-21munin-run20:47:083
1583025515,24sleep015843-21munin-run22:27:090
1355825518,21sleep20-21swapper/200:02:342
1355825518,21sleep20-21swapper/200:02:342
853825413,24sleep38544-21df_inode23:47:333
853825413,24sleep38544-21df_inode23:47:333
693825415,19sleep20-21swapper/221:57:362
326552549,32sleep0367-21snmpd23:19:160
2929125415,23sleep228807-21apt-get19:42:202
2929125415,23sleep228807-21apt-get19:42:202
152882544,12sleep20-21swapper/220:40:212
152882544,12sleep20-21swapper/220:40:212
1182425411,22sleep30-21swapper/322:12:433
2929425318,16sleep00-21swapper/019:42:220
2929425318,16sleep00-21swapper/019:42:220
2705225314,23sleep326944-21/usr/sbin/munin23:02:363
2661825311,24sleep326613-21apt-get21:17:323
2350325313,16sleep30-21swapper/319:22:473
2273525311,24sleep222728-21apt-get00:32:292
1511325313,20sleep00-21swapper/000:07:410
1337425312,25sleep313370-21apt-get00:02:273
1337425312,25sleep313370-21apt-get00:02:273
120482535,11sleep30-21swapper/320:31:503
3208725214,24sleep332097-21ntp_states23:17:503
1609425211,21sleep20-21swapper/222:27:312
76625112,23sleep2803-21ntp_states21:37:422
246882517,17sleep00-21swapper/019:27:410
1025125117,20sleep210295-21awk22:07:412
1025125117,20sleep210295-21awk22:07:412
3109125012,21sleep031144-21irqstats19:47:390
2642625011,27sleep0367-21snmpd22:58:580
2642625011,27sleep0367-21snmpd22:58:580
2511725014,15sleep20-21swapper/219:27:512
1788425015,17sleep20-21swapper/220:47:482
165025013,16sleep30-21swapper/321:42:113
452324917,15sleep30-21swapper/323:32:523
368124812,22sleep33708-21irqstats21:47:403
2815824816,18sleep20-21swapper/221:22:322
1790724811,21sleep217911-21df_inode19:07:322
1790724811,21sleep217911-21df_inode19:07:322
958224713,19sleep21-21systemd22:07:092
958224713,19sleep21-21systemd22:07:092
278152475,15sleep20-21swapper/221:21:352
24082469,18sleep20-21swapper/221:42:462
1655424612,15sleep30-21swapper/322:27:443
2373724311,15sleep20-21swapper/221:07:382
1057024312,16sleep20-21swapper/220:27:092
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional