You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-08 - 08:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Sep 08, 2024 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
269942356210,3539sleep125427-21kworker/1:021:52:571
4825234837,3463sleep11608-21kworker/1:020:41:381
168172143414,1401sleep316810-21apt-get23:04:523
2593262312,533sleep02747-21latency_hist22:19:400
2838724877,424sleep028423-21ntpq23:40:340
2781224247,404sleep027856-21ntpq21:55:020
28622240712,377sleep228696-21find23:40:402
921234559,207sleep01070-21grep23:55:120
2624123328,265sleep026327-21fschecks_time23:34:570
1093123208,296sleep011083-21latency_hist00:29:400
207042317168,129sleep00-21swapper/021:34:390
1942122658,201sleep0374-21snmpd23:12:000
14472225517,185sleep00-21swapper/021:14:400
12854225084,149sleep012888-21proc_pri19:25:050
8168224584,26sleep07068-21cyclictest20:54:370
13946224512,219sleep00-21swapper/019:29:530
25953224279,84sleep025995-21fschecks_count21:49:570
7540223781,74sleep07603-21fschecks_time22:34:580
2205522369,212sleep022118-21switchtime23:20:070
136422228,204sleep0374-21snmpd20:30:270
31472221425,134sleep00-21swapper/023:50:030
19327221027,112sleep019389-21processes19:45:040
23222220714,176sleep023255-21cron19:59:370
12406220718,175sleep312460-21sed22:50:033
24285220376,62sleep017608-1kworker/0:1H21:44:490
3412220016,158sleep00-21swapper/000:04:570
3412220016,158sleep00-21swapper/000:04:570
9962219924,90sleep09230-21/usr/sbin/munin19:15:070
1718521917,125sleep017221-21ntpq23:05:020
17990218015,144sleep017820-21/usr/sbin/munin21:24:580
666421769,150sleep06797-21latency_hist20:49:410
32706217512,150sleep0340-21timerandwakeup22:10:080
6299216416,71sleep00-21swapper/022:30:020
1543621640,41sleep315440-21fschecks_time22:59:593
4078216310,138sleep04095-21kernelversion20:40:120
26638216113,132sleep126705-21systemctl23:35:061
12223215712,15sleep00-21swapper/022:50:000
4103215614,19sleep30-21swapper/322:24:403
23147215612,21sleep00-21swapper/023:24:560
23147215612,21sleep00-21swapper/023:24:560
16222214619,56sleep00-21swapper/019:35:100
16222214619,56sleep00-21swapper/019:35:100
8711214418,109sleep3493-21cron22:39:373
4849214410,121sleep04940-21mailstats22:25:090
912214019,107sleep2961-21unin-run22:14:382
21421213826,92sleep020919-21/usr/sbin/munin21:35:080
21421213826,92sleep020919-21/usr/sbin/munin21:35:080
17026213820,39sleep017166-21unixbench_singl21:20:140
11949213833,47sleep00-21swapper/019:22:380
9522213713,110sleep39572-21sed19:14:583
14364213526,89sleep114522-21unixbench_multi22:55:101
2793213412,107sleep12849-21cpuspeed22:19:531
617621318,110sleep00-21swapper/000:14:380
617621318,110sleep00-21swapper/000:14:380
21727213112,101sleep021855-21latency_hist19:54:410
10333213115,64sleep210334-21sshd19:17:112
271621306,60sleep07068-21cyclictest00:00:370
271621306,60sleep07068-21cyclictest00:00:370
15837213012,57sleep00-21swapper/021:17:510
15837213012,57sleep00-21swapper/021:17:510
23577212830,12sleep00-21swapper/021:42:020
8769212760,32sleep00-21swapper/020:55:020
3889212721,88sleep33905-21sort20:39:593
12165212630,10sleep00-21swapper/000:31:500
24421212526,79sleep124513-21cut21:45:061
31171212416,92sleep331259-21fschecks_time23:49:583
16384212218,83sleep316397-21fschecks_count21:20:003
24782212143,65sleep024846-21cat23:30:000
1267521198,93sleep00-21swapper/000:34:530
1182521166,98sleep00-21swapper/022:49:230
29549211340,53sleep00-21swapper/020:19:400
17284211221,74sleep317302-21/usr/sbin/munin19:39:503
17223211014,80sleep117268-21ntp_states23:05:031
2541521098,87sleep325416-21sshd21:47:383
3064210810,84sleep03155-21load22:20:010
29786210735,56sleep00-21swapper/020:19:530
24266210516,19sleep00-21swapper/020:00:100
11581210513,78sleep211587-21ntp_states00:30:072
25601210411,24sleep00-21swapper/020:05:110
19762210417,72sleep219806-21ntpq21:30:012
5829210330,58sleep05869-21fschecks_time19:04:580
31862210313,75sleep131999-21latency_hist22:09:401
13764210333,32sleep00-21swapper/022:54:560
11298210313,74sleep111307-21ntpq19:20:051
30966210114,72sleep20-21swapper/220:23:382
30311210123,59sleep130319-21grep22:04:401
29160210135,52sleep129161-21sshd23:43:381
6292210037,34sleep20-21swapper/219:05:082
18377210018,66sleep118448-21sensors_temp21:25:071
1089429933,34sleep00-21swapper/021:00:110
767429819,61sleep011-21ksoftirqd/019:10:030
968629723,52sleep19810-21timerwakeupswit22:40:081
968629723,52sleep19810-21timerwakeupswit22:40:081
2949029730,11sleep00-21swapper/022:00:070
480729511,70sleep34319-21/usr/sbin/munin22:25:083
1451429519,57sleep114447-21latency_hist21:14:411
278329416,60sleep32881-21users20:35:103
278329416,60sleep32881-21users20:35:103
1355529418,38sleep313598-21ntpq21:10:033
661129315,61sleep26753-21unixbench_singl22:30:082
2795729353,22sleep00-21swapper/020:14:400
2365929316,64sleep323767-21timerandwakeup23:25:073
2365929316,64sleep323767-21timerandwakeup23:25:073
201902938,16sleep00-21swapper/023:15:030
1662429118,18sleep00-21swapper/023:04:410
177892909,13sleep20-21swapper/219:40:172
68582887,17sleep00-21swapper/000:15:060
308292888,15sleep00-21swapper/022:04:590
341628728,36sleep10-21swapper/100:04:581
341628728,36sleep10-21swapper/100:04:581
321772877,13sleep10-21swapper/123:52:381
3004628718,55sleep330064-21uname20:20:003
2000728724,42sleep120073-21fschecks_time23:15:001
179012879,37sleep00-21swapper/019:40:190
1205628719,44sleep211571-21/usr/sbin/munin21:05:052
727128618,44sleep37290-21apt-get19:09:493
2472228618,49sleep324776-21fschecks_time23:29:593
1012528618,53sleep310176-21fschecks_count20:59:573
31222855,69sleep00-21swapper/020:38:060
31222855,69sleep00-21swapper/020:38:060
2952828511,54sleep00-21swapper/023:44:540
2777928513,51sleep327387-21/usr/sbin/munin21:55:013
2189928515,53sleep321481-21/usr/sbin/munin23:20:023
193912840,7sleep10-21swapper/119:45:041
164192849,11sleep10-21swapper/119:35:241
164192849,11sleep10-21swapper/119:35:241
2166928320,34sleep30-21swapper/321:35:113
2166928320,34sleep30-21swapper/321:35:113
84212827,63sleep08456-21ntpq00:20:030
819128221,38sleep18218-21munin-run20:54:381
571528213,55sleep15727-21netstat20:45:011
2519728214,52sleep325296-21cut20:05:023
91062817,13sleep00-21swapper/022:39:550
91062817,13sleep00-21swapper/022:39:550
313332819,35sleep00-21swapper/020:24:530
1968028118,16sleep10-21swapper/123:14:391
143528125,35sleep21479-40mandb23:59:382
297528013,26sleep20-21swapper/222:19:592
2486328023,36sleep225011-21cat20:04:402
1836427922,33sleep318361-21apt-get23:09:493
118322798,59sleep011571-21/usr/sbin/munin21:04:590
1097927815,49sleep311023-21ntp_offset22:45:023
902927714,48sleep29093-21/usr/sbin/munin22:39:532
902927714,48sleep29093-21/usr/sbin/munin22:39:532
2797027724,33sleep327967-21apt-get23:39:523
2790627713,51sleep224079-21systemd-journal20:14:382
2757727621,33sleep327765-21gpgv23:39:403
2109427614,46sleep121232-21grep19:50:101
1415027617,14sleep20-21swapper/222:55:072
75262758,59sleep30-21swapper/300:17:243
792027427,27sleep10-21swapper/122:35:071
758727412,25sleep30-21swapper/320:50:123
565227420,29sleep20-21swapper/222:29:392
2644927422,36sleep326541-21/usr/sbin/munin21:50:093
231252745,11sleep20-21swapper/219:58:222
1030927424,30sleep20-21swapper/221:00:002
96227322,30sleep30-21swapper/322:14:393
3264627318,26sleep30-21swapper/320:29:413
3153027312,45sleep20-21swapper/223:50:052
300702730,8sleep20-21swapper/223:45:072
294452737,48sleep20-21swapper/222:00:052
1375327310,14sleep30-21swapper/322:54:553
879327218,25sleep20-21swapper/222:39:402
802627217,42sleep38087-21cpuspeed00:19:543
778527220,23sleep10-21swapper/100:19:401
616027224,32sleep16213-21latency22:29:591
3172227217,27sleep20-21swapper/220:25:032
272627220,25sleep20-21swapper/200:00:392
272627220,25sleep20-21swapper/200:00:392
1534927218,24sleep20-21swapper/222:59:592
1213927215,33sleep10-21swapper/122:49:571
2091827111,15sleep30-21swapper/319:50:073
1122427110,39sleep10-21swapper/122:45:081
3084027023,18sleep10-21swapper/123:49:391
231842705,11sleep30-21swapper/319:59:083
2148027029,19sleep10-21swapper/121:35:091
2148027029,19sleep10-21swapper/121:35:091
2079627014,27sleep20-21swapper/219:50:022
1248927015,25sleep30-21swapper/321:05:103
891226915,38sleep38919-21/usr/sbin/munin20:55:053
869426919,22sleep20-21swapper/200:20:092
589726916,23sleep30-21swapper/320:45:073
2724426916,36sleep127324-21find20:10:081
217426924,27sleep30-21swapper/300:00:033
217426924,27sleep30-21swapper/300:00:033
2518626815,36sleep225264-21ls23:30:102
231922687,52sleep10-21swapper/119:59:141
90826724,26sleep30-21swapper/320:30:053
703426721,20sleep10-21swapper/119:09:391
534726722,23sleep20-21swapper/200:10:082
534726722,23sleep20-21swapper/200:10:082
2381026714,24sleep30-21swapper/320:00:013
2160126725,24sleep20-21swapper/223:19:572
184492679,14sleep20-21swapper/221:25:082
76722666,49sleep20-21swapper/200:19:122
478926623,27sleep14796-21ntp_states22:25:061
332726617,32sleep33323-21apt-get00:04:523
332726617,32sleep33323-21apt-get00:04:523
3180726617,35sleep231829-21idleruntime-cro22:09:372
3104326620,29sleep131071-21munin-run20:24:381
2259226613,19sleep30-21swapper/321:39:553
114526613,38sleep21292-21unixbench_singl20:30:082
962226517,26sleep10-21swapper/119:15:001
3013726514,28sleep30-21swapper/323:45:093
2562526525,21sleep20-21swapper/221:49:392
2199326510,44sleep20-21swapper/221:36:592
2199326510,44sleep20-21swapper/221:36:592
211752658,46sleep20-21swapper/223:19:292
2312826411,38sleep223135-21diskstats23:24:542
2312826411,38sleep223135-21diskstats23:24:542
174626414,21sleep10-21swapper/120:34:401
1370726419,19sleep30-21swapper/319:29:403
1084626419,25sleep20-21swapper/219:19:542
95332636,47sleep20-21swapper/220:57:182
38126314,21sleep10-21swapper/123:55:011
3210426310,20sleep10-21swapper/120:25:111
2945226321,22sleep30-21swapper/322:00:063
1954026318,25sleep30-21swapper/319:45:073
1862926322,19sleep10-21swapper/119:44:401
1836826312,30sleep20-21swapper/223:09:522
1434526318,28sleep113919-21/usr/sbin/munin19:30:021
3218626211,35sleep132246-21fschecks_time22:09:561
308122627,12sleep30-21swapper/322:04:583
141026214,24sleep30-21swapper/322:14:593
48142618,12sleep30-21swapper/300:09:403
48142618,12sleep30-21swapper/300:09:403
276752617,13sleep10-21swapper/121:54:591
2353026119,26sleep123587-21/usr/sbin/munin23:25:051
2353026119,26sleep123587-21/usr/sbin/munin23:25:051
1394326115,28sleep213939-21apt-get19:29:512
1325626121,21sleep20-21swapper/221:09:582
1275626117,25sleep312759-21/usr/sbin/munin00:34:563
1214526114,34sleep312168-21cat19:24:393
82082606,12sleep10-21swapper/119:10:121
2749826020,17sleep20-21swapper/221:54:562
1597626017,22sleep10-21swapper/121:19:381
1597626017,22sleep10-21swapper/121:19:381
1543326020,19sleep20-21swapper/221:15:112
1543326020,19sleep20-21swapper/221:15:112
1296726023,22sleep20-21swapper/219:25:072
122642607,43sleep10-21swapper/100:33:081
846125913,19sleep10-21swapper/120:54:551
341725915,27sleep13585-21cut20:39:401
341725915,27sleep13585-21cut20:39:401
267062598,32sleep222689-1kworker/2:1H23:35:072
2567825917,23sleep10-21swapper/120:05:121
1742125917,27sleep217509-21timerandwakeup23:05:082
1650725917,19sleep10-21swapper/121:20:021
1400825920,19sleep20-21swapper/200:39:392
1088325922,23sleep30-21swapper/300:29:393
526725818,23sleep10-21swapper/100:10:031
526725818,23sleep10-21swapper/100:10:031
47642588,39sleep20-21swapper/220:40:502
2630025810,11sleep20-21swapper/220:08:522
2382025815,25sleep223905-21apt-config21:44:382
1110325815,17sleep30-21swapper/300:29:513
570925714,25sleep35704-21apt-get19:04:533
570425716,24sleep35629-21latency_hist22:29:403
2817325714,27sleep328192-21apt-get20:14:513
1594625716,25sleep315952-21ntp_states19:35:053
1594625716,25sleep315952-21ntp_states19:35:053
572825617,17sleep20-21swapper/220:45:022
3262725616,19sleep30-21swapper/323:54:543
2877325623,19sleep10-21swapper/120:15:091
2607725614,23sleep226134-21irqstats21:49:592
195725615,25sleep21974-21/usr/sbin/munin20:34:492
195725615,25sleep21974-21/usr/sbin/munin20:34:492
1948025614,23sleep10-21swapper/121:29:561
1605725619,17sleep30-21swapper/321:19:403
1605725619,17sleep30-21swapper/321:19:403
2353825513,26sleep123535-21apt-get19:59:511
2260825515,19sleep10-21swapper/121:39:571
2033525514,18sleep30-21swapper/323:15:063
902525412,26sleep39022-21apt-get22:39:503
902525412,26sleep39022-21apt-get22:39:503
718425417,20sleep20-21swapper/220:50:032
1553325420,18sleep10-21swapper/123:00:021
2887025316,17sleep20-21swapper/220:15:102
1763325316,21sleep217574-21latency_hist21:24:402
128925314,17sleep10-21swapper/122:14:571
756725215,17sleep20-21swapper/219:10:012
178192529,17sleep30-21swapper/321:24:493
177425213,25sleep11749-21/usr/sbin/munin23:59:531
177425213,25sleep11749-21/usr/sbin/munin23:59:531
1691425213,22sleep217082-21latency_hist19:39:412
1691425213,22sleep217082-21latency_hist19:39:412
1221025214,23sleep212216-21df_inode22:49:582
859925115,21sleep18692-21cat00:20:081
2745625112,14sleep30-21swapper/320:10:103
2509925111,22sleep125185-21mailstats23:30:091
1278825111,22sleep10-21swapper/119:25:021
1135925116,20sleep10-21swapper/121:04:401
2067125014,21sleep31-21systemd21:34:383
1930725014,14sleep20-21swapper/219:45:022
990224911,15sleep20-21swapper/200:25:022
369724816,18sleep20-21swapper/200:05:032
369724816,18sleep20-21swapper/200:05:032
2177624810,13sleep10-21swapper/123:20:001
1285624813,18sleep10-21swapper/100:34:581
965524711,21sleep19659-21diskstats00:24:571
1204924612,21sleep112055-21ntp_states21:05:041
13392457,17sleep20-21swapper/222:14:582
1394424312,17sleep30-21swapper/319:29:523
115992428,26sleep20-21swapper/222:46:352
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional