You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-22 - 12:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sat Jun 22, 2024 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305562112012,1098sleep316943-1kworker/3:1H20:38:143
28027247238,409sleep128183-21grep23:53:351
11227243798,196sleep1231-21systemd-journal23:02:561
474623847,365sleep14788-21ntpq20:58:271
13583227418,240sleep313596-21cut23:08:273
324762271124,26sleep10-21swapper/122:27:591
21752223318,107sleep121791-21sh20:12:561
26626223119,194sleep10-21swapper/122:08:201
184122160,30sleep131503-21cyclictest22:32:591
5841219812,164sleep15928-21unixbench_singl22:43:341
512121980,13sleep325795-21kworker/3:222:43:143
13432219432,25sleep113398-21latency_hist21:27:581
8022218913,154sleep18231-21latency_hist22:52:581
334218214,17sleep00-21swapper/022:28:060
9119218115,132sleep10-21swapper/121:13:211
4296218118,21sleep10-21swapper/100:23:151
32361217710,24sleep10-21swapper/119:03:351
3216217619,23sleep10-21swapper/122:37:561
286421758,16sleep30-21swapper/320:53:163
2567217518,140sleep02648-21timerandwakeup22:33:310
28529217315,22sleep10-21swapper/120:32:591
30794217112,143sleep130912-21latency_hist22:22:581
22293216915,133sleep10-21swapper/120:13:201
22293216915,133sleep10-21swapper/120:13:201
1901121659,85sleep119079-21cat21:43:331
2302321629,139sleep123144-21latency_hist21:57:591
27093215919,123sleep027064-40mandb23:52:570
22029215914,123sleep10-21swapper/123:33:361
17121215916,22sleep10-21swapper/119:57:581
17121215916,22sleep10-21swapper/119:57:581
2786215810,134sleep11166-21snmpd20:53:141
30538215615,120sleep130541-21apt-get20:38:081
17637215510,15sleep10-21swapper/119:58:221
17637215510,15sleep10-21swapper/119:58:221
26966215411,127sleep027063-21sed22:08:320
2044321539,22sleep10-21swapper/123:28:361
2526621527,82sleep10-21swapper/122:03:281
15345215213,117sleep10-21swapper/121:33:201
15345215213,117sleep10-21swapper/121:33:201
2420521489,125sleep124223-21cpu23:43:091
2420521489,125sleep124223-21cpu23:43:091
14864214720,21sleep10-21swapper/123:13:051
224621446,99sleep11166-21snmpd20:50:071
4919214214,26sleep00-21swapper/000:23:340
26726213510,23sleep126819-21latency_hist20:27:581
12778213210,106sleep112798-21munin-run23:07:561
4752213114,17sleep10-21swapper/119:18:251
2397321285,29sleep10-21swapper/120:18:371
1432221285,29sleep10-21swapper/123:09:501
71521275,30sleep10-21swapper/120:45:231
2392121279,102sleep11-21systemd23:42:571
7926212619,88sleep37913-21apt-get19:28:123
1653221268,102sleep116619-21perl23:18:141
14656212610,15sleep10-21swapper/119:48:251
278221258,44sleep112-21rcu_sched19:13:081
755721238,100sleep17649-21perl21:08:161
629421237,103sleep16338-21mailstats00:28:321
2013121238,16sleep10-21swapper/121:48:141
18083212310,17sleep10-21swapper/123:23:151
1264921237,16sleep10-21swapper/119:43:081
886421229,17sleep18896-21ntp_offset22:53:431
740021229,17sleep17455-21apt-config19:27:581
1121821228,99sleep111316-21cat21:18:301
949221218,18sleep19494-21apt-get19:33:071
2682212110,13sleep10-21swapper/100:18:061
2677221217,43sleep112553-1kworker/1:1H23:49:511
589721208,14sleep10-21swapper/121:03:111
231721205,28sleep10-21swapper/100:16:571
231721205,28sleep10-21swapper/100:16:571
3072021196,85sleep10-21swapper/100:03:201
169121187,14sleep10-21swapper/119:08:351
169121187,14sleep10-21swapper/119:08:351
788521176,15sleep10-21swapper/119:28:021
2334121177,37sleep10-21swapper/121:58:151
3230521160,9sleep30-21swapper/322:27:563
30523211522,79sleep030542-21grep00:03:150
11511211417,82sleep011533-21ntpq19:38:270
54521115,12sleep10-21swapper/100:09:381
54521115,12sleep10-21swapper/100:09:381
13021211121,73sleep013202-21latency_hist23:07:590
19731211013,82sleep219946-21sed21:47:582
4878210820,75sleep24917-21mailstats20:58:322
20863210815,78sleep320899-21ntpq20:08:223
1364221086,30sleep10-21swapper/121:28:071
927021070,8sleep20-21swapper/219:32:582
24574210713,78sleep024584-21/usr/sbin/munin23:43:200
24574210713,78sleep024584-21/usr/sbin/munin23:43:200
32367210614,76sleep332369-21/usr/sbin/munin00:08:233
32367210614,76sleep332369-21/usr/sbin/munin00:08:233
2976221065,11sleep10-21swapper/120:34:371
7948210414,74sleep08012-21cut21:08:270
25669210412,74sleep025716-21/usr/sbin/munin22:03:350
2018021045,12sleep10-21swapper/120:07:431
2018021045,12sleep10-21swapper/120:07:431
5519210313,72sleep05532-21/usr/sbin/munin22:43:280
12742210313,34sleep312832-21perl19:43:143
1219121035,30sleep10-21swapper/119:40:461
9682210216,52sleep20-21swapper/222:57:562
452121025,29sleep10-21swapper/122:39:501
26723210210,76sleep326751-21/usr/sbin/munin22:08:223
2886121015,30sleep10-21swapper/123:58:031
2831321015,27sleep10-21swapper/122:13:211
1778321016,29sleep10-21swapper/121:39:291
7688210012,71sleep07883-21cat19:28:010
17942210014,71sleep218004-21sed19:58:312
17942210014,71sleep218004-21sed19:58:312
246729715,63sleep02527-21sendmail-msp20:52:570
1580629716,67sleep015809-21grep19:53:120
1580629716,67sleep015809-21grep19:53:120
1492729719,55sleep214931-21cron21:32:562
468429520,54sleep30-21swapper/319:18:233
2748929421,58sleep327554-21idleruntime23:53:213
1838529410,67sleep31166-21snmpd19:59:583
1838529410,67sleep31166-21snmpd19:59:583
3210529319,57sleep232080-21/usr/sbin/munin20:43:102
283329317,60sleep22895-21/usr/sbin/munin00:18:152
35042919,14sleep30-21swapper/322:38:103
1461129114,59sleep014173-21/usr/sbin/munin19:48:240
94922907,17sleep00-21swapper/022:55:320
2922829014,16sleep00-21swapper/020:33:260
2621529010,65sleep326307-21users20:23:323
18122907,63sleep00-21swapper/000:13:290
18122907,63sleep00-21swapper/000:13:290
2386428919,50sleep023844-21apt-get20:18:160
2583928816,56sleep225871-21fschecks_count23:48:152
1292128824,34sleep30-21swapper/323:07:583
1327228715,58sleep213288-21awk23:08:062
2460928617,55sleep224636-21ntp_kernel_pll_23:43:222
2460928617,55sleep224636-21ntp_kernel_pll_23:43:222
1570728613,54sleep215793-21/usr/sbin/munin23:13:312
635128516,47sleep25755-21/usr/sbin/munin00:28:332
822228419,42sleep08285-21latency19:28:370
2306028416,54sleep323073-21ntpq23:38:273
901928315,52sleep39068-21expr22:53:493
2138528319,40sleep321416-21munin-run21:52:563
34012820,7sleep30-21swapper/300:18:273
3175928231,28sleep20-21swapper/200:07:572
320002810,6sleep00-21swapper/000:08:110
320002810,6sleep00-21swapper/000:08:110
2580828112,50sleep20-21swapper/220:23:232
203628117,35sleep20-21swapper/200:13:312
203628117,35sleep20-21swapper/200:13:312
2029328121,17sleep30-21swapper/323:28:343
150028119,18sleep30-21swapper/300:13:193
150028119,18sleep30-21swapper/300:13:193
1987328017,49sleep3306-21systemd-logind20:03:493
1987328017,49sleep3306-21systemd-logind20:03:493
1177928018,37sleep30-21swapper/319:38:343
11282800,6sleep20-21swapper/222:28:442
301727815,32sleep00-21swapper/019:13:190
2430227814,47sleep324401-21grep23:43:153
2430227814,47sleep324401-21grep23:43:153
30427714,45sleep2397-21unixbench_singl00:08:342
30427714,45sleep2397-21unixbench_singl00:08:342
785827614,44sleep37937-21systemd-run22:52:563
754227520,27sleep30-21swapper/319:28:003
15142756,47sleep20-21swapper/220:48:222
1282927537,24sleep212908-21munin-run23:07:572
999227423,33sleep00-21swapper/019:33:300
3229927425,30sleep232303-21modprobe22:27:562
3074027416,45sleep030776-21switchtime18:58:290
2800827414,43sleep028017-21/usr/sbin/munin22:13:120
250192749,47sleep325071-21if_eth022:03:223
2956627324,27sleep30-21swapper/320:33:343
2865427317,34sleep00-21swapper/023:57:580
2119827317,31sleep20-21swapper/223:33:172
2834027216,28sleep30-21swapper/322:13:233
2798427213,44sleep028109-21uptime20:28:460
2338727225,24sleep20-21swapper/220:17:582
2338727225,24sleep20-21swapper/220:17:582
1615127212,16sleep30-21swapper/323:17:563
1467627217,33sleep314173-21/usr/sbin/munin19:48:273
1340627215,27sleep00-21swapper/023:08:240
2945027129,21sleep20-21swapper/220:33:332
971627024,25sleep30-21swapper/321:13:353
478727010,46sleep24849-21sensors00:23:322
397227015,39sleep23990-21/usr/sbin/munin22:38:262
3052527027,23sleep20-21swapper/218:58:232
1804827025,26sleep00-21swapper/019:58:330
1804827025,26sleep00-21swapper/019:58:330
1058427020,21sleep20-21swapper/222:58:332
2935926914,21sleep20-21swapper/223:58:282
2429326917,26sleep20-21swapper/220:18:472
1707026921,28sleep30-21swapper/319:57:563
1707026921,28sleep30-21swapper/319:57:563
2075826828,21sleep20-21swapper/221:48:392
1654626824,24sleep30-21swapper/321:37:583
1654626824,24sleep30-21swapper/321:37:583
62632679,41sleep06291-21ps00:28:310
350126712,37sleep00-21swapper/022:38:070
301592678,43sleep130133-21/usr/sbin/munin18:58:141
2007926713,35sleep020142-21/usr/sbin/munin23:28:300
190022677,40sleep20-21swapper/220:03:162
190022677,40sleep20-21swapper/220:03:162
1457426714,24sleep00-21swapper/021:28:340
99126615,24sleep30-21swapper/320:47:593
3116426613,39sleep31166-21snmpd18:59:453
2451726622,22sleep20-21swapper/222:02:562
2048526610,28sleep00-21swapper/020:08:110
1658126621,24sleep00-21swapper/021:37:590
1658126621,24sleep00-21swapper/021:37:590
3020226520,21sleep00-21swapper/000:02:590
2785826518,30sleep227911-21grep23:53:322
2457026518,32sleep024647-21cut22:02:580
2391726516,31sleep023918-21cron23:42:560
1893126519,23sleep00-21swapper/021:43:320
1690526517,31sleep316932-21/usr/sbin/munin23:18:243
3097726418,21sleep20-21swapper/220:38:372
2347526415,33sleep323621-21latency_hist20:17:593
2347526415,33sleep323621-21latency_hist20:17:593
1922726414,23sleep30-21swapper/321:43:353
1578126415,33sleep215800-21/usr/sbin/munin21:33:332
1578126415,33sleep215800-21/usr/sbin/munin21:33:332
1131726422,22sleep20-21swapper/221:18:312
657626318,26sleep26594-21apt-get22:48:072
37052635,12sleep00-21swapper/020:53:370
3247526321,27sleep032500-21awk20:43:200
2211126315,30sleep222128-21/usr/sbin/munin21:53:272
2021526319,27sleep220291-21ls23:28:332
1493226324,20sleep30-21swapper/321:32:573
105326310,27sleep20-21swapper/219:08:042
105326310,27sleep20-21swapper/219:08:042
84826217,20sleep30-21swapper/322:28:383
2930426217,29sleep30-21swapper/323:58:253
220862626,12sleep00-21swapper/023:34:200
196026221,23sleep20-21swapper/222:33:132
1238926220,25sleep20-21swapper/221:23:192
681026116,25sleep00-21swapper/022:48:170
642726121,22sleep00-21swapper/021:03:310
2517626118,22sleep20-21swapper/222:03:252
238462619,23sleep30-21swapper/320:18:073
2179526113,18sleep30-21swapper/321:53:203
2160926111,18sleep30-21swapper/323:33:303
1265026113,26sleep20-21swapper/219:43:092
765026016,20sleep20-21swapper/221:08:172
636426020,19sleep20-21swapper/221:03:292
2586026011,33sleep025866-21ntp_states20:23:250
1087526013,20sleep00-21swapper/021:18:210
102092608,14sleep30-21swapper/319:33:333
486925920,18sleep20-21swapper/219:18:302
326225917,26sleep33268-21ntp_states19:13:263
3182725915,27sleep331831-21diskstats19:03:203
294602598,12sleep00-21swapper/022:18:090
2266225917,25sleep222656-21apt-get23:38:132
176472597,11sleep00-21swapper/023:22:170
113162597,13sleep20-21swapper/219:38:202
101302595,42sleep00-21swapper/021:15:580
1929725815,28sleep019364-21latency_hist23:27:580
1859225813,32sleep218665-21find23:23:282
1706825810,29sleep317099-21/usr/sbin/munin21:38:223
1666525816,25sleep216711-21users19:53:352
1666525816,25sleep216711-21users19:53:352
310362577,12sleep00-21swapper/020:38:390
2790925714,25sleep227928-21apt-get22:13:062
1127825718,17sleep30-21swapper/323:02:583
720525613,29sleep37174-21latency_hist21:07:583
499325617,23sleep05097-21sed19:18:320
1891025615,25sleep018904-21apt-get20:03:100
1891025615,25sleep018904-21apt-get20:03:100
179412567,37sleep00-21swapper/021:41:300
166525619,18sleep30-21swapper/319:08:333
166525619,18sleep30-21swapper/319:08:333
146632567,13sleep20-21swapper/219:48:262
1301425613,26sleep013020-21memory19:43:190
642025520,15sleep20-21swapper/219:23:262
505725515,20sleep20-21swapper/222:43:142
3193025520,21sleep20-21swapper/219:03:232
2853425515,17sleep20-21swapper/220:33:002
2700925516,24sleep327091-21systemd-run23:52:563
2207225513,25sleep022066-21apt-get20:13:110
2207225513,25sleep022066-21apt-get20:13:110
1039225516,25sleep00-21swapper/022:58:270
1033625517,18sleep20-21swapper/221:17:582
3111925413,24sleep031126-21/usr/sbin/munin22:23:170
2743425413,19sleep20-21swapper/222:10:362
2716225411,30sleep327183-21awk20:28:083
2379725415,21sleep30-21swapper/321:58:293
1488925412,25sleep014884-21apt-get23:13:080
878825318,18sleep30-21swapper/321:12:583
3007625311,19sleep20-21swapper/222:18:312
2994925317,19sleep30-21swapper/322:18:283
1819625313,23sleep318243-21/usr/sbin/munin23:23:173
128325315,24sleep31289-21df_inode20:48:153
1110225312,14sleep30-21swapper/321:18:283
575425220,17sleep30-21swapper/300:28:053
460925213,23sleep04613-21irqstats20:58:220
325725210,15sleep30-21swapper/322:37:583
280025212,17sleep20-21swapper/219:13:092
2094625111,14sleep20-21swapper/220:08:252
122725113,22sleep01222-21apt-get19:08:140
122725113,22sleep01222-21apt-get19:08:140
1216625113,15sleep00-21swapper/021:23:130
267625012,22sleep02679-21apt-get00:18:050
2170725010,27sleep00-21swapper/021:53:130
2008625011,22sleep019614-21apt-get21:48:000
3201724813,17sleep00-21swapper/019:03:250
1720124815,17sleep20-21swapper/221:38:252
305924713,19sleep20-21swapper/220:53:232
1194324715,15sleep30-21swapper/321:23:003
3051724610,21sleep330521-21diskstats00:03:143
1643524610,20sleep216431-21apt-get23:18:072
1011424613,20sleep20-21swapper/219:33:322
479824511,16sleep30-21swapper/320:58:293
42372398,17sleep30-21swapper/300:23:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional