You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-26 - 07:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Apr 26, 2024 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
49422198214,1948sleep04935-21apt-get22:46:300
3776250810,483sleep13806-21ntpq22:41:421
3776250810,483sleep13806-21ntpq22:41:421
19045248077,386sleep319135-21fschecks_time23:31:373
2688724707,427sleep326934-21ntpq22:11:413
2688724707,427sleep326934-21ntpq22:11:413
426323688,347sleep04303-21ntpq00:26:410
29760230810,207sleep329830-21basename22:21:343
22754229422,251sleep122775-21fschecks_count20:16:331
28152228136,217sleep30-21swapper/322:16:333
24720226879,28sleep30-21swapper/320:21:443
462122287,209sleep31166-21snmpd22:45:523
462122287,209sleep31166-21snmpd22:45:523
25983220442,145sleep326044-21switchtime23:51:483
2811219744,136sleep30-21swapper/300:21:443
2811219744,136sleep30-21swapper/300:21:443
230721928,172sleep32315-21awk22:36:443
230721928,172sleep32315-21awk22:36:443
16681218817,155sleep216651-21latency_hist21:41:162
16302218613,158sleep116304-21sh23:21:421
13983218539,123sleep30-21swapper/319:46:493
17062218112,106sleep30-21swapper/321:41:373
23978218011,154sleep023952-21latency_hist20:21:170
1093217619,137sleep21251-21latency_hist20:51:182
30665217225,131sleep130658-21apt-get20:41:281
2491217066,92sleep32557-21sed19:11:453
17158216716,128sleep30-21swapper/323:26:173
4848216011,133sleep35003-21latency_hist19:21:183
16895215682,29sleep30-21swapper/319:56:463
10414215416,119sleep3311-21in:imuxsock21:21:143
1043121539,131sleep310483-21cat19:36:443
32262215114,123sleep332378-21timerandwakeup00:11:463
32262215114,123sleep332378-21timerandwakeup00:11:463
337214810,122sleep3379-21fschecks_count22:31:313
31349214821,108sleep331431-21mailstats19:01:443
19726214324,95sleep219795-21tail23:31:532
680321399,45sleep30-21swapper/300:36:173
26699213923,97sleep21-21systemd23:56:152
32430213813,63sleep332458-21modprobe22:31:153
284221386,11sleep00-21swapper/019:11:500
1198213615,96sleep30-21swapper/300:16:393
1198213615,96sleep30-21swapper/300:16:393
10467213620,99sleep210573-21latency_hist21:21:172
9616213436,41sleep39613-21apt-get23:01:263
12889213210,107sleep312983-21cat21:26:483
28947213114,94sleep30-21swapper/300:01:413
28947213114,94sleep30-21swapper/300:01:413
9720212815,96sleep30-21swapper/319:36:153
32320212818,95sleep332334-21fschecks_count20:46:343
17928212487,19sleep30-21swapper/323:26:453
2738921236,14sleep30-21swapper/320:31:333
20795212312,43sleep30-21swapper/320:11:143
3244521218,96sleep30-21swapper/319:06:323
8624211915,88sleep08691-21switchtime22:56:470
18594211914,41sleep117964-21/usr/sbin/munin20:01:481
26663211819,80sleep126696-21munin-plugin-st23:56:141
25463211817,80sleep225536-21cut22:06:432
1619421177,53sleep316268-21latency_hist19:56:173
1619421177,53sleep316268-21latency_hist19:56:173
856721169,43sleep31-21systemd22:56:463
704121159,90sleep37107-21wc22:51:453
3069421158,40sleep30-21swapper/320:41:463
21274211466,30sleep321311-21sendmail21:56:153
1430021139,39sleep30-21swapper/323:16:283
21170211116,73sleep121230-21fschecks_time20:11:331
5342211019,54sleep30-21swapper/322:46:413
2963321109,39sleep30-21swapper/320:36:543
1806821107,87sleep30-21swapper/320:01:343
2385821088,88sleep323844-21apt-get23:46:313
2366921088,43sleep31166-21snmpd20:17:573
3251621066,15sleep10-21swapper/100:11:491
3251621066,15sleep10-21swapper/100:11:491
11142210612,79sleep111161-21apt-get23:06:251
350621059,84sleep01166-21snmpd00:25:080
350621059,84sleep01166-21snmpd00:25:080
336521046,35sleep30-21swapper/320:56:443
24676210312,76sleep124692-21open_inodes20:21:421
24395210319,53sleep30-21swapper/322:03:013
8179210223,59sleep28253-21find21:11:492
2273021026,13sleep30-21swapper/321:59:343
1923921028,78sleep119364-21uptime21:46:481
4252210113,67sleep30-21swapper/300:26:403
2169521005,12sleep30-21swapper/320:11:473
15134210010,78sleep315136-21munin-run21:36:163
15134210010,78sleep315136-21munin-run21:36:163
88182996,38sleep30-21swapper/321:15:583
786929814,65sleep30-21swapper/319:31:153
77929814,20sleep20-21swapper/200:16:262
77929814,20sleep20-21swapper/200:16:262
1196229848,39sleep311987-21munin-run21:26:153
2552629720,20sleep00-21swapper/020:26:170
1782329711,62sleep20-21swapper/221:41:532
1242829662,15sleep30-21swapper/323:11:153
3140329513,69sleep131458-21sed00:11:161
3116529426,24sleep00-21swapper/022:26:320
1513029416,56sleep214828-21/usr/sbin/munin19:51:402
1513029416,56sleep214828-21/usr/sbin/munin19:51:402
1429929327,42sleep214296-21apt-get23:16:272
1197229315,60sleep00-21swapper/023:06:480
668529221,53sleep06727-21awk00:36:140
287862925,13sleep20-21swapper/200:01:362
287862925,13sleep20-21swapper/200:01:362
270062925,11sleep30-21swapper/320:31:043
397029017,58sleep24062-21tune2fs00:26:352
124862908,13sleep00-21swapper/021:26:360
1546828921,53sleep30-21swapper/321:36:333
1203428910,67sleep312077-21latency19:41:403
3082828832,32sleep00-21swapper/019:01:250
224372889,67sleep322462-21netstat23:41:393
515828720,49sleep25214-21sort19:21:372
506128720,51sleep30-21swapper/319:21:303
2925828713,62sleep229275-21netstat20:36:452
2698228759,16sleep326978-21apt-get23:56:243
258752876,11sleep30-21swapper/322:07:093
185532878,59sleep30-21swapper/321:46:343
127142868,60sleep30-21swapper/323:11:273
2822228511,24sleep00-21swapper/020:31:500
250628419,11sleep30-21swapper/320:55:103
3132128327,26sleep20-21swapper/219:01:432
2335628323,37sleep123374-21apt22:01:251
2169328319,10sleep30-21swapper/323:40:053
2169328319,10sleep30-21swapper/323:40:053
578628111,15sleep30-21swapper/321:06:173
2437428111,54sleep223824-21/usr/sbin/munin23:47:072
233932810,5sleep20-21swapper/220:16:462
700928019,11sleep30-21swapper/321:07:463
1693428014,47sleep116408-21/usr/sbin/munin19:56:481
1257528012,17sleep20-21swapper/219:41:512
938327912,51sleep19478-21latency_hist23:01:161
3109627915,11sleep30-21swapper/300:07:593
2215727914,50sleep222192-21fschecks_count23:41:332
267542786,11sleep10-21swapper/120:27:521
2561727818,41sleep025643-21/usr/sbin/munin23:51:370
130527816,39sleep00-21swapper/019:11:170
301527718,38sleep20-21swapper/200:21:472
301527718,38sleep20-21swapper/200:21:472
2980427718,31sleep00-21swapper/000:06:140
2980427718,31sleep00-21swapper/000:06:140
2339927720,32sleep023533-21apt-config23:46:160
137027715,42sleep11547-21latency_hist19:11:181
288327619,34sleep10-21swapper/120:56:261
2469927519,25sleep00-21swapper/022:06:170
209827519,27sleep00-21swapper/000:21:160
209827519,27sleep00-21swapper/000:21:160
2088827519,29sleep20-21swapper/223:36:372
2088827519,29sleep20-21swapper/223:36:372
1022527516,32sleep110202-21apt-get19:36:381
554627419,21sleep00-21swapper/019:21:470
267027414,41sleep02776-21latency_hist20:56:160
1978127415,44sleep219937-21date21:51:172
1807427420,34sleep00-21swapper/020:01:350
2600427313,46sleep026033-21/usr/sbin/munin20:26:370
2177427323,28sleep20-21swapper/221:56:342
1339727326,23sleep20-21swapper/223:11:462
924027227,23sleep10-21swapper/121:16:321
456827220,28sleep10-21swapper/100:26:501
3091527215,37sleep130935-21ls22:26:161
2081927219,29sleep10-21swapper/121:51:491
160827211,41sleep01547-21latency_hist22:36:160
1201527218,22sleep10-21swapper/121:26:171
481627116,41sleep24846-21latency_hist19:21:162
2897727117,40sleep129021-21grep20:36:391
171582718,12sleep10-21swapper/121:41:391
141992710,11sleep132226-21cobalt_printf23:16:171
165262707,13sleep00-21swapper/021:39:530
531026916,35sleep25341-21/usr/sbin/munin22:46:402
376326914,38sleep03775-21/usr/sbin/munin22:41:410
376326914,38sleep03775-21/usr/sbin/munin22:41:410
1043726915,39sleep01-21systemd21:21:150
733326820,25sleep20-21swapper/221:11:162
321726815,40sleep23275-21unin-run19:16:162
3210726814,23sleep00-21swapper/000:11:430
3210726814,23sleep00-21swapper/000:11:430
2937626817,26sleep10-21swapper/122:21:161
2891226813,28sleep10-21swapper/100:01:391
2891226813,28sleep10-21swapper/100:01:391
1577526818,23sleep10-21swapper/121:36:391
1152526814,34sleep011573-21grep19:41:170
87672678,49sleep00-21swapper/021:15:180
319726717,35sleep13214-21sh19:16:151
3085526725,28sleep130910-21cpuspeed19:01:301
2454126720,25sleep10-21swapper/123:47:101
193626720,23sleep20-21swapper/222:36:352
193626720,23sleep20-21swapper/222:36:352
1339626719,21sleep20-21swapper/219:46:362
714526620,17sleep10-21swapper/119:26:531
3230726613,24sleep10-21swapper/120:46:321
3093726612,38sleep231097-21sed22:26:172
2693726615,34sleep026939-21ntp_states22:11:420
2693726615,34sleep026939-21ntp_states22:11:420
2530426610,37sleep024909-21/usr/sbin/munin22:06:370
2337526614,32sleep222885-21apt-get22:01:272
1016726615,22sleep00-21swapper/023:01:470
419626511,22sleep10-21swapper/119:16:511
2775226520,28sleep127780-21munin-run22:16:151
2775226520,28sleep127780-21munin-run22:16:151
144362658,18sleep00-21swapper/019:49:320
142762656,18sleep00-21swapper/023:16:250
976426426,20sleep00-21swapper/019:36:160
395726411,12sleep00-21swapper/020:58:270
2932826419,23sleep00-21swapper/022:21:150
270572647,16sleep10-21swapper/123:56:301
1351126419,29sleep213530-21munin-run21:31:152
1233626415,29sleep112343-21/usr/sbin/munin21:26:331
118622645,45sleep20-21swapper/221:24:572
1030426417,25sleep20-21swapper/223:01:492
965726318,26sleep29719-21modprobe19:36:142
54012636,45sleep20-21swapper/200:31:242
3063926317,27sleep230156-21apt-get20:41:252
709826218,25sleep00-21swapper/019:26:520
3250026217,21sleep20-21swapper/222:31:182
3085826214,19sleep00-21swapper/000:06:470
306632626,13sleep00-21swapper/020:41:270
3036226214,20sleep20-21swapper/200:06:372
1059126218,28sleep210619-21/usr/sbin/munin19:36:472
657626119,22sleep00-21swapper/022:51:340
324282618,16sleep10-21swapper/122:31:131
2547226120,25sleep225498-21/usr/sbin/munin20:26:152
1438726117,25sleep00-21swapper/021:31:490
1438726117,25sleep00-21swapper/021:31:490
2915226015,29sleep029164-21/usr/sbin/munin20:36:430
2215426018,23sleep10-21swapper/123:41:321
216282608,11sleep20-21swapper/220:11:462
2154526019,20sleep00-21swapper/020:11:440
1748126020,24sleep017562-21grep23:26:340
152726014,21sleep20-21swapper/222:36:142
1439126017,21sleep10-21swapper/123:16:341
1107226013,32sleep011100-21nfs4_client21:21:400
494325917,25sleep10-21swapper/122:46:311
46725910,16sleep10-21swapper/122:31:331
2702025922,22sleep20-21swapper/220:31:162
269892598,14sleep00-21swapper/023:56:290
1643325915,26sleep016427-21apt-get19:56:300
1406525913,22sleep00-21swapper/023:16:170
115125911,34sleep11183-21awk00:16:371
115125911,34sleep11183-21awk00:16:371
642525820,21sleep10-21swapper/121:06:391
359125815,27sleep13620-21munin-run00:26:141
359125815,27sleep13620-21munin-run00:26:141
2516925815,28sleep027323-1kworker/0:0H23:51:170
2052125815,27sleep020515-21apt-get23:36:270
2052125815,27sleep020515-21apt-get23:36:270
2002025819,18sleep00-21swapper/020:06:440
1835725817,22sleep20-21swapper/220:01:412
1485125816,26sleep014847-21apt-get19:51:260
1485125816,26sleep014847-21apt-get19:51:260
649225717,17sleep10-21swapper/122:51:311
2061025715,23sleep120702-21perl23:36:321
2061025715,23sleep120702-21perl23:36:321
1646325717,18sleep20-21swapper/223:21:472
575825614,23sleep15797-21memory00:31:391
348025615,25sleep23487-21/usr/sbin/munin22:41:352
348025615,25sleep23487-21/usr/sbin/munin22:41:352
204425616,25sleep21306-21/usr/sbin/munin20:51:502
1902025610,30sleep019027-21df_inode23:31:340
184525615,21sleep10-21swapper/122:36:301
184525615,21sleep10-21swapper/122:36:301
1803925615,17sleep10-21swapper/123:26:471
1093125621,20sleep110903-21latency_hist23:06:161
54482556,10sleep20-21swapper/221:02:442
283872558,10sleep10-21swapper/120:33:571
1774825513,20sleep20-21swapper/220:01:172
1176325513,25sleep111758-21apt-get19:41:301
697925414,24sleep26986-21memory19:26:462
3196725414,24sleep232006-21memory00:11:372
3196725414,24sleep232006-21memory00:11:372
2871425421,19sleep20-21swapper/222:16:492
2301425416,16sleep10-21swapper/122:01:171
229152549,28sleep023011-21apt-config22:01:160
205022547,37sleep20-21swapper/220:07:302
2025825413,23sleep020285-21irqstats21:51:350
184425413,24sleep01838-21apt-get22:36:290
184425413,24sleep01838-21apt-get22:36:290
176062548,12sleep00-21swapper/021:41:500
1116925417,18sleep20-21swapper/223:06:302
851525315,21sleep10-21swapper/119:31:531
806625314,24sleep28061-21apt-get22:56:292
426825317,15sleep00-21swapper/021:01:170
2005825314,24sleep220086-21df21:51:292
178712535,11sleep20-21swapper/223:26:442
648825211,24sleep26484-21apt-get22:51:282
462425215,19sleep10-21swapper/121:01:341
348425215,22sleep23536-21sort20:56:472
2961325213,23sleep029608-21apt-get22:21:270
2648225211,24sleep226478-21apt-get22:11:282
2648225211,24sleep226478-21apt-get22:11:282
1377425214,25sleep213781-21cat21:31:272
1377425214,25sleep213781-21cat21:31:272
620625116,20sleep06246-21idleruntime21:06:340
3234825114,16sleep00-21swapper/020:46:360
140625112,24sleep11412-21df_inode20:51:331
812225014,14sleep10-21swapper/121:11:481
1584024913,21sleep015842-21apt-get23:21:250
12542499,16sleep20-21swapper/219:11:152
2778324814,17sleep20-21swapper/220:31:442
2518224811,21sleep125203-21irqstats22:06:341
1980324816,17sleep10-21swapper/120:06:361
1972824814,17sleep00-21swapper/021:51:150
1902824814,19sleep119034-21diskstats23:31:351
152732489,14sleep10-21swapper/119:51:431
152732489,14sleep10-21swapper/119:51:431
348924712,20sleep03490-21/usr/sbin/munin19:16:260
636124513,18sleep10-21swapper/119:26:161
145682449,19sleep114595-21munin-run19:51:151
1237424410,15sleep00-21swapper/019:41:490
1446124210,12sleep10-21swapper/121:31:501
1446124210,12sleep10-21swapper/121:31:501
2741624011,14sleep20-21swapper/223:56:402
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional