You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 15:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Apr 27, 2024 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21680259716,567sleep221719-21ntpq23:21:422
818124820,16sleep37700-21kworker/3:100:21:313
818124820,16sleep37700-21kworker/3:100:21:313
13227245112,423sleep313233-21/usr/sbin/munin21:11:563
5474231919,205sleep15487-21fschecks_count19:06:311
20061227275,182sleep120102-21latency19:51:381
7435225166,102sleep17495-21fschecks_time19:11:311
712224929,196sleep1742-21munin-plugin-st22:16:111
14839224016,202sleep114965-21latency_hist23:01:131
31951222992,67sleep10-21swapper/120:31:111
24104222563,87sleep10-21swapper/121:46:351
7804221716,158sleep17956-21latency_hist00:21:131
2943220412,175sleep32967-21netstat22:21:343
2360220315,95sleep10-21swapper/120:37:091
274622029,181sleep01166-21snmpd00:02:180
28942219618,147sleep10-21swapper/120:21:141
12717219538,118sleep00-21swapper/021:11:210
3274721809,158sleep1380-21latency23:56:581
32347218017,89sleep15168-21cyclictest23:56:171
10119217821,134sleep010173-21sendmail_mailqu21:01:440
32327217416,91sleep15168-21cyclictest20:31:321
32248217329,34sleep30-21swapper/323:56:153
8565216914,85sleep10-21swapper/100:21:421
8565216914,85sleep10-21swapper/100:21:421
27643216919,133sleep10-21swapper/123:41:151
2736121688,103sleep127437-21sed20:16:131
1823821636,13sleep10-21swapper/123:11:271
16963216211,132sleep10-21swapper/119:41:361
629216018,127sleep2703-21find23:57:072
2021521536,11sleep10-21swapper/121:32:541
25898214811,74sleep10-21swapper/123:36:101
8998214359,15sleep10-21swapper/119:16:281
5191214358,19sleep10-21swapper/120:46:551
13493214112,115sleep26753-21munin-node22:56:222
1099213750,21sleep10-21swapper/122:16:271
1368321365,33sleep10-21swapper/121:12:061
25726213520,94sleep025733-21modprobe20:11:110
12506213510,113sleep112615-21latency_hist21:11:141
2401421345,34sleep10-21swapper/120:04:061
21828213028,89sleep121857-21ps19:56:381
13277212913,25sleep10-21swapper/122:56:131
5866212815,94sleep05917-21/usr/sbin/munin19:06:420
4253212416,22sleep10-21swapper/100:06:481
1459621245,103sleep10-21swapper/121:16:351
17668212325,75sleep217651-21apt-get21:26:292
1453121239,13sleep00-21swapper/022:57:530
16456212117,77sleep00-21swapper/019:41:150
26031212018,70sleep10-21swapper/121:51:401
26031212018,70sleep10-21swapper/121:51:401
21394212014,92sleep021468-21find21:36:450
11329212021,17sleep10-21swapper/119:21:491
3027621195,102sleep10-21swapper/123:47:251
14065211981,18sleep10-21swapper/119:31:421
2405321185,57sleep10-21swapper/123:27:221
2405321185,57sleep10-21swapper/123:27:221
25192211716,88sleep325158-21latency_hist21:51:133
2486321177,95sleep124883-21ntpq20:06:361
32117211622,73sleep232149-21munin-run23:56:102
7760211511,89sleep17807-21systemctl20:56:111
28869211515,85sleep328932-21memory22:01:393
7782211411,19sleep00-21swapper/000:21:120
31407211422,67sleep331489-21apt-key22:11:123
842821108,88sleep08465-21grep00:21:370
842821108,88sleep08465-21grep00:21:370
5266211014,44sleep10-21swapper/122:29:321
1974021108,36sleep10-21swapper/123:16:241
16600210923,52sleep216619-21apt-get23:06:232
21045210817,75sleep221046-21cron19:56:112
2890521070,6sleep00-21swapper/020:21:120
12174210711,78sleep212267-21/usr/sbin/munin00:31:452
978221068,86sleep19301-21apt-get00:26:181
7071210522,46sleep17184-21latency_hist22:36:141
13521210418,63sleep313496-21/usr/sbin/munin22:56:273
856421039,79sleep18581-21modprobe22:41:111
6237210331,62sleep16239-21munin-run00:16:131
2178621036,15sleep30-21swapper/323:21:473
29122210220,56sleep229140-21apt-get20:21:232
23082210214,69sleep222575-21apt-get23:26:222
23082210214,69sleep222575-21apt-get23:26:222
17487210226,14sleep10-21swapper/123:06:471
15649210024,65sleep115714-21sed23:01:421
1150221005,10sleep10-21swapper/122:49:101
265929922,15sleep10-21swapper/122:21:281
905629810,73sleep19106-21if_eth022:41:311
178029810,23sleep10-21swapper/100:01:221
329829719,63sleep10-21swapper/120:41:381
2612729718,59sleep226272-21sed21:51:412
2612729718,59sleep226272-21sed21:51:412
1875229711,70sleep018842-21sed23:11:420
1043629710,71sleep310522-21cat00:27:093
2940429621,15sleep10-21swapper/120:21:341
1622629614,64sleep00-21swapper/023:04:480
125312965,10sleep10-21swapper/100:34:591
108482958,13sleep00-21swapper/019:21:400
1061329518,66sleep10-21swapper/121:02:371
270942949,15sleep10-21swapper/121:56:321
1768429420,14sleep10-21swapper/121:26:401
306422939,18sleep10-21swapper/122:06:421
2270129312,68sleep122784-21apt-config23:26:131
1095829317,53sleep210918-21latency_hist21:06:142
660729216,61sleep36675-21fschecks_time00:16:353
2554629214,62sleep225555-21/usr/sbin/munin23:31:522
2186029114,63sleep221910-21sendmail_mailst19:56:392
539429024,41sleep35422-21munin-run22:31:113
111429018,43sleep31151-21fschecks_count22:16:293
91082895,10sleep10-21swapper/120:58:091
40228815,56sleep3511-21sed20:31:463
379328812,59sleep10-21swapper/119:01:221
227292879,66sleep122749-21ntp_kernel_err21:41:381
1863728714,56sleep218675-21processes23:11:402
1728028730,27sleep30-21swapper/321:26:143
1242428718,53sleep10-21swapper/119:26:381
2728028615,55sleep327308-21munin-plugin-st20:16:113
231728615,54sleep02358-21grep20:37:080
1587328621,49sleep10-21swapper/121:21:231
141322850,7sleep20-21swapper/219:31:422
104728522,33sleep00-21swapper/020:36:110
3218828420,49sleep032234-21sed22:11:500
2888328418,11sleep10-21swapper/123:44:221
194142845,33sleep10-21swapper/119:50:041
1777328418,40sleep017861-21fschecks_time21:26:470
298702838,17sleep129978-21sed22:06:131
2173828318,11sleep10-21swapper/121:37:141
747328212,56sleep27474-21sendmail-mta00:17:472
83422810,7sleep20-21swapper/200:21:342
83422810,7sleep20-21swapper/200:21:342
2650928114,37sleep30-21swapper/323:36:493
246382817,14sleep30-21swapper/323:31:243
2047028128,34sleep320510-21idleruntime-cro21:36:113
1632528118,10sleep10-21swapper/119:40:111
449428022,40sleep04622-21latency_hist20:46:130
1166028019,38sleep311171-21/usr/sbin/munin21:06:433
1746427920,40sleep317562-21unixbench_singl19:41:463
1490727917,39sleep014997-21sort19:36:140
1276427922,31sleep012906-21grep19:26:450
1035927914,50sleep210435-21systemctl00:27:082
464827814,49sleep24742-21latency_hist00:11:132
92952770,9sleep30-21swapper/322:41:363
3020927717,45sleep230300-21fschecks_time22:06:312
2786727725,32sleep20-21swapper/220:16:342
742427612,14sleep00-21swapper/019:11:300
511327626,31sleep35130-21munin-run19:06:123
388727627,27sleep00-21swapper/019:01:300
2698227616,39sleep227124-21unixbench_singl23:37:002
1331627619,24sleep20-21swapper/222:56:142
647227512,49sleep36582-21sshd22:31:483
587627420,31sleep20-21swapper/220:47:082
2982927416,38sleep029919-21timerwakeupswit20:21:450
259927425,29sleep30-21swapper/320:37:483
1738827417,38sleep017486-21sed23:06:460
1523327413,40sleep315242-21/usr/sbin/munin19:36:333
343227319,26sleep30-21swapper/300:06:293
2076327314,44sleep220771-21acpi21:36:222
1869127320,31sleep30-21swapper/323:11:413
754927212,37sleep00-21swapper/022:36:330
2101527215,27sleep00-21swapper/023:21:110
190527220,34sleep21938-21fschecks_count00:01:522
2756427113,40sleep227619-21/usr/sbin/munin21:56:452
2523527113,42sleep025382-21latency_hist21:51:140
1165827116,37sleep011699-21systemd-run22:51:110
689927017,29sleep00-21swapper/020:51:380
3092827014,24sleep30-21swapper/320:26:333
2641827014,43sleep026461-21idleruntime23:36:460
624426811,41sleep26320-21sed00:16:142
515526717,28sleep00-21swapper/000:11:310
459626715,32sleep04626-21munin-run00:11:110
3050526718,33sleep030540-21ntp_kernel_pll_22:06:370
1263626715,36sleep312760-21sed19:26:443
123452679,13sleep00-21swapper/022:51:390
754726618,29sleep37597-21/usr/sbin/munin19:11:333
3229726610,39sleep032345-21cat23:56:160
2865826624,20sleep20-21swapper/222:01:342
130926618,27sleep30-21swapper/300:01:123
2364126518,22sleep30-21swapper/321:46:123
2158226514,32sleep021648-21ls23:21:400
1624526517,29sleep216279-21/usr/sbin/munin21:21:362
1469626517,25sleep30-21swapper/321:16:373
907026417,34sleep09121-21fschecks_time19:16:310
872226420,28sleep28835-21sed20:56:452
2605126413,22sleep00-21swapper/023:36:140
2569426423,21sleep00-21swapper/021:51:300
2569426423,21sleep00-21swapper/021:51:300
1420126417,25sleep30-21swapper/319:31:443
103152648,13sleep00-21swapper/000:27:060
3232026312,21sleep30-21swapper/322:11:533
2359426311,39sleep223640-21sh21:46:112
1734026316,28sleep217463-21/usr/sbin/munin19:41:452
552826224,17sleep30-21swapper/320:47:053
506126217,26sleep25090-21munin-run19:06:112
409826213,34sleep24104-21sort22:26:212
2324526219,15sleep30-21swapper/323:26:563
2324526219,15sleep30-21swapper/323:26:563
2144426215,35sleep321470-21fschecks_count19:56:283
1504226224,17sleep00-21swapper/021:16:460
143312626,13sleep20-21swapper/221:16:282
2972726117,29sleep029769-21ntpq23:46:470
2368326121,19sleep00-21swapper/023:27:090
2368326121,19sleep00-21swapper/023:27:090
175726116,29sleep31274-21apt-get00:01:203
1699126117,22sleep30-21swapper/323:06:363
1073026116,26sleep210769-21/usr/sbin/munin22:46:372
933726012,21sleep20-21swapper/222:41:402
92142606,42sleep00-21swapper/020:59:290
796626020,23sleep20-21swapper/222:36:432
599226015,24sleep20-21swapper/219:06:442
1934426011,38sleep00-21swapper/019:49:100
1718826012,23sleep00-21swapper/019:41:430
1571526016,29sleep215839-21timerwakeupswit23:01:432
441125917,26sleep24449-21modprobe20:46:112
244632598,37sleep224509-21grep20:06:252
1985325914,21sleep20-21swapper/223:16:332
1303725921,17sleep30-21swapper/322:54:053
770625818,18sleep20-21swapper/219:11:372
3209925821,23sleep2313-21rs:main2
2965125811,30sleep229694-21memory23:46:452
270022585,11sleep30-21swapper/321:56:263
2197625811,19sleep00-21swapper/019:56:410
1981925817,17sleep00-21swapper/021:31:440
1169425822,22sleep00-21swapper/021:06:440
1054825815,27sleep310558-21diskstats19:21:313
1023525817,27sleep20-21swapper/219:21:132
2583525711,23sleep30-21swapper/320:11:143
1145025714,19sleep30-21swapper/300:31:293
1101025716,19sleep30-21swapper/322:46:463
412325614,25sleep04119-21apt-get22:26:240
3253025615,24sleep032531-21diskstats23:56:330
2662825615,25sleep226752-21sed20:11:422
2421725615,25sleep21-21systemd20:06:132
2291625614,25sleep322969-21cpuspeed20:01:253
2112425617,18sleep00-21swapper/019:56:130
1975625614,25sleep319752-21apt-get23:16:263
173125613,26sleep20-21swapper/220:36:542
1713525615,26sleep017137-21cron21:26:110
1553825618,24sleep315562-21proc_pri23:01:403
56062558,11sleep00-21swapper/020:47:060
55952558,12sleep30-21swapper/319:06:333
529125516,22sleep30-21swapper/300:11:343
3069025515,25sleep230686-21apt-get20:26:242
2739725512,28sleep027422-21/usr/sbin/munin21:56:390
2731425514,23sleep027359-21systemd-run20:16:120
2309825515,20sleep00-21swapper/020:01:300
339825410,18sleep30-21swapper/320:41:403
2759425413,20sleep30-21swapper/320:16:233
2106225415,18sleep30-21swapper/321:36:353
1888125415,18sleep20-21swapper/219:46:422
1560525413,17sleep20-21swapper/219:36:442
861525315,20sleep30-21swapper/322:41:133
659225310,18sleep20-21swapper/220:51:322
636325314,18sleep20-21swapper/222:31:472
3223825312,20sleep20-21swapper/220:31:252
2385925313,24sleep223878-21apt-get21:46:242
183525312,22sleep21932-21users22:16:442
963425211,24sleep39630-21apt-get21:01:263
719225213,20sleep30-21swapper/320:51:453
303025212,25sleep23036-21ntp_states22:21:372
2979025214,21sleep30-21swapper/323:46:513
2040325212,17sleep30-21swapper/319:51:483
328025112,18sleep00-21swapper/022:21:430
1137125111,23sleep011437-21df00:31:260
612425013,21sleep06140-21ntp_states22:31:410
2776625010,24sleep327758-21apt-get23:41:263
2542325012,22sleep325420-21apt-get21:51:213
2542325012,22sleep325420-21apt-get21:51:213
1201625014,20sleep212012-21apt-get19:26:222
1062225010,16sleep20-21swapper/219:21:342
10252509,18sleep00-21swapper/022:16:220
897224911,14sleep00-21swapper/022:41:300
3044524913,22sleep330433-21latency_hist20:26:133
116532499,23sleep211659-21ntp_states21:06:422
1765324811,18sleep30-21swapper/321:26:223
932724717,15sleep30-21swapper/319:16:383
915324712,21sleep31166-21snmpd20:58:443
2814024712,19sleep20-21swapper/223:41:412
2233624711,21sleep022331-21apt-get21:41:250
1969924714,17sleep20-21swapper/221:31:422
3109424516,15sleep00-21swapper/020:26:380
277682447,13sleep00-21swapper/023:41:270
1844724310,13sleep30-21swapper/319:46:313
285372429,18sleep028556-21apt-get22:01:240
127392397,15sleep20-21swapper/221:11:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional