You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-20 - 15:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sat Apr 20, 2024 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14280251185,5101sleep213013-21kworker/2:200:27:262
323312151815,1451sleep219949-1kworker/2:2H22:01:512
323312151815,1451sleep219949-1kworker/2:2H22:01:512
3059427586,737sleep219949-1kworker/2:2H21:57:272
7129264513,614sleep37122-21apt-get19:01:493
26054261415,583sleep326061-21users19:57:093
26054261415,583sleep326061-21users19:57:093
4327260110,577sleep24272-21/usr/sbin/munin22:16:372
188625608,538sleep01892-21ntpq22:07:020
11961244317,408sleep311963-21sshd22:37:263
21026239086,200sleep221155-21unixbench_multi23:07:052
21026239086,200sleep221155-21unixbench_multi23:07:052
32648235714,300sleep232690-21true20:21:362
30708233210,213sleep230761-21sendmail_mailqu23:37:022
30708233210,213sleep230761-21sendmail_mailqu23:37:022
1841422699,248sleep01166-21snmpd23:00:210
1157222620,10sleep10-21swapper/120:56:391
9453224743,176sleep20-21swapper/222:31:522
12853224319,207sleep312915-21cat22:42:033
4927223854,160sleep24957-21fschecks_count22:17:262
25339223524,200sleep225384-21unin-run21:41:362
524722299,18sleep20-21swapper/220:36:542
10814222454,153sleep210956-21unixbench_singl20:52:332
6784221643,79sleep20-21swapper/220:41:552
27263221632,84sleep227368-21fschecks_time21:46:532
27263221632,84sleep227368-21fschecks_time21:46:532
30670221313,186sleep330680-21ntpq23:36:593
30670221313,186sleep330680-21ntpq23:36:593
11699221328,114sleep211713-21fschecks_count20:56:532
11798221218,181sleep211850-21cpuspeed00:21:492
2431221146,112sleep20-21swapper/220:27:012
16312221045,78sleep20-21swapper/219:27:122
29519220717,164sleep329532-21apt20:11:353
9524220514,113sleep21-21systemd20:51:362
26718218013,112sleep20-21swapper/220:01:532
26718218013,112sleep20-21swapper/220:01:532
25600217514,125sleep21166-21snmpd21:41:482
9992216938,88sleep2288-21dbus-daemon19:11:372
3044221689,65sleep20-21swapper/220:12:072
3044221689,65sleep20-21swapper/220:12:072
580216767,83sleep20-21swapper/223:46:352
22305216073,24sleep20-21swapper/223:12:062
10063215518,124sleep310141-21grep22:32:093
11430215411,126sleep011570-21cat20:56:390
25729214968,16sleep20-21swapper/223:22:042
28901214751,82sleep229020-21cat20:07:062
2908421459,124sleep229096-21grep21:52:022
8749214252,20sleep20-21swapper/200:11:552
8749214252,20sleep20-21swapper/200:11:552
1252621425,11sleep10-21swapper/119:16:491
1829214148,74sleep20-21swapper/222:07:002
2025621409,119sleep220328-21date19:41:382
10343214013,108sleep19987-21apt-get19:11:401
12254213817,89sleep20-21swapper/219:16:382
56821346,15sleep20-21swapper/220:21:562
29274213315,93sleep129364-21sed23:32:171
29274213315,93sleep129364-21sed23:32:171
22276213250,38sleep212-21rcu_sched21:31:382
25366212937,35sleep20-21swapper/219:56:562
25366212937,35sleep20-21swapper/219:56:562
24840212920,90sleep224869-21latency_hist19:56:362
24840212920,90sleep224869-21latency_hist19:56:362
22961212717,85sleep20-21swapper/221:32:032
22961212717,85sleep20-21swapper/221:32:032
11874212717,96sleep011879-21df00:21:510
19150212650,42sleep219171-21unin-run21:21:372
14125212512,89sleep20-21swapper/219:21:552
3577212447,36sleep20-21swapper/220:31:492
3577212447,36sleep20-21swapper/220:31:492
10734212433,24sleep20-21swapper/200:17:042
19493212352,56sleep219573-21tune2fs21:21:532
5320212212,98sleep25370-21wc00:01:382
28821212216,61sleep20-21swapper/223:32:052
28821212216,61sleep20-21swapper/223:32:052
20955212014,35sleep120975-21/usr/sbin/munin19:42:051
7007211943,16sleep20-21swapper/222:22:072
18579211919,75sleep018789-21latency_hist19:36:390
12176211918,79sleep012178-21cron19:16:360
27468211810,62sleep20-21swapper/223:27:092
27468211810,62sleep20-21swapper/223:27:092
25837211610,18sleep10-21swapper/121:41:591
27715211415,83sleep027727-21/usr/sbin/munin23:27:110
27715211415,83sleep027727-21/usr/sbin/munin23:27:110
31034211210,91sleep00-21swapper/020:16:040
31034211210,91sleep00-21swapper/020:16:040
10711211025,66sleep1231-21systemd-journal22:36:351
10960210913,80sleep310008-21/usr/sbin/munin20:52:343
7661210817,70sleep17806-21latency_hist22:26:381
7191210720,70sleep07206-21sort00:06:530
22047210710,82sleep022048-21sshd21:29:260
856210613,79sleep0880-21netstat20:22:020
831221060,7sleep00-21swapper/020:46:530
32358210313,72sleep332318-21apt-get22:02:123
32358210313,72sleep332318-21apt-get22:02:123
2350921028,42sleep20-21swapper/223:16:382
2350921028,42sleep20-21swapper/223:16:382
13843210247,18sleep20-21swapper/219:21:382
25350210119,54sleep325369-21fschecks_count23:21:533
376221009,28sleep23764-21munin-run23:56:372
3008329930,33sleep10-21swapper/121:56:371
201112997,77sleep220244-21sed23:06:372
201112997,77sleep220244-21sed23:06:372
1337829911,76sleep313391-21basename21:01:593
1176329919,54sleep111901-21sed22:37:081
1151029919,35sleep20-21swapper/222:37:052
2880529822,55sleep028819-21cut21:51:560
756929712,60sleep20-21swapper/200:07:032
2160629726,57sleep321619-21idleruntime-cro23:11:353
2160629726,57sleep321619-21idleruntime-cro23:11:353
3090629523,45sleep131055-21tail23:37:051
3090629523,45sleep131055-21tail23:37:051
1534829415,63sleep115413-21latency_hist19:26:381
153462949,42sleep20-21swapper/221:07:072
52029313,66sleep1552-21sed22:02:391
52029313,66sleep1552-21sed22:02:391
479029315,61sleep34281-21apt-get22:16:463
143632939,68sleep014392-21irqstats22:46:560
2332129215,60sleep123507-21apt-key23:16:371
1480429220,56sleep314952-21grep22:47:073
1393129214,60sleep20-21swapper/222:46:382
227702917,68sleep00-21swapper/021:31:570
227702917,68sleep00-21swapper/021:31:570
2997628927,40sleep329996-21apt-get21:56:353
2371128912,61sleep323706-21apt-get23:16:483
2371128912,61sleep323706-21apt-get23:16:483
2020528931,37sleep020226-21modprobe19:41:360
783728725,28sleep00-21swapper/019:02:060
648728719,50sleep16622-21latency_hist20:41:381
2560328714,57sleep025613-21ntp_states19:57:020
2560328714,57sleep025613-21ntp_states19:57:020
1402728718,40sleep314181-21unixbench_multi00:27:073
170872867,14sleep20-21swapper/221:12:082
1119528616,54sleep311226-21latency19:12:103
2225828515,47sleep122274-21latency_hist21:31:371
834328415,55sleep38387-21grep20:46:573
376528439,23sleep30-21swapper/323:56:383
268002848,61sleep10-21swapper/120:01:561
268002848,61sleep10-21swapper/120:01:561
2370828416,53sleep123182-21apt-get23:16:461
2370828416,53sleep123182-21apt-get23:16:461
1327328410,59sleep20-21swapper/221:01:582
207512837,14sleep20-21swapper/219:41:582
179728312,56sleep11828-21/usr/sbin/munin22:06:591
171172837,14sleep20-21swapper/219:31:482
157532827,52sleep20-21swapper/200:32:152
142412829,39sleep20-21swapper/222:46:542
10532829,57sleep20-21swapper/223:46:562
274128125,22sleep30-21swapper/322:11:373
800928012,53sleep18121-21latency_hist20:46:381
211828012,52sleep02186-21grep20:26:550
145012805,64sleep00-21swapper/000:30:150
60602797,60sleep26109-21mailstats00:02:062
60602797,60sleep26109-21mailstats00:02:062
2689427911,53sleep326951-21cat20:01:583
2689427911,53sleep326951-21cat20:01:583
31427814,45sleep0368-21/usr/sbin/munin22:02:340
31427814,45sleep0368-21/usr/sbin/munin22:02:340
1627927815,43sleep016330-21sort21:11:490
982127713,27sleep00-21swapper/022:32:020
2822527726,29sleep30-21swapper/323:31:383
2822527726,29sleep30-21swapper/323:31:383
1792427722,31sleep118066-21grep22:57:071
362227616,45sleep13743-21timerwakeupswit22:12:081
2571727628,30sleep30-21swapper/321:41:573
2199027618,29sleep30-21swapper/323:11:593
85822758,33sleep20-21swapper/222:27:072
96527430,26sleep10-21swapper/120:22:071
2425527422,37sleep024336-21cut23:17:420
2425527422,37sleep024336-21cut23:17:420
179427413,45sleep31892-21cat20:26:373
87782730,7sleep30-21swapper/300:11:553
87782730,7sleep30-21swapper/300:11:553
241727313,45sleep12423-21grep23:51:471
2310527320,38sleep123180-21grep21:32:061
2310527320,38sleep123180-21grep21:32:061
155832737,10sleep20-21swapper/222:51:392
62427224,26sleep30-21swapper/323:46:363
2887727216,37sleep328572-21/usr/sbin/munin23:32:063
2887727216,37sleep328572-21/usr/sbin/munin23:32:063
2874527112,44sleep028828-21mailstats20:07:040
1543627113,42sleep115582-21latency_hist22:51:381
1295127110,39sleep112982-21/usr/sbin/munin22:42:051
126872718,18sleep00-21swapper/020:59:010
1089227121,20sleep00-21swapper/000:17:060
65327018,25sleep00-21swapper/023:46:370
3159627016,28sleep10-21swapper/123:41:371
2745327018,32sleep027595-21sed20:02:110
2745327018,32sleep027595-21sed20:02:110
230242705,11sleep20-21swapper/219:48:082
2202227018,24sleep10-21swapper/119:46:541
185082706,11sleep20-21swapper/223:01:322
1449727021,26sleep10-21swapper/121:06:371
564426922,26sleep30-21swapper/300:01:553
564426922,26sleep30-21swapper/300:01:553
427926921,25sleep00-21swapper/020:32:080
427926921,25sleep00-21swapper/020:32:080
2739126921,29sleep10-21swapper/123:27:081
2739126921,29sleep10-21swapper/123:27:081
2526826911,41sleep025335-21/usr/sbin/munin23:21:490
1902026912,32sleep00-21swapper/023:01:540
1902026912,32sleep00-21swapper/023:01:540
2533526816,31sleep125337-21cron21:41:351
2241626826,25sleep30-21swapper/319:47:083
1847326811,29sleep30-21swapper/319:36:383
1206526813,41sleep112096-21grep00:21:561
2034626712,39sleep120413-21df23:06:481
2034626712,39sleep120413-21df23:06:481
1948026729,22sleep30-21swapper/323:02:063
1948026729,22sleep30-21swapper/323:02:063
798226613,26sleep10-21swapper/122:26:541
418426618,24sleep30-21swapper/320:32:073
418426618,24sleep30-21swapper/320:32:073
3107426617,32sleep331113-21grep20:16:363
3107426617,32sleep331113-21grep20:16:363
251282668,48sleep00-21swapper/021:38:550
2043026616,31sleep020462-21grep23:06:510
2043026616,31sleep020462-21grep23:06:510
200862667,11sleep10-21swapper/119:40:031
1585726621,19sleep30-21swapper/319:27:013
1202226614,28sleep10-21swapper/120:57:001
1138726610,36sleep00-21swapper/022:36:590
1015926617,33sleep010000-21latency_hist19:11:390
2768726513,38sleep027765-21mailstats21:47:030
2768726513,38sleep027765-21mailstats21:47:030
2246326511,36sleep022499-21ntp_states19:47:090
2008426522,22sleep30-21swapper/321:22:063
1249026520,26sleep00-21swapper/019:16:480
951026423,24sleep10-21swapper/120:51:351
645526416,35sleep06486-21latency_hist20:41:370
496326416,23sleep00-21swapper/022:17:280
451726421,24sleep00-21swapper/022:16:390
367826423,23sleep03687-21modprobe23:56:350
347826412,33sleep30-21swapper/322:12:063
3269126425,19sleep30-21swapper/320:21:373
3115526421,23sleep10-21swapper/120:16:381
3115526421,23sleep10-21swapper/120:16:381
2834526422,20sleep10-21swapper/120:06:531
2587126415,34sleep125945-21find23:22:071
1510326422,23sleep30-21swapper/321:07:003
1302126417,26sleep313090-21/usr/sbin/munin19:17:033
760226313,32sleep37680-21/usr/sbin/munin00:07:043
687026318,21sleep10-21swapper/100:06:371
687026318,21sleep10-21swapper/100:06:371
2820326318,34sleep1373-21sshd21:48:411
2820326318,34sleep1373-21sshd21:48:411
1710726318,23sleep10-21swapper/119:31:471
1265326315,32sleep012659-21df_inode22:41:550
730726222,18sleep00-21swapper/020:42:090
3057626212,25sleep10-21swapper/121:57:221
2426826220,27sleep024365-21timerwakeupswit19:52:180
2426826220,27sleep024365-21timerwakeupswit19:52:180
235942628,37sleep10-21swapper/119:51:581
235942628,37sleep10-21swapper/119:51:581
1499926220,18sleep00-21swapper/000:32:000
13562627,11sleep30-21swapper/323:47:053
3105326017,22sleep30-21swapper/321:57:403
1847326019,19sleep00-21swapper/021:17:110
1156326019,18sleep30-21swapper/300:21:383
921525917,19sleep00-21swapper/022:31:380
73382595,11sleep10-21swapper/119:01:551
685925914,32sleep06869-21unin-run00:06:360
685925914,32sleep06869-21unin-run00:06:360
3148625917,16sleep10-21swapper/120:16:541
3052325916,17sleep00-21swapper/023:36:560
3052325916,17sleep00-21swapper/023:36:560
2197025914,28sleep021980-21/usr/sbin/munin23:11:560
1963525913,29sleep019668-21/usr/sbin/munin21:21:550
172622597,11sleep10-21swapper/121:13:041
1700725915,19sleep00-21swapper/022:56:370
1424925915,19sleep00-21swapper/019:21:570
135562595,11sleep00-21swapper/021:02:040
997225815,24sleep19989-21awk19:11:361
898825812,16sleep30-21swapper/319:07:013
50202586,41sleep00-21swapper/023:58:240
434925820,19sleep10-21swapper/120:32:091
434925820,19sleep10-21swapper/120:32:091
3079325814,27sleep030845-21interrupts21:57:330
2076525813,29sleep320469-21/usr/sbin/munin19:41:593
1795325810,17sleep30-21swapper/321:16:573
2110725714,23sleep00-21swapper/019:42:080
1534125714,25sleep315357-21ntp_states00:32:083
151402576,12sleep00-21swapper/021:07:010
450925615,27sleep34585-21systemctl23:57:063
30012566,38sleep30-21swapper/320:27:553
2764825613,17sleep30-21swapper/321:47:023
2764825613,17sleep30-21swapper/321:47:023
249225618,16sleep10-21swapper/120:27:041
1761525613,24sleep317625-21munin-run21:16:373
1613625618,17sleep30-21swapper/322:52:043
128682564,43sleep30-21swapper/300:24:123
483525514,24sleep14787-21apt-get22:17:171
2873125512,17sleep00-21swapper/023:32:030
2873125512,17sleep00-21swapper/023:32:030
96822546,39sleep00-21swapper/000:13:310
96822546,39sleep00-21swapper/000:13:310
96225414,19sleep10-21swapper/123:46:551
709725416,22sleep17100-21apt-get00:06:451
219632545,9sleep30-21swapper/321:28:243
1912225415,25sleep119147-21awk21:21:361
1620025412,24sleep116310-21timerandwakeup19:27:111
1465325413,20sleep30-21swapper/319:22:083
1171725411,26sleep311820-21fschecks_time20:56:543
467925313,27sleep14803-21sed23:57:081
1940725315,21sleep119401-21apt-get21:21:481
1879725313,23sleep118791-21apt-get23:01:471
1879725313,23sleep118791-21apt-get23:01:471
1465625319,21sleep10-21swapper/100:31:371
643225218,20sleep00-21swapper/022:21:520
2608225213,23sleep026088-21ntp_states21:42:060
2414425214,23sleep324148-21df21:36:523
1751125217,17sleep30-21swapper/322:56:573
642125112,17sleep30-21swapper/322:21:513
2527225114,18sleep10-21swapper/119:56:551
2527225114,18sleep10-21swapper/119:56:551
2271925110,18sleep30-21swapper/321:31:563
2271925110,18sleep30-21swapper/321:31:563
1608425113,22sleep116231-21latency_hist21:11:381
113562517,32sleep10-21swapper/100:19:381
75452507,16sleep20-21swapper/219:02:002
2368125011,23sleep323740-21fschecks_count19:52:053
2368125011,23sleep323740-21fschecks_count19:52:053
906524912,17sleep10-21swapper/100:12:021
906524912,17sleep10-21swapper/100:12:021
90492494,33sleep30-21swapper/322:30:073
3272224915,15sleep00-21swapper/020:21:380
1891424912,23sleep318440-21apt-get19:36:463
1591624912,17sleep00-21swapper/019:27:020
1566324811,21sleep115662-21apt-get22:51:461
87124713,17sleep00-21swapper/023:46:490
583324411,12sleep30-21swapper/320:37:073
1490024210,16sleep10-21swapper/100:31:531
669024110,15sleep30-21swapper/320:41:483
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional