You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-19 - 19:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Apr 19, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17134215996,1577sleep21231-1kworker/2:2H07:24:172
2223239020,353sleep22225-21/usr/sbin/munin08:19:132
1883023830,7sleep30-21swapper/309:09:403
25943235617,319sleep026012-21/usr/sbin/munin07:49:310
25943235617,319sleep026012-21/usr/sbin/munin07:49:310
4213231028,209sleep34223-21sh08:24:233
24870229919,231sleep324958-21latency_hist07:48:573
24870229919,231sleep324958-21latency_hist07:48:573
12933227717,240sleep113079-21head07:09:351
12933227717,240sleep113079-21head07:09:351
4572226771,93sleep34701-21timerwakeupswit10:19:223
6593226098,66sleep36603-21modprobe08:33:563
5419224916,218sleep05426-21diskstats08:29:160
25857223497,55sleep30-21swapper/309:39:213
11883223384,76sleep311920-21awk07:08:563
11883223384,76sleep311920-21awk07:08:563
2900222735,15sleep30-21swapper/308:19:303
7385220912,181sleep07458-21sendmail_mailtr08:34:250
5954219414,164sleep16051-21timerandwakeup08:29:301
18174219413,132sleep318210-21wc11:09:333
1216921840,19sleep38427-21cyclictest10:48:573
1216921840,19sleep38427-21cyclictest10:48:573
9278217914,63sleep30-21swapper/306:59:353
9278217914,63sleep30-21swapper/306:59:353
133882174114,43sleep30-21swapper/308:54:233
11320217097,52sleep30-21swapper/308:48:583
9638216969,88sleep39678-21ntp_kernel_pll_10:39:173
25578216837,106sleep318273-1kworker/3:0H11:39:453
20968216736,113sleep321053-21cut07:34:523
20968216736,113sleep321053-21cut07:34:523
238302165115,15sleep30-21swapper/309:29:283
3666216338,112sleep33695-21fschecks_count12:19:103
3666216338,112sleep33695-21fschecks_count12:19:103
27097216340,102sleep30-21swapper/307:54:273
27097216340,102sleep30-21swapper/307:54:273
29704215935,105sleep329714-21modprobe08:03:563
22476215818,67sleep322614-21grep09:24:473
32275214831,100sleep30-21swapper/312:04:123
32275214831,100sleep30-21swapper/312:04:123
1543121467,120sleep315157-21/usr/sbin/munin07:19:263
1543121467,120sleep315157-21/usr/sbin/munin07:19:263
2358721428,12sleep00-21swapper/007:44:140
2358721428,12sleep00-21swapper/007:44:140
15034214215,114sleep315085-21processes10:59:283
12687214127,24sleep20-21swapper/207:09:312
12687214127,24sleep20-21swapper/207:09:312
29789213911,36sleep30-21swapper/309:49:353
23211213837,85sleep30-21swapper/307:43:323
23211213837,85sleep30-21swapper/307:43:323
23082213732,15sleep30-21swapper/311:29:153
21997213773,23sleep322093-21switchtime11:24:273
12368213751,14sleep30-21swapper/310:49:103
29312213416,18sleep30-21swapper/311:49:303
21135213316,98sleep121156-21/usr/sbin/munin11:23:571
16797213132,77sleep316943-21tail11:04:333
1162621319,76sleep30-21swapper/308:49:143
8137212925,91sleep31-21systemd08:38:563
1982212912,92sleep30-21swapper/308:18:583
3274121269,102sleep332597-21/usr/sbin/munin10:04:163
6825212526,31sleep20-21swapper/210:28:582
589821257,13sleep30-21swapper/310:24:233
2483212415,18sleep30-21swapper/310:09:293
27362212213,20sleep30-21swapper/311:44:223
24650212217,85sleep124731-21latency_hist09:33:581
19056212231,78sleep319094-21ntpq07:29:233
12754211832,15sleep30-21swapper/307:09:323
12754211832,15sleep30-21swapper/307:09:323
3198221179,92sleep30-21swapper/312:03:563
817621157,10sleep30-21swapper/310:32:143
15927211517,25sleep315966-21modprobe09:03:563
3213121136,38sleep30-21swapper/310:01:213
3213121136,38sleep30-21swapper/310:01:213
616221127,39sleep30-21swapper/312:26:013
1937211228,21sleep32007-21sensors12:09:253
28864211116,78sleep028882-21apt09:49:110
2520321105,10sleep30-21swapper/309:35:253
20327210821,13sleep30-21swapper/311:19:183
1523121089,79sleep214375-21kworker/u8:108:59:312
11241210817,70sleep111304-21sendmail-msp08:48:561
9165210716,71sleep19222-21awk06:59:331
9165210716,71sleep19222-21awk06:59:331
19984210724,48sleep30-21swapper/309:18:583
1514521077,86sleep315230-21sed07:19:093
1514521077,86sleep315230-21sed07:19:093
22263210611,80sleep022323-21/usr/sbin/munin09:24:440
31572210512,74sleep031624-21users11:59:250
19347210516,26sleep00-21swapper/009:13:580
1433210314,74sleep2227-21systemd-journal10:08:562
28988210212,75sleep128999-21ntpq11:49:241
17766210226,53sleep217501-21latency_hist09:08:592
28540210021,61sleep228621-21grep07:59:182
3132829819,61sleep231418-21sed11:59:232
2153129811,46sleep321687-21latency_hist09:23:583
149129812,72sleep11574-21fschecks_time12:09:151
548229718,67sleep00-21swapper/010:24:120
183929612,69sleep21851-21fschecks_count10:09:162
1996429525,54sleep120061-21latency_hist07:33:571
386129315,56sleep03951-21grep08:24:160
2856629320,49sleep128598-21sendmail09:48:561
1958629325,47sleep019612-21munin-run11:18:550
1444329219,48sleep214588-21unixbench_multi07:14:362
1444329219,48sleep214588-21unixbench_multi07:14:362
775929121,32sleep00-21swapper/006:54:320
416529115,55sleep23905-21/usr/sbin/munin10:19:132
1087829111,65sleep010903-21fschecks_count10:44:150
1087829111,65sleep010903-21fschecks_count10:44:150
35242906,11sleep30-21swapper/310:17:243
309472906,12sleep30-21swapper/308:05:013
747428926,28sleep20-21swapper/206:54:262
326042887,11sleep30-21swapper/308:10:583
304222888,10sleep30-21swapper/309:54:153
2006228817,37sleep20-21swapper/207:33:582
1801028819,49sleep117504-21apt-get09:09:071
1658928824,45sleep116626-21sed09:04:221
3120428712,55sleep031282-21fw_packets09:59:120
3120428712,55sleep031282-21fw_packets09:59:120
301032876,10sleep30-21swapper/311:54:353
263222875,11sleep30-21swapper/307:53:183
263222875,11sleep30-21swapper/307:53:183
247382876,11sleep30-21swapper/311:36:593
95212866,11sleep30-21swapper/308:41:433
95212866,11sleep30-21swapper/308:41:433
901128625,23sleep10-21swapper/110:38:591
901128625,23sleep10-21swapper/110:38:591
192372865,11sleep30-21swapper/311:14:263
85842855,10sleep30-21swapper/310:34:083
85842855,10sleep30-21swapper/310:34:083
30152856,12sleep30-21swapper/312:15:193
283372856,10sleep30-21swapper/309:45:593
2041128513,56sleep119941-21apt-get07:34:061
2041128513,56sleep119941-21apt-get07:34:061
183342847,11sleep30-21swapper/307:28:513
1156128417,45sleep210767-21/usr/sbin/munin10:44:292
1156128417,45sleep210767-21/usr/sbin/munin10:44:292
3083128319,50sleep130860-21latency_hist09:58:571
269728320,45sleep22715-21grep12:13:562
1812628333,31sleep20-21swapper/209:09:162
140562835,11sleep30-21swapper/310:56:093
140562835,11sleep30-21swapper/310:56:093
1008728321,29sleep00-21swapper/008:44:180
910528217,48sleep09228-21timerwakeupswit08:39:260
910528217,48sleep09228-21timerwakeupswit08:39:260
1834128225,36sleep018371-21modprobe07:28:560
3238128118,45sleep132423-21latency_hist10:03:581
3238128118,45sleep132423-21latency_hist10:03:581
2585528115,51sleep125892-21sed11:39:511
174552816,10sleep30-21swapper/309:08:363
606828022,38sleep26207-21grep08:29:312
3242528024,25sleep20-21swapper/210:03:592
1738627918,42sleep117438-21latency07:24:411
63927810,53sleep00-21swapper/010:04:250
411327812,21sleep10-21swapper/110:19:121
2342927814,44sleep023320-21/usr/sbin/munin09:29:160
1311527822,35sleep113112-21apt-get08:54:131
1282327816,42sleep012931-21ls07:09:330
1282327816,42sleep012931-21ls07:09:330
376327721,35sleep13759-21apt-get08:24:091
10752776,20sleep00-21swapper/008:14:220
2604127620,30sleep20-21swapper/209:39:242
253122766,14sleep20-21swapper/211:39:002
742527512,46sleep27465-21grep10:29:232
3048327523,24sleep10-21swapper/111:58:571
1659427517,42sleep216622-21munin-run07:23:572
1659427517,42sleep216622-21munin-run07:23:572
2430327418,29sleep10-21swapper/111:33:581
870327319,25sleep20-21swapper/208:39:162
870327319,25sleep20-21swapper/208:39:162
2479027323,32sleep124815-21munin-run07:48:551
2479027323,32sleep124815-21munin-run07:48:551
1049027313,44sleep210564-21sendmail_mailtr08:44:292
439227211,15sleep00-21swapper/010:19:190
2260627219,41sleep122727-21cat07:39:311
2260627219,41sleep122727-21cat07:39:311
223462727,10sleep20-21swapper/211:24:382
1638627216,38sleep216417-21/usr/sbin/munin11:04:232
1581727222,31sleep20-21swapper/211:03:552
744927017,23sleep10-21swapper/106:54:251
1201327023,25sleep00-21swapper/007:09:090
1201327023,25sleep00-21swapper/007:09:090
297912699,13sleep00-21swapper/011:53:470
2672626920,25sleep00-21swapper/011:43:580
2188526924,24sleep10-21swapper/111:24:251
1232226911,42sleep211535-21/usr/sbin/munin08:49:292
73372687,40sleep20-21swapper/208:34:232
3198426817,32sleep00-21swapper/008:09:260
978826711,39sleep19936-21latency_hist08:43:591
836426713,38sleep18490-21date10:33:581
300526717,30sleep13007-21cron10:13:551
281602677,47sleep00-21swapper/011:48:100
2630526721,25sleep10-21swapper/109:39:311
2473326720,22sleep20-21swapper/209:33:592
2380426717,27sleep20-21swapper/207:44:202
2380426717,27sleep20-21swapper/207:44:202
1829126722,23sleep10-21swapper/111:09:351
1538226719,27sleep10-21swapper/107:19:201
1538226719,27sleep10-21swapper/107:19:201
2757026621,22sleep10-21swapper/107:54:371
2757026621,22sleep10-21swapper/107:54:371
821726521,22sleep10-21swapper/108:38:581
513426515,34sleep05202-21df12:24:120
1525126518,28sleep015411-21sed10:59:330
1483826514,28sleep10-21swapper/110:59:221
329326418,30sleep03324-21munin-run12:18:550
2019426423,27sleep00-21swapper/009:19:030
1991426413,27sleep00-21swapper/007:33:560
1037426418,25sleep20-21swapper/207:04:102
1037426418,25sleep20-21swapper/207:04:102
3203426315,30sleep132162-21latency_hist12:03:581
2223026327,20sleep00-21swapper/007:39:220
2223026327,20sleep00-21swapper/007:39:220
219412639,37sleep222000-21cat09:24:222
2124926316,25sleep20-21swapper/207:34:552
2124926316,25sleep20-21swapper/207:34:552
1838626321,22sleep20-21swapper/211:09:362
122526312,16sleep00-21swapper/012:08:580
3107026210,38sleep231064-21apt-get09:59:062
2232426210,18sleep10-21swapper/109:24:451
1049626222,21sleep00-21swapper/007:04:120
43326123,21sleep10-21swapper/108:13:591
3134126113,32sleep131490-21latency_hist08:08:581
294392615,13sleep00-21swapper/008:00:290
2488826115,31sleep224913-21munin-run11:38:552
2317326119,22sleep00-21swapper/011:29:170
215102617,14sleep00-21swapper/011:24:160
1380026113,32sleep113826-21df07:14:151
1380026113,32sleep113826-21df07:14:151
1282126110,14sleep20-21swapper/210:49:212
1247326114,22sleep00-21swapper/008:49:310
1091126116,25sleep20-21swapper/207:04:352
1091126116,25sleep20-21swapper/207:04:352
981726018,20sleep00-21swapper/010:39:220
85762607,12sleep30-21swapper/306:59:153
46222606,11sleep00-21swapper/012:21:050
46222606,11sleep00-21swapper/012:21:050
3235926015,23sleep20-21swapper/212:04:152
3235926015,23sleep20-21swapper/212:04:152
249226016,28sleep10-21swapper/108:19:201
2033526016,25sleep020371-21wc11:19:190
1755926019,20sleep00-21swapper/009:08:570
1090426014,28sleep110993-21fschecks_time10:44:161
1090426014,28sleep110993-21fschecks_time10:44:161
3155825914,27sleep231577-21apt-get08:09:112
3086225924,19sleep20-21swapper/209:58:582
269125913,31sleep02729-21/usr/sbin/munin08:19:270
2672325918,25sleep126850-21latency_hist07:54:001
2642625916,26sleep226519-21apt-config07:53:572
2642625916,26sleep226519-21apt-config07:53:572
2272825913,25sleep222868-21grep07:39:322
2272825913,25sleep222868-21grep07:39:322
2139525914,24sleep00-21swapper/007:34:570
2139525914,24sleep00-21swapper/007:34:570
1632725914,23sleep10-21swapper/111:04:221
2637325813,30sleep126425-21apt-config07:53:561
2556725825,16sleep10-21swapper/109:38:591
84012574,19sleep10-21swapper/108:39:021
66202576,10sleep10-21swapper/110:27:281
3107525713,27sleep20-21swapper/208:06:332
2789625713,25sleep00-21swapper/007:55:330
2789625713,25sleep00-21swapper/007:55:330
2775925714,24sleep227866-21users11:44:292
1436825716,25sleep014407-21/usr/sbin/munin08:58:560
411525617,20sleep10-21swapper/112:19:201
411525617,20sleep10-21swapper/112:19:201
2964625615,25sleep229788-21/usr/sbin/munin09:49:332
2902725616,19sleep10-21swapper/107:59:291
2574225613,19sleep00-21swapper/009:39:100
131942567,16sleep20-21swapper/208:54:182
1201525615,22sleep112021-21ntp_states08:49:241
924325513,24sleep29310-21df10:39:082
895425511,28sleep28955-21cron10:38:562
895425511,28sleep28955-21cron10:38:562
3051525516,18sleep20-21swapper/211:58:582
2431625517,18sleep20-21swapper/211:33:592
2151925511,28sleep221505-21/usr/sbin/munin09:23:572
2118225519,19sleep20-21swapper/211:23:582
1597825516,18sleep10-21swapper/109:03:581
1125225514,18sleep10-21swapper/107:04:461
1125225514,18sleep10-21swapper/107:04:461
97325411,26sleep21040-21memory08:14:202
3227225410,27sleep10-21swapper/112:04:101
3227225410,27sleep10-21swapper/112:04:101
311525416,16sleep00-21swapper/010:13:580
2864825412,24sleep028795-21sed09:48:590
2862625416,23sleep228678-21fschecks_count11:49:172
2552425415,16sleep00-21swapper/009:38:570
2067525416,23sleep120681-21ntp_states09:19:201
1963725417,20sleep20-21swapper/211:18:572
1821225413,25sleep018290-21mailstats11:09:340
1274525415,23sleep012790-21grep10:49:190
95152539,16sleep10-21swapper/110:39:151
87572535,37sleep00-21swapper/010:36:230
87572535,37sleep00-21swapper/010:36:230
2686925313,25sleep226387-21apt-get07:54:052
2686925313,25sleep226387-21apt-get07:54:052
192825313,17sleep10-21swapper/108:18:561
3234625212,18sleep00-21swapper/012:04:130
3234625212,18sleep00-21swapper/012:04:130
2776625213,23sleep227772-21ntp_states09:44:252
1610225210,17sleep00-21swapper/011:04:130
561825117,21sleep10-21swapper/112:24:251
49302517,32sleep20-21swapper/208:27:452
282125112,23sleep02822-21sshd10:11:320
1468025115,19sleep00-21swapper/008:59:160
139912516,35sleep00-21swapper/010:55:170
139912516,35sleep00-21swapper/010:55:170
650525017,17sleep20-21swapper/212:28:582
59825012,22sleep1624-21ntp_states10:04:221
357925011,19sleep20-21swapper/212:19:052
357925011,19sleep20-21swapper/212:19:052
2499125011,22sleep025121-21sed11:38:580
245692505,33sleep10-21swapper/111:34:471
874724911,16sleep00-21swapper/006:59:200
3016724911,22sleep130198-21munin-run09:53:571
1244724912,23sleep112451-21diskstats10:49:121
1993224811,21sleep219983-21systemd-run09:18:572
1581724811,20sleep015889-21mailstats07:19:350
1581724811,20sleep015889-21mailstats07:19:350
1375424815,16sleep00-21swapper/007:14:140
1375424815,16sleep00-21swapper/007:14:140
3045224711,21sleep030458-21ntp_states08:04:250
2763624713,14sleep10-21swapper/111:44:281
333424612,21sleep13340-21sh12:18:561
139332455,11sleep10-21swapper/110:54:331
139332455,11sleep10-21swapper/110:54:331
1924524414,17sleep10-21swapper/111:14:321
70562428,17sleep00-21swapper/010:29:130
3043824210,14sleep00-21swapper/011:58:560
36422418,15sleep10-21swapper/110:18:561
307872419,18sleep01-21systemd09:58:560
234082415,14sleep20-21swapper/209:29:142
140662419,24sleep20-21swapper/210:56:162
140662419,24sleep20-21swapper/210:56:162
36832388,15sleep20-21swapper/210:18:572
33412336,15sleep20-21swapper/212:18:572
1975323310,12sleep00-21swapper/009:16:370
198932306,11sleep10-21swapper/109:18:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional