You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 05:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Sep 17, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
183542422211,4198sleep218583-21kworker/2:022:26:082
142802399212,3962sleep221397-21kworker/2:223:57:302
30948217959,1774sleep124839-21kworker/1:121:18:081
30948217959,1774sleep124839-21kworker/1:121:18:081
2141726458,622sleep021462-21ntpq19:07:560
2141726458,622sleep021462-21ntpq19:07:560
2490924930,9sleep30-21swapper/322:47:323
297152433114,303sleep029750-21sh21:17:300
297152433114,303sleep029750-21sh21:17:300
12597239419,358sleep112609-21/usr/sbin/munin22:07:461
2961232968,240sleep02980-21modprobe19:52:300
2961232968,240sleep02980-21modprobe19:52:300
2086423275,16sleep00-21swapper/022:33:050
702231718,282sleep3710-21/usr/sbin/munin21:27:423
702231718,282sleep3710-21/usr/sbin/munin21:27:423
19088231618,146sleep00-21swapper/020:42:450
12267231019,277sleep212276-21unin-run22:07:312
27165228153,210sleep027233-21latency21:07:530
8223226320,222sleep38300-21fschecks_time23:37:533
4431225922,202sleep00-21swapper/023:23:530
10978224650,183sleep011025-21apt-key23:47:310
2669622418,193sleep026848-21date21:07:330
3724224061,166sleep03764-21uname19:53:030
289772237106,22sleep00-21swapper/019:32:310
2012622286,211sleep020199-21find00:13:150
2081222572,24sleep00-21swapper/021:32:320
25836222111,151sleep025854-21awk19:22:300
25836222111,151sleep025854-21awk19:22:300
17299222116,183sleep117396-21fschecks_time22:22:521
11638221711,185sleep011728-21latency_hist22:02:320
11638221711,185sleep011728-21latency_hist22:02:320
14745220922,62sleep00-21swapper/022:13:020
7718220825,143sleep07866-21latency_hist20:07:320
27880220723,119sleep00-21swapper/022:57:320
1187922045,11sleep00-21swapper/022:03:050
21363220311,141sleep00-21swapper/020:48:020
2872721978,154sleep028835-21cat22:58:000
5113219651,87sleep05140-21awk21:42:290
2391221839,100sleep00-21swapper/022:43:020
25181217940,24sleep00-21swapper/022:47:510
545217047,84sleep0669-21timerwakeupswit19:43:090
545217047,84sleep0669-21timerwakeupswit19:43:090
309332170103,51sleep030957-21modprobe23:07:290
309332170103,51sleep030957-21modprobe23:07:290
21710216910,144sleep021809-21timerandwakeup00:18:030
21710216910,144sleep021809-21timerandwakeup00:18:030
13776215931,73sleep013792-21awk22:12:290
963221589,13sleep10-21swapper/120:12:341
20067215163,71sleep020191-21latency_hist22:32:320
2228321468,82sleep022376-21tune2fs20:52:510
1550214413,55sleep00-21swapper/021:28:040
25094214231,63sleep00-21swapper/021:02:320
2123321405,11sleep10-21swapper/122:33:101
2339221390,13sleep20-21swapper/200:23:132
894021387,75sleep0362-21snmpd20:09:140
894021387,75sleep0362-21snmpd20:09:140
6147213410,84sleep06209-21latency_hist20:02:320
1675321335,10sleep00-21swapper/022:20:250
905421325,36sleep00-21swapper/023:38:100
15874213111,104sleep015875-21cron00:02:290
31984213013,97sleep00-21swapper/021:22:590
31984213013,97sleep00-21swapper/021:22:590
16293213018,98sleep216419-21sed22:18:012
30119212928,59sleep00-21swapper/021:17:500
30119212928,59sleep00-21swapper/021:17:500
31688212524,55sleep00-21swapper/023:08:020
31688212524,55sleep00-21swapper/023:08:020
2057212520,55sleep00-21swapper/023:17:510
14371212524,62sleep00-21swapper/023:57:320
24328212456,54sleep024389-21cut00:27:530
30892212223,72sleep130907-21fschecks_count19:37:491
30892212223,72sleep130907-21fschecks_count19:37:491
25580212216,16sleep30-21swapper/300:32:323
1768721227,100sleep22-21kthreadd00:07:392
16664212255,27sleep015934-21/usr/sbin/munin20:33:000
720212057,20sleep00-21swapper/023:12:540
720212057,20sleep00-21swapper/023:12:540
21777212020,84sleep321920-21sed19:08:033
21777212020,84sleep321920-21sed19:08:033
1845321205,60sleep00-21swapper/022:27:230
838621179,75sleep0362-21snmpd21:51:380
32119211616,38sleep032090-21latency_hist19:42:320
32119211616,38sleep032090-21latency_hist19:42:320
560721147,36sleep00-21swapper/021:42:370
2511211410,87sleep22584-21threads23:18:012
1414521149,94sleep3362-21snmpd23:55:453
969521138,56sleep013997-21kworker/u8:320:12:410
14854211048,47sleep00-21swapper/020:27:560
15747210920,70sleep30-21swapper/322:17:493
1385721095,10sleep00-21swapper/020:24:080
31384210715,74sleep331453-21/usr/sbin/munin19:38:033
31384210715,74sleep331453-21/usr/sbin/munin19:38:033
2486121075,12sleep30-21swapper/321:00:093
20202210716,62sleep120340-21grep00:13:161
19472210718,71sleep218989-21apt-get00:12:402
31310210515,65sleep331200-21/usr/sbin/munin23:07:523
31310210515,65sleep331200-21/usr/sbin/munin23:07:523
27386210427,65sleep01-21systemd19:27:300
19141210416,63sleep30-21swapper/322:27:553
26375210217,68sleep126498-21timerwakeupswit00:33:021
16626210212,75sleep116662-21wc00:02:581
5609210119,63sleep25127-21apt-get21:42:392
29244210122,17sleep00-21swapper/019:32:550
1643821017,49sleep00-21swapper/000:02:530
26626210013,73sleep326678-21cut22:52:503
22349210027,19sleep022355-21ntpq22:37:570
22349210027,19sleep022355-21ntpq22:37:570
17517210015,69sleep317664-21latency_hist00:07:323
547129928,15sleep05505-21ps23:28:000
870729842,10sleep00-21swapper/021:52:350
845229721,61sleep38462-21modprobe21:52:303
32392979,72sleep13390-21latency_hist23:22:321
2805829716,63sleep227657-21/usr/sbin/munin19:28:052
2692629719,26sleep00-21swapper/000:36:200
19952976,14sleep30-21swapper/323:17:503
782329610,71sleep27951-21timerwakeupswit21:48:012
2336329613,18sleep10-21swapper/119:12:581
214642968,60sleep10-21swapper/119:07:571
214642968,60sleep10-21swapper/119:07:571
361729419,50sleep33760-21latency_hist21:37:323
2999029415,57sleep230134-21grep19:33:182
997129216,56sleep19972-21sshd21:57:081
690029223,40sleep36919-21munin-run21:47:293
59332919,72sleep1362-21snmpd20:00:231
236682919,29sleep023790-21cat19:13:050
273722898,64sleep227433-21sendmail_mailqu21:07:592
189752887,24sleep00-21swapper/000:12:300
1666328815,59sleep216142-21/usr/sbin/munin00:02:592
1515028815,59sleep215151-21sshd22:15:102
2601528722,39sleep326170-21grep21:03:033
107242860,6sleep10-21swapper/121:57:561
107242860,6sleep10-21swapper/121:57:561
2610028520,44sleep226119-21apt-get19:22:402
2484628519,48sleep124911-21grep19:17:541
2484628519,48sleep124911-21grep19:17:541
227828526,35sleep12272-21apt-get21:32:451
118942850,5sleep20-21swapper/220:17:592
75992848,22sleep0362-21snmpd23:34:470
217202847,12sleep30-21swapper/320:50:073
1762328317,46sleep117663-21sort20:37:501
1762328317,46sleep117663-21sort20:37:501
3184628216,30sleep10-21swapper/121:22:561
3184628216,30sleep10-21swapper/121:22:561
2912128214,53sleep129245-21sed21:13:031
2912128214,53sleep129245-21sed21:13:031
35302815,37sleep00-21swapper/021:37:250
910828018,47sleep39147-21ntpq21:52:543
1261228010,55sleep112621-21systemd-run20:22:311
704027927,33sleep20-21swapper/223:32:592
648627914,50sleep26574-21fschecks_time20:02:542
558427916,18sleep20-21swapper/223:28:022
169527915,47sleep21844-21cut23:17:332
169527915,47sleep21844-21cut23:17:332
3236727815,35sleep30-21swapper/319:42:523
3236727815,35sleep30-21swapper/319:42:523
2973227714,17sleep30-21swapper/319:33:153
2319927717,40sleep223231-21latency_hist22:42:322
2319927717,40sleep223231-21latency_hist22:42:322
203062779,52sleep20-21swapper/222:32:462
1296027714,17sleep10-21swapper/120:22:521
1143827722,13sleep00-21swapper/023:47:390
2820527621,35sleep128293-21fschecks_time22:57:491
2551627620,28sleep20-21swapper/200:32:312
2215627619,17sleep30-21swapper/322:37:523
2215627619,17sleep30-21swapper/322:37:523
1369427619,42sleep113821-21unixbench_multi23:53:011
2624827514,43sleep126323-21fschecks_time19:22:511
2265027521,13sleep00-21swapper/000:22:440
2256827516,28sleep10-21swapper/120:52:561
187102758,19sleep30-21swapper/300:09:103
1921227415,31sleep20-21swapper/222:27:582
1814127446,18sleep017513-21/usr/sbin/munin20:38:010
1814127446,18sleep017513-21/usr/sbin/munin20:38:010
859027316,40sleep28596-21/usr/sbin/munin23:38:002
2499027319,39sleep325043-21ls22:47:463
1800427316,42sleep218069-21awk20:37:592
1800427316,42sleep218069-21awk20:37:592
1513327320,30sleep30-21swapper/323:57:593
867027220,30sleep18803-21head20:08:021
867027220,30sleep18803-21head20:08:021
404927118,35sleep14133-21find23:23:011
2694727114,25sleep10-21swapper/122:52:561
2632727121,20sleep20-21swapper/222:52:332
2275127126,23sleep10-21swapper/100:22:531
1251427123,31sleep212565-21sort22:07:432
43427020,31sleep20-21swapper/221:27:292
43427020,31sleep20-21swapper/221:27:292
391327016,36sleep23917-21ntp_states23:22:582
2743527013,45sleep327502-21/usr/sbin/munin21:08:003
2117927024,19sleep20-21swapper/220:48:002
124942706,12sleep00-21swapper/020:21:290
393326918,22sleep30-21swapper/319:53:063
3119926911,41sleep231201-21/usr/sbin/munin23:07:432
3119926911,41sleep231201-21/usr/sbin/munin23:07:432
2782026919,29sleep10-21swapper/119:28:001
2689726915,32sleep10-21swapper/121:07:461
2588626922,25sleep20-21swapper/219:22:322
2588626922,25sleep20-21swapper/219:22:322
2284526917,20sleep20-21swapper/219:12:382
1176526919,25sleep20-21swapper/223:48:092
1161126918,27sleep30-21swapper/323:48:063
1641026815,27sleep30-21swapper/320:32:553
1107526816,26sleep30-21swapper/320:17:333
2451826714,39sleep124658-21awk20:58:051
22692678,16sleep20-21swapper/221:32:422
1382826721,22sleep30-21swapper/322:12:323
1139626717,33sleep311429-21grep20:17:493
1102626718,19sleep10-21swapper/123:47:321
459126618,32sleep24731-21latency_hist19:57:332
2636426621,27sleep326398-21/usr/sbin/munin19:22:533
2630226619,22sleep10-21swapper/122:52:321
1947426615,35sleep319475-21munin-node00:12:413
1811026613,29sleep10-21swapper/100:07:551
1656326615,31sleep115627-21/usr/sbin/munin22:18:041
1541226618,25sleep10-21swapper/122:17:311
2586226516,23sleep10-21swapper/121:03:011
2494526516,29sleep30-21swapper/319:17:563
2494526516,29sleep30-21swapper/319:17:563
2002126528,18sleep30-21swapper/319:02:593
2002126528,18sleep30-21swapper/319:02:593
1807026515,20sleep30-21swapper/320:38:003
1807026515,20sleep30-21swapper/320:38:003
2345426421,26sleep223512-21sendmail20:57:292
1508726414,25sleep10-21swapper/120:28:011
514626319,22sleep10-21swapper/121:42:301
1099726316,26sleep30-21swapper/321:58:033
1099726316,26sleep30-21swapper/321:58:033
687526222,21sleep30-21swapper/320:03:033
39326216,24sleep10-21swapper/123:12:471
39326216,24sleep10-21swapper/123:12:471
1993726223,24sleep20-21swapper/219:02:582
1993726223,24sleep20-21swapper/219:02:582
1278426219,18sleep10-21swapper/123:52:321
923726120,19sleep20-21swapper/220:12:312
923726120,19sleep20-21swapper/220:12:312
3030126117,26sleep330343-21latency21:17:533
3030126117,26sleep330343-21latency21:17:533
2447726114,30sleep324536-21/usr/sbin/munin00:27:563
2325326115,29sleep30-21swapper/319:12:563
1968126116,28sleep319765-21awk20:43:013
975126014,20sleep30-21swapper/323:42:453
315872609,11sleep30-21swapper/321:22:503
315872609,11sleep30-21swapper/321:22:503
2964626013,26sleep329665-21apt-get23:02:403
2964626013,26sleep329665-21apt-get23:02:403
1649126015,19sleep20-21swapper/220:32:582
323272595,11sleep10-21swapper/119:42:371
323272595,11sleep10-21swapper/119:42:371
2680925921,23sleep20-21swapper/222:52:532
801125814,26sleep38020-21diskstats20:07:483
801125814,26sleep38020-21diskstats20:07:483
627725815,23sleep10-21swapper/121:43:151
2612425818,22sleep20-21swapper/200:32:552
2101425816,24sleep10-21swapper/120:47:551
1915725812,32sleep219163-21/usr/sbin/munin20:42:472
81325714,29sleep232333-21/usr/sbin/munin19:43:112
81325714,29sleep232333-21/usr/sbin/munin19:43:112
655725719,18sleep10-21swapper/123:32:461
498425713,23sleep30-21swapper/323:27:433
480225716,18sleep30-21swapper/319:57:423
3274825714,25sleep332766-21apt-get23:12:413
3274825714,25sleep332766-21apt-get23:12:413
3143325716,24sleep131483-21if_eth023:07:551
3143325716,24sleep131483-21if_eth023:07:551
306412575,11sleep10-21swapper/123:03:471
306412575,11sleep10-21swapper/123:03:471
302922577,12sleep20-21swapper/223:02:592
302922577,12sleep20-21swapper/223:02:592
1453225716,18sleep20-21swapper/220:27:502
1433225715,19sleep30-21swapper/322:12:533
107692575,31sleep20-21swapper/221:57:582
107692575,31sleep20-21swapper/221:57:582
737725613,23sleep10-21swapper/120:04:061
2894525618,18sleep30-21swapper/321:13:013
2894525618,18sleep30-21swapper/321:13:013
2125725620,20sleep20-21swapper/200:17:532
2125725620,20sleep20-21swapper/200:17:532
2118725614,24sleep30-21swapper/300:17:503
2118725614,24sleep30-21swapper/300:17:503
141132566,10sleep20-21swapper/220:27:252
133025613,20sleep20-21swapper/221:28:022
133025613,20sleep20-21swapper/221:28:022
382925514,23sleep23848-21apt-get21:37:432
344025514,24sleep33452-21apt-get23:22:423
2420625512,25sleep124200-21apt-get00:27:451
224125514,18sleep10-21swapper/119:48:271
224125514,18sleep10-21swapper/119:48:271
1945525514,24sleep019450-21apt-get19:02:440
1945525514,24sleep019450-21apt-get19:02:440
75492545,38sleep30-21swapper/323:34:093
2896425413,24sleep229016-21users22:58:022
2475625411,17sleep20-21swapper/222:47:322
2231125413,26sleep222349-21munin-run19:12:302
2231125413,26sleep222349-21munin-run19:12:302
1596025415,24sleep116027-21df20:32:451
1294025414,17sleep30-21swapper/320:22:503
2986525316,23sleep129989-21sed19:33:171
2533325313,24sleep225327-21apt-get21:02:462
170325314,24sleep31688-21apt-get19:47:483
170325314,24sleep31688-21apt-get19:47:483
761325214,21sleep17642-21ntp_states21:47:561
482925212,24sleep24821-21apt-get19:57:452
2201225212,24sleep122098-21fw_packets22:37:501
2201225212,24sleep122098-21fw_packets22:37:501
2056425212,24sleep320578-21irqstats22:32:563
195492529,31sleep119589-21ps20:42:591
1903625215,17sleep10-21swapper/122:27:531
385625116,21sleep33923-21df21:37:483
2979425114,23sleep229840-21latency_hist21:17:322
2979425114,23sleep229840-21latency_hist21:17:322
2811425113,22sleep328110-21apt-get22:57:433
2465925111,22sleep224681-21sshd20:58:062
2265825112,23sleep322653-21apt-get00:22:473
1440825112,23sleep314405-21apt-get20:27:423
986325014,17sleep20-21swapper/220:13:042
96662505,15sleep20-21swapper/221:53:142
318372505,10sleep20-21swapper/219:39:362
318372505,10sleep20-21swapper/219:39:362
1945625012,19sleep10-21swapper/119:02:451
1945625012,19sleep10-21swapper/119:02:451
108122505,36sleep20-21swapper/223:45:512
332824911,21sleep13337-21df_inode19:52:511
2143424812,21sleep121473-21memory00:17:561
2143424812,21sleep121473-21memory00:17:561
1745624811,21sleep317474-21irqstats22:22:543
156442485,31sleep10-21swapper/123:59:351
302432479,14sleep20-21swapper/221:17:522
302432479,14sleep20-21swapper/221:17:522
1222624710,13sleep10-21swapper/122:07:301
79824617,16sleep10-21swapper/121:27:481
79824617,16sleep10-21swapper/121:27:481
540824616,16sleep10-21swapper/123:27:571
279142468,14sleep30-21swapper/319:28:023
218224618,15sleep20-21swapper/219:48:242
218224618,15sleep20-21swapper/219:48:242
1507324311,20sleep214583-21/usr/sbin/munin23:57:582
2351924111,14sleep10-21swapper/122:42:511
221962406,17sleep30-21swapper/320:52:463
81362397,15sleep10-21swapper/123:37:461
1220323912,14sleep30-21swapper/322:07:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional