You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 13:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Nov 19, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
287812369710,3641sleep319716-21kworker/3:121:51:113
315925799,551sleep021808-21kworker/0:023:54:230
4323245352,379sleep34362-21wc23:59:193
2334623770,11sleep00-21swapper/021:34:280
32226225786,102sleep332325-21timerandwakeup23:44:213
32226225786,102sleep332325-21timerandwakeup23:44:213
25200225414,223sleep225268-21ls19:59:142
25200225414,223sleep225268-21ls19:59:142
13221225412,227sleep013227-21ntpq22:45:230
20091224818,211sleep120123-21fschecks_count19:44:111
15114224399,74sleep30-21swapper/321:09:123
15114224399,74sleep30-21swapper/321:09:123
17869223849,101sleep317872-21cron21:18:493
17869223849,101sleep317872-21cron21:18:493
32022223177,97sleep332141-21timerandwakeup20:19:213
5284223066,19sleep30-21swapper/320:39:123
7655221919,175sleep37785-21unixbench_multi00:09:243
2866221129,168sleep32807-21latency_hist22:13:533
32712220114,171sleep132734-21latency_hist20:23:521
27406218425,81sleep3225-21systemd-journal21:48:253
18675217974,27sleep30-21swapper/323:03:513
29622217516,145sleep329790-21rm23:38:523
13338217536,71sleep30-21swapper/322:45:263
23490217317,78sleep30-21swapper/321:34:323
5414217010,75sleep30-21swapper/300:04:073
5414217010,75sleep30-21swapper/300:04:073
628521680,37sleep26289-21unixbench_singl00:04:262
628521680,37sleep26289-21unixbench_singl00:04:262
4946216616,125sleep30-21swapper/320:38:523
2512621649,142sleep325145-21apt-get23:24:023
22590215617,121sleep322593-21gpgv21:33:513
22590215617,121sleep322593-21gpgv21:33:513
25654214922,51sleep325805-21unixbench_singl19:59:233
25654214922,51sleep325805-21unixbench_singl19:59:233
23860214960,28sleep30-21swapper/323:19:133
19387214917,63sleep30-21swapper/323:04:283
22070214710,124sleep322122-21if_eth019:50:073
22070214710,124sleep322122-21if_eth019:50:073
12271214514,69sleep312370-21timerandwakeup00:24:223
16657214016,109sleep316675-21latency_hist19:33:513
21038213718,103sleep221115-21grep23:09:232
21038213718,103sleep221115-21grep23:09:232
1362521349,53sleep30-21swapper/300:29:193
28200213216,88sleep10-21swapper/120:09:181
12676213212,56sleep30-21swapper/319:19:183
8449213011,65sleep38448-21munin-run19:08:523
1942121307,74sleep319448-21munin-run21:23:503
1188212217,25sleep30-21swapper/323:49:223
1188212217,25sleep30-21swapper/323:49:223
20007211816,83sleep220074-21/usr/sbin/munin19:44:082
23867211613,44sleep30-21swapper/319:54:273
23867211613,44sleep30-21swapper/319:54:273
25941211518,24sleep30-21swapper/321:43:523
24628211534,49sleep024624-21apt-get21:39:050
14784211423,21sleep30-21swapper/321:08:523
13380211311,88sleep213439-21basename00:29:142
22447211112,86sleep322483-21ntpq23:14:173
2380721106,11sleep20-21swapper/223:19:122
20551211039,52sleep30-21swapper/323:09:123
20551211039,52sleep30-21swapper/323:09:123
11679211020,74sleep111777-21latency_hist20:58:521
30038210920,71sleep230189-21head21:54:252
12624210932,16sleep30-21swapper/322:44:053
12624210932,16sleep30-21swapper/322:44:053
8560210820,74sleep18579-21grep00:14:071
31120210714,74sleep031139-21awk20:18:500
4650210622,41sleep10-21swapper/122:19:031
21152210636,52sleep30-21swapper/319:47:243
17115210638,13sleep30-21swapper/322:58:503
14466210620,65sleep114504-21idleruntime-cro00:33:491
8786210529,15sleep30-21swapper/319:08:563
350021049,83sleep33536-21ntpq22:14:173
3013210435,12sleep30-21swapper/323:54:223
11913210418,45sleep20-21swapper/220:59:052
10856210434,12sleep30-21swapper/320:54:343
934421027,39sleep30-21swapper/322:34:073
6204210237,14sleep30-21swapper/322:24:063
18731210234,13sleep30-21swapper/319:39:173
31139210137,47sleep30-21swapper/321:59:153
27962210127,56sleep128000-21systemctl23:33:491
9971210014,70sleep210192-21latency_hist20:53:532
19713210014,71sleep019753-21modprobe19:43:500
13170210016,48sleep113181-21modprobe21:03:491
20682987,42sleep32058-21apt-get20:29:113
205529812,72sleep02123-21/usr/sbin/munin22:09:210
571629731,36sleep30-21swapper/318:59:283
1821529622,58sleep30-21swapper/319:38:523
127729620,57sleep30-21swapper/322:08:523
724229520,18sleep20-21swapper/200:09:152
2317929416,20sleep20-21swapper/219:53:532
2307929421,35sleep20-21swapper/221:34:062
2008329412,65sleep220115-21awk21:24:172
934229323,42sleep19339-21apt-get22:34:051
909129317,55sleep30-21swapper/300:14:213
82382939,22sleep00-21swapper/000:13:490
565729337,33sleep20-21swapper/218:59:272
2678229327,13sleep30-21swapper/323:29:113
1548829352,19sleep30-21swapper/322:53:503
1193829314,64sleep311949-21modprobe19:18:493
1248929154,18sleep30-21swapper/320:59:233
1065229113,57sleep30-21swapper/300:19:213
99229019,51sleep1697-21/usr/sbin/munin23:49:151
99229019,51sleep1697-21/usr/sbin/munin23:49:151
69712909,23sleep36983-21apt-get20:44:043
987328912,61sleep19969-21latency_hist20:53:521
2017528923,51sleep10-21swapper/123:08:501
2831028858,12sleep30-21swapper/320:09:203
2453028814,55sleep224531-21sshd19:57:242
2453028814,55sleep224531-21sshd19:57:242
77632878,18sleep30-21swapper/322:29:073
701128715,51sleep17117-21users22:24:241
473628713,57sleep30-21swapper/322:19:093
1658628725,40sleep116605-21apt-get21:14:031
1658628725,40sleep116605-21apt-get21:14:031
325022866,12sleep20-21swapper/222:04:102
489628516,51sleep129310-21kworker/u8:120:38:491
3141328519,35sleep00-21swapper/020:19:060
1058528521,31sleep20-21swapper/200:19:202
747028415,52sleep27509-21wc20:44:362
2812728321,30sleep00-21swapper/020:09:170
2655928314,18sleep30-21swapper/320:04:123
1862328315,53sleep118637-21idleruntime-cro23:03:491
1270028310,22sleep10-21swapper/122:44:351
1162128317,50sleep211623-21cron20:58:492
3061528210,60sleep330625-21munin-run21:58:503
956728119,24sleep00-21swapper/019:09:220
298292817,54sleep30-21swapper/320:14:003
2433828117,49sleep124361-21idleruntime-cro21:38:501
1244328116,19sleep00-21swapper/022:43:530
1244328116,19sleep00-21swapper/022:43:530
356728012,53sleep23590-21pluginstate22:14:202
3239028010,13sleep00-21swapper/022:04:000
151728022,44sleep21518-21cpu22:09:072
137762808,60sleep313778-21apt-get19:24:063
1387227818,46sleep013967-21fschecks_time19:24:160
372627710,53sleep23739-21fschecks_count20:34:142
1260227716,44sleep112172-21apt-get22:44:031
874127613,48sleep28804-21if_err_eth000:14:122
85362767,52sleep30-21swapper/320:49:053
585427620,26sleep20-21swapper/220:39:232
2626727616,28sleep20-21swapper/221:44:122
2491327621,40sleep125051-21latency_hist23:23:521
2232427618,43sleep222364-21proc_pri19:50:142
1291727621,35sleep212928-21sh00:28:492
1119127633,32sleep025806-1kworker/0:0H00:21:370
2313327513,48sleep123178-21wc19:53:521
2313327513,48sleep123178-21wc19:53:521
197727522,34sleep21983-21modprobe23:53:492
197727522,34sleep21983-21modprobe23:53:492
1426927514,42sleep114337-21sshd19:24:251
688227419,37sleep07023-21latency_hist19:03:530
3046727424,29sleep20-21swapper/223:39:352
2957927419,30sleep10-21swapper/121:54:161
2748927417,33sleep10-21swapper/121:48:511
1421427319,32sleep214358-21grep21:04:222
607727222,29sleep20-21swapper/222:23:532
2919827213,12sleep20-21swapper/223:34:272
2540727220,33sleep00-21swapper/019:59:200
2540727220,33sleep00-21swapper/019:59:200
1408927218,26sleep10-21swapper/121:04:211
97302717,11sleep00-21swapper/000:17:410
511327118,29sleep00-21swapper/022:19:190
268727110,31sleep20-21swapper/220:29:302
2083327117,20sleep00-21swapper/023:09:170
2083327117,20sleep00-21swapper/023:09:170
1798327127,23sleep20-21swapper/222:59:222
139852710,6sleep10-21swapper/122:48:501
1145327127,23sleep10-21swapper/100:23:521
319127021,28sleep10-21swapper/122:14:111
3125927025,22sleep10-21swapper/121:59:171
2874627017,23sleep00-21swapper/023:34:200
2697527015,40sleep027024-21latency23:29:160
584026921,21sleep10-21swapper/118:59:301
444226915,38sleep14536-21/usr/sbin/munin23:59:211
257426917,32sleep20-21swapper/223:54:132
1519226912,22sleep00-21swapper/021:09:130
1519226912,22sleep00-21swapper/021:09:130
1221126923,21sleep20-21swapper/222:43:512
1221126923,21sleep20-21swapper/222:43:512
979526813,39sleep09870-21gpgv20:53:510
846326822,21sleep00-21swapper/019:08:530
824626818,23sleep00-21swapper/020:48:500
644426816,38sleep16470-21/usr/sbin/munin20:43:501
5862687,45sleep30-21swapper/320:24:103
5862687,45sleep30-21swapper/320:24:103
2252126820,32sleep222548-21munin-run21:33:502
2252126820,32sleep222548-21munin-run21:33:502
173792686,15sleep00-21swapper/022:59:080
1035526819,31sleep00-21swapper/020:54:070
712826716,34sleep17151-21fschecks_count20:44:271
3014426712,11sleep30-21swapper/323:39:083
2053626712,39sleep120441-21/usr/sbin/munin23:09:101
2053626712,39sleep120441-21/usr/sbin/munin23:09:101
2008326714,37sleep020090-21/usr/sbin/munin19:44:100
1952326715,28sleep20-21swapper/223:04:312
954926612,28sleep20-21swapper/219:09:202
669526614,28sleep06465-21apt-get20:43:530
3241726616,33sleep132412-21apt-get22:04:051
2835426610,43sleep128361-21df_inode23:34:091
2517426617,31sleep124606-21/usr/sbin/munin21:39:221
1896326613,38sleep119051-21grep19:39:231
1736526615,30sleep217372-21/usr/sbin/munin19:34:192
1645726626,21sleep10-21swapper/122:54:241
1296926623,26sleep113112-21latency_hist00:28:521
3096526520,22sleep00-21swapper/021:59:120
1459826518,30sleep114629-21/usr/sbin/munin22:49:151
51226421,21sleep10-21swapper/120:24:081
51226421,21sleep10-21swapper/120:24:081
393326415,20sleep00-21swapper/023:59:080
3017226419,24sleep10-21swapper/123:39:221
3015126418,27sleep20-21swapper/220:14:362
230122645,19sleep00-21swapper/019:53:190
230122645,19sleep00-21swapper/019:53:190
168232646,15sleep10-21swapper/119:33:541
688426316,30sleep00-21swapper/022:24:230
2471526317,31sleep224717-21entropy21:39:112
2213726317,28sleep122154-21fschecks_count23:14:111
2039626316,31sleep120550-21unixbench_multi21:24:241
196752635,10sleep30-21swapper/321:23:543
831626221,19sleep20-21swapper/222:29:222
603026215,18sleep00-21swapper/000:04:250
603026215,18sleep00-21swapper/000:04:250
3273726214,22sleep20-21swapper/220:23:532
2339226222,25sleep223336-21latency_hist23:18:522
1867226213,32sleep018761-21grep21:19:230
185826213,28sleep20-21swapper/220:28:532
185826213,28sleep20-21swapper/220:28:532
1558926215,31sleep215601-21/usr/sbin/munin19:29:142
1529726218,22sleep20-21swapper/221:09:152
1529726218,22sleep20-21swapper/221:09:152
108726213,35sleep01185-21cat20:24:230
108726213,35sleep01185-21cat20:24:230
620326117,27sleep26200-21apt-get22:24:052
2983226114,25sleep10-21swapper/120:14:021
1470126111,37sleep014702-21sshd21:08:240
105322616,43sleep00-21swapper/022:37:560
813026016,30sleep08156-21netstat22:29:160
2182426015,29sleep10-21swapper/121:29:211
2182426015,29sleep10-21swapper/121:29:211
1823626013,24sleep00-21swapper/019:38:530
1627126016,22sleep20-21swapper/222:54:212
1458826014,29sleep011-21ksoftirqd/000:33:520
1223326021,24sleep10-21swapper/119:19:081
1199026016,29sleep111966-21latency_hist19:18:511
732625915,29sleep27384-21if_eth019:04:112
6892595,10sleep00-21swapper/023:48:580
6892595,10sleep00-21swapper/023:48:580
65632596,17sleep10-21swapper/100:07:241
65632596,17sleep10-21swapper/100:07:241
2937125920,23sleep229408-21munin-run20:13:512
2773125920,21sleep00-21swapper/021:49:060
2670325911,32sleep226700-21apt-get23:29:062
214452599,27sleep00-21swapper/021:29:120
214452599,27sleep00-21swapper/021:29:120
1667725915,27sleep016822-21sed19:33:520
2979525814,20sleep00-21swapper/023:38:530
1707825820,21sleep20-21swapper/221:14:202
1707825820,21sleep20-21swapper/221:14:202
3188825718,17sleep20-21swapper/223:44:122
3188825718,17sleep20-21swapper/223:44:122
2906925716,22sleep10-21swapper/121:53:521
2830525710,27sleep20-21swapper/221:49:212
2641425714,27sleep226436-21awk23:28:492
1451425717,23sleep014597-21memory22:49:140
1375825714,27sleep213775-21apt19:24:052
2744025611,31sleep027488-21unin-run21:48:500
2427425618,16sleep00-21swapper/023:19:230
178525616,20sleep10-21swapper/122:09:141
1721425614,26sleep017288-21find21:14:220
1721425614,26sleep017288-21find21:14:220
1273525617,22sleep20-21swapper/219:19:212
339825514,17sleep00-21swapper/020:33:520
1862525517,23sleep218626-21ntp_states21:19:212
1712625515,16sleep00-21swapper/022:58:510
402025412,17sleep00-21swapper/022:14:260
2553325414,23sleep00-21swapper/023:24:190
2523325312,19sleep10-21swapper/123:24:121
2337425314,22sleep023370-21apt-get19:54:070
2337425314,22sleep023370-21apt-get19:54:070
2205225314,24sleep022035-21apt-get23:14:060
1528725313,23sleep20-21swapper/222:51:182
839825216,22sleep18408-21modprobe19:08:501
783325212,24sleep17839-21df_inode22:29:091
2646125212,25sleep126480-21apt-get20:04:051
252425212,26sleep12525-21sshd20:29:251
1947325213,23sleep119570-21date21:23:521
1897225212,23sleep018998-21df23:04:160
1583325220,17sleep10-21swapper/119:29:211
403925113,20sleep00-21swapper/020:34:200
2534625113,21sleep125383-21ntp_states19:59:171
2534625113,21sleep125383-21ntp_states19:59:171
23782519,18sleep10-21swapper/123:54:081
1402125115,17sleep20-21swapper/222:48:522
1353225118,18sleep00-21swapper/019:23:510
1258725115,20sleep00-21swapper/020:59:240
1238025113,17sleep00-21swapper/000:24:230
1068425112,26sleep210728-21cpu19:14:082
71862509,15sleep10-21swapper/100:09:141
3111525011,15sleep20-21swapper/221:59:142
2666825014,22sleep20-21swapper/220:04:152
942024912,21sleep29427-21diskstats22:34:102
43162498,20sleep20-21swapper/223:59:182
297272498,14sleep00-21swapper/021:54:190
2675524917,18sleep00-21swapper/020:04:170
3169224711,21sleep131699-21df_inode23:44:071
3169224711,21sleep131699-21df_inode23:44:071
3161624711,20sleep031613-21apt-get23:44:020
3161624711,20sleep031613-21apt-get23:44:020
154352479,16sleep10-21swapper/121:09:181
154352479,16sleep10-21swapper/121:09:181
859824512,17sleep10-21swapper/119:08:541
238352458,12sleep10-21swapper/119:54:251
238352458,12sleep10-21swapper/119:54:251
189002448,13sleep20-21swapper/219:39:202
106822439,17sleep00-21swapper/019:14:060
2016224111,15sleep00-21swapper/021:24:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional