You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 09:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Aug 27, 2025 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21802114224,1102sleep22181-21ntp_states23:44:172
214672111314,1084sleep06895-21kworker/u8:323:04:020
25098278617,750sleep225167-21/usr/sbin/munin23:14:172
1051254716,514sleep11087-21/usr/sbin/munin21:59:061
13922314147,148sleep31406-21modprobe20:18:443
27216230712,281sleep0362-21snmpd23:21:180
242402294108,71sleep324031-21/usr/sbin/munin21:29:053
198642277112,55sleep30-21swapper/319:33:463
6073225834,205sleep36141-40mandb23:58:443
13643224061,26sleep30-21swapper/300:20:203
9337222450,29sleep39458-21apt-key20:43:473
28817221839,25sleep30-21swapper/321:44:033
14793220817,177sleep214818-21tune2fs20:59:202
3265220513,100sleep33283-21cpu20:24:013
12413220351,98sleep30-21swapper/300:18:463
32375218711,150sleep30-21swapper/323:38:453
32375218711,150sleep30-21swapper/323:38:453
31543218789,47sleep3225-21systemd-journal21:53:443
29142218220,135sleep229293-21grep23:28:472
30786217715,141sleep30-21swapper/320:08:473
13200217414,48sleep30-21swapper/322:38:453
1233821709,147sleep312472-21timerwakeupswit19:09:163
1849216644,20sleep30-21swapper/323:44:103
19410216510,139sleep319483-21fschecks_time21:14:073
16304216438,17sleep30-21swapper/322:48:473
8045216215,92sleep38139-21cut00:04:053
4573215919,120sleep04658-21latency_hist20:28:460
10677215812,128sleep30-21swapper/322:29:133
21563215413,92sleep30-21swapper/321:19:163
2525215112,125sleep12575-21cpuspeed22:04:031
9596214813,17sleep20-21swapper/200:09:042
65221449,40sleep30-21swapper/323:39:183
24567214016,100sleep024601-21cpu23:14:030
18201213120,91sleep30-21swapper/322:54:083
15091213116,91sleep115183-21fschecks_time22:44:081
3533213013,101sleep23589-21latency20:24:082
5498212813,99sleep05657-21unixbench_multi23:54:170
23702212794,16sleep30-21swapper/323:09:223
16481212760,48sleep20-21swapper/221:04:092
799212516,90sleep1922-21ls23:39:201
1673421239,103sleep3362-21snmpd00:30:133
10817212315,92sleep30-21swapper/300:13:463
28096212015,93sleep328134-21ls19:59:063
27462211913,91sleep227600-21cat23:23:462
27666211716,86sleep327695-21latency_hist19:58:453
19209211710,93sleep119211-21sshd22:56:561
13808211611,51sleep30-21swapper/319:13:563
13451211513,90sleep313460-21cpu22:39:063
13091211522,50sleep20-21swapper/219:13:472
28959211414,53sleep328989-21awk23:28:443
26865211413,87sleep226936-21systemctl21:38:452
22410211220,71sleep222552-21head19:39:192
2304921118,89sleep323197-21modprobe19:43:483
2304921118,89sleep323197-21modprobe19:43:483
13880211113,23sleep00-21swapper/019:14:010
1825721108,50sleep318343-21mailstats21:09:153
15142211010,54sleep315180-21processes20:59:303
24565210922,46sleep30-21swapper/323:14:023
1694121099,84sleep316982-21ntpq22:49:113
1546221099,49sleep315491-21awk00:28:433
1468210848,19sleep30-21swapper/321:59:173
1014321088,87sleep310178-21ntpq20:44:243
2607021079,47sleep31-21systemd19:53:453
22161210717,73sleep322222-21sendmail_mailqu19:39:153
18180210618,75sleep218200-21fschecks_count22:54:072
21447210510,55sleep30-21swapper/319:38:473
6890210411,76sleep06972-21mailstats23:59:170
1787210421,44sleep30-21swapper/320:19:043
2739921038,39sleep30-21swapper/321:39:103
21432210314,71sleep120950-21apt-get23:03:551
3100121029,75sleep30-21swapper/320:08:593
15275210214,71sleep215422-21latency_hist19:18:472
764521018,40sleep311481-21cyclictest22:19:153
1693721018,43sleep317083-21sed21:04:183
1633721010,31sleep016341-21unixbench_singl19:19:170
2146521008,57sleep321461-21apt-get23:04:013
11040210015,60sleep211165-21latency_hist20:48:472
261542998,35sleep30-21swapper/323:19:033
173782999,55sleep317395-21netstat19:24:093
112792999,43sleep30-21swapper/320:49:003
830629813,64sleep17914-21/usr/sbin/munin20:39:091
2779329813,70sleep027870-21timerandwakeup21:39:190
2033229850,21sleep30-21swapper/322:59:163
1402429831,32sleep30-21swapper/320:57:483
1402429831,32sleep30-21swapper/320:57:483
1309729811,70sleep113109-21/usr/sbin/munin20:54:071
1309729811,70sleep113109-21/usr/sbin/munin20:54:071
185282976,79sleep04981-1kworker/0:0H19:28:510
1299629721,60sleep113089-21cut19:13:461
414229610,44sleep30-21swapper/322:09:023
3220529614,66sleep232236-21/usr/sbin/munin21:54:112
3013429611,70sleep330264-21sort23:29:333
3013429611,70sleep330264-21sort23:29:333
2945929518,57sleep228979-21apt-get23:28:552
2945929518,57sleep228979-21apt-get23:28:552
1597229413,66sleep016117-21latency_hist21:03:470
56182937,33sleep30-21swapper/322:13:583
2624129321,54sleep126273-21tune2fs23:19:091
265422927,34sleep30-21swapper/321:34:383
180329222,49sleep11842-21cut20:19:061
1014729224,31sleep30-21swapper/319:04:033
762129125,44sleep17651-21modprobe20:38:441
227572917,31sleep30-21swapper/321:24:083
2167029114,58sleep021821-21grep21:19:170
376329016,55sleep30-21swapper/323:49:153
376329016,55sleep30-21swapper/323:49:153
3142428913,26sleep10-21swapper/120:09:131
19852899,65sleep02017-21/usr/sbin/munin20:19:100
58122885,11sleep30-21swapper/320:30:273
923428725,38sleep20-21swapper/222:24:152
326492879,64sleep032658-21df_inode20:14:050
636428539,34sleep30-21swapper/323:58:483
87552847,61sleep30-21swapper/322:24:003
835728411,55sleep10-21swapper/122:22:201
1746828423,40sleep117480-21modprobe21:08:441
1057828314,55sleep110609-21latency22:29:111
2918228215,51sleep129208-21latency_hist20:03:441
1159928119,25sleep00-21swapper/000:14:150
34732805,12sleep30-21swapper/322:04:503
2177228023,39sleep121814-21fschecks_count19:39:051
1135428031,28sleep00-21swapper/019:08:440
2899227921,35sleep029026-21munin-run23:28:450
2707127916,39sleep027220-21grep19:54:180
2630427915,44sleep126454-21grep21:34:201
2070827917,48sleep120788-21awk19:34:171
3229627821,37sleep132323-21munin-run23:38:431
3229627821,37sleep132323-21munin-run23:38:431
3101227821,37sleep00-21swapper/021:49:210
2902927822,33sleep10-21swapper/123:28:461
2143327822,32sleep121493-21find21:19:141
948227716,41sleep19621-21unixbench_singl22:24:181
925127715,44sleep29336-21dpkg20:43:462
3102927713,47sleep031096-21/usr/sbin/munin20:09:040
2414827716,30sleep20-21swapper/221:29:042
175782770,8sleep20-21swapper/200:34:062
1493427724,23sleep10-21swapper/100:24:171
92427613,44sleep21065-21/usr/sbin/munin23:39:212
31627623,41sleep30-21swapper/321:54:453
1324827626,27sleep00-21swapper/022:38:460
3210127517,29sleep00-21swapper/021:54:090
307462759,13sleep00-21swapper/023:33:420
307462759,13sleep00-21swapper/023:33:420
297427526,29sleep02987-21cron23:48:440
1672427517,44sleep30-21swapper/319:23:453
1292227526,30sleep20-21swapper/220:54:042
1292227526,30sleep20-21swapper/220:54:042
927027418,40sleep09415-21latency_hist00:08:460
711427418,41sleep30-21swapper/320:34:223
699027422,30sleep20-21swapper/220:34:212
685427422,28sleep00-21swapper/020:34:190
292027424,27sleep20-21swapper/222:04:122
2360227414,28sleep20-21swapper/219:44:112
2360227414,28sleep20-21swapper/219:44:112
1848027431,11sleep30-21swapper/300:37:403
2791527324,23sleep20-21swapper/219:59:012
2328927317,41sleep123408-21users21:24:181
1573227326,30sleep20-21swapper/219:19:082
1706427219,23sleep00-21swapper/019:24:020
852027116,24sleep20-21swapper/222:23:462
616227111,45sleep16318-21cat23:58:461
518027115,23sleep20-21swapper/220:29:092
481027116,29sleep10-21swapper/122:09:161
2551727122,27sleep025654-21sed19:49:180
2272227120,34sleep222747-21munin-run23:08:442
1142627120,22sleep20-21swapper/219:08:462
766627011,43sleep07672-21ntp_states22:19:160
2727527022,25sleep10-21swapper/121:39:081
1849527013,43sleep018539-21ntpq22:54:130
1397627025,25sleep10-21swapper/119:14:061
1132827014,38sleep111420-21/usr/sbin/munin00:14:081
570526922,31sleep15737-21tune2fs22:14:041
3159426910,17sleep10-21swapper/121:53:461
297826921,31sleep02984-21ntp_states22:04:140
2107226916,34sleep121265-21sed23:03:471
123326925,20sleep00-21swapper/021:59:100
91682686,11sleep10-21swapper/100:08:061
3080326818,25sleep10-21swapper/123:33:471
3080326818,25sleep10-21swapper/123:33:471
2536926821,22sleep10-21swapper/121:33:471
2078926817,30sleep220922-21unixbench_multi19:34:182
696826727,20sleep20-21swapper/222:18:462
1868126719,26sleep10-21swapper/100:38:471
766926610,13sleep20-21swapper/200:03:432
72126613,16sleep00-21swapper/023:39:190
3212626615,26sleep10-21swapper/121:54:101
3036026615,16sleep20-21swapper/221:49:062
1028426612,38sleep210302-21apt22:28:592
319622659,46sleep0362-21snmpd23:34:260
319622659,46sleep0362-21snmpd23:34:260
2166026511,41sleep00-21swapper/019:38:560
208526515,21sleep20-21swapper/220:19:122
1753526516,27sleep00-21swapper/021:08:470
1706526511,37sleep117144-21date00:33:461
167626511,36sleep21825-21unixbench_singl21:59:202
1584026523,23sleep00-21swapper/000:29:050
1140326518,25sleep111424-21munin-run19:08:451
1014926514,24sleep10-21swapper/100:09:161
3112326416,27sleep20-21swapper/223:34:042
3112326416,27sleep20-21swapper/223:34:042
1294326411,35sleep012995-21sh19:13:450
937026313,33sleep09481-21sed22:24:170
551926313,31sleep15659-21/usr/sbin/munin20:29:161
2328926311,32sleep222993-21/usr/sbin/munin23:09:132
1709726319,28sleep217107-21/usr/sbin/munin19:24:042
508126214,23sleep00-21swapper/020:29:070
457626219,24sleep20-21swapper/223:53:462
457626219,24sleep20-21swapper/223:53:462
273342625,12sleep20-21swapper/221:39:092
1981526214,21sleep00-21swapper/022:59:050
1213326215,31sleep212201-21latency22:34:082
53052616,40sleep00-21swapper/022:13:230
2894126117,29sleep129000-21/usr/sbin/munin21:44:051
950626017,20sleep00-21swapper/020:43:480
251392607,11sleep00-21swapper/021:31:280
2331226011,33sleep023319-21diskstats19:44:050
2331226011,33sleep023319-21diskstats19:44:050
1911426016,25sleep20-21swapper/219:29:172
181982606,12sleep00-21swapper/000:34:180
1680426018,20sleep00-21swapper/022:49:080
1547326010,28sleep20-21swapper/222:44:172
1363326016,29sleep013691-21/usr/sbin/munin22:39:110
1007926014,21sleep00-21swapper/000:09:150
496325917,23sleep20-21swapper/222:09:172
2604425916,27sleep2225-21systemd-journal19:53:442
1677625911,24sleep10-21swapper/119:23:471
1312725915,28sleep113190-21grep00:19:131
2652025815,26sleep126573-21/usr/sbin/munin19:54:071
1462325814,18sleep10-21swapper/120:58:551
2964725712,27sleep00-21swapper/021:44:200
2493425713,20sleep10-21swapper/123:14:121
2050925718,21sleep00-21swapper/019:34:120
177742577,18sleep20-21swapper/221:09:012
1777225714,26sleep117766-21apt-get21:09:001
1721925713,25sleep117345-21/usr/sbin/munin22:49:171
1155225733,12sleep00-21swapper/022:33:250
983125621,18sleep10-21swapper/120:44:181
772125615,20sleep10-21swapper/100:03:471
688825615,22sleep06907-21sendmail22:18:440
2621425610,33sleep026303-21timerandwakeup21:34:190
1481625611,29sleep014933-21timerandwakeup00:24:160
147425610,23sleep20-21swapper/220:18:472
534525513,24sleep25417-21sensors_temp23:54:152
2380725512,28sleep023817-21munin-run21:28:450
2301725513,25sleep123012-21apt-get23:09:031
1966825514,25sleep219664-21apt-get22:58:592
1785325512,17sleep00-21swapper/021:09:030
77922545,12sleep20-21swapper/200:03:482
2454625420,17sleep10-21swapper/119:48:461
2454625420,17sleep10-21swapper/119:48:461
2071825418,16sleep20-21swapper/221:18:462
1319225415,17sleep20-21swapper/200:19:142
1290725416,17sleep00-21swapper/020:54:020
1290725416,17sleep00-21swapper/020:54:020
3004425311,18sleep20-21swapper/221:48:462
2675325314,23sleep226789-21ntp_states19:54:122
219992537,16sleep20-21swapper/221:21:002
1751225313,24sleep116989-21/usr/sbin/munin19:24:151
2499325213,18sleep10-21swapper/119:49:071
2257825213,24sleep222584-21df_inode21:24:032
1879325213,23sleep118830-21irqstats19:29:081
1470325212,25sleep114731-21munin-run22:43:441
129962526,35sleep10-21swapper/122:37:061
1273725215,23sleep012808-21cut00:19:040
1136825216,16sleep00-21swapper/020:49:050
70025113,23sleep10-21swapper/120:14:181
322142516,33sleep20-21swapper/220:12:502
1420125113,21sleep214198-21apt-get00:23:582
353825011,22sleep13631-21memory23:49:091
353825011,22sleep13631-21memory23:49:091
2956125012,21sleep029562-21diskstats20:04:070
1948525013,19sleep00-21swapper/021:14:080
172125012,21sleep11715-21apt-get23:44:041
1594725010,28sleep0362-21snmpd22:44:520
1105825013,20sleep211125-21df00:14:012
839724915,20sleep08398-21ntp_states20:39:120
2923624912,16sleep00-21swapper/020:03:470
1127824912,21sleep211274-21apt-get20:48:592
2996024816,17sleep20-21swapper/220:04:182
2956324812,14sleep10-21swapper/120:04:081
1954424811,21sleep119592-21irqstats21:14:091
1691024812,13sleep20-21swapper/222:49:102
1372124812,17sleep20-21swapper/222:39:132
1165824810,15sleep10-21swapper/120:49:101
334624610,14sleep00-21swapper/023:49:040
334624610,14sleep00-21swapper/023:49:040
2348624511,18sleep023550-21ps23:09:190
159652457,14sleep20-21swapper/200:29:072
333824010,15sleep10-21swapper/120:24:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional