You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-07 - 05:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Nov 07, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
262802192712,1898sleep326267-21apt-get19:24:563
24442240512,380sleep124434-21apt22:44:471
12705236433,319sleep012767-21cat20:24:550
12705236433,319sleep012767-21cat20:24:550
577323458,325sleep0362-21snmpd20:01:460
1541234238,185sleep01698-21unixbench_singl21:30:020
2586623216,303sleep0195-21jbd2/mmcblk2p2-19:23:030
231632264135,115sleep00-21swapper/022:39:580
6466224961,52sleep00-21swapper/020:04:530
20474223018,134sleep00-21swapper/019:09:350
8732222315,188sleep18762-21wc23:35:051
834322209,197sleep08047-21/usr/sbin/munin23:34:550
29557221818,185sleep329615-21/usr/sbin/munin19:34:533
29557221818,185sleep329615-21/usr/sbin/munin19:34:533
22652221110,146sleep022625-21latency_hist22:39:330
378922099,186sleep03890-21timerandwakeup19:55:040
20796220728,142sleep020859-21mailstats20:50:020
1894822079,142sleep019093-21unixbench_singl22:25:040
1894822079,142sleep019093-21unixbench_singl22:25:040
10175220715,60sleep00-21swapper/021:59:330
7920220470,81sleep00-21swapper/020:09:540
18896220415,128sleep019072-21latency_hist19:04:340
1361521976,146sleep00-21swapper/023:50:050
25689219114,117sleep025835-21unixbench_multi21:05:040
25689219114,117sleep025835-21unixbench_multi21:05:040
25679218556,48sleep00-21swapper/000:30:010
31524218210,112sleep031602-21sort19:40:030
31524218210,112sleep031602-21sort19:40:030
24703218116,103sleep024718-21modprobe21:04:310
20067216616,132sleep020032-21latency_hist20:49:330
621216341,72sleep0712-21switchtime19:45:030
29128216132,45sleep00-21swapper/022:59:480
28298216036,66sleep00-21swapper/019:30:020
28298216036,66sleep00-21swapper/019:30:020
26260216018,126sleep026325-21latency22:49:550
26260216018,126sleep026325-21latency22:49:550
27580215815,127sleep127556-21latency_hist19:29:341
24973215717,61sleep00-21swapper/000:29:330
24973215717,61sleep00-21swapper/000:29:330
27271215325,100sleep127400-21tail21:10:051
1699621477,129sleep0195-21jbd2/mmcblk2p2-00:02:570
18348214427,82sleep0362-21snmpd00:05:230
2682921427,120sleep00-21swapper/021:09:560
1213921415,36sleep00-21swapper/023:45:490
2211521397,15sleep00-21swapper/000:19:480
2211521397,15sleep00-21swapper/000:19:480
27795213819,104sleep327814-21apt-get19:29:443
27795213819,104sleep327814-21apt-get19:29:443
22268213818,100sleep222308-21fschecks_time00:19:582
22268213818,100sleep222308-21fschecks_time00:19:582
23639213511,111sleep023715-21timerwakeupswit19:15:280
1809213313,53sleep00-21swapper/023:14:530
1809213313,53sleep00-21swapper/023:14:530
4774213253,32sleep00-21swapper/023:24:500
3168621318,75sleep0362-21snmpd21:24:370
3168621318,75sleep0362-21snmpd21:24:370
1964421308,56sleep00-21swapper/000:10:070
1964421308,56sleep00-21swapper/000:10:070
1082421280,13sleep30-21swapper/321:59:563
4975212717,92sleep25049-21latency23:24:562
21109212430,49sleep00-21swapper/019:09:470
21109212430,49sleep00-21swapper/019:09:470
6083212312,20sleep30-21swapper/323:29:343
24335212317,32sleep20-21swapper/219:19:342
26459212114,51sleep00-21swapper/019:25:010
94321166,98sleep00-21swapper/023:10:170
94321166,98sleep00-21swapper/023:10:170
3239211422,68sleep23303-21sort23:19:522
28631211343,17sleep00-21swapper/021:15:010
21968211325,66sleep222048-21tune2fs20:54:522
14772211315,86sleep00-21swapper/022:14:180
27550211257,13sleep00-21swapper/022:54:430
27550211257,13sleep00-21swapper/022:54:430
15460211219,28sleep20-21swapper/220:34:342
21022210928,57sleep221042-21apt22:34:312
2132210752,13sleep00-21swapper/019:50:040
4554210547,45sleep04634-21xargs21:40:080
19513210318,26sleep20-21swapper/219:04:572
10154210320,54sleep310294-21grep20:15:053
6184210248,20sleep06216-21nfs_client21:45:570
1005210118,66sleep11058-21/usr/sbin/munin21:29:511
1450029911,71sleep214559-21cat23:55:002
3160329812,70sleep131719-21timerwakeupswit19:40:041
3160329812,70sleep131719-21timerwakeupswit19:40:041
232062988,71sleep023342-21cat20:59:350
1110229718,57sleep111243-21unixbench_multi22:00:031
301529642,38sleep00-21swapper/021:35:030
2488929629,55sleep024896-21ntp_states22:44:580
3126629521,59sleep031376-21latency_hist21:24:330
2039429520,60sleep320460-21latency_hist00:14:343
2039429520,60sleep320460-21latency_hist00:14:343
1756529535,42sleep00-21swapper/020:39:570
2301229420,53sleep122866-21/usr/sbin/munin22:39:551
1766029429,54sleep0362-21snmpd22:21:170
124972947,16sleep30-21swapper/320:24:463
124972947,16sleep30-21swapper/320:24:463
3169329119,42sleep231215-21apt-get21:24:432
3169329119,42sleep231215-21apt-get21:24:432
1012629030,38sleep10-21swapper/123:40:011
889928915,59sleep28900-21sshd20:11:582
578928911,27sleep10-21swapper/121:45:251
242392890,12sleep20-21swapper/200:25:062
242392890,12sleep20-21swapper/200:25:062
2136528918,57sleep221381-21uname19:09:532
2136528918,57sleep221381-21uname19:09:532
788428713,60sleep38027-21latency_hist23:34:343
2081628713,59sleep120852-21uname00:15:091
2081628713,59sleep120852-21uname00:15:091
203692879,40sleep020446-21grep22:30:050
573328630,35sleep35728-21apt-get21:44:473
2258428616,54sleep322618-21ps00:20:073
2258428616,54sleep322618-21ps00:20:073
376628523,44sleep13128-21/usr/sbin/munin23:20:031
2297528520,45sleep322991-21fschecks_count22:39:533
517728414,50sleep25193-21ntp_states20:00:032
2001728416,17sleep30-21swapper/320:49:323
1802128426,39sleep118061-21ps19:00:001
3119328324,38sleep231219-21munin-run21:24:312
2215228318,19sleep10-21swapper/120:54:551
3083328210,51sleep00-21swapper/023:04:520
2685828222,26sleep20-21swapper/219:25:112
2491028220,43sleep324948-21processes22:45:013
2351928213,51sleep223524-21diskstats20:59:492
2185328221,14sleep00-21swapper/000:19:320
2185328221,14sleep00-21swapper/000:19:320
129272826,37sleep00-21swapper/022:05:390
1088028211,53sleep0225-21systemd-journal23:44:320
75542815,65sleep00-21swapper/023:32:010
44992817,59sleep04645-21sed23:24:340
227402810,6sleep30-21swapper/320:55:053
2156028117,21sleep30-21swapper/319:09:583
2156028117,21sleep30-21swapper/319:09:583
395128014,27sleep20-21swapper/221:39:492
1123428013,51sleep111235-21interrupts20:19:561
1123428013,51sleep111235-21interrupts20:19:561
1050128020,44sleep00-21swapper/021:59:490
1021928021,27sleep10-21swapper/121:59:341
3138027922,31sleep10-21swapper/121:24:341
3138027922,31sleep10-21swapper/121:24:341
310272790,5sleep30-21swapper/323:04:543
2960327917,45sleep229816-21sed21:19:352
2093227915,49sleep120944-21df_inode19:09:401
2093227915,49sleep120944-21df_inode19:09:401
1817727916,29sleep30-21swapper/322:24:433
1817727916,29sleep30-21swapper/322:24:433
168527921,37sleep21574-21/usr/sbin/munin19:49:542
149312798,57sleep015056-21cat23:55:100
106852798,60sleep2362-21snmpd23:42:042
407327821,28sleep10-21swapper/121:39:561
2823127814,46sleep30-21swapper/321:14:513
2647227820,35sleep126483-21modprobe00:34:301
949127718,13sleep00-21swapper/020:14:500
1397727724,31sleep214001-21munin-run23:54:312
688727617,30sleep20-21swapper/223:30:092
264672768,13sleep20-21swapper/222:50:012
264672768,13sleep20-21swapper/222:50:012
2534927617,25sleep20-21swapper/219:20:232
2470327619,43sleep324719-21modprobe19:19:473
1407027617,13sleep00-21swapper/020:29:480
83182757,17sleep30-21swapper/323:34:543
3249827524,28sleep20-21swapper/219:44:492
2981427519,14sleep00-21swapper/019:34:580
2981427519,14sleep00-21swapper/019:34:580
2585627519,32sleep30-21swapper/300:30:043
2044727520,36sleep120588-21sed22:30:061
2031027518,43sleep120336-21latency_hist00:14:321
2031027518,43sleep120336-21latency_hist00:14:321
1508227518,12sleep00-21swapper/022:14:430
674627417,27sleep30-21swapper/320:05:003
2791227423,23sleep20-21swapper/221:14:342
2087227415,43sleep120948-21sed20:50:031
2040827424,32sleep120433-21modprobe19:09:321
1143927420,12sleep00-21swapper/020:20:030
1143927420,12sleep00-21swapper/020:20:030
363027311,23sleep30-21swapper/323:20:013
3122827318,29sleep30-21swapper/321:24:323
2888027321,29sleep10-21swapper/122:59:321
2888027321,29sleep10-21swapper/122:59:321
2131227314,18sleep30-21swapper/300:15:193
2131227314,18sleep30-21swapper/300:15:193
1756127328,20sleep20-21swapper/218:59:482
1206927323,25sleep20-21swapper/222:04:522
796927219,38sleep38130-21awk21:50:043
436327214,43sleep34408-21/usr/sbin/munin21:40:023
2966627235,22sleep10-21swapper/119:34:551
2966627235,22sleep10-21swapper/119:34:551
2415727218,40sleep124238-21mailstats00:25:061
2415727218,40sleep124238-21mailstats00:25:061
1532627222,30sleep315332-21modprobe20:34:313
56632715,10sleep30-21swapper/320:00:203
196827115,38sleep31997-21/usr/sbin/munin23:14:563
196827115,38sleep31997-21/usr/sbin/munin23:14:563
77562708,50sleep07794-21processes21:50:010
71527015,29sleep10-21swapper/119:45:041
186827015,30sleep10-21swapper/119:49:571
1616627023,25sleep10-21swapper/123:59:591
1135527017,28sleep10-21swapper/123:44:531
683826818,28sleep10-21swapper/120:05:021
646426815,35sleep15976-21apt-get23:29:441
2782726821,28sleep30-21swapper/321:14:313
2754026813,36sleep327541-21/usr/sbin/munin22:54:423
2754026813,36sleep327541-21/usr/sbin/munin22:54:423
2029326822,22sleep30-21swapper/322:30:043
143026815,22sleep30-21swapper/321:30:013
137826813,25sleep30-21swapper/319:49:343
3225926723,18sleep30-21swapper/319:44:343
3225926723,18sleep30-21swapper/319:44:343
2985926715,26sleep20-21swapper/219:35:002
2985926715,26sleep20-21swapper/219:35:002
2891126718,26sleep20-21swapper/222:59:332
2891126718,26sleep20-21swapper/222:59:332
1400726722,25sleep30-21swapper/322:10:023
123726710,41sleep21274-21ntp_states21:29:562
311292665,16sleep10-21swapper/121:23:421
3109026618,32sleep131127-21sshd23:04:571
2809026619,27sleep20-21swapper/219:29:552
2809026619,27sleep20-21swapper/219:29:552
2351126612,41sleep123518-21df_inode20:59:481
1854826623,27sleep10-21swapper/120:44:341
1777326615,24sleep20-21swapper/220:40:032
1625226620,25sleep10-21swapper/120:35:061
1416026616,32sleep214246-21grep20:29:542
1352226622,21sleep10-21swapper/122:09:471
2584026521,22sleep10-21swapper/121:05:051
2584026521,22sleep10-21swapper/121:05:051
1872526525,22sleep30-21swapper/300:09:343
1800126511,26sleep20-21swapper/222:24:342
1123626519,29sleep211256-21/usr/sbin/munin20:19:572
1123626519,29sleep211256-21/usr/sbin/munin20:19:572
1641826417,29sleep30-21swapper/322:19:343
1433826415,36sleep114365-21cut20:29:571
2973526319,27sleep30-21swapper/323:00:033
1952326310,38sleep119527-21latency_hist22:29:331
1952326310,38sleep119527-21latency_hist22:29:331
113532638,12sleep20-21swapper/222:00:202
3243426217,27sleep231946-21apt-get23:09:432
3243426217,27sleep231946-21apt-get23:09:432
1382026212,36sleep113829-21/usr/sbin/munin20:29:321
1382026212,36sleep113829-21/usr/sbin/munin20:29:321
2375426115,25sleep30-21swapper/300:24:563
2375426115,25sleep30-21swapper/300:24:563
156482617,14sleep00-21swapper/020:34:480
152612615,9sleep30-21swapper/323:55:333
59632609,17sleep10-21swapper/123:29:311
2948526014,25sleep129598-21apt-config21:19:341
2483326015,21sleep30-21swapper/321:04:343
2293426017,21sleep10-21swapper/119:14:491
1662826015,29sleep316646-21/usr/sbin/munin22:19:423
1607826011,20sleep30-21swapper/323:59:573
1228226013,33sleep312351-21sed20:24:343
1228226013,33sleep312351-21sed20:24:343
9392596,12sleep10-21swapper/123:10:141
9392596,12sleep10-21swapper/123:10:141
522525917,27sleep15288-21mailstats20:00:061
3043625913,19sleep10-21swapper/123:04:331
2603025915,27sleep10-21swapper/119:24:331
2184525912,31sleep31-21systemd22:35:003
207025913,28sleep32127-21/usr/sbin/munin19:50:033
2012225921,22sleep10-21swapper/120:49:341
1796125915,21sleep10-21swapper/122:24:331
58425811,34sleep3657-21cat21:29:333
58425811,34sleep3657-21cat21:29:333
389425813,24sleep10-21swapper/119:55:051
2725125819,23sleep1284-21dbus-daemon22:54:321
2725125819,23sleep1284-21dbus-daemon22:54:321
138312588,11sleep10-21swapper/123:52:401
1230825816,22sleep20-21swapper/223:47:572
639125711,24sleep20-21swapper/221:46:032
2595025712,33sleep2362-21snmpd21:05:302
2595025712,33sleep2362-21snmpd21:05:302
2577525713,23sleep20-21swapper/200:30:032
2352425717,17sleep30-21swapper/319:15:153
212292575,10sleep20-21swapper/220:50:122
1557425718,24sleep115611-21modprobe23:59:311
920125613,17sleep10-21swapper/120:14:351
3075725613,27sleep230759-21diskstats23:04:502
294625614,18sleep30-21swapper/321:35:023
1963325618,17sleep30-21swapper/319:05:023
1922325611,27sleep219247-21/usr/sbin/munin20:45:032
1887125622,19sleep30-21swapper/319:04:333
2269825514,18sleep10-21swapper/100:20:091
2269825514,18sleep10-21swapper/100:20:091
2134025513,25sleep121334-21apt-get22:34:461
1751025510,30sleep217605-21fschecks_time00:04:532
886325414,24sleep28859-21apt-get21:54:472
2512425419,16sleep20-21swapper/222:45:042
1598925414,17sleep20-21swapper/223:59:562
11852547,10sleep30-21swapper/323:13:153
11852547,10sleep30-21swapper/323:13:153
103532546,37sleep10-21swapper/120:15:201
862625314,16sleep10-21swapper/121:54:341
371025315,23sleep23787-21cut21:39:332
2508625314,23sleep325091-21diskstats21:04:503
2508625314,23sleep325091-21diskstats21:04:503
1828325318,22sleep20-21swapper/222:24:502
1828325318,22sleep20-21swapper/222:24:502
160225311,25sleep21596-21apt-get23:14:472
160225311,25sleep21596-21apt-get23:14:472
1388825319,16sleep20-21swapper/222:09:572
1269425312,22sleep312713-21apt-get23:49:443
1262025311,17sleep10-21swapper/122:05:031
969025218,14sleep20-21swapper/220:14:562
1383025218,19sleep213972-21latency_hist20:29:332
1383025218,19sleep213972-21latency_hist20:29:332
1013125220,19sleep3284-21dbus-daemon21:59:323
2640625110,17sleep30-21swapper/322:49:583
2640625110,17sleep30-21swapper/322:49:583
240225111,25sleep12395-21apt-get21:34:481
2191425115,17sleep20-21swapper/200:19:342
2191425115,17sleep20-21swapper/200:19:342
625725017,18sleep26258-21/usr/sbin/munin20:04:432
333925011,19sleep20-21swapper/219:54:542
3106925010,15sleep20-21swapper/219:39:522
3106925010,15sleep20-21swapper/219:39:522
200822505,34sleep20-21swapper/200:12:312
200822505,34sleep20-21swapper/200:12:312
1517725011,15sleep20-21swapper/222:14:492
1245924912,16sleep30-21swapper/322:05:013
991924812,17sleep30-21swapper/323:39:553
938124814,15sleep20-21swapper/223:39:342
359324814,18sleep30-21swapper/319:54:593
2805124811,18sleep10-21swapper/119:29:541
2805124811,18sleep10-21swapper/119:29:541
2297124813,20sleep222974-21diskstats22:39:522
2163024816,18sleep30-21swapper/320:54:333
1562624811,22sleep315644-21apt-get20:34:473
1243024810,17sleep30-21swapper/323:49:323
773724710,15sleep20-21swapper/221:49:592
2134124715,16sleep20-21swapper/222:34:472
160124712,17sleep10-21swapper/123:14:461
160124712,17sleep10-21swapper/123:14:461
1728224616,15sleep30-21swapper/320:39:523
82672457,16sleep30-21swapper/320:10:013
174142449,17sleep10-21swapper/100:04:471
2967224312,15sleep20-21swapper/223:00:022
151092438,14sleep10-21swapper/122:14:481
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional