You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-31 - 04:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Wed Dec 31, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
64202654810,6525sleep29311-21kworker/2:100:13:302
13275271034,533sleep113427-21unixbench_singl22:49:311
21061243463,176sleep10-21swapper/119:49:141
15915240715,377sleep215917-21irqstats22:59:212
2121424067,360sleep121283-21sed21:33:591
116102319184,117sleep10-21swapper/122:44:301
1749223130,11sleep00-21swapper/019:38:580
3868230229,239sleep13920-21cat00:04:211
25722275134,115sleep10-21swapper/123:59:281
25722275134,115sleep10-21swapper/123:59:281
6978226318,226sleep27095-21cat22:29:282
8157225150,144sleep18194-21sed19:09:241
2916223134,153sleep12960-21sendmail22:18:571
27791222662,65sleep127808-21modprobe23:38:571
27791222662,65sleep127808-21modprobe23:38:571
12728222626,132sleep10-21swapper/100:34:001
31635222430,131sleep10-21swapper/123:49:191
19410222140,58sleep119529-21timerwakeupswit23:09:291
2595121969,172sleep126038-21latency_hist21:49:011
2595121969,172sleep126038-21latency_hist21:49:011
14220219640,84sleep114244-21modprobe00:38:571
21504219130,17sleep10-21swapper/123:18:301
21504219130,17sleep10-21swapper/123:18:301
28992217653,65sleep10-21swapper/121:58:581
427521757,158sleep1369-21snmpd22:21:121
22979217511,149sleep323018-21ntpq19:54:253
1833821748,28sleep117851-21apt-get21:24:101
611217012,26sleep10-21swapper/123:54:191
825216712,140sleep2974-21latency_hist20:29:002
10330216716,135sleep310335-21sort20:59:093
9677216119,124sleep29618-21latency_hist00:24:002
2260921578,40sleep10-21swapper/119:54:151
13224215723,112sleep10-21swapper/121:09:001
25812215514,118sleep10-21swapper/120:04:161
25812215514,118sleep10-21swapper/120:04:161
5685215421,97sleep10-21swapper/120:44:131
4379215216,112sleep10-21swapper/120:39:211
3150621479,84sleep10-21swapper/122:04:281
3150621479,84sleep10-21swapper/122:04:281
6896214523,64sleep10-21swapper/122:29:271
1592321449,115sleep10-21swapper/119:34:001
9158213863,21sleep10-21swapper/120:54:301
30632213718,38sleep330666-21irqstats20:19:203
29894213732,43sleep10-21swapper/123:44:141
65821346,12sleep10-21swapper/120:27:581
2366421327,84sleep10-21swapper/123:24:191
773221299,63sleep17792-21find20:49:381
773221299,63sleep17792-21find20:49:381
3010021298,38sleep10-21swapper/121:59:471
14536212628,64sleep114570-21ps22:54:271
8535212224,84sleep18655-21timerwakeupswit22:34:281
26055212227,61sleep1369-21snmpd23:31:261
2672021208,78sleep126798-21xargs21:49:301
2672021208,78sleep126798-21xargs21:49:301
16870212025,58sleep10-21swapper/121:19:211
16870212025,58sleep10-21swapper/121:19:211
964021188,51sleep19676-21latency_hist00:23:591
21662211822,37sleep10-21swapper/121:34:191
21662211822,37sleep10-21swapper/121:34:191
21593211812,25sleep30-21swapper/319:49:283
2485721179,92sleep124981-21sed19:59:321
2485721179,92sleep124981-21sed19:59:321
2185521168,42sleep10-21swapper/123:19:111
9632211410,69sleep19639-21fschecks_time22:39:211
17738211325,26sleep10-21swapper/119:39:171
17738211325,26sleep10-21swapper/119:39:171
17738211325,26sleep10-21swapper/119:39:171
5261211211,56sleep10-21swapper/122:24:271
3144211228,72sleep13237-21switchtime20:34:281
3144211228,72sleep13237-21switchtime20:34:281
2468221110,10sleep30-21swapper/319:59:293
2468221110,10sleep30-21swapper/319:59:293
18337211016,74sleep017851-21apt-get21:24:090
1623921098,36sleep10-21swapper/119:34:161
13951210930,68sleep114032-21mailstats21:09:281
473121089,17sleep00-21swapper/000:06:340
1956921087,66sleep119376-21apt-get19:44:541
1956921087,66sleep119376-21apt-get19:44:541
12751210810,83sleep212753-21sshd19:23:312
29062210716,69sleep329212-21grep23:39:443
16259210738,13sleep10-21swapper/121:18:491
7432210634,53sleep37474-21modprobe19:08:583
3308210512,30sleep23317-21/usr/sbin/munin22:19:152
27895210529,38sleep128019-21cat20:09:291
27895210529,38sleep128019-21cat20:09:291
26048210412,78sleep226051-21sshd23:31:232
21996210315,64sleep321831-21/usr/sbin/munin23:19:173
3208421027,83sleep00-21swapper/022:06:290
3208421027,83sleep00-21swapper/022:06:290
7365210113,74sleep07424-21tune2fs20:49:290
7365210113,74sleep07424-21tune2fs20:49:290
24332210121,51sleep00-21swapper/021:43:590
785929814,65sleep07966-21grep19:09:190
1091629826,61sleep10-21swapper/119:18:091
3015129719,64sleep230183-21latency_hist20:18:592
2019829715,20sleep00-21swapper/023:14:050
176629729,30sleep11901-21unixbench_singl20:29:321
912529625,48sleep29178-21cut22:38:572
63222967,39sleep30-21swapper/322:29:133
2415529631,11sleep10-21swapper/121:42:191
119929631,11sleep10-21swapper/122:11:471
1735629416,58sleep017414-21/usr/sbin/munin23:04:190
1585729424,29sleep10-21swapper/122:59:201
1166829433,19sleep10-21swapper/121:03:591
2881629326,51sleep10-21swapper/120:14:141
72932929,47sleep17361-21sshd00:14:301
750429130,20sleep10-21swapper/119:09:001
1587129120,51sleep315889-21grep19:33:583
1344329110,21sleep10-21swapper/119:24:231
1024429112,39sleep10-21swapper/100:24:231
808029015,57sleep38128-21/usr/sbin/munin22:34:183
78702909,21sleep00-21swapper/022:34:110
1624628922,43sleep216262-21fschecks_count19:34:172
1383628917,55sleep213953-21latency_hist22:54:002
3085728813,35sleep10-21swapper/120:19:261
625728726,28sleep10-21swapper/100:11:231
371028713,59sleep0314-21rs:main0
371028713,59sleep0314-21rs:main0
31528712,58sleep3378-21awk22:09:223
1354928722,43sleep313564-21fschecks_count21:09:183
1252128627,11sleep10-21swapper/119:20:311
2999128523,30sleep20-21swapper/223:44:192
1938228514,55sleep219002-21apt-get19:44:142
1938228514,55sleep219002-21apt-get19:44:142
1938228514,55sleep219002-21apt-get19:44:142
1217128517,48sleep212323-21unixbench_singl00:29:332
41422839,14sleep20-21swapper/220:39:132
2918428317,48sleep329215-21/usr/sbin/munin20:14:243
1826628322,29sleep10-21swapper/123:06:351
2608928218,30sleep00-21swapper/021:49:080
2608928218,30sleep00-21swapper/021:49:080
1529528223,38sleep015379-21sshd19:29:300
2454828112,53sleep224579-21/usr/sbin/munin19:59:252
2454828112,53sleep224579-21/usr/sbin/munin19:59:252
626728019,41sleep06354-21fschecks_time19:04:190
2965128013,44sleep229762-21cut23:43:592
1878128012,55sleep318824-21ntp_kernel_err21:24:563
3273127919,31sleep00-21swapper/023:53:580
2531227922,31sleep00-21swapper/021:44:320
2531227922,31sleep00-21swapper/021:44:320
2393927821,35sleep223984-21latency_hist19:59:002
621727714,43sleep26291-21/usr/sbin/munin20:44:302
3169827715,46sleep231794-21latency_hist20:23:592
228272770,6sleep00-21swapper/019:54:210
2116327728,28sleep30-21swapper/321:33:573
103952779,39sleep00-21swapper/019:14:280
544227620,29sleep30-21swapper/320:43:593
203612768,16sleep00-21swapper/021:29:260
1438027623,33sleep20-21swapper/219:29:002
2365527513,32sleep20-21swapper/221:39:292
1957727523,29sleep30-21swapper/319:45:003
1957727523,29sleep30-21swapper/319:45:003
1474927523,24sleep00-21swapper/022:54:300
2392027416,41sleep023984-21ps23:24:270
1364727419,23sleep20-21swapper/200:34:302
1232427422,28sleep30-21swapper/300:29:343
68727317,27sleep00-21swapper/022:09:320
433527319,31sleep30-21swapper/300:04:313
3102127313,47sleep30-21swapper/322:04:143
3102127313,47sleep30-21swapper/322:04:143
2788727319,39sleep328008-21apt-key23:38:593
2788727319,39sleep328008-21apt-key23:38:593
2708427320,36sleep027229-21sed20:09:000
2443827325,23sleep20-21swapper/223:25:302
2303027315,37sleep323004-21/usr/sbin/munin21:39:143
1347227322,30sleep00-21swapper/000:34:270
1105727310,48sleep311064-21df_inode22:44:153
854227217,25sleep20-21swapper/220:53:592
854227217,25sleep20-21swapper/220:53:592
1217427215,39sleep212190-21/usr/sbin/munin21:04:202
323527116,25sleep00-21swapper/022:19:130
3164827122,23sleep00-21swapper/020:23:570
3157527118,28sleep20-21swapper/222:04:292
3157527118,28sleep20-21swapper/222:04:292
2908227123,27sleep20-21swapper/221:59:002
2835027118,37sleep10-21swapper/123:39:131
2741927117,33sleep20-21swapper/221:53:592
2741927117,33sleep20-21swapper/221:53:592
2928527017,21sleep30-21swapper/321:59:013
2816627020,34sleep028244-21mailstats21:54:290
1695127018,37sleep016984-21latency_hist23:03:590
1532527020,34sleep315329-21cron22:58:573
1447127022,26sleep20-21swapper/222:54:242
1031327026,26sleep30-21swapper/319:14:273
777626915,30sleep20-21swapper/200:15:572
2765026916,37sleep327694-21ntpq20:09:233
2765026916,37sleep327694-21ntpq20:09:233
2582426916,39sleep21-21systemd21:48:582
2582426916,39sleep21-21systemd21:48:582
2142926914,37sleep221463-21/usr/sbin/munin19:49:232
1011926915,39sleep310153-21irqstats00:24:213
2681126812,42sleep00-21swapper/023:34:240
2681126812,42sleep00-21swapper/023:34:240
1562026815,31sleep30-21swapper/322:59:103
3233926713,38sleep032384-21ntpq20:24:260
249926716,37sleep22537-21ntpq23:59:252
249926716,37sleep22537-21ntpq23:59:252
1728126725,20sleep30-21swapper/323:04:183
310282665,51sleep00-21swapper/023:47:240
19482657,13sleep30-21swapper/323:59:023
16522658,48sleep00-21swapper/023:57:170
308826411,39sleep30-21swapper/300:00:193
308826411,39sleep30-21swapper/300:00:193
2738526416,29sleep227446-21/usr/sbin/munin23:34:342
2738526416,29sleep227446-21/usr/sbin/munin23:34:342
229826416,29sleep22410-21sort22:14:302
1912126412,37sleep219263-21unixbench_singl21:25:042
88426316,28sleep21015-21/usr/sbin/munin22:09:342
652926316,22sleep00-21swapper/022:29:180
2518726322,22sleep30-21swapper/321:44:313
2518726322,22sleep30-21swapper/321:44:313
1954926317,28sleep019568-21modprobe19:44:530
1954926317,28sleep019568-21modprobe19:44:530
1199426313,32sleep312048-21fschecks_count21:04:173
972226214,32sleep09816-21memory22:39:240
944826210,40sleep39449-21apt-get22:39:153
884026216,28sleep38919-21systemctl00:19:293
623826215,28sleep00-21swapper/022:29:020
2511626218,19sleep30-21swapper/323:29:183
2078326221,20sleep20-21swapper/223:14:272
2078326221,20sleep20-21swapper/223:14:272
1550926221,26sleep30-21swapper/321:14:273
1223226217,19sleep00-21swapper/019:19:490
1173626218,27sleep211881-21grep22:44:312
350326118,25sleep33562-21/usr/sbin/munin22:19:203
2345326115,21sleep30-21swapper/323:24:133
2027926111,20sleep00-21swapper/023:14:130
2027926111,20sleep00-21swapper/023:14:130
1948426120,26sleep319376-21apt-get19:44:243
1948426120,26sleep319376-21apt-get19:44:243
1948426120,26sleep319376-21apt-get19:44:243
175242614,17sleep30-21swapper/321:19:523
175242614,17sleep30-21swapper/321:19:523
1673326117,28sleep216795-21/usr/sbin/munin21:19:182
1673326117,28sleep216795-21/usr/sbin/munin21:19:182
1392626115,29sleep013950-21/usr/sbin/munin21:09:270
1308426119,19sleep30-21swapper/322:49:283
100226119,21sleep20-21swapper/223:54:282
83652606,11sleep30-21swapper/320:52:183
83652606,11sleep30-21swapper/320:52:183
711126019,20sleep30-21swapper/300:14:243
299526014,32sleep33091-21latency_hist22:18:593
2166526017,27sleep021742-21/usr/sbin/munin19:49:300
1803526013,31sleep018228-21apt-config21:24:010
1803526013,31sleep018228-21apt-config21:24:010
1432626012,32sleep314448-21cat00:38:593
1182226017,21sleep30-21swapper/319:19:403
1097926013,31sleep211003-21munin-run19:18:582
801925916,23sleep20-21swapper/219:09:212
2145225915,26sleep321447-21apt-get21:34:133
2145225915,26sleep321447-21apt-get21:34:133
214232596,41sleep00-21swapper/021:34:060
214232596,41sleep00-21swapper/021:34:060
157612596,42sleep20-21swapper/219:32:312
1154025918,19sleep00-21swapper/022:44:290
842525817,20sleep30-21swapper/319:09:313
197752585,11sleep20-21swapper/223:11:122
987925713,19sleep29876-21apt-get00:24:122
674725714,26sleep06742-21apt-get00:14:130
485325716,25sleep24860-21diskstats22:24:162
325852577,11sleep20-21swapper/223:52:352
323825718,20sleep20-21swapper/220:34:292
323825718,20sleep20-21swapper/220:34:292
3002725713,28sleep030099-21sensors_temp21:59:460
2782325718,22sleep00-21swapper/020:09:280
2782325718,22sleep00-21swapper/020:09:280
2570125710,18sleep20-21swapper/220:04:092
2570125710,18sleep20-21swapper/220:04:092
219625718,25sleep02213-21uniq23:59:190
219625718,25sleep02213-21uniq23:59:190
2194325715,17sleep20-21swapper/223:19:162
1172725716,16sleep20-21swapper/221:04:002
900025613,25sleep29030-21if_err_eth020:54:272
566325612,19sleep00-21swapper/020:44:120
944725514,25sleep29441-21apt-get22:39:142
878225513,25sleep08792-21ntp_states00:19:260
51872558,36sleep00-21swapper/020:41:130
2834925512,26sleep028339-21apt-get23:39:120
2580225514,24sleep025811-21diskstats20:04:150
651325419,18sleep30-21swapper/300:14:003
2765725416,21sleep327654-21apt-get21:54:123
2610025416,20sleep30-21swapper/320:04:223
2610025416,20sleep30-21swapper/320:04:223
1771125417,23sleep00-21swapper/019:39:120
1771125417,23sleep00-21swapper/019:39:120
1771125417,23sleep00-21swapper/019:39:120
1500625415,23sleep215074-21df21:14:142
2658425317,18sleep30-21swapper/321:49:283
2658425317,18sleep30-21swapper/321:49:283
25442537,19sleep30-21swapper/320:34:103
25442537,19sleep30-21swapper/320:34:103
214225316,14sleep00-21swapper/022:14:280
2038025315,18sleep20-21swapper/221:29:282
1957625312,24sleep219376-21apt-get19:44:592
1957625312,24sleep219376-21apt-get19:44:592
1372225326,13sleep20-21swapper/221:09:212
130525311,29sleep01384-21sed20:29:220
614025214,22sleep36142-21ntp_states20:44:273
365825212,23sleep23652-21apt-get00:04:142
293222527,15sleep00-21swapper/020:14:290
2027725212,23sleep320272-21apt-get23:14:123
2027725212,23sleep320272-21apt-get23:14:123
887125113,23sleep08874-21diskstats20:54:210
523025112,22sleep35236-21ntp_states22:24:253
2856425114,22sleep228575-21munin-run20:13:582
2856425114,22sleep228575-21munin-run20:13:582
171282515,11sleep30-21swapper/319:35:153
1432925111,24sleep014355-21munin-run19:28:580
102725114,21sleep21024-21apt-get20:29:122
98572504,34sleep30-21swapper/320:56:343
565025012,16sleep00-21swapper/000:09:240
3194125013,23sleep332005-21/usr/sbin/munin20:24:173
3084725011,17sleep20-21swapper/222:04:002
1727625012,15sleep20-21swapper/223:04:172
1544725011,23sleep015458-21ntp_states21:14:240
1378525012,23sleep013817-21munin-run22:53:580
1350925014,20sleep313528-21ntp_states19:24:253
2629924914,14sleep00-21swapper/023:34:000
144724912,21sleep31483-21ntp_states20:29:253
1081224913,21sleep010818-21ntp_states20:59:260
3170824811,15sleep30-21swapper/323:49:213
2171924810,18sleep20-21swapper/221:34:202
2171924810,18sleep20-21swapper/221:34:202
1150324811,16sleep00-21swapper/000:29:190
2177324711,16sleep00-21swapper/023:19:010
1117524613,17sleep00-21swapper/000:28:580
625324514,14sleep20-21swapper/219:04:172
3009124512,19sleep330141-21if_eth023:44:203
187332459,18sleep30-21swapper/323:09:113
2417924410,18sleep00-21swapper/019:59:150
2417924410,18sleep00-21swapper/019:59:150
123732419,14sleep00-21swapper/021:04:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional