You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 16:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sat May 23, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29493225296,2491sleep321995-21kworker/3:010:02:003
5756262538,495sleep3194-21jbd2/mmcblk2p2-12:13:593
5756262538,495sleep3194-21jbd2/mmcblk2p2-12:13:593
7215252123,486sleep37240-21ntpq10:34:223
25921241912,392sleep025957-21ntpq11:34:190
15014241113,384sleep015057-21ntpq10:59:180
162492346155,45sleep30-21swapper/309:19:243
4613234513,316sleep14652-21ntpq12:09:171
24898231416,279sleep224962-21/usr/sbin/munin08:04:172
24898231416,279sleep224962-21/usr/sbin/munin08:04:172
8305228922,248sleep18357-21latency_hist07:13:561
8305228922,248sleep18357-21latency_hist07:13:561
4326223724,136sleep34361-21fschecks_count12:09:103
1415222611,200sleep21566-21latency_hist08:33:562
5121221138,136sleep35157-21ntpq08:44:193
20607220633,78sleep30-21swapper/309:34:183
30459219132,118sleep329949-21apt-get11:49:083
910621867,163sleep39252-21grep10:39:273
2197218213,85sleep30-21swapper/308:34:263
28760217913,152sleep028771-21awk11:44:120
25638217945,119sleep325693-21fschecks_count11:34:143
10484217817,51sleep30-21swapper/312:29:093
32588217433,104sleep332681-21cat11:54:253
32588217433,104sleep332681-21cat11:54:253
215521748,46sleep32157-21apt-get10:19:083
215521748,46sleep32157-21apt-get10:19:083
26893217122,125sleep126986-21fschecks_time09:54:151
9020216650,101sleep39109-21fschecks_time12:24:133
17990216635,85sleep318000-21modprobe07:43:543
1224821669,73sleep312286-21munin-run09:08:543
6742216585,65sleep36929-21cat07:08:573
6742216585,65sleep36929-21cat07:08:573
2809221518,17sleep00-21swapper/008:14:160
9010214811,123sleep29022-21fschecks_count07:14:132
23321214712,74sleep323354-21fschecks_time07:59:213
23321214712,74sleep323354-21fschecks_time07:59:213
9521214517,105sleep39601-21fschecks_time08:59:133
15372214524,99sleep1225-21systemd-journal09:18:541
12451214212,88sleep312488-21awk07:24:203
4241214015,109sleep04306-21cpuspeed12:09:070
21510213910,89sleep321603-21sed11:19:243
891621358,69sleep30-21swapper/307:14:063
11477213310,104sleep011546-21mailstats09:04:230
22861213135,80sleep323068-21cat07:58:563
22666213113,96sleep122730-21dpkg07:58:541
3285213014,79sleep33363-21grep08:39:143
9017212862,51sleep29019-21diskstats12:24:122
637221286,35sleep30-21swapper/308:49:123
30064212749,56sleep30-21swapper/311:48:553
30076212617,93sleep330143-21idleruntime10:04:143
30076212617,93sleep330143-21idleruntime10:04:143
20665212647,24sleep30-21swapper/311:18:553
1880212618,55sleep30-21swapper/311:59:253
2957212332,71sleep13086-21latency_hist08:38:561
1897212318,80sleep00-21swapper/008:34:190
26925212110,95sleep327042-21timerwakeupswit08:09:263
26925212110,95sleep327042-21timerwakeupswit08:09:263
23804212019,63sleep323953-21latency_hist11:28:573
25856211911,92sleep025930-21find09:49:250
17510211712,92sleep017580-21cat09:24:180
944211511,29sleep30-21swapper/310:14:193
944211511,29sleep30-21swapper/310:14:193
31241211417,80sleep231372-21latency_hist10:08:572
31241211417,80sleep231372-21latency_hist10:08:572
12835211474,20sleep30-21swapper/310:53:563
3267321116,93sleep00-21swapper/010:13:030
26987211114,41sleep227045-21if_eth009:54:162
24365211015,22sleep30-21swapper/309:44:363
12272211010,84sleep012301-21irqstats12:34:150
28426210945,26sleep328460-21latency_hist11:43:553
19538210814,14sleep30-21swapper/307:48:553
12213210810,20sleep30-21swapper/312:34:143
13716210731,60sleep313840-21sed10:54:293
14654210610,25sleep30-21swapper/309:14:243
11541210510,79sleep211566-21/usr/sbin/munin10:49:112
3709210411,75sleep23759-21/usr/sbin/munin10:24:122
3709210411,75sleep23759-21/usr/sbin/munin10:24:122
23094210412,19sleep30-21swapper/311:24:253
11250210348,20sleep311323-21latency_hist10:48:563
31626210037,16sleep30-21swapper/310:09:153
17483210012,73sleep117603-21latency_hist11:08:561
17483210012,73sleep117603-21latency_hist11:08:561
1622210029,46sleep21619-21apt-get08:34:092
827229927,17sleep38309-21ntp_states08:54:213
2737329914,22sleep30-21swapper/309:54:253
2676129913,68sleep026783-21/usr/sbin/munin08:09:230
2676129913,68sleep026783-21/usr/sbin/munin08:09:230
2105029945,20sleep3363-21snmpd07:53:273
3219829815,23sleep20-21swapper/211:54:162
3219829815,23sleep20-21swapper/211:54:162
1458829815,25sleep20-21swapper/209:14:232
303422977,41sleep3363-21snmpd08:19:283
100162979,75sleep010024-21df10:44:120
612829521,55sleep16129-21sshd07:05:001
2497629543,15sleep30-21swapper/311:30:143
1498429526,51sleep30-21swapper/310:59:173
1428129539,18sleep314402-21sed07:29:253
2376529415,64sleep223803-21cut11:28:562
2376529415,64sleep223803-21cut11:28:562
229132945,11sleep00-21swapper/011:24:220
229132945,11sleep00-21swapper/011:24:220
112232948,50sleep311369-21grep07:19:273
81729317,62sleep0832-21kworker/u8:311:58:540
81729317,62sleep0832-21kworker/u8:311:58:540
642929323,49sleep00-21swapper/007:08:540
183002936,40sleep30-21swapper/309:26:233
1781129311,69sleep217818-21df_inode11:09:262
1781129311,69sleep217818-21df_inode11:09:262
172902936,12sleep30-21swapper/311:07:053
172902936,12sleep30-21swapper/311:07:053
153029217,19sleep00-21swapper/011:59:180
2885729115,30sleep30-21swapper/308:15:443
1902829112,66sleep319086-21cat11:13:553
1902829112,66sleep319086-21cat11:13:553
65192907,49sleep36546-21munin-run10:33:543
553529012,60sleep05238-21/usr/sbin/munin10:29:180
341628919,19sleep00-21swapper/010:23:530
341628919,19sleep00-21swapper/010:23:530
267228816,54sleep22762-21mailstats10:19:272
267228816,54sleep22762-21mailstats10:19:272
263022885,37sleep30-21swapper/309:51:143
1915428811,36sleep30-21swapper/309:29:183
1181628811,61sleep011945-21latency_hist07:23:570
581728730,28sleep20-21swapper/207:04:252
526128743,25sleep30-21swapper/307:04:133
232092878,67sleep323263-21munin-run09:43:553
232092878,67sleep323263-21munin-run09:43:553
2004928623,28sleep10-21swapper/111:14:251
1048528618,23sleep20-21swapper/207:19:062
3000128412,14sleep00-21swapper/010:04:110
3000128412,14sleep00-21swapper/010:04:110
2672028410,14sleep20-21swapper/208:09:212
2672028410,14sleep20-21swapper/208:09:212
1302428412,36sleep30-21swapper/309:09:233
83682836,36sleep30-21swapper/312:20:213
210682830,7sleep10-21swapper/111:19:131
62772829,21sleep10-21swapper/108:49:061
2113728222,32sleep00-21swapper/007:53:570
1119928119,38sleep10-21swapper/110:48:541
2451028022,38sleep224545-21latency_hist08:03:562
2451028022,38sleep224545-21latency_hist08:03:562
911427917,39sleep09130-21modprobe08:58:540
587627914,52sleep05880-21entropy12:14:120
587627914,52sleep05880-21entropy12:14:120
399227915,51sleep04021-21latency_hist12:08:550
1211427919,29sleep10-21swapper/112:34:121
295927827,27sleep10-21swapper/112:04:161
193127814,23sleep00-21swapper/010:18:570
193127814,23sleep00-21swapper/010:18:570
175027715,42sleep21879-21timerwakeupswit11:59:242
1183327724,28sleep20-21swapper/212:33:562
869227620,37sleep18814-21munin-node07:14:001
245127615,19sleep10-21swapper/112:03:561
1724127626,23sleep10-21swapper/109:24:111
579227524,30sleep10-21swapper/112:14:091
579227524,30sleep10-21swapper/112:14:091
3228627516,33sleep232372-21latency_hist08:28:562
3252927420,26sleep00-21swapper/008:29:100
3205727418,23sleep10-21swapper/110:09:251
2263327428,29sleep022661-21munin-plugin-st07:58:530
1437727419,37sleep214503-21latency_hist10:58:562
11952747,34sleep30-21swapper/308:32:083
714527321,22sleep20-21swapper/212:18:552
714527321,22sleep20-21swapper/212:18:552
2501227314,26sleep00-21swapper/009:48:570
135852739,19sleep00-21swapper/007:29:050
529627215,41sleep05360-21/usr/sbin/munin08:44:240
445927219,38sleep24476-21awk08:43:542
267972729,49sleep00-21swapper/009:54:100
163562728,11sleep00-21swapper/011:04:150
163562728,11sleep00-21swapper/011:04:150
869327120,24sleep20-21swapper/212:23:552
41032716,28sleep30-21swapper/310:24:213
41032716,28sleep30-21swapper/310:24:213
3195627110,50sleep30-21swapper/308:25:173
3062727115,37sleep130702-21fschecks_time11:50:111
2758127118,29sleep10-21swapper/111:39:201
260452716,12sleep20-21swapper/209:49:272
1525427113,43sleep015260-21diskstats07:34:260
1331627119,33sleep113335-21idleruntime-cro07:28:541
46282705,52sleep00-21swapper/010:24:300
46282705,52sleep00-21swapper/010:24:300
1837027014,40sleep218392-21fschecks_count07:44:142
1341427015,38sleep013445-21nfs_client10:54:220
2221126919,23sleep20-21swapper/211:23:552
2166326912,44sleep021706-21cut09:38:550
172722699,12sleep20-21swapper/207:39:252
1066826913,20sleep00-21swapper/012:29:140
360126818,27sleep10-21swapper/108:39:211
2917426818,29sleep20-21swapper/211:44:222
152452688,45sleep30-21swapper/307:34:253
86452677,14sleep00-21swapper/012:23:530
677726716,29sleep20-21swapper/208:49:232
2877226721,29sleep128879-21fschecks_time11:44:131
2496326710,14sleep30-21swapper/308:04:183
2496326710,14sleep30-21swapper/308:04:183
115226722,24sleep10-21swapper/110:14:251
115226722,24sleep10-21swapper/110:14:251
2838426610,42sleep21-21systemd11:43:542
277626617,32sleep22839-21cut12:04:132
259462668,47sleep00-21swapper/008:08:060
259462668,47sleep00-21swapper/008:08:060
2258626616,31sleep10-21swapper/109:39:251
2258626616,31sleep10-21swapper/109:39:251
839326516,27sleep10-21swapper/110:39:091
761626512,34sleep00-21swapper/007:09:550
761626512,34sleep00-21swapper/007:09:550
533226515,34sleep25339-21df_inode10:29:122
36302658,10sleep30-21swapper/312:04:513
2846326513,37sleep028613-21cat11:43:560
2184326516,26sleep20-21swapper/207:54:232
2010326521,23sleep10-21swapper/107:49:171
1210526515,35sleep00-21swapper/010:49:250
3043526421,23sleep00-21swapper/011:49:050
2330226415,22sleep00-21swapper/009:43:560
2330226415,22sleep00-21swapper/009:43:560
2316726417,31sleep223197-21munin-run09:43:542
2316726417,31sleep223197-21munin-run09:43:542
1938826414,33sleep119462-21find09:29:241
1386426417,24sleep00-21swapper/009:13:560
572526314,34sleep15747-21pluginstate10:29:231
191882637,12sleep00-21swapper/009:29:190
1683326319,22sleep00-21swapper/007:39:150
1264326323,18sleep20-21swapper/209:09:142
794426215,31sleep18031-21fschecks_time08:54:151
243392625,46sleep00-21swapper/008:02:210
243392625,46sleep00-21swapper/008:02:210
2019426216,27sleep020211-21apt-get11:14:370
900326113,20sleep20-21swapper/210:39:262
691626118,26sleep27012-21grep10:34:172
483926115,19sleep20-21swapper/208:44:142
35682617,11sleep00-21swapper/008:39:190
2892526121,21sleep20-21swapper/209:59:252
2680726122,21sleep20-21swapper/211:38:532
2571226117,18sleep10-21swapper/109:49:221
1772326116,27sleep217783-21/usr/sbin/munin09:24:242
3223426018,26sleep032270-21munin-run08:28:540
2424226011,34sleep224275-21irqstats11:29:162
1962926016,19sleep10-21swapper/107:48:571
194452606,14sleep00-21swapper/007:47:430
124626010,35sleep21367-21timerwakeupswit10:14:262
124626010,35sleep21367-21timerwakeupswit10:14:262
2419425919,21sleep10-21swapper/111:29:151
2125825916,24sleep00-21swapper/011:19:170
2051925913,28sleep120572-21/usr/sbin/munin09:34:161
2029425915,27sleep220309-21/usr/sbin/munin07:49:222
1699325922,18sleep20-21swapper/209:23:552
1633225921,22sleep00-21swapper/009:19:250
1593325916,26sleep216035-21sed11:03:572
1490725916,20sleep20-21swapper/207:33:562
140332594,20sleep10-21swapper/110:55:281
1315625925,18sleep10-21swapper/109:09:251
1277325919,19sleep20-21swapper/212:34:252
1074925914,29sleep110807-21latency_hist09:03:561
987725813,30sleep09878-21sshd07:15:000
943225818,24sleep09428-21apt-get08:59:060
3068525816,20sleep10-21swapper/108:23:541
1158725814,29sleep111691-21grep10:49:141
3070425716,22sleep20-21swapper/211:50:122
2495725721,21sleep10-21swapper/109:48:541
231225720,22sleep10-21swapper/110:19:181
231225720,22sleep10-21swapper/110:19:181
1769625714,23sleep10-21swapper/111:09:041
1769625714,23sleep10-21swapper/111:09:041
718625616,22sleep27247-21perl07:09:452
718625616,22sleep27247-21perl07:09:452
438225619,22sleep10-21swapper/110:24:271
438225619,22sleep10-21swapper/110:24:271
2021525611,20sleep20-21swapper/211:14:392
1649425610,18sleep20-21swapper/207:38:572
85522555,10sleep10-21swapper/112:22:411
5352558,12sleep10-21swapper/108:29:231
2475225515,24sleep124745-21apt-get08:04:121
2475225515,24sleep124745-21apt-get08:04:121
212972554,11sleep20-21swapper/209:35:062
1205825515,24sleep012053-21apt-get07:24:090
1068425514,18sleep20-21swapper/210:44:272
817325417,21sleep18327-21latency_hist10:38:561
3145325415,25sleep231472-21apt-get10:09:062
2950625416,23sleep229512-21diskstats08:19:112
2940525412,24sleep029423-21apt-get08:19:040
2873725410,25sleep00-21swapper/009:59:180
2847325414,23sleep128596-21ls08:14:251
1682625420,19sleep20-21swapper/211:04:252
1682625420,19sleep20-21swapper/211:04:252
1566825415,20sleep10-21swapper/109:19:061
1410325416,20sleep10-21swapper/107:29:231
839225311,25sleep08386-21apt-get10:39:080
2821125311,27sleep228234-21netstat08:14:182
1203425310,17sleep20-21swapper/207:24:072
87725213,23sleep1895-21munin-run11:58:551
87725213,23sleep1895-21munin-run11:58:551
736625214,16sleep00-21swapper/010:34:270
676125214,20sleep00-21swapper/008:49:210
2293725215,18sleep10-21swapper/111:24:231
2293725215,18sleep10-21swapper/111:24:231
1842325216,16sleep00-21swapper/011:09:400
1842325216,16sleep00-21swapper/011:09:400
1455825213,24sleep014564-21ntp_states09:14:200
1154825215,15sleep10-21swapper/109:04:241
536125117,15sleep10-21swapper/108:44:251
3107625112,21sleep131185-21perl08:24:141
2969025114,23sleep129769-21latency_hist10:03:561
2678425112,24sleep11-21systemd08:09:241
2678425112,24sleep11-21systemd08:09:241
2342325113,14sleep10-21swapper/107:59:231
2342325113,14sleep10-21swapper/107:59:231
1517625111,24sleep115170-21apt-get07:34:231
1163125114,20sleep211752-21sed09:04:252
893125012,21sleep08927-21apt-get12:24:050
78532509,17sleep20-21swapper/208:54:072
3053725010,18sleep10-21swapper/110:04:241
3053725010,18sleep10-21swapper/110:04:241
2398125013,20sleep023999-21apt-get11:29:060
465424911,20sleep24696-21ntp_states12:09:182
2428524910,23sleep224363-21ls09:44:352
1179024912,23sleep10-21swapper/112:33:551
1878524614,16sleep20-21swapper/209:29:082
1673524612,16sleep10-21swapper/107:39:081
939924515,15sleep20-21swapper/208:59:002
1487524512,20sleep114888-21irqstats10:59:151
1072424311,15sleep10-21swapper/107:19:171
915324111,15sleep10-21swapper/112:24:151
257632417,16sleep10-21swapper/111:34:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional