You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 08:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Jan 02, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19778234568,3435sleep031503-21kworker/0:122:05:300
24428289710,872sleep130351-1kworker/1:0H20:39:081
1634263012,601sleep11679-21/usr/sbin/munin21:09:201
1634263012,601sleep11679-21/usr/sbin/munin21:09:201
2365924737,456sleep3369-21snmpd20:37:043
31458231639,261sleep231478-21users20:59:282
4353227581,176sleep33865-21/usr/sbin/munin22:59:183
25126224521,203sleep325267-21unixbench_singl20:39:553
25994224478,60sleep30-21swapper/320:44:113
20652224317,214sleep116920-21kworker/u8:022:09:111
2256222253,85sleep30-21swapper/300:38:533
2998022219,196sleep029876-21/usr/sbin/munin22:39:090
2998022219,196sleep029876-21/usr/sbin/munin22:39:090
10619222089,57sleep310101-21/usr/sbin/munin19:54:213
31877221913,191sleep131922-21fschecks_time22:44:471
9257221855,87sleep39393-21unixbench_singl23:14:243
18001221213,186sleep218060-21fschecks_count20:19:112
25313220346,88sleep325338-21munin-plugin-st00:08:503
25313220346,88sleep325338-21munin-plugin-st00:08:503
30331220013,173sleep330440-21sed20:58:523
31884219416,158sleep332011-21latency_hist21:03:533
31591219316,162sleep331643-21latency_hist00:28:523
10315218710,40sleep30-21swapper/323:19:133
23772217511,150sleep323789-21fschecks_count22:19:103
17235216782,61sleep30-21swapper/321:59:073
17235216782,61sleep30-21swapper/321:59:073
14552216712,139sleep114515-21latency_hist20:08:531
20760216410,14sleep30-21swapper/320:28:533
915216170,32sleep3928-21fschecks_count19:24:103
2803216025,72sleep32297-21/usr/sbin/munin22:54:223
18283216016,132sleep318315-21awk20:19:163
93321507,121sleep00-21swapper/019:24:110
15986214916,115sleep316006-21kworker/u8:023:38:503
3275214610,123sleep03336-21cat21:15:160
25607213814,110sleep125649-21sed22:24:171
17819213714,100sleep20-21swapper/223:44:022
1165121349,110sleep311718-21sort23:24:093
31920213117,94sleep231972-21fschecks_count00:29:112
27045212911,45sleep30-21swapper/322:29:153
10455212811,97sleep30-21swapper/321:35:303
10455212811,97sleep30-21swapper/321:35:303
383221278,60sleep33977-21unixbench_singl21:15:283
834221269,102sleep38414-21latency_hist19:48:523
25770212616,94sleep225836-21swap22:24:222
15713212612,99sleep015735-21df21:54:080
15713212612,99sleep015735-21df21:54:080
14750212112,88sleep30-21swapper/321:49:233
26482211922,79sleep326514-21idleruntime-cro22:28:513
31504211712,48sleep331560-21latency_hist19:18:533
6127211624,64sleep26283-21grep23:04:262
17233211559,40sleep217226-21apt-get21:59:062
17233211559,40sleep217226-21apt-get21:59:062
26730211340,49sleep326744-21cut19:04:123
19166211219,77sleep119180-21latency_hist20:23:501
3174221109,85sleep30-21swapper/319:19:053
22269211031,67sleep322273-21munin-run23:58:533
27234210927,61sleep127249-21cut00:14:091
22594210912,80sleep30-21swapper/322:14:213
22594210912,80sleep30-21swapper/322:14:213
4766210829,67sleep34823-21basename21:19:193
21870210819,74sleep321984-21latency_hist22:13:533
9901210675,15sleep30-21swapper/319:53:533
7444210613,20sleep30-21swapper/321:28:523
6220210615,71sleep26232-21/usr/sbin/munin21:24:102
32295210612,14sleep30-21swapper/300:29:203
2302210669,18sleep30-21swapper/321:11:323
2302210669,18sleep30-21swapper/321:11:323
1728021065,61sleep30-21swapper/320:14:523
3922210523,63sleep30-21swapper/319:34:073
30122210512,76sleep130223-21cut00:23:531
30122210512,76sleep130223-21cut00:23:531
18997210528,59sleep30-21swapper/322:04:163
12027210311,18sleep00-21swapper/019:59:150
6287210216,18sleep30-21swapper/323:04:273
2643121028,75sleep130351-1kworker/1:0H20:44:211
21003210124,14sleep30-21swapper/320:29:093
30364210015,66sleep230379-21/usr/sbin/munin22:39:192
30364210015,66sleep230379-21/usr/sbin/munin22:39:192
17556210013,73sleep317598-21true23:43:513
11696210023,14sleep30-21swapper/319:59:063
2868229920,14sleep30-21swapper/319:09:203
2721829944,39sleep30-21swapper/300:14:063
1285229920,15sleep30-21swapper/321:44:183
311202985,59sleep30-21swapper/322:43:243
311202985,59sleep30-21swapper/322:43:243
3052229811,71sleep130531-21/usr/sbin/munin19:14:151
3267329732,46sleep30-21swapper/321:04:263
1609329712,66sleep011385-21kworker/u8:220:13:520
2795729652,21sleep30-21swapper/320:49:203
2308529661,21sleep323170-21switchtime23:59:213
1079329614,66sleep210857-21latency_hist21:38:522
1079329614,66sleep210857-21latency_hist21:38:522
3024429514,62sleep030134-21/usr/sbin/munin19:14:100
148782957,68sleep00-21swapper/021:49:240
336329416,53sleep22977-21/usr/sbin/munin21:15:182
1852029420,12sleep30-21swapper/323:44:243
1380429413,60sleep30-21swapper/323:29:223
8752928,68sleep3965-21perl22:49:173
2291129221,50sleep022521-21/usr/sbin/munin20:34:190
1566229211,17sleep312-21rcu_sched21:54:073
1566229211,17sleep312-21rcu_sched21:54:073
3165129112,60sleep30-21swapper/322:44:003
2859629114,55sleep028622-21netstat19:09:170
2691029127,41sleep226932-21kernelversion19:04:152
1319129010,64sleep013254-21cpuspeed23:29:070
1019329015,22sleep10-21swapper/123:19:111
753828919,48sleep07725-21latency_hist21:28:530
297192895,49sleep30-21swapper/300:20:163
297192895,49sleep30-21swapper/300:20:163
2587828911,44sleep325951-21latency00:09:143
2587828911,44sleep325951-21latency00:09:143
166928913,56sleep01786-21unixbench_singl00:34:260
1243928920,46sleep112589-21grep19:59:241
2850328816,48sleep00-21swapper/000:18:520
2704428821,30sleep10-21swapper/119:04:181
1623528818,55sleep216307-21switchtime21:54:222
1623528818,55sleep216307-21switchtime21:54:222
430328715,59sleep14350-21tr19:34:171
36292879,14sleep30-21swapper/319:33:513
198012876,15sleep30-21swapper/323:49:163
77702867,14sleep30-21swapper/323:09:243
62482866,21sleep20-21swapper/219:39:232
1140628512,57sleep211450-21date23:23:522
121012840,6sleep20-21swapper/219:59:162
1137828433,30sleep00-21swapper/019:58:500
89292836,23sleep30-21swapper/321:30:413
89292836,23sleep30-21swapper/321:30:413
2947728315,57sleep30-21swapper/322:37:313
2273028311,58sleep122789-21if_err_eth020:34:161
713228216,51sleep17133-21sshd21:25:311
2808028220,34sleep00-21swapper/022:33:520
2442828217,49sleep024439-21/usr/sbin/munin00:04:150
2442828217,49sleep024439-21/usr/sbin/munin00:04:150
2001028215,52sleep220072-21switchtime20:24:242
1071428216,44sleep010717-21cron21:38:500
1071428216,44sleep010717-21cron21:38:500
33372810,5sleep10-21swapper/121:15:161
2947828120,43sleep129480-21cron19:13:501
2753128116,50sleep127591-21fschecks_time20:49:101
2230928125,28sleep10-21swapper/123:58:551
1150028113,55sleep211552-21processes21:39:202
873928014,53sleep08795-21awk23:14:130
3103727914,43sleep130551-21/usr/sbin/munin20:59:231
3083327919,28sleep00-21swapper/000:24:210
261027916,38sleep22806-21gpgv21:13:532
261027916,38sleep22806-21gpgv21:13:532
1709327912,30sleep10-21swapper/120:14:261
1467327917,49sleep214749-21sed21:49:222
2842227818,40sleep228453-21grep00:18:502
2405427817,24sleep00-21swapper/022:19:150
2167427813,47sleep021811-21sed23:54:260
1764027826,20sleep10-21swapper/123:43:531
1409227814,18sleep20-21swapper/220:04:262
2958027715,45sleep229643-21modprobe22:38:512
2953127722,31sleep00-21swapper/020:54:210
217982776,11sleep00-21swapper/022:13:340
84172769,12sleep00-21swapper/019:48:530
195842760,7sleep20-21swapper/223:49:102
99752745,12sleep00-21swapper/023:18:530
1266327416,31sleep20-21swapper/221:44:132
122162747,56sleep30-21swapper/321:42:573
858327322,28sleep20-21swapper/219:49:082
2690327322,22sleep00-21swapper/000:13:510
2690327322,22sleep00-21swapper/000:13:510
63892726,51sleep30-21swapper/319:39:243
544427218,29sleep00-21swapper/023:04:080
3115527219,31sleep10-21swapper/122:43:501
3115527219,31sleep10-21swapper/122:43:501
2608127214,36sleep125581-21/usr/sbin/munin00:09:201
2608127214,36sleep125581-21/usr/sbin/munin00:09:201
1163527217,34sleep011709-21threads21:39:220
86627113,14sleep20-21swapper/222:49:162
788927116,23sleep00-21swapper/021:29:010
788927116,23sleep00-21swapper/021:29:010
53127121,26sleep20-21swapper/222:48:522
2870727118,30sleep20-21swapper/220:53:512
2317427113,39sleep023290-21ls23:59:220
1812627115,22sleep00-21swapper/020:19:130
1802927115,25sleep00-21swapper/023:44:120
2020027024,20sleep00-21swapper/020:24:260
3227626915,22sleep20-21swapper/219:19:202
3149026919,28sleep20-21swapper/219:18:522
160012695,53sleep30-21swapper/320:13:193
463426832,18sleep00-21swapper/019:34:240
2711126823,26sleep00-21swapper/019:04:210
2389426818,25sleep10-21swapper/122:19:121
141512685,51sleep30-21swapper/320:04:273
996926717,25sleep10-21swapper/121:34:231
996926717,25sleep10-21swapper/121:34:231
978826719,21sleep10-21swapper/123:18:501
679926721,21sleep00-21swapper/019:43:530
3154226720,25sleep10-21swapper/100:28:501
2166926719,25sleep00-21swapper/020:29:260
1542326716,34sleep115556-21latency_hist21:53:531
520426616,20sleep10-21swapper/123:03:531
460126611,12sleep20-21swapper/221:19:142
459426614,36sleep14584-21apt-get21:19:091
3532666,19sleep10-21swapper/100:30:191
3190526624,24sleep10-21swapper/119:19:111
3025126620,29sleep130314-21modprobe20:58:501
2805626611,39sleep028130-21threads20:49:210
1676726620,25sleep00-21swapper/023:39:200
1192226615,22sleep20-21swapper/223:24:162
5842655,19sleep00-21swapper/000:33:170
482626510,14sleep00-21swapper/021:19:200
3183326520,21sleep031903-21grep19:19:100
2899826518,22sleep20-21swapper/220:54:072
240726531,16sleep10-21swapper/122:54:121
1012226516,26sleep20-21swapper/219:54:042
316642645,12sleep00-21swapper/021:01:360
2869726412,38sleep128727-21awk22:34:181
2377326415,32sleep11-21systemd00:03:511
850226311,34sleep28521-21apt-get23:14:032
2831326323,16sleep20-21swapper/219:09:112
26382639,24sleep10-21swapper/119:29:121
2274826313,24sleep10-21swapper/122:14:231
2274826313,24sleep10-21swapper/122:14:231
2140126312,31sleep20-21swapper/220:29:202
1835126319,21sleep10-21swapper/123:44:221
16012636,11sleep00-21swapper/021:09:190
16012636,11sleep00-21swapper/021:09:190
98726218,17sleep20-21swapper/221:08:532
91026220,27sleep2914-21diskstats19:24:092
612526216,31sleep10-21swapper/119:39:221
2267326216,31sleep022747-21sed22:14:220
2267326216,31sleep022747-21sed22:14:220
1983626224,21sleep10-21swapper/120:24:191
1449126219,23sleep10-21swapper/123:33:521
130126222,19sleep10-21swapper/119:24:201
743826115,31sleep127124-21ntpd19:44:181
132626116,22sleep10-21swapper/122:49:271
8122606,13sleep00-21swapper/021:07:250
3249526017,22sleep10-21swapper/121:04:191
244602606,11sleep20-21swapper/220:39:262
1521126014,30sleep115271-21grep23:34:201
82032595,20sleep00-21swapper/023:13:240
2499125919,23sleep225149-21date22:23:532
744025813,24sleep27446-21ntp_states23:09:192
3200525810,32sleep032099-21memory22:44:500
2137225813,25sleep10-21swapper/120:29:191
1893325811,28sleep20-21swapper/222:04:152
1519525813,28sleep015210-21/usr/sbin/munin23:34:190
1337325814,25sleep113481-21users21:44:281
1325625817,19sleep00-21swapper/021:44:270
1750225719,18sleep10-21swapper/121:59:141
1750225719,18sleep10-21swapper/121:59:141
81092565,42sleep20-21swapper/219:46:292
782425614,19sleep00-21swapper/019:44:250
544525616,22sleep15513-21df23:04:091
3065025612,27sleep230658-21/usr/sbin/munin00:24:152
202992566,13sleep00-21swapper/023:49:250
1763325619,19sleep00-21swapper/021:59:170
1763325619,19sleep00-21swapper/021:59:170
1699425616,23sleep117134-21latency_hist21:58:531
1699425616,23sleep117134-21latency_hist21:58:531
1697125611,25sleep00-21swapper/020:14:250
141325614,18sleep10-21swapper/100:34:231
126625615,24sleep00-21swapper/022:49:260
2725725513,17sleep10-21swapper/122:29:211
2589725513,26sleep225891-21apt-get20:44:062
2247825513,24sleep20-21swapper/222:14:162
2247825513,24sleep20-21swapper/222:14:162
2003525514,25sleep120057-21munin-run22:08:501
1374125518,20sleep20-21swapper/223:29:212
1103425514,24sleep111027-21apt-get21:39:071
2432025414,23sleep023819-21apt-get20:38:590
2413925412,26sleep224145-21diskstats00:04:082
2413925412,26sleep224145-21diskstats00:04:082
2254525414,23sleep222539-21apt-get20:34:092
1213425414,23sleep012196-21/usr/sbin/munin23:24:220
2685125312,26sleep226857-21diskstats22:29:102
1938825314,22sleep119455-21df23:49:061
1682125311,29sleep216891-21tr20:14:232
1627825315,21sleep216297-21apt-get23:39:022
1479125313,23sleep214784-21apt-get20:09:062
852925212,26sleep28620-21sed21:29:482
852925212,26sleep28620-21sed21:29:482
235025216,21sleep22370-21apt-get19:29:012
2148525215,23sleep221548-21switchtime23:54:242
706725112,22sleep17071-21diskstats23:09:091
304225114,18sleep20-21swapper/222:54:252
283302509,16sleep00-21swapper/022:34:080
2523125012,23sleep025276-21/usr/sbin/munin22:24:070
1333425013,24sleep013340-21cat20:04:110
792024912,21sleep10-21swapper/121:29:061
792024912,21sleep10-21swapper/121:29:061
392124910,23sleep23917-21apt-get19:34:062
2405924912,21sleep124056-21apt-get00:04:031
2405924912,21sleep124056-21apt-get00:04:031
1572824911,24sleep215730-21sshd23:35:302
148712498,20sleep10-21swapper/120:09:091
135724912,19sleep20-21swapper/200:34:202
1164924912,21sleep111644-21apt-get23:24:071
1058024910,16sleep10-21swapper/119:54:191
341124810,24sleep03413-21sshd22:56:450
2300624812,16sleep20-21swapper/223:59:202
2063524811,13sleep20-21swapper/222:09:082
474724711,21sleep04794-21users22:59:240
44992479,22sleep23865-21/usr/sbin/munin22:59:212
2929624715,18sleep229365-21expr00:19:222
2929624715,18sleep229365-21expr00:19:222
267462478,17sleep00-21swapper/020:44:240
86662458,12sleep10-21swapper/119:49:111
258492448,14sleep20-21swapper/200:09:132
258492448,14sleep20-21swapper/200:09:132
237524411,16sleep00-21swapper/019:29:030
288672385,12sleep00-21swapper/000:19:120
288672385,12sleep00-21swapper/000:19:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional