You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 14:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Mar 20, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21671287213,844sleep021677-21df_inode00:13:100
25916272812,697sleep326046-21/usr/sbin/munin19:18:213
15645258912,561sleep015653-21ntpq20:28:180
13716251816,483sleep213718-21sshd22:03:272
13716251816,483sleep213718-21sshd22:03:272
3706237676,283sleep23719-21irqstats21:33:112
20063234620,306sleep320120-21/usr/sbin/munin20:43:113
21277232229,274sleep221344-21/usr/sbin/munin19:07:512
14581232211,264sleep114073-21apt-get22:08:021
791922730,6sleep30-21swapper/323:28:123
7881225489,150sleep17930-21latency_hist21:47:521
9734222751,133sleep19842-21timerandwakeup23:33:201
6692221982,45sleep16691-21pluginstate20:02:521
25046220714,175sleep325167-21sed22:38:203
31130220611,179sleep131253-21latency_hist19:37:531
31130220611,179sleep131253-21latency_hist19:37:531
10631220539,148sleep110734-21timerandwakeup20:13:221
3267220249,36sleep13308-21ntp_kernel_err23:13:131
16687220243,89sleep116763-21switchtime22:13:191
22671220112,167sleep122694-21munin-run20:52:491
23201219934,27sleep10-21swapper/120:53:061
24911219714,24sleep10-21swapper/122:38:181
32338219514,167sleep032342-21diskstats23:03:070
14165219314,64sleep10-21swapper/123:48:131
26527218610,162sleep126596-21switchtime22:43:191
29521218313,140sleep10-21swapper/121:13:081
2900217816,146sleep02893-21apt-get23:13:040
2651217616,143sleep12786-21grep21:28:341
23261217645,76sleep10-21swapper/100:18:081
3233021707,18sleep30-21swapper/323:03:063
10704217014,25sleep10-21swapper/123:38:041
5142216816,137sleep25268-21cat23:18:232
5142216816,137sleep25268-21cat23:18:232
24118216820,130sleep124133-21modprobe22:37:501
16220216812,142sleep216225-21entropy22:13:082
16737216710,19sleep10-21swapper/123:57:541
26038216620,18sleep10-21swapper/100:27:531
19599216613,137sleep319664-21fschecks_time22:23:103
1994021648,16sleep10-21swapper/120:43:091
23647216019,114sleep323797-21grep22:34:003
21333215913,133sleep021432-21cut22:28:120
21333215913,133sleep021432-21cut22:28:120
26464215612,123sleep00-21swapper/022:43:180
23961215411,126sleep124006-21ntpq19:13:151
1854821509,102sleep117933-21/usr/sbin/munin22:18:241
1854821509,102sleep117933-21/usr/sbin/munin22:18:241
17218214915,39sleep10-21swapper/120:33:201
454214317,16sleep10-21swapper/119:43:061
2116421398,89sleep121034-21/usr/sbin/munin22:28:091
2116421398,89sleep121034-21/usr/sbin/munin22:28:091
5080213312,28sleep35275-21apt-config19:57:533
27980213313,48sleep128053-21grep21:08:121
1438213214,100sleep11520-21cut23:08:091
6335213021,96sleep26450-21sed21:42:522
15106213022,88sleep315157-21latency_hist23:52:513
27005212992,20sleep10-21swapper/100:28:211
2220421278,40sleep10-21swapper/100:13:241
18309212720,75sleep218303-21apt-get20:38:052
28340212635,68sleep10-21swapper/119:28:081
20123212612,25sleep020154-21kernelversion20:43:120
24847212283,18sleep10-21swapper/120:58:061
820321198,78sleep10-21swapper/121:48:081
8186211613,81sleep28288-21/usr/sbin/munin23:28:202
2383821120,8sleep30-21swapper/319:13:123
18769211213,83sleep318773-21ntpq20:38:183
14200211215,80sleep314292-21apt-config22:07:523
14200211215,80sleep314292-21apt-config22:07:523
9667211010,20sleep10-21swapper/121:53:051
8924210913,46sleep10-21swapper/120:08:171
8924210913,46sleep10-21swapper/120:08:171
6215210815,47sleep16009-21/usr/sbin/munin23:23:101
561210612,16sleep10-21swapper/121:23:171
24135210619,67sleep224214-21sort19:13:202
27995210520,25sleep30-21swapper/322:48:173
12894210410,46sleep113001-21fschecks_time22:03:101
12894210410,46sleep113001-21fschecks_time22:03:101
6034210314,76sleep06045-21grep23:23:050
5061210216,69sleep10-21swapper/123:18:221
5061210216,69sleep10-21swapper/123:18:221
303062998,40sleep10-21swapper/122:57:271
1228129913,54sleep112317-21ntpq20:19:031
3244029823,41sleep10-21swapper/119:40:271
1713429812,66sleep017179-21/usr/sbin/munin20:33:150
2879529720,41sleep10-21swapper/100:34:101
2232829728,42sleep222358-21munin-run22:32:502
2232829728,42sleep222358-21munin-run22:32:502
2230629716,63sleep222453-21grep00:13:252
2207229744,28sleep30-21swapper/300:13:223
2196729713,70sleep022044-21switchtime20:48:200
1545329713,68sleep315469-21kernelversion20:28:133
383329618,40sleep10-21swapper/121:33:141
3267629612,23sleep00-21swapper/019:42:530
2481329616,58sleep324853-21fschecks_count00:23:103
690229522,41sleep10-21swapper/120:03:001
3110729517,58sleep230997-21/usr/sbin/munin21:18:102
3110729517,58sleep230997-21/usr/sbin/munin21:18:102
2865429518,44sleep10-21swapper/122:51:051
2640629514,42sleep10-21swapper/119:22:511
1953829522,41sleep10-21swapper/120:42:281
3233029432,49sleep132449-21latency_hist21:22:531
3233029432,49sleep132449-21latency_hist21:22:531
3036629417,51sleep329768-21/usr/sbin/munin19:33:193
3036629417,51sleep329768-21/usr/sbin/munin19:33:193
2707529418,39sleep10-21swapper/119:23:161
2005929418,61sleep10-21swapper/122:23:201
479029319,40sleep10-21swapper/119:57:311
412629317,57sleep24201-21grep19:53:202
3223229314,41sleep10-21swapper/123:03:001
1807129319,40sleep10-21swapper/100:01:131
1360929310,41sleep10-21swapper/120:23:021
1329429318,41sleep10-21swapper/123:45:061
855929219,40sleep10-21swapper/123:29:161
3189029218,39sleep10-21swapper/123:01:471
232732928,16sleep00-21swapper/020:53:090
215992927,36sleep10-21swapper/119:08:021
2134529215,59sleep321406-21latency_hist19:07:523
184862927,35sleep10-21swapper/100:03:021
1218629219,39sleep10-21swapper/121:58:461
3148329110,65sleep331503-21/usr/sbin/munin21:18:193
3148329110,65sleep331503-21/usr/sbin/munin21:18:193
278129115,58sleep32903-21users19:48:213
2758529118,40sleep10-21swapper/121:07:471
2096129116,42sleep10-21swapper/100:08:311
1867629110,67sleep018737-21if_eth000:03:090
984328921,45sleep29982-21grep23:33:212
276392897,68sleep027782-21latency_hist00:32:520
273102890,7sleep30-21swapper/319:23:213
1820028922,43sleep018226-21latency_hist00:02:490
1498128914,31sleep30-21swapper/320:27:533
1518928811,45sleep10-21swapper/120:28:071
2292828615,46sleep023107-21latency_hist19:12:530
50912846,44sleep10-21swapper/121:38:071
257728416,17sleep10-21swapper/119:48:191
218828417,48sleep02246-21/usr/sbin/munin19:48:090
184942848,56sleep00-21swapper/020:38:110
282102830,8sleep20-21swapper/222:48:192
86752820,5sleep00-21swapper/021:48:180
3028928215,32sleep20-21swapper/219:33:182
3028928215,32sleep20-21swapper/219:33:182
2038228228,27sleep10-21swapper/119:03:141
3243928121,34sleep20-21swapper/223:03:092
195628118,42sleep02040-21wc23:08:200
2672928019,33sleep326851-21head22:43:213
1857927925,27sleep30-21swapper/300:03:083
363827824,37sleep33650-21/usr/sbin/munin19:53:093
1040927815,42sleep310496-21unixbench_multi21:53:233
823927719,34sleep00-21swapper/020:07:510
794927711,51sleep07993-21ntpq23:28:140
657227717,35sleep26566-21apt-get21:43:042
481427716,37sleep04871-21sendmail-msp19:57:500
1419327714,42sleep214227-21/usr/sbin/munin23:48:142
776227615,44sleep27828-21users20:03:212
345127613,32sleep00-21swapper/021:33:050
2969027621,28sleep20-21swapper/222:53:212
2950227624,35sleep329513-21modprobe19:32:503
244372768,49sleep00-21swapper/022:38:050
1092527627,25sleep20-21swapper/221:57:502
53827522,32sleep00-21swapper/019:43:090
2641427518,35sleep326444-21fschecks_count21:03:093
1662127522,30sleep00-21swapper/022:13:180
1234327518,33sleep30-21swapper/323:43:103
1114627510,52sleep011157-21ntp_states23:38:150
2659827414,30sleep20-21swapper/222:43:202
2598527415,38sleep226025-21modprobe00:27:492
187582747,13sleep20-21swapper/200:03:112
1740527419,36sleep317407-21cron22:17:493
669327320,30sleep30-21swapper/321:43:103
539327311,49sleep05422-21nfs4_client21:38:140
3116327321,27sleep20-21swapper/222:58:172
3090927317,32sleep30-21swapper/322:58:103
1535727313,35sleep015507-21sed22:09:060
1256227315,17sleep00-21swapper/022:02:530
113082739,14sleep20-21swapper/221:58:062
424627216,42sleep34309-21latency_hist23:17:523
2724127221,18sleep20-21swapper/219:23:212
2704027216,37sleep027188-21unixbench_singl21:03:220
259442727,12sleep00-21swapper/000:27:190
2675027016,35sleep226811-21grep00:28:182
1649627020,27sleep20-21swapper/220:32:522
410026917,32sleep34228-21ls21:33:213
2453726915,24sleep30-21swapper/320:57:523
2075726918,32sleep220012-21/usr/sbin/munin00:08:202
949526818,26sleep30-21swapper/323:33:133
2443926820,22sleep30-21swapper/300:22:503
1422126821,35sleep014290-21swap20:23:500
3038626720,29sleep330407-21apt22:57:493
2214526710,40sleep222213-21/usr/sbin/munin19:08:192
2040426714,39sleep320443-21ntpq00:08:133
607026618,24sleep20-21swapper/219:58:372
3275026620,28sleep3318-21/usr/sbin/munin21:23:113
181826614,37sleep21883-21processes23:08:182
1793226616,27sleep217430-21apt-get22:18:012
1793226616,27sleep217430-21apt-get22:18:012
2920026519,25sleep20-21swapper/221:12:532
2681326511,12sleep30-21swapper/300:28:193
265726516,33sleep22780-21timerwakeupswit19:48:202
2330226518,20sleep00-21swapper/019:12:570
1777626423,17sleep30-21swapper/322:17:533
1777626423,17sleep30-21swapper/322:17:533
1189726423,30sleep30-21swapper/320:18:403
118172649,21sleep00-21swapper/020:18:010
894426317,17sleep30-21swapper/320:08:193
894426317,17sleep30-21swapper/320:08:193
2843726315,21sleep00-21swapper/000:33:200
2545626317,24sleep00-21swapper/020:58:220
1110126313,26sleep30-21swapper/323:38:143
1073626319,22sleep20-21swapper/220:13:232
320726216,29sleep20-21swapper/221:32:512
2989726223,22sleep20-21swapper/221:13:172
2077126221,19sleep20-21swapper/222:27:492
1589426216,20sleep00-21swapper/023:53:210
302952618,41sleep00-21swapper/022:57:180
2670026119,18sleep00-21swapper/019:23:060
1505226112,29sleep215100-21/usr/sbin/munin22:09:002
1429426114,32sleep014073-21apt-get22:07:530
1429426114,32sleep014073-21apt-get22:07:530
1017026116,25sleep30-21swapper/320:13:113
2504526011,14sleep20-21swapper/220:58:122
1807226022,19sleep20-21swapper/220:37:532
1608726017,21sleep20-21swapper/223:53:232
966225911,19sleep00-21swapper/023:33:190
3181125920,17sleep30-21swapper/319:38:153
2269925919,20sleep20-21swapper/220:52:502
2236325912,27sleep30-21swapper/322:32:513
2236325912,27sleep30-21swapper/322:32:513
1999625913,27sleep020058-21/usr/sbin/munin22:23:190
1273625912,28sleep012822-21cut23:43:200
534425818,21sleep00-21swapper/019:58:030
3256725817,26sleep032562-21apt-get21:23:040
3109225811,18sleep00-21swapper/021:18:080
3109225811,18sleep00-21swapper/021:18:080
2951625814,27sleep029553-21munin-run19:32:510
106925817,20sleep30-21swapper/319:43:203
2442625711,32sleep224427-21sshd20:57:272
1252325713,26sleep20-21swapper/223:43:132
915425614,24sleep29280-21unixbench_multi20:08:222
915425614,24sleep29280-21unixbench_multi20:08:222
701625611,32sleep213441-21snmpd23:24:202
458525612,18sleep30-21swapper/323:18:123
458525612,18sleep30-21swapper/323:18:123
2907025613,24sleep20-21swapper/219:28:212
2789625614,26sleep327905-21/usr/sbin/munin00:33:073
2782025613,21sleep20-21swapper/200:33:022
2542025614,22sleep225513-21memory19:18:122
2274125613,27sleep322881-21date20:52:513
168922569,20sleep30-21swapper/320:33:103
603225513,25sleep36027-21apt-get23:23:043
30192556,12sleep00-21swapper/021:30:590
287082556,37sleep00-21swapper/022:51:450
2336125514,22sleep223367-21ntp_states22:33:552
2003525513,26sleep020030-21apt-get00:08:020
1980125518,19sleep00-21swapper/019:02:530
1846825518,22sleep00-21swapper/022:18:230
1846825518,22sleep00-21swapper/022:18:230
1100625514,19sleep00-21swapper/021:57:520
1068625516,22sleep210698-21apt-get23:38:012
743325414,20sleep30-21swapper/320:03:183
454025415,20sleep00-21swapper/023:18:090
454025415,20sleep00-21swapper/023:18:090
233725413,25sleep32374-21ntp_states21:28:283
1148725416,22sleep011311-21latency_hist20:17:530
93752538,36sleep30-21swapper/321:52:463
847725313,23sleep08473-21apt-get20:08:040
847725313,23sleep08473-21apt-get20:08:040
502625314,23sleep35021-21apt-get21:38:053
3070225315,25sleep030697-21apt-get22:58:030
173325314,24sleep2281-21dbus-daemon19:47:512
1368925315,24sleep213711-21df20:23:082
1178725314,23sleep30-21swapper/321:58:203
3006425210,15sleep00-21swapper/019:33:120
3006425210,15sleep00-21swapper/019:33:120
2477425215,19sleep30-21swapper/320:58:043
2440825213,20sleep00-21swapper/000:22:270
2338425218,17sleep00-21swapper/022:33:570
170252525,11sleep20-21swapper/223:58:112
1445725214,17sleep00-21swapper/023:48:200
343125112,25sleep23500-21/usr/sbin/munin23:13:182
2943425113,20sleep00-21swapper/021:13:030
188425113,22sleep31954-21sensors_temp23:08:193
2765425015,21sleep227755-21latency_hist21:07:522
1473325015,19sleep30-21swapper/322:08:533
92222496,9sleep20-21swapper/221:50:482
3176524911,22sleep231809-21ntpq19:38:142
2563324912,14sleep00-21swapper/019:18:180
3265424811,15sleep20-21swapper/221:23:102
2796424812,21sleep327973-21df_inode21:08:093
2291124811,22sleep322925-21munin-node19:12:523
2983324717,15sleep30-21swapper/321:13:143
2816424713,17sleep00-21swapper/021:08:150
2573524711,21sleep225744-21/usr/sbin/munin22:42:512
1535024712,20sleep315420-21df23:53:073
742724611,14sleep00-21swapper/021:43:230
1185124611,19sleep211836-21apt-get20:18:072
2385924310,15sleep20-21swapper/200:18:212
2022223811,14sleep20-21swapper/220:43:142
169332387,16sleep30-21swapper/323:58:043
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional