You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 10:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed Jul 08, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19001284118,807sleep019046-21/usr/sbin/munin21:59:330
28287266313,632sleep128289-21sshd20:46:271
18017245811,433sleep118055-21ntpq20:14:331
6982243357,355sleep37013-21fschecks_count23:04:273
14054239722,252sleep314144-21fschecks_time21:44:303
14054239722,252sleep314144-21fschecks_time21:44:303
28046232114,293sleep128056-21ntpq19:04:351
2085023206,300sleep320893-21unin-run23:49:083
19370229777,198sleep319519-21unixbench_singl21:59:403
23342225745,196sleep323466-21timerwakeupswit23:54:383
31601220856,131sleep331634-21modprobe20:59:083
2570219818,156sleep30-21swapper/321:09:093
14150219313,164sleep014207-21if_eth021:44:310
14150219313,164sleep014207-21if_eth021:44:310
90821918,168sleep31023-21sed00:29:093
12836218613,160sleep312727-21/usr/sbin/munin19:59:253
12836218613,160sleep312727-21/usr/sbin/munin19:59:253
30588218412,155sleep330682-21cut00:19:243
30588218412,155sleep330682-21cut00:19:243
12836218415,155sleep1364-21snmpd23:24:071
28492217632,126sleep328495-21cron20:49:073
9995217559,43sleep30-21swapper/323:14:243
24640217511,90sleep30-21swapper/323:59:343
3529217325,70sleep33660-21latency_hist22:54:103
16304217117,135sleep316384-21fschecks_time23:34:273
252902167102,22sleep325446-21gpgv20:39:103
15402216718,128sleep115551-21grep23:29:361
13730216213,133sleep013816-21ls23:24:390
23936215628,110sleep123999-40mandb23:59:081
17023215614,53sleep30-21swapper/321:54:193
1117621569,132sleep00-21swapper/019:54:220
23367215511,124sleep323459-21fschecks_time22:14:303
23367215511,124sleep323459-21fschecks_time22:14:303
1203121558,106sleep312040-21ntpq23:19:333
3212521529,77sleep332168-21if_eth020:59:283
5342214819,106sleep30-21swapper/322:59:233
21615214873,25sleep321678-21mailstats23:49:373
26405214624,98sleep026525-21grep20:39:480
26405214624,98sleep026525-21grep20:39:480
30290214514,114sleep330354-21mailstats22:35:133
18862214413,55sleep318961-21grep23:40:533
15507214182,37sleep30-21swapper/321:49:263
3499214020,99sleep30-21swapper/321:09:423
856621389,21sleep30-21swapper/323:09:283
1622213213,61sleep30-21swapper/319:24:103
18004212730,72sleep10-21swapper/123:39:171
29157212318,10sleep30-21swapper/319:09:173
3952212224,79sleep04001-21basename00:39:070
28309212136,62sleep228469-21grep19:04:402
20856212120,73sleep30-21swapper/320:24:253
13529212113,39sleep30-21swapper/323:24:343
18212211784,15sleep30-21swapper/320:14:403
14215211620,71sleep014415-21latency_hist20:04:110
14215211620,71sleep014415-21latency_hist20:04:110
22296211515,10sleep30-21swapper/320:29:173
25053211413,82sleep025167-21users23:59:430
22097211414,83sleep122128-21/usr/sbin/munin22:09:311
22097211414,83sleep122128-21/usr/sbin/munin22:09:311
3253021128,36sleep30-21swapper/319:19:093
3253021128,36sleep30-21swapper/319:19:093
16762211218,79sleep01-21systemd21:54:080
27692211134,49sleep327784-21fschecks_time19:04:293
25970211011,82sleep326038-21latency20:39:393
25970211011,82sleep326038-21latency20:39:393
1214621095,11sleep30-21swapper/319:55:263
31508210817,78sleep131545-21ntp_kernel_err19:14:351
31508210817,78sleep131545-21ntp_kernel_err19:14:351
10396210818,11sleep30-21swapper/321:32:153
31765210622,63sleep131784-21idleruntime-cro00:24:071
31765210622,63sleep131784-21idleruntime-cro00:24:071
341210523,62sleep132619-21latency_hist22:44:111
534121047,20sleep20-21swapper/222:59:222
4077210411,75sleep30-21swapper/319:29:433
1511210414,77sleep01522-21/usr/sbin/munin00:29:310
7208210314,75sleep07254-21apt-config21:24:080
28325210361,23sleep30-21swapper/322:29:313
26538210310,76sleep326685-21head00:04:393
2304021039,78sleep323171-21latency_hist22:14:093
2304021039,78sleep323171-21latency_hist22:14:093
1972210319,24sleep20-21swapper/222:49:082
1283210314,68sleep0640-21/usr/sbin/munin22:44:420
8756210256,16sleep30-21swapper/319:44:403
32028210217,71sleep132036-21acpi00:24:191
18013210222,61sleep217517-21apt-get23:39:182
9131210115,70sleep19292-21awk23:09:421
25481210116,28sleep00-21swapper/022:19:390
25481210116,28sleep00-21swapper/022:19:390
5050210012,72sleep35189-21grep21:14:433
331721009,75sleep33439-21cat00:34:383
77902998,78sleep37818-21munin-run19:44:083
417029917,63sleep24212-21ntp_offset22:54:332
1126629925,53sleep211329-21cut19:54:282
3005029823,58sleep30-21swapper/320:54:083
249529811,72sleep22196-21/usr/sbin/munin22:49:332
1310329814,64sleep113256-21/usr/sbin/munin21:39:401
3035829714,69sleep030422-21switchtime22:35:140
2736529710,69sleep30-21swapper/300:09:233
2358229721,56sleep123604-21munin-plugin-st20:34:071
229129712,59sleep00-21swapper/022:49:270
3055629623,51sleep230672-21grep22:35:162
2878129613,18sleep10-21swapper/120:49:211
277329630,54sleep30-21swapper/319:24:423
2017829617,62sleep30-21swapper/322:04:203
2017829617,62sleep30-21swapper/322:04:203
18022969,73sleep31929-21cat00:29:383
1267829617,65sleep012734-21latency21:39:300
65029518,61sleep30-21swapper/322:44:213
606429518,12sleep30-21swapper/319:37:143
2860529511,65sleep227935-21/usr/sbin/munin22:29:392
3154629414,58sleep231591-21ntp_states19:14:362
3154629414,58sleep231591-21ntp_states19:14:362
2580529424,60sleep30-21swapper/322:20:033
2580529424,60sleep30-21swapper/322:20:033
1088229418,10sleep30-21swapper/319:54:053
42192936,11sleep00-21swapper/021:14:110
35982930,8sleep00-21swapper/021:09:420
2742629318,10sleep30-21swapper/322:26:003
2742629318,10sleep30-21swapper/322:26:003
2673829310,12sleep10-21swapper/120:41:261
2673829310,12sleep10-21swapper/120:41:261
1371629357,13sleep30-21swapper/321:44:103
1700429219,12sleep30-21swapper/320:09:433
1586029252,17sleep30-21swapper/320:09:103
1586029252,17sleep30-21swapper/320:09:103
55002915,12sleep20-21swapper/219:34:382
290929113,63sleep22969-21fschecks_time21:09:292
76992907,14sleep30-21swapper/321:24:153
390829014,56sleep13987-21/usr/sbin/munin19:29:411
1559628918,11sleep30-21swapper/323:29:513
1181428918,10sleep30-21swapper/321:35:343
965628813,50sleep00-21swapper/021:29:330
714228818,11sleep30-21swapper/321:23:183
483428816,10sleep30-21swapper/322:56:403
471728820,19sleep20-21swapper/219:34:102
1013728831,32sleep10-21swapper/119:49:391
3256528718,12sleep30-21swapper/322:43:443
2776728714,21sleep10-21swapper/100:09:331
2208528722,19sleep00-21swapper/020:29:100
145752878,12sleep20-21swapper/220:04:412
145752878,12sleep20-21swapper/220:04:412
3259528612,53sleep232610-21apt22:44:082
2938428612,58sleep129439-21apt-key22:34:101
292882860,7sleep00-21swapper/022:34:080
2799928630,36sleep028044-21ntpq19:04:340
2046628615,12sleep30-21swapper/320:23:523
3003028518,10sleep30-21swapper/300:17:063
3003028518,10sleep30-21swapper/300:17:063
1080928512,48sleep210828-21apt-get21:34:202
844528315,21sleep20-21swapper/223:09:232
1451928314,56sleep014047-21apt-get20:04:190
1451928314,56sleep014047-21apt-get20:04:190
3184928228,28sleep00-21swapper/019:14:420
3184928228,28sleep00-21swapper/019:14:420
2084228219,44sleep220848-21modprobe23:49:072
1656828218,47sleep216629-21/usr/sbin/munin20:09:382
251527928,34sleep12534-21apt21:09:071
295792780,6sleep10-21swapper/100:14:371
295792780,6sleep10-21swapper/100:14:371
385227722,27sleep00-21swapper/022:54:270
2641527713,44sleep026464-21cpuspeed22:24:240
2641527713,44sleep026464-21cpuspeed22:24:240
1681127718,29sleep10-21swapper/121:54:091
122727714,45sleep0752-21apt-get21:04:180
661627612,47sleep26630-21fschecks_count19:39:272
2732627620,29sleep00-21swapper/020:44:280
1280327615,38sleep112727-21/usr/sbin/munin19:59:231
1280327615,38sleep112727-21/usr/sbin/munin19:59:231
2384627519,24sleep10-21swapper/120:34:201
194427522,30sleep00-21swapper/019:24:270
3087227412,42sleep030920-21ntpq00:19:290
3087227412,42sleep030920-21ntpq00:19:290
2355227422,27sleep20-21swapper/222:14:332
2355227422,27sleep20-21swapper/222:14:332
1834127412,44sleep018399-21/usr/sbin/munin23:40:420
1681327415,38sleep216895-21find23:34:382
86827318,28sleep00-21swapper/021:04:100
2610427317,26sleep20-21swapper/220:39:422
2610427317,26sleep20-21swapper/220:39:422
219527320,36sleep12197-21/usr/sbin/munin22:49:191
2144027319,35sleep221465-21munin-run22:09:082
2144027319,35sleep221465-21munin-run22:09:082
2054327319,27sleep10-21swapper/122:04:301
2054327319,27sleep10-21swapper/122:04:301
1670627315,41sleep016783-21/usr/sbin/munin20:09:400
1036427319,29sleep00-21swapper/023:14:340
3200127124,27sleep10-21swapper/120:59:261
3009427126,25sleep00-21swapper/020:54:090
2986627117,34sleep030012-21sed19:09:380
2433527112,44sleep224342-21sh23:59:252
783127016,39sleep00-21swapper/019:44:090
573127014,41sleep05771-21/usr/sbin/munin22:59:320
2324927019,32sleep223339-21switchtime23:54:372
228562707,51sleep00-21swapper/022:12:250
228562707,51sleep00-21swapper/022:12:250
2169127026,21sleep00-21swapper/023:49:380
1966227010,15sleep10-21swapper/120:19:341
125027016,28sleep21246-21apt-get21:04:202
635026918,30sleep20-21swapper/221:19:342
481326917,25sleep20-21swapper/221:14:372
2955326919,26sleep10-21swapper/119:09:311
2520426923,28sleep225221-21munin-run20:39:092
2241326930,23sleep20-21swapper/220:29:282
2149726911,44sleep221508-21/usr/sbin/munin23:49:322
2886826817,25sleep20-21swapper/220:49:262
186126814,22sleep10-21swapper/119:24:241
770326720,23sleep20-21swapper/221:24:182
672226715,35sleep16851-21awk21:19:411
3026726716,29sleep011-21ksoftirqd/000:19:090
3026726716,29sleep011-21ksoftirqd/000:19:090
2608426724,26sleep20-21swapper/200:04:302
2238926719,28sleep122419-21munin-run23:54:071
1514926711,43sleep115238-21switchtime20:04:571
1514926711,43sleep115238-21switchtime20:04:571
1350226716,21sleep20-21swapper/223:24:332
1217026715,29sleep00-21swapper/021:39:100
905426614,18sleep00-21swapper/023:09:410
820526615,33sleep08237-21proc_pri21:24:410
326526615,21sleep10-21swapper/121:09:361
2089826617,24sleep10-21swapper/123:49:111
187792667,49sleep20-21swapper/220:17:252
1678426616,29sleep116949-21sed20:09:411
1366726614,35sleep213715-21unin-run21:44:092
39226520,22sleep00-21swapper/019:19:270
2534926515,22sleep20-21swapper/222:19:372
2534926515,22sleep20-21swapper/222:19:372
514026425,19sleep10-21swapper/119:34:291
1989226416,30sleep219923-21awk22:04:072
113726415,21sleep10-21swapper/122:44:391
818826312,33sleep08207-21munin-run23:09:090
493126316,27sleep04950-21apt-get19:34:200
2958526320,24sleep20-21swapper/219:09:322
2058926314,25sleep20-21swapper/220:24:112
1776226317,21sleep00-21swapper/021:54:400
152802639,39sleep015400-21timerandwakeup23:29:350
3026126212,35sleep130267-21ntp_states22:35:111
2797026211,14sleep20-21swapper/200:09:382
2618426211,12sleep00-21swapper/000:04:320
261652629,17sleep10-21swapper/122:24:081
261652629,17sleep10-21swapper/122:24:081
1503726218,24sleep20-21swapper/223:29:292
1403526212,32sleep114043-21diskstats21:44:281
1403526212,32sleep114043-21diskstats21:44:281
313102617,12sleep20-21swapper/220:55:242
3108726121,18sleep00-21swapper/020:54:530
1764726119,27sleep017691-21/usr/sbin/munin20:14:240
109962617,12sleep10-21swapper/119:54:111
476926012,23sleep10-21swapper/121:14:361
263852605,19sleep20-21swapper/222:24:182
263852605,19sleep20-21swapper/222:24:182
338425912,19sleep00-21swapper/019:29:190
3214325916,25sleep232235-21perl00:24:282
2901225921,20sleep20-21swapper/200:14:272
2901225921,20sleep20-21swapper/200:14:272
2454725924,19sleep10-21swapper/122:19:081
2454725924,19sleep10-21swapper/122:19:081
20682598,10sleep20-21swapper/200:29:492
1728625917,25sleep217320-21irqstats21:54:302
1638825923,19sleep00-21swapper/023:34:280
1326025917,19sleep10-21swapper/123:24:281
3050125816,25sleep230495-21apt-get00:19:192
3050125816,25sleep230495-21apt-get00:19:192
207922588,36sleep00-21swapper/023:48:290
737825714,21sleep10-21swapper/123:04:381
3168025717,16sleep10-21swapper/120:59:101
235132576,39sleep00-21swapper/020:33:140
1559025710,19sleep20-21swapper/221:49:302
1129025713,20sleep10-21swapper/121:34:351
558825616,17sleep10-21swapper/122:59:291
37102567,37sleep20-21swapper/200:36:032
3212825617,17sleep00-21swapper/000:24:260
316325620,17sleep20-21swapper/219:29:092
2063725616,24sleep020643-21ntp_states22:04:330
2063725616,24sleep020643-21ntp_states22:04:330
1299025614,25sleep213024-21/usr/sbin/munin19:59:292
1299025614,25sleep213024-21/usr/sbin/munin19:59:292
950625516,23sleep19555-21irqstats21:29:301
497225517,27sleep10-21swapper/122:58:281
285232557,39sleep00-21swapper/000:12:510
2763025514,17sleep20-21swapper/220:44:342
2728925514,24sleep127317-21users22:24:411
2728925514,24sleep127317-21users22:24:411
1862625513,25sleep118621-21apt-get21:59:221
1840925511,21sleep10-21swapper/121:59:091
1462725513,16sleep20-21swapper/221:44:412
1462725513,16sleep20-21swapper/221:44:412
14562558,38sleep10-21swapper/119:22:381
1238925514,24sleep012475-21users23:19:380
1184125516,22sleep20-21swapper/223:19:282
1035725516,18sleep00-21swapper/019:49:420
999425412,24sleep29988-21apt-get23:14:232
854325422,17sleep00-21swapper/019:44:370
773325414,24sleep17730-21apt-get21:24:211
2552925417,22sleep1278-21dbus-daemon00:04:081
2277125413,19sleep00-21swapper/023:54:260
1672425420,18sleep10-21swapper/123:34:371
789125316,21sleep28041-21modprobe19:44:112
2333125312,16sleep00-21swapper/022:14:270
2333125312,16sleep00-21swapper/022:14:270
962325212,24sleep29621-21apt-get19:49:192
732525213,24sleep27349-21ntp_states23:04:342
274425111,22sleep12768-21sh00:34:241
2225325115,16sleep20-21swapper/222:09:362
2225325115,16sleep20-21swapper/222:09:362
2203225112,23sleep222063-21munin-run20:29:082
1930125110,16sleep00-21swapper/020:19:250
661125011,22sleep16615-21diskstats19:39:261
3214325012,20sleep032241-21users22:39:420
939124915,18sleep10-21swapper/119:49:101
3276524912,21sleep232762-21apt-get19:19:212
152324814,15sleep10-21swapper/100:29:321
2581424710,18sleep10-21swapper/100:04:201
3191824611,15sleep20-21swapper/220:59:232
1023524611,21sleep110264-21irqstats23:14:311
92842459,19sleep20-21swapper/221:29:232
248442457,25sleep00-21swapper/020:35:280
2117024213,15sleep10-21swapper/120:24:311
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional