You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 12:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Mar 30, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
307362157517,1542sleep230728-21/usr/sbin/munin00:32:162
89442125022,1212sleep18946-21ntp_states21:37:411
289682102016,986sleep229035-21df22:42:322
1350429476,932sleep11988-1kworker/1:1H21:52:391
31107275413,725sleep331119-21/usr/sbin/munin19:22:393
133862601566,21sleep213388-21sshd23:35:022
2966723997,357sleep029703-21ntpq19:17:390
32758234164,153sleep0355-21fschecks_time19:28:010
32758234164,153sleep0355-21fschecks_time19:28:010
1961022869,55sleep00-21swapper/023:55:070
1961022869,55sleep00-21swapper/023:55:070
13422227013,190sleep013497-21switchtime20:07:470
2396822698,55sleep00-21swapper/020:42:360
32329226248,23sleep00-21swapper/022:52:350
2559822619,207sleep025638-21latency_hist22:32:160
607622535,211sleep00-21swapper/019:46:510
3011422528,201sleep030209-21timerwakeupswit00:27:460
30961224638,130sleep00-21swapper/000:32:260
1543022409,218sleep00-21swapper/021:57:430
8571223910,23sleep00-21swapper/019:52:520
7103223944,135sleep07132-21grep23:17:150
13508223342,121sleep013562-21fschecks_time21:52:420
1328221638,126sleep01476-21unixbench_multi21:12:490
8192219816,164sleep18171-21apt-get19:52:371
2017521985,10sleep00-21swapper/020:30:040
27088218161,64sleep027192-21latency_hist19:12:180
27088218161,64sleep027192-21latency_hist19:12:180
27182217447,70sleep00-21swapper/022:37:170
20393216920,131sleep320461-21latency_hist20:32:173
4653216856,58sleep00-21swapper/023:07:380
4653216856,58sleep00-21swapper/023:07:380
29401216717,137sleep00-21swapper/020:58:230
29967216537,106sleep00-21swapper/021:02:270
29967216537,106sleep00-21swapper/021:02:270
26944216514,134sleep327018-21find00:17:433
26944216514,134sleep327018-21find00:17:433
22832216226,116sleep00-21swapper/022:22:350
32396215410,129sleep032509-21latency_hist19:27:200
2702321546,134sleep027143-21cat00:17:440
2702321546,134sleep027143-21cat00:17:440
14781215315,124sleep314826-21awk20:12:423
23292215210,126sleep023382-21cut00:07:350
22577215211,126sleep022613-21ntpq20:37:390
9207215022,110sleep00-21swapper/023:22:280
18049214737,44sleep00-21swapper/020:22:450
22280214629,96sleep00-21swapper/000:02:460
11398214615,116sleep011473-21fschecks_time20:02:340
30477214420,108sleep230558-21cut21:02:452
30477214420,108sleep230558-21cut21:02:452
610621438,121sleep06236-21grep21:27:470
16362214324,54sleep00-21swapper/020:17:400
1267214230,53sleep01356-21fschecks_time22:57:320
20108214011,87sleep020191-21mailstats22:12:460
20108214011,87sleep020191-21mailstats22:12:460
3202921347,113sleep00-21swapper/021:07:440
3202921347,113sleep00-21swapper/021:07:440
3093213213,70sleep03245-21latency_hist19:37:180
1255221316,28sleep00-21swapper/021:48:260
1255221316,28sleep00-21swapper/021:48:260
12948213017,56sleep00-21swapper/023:32:460
2547212926,41sleep00-21swapper/021:17:370
7378212711,16sleep20-21swapper/221:32:412
7378212711,16sleep20-21swapper/221:32:412
5916212615,94sleep25983-21/usr/sbin/munin21:27:452
28016212612,70sleep028096-21xargs22:37:450
11075212611,68sleep011084-21latency_hist20:02:170
19366212310,56sleep019456-21sort22:12:170
765321228,93sleep031865-1kworker/0:2H23:17:270
21743212211,47sleep00-21swapper/022:17:460
6800212012,97sleep06838-21ntp_kernel_pll_19:47:430
3788211911,29sleep00-21swapper/019:37:460
28119211917,85sleep128144-21idleruntime-cro20:57:151
17582211823,23sleep10-21swapper/123:47:451
2028021166,12sleep20-21swapper/222:12:482
2028021166,12sleep20-21swapper/222:12:482
1791921156,11sleep00-21swapper/023:48:170
27420211425,64sleep127414-21apt-get22:37:301
2443211417,80sleep22499-21idleruntime21:17:352
23476211420,80sleep323558-21sed00:07:383
2279211324,73sleep12290-21cpu21:17:301
864211223,71sleep3588-21cron22:57:153
15496211214,82sleep115565-21awk21:57:441
3821211124,39sleep00-21swapper/023:04:590
5370210828,40sleep10-21swapper/121:27:271
24967210719,64sleep325109-21grep22:27:473
2126210715,72sleep31703-21/usr/sbin/munin19:33:003
30761210617,18sleep10-21swapper/121:02:481
30761210617,18sleep10-21swapper/121:02:481
1961210424,61sleep21995-21munin-run21:17:152
8194210326,56sleep38195-21apt-get19:52:393
10045210318,71sleep210104-21cpuspeed21:42:312
5858210212,76sleep15914-21cpuspeed23:12:481
20718210231,56sleep020808-21fschecks_time20:32:340
19172210112,64sleep219062-21/usr/sbin/munin20:27:352
28966210034,17sleep00-21swapper/022:42:300
2817321007,36sleep00-21swapper/000:22:370
2444929919,63sleep024481-21latency19:02:370
2444929919,63sleep024481-21latency19:02:370
979629840,22sleep00-21swapper/021:42:170
3177129828,56sleep031805-21munin-run22:52:150
1385529834,17sleep00-21swapper/023:37:260
210482976,40sleep021051-21modprobe23:58:140
210482976,40sleep021051-21modprobe23:58:140
2905229635,37sleep129113-21fschecks_time22:42:351
1201329622,51sleep112035-21modprobe23:32:141
261042957,39sleep026190-21cat00:17:160
261042957,39sleep026190-21cat00:17:160
1026329538,44sleep00-21swapper/021:42:370
715929430,47sleep00-21swapper/021:32:350
715929430,47sleep00-21swapper/021:32:350
1541329412,68sleep315428-21/usr/sbin/munin21:57:423
1296229414,64sleep213066-21fschecks_time20:07:362
1659429318,57sleep216655-21fschecks_time22:02:342
1168829314,60sleep211774-21/usr/sbin/munin21:47:362
1168829314,60sleep211774-21/usr/sbin/munin21:47:362
3262429213,62sleep232120-21apt-get19:27:262
3262429213,62sleep232120-21apt-get19:27:262
2815129221,57sleep328171-21cut00:22:363
2351929219,19sleep20-21swapper/220:42:172
1677729218,59sleep116839-21wc22:02:371
1468228725,39sleep114746-21cat20:12:401
1042928615,54sleep210541-21cat19:57:472
513628521,42sleep35222-21grep21:27:173
2142028521,31sleep30-21swapper/300:02:173
2142028521,31sleep30-21swapper/300:02:173
1672728433,13sleep00-21swapper/022:02:360
2661428321,46sleep00-21swapper/020:52:170
1313728324,26sleep10-21swapper/121:52:191
1313728324,26sleep10-21swapper/121:52:191
2662428223,27sleep10-21swapper/120:52:181
258332820,6sleep30-21swapper/319:07:313
258332820,6sleep30-21swapper/319:07:313
274742819,55sleep00-21swapper/020:52:450
2165928120,27sleep10-21swapper/100:02:311
1536228119,12sleep00-21swapper/020:13:200
899828014,46sleep29144-21latency_hist23:22:182
1627528021,42sleep00-21swapper/023:42:470
819327910,51sleep20-21swapper/219:52:382
720027921,33sleep27332-21apt-config23:17:172
35027916,41sleep1469-21unixbench_singl22:52:451
1673327924,27sleep10-21swapper/123:47:171
2877627819,10sleep00-21swapper/019:14:360
399427721,39sleep14028-21awk23:07:141
278932777,16sleep10-21swapper/100:22:301
2451727717,40sleep224665-21latency_hist00:12:172
165627726,32sleep10-21swapper/122:57:411
1287427715,44sleep312943-21switchtime23:32:453
237027614,42sleep22467-21/usr/sbin/munin19:33:082
2368227622,30sleep123746-21find00:07:441
207122767,41sleep30-21swapper/320:32:333
1575927614,46sleep215898-21cut20:17:182
1102427618,45sleep00-21swapper/023:27:370
2530527521,17sleep20-21swapper/220:47:302
2399727523,32sleep324005-21modprobe22:27:143
1800627521,29sleep30-21swapper/320:22:443
1668127518,39sleep316704-21modprobe23:47:143
1628927525,24sleep20-21swapper/222:02:182
94942746,56sleep30-21swapper/321:38:563
376627318,39sleep33786-21/usr/sbin/munin19:37:453
2936727316,40sleep329412-21/usr/sbin/munin22:42:413
2168627323,31sleep321740-21awk22:17:453
849427215,16sleep20-21swapper/221:37:302
615927226,25sleep20-21swapper/219:47:172
38292728,18sleep00-21swapper/021:22:340
85827112,46sleep213441-21snmpd22:57:102
666127116,36sleep16371-21/usr/sbin/munin19:47:401
323727122,23sleep20-21swapper/223:02:432
2674227116,41sleep226784-21ntp_offset00:17:382
2674227116,41sleep226784-21ntp_offset00:17:382
2492827124,28sleep125076-21/usr/sbin/munin19:02:461
2492827124,28sleep125076-21/usr/sbin/munin19:02:461
1814227114,23sleep10-21swapper/120:22:461
1411527116,28sleep20-21swapper/223:37:362
1334927113,44sleep313419-21mailstats20:07:463
673127017,22sleep30-21swapper/321:32:183
2905127021,30sleep20-21swapper/220:57:482
2549027022,30sleep325508-21munin-run19:07:163
2549027022,30sleep325508-21munin-run19:07:163
1977827025,29sleep20-21swapper/223:57:152
1977827025,29sleep20-21swapper/223:57:152
1725127017,25sleep30-21swapper/320:22:153
1365727026,23sleep30-21swapper/321:52:463
124412709,14sleep30-21swapper/321:47:493
124412709,14sleep30-21swapper/321:47:493
299992698,25sleep30-21swapper/319:17:463
293672698,13sleep10-21swapper/119:17:321
2592426920,23sleep10-21swapper/122:32:331
2223026919,30sleep30-21swapper/300:02:453
941526813,24sleep30-21swapper/323:22:353
385126810,43sleep13917-21/usr/sbin/munin19:37:471
3135126819,19sleep20-21swapper/219:22:462
2811426812,40sleep328156-21/usr/sbin/munin19:12:413
68526717,28sleep10-21swapper/119:28:101
68526717,28sleep10-21swapper/119:28:101
3090826727,19sleep30-21swapper/322:47:383
2064426713,36sleep220712-21/usr/sbin/munin23:57:432
2064426713,36sleep220712-21/usr/sbin/munin23:57:432
130752679,41sleep113201-21/usr/sbin/munin23:32:471
1178026713,37sleep111844-21ps20:02:431
2019326619,32sleep120279-21/usr/sbin/munin22:12:471
2019326619,32sleep120279-21/usr/sbin/munin22:12:471
3025226517,34sleep30-21swapper/321:02:383
3025226517,34sleep30-21swapper/321:02:383
302102656,15sleep20-21swapper/219:17:492
2412526518,21sleep30-21swapper/320:42:393
1982026522,25sleep319863-21unin-run23:57:163
1982026522,25sleep319863-21unin-run23:57:163
1803226519,30sleep118034-21/usr/sbin/munin22:07:251
920926421,23sleep20-21swapper/223:22:302
2330926415,19sleep20-21swapper/222:22:452
2165326413,22sleep20-21swapper/222:17:442
2096326419,26sleep321105-21latency_hist22:17:173
2096326419,26sleep321105-21latency_hist22:17:173
1198226412,36sleep211983-21sshd23:31:512
101332647,10sleep10-21swapper/123:22:581
383026319,28sleep13897-21/usr/sbin/munin21:22:351
289002639,12sleep30-21swapper/320:57:453
40126221,20sleep10-21swapper/121:12:171
40126221,20sleep10-21swapper/121:12:171
3136926224,21sleep30-21swapper/321:07:183
3136926224,21sleep30-21swapper/321:07:183
3059526213,33sleep130607-21/usr/sbin/munin22:47:321
2602326216,31sleep226057-21basename00:17:142
2602326216,31sleep226057-21basename00:17:142
2080926219,21sleep10-21swapper/120:32:351
1814426217,23sleep20-21swapper/222:07:352
1436226218,27sleep10-21swapper/123:37:441
733626119,21sleep30-21swapper/323:17:183
3074926111,35sleep330901-21sed00:32:173
2402626118,21sleep10-21swapper/120:42:371
1783826120,25sleep217907-21memory20:22:392
1542626114,29sleep215419-21apt-get23:42:292
648526022,17sleep20-21swapper/219:47:372
6122605,11sleep10-21swapper/121:12:211
2829126020,18sleep20-21swapper/200:22:392
2671026021,23sleep10-21swapper/100:17:371
2671026021,23sleep10-21swapper/100:17:371
2605826015,25sleep326102-21munin-run00:17:153
2605826015,25sleep326102-21munin-run00:17:153
2578826020,21sleep30-21swapper/320:47:443
17602607,10sleep30-21swapper/321:15:183
1471426018,27sleep214749-21latency_hist21:57:172
1041226015,32sleep310448-21ntpq21:42:403
1004326016,27sleep110037-21apt-get21:42:301
897125916,28sleep18996-21cat23:22:171
294412596,12sleep10-21swapper/100:27:311
1730625917,17sleep20-21swapper/220:22:182
170225913,24sleep10-21swapper/119:32:261
118692595,13sleep30-21swapper/323:30:233
100125918,22sleep21145-21modprobe22:57:182
824825815,20sleep10-21swapper/121:37:171
824825815,20sleep10-21swapper/121:37:171
331525821,23sleep30-21swapper/323:02:443
2792325814,26sleep328013-21grep22:37:443
2512825825,18sleep10-21swapper/100:12:411
2512825825,18sleep10-21swapper/100:12:411
502425720,18sleep30-21swapper/323:07:453
502425720,18sleep30-21swapper/323:07:453
3232425722,15sleep20-21swapper/200:37:172
3214925710,21sleep10-21swapper/119:27:171
3163725715,28sleep231644-21forks21:07:332
3163725715,28sleep231644-21forks21:07:332
2819925722,18sleep30-21swapper/320:57:173
2623925716,19sleep30-21swapper/322:32:403
2443125719,17sleep20-21swapper/220:42:462
1853125716,25sleep318590-21/usr/sbin/munin22:07:443
1776825716,17sleep30-21swapper/322:07:153
1697225713,25sleep20-21swapper/223:47:302
108852575,14sleep30-21swapper/320:00:273
483025616,25sleep14897-21/usr/sbin/munin23:07:431
483025616,25sleep14897-21/usr/sbin/munin23:07:431
11462569,11sleep30-21swapper/322:57:193
494425523,17sleep10-21swapper/119:42:361
455325517,24sleep24630-21latency23:07:362
455325517,24sleep24630-21latency23:07:362
2415625513,24sleep224174-21apt-get19:02:272
2415625513,24sleep224174-21apt-get19:02:272
2273525516,23sleep122732-21apt-get22:22:281
2229425517,16sleep30-21swapper/320:37:343
2228425514,24sleep122291-21diskstats20:37:321
1879125512,17sleep10-21swapper/123:52:381
1879125512,17sleep10-21swapper/123:52:381
1734725519,22sleep317391-21ntpq23:47:393
1286525512,25sleep112859-21apt-get20:07:311
3184025412,25sleep131888-21users00:32:481
2164625413,26sleep121652-21ntp_states22:17:431
143825419,19sleep30-21swapper/319:32:163
143825419,19sleep30-21swapper/319:32:163
1108725417,16sleep10-21swapper/120:02:181
422325314,19sleep20-21swapper/221:22:442
2780125319,16sleep10-21swapper/119:12:351
2723725322,15sleep20-21swapper/220:52:392
2436025313,16sleep10-21swapper/122:27:331
2296825315,16sleep20-21swapper/200:07:162
71172526,11sleep30-21swapper/321:32:343
71172526,11sleep30-21swapper/321:32:343
460025213,23sleep24610-21munin-run19:42:162
2808225216,21sleep20-21swapper/219:12:402
75512515,10sleep30-21swapper/319:50:533
526725112,16sleep30-21swapper/319:42:423
3098525114,22sleep330980-21apt-get00:32:293
2585325112,18sleep10-21swapper/119:07:341
2585325112,18sleep10-21swapper/119:07:341
2576925111,24sleep225832-21/usr/sbin/munin19:07:312
2576925111,24sleep225832-21/usr/sbin/munin19:07:312
2455525115,21sleep20-21swapper/222:27:382
200772516,32sleep10-21swapper/120:28:521
1172425113,22sleep211763-21ntp_states20:02:402
2084324912,14sleep20-21swapper/220:32:362
765824812,21sleep37650-21apt-get23:17:303
2752824811,20sleep227625-21perl22:37:352
3087724714,14sleep20-21swapper/222:47:372
2710024712,20sleep227125-21munin-run22:37:152
26922479,20sleep32729-21ntp_states21:17:403
1433224715,16sleep30-21swapper/323:37:413
1158024711,15sleep30-21swapper/320:02:373
2843924612,21sleep128433-21apt-get20:57:311
1640324611,20sleep116445-21ntp_states20:17:411
2009824510,20sleep120095-21apt-get23:57:261
2009824510,20sleep120095-21apt-get23:57:261
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional