You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-14 - 05:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Mar 14, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18334234765,3458sleep115274-21kworker/1:021:30:311
32590227707,2747sleep24936-21kworker/u8:220:33:282
32590227707,2747sleep24936-21kworker/u8:220:33:282
209462204319,2011sleep119486-21kworker/1:019:55:321
8010260012,571sleep08054-21ntpq00:23:350
8586251815,485sleep28728-21latency_hist19:18:142
17823238313,353sleep217900-21fschecks_time23:13:282
10526232426,241sleep210602-21sed22:48:412
9763228284,168sleep20-21swapper/221:03:312
6572227083,102sleep26540-21latency_hist19:13:142
11993225878,161sleep212043-21idleruntime-cro00:38:102
7910225342,196sleep27976-21latency00:23:332
3655224710,221sleep03694-21ntpq20:43:350
3655224710,221sleep03694-21ntpq20:43:350
143922417,221sleep21505-21fschecks_time00:03:302
25885223680,85sleep225970-21latency_hist20:13:132
6047222919,77sleep20-21swapper/200:18:282
14456222424,185sleep214518-21wc21:18:332
25251222213,195sleep325296-21if_eth021:53:313
25251222213,195sleep325296-21if_eth021:53:313
6555221116,107sleep26572-21modprobe22:38:112
278872210109,22sleep20-21swapper/222:03:102
73422056,166sleep2752-21fschecks_count22:18:332
13947220362,128sleep214000-21latency_hist21:18:132
20653219426,151sleep220651-21munin-run23:23:132
7075218545,91sleep20-21swapper/220:53:432
24757218222,132sleep024785-21munin-run21:53:110
30421217721,133sleep20-21swapper/223:53:122
2115321749,81sleep221215-21modprobe19:58:112
20088216863,39sleep220138-21readlink21:38:112
11541216810,89sleep20-21swapper/221:08:402
26487216717,133sleep126501-21awk23:38:381
26487216717,133sleep126501-21awk23:38:381
1178621628,118sleep211875-21cat22:53:332
3270721587,26sleep20-21swapper/223:58:382
13149215711,82sleep20-21swapper/219:33:122
9370215229,109sleep29462-21cat19:18:392
9370215229,109sleep29462-21cat19:18:392
2997214910,123sleep23092-21fschecks_time00:08:312
14940214917,116sleep114993-21idleruntime-cro19:38:121
14940214917,116sleep114993-21idleruntime-cro19:38:121
7702213911,113sleep07767-21cat20:58:130
32087213829,94sleep232193-21latency_hist23:58:132
23917213328,87sleep223920-21cron23:33:102
12844213212,20sleep10-21swapper/121:13:321
26502213131,42sleep226562-21find23:38:392
26502213131,42sleep226562-21find23:38:392
5071212728,87sleep25186-21timerandwakeup00:13:402
2997321278,104sleep230117-21grep20:23:432
6316212615,97sleep16330-21kernelversion00:18:351
28417212642,72sleep228404-21apt-get22:03:272
1352212610,103sleep00-21swapper/000:03:240
4376212531,67sleep14541-21grep19:03:401
17542212527,36sleep20-21swapper/219:43:422
27950212423,79sleep028080-21apt-config22:03:120
26096212311,96sleep25016-21cyclictest20:13:212
9029212214,40sleep20-21swapper/222:43:442
3200021227,29sleep232064-21mailstats22:13:402
29586211913,92sleep029607-21awk23:48:380
2349921197,71sleep20-21swapper/221:48:252
671221180,7sleep30-21swapper/319:13:143
5668211411,88sleep35272-21/usr/sbin/munin22:33:363
3734211329,70sleep23735-21apt-get22:28:222
22437211310,86sleep222501-21mailstats21:43:402
4807211225,75sleep213441-21snmpd20:48:262
25706211220,76sleep125835-21sed21:53:411
25706211220,76sleep125835-21sed21:53:411
22440211229,72sleep213441-21snmpd19:59:242
19373211212,87sleep113441-21snmpd19:50:111
3341211131,67sleep23355-21grep20:43:292
3341211131,67sleep23355-21grep20:43:292
2597621117,92sleep213441-21snmpd21:53:582
2597621117,92sleep213441-21snmpd21:53:582
32579210917,70sleep132074-21apt-get20:33:231
32579210917,70sleep132074-21apt-get20:33:231
31303210972,19sleep20-21swapper/220:28:392
13557210931,66sleep213619-21uname22:58:382
31476210715,76sleep031580-21ls20:28:410
2205210711,25sleep20-21swapper/220:38:412
2205210711,25sleep20-21swapper/220:38:412
22423210425,37sleep20-21swapper/223:28:132
20351210416,73sleep320359-21ntp_states19:53:403
15535210411,76sleep315685-21cut21:23:153
2401821036,11sleep20-21swapper/220:04:452
14646210324,21sleep20-21swapper/223:03:262
21709210126,64sleep221713-21munin-run21:43:122
19922210112,73sleep120000-21/usr/sbin/munin23:18:391
16872210112,74sleep016881-21diskstats19:43:280
1051621017,80sleep210666-21latency_hist00:33:142
1051621017,80sleep210666-21latency_hist00:33:142
28872210029,39sleep228882-21munin-run23:48:122
10869210015,30sleep011015-21latency_hist21:08:140
2302429913,68sleep023025-21/usr/sbin/munin20:03:260
3160129820,52sleep131683-21cut22:13:311
2171429817,22sleep30-21swapper/321:43:133
2091029820,59sleep021057-21latency_hist23:23:150
1859129821,16sleep20-21swapper/219:48:342
2158129713,64sleep021616-21proc_pri23:23:430
168492978,73sleep20-21swapper/221:26:202
86162967,75sleep28738-21cat20:58:432
763929516,66sleep37667-21cpu00:23:263
161922958,15sleep20-21swapper/223:08:232
108062949,48sleep25016-21cyclictest19:23:392
492929329,40sleep04959-21munin-run19:08:110
2502229322,11sleep20-21swapper/223:34:032
2326529316,61sleep323411-21latency_hist21:48:143
1440429313,64sleep114455-21irqstats21:18:321
2155229229,46sleep321567-21cut19:58:283
3159629114,58sleep00-21swapper/022:13:300
2306029116,59sleep323102-21/usr/sbin/munin23:28:353
182532915,53sleep20-21swapper/221:29:312
1408429110,16sleep20-21swapper/219:34:052
1408429110,16sleep20-21swapper/219:34:052
958629018,54sleep39623-21ntpq00:28:343
958629018,54sleep39623-21ntpq00:28:343
1534129021,46sleep00-21swapper/019:38:310
1534129021,46sleep00-21swapper/019:38:310
299652887,17sleep20-21swapper/222:08:232
289312886,51sleep20-21swapper/220:22:462
2334128820,41sleep123479-21unixbench_multi23:28:411
168428816,34sleep00-21swapper/020:38:220
168428816,34sleep00-21swapper/020:38:220
270552867,67sleep00-21swapper/020:14:000
1750128620,51sleep317580-21latency_hist23:13:123
1197528618,51sleep012035-21fschecks_time19:28:300
165802857,19sleep216732-21cat19:43:132
165802857,19sleep216732-21cat19:43:132
80282847,66sleep20-21swapper/222:41:502
264728426,45sleep22167-21/usr/sbin/munin22:23:372
1158328411,58sleep011590-21df_inode22:53:270
2790428320,46sleep327947-21unin-run22:03:113
2637328212,57sleep026385-21latency_hist21:58:120
2637328212,57sleep026385-21latency_hist21:58:120
230262820,7sleep10-21swapper/120:03:261
149828226,27sleep30-21swapper/320:38:133
149828226,27sleep30-21swapper/320:38:133
2231828111,24sleep10-21swapper/121:43:351
151828122,36sleep31575-21sshd00:03:313
2932928028,27sleep10-21swapper/123:48:311
3130527926,33sleep00-21swapper/022:13:140
2876227911,59sleep00-21swapper/023:47:170
2398727910,43sleep00-21swapper/023:33:120
1126327913,47sleep310707-21/usr/sbin/munin00:33:393
689427824,27sleep00-21swapper/019:13:160
424127722,27sleep00-21swapper/000:13:130
413227730,24sleep20-21swapper/219:03:372
2366127722,30sleep10-21swapper/121:48:321
2108027732,26sleep30-21swapper/323:23:223
2114527615,42sleep021288-21unixbench_singl21:38:420
946327523,27sleep30-21swapper/319:18:403
946327523,27sleep30-21swapper/319:18:403
663627519,25sleep00-21swapper/022:38:130
1060327515,38sleep010617-21/usr/sbin/munin19:23:330
2583427410,13sleep00-21swapper/020:13:110
1728027442,17sleep30-21swapper/319:43:393
925827322,24sleep00-21swapper/021:03:130
506027226,19sleep10-21swapper/122:33:131
1277927210,42sleep012572-21/usr/sbin/munin21:13:310
3001727113,23sleep10-21swapper/122:08:261
1204927122,24sleep10-21swapper/121:09:481
111127112,42sleep31151-21ps22:18:423
309427016,40sleep33152-21if_eth000:08:323
1989827012,39sleep019965-21df19:53:290
168822705,11sleep10-21swapper/119:43:291
3201226925,23sleep10-21swapper/123:58:121
3192626924,27sleep331965-21systemctl23:58:103
2895226914,37sleep128984-21proc_pri22:04:351
182002699,31sleep30-21swapper/323:13:373
1501526921,26sleep30-21swapper/323:03:353
3005026816,36sleep330060-21forks22:08:283
2933426817,37sleep129364-21grep20:23:301
2808526819,29sleep10-21swapper/122:03:141
1466326832,24sleep014722-21sendmail_mailqu21:18:390
716926716,36sleep07173-21systemctl00:21:290
644726714,22sleep10-21swapper/120:53:291
2973726721,19sleep30-21swapper/323:48:413
1897326716,33sleep319089-21sed19:48:433
890026618,30sleep08964-21grep22:43:420
500726615,35sleep05068-21if_eth020:48:320
49826618,23sleep30-21swapper/323:58:433
271782667,16sleep00-21swapper/023:41:480
271782667,16sleep00-21swapper/023:41:480
612626514,21sleep10-21swapper/120:53:131
2157126514,20sleep00-21swapper/019:58:290
1800626520,28sleep10-21swapper/123:13:311
1207226520,23sleep00-21swapper/000:38:120
115526522,21sleep00-21swapper/022:18:430
821426415,28sleep10-21swapper/120:58:341
572726420,28sleep15749-21munin-run00:18:111
3242326417,19sleep00-21swapper/023:58:310
31792645,50sleep00-21swapper/022:24:400
294962646,11sleep00-21swapper/022:05:510
254622645,16sleep10-21swapper/123:35:031
1214226421,23sleep30-21swapper/322:53:423
1161526420,23sleep30-21swapper/321:08:413
1141726422,22sleep10-21swapper/100:33:411
747826315,21sleep10-21swapper/119:13:331
2151526319,28sleep121549-21/usr/sbin/munin23:23:401
1320026316,22sleep30-21swapper/319:33:133
1045426318,25sleep10-21swapper/122:48:401
744026220,18sleep30-21swapper/300:23:133
3158726220,26sleep331594-21df_inode22:13:293
301962627,10sleep00-21swapper/020:24:090
227292625,11sleep10-21swapper/120:03:071
1832426215,21sleep00-21swapper/019:48:240
1501826220,19sleep30-21swapper/319:38:143
1501826220,19sleep30-21swapper/319:38:143
147082627,12sleep00-21swapper/019:35:120
147082627,12sleep00-21swapper/019:35:120
805626118,25sleep10-21swapper/100:23:361
580526117,18sleep30-21swapper/300:18:133
34012617,41sleep10-21swapper/122:27:321
2784326119,27sleep327930-21sed23:43:333
2016826121,19sleep00-21swapper/021:38:130
1585526115,17sleep30-21swapper/321:23:313
764926016,25sleep37700-21munin-run20:58:123
758526016,25sleep10-21swapper/122:38:421
671226014,20sleep30-21swapper/319:13:153
2837526021,16sleep10-21swapper/120:18:411
185072605,17sleep00-21swapper/021:32:440
1630826011,34sleep116398-21switchtime21:23:411
151242609,26sleep00-21swapper/023:03:400
1099826018,20sleep10-21swapper/119:23:421
3210025918,22sleep332189-21gpgv20:33:123
2621725919,20sleep30-21swapper/320:13:303
1745125920,18sleep10-21swapper/123:13:101
1355025917,26sleep113551-21ntp_states22:58:371
839625815,26sleep38415-21apt-get22:43:253
634925813,31sleep16350-21sshd22:35:311
32125821,20sleep10-21swapper/123:58:411
318002586,11sleep10-21swapper/120:29:471
2924725816,26sleep329244-21apt-get20:23:233
184222585,10sleep30-21swapper/321:31:383
1562525818,20sleep10-21swapper/119:38:361
1562525818,20sleep10-21swapper/119:38:361
695825714,28sleep06970-21fschecks_count22:38:290
419825716,24sleep34204-21ntp_states22:28:393
2578725713,31sleep025812-21cat23:38:130
169212575,40sleep00-21swapper/021:27:150
1010525719,22sleep010207-21latency_hist19:23:130
1010525719,22sleep010207-21latency_hist19:23:130
967825617,22sleep10-21swapper/122:48:111
549125612,22sleep10-21swapper/119:08:321
383425620,18sleep10-21swapper/120:43:401
383425620,18sleep10-21swapper/120:43:401
3261725615,24sleep032576-21apt-get20:33:460
3261725615,24sleep032576-21apt-get20:33:460
2828125618,17sleep00-21swapper/020:18:400
2774525621,17sleep10-21swapper/123:43:311
2057625618,15sleep10-21swapper/121:38:311
1954925617,24sleep119684-21unixbench_singl21:33:421
1689225619,16sleep00-21swapper/023:08:410
985925514,20sleep30-21swapper/321:03:323
56272559,32sleep05662-21ntpq19:08:350
3196925514,25sleep032011-21unin-run23:58:110
290325514,24sleep12896-21apt-get00:08:261
289925517,21sleep30-21swapper/322:23:433
2770225516,22sleep327699-21apt-get20:18:233
2619925513,18sleep10-21swapper/120:13:281
2602225514,24sleep326018-21apt-get23:38:243
2602225514,24sleep326018-21apt-get23:38:243
1931125512,24sleep319317-21apt-get23:18:213
1259725518,17sleep30-21swapper/321:13:263
1189125516,24sleep111958-21df19:28:271
994325415,22sleep09962-21apt-get22:48:220
2410625419,18sleep30-21swapper/321:48:423
196125415,17sleep10-21swapper/100:03:411
1525425411,25sleep315247-21apt-get19:38:263
1525425411,25sleep315247-21apt-get19:38:263
1049025416,24sleep110514-21cat00:33:131
1049025416,24sleep110514-21cat00:33:131
493225315,20sleep00-21swapper/000:13:380
386325311,16sleep00-21swapper/022:28:320
366225312,24sleep33660-21apt-get19:03:223
2906325314,16sleep30-21swapper/322:04:373
1164325314,23sleep311652-21munin-run19:28:123
999425213,25sleep010067-21sendmail_mailtr21:03:380
24932529,17sleep10-21swapper/120:38:441
24932529,17sleep10-21swapper/120:38:441
2329425219,16sleep30-21swapper/320:03:333
1151325213,24sleep111508-21apt-get22:53:241
879025114,21sleep18787-21apt-get19:18:211
879025114,21sleep18787-21apt-get19:18:211
335925112,21sleep33462-21perl20:43:303
335925112,21sleep33462-21perl20:43:303
2457225112,23sleep324567-21apt-get20:08:263
2222625117,17sleep00-21swapper/021:43:340
1048625114,22sleep010488-21systemd-run00:33:120
1048625114,22sleep010488-21systemd-run00:33:120
956825016,18sleep10-21swapper/121:03:261
924625014,21sleep39256-21munin-run21:03:123
214025013,22sleep32164-21ntp_states20:38:383
214025013,22sleep32164-21ntp_states20:38:383
194325013,20sleep11952-21unin-run22:23:121
1924025012,21sleep319282-21ntp_states21:33:363
1518425013,22sleep115255-21sensors23:03:421
1335525015,15sleep00-21swapper/022:58:320
1060325012,18sleep30-21swapper/322:48:423
687124912,22sleep36867-21apt-get22:38:243
480824914,18sleep30-21swapper/320:48:273
1794824915,18sleep00-21swapper/023:13:300
1435524910,23sleep314382-21munin-run23:03:113
2474624813,17sleep10-21swapper/120:08:321
2444324110,15sleep30-21swapper/323:33:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional