You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-04 - 14:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sat Apr 04, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5912211416,1124sleep03553-1kworker/0:0H19:47:300
2706926647,641sleep027119-21unin-run00:21:580
32056263120,588sleep232130-21/usr/sbin/munin22:52:272
10345263014,601sleep310373-21/usr/sbin/munin23:27:213
10345263014,601sleep310373-21/usr/sbin/munin23:27:213
995025496,532sleep09961-21ntpq21:42:230
1202224830,45sleep011-21ksoftirqd/020:07:000
993324657,414sleep013441-21snmpd19:57:360
8009243540,332sleep00-21swapper/019:52:270
13364240019,333sleep013383-21uname23:37:200
14587230715,275sleep214660-21find20:13:212
1693923027,279sleep017086-21awk23:47:290
1693923027,279sleep017086-21awk23:47:290
22093227920,59sleep00-21swapper/020:37:220
27939225622,217sleep327941-21sshd20:57:123
12903225033,197sleep013055-21latency_hist23:36:590
17412224113,213sleep217448-21ntpq20:22:222
15492223342,179sleep03553-1kworker/0:0H20:17:110
15492223342,179sleep03553-1kworker/0:0H20:17:110
21908223136,135sleep021935-21basename22:21:570
12580223013,202sleep312586-21/usr/sbin/munin20:07:243
20222222344,118sleep020391-21latency_hist22:17:000
1405822160,17sleep10-21swapper/120:13:071
24718221219,124sleep024867-21unixbench_singl00:12:290
322142211100,64sleep00-21swapper/022:52:300
17970221014,179sleep118105-21awk22:07:361
1454222008,135sleep00-21swapper/021:57:210
31565219515,138sleep031638-21awk21:07:260
28468218216,90sleep028524-21basename19:16:570
15906217937,67sleep025045-21cyclictest23:46:570
2525221789,40sleep00-21swapper/019:07:120
2525221789,40sleep00-21swapper/019:07:120
13320217813,62sleep00-21swapper/021:52:290
11136217739,79sleep00-21swapper/021:47:160
2322217312,64sleep02470-21latency_hist19:37:000
2887216632,110sleep33034-21grep23:02:303
27443216619,132sleep227484-21uniq00:22:162
987721630,7sleep30-21swapper/323:27:003
18169216210,141sleep113441-21snmpd20:25:151
29246216133,68sleep00-21swapper/019:17:260
24232215671,17sleep00-21swapper/022:27:260
19814215339,54sleep00-21swapper/023:57:270
6849214632,99sleep06903-21processes21:32:260
6849214632,99sleep06903-21processes21:32:260
15179214529,100sleep015204-21munin-run20:16:590
1337821438,117sleep013407-21idleruntime-cro20:11:580
22824214030,98sleep022866-21tr00:07:200
7349213413,109sleep07411-21uname23:17:250
27105213411,81sleep027243-21unixbench_singl20:52:280
30907213235,46sleep00-21swapper/019:22:270
22568213035,81sleep022604-21ntp_kernel_err22:22:210
11496212919,87sleep00-21swapper/023:32:070
27753212515,96sleep027789-21ntp_kernel_pll_00:22:220
11285212533,74sleep00-21swapper/023:31:590
11285212533,74sleep00-21swapper/023:31:590
1327212137,40sleep00-21swapper/022:57:300
30539212014,23sleep230634-21perl00:32:172
4649211911,92sleep14714-21switchtime19:42:271
4649211911,92sleep14714-21switchtime19:42:271
3836211849,54sleep03893-21grep23:07:150
29547211818,78sleep10-21swapper/100:27:281
21456211828,76sleep021544-21cat00:02:270
2139721169,92sleep021448-21sendmail20:36:570
17648211662,16sleep00-21swapper/020:22:280
23688211519,75sleep123721-21munin-run00:11:571
29867211310,25sleep00-21swapper/021:02:220
1594221130,7sleep10-21swapper/122:02:181
17030211122,73sleep00-21swapper/022:06:580
620211019,18sleep20-21swapper/222:57:082
19254210929,65sleep019368-21wc20:27:290
18590210911,58sleep00-21swapper/022:11:590
24782210820,72sleep124787-21sort20:47:081
10842210811,80sleep110901-21latency20:02:221
9264210745,43sleep00-21swapper/021:41:590
9264210745,43sleep00-21swapper/021:41:590
6208210616,15sleep20-21swapper/223:12:302
28890210414,70sleep128403-21/usr/sbin/munin22:42:251
31580210339,54sleep030722-1kworker/u9:100:35:180
20619210311,77sleep320114-21apt-get22:17:073
55021029,75sleep132582-21/usr/sbin/munin21:12:241
3047210210,71sleep00-21swapper/021:22:000
19089210218,68sleep119135-21cat23:56:591
13142210213,70sleep213208-21/usr/sbin/munin21:52:272
6564210120,61sleep16595-21idleruntime-cro23:16:571
2803621018,71sleep00-21swapper/020:57:170
5315210034,16sleep00-21swapper/023:12:120
31995210017,68sleep132055-21processes22:52:261
29006210019,55sleep229089-21cut00:27:172
386229943,40sleep00-21swapper/021:22:290
2064429916,66sleep220114-21apt-get22:17:102
2480729822,18sleep00-21swapper/020:47:110
3053829713,66sleep330588-21/usr/sbin/munin19:22:173
219872969,12sleep20-21swapper/220:37:202
278812935,42sleep00-21swapper/022:38:360
2373929315,58sleep323826-21latency_hist00:11:593
1933129335,35sleep319302-21/usr/sbin/munin23:57:133
3021729223,17sleep00-21swapper/000:31:590
2417229222,17sleep00-21swapper/020:42:310
1425929225,43sleep314339-21fschecks_time21:57:163
1257329113,56sleep212094-21/usr/sbin/munin20:07:232
1155529115,59sleep211642-21basename21:47:262
921229016,55sleep29243-21awk21:41:572
921229016,55sleep29243-21awk21:41:572
44742908,43sleep04518-21ntpq19:42:210
44742908,43sleep04518-21ntpq19:42:210
3037528814,59sleep230409-21/usr/sbin/munin22:47:232
2762328828,42sleep127648-21munin-run20:56:571
216892889,16sleep30-21swapper/320:37:093
690428718,54sleep16983-21mailstats21:32:271
690428718,54sleep16983-21mailstats21:32:271
504228714,34sleep20-21swapper/221:27:192
2905428719,54sleep329079-21kernelversion19:17:213
2818828612,52sleep20-21swapper/220:57:192
1643528612,59sleep216468-21irqstats23:47:192
1643528612,59sleep216468-21irqstats23:47:192
995928514,58sleep19960-21cron19:57:571
2696928518,44sleep227007-21grep22:37:172
2505928520,43sleep10-21swapper/122:31:591
968828429,13sleep00-21swapper/023:26:580
2016228419,38sleep30-21swapper/322:16:593
2530428316,48sleep125298-21apt-get22:32:111
1820128312,55sleep218202-21ntp_states23:52:242
2995328228,38sleep00-21swapper/022:47:080
249972819,33sleep025113-21latency_hist19:07:000
181062810,9sleep20-21swapper/222:07:362
763728022,41sleep37682-21idleruntime-cro21:36:573
763728022,41sleep37682-21idleruntime-cro21:36:573
2558028023,24sleep30-21swapper/300:17:003
2167528012,53sleep10-21swapper/122:18:551
49212798,56sleep04966-21fschecks_count21:27:170
2784127912,52sleep327961-21timerwakeupswit19:12:283
2607527918,36sleep226095-21cron20:51:582
2406527922,25sleep20-21swapper/219:02:222
2403427920,25sleep10-21swapper/119:02:211
255127824,39sleep00-21swapper/019:37:130
641027715,42sleep15860-21/usr/sbin/munin19:48:041
471827716,29sleep20-21swapper/219:42:282
471827716,29sleep20-21swapper/219:42:282
3242127714,47sleep232568-21modprobe21:12:012
2861627718,44sleep328665-21unin-run00:26:573
263022770,8sleep10-21swapper/100:17:301
22092778,18sleep00-21swapper/023:02:150
1188527718,28sleep30-21swapper/320:07:003
1123427723,35sleep211264-21munin-run23:31:572
1123427723,35sleep211264-21munin-run23:31:572
82082765,46sleep20-21swapper/219:52:292
285142768,56sleep028528-21grep22:42:160
1603127612,47sleep216102-21/usr/sbin/munin20:17:292
1603127612,47sleep216102-21/usr/sbin/munin20:17:292
2451927510,50sleep324561-21systemd-run20:46:583
294827420,39sleep22984-21ntpq19:37:232
1466327437,17sleep30-21swapper/320:13:223
2334527319,37sleep123379-21cut20:42:161
150127314,18sleep00-21swapper/021:16:590
107027326,23sleep10-21swapper/119:32:171
73427222,35sleep00-21swapper/019:32:000
2899727214,28sleep20-21swapper/219:17:202
1680527217,24sleep10-21swapper/120:22:001
1680527217,24sleep10-21swapper/120:22:001
463627122,21sleep30-21swapper/321:27:003
3179627117,36sleep30-21swapper/322:52:203
1988627118,27sleep10-21swapper/123:57:281
3164327016,17sleep10-21swapper/119:27:001
281027023,24sleep30-21swapper/319:37:203
2089527029,20sleep10-21swapper/120:32:301
3132726920,32sleep331473-21sed22:52:003
2842626916,30sleep328422-21apt-get22:42:113
2697726931,25sleep327101-21sed20:52:273
2511126917,28sleep20-21swapper/222:32:002
2392426920,28sleep30-21swapper/319:02:193
560126815,25sleep10-21swapper/121:27:301
313402680,8sleep30-21swapper/300:32:303
3102226819,35sleep131023-21sshd22:49:111
2375026811,21sleep30-21swapper/322:27:133
196426822,26sleep10-21swapper/123:01:591
1795226812,15sleep025045-21cyclictest23:52:180
3171026722,29sleep131740-21munin-run00:36:571
31192675,18sleep20-21swapper/223:02:332
263472676,37sleep20-21swapper/222:34:102
182526716,24sleep10-21swapper/121:17:171
1595026712,22sleep30-21swapper/320:17:263
1595026712,22sleep30-21swapper/320:17:263
3236626621,18sleep20-21swapper/219:27:252
2678426624,21sleep20-21swapper/219:12:032
255182669,15sleep00-21swapper/022:32:190
2540926619,22sleep20-21swapper/200:16:592
756326522,25sleep30-21swapper/323:17:283
748526520,29sleep27562-21/usr/sbin/munin23:17:272
3132926511,35sleep131362-21latency21:07:191
1919526518,25sleep10-21swapper/122:12:211
1633626519,31sleep216438-21sed22:02:282
997526417,30sleep19970-21apt-get23:27:111
997526417,30sleep19970-21apt-get23:27:111
771726412,24sleep20-21swapper/221:37:002
771726412,24sleep20-21swapper/221:37:002
1991326418,18sleep30-21swapper/320:32:003
1880026416,21sleep20-21swapper/220:27:182
741326312,24sleep10-21swapper/123:17:261
415126318,30sleep20-21swapper/223:07:212
380826321,22sleep10-21swapper/119:41:581
325926310,39sleep13272-21acpi21:22:091
2939126314,34sleep229392-21sshd22:43:112
1441026318,21sleep30-21swapper/323:41:593
1643926212,19sleep30-21swapper/322:02:293
867426116,22sleep30-21swapper/321:37:303
867426116,22sleep30-21swapper/321:37:303
551726120,24sleep35713-21gpgv19:47:013
551726120,24sleep35713-21gpgv19:47:013
2980226118,26sleep30-21swapper/300:27:303
2443026111,35sleep324436-21ntp_states00:12:243
1259526117,27sleep312590-21apt-get21:52:113
1252726114,19sleep10-21swapper/120:07:221
1235726117,23sleep30-21swapper/321:51:593
1159326112,31sleep211602-21df_inode23:32:132
1159326112,31sleep211602-21df_inode23:32:132
647026014,29sleep26540-21sensors_temp19:48:052
2518826012,34sleep325231-21tr20:47:223
2246926016,23sleep20-21swapper/222:22:192
2069426025,22sleep30-21swapper/320:32:283
1820826019,18sleep30-21swapper/323:52:253
958925913,23sleep10-21swapper/121:42:161
75225915,25sleep3855-21perl22:57:173
2302325914,28sleep123090-21latency_hist20:41:591
1736225917,26sleep317368-21df_inode22:07:223
140925912,32sleep327-21ksoftirqd/319:32:233
901525814,25sleep29085-21df19:57:142
529425818,23sleep30-21swapper/323:12:113
2688525811,19sleep30-21swapper/322:37:143
2643725813,23sleep20-21swapper/220:52:142
2532225818,23sleep125408-21gpgv00:16:581
1745025815,25sleep317495-21ntp_states20:22:233
783125716,25sleep37865-21/usr/sbin/munin19:52:223
2981525712,31sleep329865-21memory21:02:213
263332575,11sleep10-21swapper/119:09:401
263332575,11sleep10-21swapper/119:09:401
2573525715,20sleep30-21swapper/319:07:283
2573525715,20sleep30-21swapper/319:07:283
2166725715,21sleep20-21swapper/200:02:292
1355825710,34sleep113566-21sh23:37:251
95222564,16sleep10-21swapper/123:24:491
3081125624,18sleep10-21swapper/121:06:591
2768725616,19sleep10-21swapper/119:12:261
2310125617,18sleep20-21swapper/200:07:262
1882425614,24sleep218821-21apt-get22:12:092
180925611,20sleep30-21swapper/321:17:153
1595825617,19sleep20-21swapper/223:46:592
1417125613,26sleep214165-21apt-get21:57:112
136925618,19sleep20-21swapper/219:32:222
593125515,22sleep30-21swapper/319:47:463
3033325517,17sleep20-21swapper/219:22:122
2534725525,17sleep20-21swapper/219:07:182
2534725525,17sleep20-21swapper/219:07:182
2110325517,22sleep221104-21sshd20:33:122
888425413,24sleep28890-21ntp_states23:22:242
743725413,24sleep17429-21apt-get19:52:121
61325412,17sleep30-21swapper/321:12:263
487025417,21sleep10-21swapper/123:10:111
2883925411,27sleep128850-21vmstat20:57:301
1638125414,25sleep116434-21/usr/sbin/munin23:47:181
1638125414,25sleep116434-21/usr/sbin/munin23:47:181
1277525412,26sleep10-21swapper/121:52:181
1065925419,19sleep20-21swapper/220:02:182
996925316,20sleep20-21swapper/221:42:252
922325312,22sleep39280-21irqstats19:57:193
3188225312,20sleep10-21swapper/119:27:121
2976725314,23sleep229472-21/usr/sbin/munin21:02:202
2712125311,28sleep227231-21latency_hist00:22:002
2154525319,15sleep10-21swapper/100:02:281
3151425213,23sleep231520-21ntp_states21:07:242
2877225213,23sleep128783-21apt-get19:17:111
2821625217,18sleep10-21swapper/122:42:011
225242529,19sleep30-21swapper/322:22:203
1511625214,21sleep10-21swapper/123:42:261
540025118,17sleep30-21swapper/321:27:283
2219625112,21sleep122191-21apt-get22:22:101
374725011,25sleep23781-21ps21:22:272
2545225013,20sleep30-21swapper/322:32:173
17412508,16sleep20-21swapper/221:17:142
128602509,16sleep30-21swapper/323:36:583
128602509,16sleep30-21swapper/323:36:583
104242495,16sleep30-21swapper/321:42:393
24252488,16sleep10-21swapper/123:02:201
2127724816,18sleep30-21swapper/300:02:223
383024710,19sleep30-21swapper/323:07:143
1984924712,16sleep10-21swapper/120:31:581
172182474,33sleep30-21swapper/323:48:583
172182474,33sleep30-21swapper/323:48:583
1464624710,21sleep30-21swapper/323:42:113
68624612,17sleep30-21swapper/319:31:593
1117124612,18sleep111257-21perl21:47:171
838124513,15sleep10-21swapper/121:37:241
838124513,15sleep10-21swapper/121:37:241
310622458,19sleep331129-21df21:07:133
2735624510,18sleep10-21swapper/100:22:111
2338024514,15sleep20-21swapper/220:42:172
139052457,17sleep20-21swapper/223:37:302
115142459,14sleep10-21swapper/123:32:081
115142459,14sleep10-21swapper/123:32:081
142472449,18sleep114253-21df_inode21:57:141
197832408,18sleep219789-21ntp_states23:57:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional