You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 11:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Thu Apr 09, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
292742682018,6788sleep029255-21kworker/0:120:38:080
16607216107,1561sleep131375-1kworker/1:2H21:41:551
13985214847,1465sleep1191-21jbd2/mmcblk2p2-19:51:451
14347293513,904sleep314385-21memory19:52:033
13359284125,803sleep113405-21ntpq21:32:051
13359284125,803sleep113405-21ntpq21:32:051
2328528295,810sleep218729-1kworker/2:1H23:46:562
942527006,682sleep19466-21ntpq23:02:041
3207241413,384sleep23246-21ntpq22:42:102
13490234250,239sleep113619-21latency_hist23:16:421
11664230521,232sleep111814-21unixbench_singl19:42:121
11664230521,232sleep111814-21unixbench_singl19:42:121
20571230012,267sleep320590-21fschecks_count20:11:583
19027229130,243sleep119069-21fschecks_count20:07:001
19027229130,243sleep119069-21fschecks_count20:07:001
2791522889,266sleep027959-21ntpq22:17:040
2791522889,266sleep027959-21ntpq22:17:040
4225228233,149sleep14271-21grep19:21:561
4009225356,131sleep10-21swapper/119:21:441
24468223349,145sleep124496-21fschecks_count22:06:591
11115222444,87sleep10-21swapper/121:26:401
26615221930,47sleep126753-21grep22:12:121
26615221930,47sleep126753-21grep22:12:121
508922167,196sleep15121-21cpu00:31:551
508922167,196sleep15121-21cpu00:31:551
24352221416,114sleep124492-21unixbench_singl20:22:141
16261221422,107sleep116465-21latency_hist21:41:421
8934219837,66sleep19097-21cut19:36:421
965218432,134sleep1981-21modprobe22:36:391
9390217838,79sleep10-21swapper/119:37:071
32343217826,132sleep032352-21modprobe00:16:390
17474217728,61sleep10-21swapper/120:01:591
7124217626,115sleep17160-21ntpq21:12:061
5703216716,41sleep10-21swapper/122:51:421
7071216614,103sleep17156-21apt-key19:31:411
22167216611,139sleep322256-21fschecks_time20:16:593
11201216526,104sleep111286-21timerandwakeup23:07:091
16426216412,136sleep016434-21awk19:58:430
3256721635,120sleep10-21swapper/119:07:551
8393216221,121sleep10-21swapper/121:16:581
4459216117,96sleep34523-21fschecks_time22:46:593
27409215911,130sleep127565-21grep20:32:121
17202215530,86sleep117252-21awk23:27:041
1099215412,91sleep10-21swapper/119:11:581
28242215151,50sleep10-21swapper/100:02:021
28242215151,50sleep10-21swapper/100:02:021
1298214810,35sleep10-21swapper/100:17:491
1298214810,35sleep10-21swapper/100:17:491
10459214612,118sleep010537-21find21:22:100
4185214310,48sleep10-21swapper/121:02:081
4185214310,48sleep10-21swapper/121:02:081
16078213913,110sleep216095-21idleruntime-cro21:41:392
30129213815,110sleep330225-21timerandwakeup00:07:093
30129213815,110sleep330225-21timerandwakeup00:07:093
17153213733,58sleep10-21swapper/120:01:421
31925213611,111sleep032028-21cat22:31:420
1971121328,86sleep119727-21latency_hist23:36:411
112421318,85sleep11246-21sed20:52:111
112421318,85sleep11246-21sed20:52:111
26987213019,35sleep00-21swapper/020:32:020
26392212931,67sleep226411-21apt-get23:56:502
30866212637,53sleep130885-21kernelversion22:27:021
20167212330,50sleep10-21swapper/121:52:091
5715212227,38sleep10-21swapper/121:07:091
22654212120,69sleep20-21swapper/222:01:432
7725212028,54sleep113441-21snmpd22:57:031
27778212029,39sleep131394-21cyclictest22:17:001
27778212029,39sleep131394-21cyclictest22:17:001
31478211934,50sleep131365-21/usr/sbin/munin20:46:581
31478211934,50sleep131365-21/usr/sbin/munin20:46:581
5297211814,27sleep30-21swapper/300:32:013
5297211814,27sleep30-21swapper/300:32:013
30371211729,73sleep10-21swapper/122:26:421
26537211710,93sleep026612-21find22:12:110
26537211710,93sleep026612-21find22:12:110
2262421175,16sleep00-21swapper/023:45:000
29693211613,87sleep029816-21timerwakeupswit22:22:120
2424211440,21sleep10-21swapper/120:57:031
31147211321,67sleep231248-21latency_hist20:46:422
31147211321,67sleep231248-21latency_hist20:46:422
29978211314,85sleep029998-21open_inodes00:07:060
29978211314,85sleep029998-21open_inodes00:07:060
30015211217,16sleep30-21swapper/319:01:553
6014211129,40sleep10-21swapper/122:52:061
27725211110,84sleep10-21swapper/100:01:411
251121117,41sleep12572-21mailstats00:22:091
251121117,41sleep12572-21mailstats00:22:091
15803211118,26sleep215822-21apt-get19:56:512
3484211010,62sleep13612-21wc22:42:171
24155211011,61sleep124302-21latency_hist22:06:431
29916210941,30sleep10-21swapper/120:41:571
29916210941,30sleep10-21swapper/120:41:571
26089210914,77sleep026147-40mandb23:56:400
29619210711,83sleep329688-21expr22:22:113
7763210613,73sleep07824-21idleruntime19:32:330
14956210615,34sleep10-21swapper/123:21:141
22574210531,42sleep10-21swapper/123:44:211
2185521058,85sleep121940-21switchtime21:57:091
5676210419,26sleep00-21swapper/022:51:410
20548210420,66sleep30-21swapper/323:37:083
22542210322,62sleep322550-21modprobe22:01:403
18339210315,70sleep218129-21/usr/sbin/munin21:47:012
30776210147,15sleep10-21swapper/100:11:391
30776210147,15sleep10-21swapper/100:11:391
15279210014,26sleep215346-21df23:21:562
15279210014,26sleep215346-21df23:21:562
641629915,61sleep26438-21latency_hist00:36:402
641629915,61sleep26438-21latency_hist00:36:402
215752995,12sleep10-21swapper/120:14:231
404929712,67sleep04122-21awk00:27:090
292232975,12sleep10-21swapper/120:37:291
2678629614,19sleep30-21swapper/323:57:033
881829527,46sleep38884-21modprobe19:36:403
1946429514,66sleep319484-21cat21:51:423
990829416,60sleep29933-21df_inode21:21:562
232692945,35sleep10-21swapper/120:20:531
1162629423,31sleep10-21swapper/121:27:001
1845629316,55sleep218471-21cut23:31:582
381229129,50sleep13839-21kernelversion00:27:021
3069429128,21sleep10-21swapper/119:02:101
1149329016,53sleep311554-21fschecks_time21:26:583
407728916,52sleep34100-21/usr/sbin/munin21:02:063
407728916,52sleep34100-21/usr/sbin/munin21:02:063
1657128918,54sleep316646-21timerandwakeup19:58:463
124672898,67sleep10-21swapper/119:46:521
124672898,67sleep10-21swapper/119:46:521
64512889,12sleep20-21swapper/219:27:102
643328819,51sleep36466-21awk21:11:393
2330928828,41sleep10-21swapper/123:47:161
124728720,43sleep2457-21/usr/sbin/munin20:52:122
124728720,43sleep2457-21/usr/sbin/munin20:52:122
1527828624,16sleep10-21swapper/123:21:551
1527828624,16sleep10-21swapper/123:21:551
2553628430,18sleep10-21swapper/120:27:031
1498828413,56sleep215021-21idleruntime-cro23:21:402
1948928328,32sleep00-21swapper/021:51:430
109028314,46sleep31174-21awk00:17:473
109028314,46sleep31174-21awk00:17:473
881528218,48sleep08894-21awk21:17:100
725828217,51sleep07373-21latency_hist22:56:420
2482528213,54sleep024836-21ntp_states22:07:060
235112820,8sleep30-21swapper/323:47:293
3264928118,50sleep232656-21modprobe20:51:392
3264928118,50sleep232656-21modprobe20:51:392
323552819,38sleep10-21swapper/100:16:401
2959828116,21sleep30-21swapper/320:41:433
2491028113,51sleep224914-21diskstats23:51:572
1609828113,54sleep316151-21rm21:41:403
1056928117,44sleep210577-21/usr/sbin/munin23:06:492
1915127817,24sleep30-21swapper/320:07:023
1915127817,24sleep30-21swapper/320:07:023
2536327723,34sleep10-21swapper/123:52:081
1428727715,40sleep213994-21/usr/sbin/munin19:52:022
1749827624,29sleep20-21swapper/220:02:002
61372757,55sleep00-21swapper/021:07:490
3199727522,25sleep30-21swapper/319:07:043
3106327521,32sleep231059-21apt-get00:11:522
1894127520,32sleep018934-21apt-get20:06:540
1894127520,32sleep018934-21apt-get20:06:540
125327515,38sleep31254-21/usr/sbin/munin22:36:503
976027427,25sleep30-21swapper/319:37:173
905027427,30sleep00-21swapper/023:01:550
82112740,7sleep30-21swapper/322:57:123
467727416,34sleep04683-21ntp_states19:22:070
1825327423,33sleep10-21swapper/121:47:001
1787527417,38sleep317940-21sendmail_mailst20:02:093
3266127312,16sleep30-21swapper/322:32:093
1979127315,38sleep019882-21fschecks_time21:51:590
2104027216,36sleep221165-21grep21:56:422
1493727215,32sleep00-21swapper/021:37:170
1322527220,32sleep213263-21wc21:32:022
1322527220,32sleep213263-21wc21:32:022
811327116,40sleep28210-21/usr/sbin/munin22:57:122
3024927124,23sleep00-21swapper/019:02:000
2149627112,46sleep221533-21cpu23:41:552
1994727118,13sleep10-21swapper/123:36:531
175927112,44sleep01821-21sshd22:37:070
2579927014,38sleep025914-21/usr/sbin/munin20:27:100
1707227022,25sleep00-21swapper/021:42:100
2804826912,15sleep00-21swapper/000:01:570
2804826912,15sleep00-21swapper/000:01:570
1907226913,40sleep219150-21/usr/sbin/munin20:07:012
1907226913,40sleep219150-21/usr/sbin/munin20:07:012
1185526911,44sleep011903-21systemd-run23:11:400
1100626920,28sleep00-21swapper/019:41:580
1100626920,28sleep00-21swapper/019:41:580
673726813,39sleep26834-21awk22:52:202
2424026817,28sleep00-21swapper/020:22:130
2146826818,29sleep00-21swapper/021:57:000
1883326815,28sleep30-21swapper/323:32:073
1682726810,19sleep00-21swapper/023:26:550
1531626815,35sleep015351-21apt-key19:56:410
1504026813,41sleep015012-21latency_hist23:21:420
985126722,24sleep20-21swapper/219:37:202
2125826725,21sleep00-21swapper/023:41:410
840026618,19sleep20-21swapper/221:16:592
410226620,28sleep00-21swapper/021:02:070
410226620,28sleep00-21swapper/021:02:070
3196726613,18sleep20-21swapper/219:07:032
2282426620,26sleep30-21swapper/323:46:413
330526518,32sleep03372-21latency_hist00:26:410
330526518,32sleep03372-21latency_hist00:26:410
2763326513,21sleep30-21swapper/322:16:583
2763326513,21sleep30-21swapper/322:16:583
1870226511,38sleep00-21swapper/020:06:420
16672658,11sleep00-21swapper/020:55:410
16672658,11sleep00-21swapper/020:55:410
1286626517,22sleep00-21swapper/019:47:030
1286626517,22sleep00-21swapper/019:47:030
3257626417,29sleep332767-21apt-config00:16:423
2836426414,33sleep228373-21/usr/sbin/munin20:36:572
2552926425,21sleep30-21swapper/323:52:103
2487826419,28sleep31-21systemd20:26:403
193662648,11sleep00-21swapper/023:33:130
718226316,31sleep37188-21ntp_states21:12:083
384026316,31sleep23911-21memory00:27:032
3009926323,20sleep20-21swapper/219:01:582
2565526316,32sleep225726-21tr20:27:082
245426311,37sleep32455-21ntp_states00:22:073
245426311,37sleep32455-21ntp_states00:22:073
1266426313,34sleep012667-21cron21:31:400
1131126317,19sleep20-21swapper/219:42:052
1131126317,19sleep20-21swapper/219:42:052
584026214,31sleep05849-21df_inode19:26:560
2411226221,26sleep223600-21/usr/sbin/munin20:22:112
2344226215,20sleep00-21swapper/022:02:090
232426218,11sleep10-21swapper/122:38:541
2175326215,29sleep021782-21munin-run20:16:400
701426124,19sleep30-21swapper/319:31:313
482626118,29sleep24833-21ntp_states22:47:072
342826114,31sleep03573-21sed21:01:430
2819726117,22sleep20-21swapper/222:17:102
2819726117,22sleep20-21swapper/222:17:102
2463026118,19sleep00-21swapper/023:51:430
224626116,24sleep20-21swapper/220:57:002
1271426122,17sleep20-21swapper/221:31:422
1255226116,30sleep212564-21fschecks_count19:46:572
1255226116,30sleep212564-21fschecks_count19:46:572
103602615,42sleep30-21swapper/323:06:423
946826018,25sleep29500-21ntp_states23:02:052
79202606,44sleep20-21swapper/221:15:462
333326015,21sleep30-21swapper/322:42:153
294526021,19sleep00-21swapper/019:17:020
214026017,27sleep32146-21diskstats20:56:573
1682526018,26sleep316820-21apt-get23:26:543
117126017,28sleep31180-21diskstats19:12:003
76425919,24sleep20-21swapper/200:17:382
76425919,24sleep20-21swapper/200:17:382
3107025915,29sleep331096-21pluginstate22:27:083
94212585,11sleep30-21swapper/321:19:473
787125817,24sleep20-21swapper/219:32:352
644325819,21sleep30-21swapper/322:52:163
2638325816,17sleep30-21swapper/322:12:063
2638325816,17sleep30-21swapper/322:12:063
2334425823,21sleep00-21swapper/020:21:400
1661125811,20sleep30-21swapper/321:41:573
1554625820,19sleep00-21swapper/023:22:020
1554625820,19sleep00-21swapper/023:22:020
1099825815,27sleep310897-21/usr/sbin/munin19:41:573
1099825815,27sleep310897-21/usr/sbin/munin19:41:573
1006925817,23sleep310126-21irqstats21:22:013
309025717,23sleep33127-21ntp_states19:17:053
2944025714,27sleep229484-21ntpq22:22:062
1283425714,26sleep312864-21awk19:47:023
1283425714,26sleep312864-21awk19:47:023
127825713,28sleep21274-21apt-get22:36:532
60592566,15sleep00-21swapper/000:32:590
60592566,15sleep00-21swapper/000:32:590
43025612,23sleep032366-21apt-get00:16:510
43025612,23sleep032366-21apt-get00:16:510
361625613,26sleep23610-21apt-get21:01:532
361625613,26sleep23610-21apt-get21:01:532
283425614,25sleep02828-21apt-get22:42:000
22632565,15sleep20-21swapper/219:15:272
1419825620,15sleep30-21swapper/323:17:083
933725510,30sleep08848-21apt-get19:36:450
749525514,24sleep07491-21apt-get22:56:540
3067325517,21sleep30-21swapper/320:42:113
3067325517,21sleep30-21swapper/320:42:113
284225518,23sleep20-21swapper/219:17:002
719225420,20sleep00-21swapper/021:12:090
3139725416,20sleep30-21swapper/320:46:563
3139725416,20sleep30-21swapper/320:46:563
3069025421,18sleep20-21swapper/222:26:592
2671525413,23sleep326711-21apt-get20:31:533
2305225415,24sleep323101-21irqstats22:02:003
1646725414,15sleep20-21swapper/221:41:432
155325415,17sleep00-21swapper/019:12:100
2517025313,23sleep325166-21apt-get20:26:523
196742535,36sleep30-21swapper/321:51:503
1737925311,24sleep217440-21/usr/sbin/munin23:27:092
1581125317,18sleep30-21swapper/323:22:093
1581125317,18sleep30-21swapper/323:22:093
1421525311,29sleep014256-21munin-run21:36:400
1421525311,29sleep014256-21munin-run21:36:400
422125216,20sleep24218-21apt-get19:21:532
354525214,21sleep33612-21df00:26:563
3270125214,17sleep00-21swapper/020:51:410
3270125214,17sleep00-21swapper/020:51:410
3051125213,18sleep20-21swapper/200:08:172
3051125213,18sleep20-21swapper/200:08:172
271232526,35sleep20-21swapper/222:15:442
271232526,35sleep20-21swapper/222:15:442
244225215,23sleep20-21swapper/200:22:062
244225215,23sleep20-21swapper/200:22:062
563325112,22sleep25639-21ntp_states21:07:062
3274125113,19sleep00-21swapper/022:32:100
3262525112,15sleep20-21swapper/222:32:082
299062517,16sleep00-21swapper/020:41:560
299062517,16sleep00-21swapper/020:41:560
2836025115,18sleep30-21swapper/300:02:043
2836025115,18sleep30-21swapper/300:02:043
2649125112,23sleep026500-21diskstats23:56:560
2449725113,20sleep224589-21perl22:07:002
2437825113,22sleep324375-21apt-get22:06:533
2001525118,17sleep20-21swapper/223:36:552
2806525012,22sleep228153-21fschecks_time00:01:592
2806525012,22sleep228153-21fschecks_time00:01:592
277132506,34sleep20-21swapper/220:33:182
2313225012,18sleep20-21swapper/220:19:082
1216425011,15sleep20-21swapper/223:11:542
552724916,16sleep30-21swapper/319:26:423
2283224911,21sleep222902-21df22:01:552
1326424912,21sleep313323-21memory21:32:033
1326424912,21sleep313323-21memory21:32:033
1057824910,15sleep30-21swapper/323:06:503
3204224810,22sleep032053-21ntp_states19:07:050
97324715,18sleep30-21swapper/320:52:093
97324715,18sleep30-21swapper/320:52:093
1259424710,21sleep012600-21ntp_states23:12:050
1221424714,17sleep30-21swapper/323:11:553
2146724613,17sleep20-21swapper/220:13:062
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional