You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 05:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Mon Mar 23, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
218112148910,1463sleep021804-1kworker/0:1H21:27:510
268362142531,1377sleep326872-21/usr/sbin/munin21:43:003
268362142531,1377sleep326872-21/usr/sbin/munin21:43:003
22625211877,1167sleep2191-21jbd2/mmcblk2p2-19:47:462
20763281114,783sleep020830-21df23:07:540
20763281114,783sleep020830-21df23:07:540
29345265216,621sleep129360-21/usr/sbin/munin20:08:081
9243252314,494sleep19244-21sshd22:28:351
3021624270,12sleep10-21swapper/123:37:501
3021624270,12sleep10-21swapper/123:37:501
3803239722,254sleep33870-21kworker/u8:223:57:393
2422230413,275sleep22447-21cat20:27:422
138362297132,26sleep30-21swapper/321:03:043
18268229614,266sleep018354-21latency_hist21:17:420
18268229614,266sleep018354-21latency_hist21:17:420
275192292124,24sleep30-21swapper/320:03:013
8658229018,256sleep38803-21latency_hist20:47:423
214512280127,23sleep30-21swapper/321:27:423
28769226113,96sleep30-21swapper/323:32:523
28769226113,96sleep30-21swapper/323:32:523
311842255113,31sleep331308-21latency_hist21:57:413
30521224311,213sleep330409-21/usr/sbin/munin20:13:023
30521224311,213sleep330409-21/usr/sbin/munin20:13:023
10899223024,18sleep30-21swapper/300:18:023
18251222316,190sleep118334-21/usr/sbin/munin22:58:101
3708221120,52sleep33723-21fschecks_count22:12:593
2633220924,40sleep30-21swapper/320:27:523
820220716,173sleep3823-21cron20:22:393
25289220716,164sleep325370-21fschecks_time21:37:593
10516220212,170sleep310553-21fschecks_count20:52:583
28681219811,170sleep028829-21cat20:07:420
9297218815,157sleep39317-21kernelversion00:13:143
5767218813,155sleep05818-21fschecks_count00:02:570
1658321850,34sleep216587-21fschecks_time19:27:592
2349218016,148sleep32501-21latency_hist23:52:423
19611217716,126sleep319644-21sed23:03:053
19611217716,126sleep319644-21sed23:03:053
18025217611,146sleep317633-21/usr/sbin/munin22:58:043
9087217312,134sleep30-21swapper/320:48:023
10205217224,27sleep30-21swapper/319:08:073
21987217015,141sleep322006-21fschecks_time21:28:023
17746216910,144sleep317828-21latency_hist19:32:413
27080216419,114sleep30-21swapper/323:28:003
301215910,134sleep3460-21sed22:02:423
24345215811,128sleep324478-21awk21:33:233
45832157115,18sleep30-21swapper/320:33:043
45832157115,18sleep30-21swapper/320:33:043
23232215610,15sleep30-21swapper/319:48:123
2014321499,16sleep00-21swapper/021:22:590
16828214914,120sleep316866-21date00:37:403
6921214812,98sleep36983-21idleruntime22:23:003
14965214615,114sleep315010-21gpgv00:32:403
12010214544,13sleep30-21swapper/320:57:553
491214412,96sleep30-21swapper/322:02:503
7420214311,12sleep10-21swapper/120:42:591
12356214316,106sleep30-21swapper/319:13:123
29606214015,65sleep329658-21sort21:52:413
29606214015,65sleep329658-21sort21:52:413
25516213915,105sleep225525-21diskstats23:22:582
16669213912,81sleep30-21swapper/321:12:403
16669213912,81sleep30-21swapper/321:12:403
26247213814,109sleep30-21swapper/319:58:093
11187213817,105sleep211323-21unixbench_singl19:12:422
16349213513,19sleep30-21swapper/322:53:003
4505213317,104sleep3357-21ntpd23:58:073
1239021329,107sleep30-21swapper/300:23:033
30194213012,61sleep30-21swapper/320:12:423
1187213010,88sleep30-21swapper/320:22:593
22402212917,55sleep30-21swapper/323:12:583
800212783,27sleep30-21swapper/323:47:423
21681212727,19sleep30-21swapper/319:43:113
2669921238,15sleep30-21swapper/323:27:393
1166421218,14sleep30-21swapper/322:38:023
5680211911,50sleep30-21swapper/300:02:523
17183211910,50sleep30-21swapper/321:13:013
17183211910,50sleep30-21swapper/321:13:013
8423211812,51sleep28511-21fschecks_time22:28:002
10184211816,86sleep210201-21cat20:52:412
24205211610,49sleep30-21swapper/319:52:563
1981321169,48sleep30-21swapper/321:22:413
17744211617,15sleep20-21swapper/222:57:592
8911211211,85sleep18508-21apt-get00:12:431
1388821127,47sleep30-21swapper/322:43:153
1084211112,17sleep30-21swapper/323:47:583
31665211051,41sleep331765-21latency_hist23:42:413
808821096,44sleep30-21swapper/300:08:133
7805210910,85sleep27864-21sendmail_mailqu20:43:092
12417210811,16sleep20-21swapper/220:58:062
6419210714,71sleep16571-21unixbench_singl00:03:101
1057621076,43sleep30-21swapper/322:33:093
18058210511,16sleep30-21swapper/319:32:583
12249210418,66sleep012266-21fschecks_count00:23:000
15533210310,82sleep0367-21sshd22:49:210
6182210210,75sleep26219-21ntp_kernel_pll_20:38:082
6182210210,75sleep26219-21ntp_kernel_pll_20:38:082
529421029,19sleep35183-21/usr/sbin/munin22:18:003
26227210211,78sleep126237-21ntp_states19:58:071
2170621027,41sleep30-21swapper/323:08:383
2170621027,41sleep30-21swapper/323:08:383
941221015,41sleep30-21swapper/322:30:433
1599821016,42sleep30-21swapper/319:25:433
14977210120,63sleep214995-21fschecks_count19:23:002
912329919,65sleep09144-21fschecks_count00:13:110
769429911,71sleep17696-21sshd22:23:351
3151729815,61sleep10-21swapper/121:58:001
1540229831,42sleep215357-21apt-get21:08:192
1540229831,42sleep215357-21apt-get21:08:192
553929713,71sleep05569-21latency_hist20:37:420
553929713,71sleep05569-21latency_hist20:37:420
265229710,24sleep30-21swapper/322:08:113
362129622,41sleep00-21swapper/022:12:520
2894229520,53sleep228984-21fschecks_time23:33:192
2894229520,53sleep228984-21fschecks_time23:33:192
2241929515,58sleep222474-21latency_hist19:47:422
2082129556,22sleep10-21swapper/119:42:411
213812946,14sleep00-21swapper/019:43:040
1014329416,62sleep010172-21idleruntime-cro20:52:390
2966429316,62sleep029808-21modprobe21:52:420
2003929315,62sleep220101-21wc19:38:102
174132930,30sleep017411-21systemd-run22:57:400
1818629218,55sleep118190-21cron21:17:391
1818629218,55sleep118190-21cron21:17:391
3238629121,54sleep132430-21ntpq20:18:051
947229022,50sleep09525-21systemd-run19:07:400
947229022,50sleep09525-21systemd-run19:07:400
2010228913,57sleep30-21swapper/319:38:113
320682887,14sleep20-21swapper/223:43:002
2994328820,51sleep230049-21tune2fs21:53:002
1169328828,42sleep011728-21idleruntime-cro20:57:400
282828713,58sleep02882-21basename20:28:010
250732855,24sleep30-21swapper/323:21:383
1850728515,35sleep20-21swapper/221:17:562
154732856,25sleep30-21swapper/322:48:363
68902845,25sleep30-21swapper/320:40:433
68902845,25sleep30-21swapper/320:40:433
834128323,24sleep10-21swapper/119:03:051
834128323,24sleep10-21swapper/119:03:051
1983228312,55sleep019900-21ls19:38:030
818128214,53sleep319808-1kworker/3:2H19:02:513
818128214,53sleep319808-1kworker/3:2H19:02:513
142642827,43sleep30-21swapper/319:18:213
1111028222,36sleep111170-21munin-node19:12:411
2044127917,45sleep120479-21/usr/sbin/munin21:23:041
1144127914,40sleep010589-21/usr/sbin/munin00:18:110
2711927815,28sleep00-21swapper/021:43:050
2711927815,28sleep00-21swapper/021:43:050
1416527815,20sleep20-21swapper/200:28:072
1037727816,16sleep20-21swapper/200:17:412
3052027714,50sleep130645-21sed21:53:111
2733827714,47sleep227339-21cpu20:02:562
2488627710,16sleep00-21swapper/021:37:400
955727625,29sleep20-21swapper/219:07:422
955727625,29sleep20-21swapper/219:07:422
486227616,38sleep25002-21head23:58:142
2554227617,34sleep00-21swapper/019:57:410
1347427615,46sleep113553-21date00:27:411
2842427513,45sleep128644-21sed23:32:421
62227414,49sleep00-21swapper/023:46:340
544427414,39sleep00-21swapper/000:02:410
2922427412,49sleep213441-21snmpd21:48:242
2922427412,49sleep213441-21snmpd21:48:242
2347227415,37sleep223473-21gpgv21:33:102
230827410,17sleep20-21swapper/223:52:412
230827410,17sleep20-21swapper/223:52:412
1332727322,24sleep10-21swapper/121:02:411
1091727314,45sleep110999-21mailstats20:53:091
2302527213,45sleep123049-21netstat19:48:061
191882727,36sleep20-21swapper/223:02:522
191882727,36sleep20-21swapper/223:02:522
1046427220,33sleep210574-21ls22:33:082
2770327118,21sleep20-21swapper/223:28:152
2192627124,27sleep10-21swapper/121:28:001
1511727122,29sleep115324-21gpgv00:32:421
1303627115,24sleep20-21swapper/222:42:572
776827014,24sleep00-21swapper/000:08:100
315827011,39sleep13234-21/usr/sbin/munin20:28:101
2331327021,28sleep00-21swapper/019:48:130
142127020,27sleep10-21swapper/122:03:121
1215727011,38sleep212203-21ntp_offset19:13:072
746526918,30sleep27604-21/usr/sbin/munin22:23:112
3035926921,34sleep130394-21cpu23:37:551
204926910,43sleep02068-21apt-get22:07:510
1271226916,22sleep00-21swapper/020:58:120
794226817,26sleep20-21swapper/200:08:122
422126817,37sleep14226-21entropy23:58:011
1394126814,36sleep214019-21/usr/sbin/munin19:18:112
81462676,15sleep10-21swapper/100:08:151
74132677,45sleep00-21swapper/020:42:580
2623826712,40sleep226245-21sh19:58:082
3016526619,27sleep00-21swapper/023:37:500
3016526619,27sleep00-21swapper/023:37:500
1465726619,23sleep10-21swapper/119:22:421
1424426620,26sleep00-21swapper/022:47:400
588126520,21sleep10-21swapper/120:38:031
588126520,21sleep10-21swapper/120:38:031
1622526523,23sleep10-21swapper/119:27:431
1501726512,37sleep015114-21gpgv00:32:410
1310126512,33sleep2367-21sshd00:23:342
62126413,31sleep0723-21fschecks_time22:02:590
388426419,24sleep20-21swapper/222:13:022
263782647,17sleep00-21swapper/023:23:320
2239126411,26sleep20-21swapper/223:12:572
2136426416,35sleep221444-21timerandwakeup23:08:082
2136426416,35sleep221444-21timerandwakeup23:08:082
19392645,45sleep00-21swapper/023:48:350
19392645,45sleep00-21swapper/023:48:350
1816426419,22sleep00-21swapper/022:58:090
1356826416,32sleep013561-21apt-get21:02:560
2880026314,29sleep00-21swapper/021:48:100
2880026314,29sleep00-21swapper/021:48:100
2719926320,27sleep226725-21/usr/sbin/munin21:43:072
2719926320,27sleep226725-21/usr/sbin/munin21:43:072
1910926322,26sleep10-21swapper/121:18:111
1686826318,21sleep00-21swapper/000:37:410
134472639,13sleep10-21swapper/122:43:091
1079726319,27sleep01-21systemd22:33:350
651826214,27sleep10-21swapper/122:22:421
2405426212,20sleep10-21swapper/121:33:171
2256426217,24sleep10-21swapper/123:13:001
1846826212,19sleep20-21swapper/219:33:092
1741426215,23sleep20-21swapper/221:13:082
1741426215,23sleep20-21swapper/221:13:082
429026118,24sleep10-21swapper/120:32:581
429026118,24sleep10-21swapper/120:32:581
2537326111,36sleep025437-21cat21:38:000
1744026116,21sleep10-21swapper/122:57:421
11962618,38sleep00-21swapper/020:23:000
337826020,19sleep10-21swapper/122:12:411
3199126014,28sleep132066-21fschecks_time23:42:591
1709226014,24sleep10-21swapper/119:28:121
42072596,12sleep20-21swapper/220:32:552
42072596,12sleep20-21swapper/220:32:552
2921325911,32sleep128277-21/usr/sbin/munin21:48:151
2921325911,32sleep128277-21/usr/sbin/munin21:48:151
2547425915,29sleep225504-21latency21:38:022
1978725913,24sleep00-21swapper/023:03:100
1978725913,24sleep00-21swapper/023:03:100
1903925915,19sleep00-21swapper/021:18:100
1806925915,27sleep018076-21/usr/sbin/munin19:32:590
38922588,33sleep13989-21cat23:57:411
322325812,19sleep00-21swapper/023:53:120
3044125813,28sleep130511-21df20:13:001
3044125813,28sleep130511-21df20:13:001
2202825815,28sleep11-21systemd23:12:391
2202825815,28sleep11-21systemd23:12:391
2062325817,26sleep220697-21cat21:23:092
1584425817,19sleep10-21swapper/122:52:411
1496125815,23sleep10-21swapper/121:07:421
1088325814,20sleep20-21swapper/200:18:012
107125815,19sleep20-21swapper/222:03:092
1039225812,27sleep110463-21/usr/sbin/munin22:33:071
896425717,21sleep10-21swapper/120:48:001
518225713,28sleep25184-21/usr/sbin/munin22:17:502
3201925717,19sleep00-21swapper/020:17:560
3189025713,25sleep20-21swapper/221:58:092
272842577,20sleep10-21swapper/121:43:101
272842577,20sleep10-21swapper/121:43:101
187262577,12sleep10-21swapper/119:33:121
1650825714,17sleep20-21swapper/222:53:032
589325616,19sleep00-21swapper/022:18:130
2447225613,25sleep10-21swapper/119:53:021
2428925618,19sleep20-21swapper/219:52:592
2409025614,26sleep224096-21ntp_states21:33:182
1533825614,26sleep114844-21apt-get21:07:501
1533825614,26sleep114844-21apt-get21:07:501
1310525613,24sleep10-21swapper/119:17:421
79392556,38sleep00-21swapper/022:26:400
2895325517,22sleep228962-21df_inode20:07:572
2831525514,24sleep028422-21apt-config23:32:410
2389125512,25sleep223885-21apt-get23:17:532
157225520,17sleep10-21swapper/120:23:091
3209125413,23sleep232097-21df_inode20:17:582
2760825416,18sleep10-21swapper/120:03:031
254425415,23sleep22540-21apt-get23:52:532
2111125416,24sleep121146-21grep19:42:571
159022547,32sleep015951-21processes21:08:580
159022547,32sleep015951-21processes21:08:580
1546425414,23sleep114979-21apt-get00:32:501
143572545,39sleep00-21swapper/019:19:310
1176025411,24sleep011753-21apt-get19:12:570
2629225312,21sleep10-21swapper/123:23:131
1966625313,25sleep119672-21ntp_states23:03:071
1966625313,25sleep119672-21ntp_states23:03:071
1141325312,25sleep111432-21apt-get22:37:521
3202625217,20sleep00-21swapper/021:58:110
3035825211,24sleep030321-21apt-get23:37:540
292725215,17sleep20-21swapper/220:28:032
2520025215,22sleep10-21swapper/123:22:411
2408025210,24sleep00-21swapper/023:17:590
1274525216,17sleep20-21swapper/222:42:412
3054025112,23sleep20-21swapper/223:38:002
2578025111,22sleep025775-21apt-get19:57:530
1689025111,15sleep00-21swapper/019:28:060
1592225115,22sleep215965-21ntpq00:33:282
1379825116,21sleep20-21swapper/221:03:032
890325017,17sleep00-21swapper/022:28:100
412525017,17sleep10-21swapper/122:13:091
1112425010,23sleep111217-21mailstats00:18:081
694324912,21sleep26972-21munin-run00:07:392
102562499,18sleep00-21swapper/019:08:080
2896924811,23sleep029029-21fschecks_time20:07:590
2651124811,16sleep10-21swapper/121:42:421
159052475,30sleep00-21swapper/019:24:310
1177324710,22sleep211849-21sed20:57:422
341024610,17sleep20-21swapper/222:12:422
1143824612,18sleep00-21swapper/022:37:550
99624511,18sleep1993-21apt-get23:47:521
99624511,18sleep1993-21apt-get23:47:521
3043724513,17sleep20-21swapper/220:12:572
3043724513,17sleep20-21swapper/220:12:572
1460424513,17sleep20-21swapper/222:47:582
2230524211,16sleep00-21swapper/023:12:510
305262417,13sleep00-21swapper/020:13:030
305262417,13sleep00-21swapper/020:13:030
605824011,14sleep00-21swapper/020:38:060
605824011,14sleep00-21swapper/020:38:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional