You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 03:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Thu Apr 23, 2026 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
102792128029,12771sleep2191-21jbd2/mmcblk2p2-23:55:512
107542318860,3111sleep110247-21apt-get23:56:021
107542318860,3111sleep110247-21apt-get23:56:021
1972426857,663sleep219757-21unin-run21:00:512
22293238311,356sleep022337-21ntpq00:31:150
12886235893,195sleep20-21swapper/200:01:202
19215232226,283sleep219251-21ntpq00:21:152
4489228712,261sleep34514-21fschecks_count20:11:093
4075228068,188sleep24199-21latency_hist23:35:532
21336226614,153sleep221445-21latency_hist21:05:522
18018226415,161sleep20-21swapper/222:36:172
1773225752,190sleep21813-21proc_pri20:01:192
1773225752,190sleep21813-21proc_pri20:01:192
8221224530,201sleep28360-21grep20:21:222
11471223216,192sleep211609-21unixbench_multi23:56:282
11471223216,192sleep211609-21unixbench_multi23:56:282
10830223254,154sleep210846-21modprobe22:15:502
8667222057,114sleep21-21systemd23:50:502
8667222057,114sleep21-21systemd23:50:502
2078221747,85sleep21763-21/usr/sbin/munin21:46:142
22166220111,78sleep20-21swapper/221:06:172
2711219214,163sleep32727-21awk23:31:013
323721909,143sleep23279-21ntpq20:06:162
12336218715,153sleep012337-21cpu00:01:060
1074217817,146sleep01090-21sed20:00:520
1074217817,146sleep01090-21sed20:00:520
13761217211,54sleep20-21swapper/220:41:052
6607216924,29sleep20-21swapper/220:16:212
6607216924,29sleep20-21swapper/220:16:212
6488216916,132sleep16609-21timerwakeupswit23:41:211
6488216916,132sleep16609-21timerwakeupswit23:41:211
15327216826,27sleep20-21swapper/200:10:532
27409216555,96sleep227414-21modprobe23:06:022
1133121648,117sleep211393-21cat22:16:102
14140216113,133sleep014184-21ntpq20:41:150
16585216066,24sleep20-21swapper/222:31:212
3492215719,119sleep03609-21timerwakeupswit20:06:220
28016215610,132sleep328091-21switchtime23:06:473
2498215615,95sleep22549-21latency_hist23:30:522
9327215156,20sleep20-21swapper/222:10:532
27127215120,106sleep20-21swapper/223:05:532
21536214938,96sleep20-21swapper/219:21:132
21536214938,96sleep20-21swapper/219:21:132
4911214816,112sleep14978-21/usr/sbin/munin21:56:071
3222214713,121sleep13230-21needreboot20:06:151
23686214410,114sleep00-21swapper/021:11:150
23686214410,114sleep00-21swapper/021:11:150
21767214410,17sleep00-21swapper/021:06:020
7207214319,100sleep37357-21grep22:01:223
25392214248,22sleep20-21swapper/223:00:532
22443213924,85sleep10-21swapper/121:06:241
4399213513,109sleep34406-21forks23:36:103
4399213513,109sleep34406-21forks23:36:103
24127213545,78sleep224154-21awk19:30:502
15841213522,92sleep115846-21fschecks_count00:11:491
952213333,20sleep20-21swapper/223:25:522
952213333,20sleep20-21swapper/223:25:522
15777213347,33sleep215841-21ps20:46:182
2884121328,112sleep228849-21systemd-run19:45:522
2884121328,112sleep228849-21systemd-run19:45:522
32478212910,104sleep132522-21ntpq23:21:141
32478212910,104sleep132522-21ntpq23:21:141
2599821267,79sleep226066-21df19:36:072
2567621257,105sleep21-21systemd19:35:512
17269212569,40sleep217338-21df00:16:062
7777212316,89sleep27902-21latency_hist22:05:532
699212246,63sleep2710-21open_files21:41:182
30555212142,44sleep230624-21/usr/sbin/munin23:16:072
32010212030,76sleep232160-21latency_hist19:55:542
21776212029,58sleep20-21swapper/222:48:212
17509212012,86sleep217638-21timerwakeupswit20:51:222
13518212042,26sleep20-21swapper/222:21:272
21421211814,83sleep121566-21/usr/sbin/munin22:46:221
31228211716,43sleep20-21swapper/221:36:092
22006211718,73sleep00-21swapper/019:21:230
22006211718,73sleep00-21swapper/019:21:230
22466211620,75sleep321981-21apt-get22:51:003
19618211617,82sleep119624-21/usr/sbin/munin22:41:171
19618211617,82sleep119624-21/usr/sbin/munin22:41:171
17591211615,86sleep017608-21apt22:36:030
3263221157,92sleep20-21swapper/219:56:182
3263221157,92sleep20-21swapper/219:56:182
14223211514,83sleep314246-21/usr/sbin/munin20:41:183
18659211241,22sleep20-21swapper/219:11:242
2740021095,41sleep20-21swapper/221:21:312
23270210918,43sleep20-21swapper/222:51:242
4402210831,24sleep216044-21cyclictest20:11:042
3056721085,13sleep20-21swapper/221:32:252
19708210719,69sleep119723-21apt21:00:501
15028210626,21sleep20-21swapper/222:26:202
2043321059,58sleep220558-21sed19:16:212
25872210424,12sleep20-21swapper/221:16:462
19625210414,68sleep219636-21open_files22:41:182
19625210414,68sleep219636-21open_files22:41:182
28799210319,66sleep028820-21modprobe19:45:500
28799210319,66sleep028820-21modprobe19:45:500
20471210314,67sleep020556-21fschecks_time00:26:100
25718210112,74sleep325795-21fschecks_time23:01:103
24583210117,70sleep324670-21mailstats22:56:213
24179210112,68sleep024263-21wc19:30:520
17353210113,73sleep117395-21fschecks_count00:16:091
14331210116,25sleep10-21swapper/120:41:201
1194210120,61sleep11237-21cpu23:26:071
2943021007,81sleep229464-21ps23:11:182
716929916,66sleep07293-21latency_hist23:45:520
716929916,66sleep07293-21latency_hist23:45:520
1021529918,66sleep110276-21awk23:55:501
1701729714,68sleep317040-21latency_hist00:15:513
321529615,65sleep03276-21grep23:31:190
36229417,60sleep3472-21ls19:56:233
36229417,60sleep3472-21ls19:56:233
1788029413,66sleep317592-21/usr/sbin/munin22:36:143
885129316,17sleep00-21swapper/020:25:520
373429240,17sleep20-21swapper/221:51:152
2957229212,64sleep329620-21awk21:31:053
1677529221,44sleep016923-21grep22:31:230
1632229113,60sleep016389-21sshd19:06:250
841329017,53sleep07990-21/usr/sbin/munin22:06:150
214442908,13sleep20-21swapper/200:28:302
3005828818,51sleep030143-21cut21:31:180
2453228814,58sleep124541-21ntpq22:56:191
1351328829,39sleep113529-21latency_hist20:40:511
1351328829,39sleep113529-21latency_hist20:40:511
118982888,67sleep21-21systemd20:35:522
118982888,67sleep21-21systemd20:35:522
41732870,6sleep10-21swapper/121:51:211
2905528719,50sleep129091-21fschecks_count23:11:091
2126828713,56sleep121280-21vmstat00:26:231
2862728617,45sleep028706-21find21:26:220
2813328411,57sleep128259-21unixbench_multi23:06:491
1693728412,56sleep016946-21/usr/sbin/munin20:51:080
1630928433,32sleep30-21swapper/322:31:143
969928314,19sleep10-21swapper/120:26:221
732328323,27sleep10-21swapper/120:20:521
732328323,27sleep10-21swapper/120:20:521
2307328340,10sleep20-21swapper/200:34:322
1591028335,26sleep30-21swapper/319:05:563
838428216,47sleep38410-21netstat22:06:143
567428219,36sleep05705-21modprobe20:15:510
401428218,38sleep04172-21unixbench_multi21:51:210
292122825,12sleep20-21swapper/221:29:552
193192826,67sleep00-21swapper/019:14:040
1616828239,30sleep20-21swapper/200:11:582
973128120,45sleep39774-21/usr/sbin/munin22:11:113
269732818,59sleep02-21kthreadd19:37:140
2519728122,32sleep30-21swapper/321:16:113
2429628135,12sleep20-21swapper/221:11:542
2429628135,12sleep20-21swapper/221:11:542
133502816,44sleep20-21swapper/220:39:152
133502816,44sleep20-21swapper/220:39:152
2908728028,15sleep20-21swapper/219:46:052
103342807,40sleep20-21swapper/220:30:392
591827929,18sleep20-21swapper/221:57:292
2361527920,35sleep323681-21latency21:11:143
2361527920,35sleep323681-21latency21:11:143
150682796,62sleep00-21swapper/000:09:540
586027835,30sleep20-21swapper/223:41:052
586027835,30sleep20-21swapper/223:41:052
3073327824,28sleep00-21swapper/019:51:110
2233927811,14sleep10-21swapper/100:31:161
175427819,45sleep31816-21expr23:26:213
2616727716,23sleep10-21swapper/123:01:201
2375727722,36sleep023780-21munin-plugin-st22:55:500
2211227723,37sleep322257-21latency_hist22:50:523
1999927715,41sleep119973-21/usr/sbin/munin21:01:071
1514027720,40sleep315155-21apt-get00:10:503
77427517,27sleep00-21swapper/021:41:200
22252756,12sleep00-21swapper/020:01:250
22252756,12sleep00-21swapper/020:01:250
1748727517,28sleep10-21swapper/119:10:501
3185327417,43sleep331880-21latency_hist23:20:523
2144727420,33sleep321648-21apt-config21:05:533
1982527419,26sleep00-21swapper/019:16:060
1359127417,40sleep313721-21timerwakeupswit22:21:283
1045827436,22sleep010612-21apt-config23:55:530
1040027411,48sleep010505-21latency_hist20:30:530
3197527314,18sleep10-21swapper/119:55:531
269402735,12sleep10-21swapper/121:21:181
2322827316,43sleep323241-21needreboot19:26:153
2179227310,48sleep121856-21ps19:21:201
2179227310,48sleep121856-21ps19:21:201
1609827322,34sleep20-21swapper/219:06:032
2490227213,25sleep00-21swapper/021:16:000
21802719,18sleep00-21swapper/021:46:160
1945727113,15sleep10-21swapper/119:15:511
489027036,17sleep20-21swapper/223:36:212
489027036,17sleep20-21swapper/223:36:212
2772927011,39sleep027389-21/usr/sbin/munin23:06:400
148432709,13sleep00-21swapper/022:26:140
1091027019,23sleep00-21swapper/022:15:520
3039526924,23sleep10-21swapper/119:50:521
2045226913,42sleep320464-21fschecks_count00:26:093
175242696,16sleep217536-21munin-run19:10:512
1245926922,28sleep312551-21apt-config22:20:523
596526814,26sleep30-21swapper/320:16:023
596526814,26sleep30-21swapper/320:16:023
2802526814,38sleep313441-21snmpd21:26:043
1428326820,27sleep10-21swapper/100:06:161
450726720,33sleep14562-21if_eth023:36:121
450726720,33sleep14562-21if_eth023:36:121
234192676,12sleep10-21swapper/119:26:211
151926719,20sleep30-21swapper/320:01:123
151926719,20sleep30-21swapper/320:01:123
85862669,12sleep10-21swapper/122:06:201
467126630,20sleep30-21swapper/321:55:523
321002669,12sleep00-21swapper/023:21:050
321002669,12sleep00-21swapper/023:21:050
2674926616,24sleep00-21swapper/021:21:130
2606826614,32sleep30-21swapper/319:36:083
1662726617,27sleep10-21swapper/120:50:531
1191726614,36sleep112045-21uptime22:16:221
60726522,27sleep30-21swapper/321:41:143
3267526514,38sleep3301-21switchtime23:21:213
3267526514,38sleep3301-21switchtime23:21:213
299962658,13sleep10-21swapper/121:31:151
198362658,15sleep10-21swapper/100:22:431
1462926519,28sleep114627-21apt-get19:01:021
623726419,28sleep30-21swapper/323:41:153
623726419,28sleep30-21swapper/323:41:153
2904026420,25sleep30-21swapper/323:11:073
246252646,11sleep30-21swapper/319:31:123
2007726425,23sleep00-21swapper/000:25:490
1619526415,34sleep016218-21pluginstate00:12:000
155026420,22sleep10-21swapper/121:45:531
1384126414,32sleep113854-21/usr/sbin/munin22:21:301
2129526314,29sleep321313-21/usr/sbin/munin19:21:023
2129526314,29sleep321313-21/usr/sbin/munin19:21:023
416726221,17sleep30-21swapper/320:10:533
3269226217,26sleep10-21swapper/119:56:211
3269226217,26sleep10-21swapper/119:56:211
1591526214,27sleep00-21swapper/020:46:200
927426115,20sleep30-21swapper/320:26:123
626226123,20sleep30-21swapper/322:00:533
3188526113,21sleep00-21swapper/023:20:530
2954426115,30sleep029643-21cat23:11:200
2579326114,29sleep025942-21latency_hist19:35:530
256452617,10sleep10-21swapper/119:35:251
1824826114,30sleep30-21swapper/300:17:163
1736726120,18sleep10-21swapper/122:35:521
1295926114,27sleep30-21swapper/300:01:213
47772608,12sleep00-21swapper/023:36:190
47772608,12sleep00-21swapper/023:36:190
2701026014,27sleep327081-21sensors_temp21:21:203
218882605,46sleep00-21swapper/022:49:480
211612605,12sleep00-21swapper/021:04:370
1820626015,31sleep118347-21sed20:55:541
1683826016,31sleep316932-21cat19:06:453
1372326017,24sleep00-21swapper/022:21:290
310225918,17sleep10-21swapper/121:50:541
1761825918,22sleep10-21swapper/122:36:081
1571225911,25sleep315283-21/usr/sbin/munin20:46:143
116725912,32sleep01168-21munin-node23:26:020
1142025915,28sleep011512-21/usr/sbin/munin22:16:120
1072825915,30sleep110784-21tune2fs20:31:101
1072825915,30sleep110784-21tune2fs20:31:101
3124425819,19sleep30-21swapper/319:51:233
23072585,10sleep10-21swapper/120:02:271
23072585,10sleep10-21swapper/120:02:271
2179625819,24sleep321787-21apt-get21:06:053
2034825813,29sleep120432-21cat19:16:201
1976125810,23sleep00-21swapper/021:00:530
1753725817,21sleep00-21swapper/000:16:120
1280925815,21sleep30-21swapper/320:36:213
1280925815,21sleep30-21swapper/320:36:213
1033225816,18sleep30-21swapper/323:55:523
2248625717,24sleep122490-21apt-get22:51:021
2188925714,25sleep321908-21apt-get00:31:013
1987125719,17sleep30-21swapper/319:16:093
1693825714,30sleep017060-21cat19:06:460
1465325717,22sleep10-21swapper/122:26:111
1221625714,27sleep012209-21apt-get20:36:060
1221625714,27sleep012209-21apt-get20:36:060
886525615,21sleep10-21swapper/120:25:531
440125614,26sleep14398-21apt-get20:11:031
3055425613,25sleep130548-21apt-get23:16:061
2808925617,17sleep30-21swapper/319:41:213
2808925617,17sleep30-21swapper/319:41:213
2665725611,33sleep013441-21snmpd23:02:490
1868425615,16sleep30-21swapper/319:11:253
1448025613,24sleep314475-21apt-get22:26:053
956225514,24sleep19556-21apt-get22:11:051
2764525413,16sleep10-21swapper/119:41:111
2764525413,16sleep10-21swapper/119:41:111
1943325415,24sleep019491-21sed22:41:120
1943325415,24sleep019491-21sed22:41:120
760825313,19sleep30-21swapper/323:46:113
760825313,19sleep30-21swapper/323:46:113
647425311,22sleep00-21swapper/020:16:190
647425311,22sleep00-21swapper/020:16:190
3034525310,20sleep30-21swapper/319:50:503
1856725313,19sleep30-21swapper/320:56:123
178425314,22sleep31781-21apt-get21:46:033
1764025313,16sleep30-21swapper/320:51:233
740725212,23sleep17400-21apt-get23:46:051
740725212,23sleep17400-21apt-get23:46:051
3240825212,23sleep032452-21systemd-run21:40:510
314625213,23sleep13191-21ntp_states23:31:161
2334225215,22sleep123339-21apt-get21:11:041
2334225215,22sleep123339-21apt-get21:11:041
115972525,38sleep00-21swapper/020:31:580
115972525,38sleep00-21swapper/020:31:580
3245325114,19sleep10-21swapper/121:40:521
2415625117,19sleep00-21swapper/022:56:100
2001825113,22sleep320065-21users22:41:233
2001825113,22sleep320065-21users22:41:233
1103325112,23sleep011062-21irqstats23:56:170
1103325112,23sleep011062-21irqstats23:56:170
1075125112,22sleep310247-21apt-get23:56:003
1075125112,22sleep310247-21apt-get23:56:003
694825012,22sleep07010-21/usr/sbin/munin22:01:180
668725013,21sleep16735-21irqstats22:01:111
85422494,35sleep30-21swapper/320:22:323
3112324912,23sleep331142-21apt-get21:36:023
2909024914,19sleep00-21swapper/019:46:070
1881824911,22sleep318814-21apt-get00:21:043
123602498,16sleep10-21swapper/100:01:071
1117524911,23sleep311245-21sensors_temp20:31:203
1117524911,23sleep311245-21sensors_temp20:31:203
536624811,21sleep05372-21ntp_states21:56:180
2725224810,17sleep10-21swapper/119:40:511
740524510,18sleep00-21swapper/023:46:040
740524510,18sleep00-21swapper/023:46:040
2492924511,17sleep10-21swapper/121:16:051
2802824210,17sleep10-21swapper/121:26:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional