You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-17 - 09:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Apr 17, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7148212058,1184sleep12314-21kworker/1:123:58:531
15798258616,552sleep315834-21/usr/sbin/munin19:16:313
15798258616,552sleep315834-21/usr/sbin/munin19:16:313
331424237,404sleep23356-21ntpq23:46:392
1820523347,293sleep213441-21snmpd19:22:112
19179229830,200sleep219206-21awk19:26:362
22621226415,175sleep222777-21unixbench_multi19:36:452
32133226147,170sleep232180-21ntp_kernel_pll_21:51:362
299212258123,120sleep2292-21in:imuxsock21:46:112
299212258123,120sleep2292-21in:imuxsock21:46:112
6809225612,55sleep20-21swapper/222:16:112
170212254149,52sleep20-21swapper/219:21:142
170212254149,52sleep20-21swapper/219:21:142
24855224615,214sleep324899-21ntpq23:11:363
2666622429,57sleep20-21swapper/219:51:312
14666223916,204sleep114782-21latency_hist22:41:141
16037223117,198sleep016054-21uniq21:01:320
3177422307,211sleep025715-21kworker/0:223:33:450
17721222438,121sleep20-21swapper/221:06:222
5078221814,187sleep15152-21find23:51:421
9793221113,182sleep09847-21fschecks_count20:41:280
25745219553,71sleep20-21swapper/223:16:112
19887219357,19sleep20-21swapper/221:11:432
32284219215,161sleep032278-21apt-get23:36:270
32284219215,161sleep032278-21apt-get23:36:270
143102189101,63sleep20-21swapper/200:21:232
14821218620,147sleep014883-21/usr/sbin/munin00:21:410
4554218412,158sleep14622-21sed22:06:421
25116217644,71sleep20-21swapper/219:46:302
28858217217,141sleep328902-21if_err_eth021:41:323
28858217217,141sleep328902-21if_err_eth021:41:323
433216415,86sleep2490-21fschecks_count20:11:302
16325216329,60sleep216373-21ntpq21:01:382
15695216213,132sleep115621-21latency_hist00:26:151
14853215930,71sleep214887-21proc_pri20:56:372
302215310,127sleep2373-21mailstats23:36:422
302215310,127sleep2373-21mailstats23:36:422
27321215332,57sleep2291-21systemd-logind23:21:122
12528215133,103sleep212552-21modprobe20:51:122
23757214746,58sleep20-21swapper/219:41:382
21695214714,116sleep321748-21cat19:36:143
669221459,120sleep26802-21timerwakeupswit23:56:402
1718213712,78sleep2858-21/usr/sbin/munin21:56:452
1718213712,78sleep2858-21/usr/sbin/munin21:56:452
30781213619,18sleep20-21swapper/223:31:292
667221348,74sleep26707-21fschecks_count20:31:332
3458213317,97sleep00-21swapper/020:21:210
12584213211,81sleep212737-21sed00:16:142
30364213013,102sleep030428-21idleruntime21:46:340
30364213013,102sleep030428-21idleruntime21:46:340
21536213043,55sleep20-21swapper/221:16:442
874321285,113sleep00-21swapper/000:04:290
32498212813,97sleep132636-21unixbench_singl21:51:441
30396212825,80sleep030528-21unixbench_multi20:01:430
20757212663,46sleep20-21swapper/219:31:362
9916212565,17sleep20-21swapper/220:41:312
1786221257,79sleep20-21swapper/200:31:372
1786221257,79sleep20-21swapper/200:31:372
10746212522,87sleep010767-21fschecks_time19:01:460
28424212415,59sleep20-21swapper/221:41:152
13370212419,91sleep313451-21switchtime20:51:413
18887212315,18sleep00-21swapper/019:26:290
1223121237,59sleep20-21swapper/222:31:382
12874211946,32sleep212960-21cut00:16:302
313221175,41sleep20-21swapper/220:20:222
1443621166,11sleep20-21swapper/222:38:522
2502221155,11sleep20-21swapper/221:28:232
2502221155,11sleep20-21swapper/221:28:232
20652211526,27sleep10-21swapper/121:16:141
11898211544,48sleep311918-21ls19:06:133
1119121145,42sleep20-21swapper/222:27:302
1409821138,93sleep014067-21latency_hist00:21:130
852921129,90sleep08544-21netstat20:36:360
27613211243,13sleep20-21swapper/223:21:282
7200211112,18sleep30-21swapper/323:59:313
2229721115,43sleep20-21swapper/223:04:532
12165211160,15sleep20-21swapper/219:06:242
13480211012,80sleep013567-21fw_packets22:36:320
17809210711,80sleep117958-21latency_hist22:51:151
17554210511,75sleep317232-21apt-get21:06:153
29470210413,70sleep30-21swapper/320:01:133
14806210320,59sleep314949-21grep19:11:423
15972210119,66sleep115979-21/usr/sbin/munin00:26:311
5011210020,62sleep20-21swapper/220:26:242
16256210023,18sleep20-21swapper/200:26:362
348129944,15sleep20-21swapper/220:21:232
146229911,18sleep20-21swapper/223:41:302
1080329910,71sleep211013-21gpgv00:11:132
815929820,42sleep10-21swapper/120:36:251
2045529822,56sleep120562-21fschecks_time19:31:311
863229731,54sleep20-21swapper/200:03:062
1645929711,71sleep316529-21df22:46:293
396029620,21sleep20-21swapper/222:06:222
2373729630,17sleep223746-21munin-run21:26:132
2373729630,17sleep223746-21munin-run21:26:132
3125129439,36sleep20-21swapper/220:06:212
2829229411,60sleep127586-21/usr/sbin/munin23:21:431
2301529437,39sleep20-21swapper/223:06:382
86752938,71sleep28711-21cpu22:21:282
377029311,64sleep20-21swapper/222:06:142
2830229310,19sleep20-21swapper/219:56:322
2220129311,65sleep322204-21sshd23:03:413
1469829312,63sleep314708-21/usr/sbin/munin00:21:363
824629215,62sleep38256-21forks20:36:313
75182928,74sleep313441-21snmpd20:32:033
1799429229,49sleep217996-21apt-get22:51:242
1781429213,66sleep117860-21ntp_kernel_pll_00:31:361
1781429213,66sleep117860-21ntp_kernel_pll_00:31:361
2821729123,46sleep028291-21awk23:21:420
2084729123,51sleep020861-21sh23:01:110
1985129136,33sleep319879-21kernelversion22:56:343
3194329022,48sleep132100-21tail20:06:411
2736529012,64sleep327396-21latency_hist23:21:133
2089929022,15sleep20-21swapper/223:01:132
158829018,53sleep11716-21unixbench_multi21:56:441
158829018,53sleep11716-21unixbench_multi21:56:441
1527429022,14sleep20-21swapper/222:41:352
1420329036,14sleep20-21swapper/219:11:292
809028929,49sleep20-21swapper/222:17:252
2665928913,57sleep10-21swapper/119:51:301
271228818,53sleep02717-21kernelversion22:01:360
1850628819,46sleep018635-21unixbench_multi21:07:370
1291228717,24sleep10-21swapper/120:51:311
2971128618,14sleep20-21swapper/220:01:242
2784628613,53sleep327882-21/usr/sbin/munin23:21:333
2098728615,58sleep321090-21fschecks_time21:16:333
1716728518,19sleep20-21swapper/222:46:442
2138728411,58sleep021388-21sshd19:33:160
549828331,19sleep20-21swapper/223:53:132
317628311,57sleep33199-21irqstats23:46:363
2912928331,36sleep20-21swapper/223:26:202
2598828333,13sleep20-21swapper/221:31:382
2550628318,31sleep30-21swapper/321:31:223
2290428317,39sleep322412-21apt-get23:06:233
1608328331,33sleep00-21swapper/019:16:360
1608328331,33sleep00-21swapper/019:16:360
3199628215,53sleep332032-21irqstats21:51:333
1786728215,42sleep017974-21cat19:21:410
872628111,21sleep28794-21cat20:36:422
3752810,6sleep30-21swapper/323:36:423
3752810,6sleep30-21swapper/323:36:423
2951528126,26sleep00-21swapper/023:26:350
2646428016,49sleep026507-21ntpq23:16:380
258352787,43sleep30-21swapper/319:46:443
162327821,26sleep00-21swapper/020:16:120
1025827718,47sleep310325-21wc20:41:403
2932527611,56sleep00-21swapper/019:59:530
2469227624,39sleep30-21swapper/319:45:353
244322766,53sleep00-21swapper/021:26:390
244322766,53sleep00-21swapper/021:26:390
1646327620,30sleep00-21swapper/000:26:420
1246727610,17sleep10-21swapper/100:16:091
2590427516,35sleep00-21swapper/021:31:360
1360927512,43sleep30-21swapper/300:16:443
1116227522,33sleep30-21swapper/300:11:223
2747327416,25sleep10-21swapper/121:36:371
1181427428,26sleep10-21swapper/120:46:431
898027330,20sleep30-21swapper/300:06:143
329527311,49sleep13313-21netstat23:46:381
306642737,18sleep230676-21ntpq21:46:402
306642737,18sleep230676-21ntpq21:46:402
2378327324,20sleep00-21swapper/021:26:150
2161627310,26sleep00-21swapper/023:01:400
2634227218,34sleep126440-21date19:51:141
1804927230,23sleep30-21swapper/300:31:423
1804927230,23sleep30-21swapper/300:31:423
436727118,26sleep00-21swapper/022:06:370
3031027115,39sleep330391-21threads20:01:423
127102719,50sleep013441-21snmpd22:32:000
1218127121,25sleep10-21swapper/122:31:371
83532708,17sleep00-21swapper/022:20:460
740327020,30sleep00-21swapper/000:01:140
253912709,13sleep00-21swapper/023:11:450
2359927014,36sleep023718-21timerwakeupswit23:06:530
2331127010,14sleep00-21swapper/019:41:150
1305927013,43sleep113090-21latency00:16:331
733326922,27sleep30-21swapper/322:16:333
2146626916,24sleep10-21swapper/121:16:431
1517526915,36sleep015190-21/usr/sbin/munin22:41:330
1277526915,31sleep312143-21/usr/sbin/munin19:06:423
887526817,36sleep18890-21apt-get00:06:111
780226821,25sleep10-21swapper/122:16:431
524426823,25sleep30-21swapper/322:11:123
221726820,29sleep32361-21latency_hist22:01:143
221726820,29sleep32361-21latency_hist22:01:143
1700226820,23sleep00-21swapper/022:46:420
128132688,16sleep20-21swapper/220:51:242
1109526818,31sleep10-21swapper/119:01:561
2824326721,28sleep10-21swapper/119:56:311
2340526717,24sleep10-21swapper/123:06:501
211926721,23sleep30-21swapper/320:16:323
1213726712,16sleep00-21swapper/019:06:180
1202426714,23sleep30-21swapper/320:46:453
61826614,33sleep1634-21/usr/sbin/munin20:11:331
39472666,13sleep10-21swapper/120:21:391
1995326618,31sleep119984-21/usr/sbin/munin22:56:361
2244226515,32sleep122529-21gpgv23:06:131
2201126522,22sleep00-21swapper/019:36:310
1279026527,22sleep00-21swapper/020:51:220
1248626516,29sleep30-21swapper/322:31:433
582126418,21sleep00-21swapper/023:56:130
2964026413,32sleep129702-21/usr/sbin/munin23:26:401
252826418,29sleep12573-21grep22:01:331
1570226427,19sleep10-21swapper/121:01:121
1116826411,40sleep00-21swapper/022:27:120
1083726422,22sleep20-21swapper/219:01:492
2665226311,38sleep026658-21df_inode19:51:290
1934226311,36sleep319485-21latency_hist22:56:143
1061426310,39sleep110615-21sshd00:10:321
324726215,32sleep03259-21latency_hist20:21:130
255326213,25sleep10-21swapper/120:16:421
1727426219,25sleep117302-21munin-run21:06:131
1637526216,28sleep316461-21mailstats00:26:413
1235226215,27sleep10-21swapper/119:06:311
936726115,32sleep313441-21snmpd00:06:183
2605726114,18sleep10-21swapper/121:31:411
2324426117,27sleep30-21swapper/319:41:143
1620226113,28sleep316224-21irqstats21:01:353
160412615,10sleep30-21swapper/322:44:353
894526017,22sleep10-21swapper/122:21:361
566126019,19sleep10-21swapper/120:26:431
3131326020,19sleep30-21swapper/323:31:423
162202607,13sleep10-21swapper/119:16:411
162202607,13sleep10-21swapper/119:16:411
82225911,23sleep30-21swapper/320:11:393
771725915,19sleep00-21swapper/020:34:340
684625915,29sleep16859-21irqstats20:31:361
2270525914,29sleep122797-21/usr/sbin/munin21:21:311
2270525914,29sleep122797-21/usr/sbin/munin21:21:311
1046625916,23sleep10-21swapper/120:41:421
3155725816,25sleep131610-21users23:31:441
2475025816,23sleep10-21swapper/123:11:341
2292025814,20sleep30-21swapper/321:21:373
2292025814,20sleep30-21swapper/321:21:373
1438725818,23sleep314384-21apt-get20:56:223
980225720,22sleep30-21swapper/322:24:323
294725715,20sleep30-21swapper/322:01:433
2219925715,19sleep00-21swapper/021:21:140
1967425714,23sleep00-21swapper/021:11:370
195225721,17sleep10-21swapper/123:41:411
1392325717,24sleep10-21swapper/119:11:201
133925717,24sleep01345-21ntp_states21:56:390
133925717,24sleep01345-21ntp_states21:56:390
683225615,23sleep36886-21munin-run22:16:123
3184525617,20sleep00-21swapper/021:51:300
2908725613,26sleep029116-21ntp_states21:41:370
2908725613,26sleep029116-21ntp_states21:41:370
2760325619,16sleep30-21swapper/321:36:433
141825614,24sleep31447-21df23:41:273
937425514,20sleep00-21swapper/000:06:230
63032558,13sleep00-21swapper/022:11:450
3240425517,18sleep10-21swapper/123:36:331
3240425517,18sleep10-21swapper/123:36:331
292725514,25sleep02921-21apt-get23:46:290
1947725512,18sleep10-21swapper/119:26:431
1900325513,23sleep319057-21/usr/sbin/munin19:26:333
1746925514,23sleep317525-21irqstats19:21:323
1487925515,20sleep10-21swapper/122:41:221
1128125517,20sleep00-21swapper/000:11:320
2250825412,27sleep022515-21sh21:21:260
1947225419,18sleep30-21swapper/321:11:323
1823625411,27sleep118237-21sshd19:22:351
1495925414,23sleep00-21swapper/019:11:440
3134025314,24sleep031344-21diskstats20:06:270
2416725317,19sleep10-21swapper/121:26:321
2416725317,19sleep10-21swapper/121:26:321
1464825313,17sleep00-21swapper/020:56:310
1335525313,24sleep313346-21apt-get22:36:273
1032325314,17sleep30-21swapper/322:26:343
580625212,25sleep35818-21munin-run23:56:123
391425214,23sleep33938-21ntp_states20:21:373
3152225219,16sleep30-21swapper/321:51:133
3152225219,16sleep30-21swapper/321:51:133
2874625217,15sleep10-21swapper/121:41:301
2874625217,15sleep10-21swapper/121:41:301
3147125118,17sleep10-21swapper/121:51:111
3147125118,17sleep10-21swapper/121:51:111
2543325118,18sleep00-21swapper/019:46:370
2478225117,16sleep10-21swapper/119:46:131
2254525113,25sleep122620-21timerandwakeup19:36:441
1906525116,16sleep10-21swapper/121:11:141
1257925116,15sleep00-21swapper/020:51:140
1030225114,21sleep110308-21diskstats22:26:321
592125014,16sleep10-21swapper/122:11:381
867324912,18sleep00-21swapper/022:21:260
2942924914,20sleep329511-21sed23:26:343
181602499,23sleep018292-21unixbench_singl00:31:430
181602499,23sleep018292-21unixbench_singl00:31:430
309722476,16sleep10-21swapper/120:06:121
1821824714,18sleep10-21swapper/122:51:351
3166124613,16sleep30-21swapper/320:06:343
528324512,17sleep00-21swapper/020:26:340
180082459,20sleep018075-21df22:51:300
541524410,16sleep30-21swapper/320:26:373
266972448,15sleep30-21swapper/319:51:323
271102439,16sleep00-21swapper/021:36:240
2479724010,15sleep30-21swapper/321:26:463
2479724010,15sleep30-21swapper/321:26:463
13732409,18sleep00-21swapper/023:41:240
260632389,17sleep10-21swapper/123:16:261
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional