You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 08:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Apr 15, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20154247312,446sleep120193-21ntpq22:56:431
26581245218,421sleep226596-21fschecks_count19:51:382
21199233424,233sleep20-21swapper/223:01:212
1376230618,83sleep20-21swapper/221:56:562
27524230133,251sleep227546-21awk23:21:192
4263225574,114sleep24166-21/usr/sbin/munin22:06:332
23439225568,170sleep223451-21munin-run21:26:192
13553222610,197sleep215044-1kworker/2:0H19:11:332
13553222610,197sleep215044-1kworker/2:0H19:11:332
5936222223,120sleep25970-21modprobe23:56:182
24504221310,193sleep013441-21snmpd19:44:210
5708221260,86sleep20-21swapper/220:26:492
32170221020,172sleep032208-21idleruntime-cro23:36:180
18431220782,34sleep218454-21awk19:26:192
1313621877,144sleep213175-21ntpq20:51:442
17213217813,93sleep217364-21latency_hist21:06:212
6354216820,93sleep20-21swapper/223:56:392
16866216241,58sleep2222-21systemd-journal00:31:182
30213215410,37sleep20-21swapper/223:26:522
26517215262,14sleep20-21swapper/221:33:142
6016214772,41sleep20-21swapper/222:11:422
2911421457,34sleep20-21swapper/223:26:192
24809214512,115sleep124906-21awk21:27:311
16785214260,66sleep216670-21apt-get19:21:222
27834214114,109sleep127930-21cat19:56:211
18937214020,94sleep119085-21sed22:51:511
14290213953,20sleep20-21swapper/220:56:282
31485213812,107sleep026767-21kworker/u8:021:51:210
8428213558,16sleep20-21swapper/200:01:512
11115213248,38sleep30-21swapper/319:06:213
24390212817,80sleep30-21swapper/323:11:193
25301212713,98sleep325451-21cat21:31:213
1909221269,18sleep20-21swapper/219:26:432
17701212218,82sleep20-21swapper/200:31:472
26256212110,99sleep226290-21cat19:51:212
137221215,10sleep00-21swapper/020:14:080
12586212111,96sleep212681-21timerandwakeup22:31:492
10603211818,70sleep20-21swapper/222:26:402
31631211517,81sleep331782-21sed23:31:493
12272211425,73sleep212367-21sed00:16:202
1552211116,74sleep31562-21/usr/sbin/munin23:41:303
395121100,7sleep10-21swapper/123:46:481
12440211016,68sleep311955-21/usr/sbin/munin22:31:463
10913211026,58sleep310908-21apt-get00:11:333
3809210911,25sleep20-21swapper/220:21:452
5973210817,74sleep36005-40mandb23:56:193
31179210876,16sleep20-21swapper/223:31:402
14062210813,78sleep314108-21grep00:21:343
115921066,11sleep20-21swapper/223:39:352
13821210520,17sleep20-21swapper/200:21:212
12679210533,57sleep212703-21munin-run19:11:202
5600210416,18sleep10-21swapper/120:26:481
28442210416,65sleep20-21swapper/221:41:212
15577210416,75sleep015616-21processes22:41:480
14295210411,79sleep014311-21awk20:56:300
3220621037,80sleep232270-21mailstats21:51:472
22309210320,56sleep022454-21head23:01:550
21826210211,76sleep221829-21cron21:21:192
17916210231,59sleep217940-21munin-run22:51:192
7862210118,62sleep20-21swapper/220:36:222
23105210165,18sleep20-21swapper/223:06:292
2269121016,10sleep20-21swapper/223:04:232
18820210158,22sleep00-21swapper/022:51:500
7331210070,16sleep20-21swapper/222:16:352
367821007,37sleep213441-21snmpd22:03:262
3632210022,15sleep20-21swapper/223:46:462
31276210020,56sleep131368-21fschecks_time20:06:381
39229927,17sleep20-21swapper/220:11:402
254232997,13sleep00-21swapper/019:46:470
1820429914,57sleep20-21swapper/222:51:312
134172998,11sleep00-21swapper/020:51:500
3090829824,16sleep20-21swapper/221:46:552
2220329811,74sleep122277-21fschecks_time21:21:381
220872975,12sleep30-21swapper/319:36:413
220872975,12sleep30-21swapper/319:36:413
1555629710,67sleep20-21swapper/222:41:462
1715129622,38sleep10-21swapper/100:31:291
1406129623,18sleep20-21swapper/200:21:332
2390229410,66sleep20-21swapper/219:41:492
2807429320,54sleep20-21swapper/219:56:342
2064029318,22sleep20-21swapper/219:31:442
68712928,12sleep00-21swapper/020:31:440
1785729211,15sleep20-21swapper/221:06:462
1219129215,20sleep00-21swapper/000:16:180
1188129224,43sleep112036-21unixbench_multi20:46:511
1165929210,69sleep211722-21processes20:46:482
87172908,37sleep20-21swapper/220:36:512
594729020,51sleep05719-21/usr/sbin/munin22:11:400
52129024,47sleep1544-21munin-run21:56:191
246802909,16sleep20-21swapper/223:11:342
713628912,63sleep37219-21/usr/sbin/munin20:31:513
308892899,68sleep230917-21basename20:06:192
231492899,68sleep223157-21munin-run19:41:202
231492899,68sleep223157-21munin-run19:41:202
2067028918,56sleep320709-21ntp_kernel_pll_19:31:453
1761728912,63sleep317618-21sshd22:47:293
1518328914,59sleep115190-21df_inode22:41:371
29912885,10sleep20-21swapper/220:19:392
26192885,10sleep20-21swapper/223:43:312
853928725,35sleep30-21swapper/300:01:523
2284028734,31sleep322841-21cron23:06:183
1359228711,60sleep013634-21grep19:11:590
1359228711,60sleep013634-21grep19:11:590
1777528617,50sleep317907-21ls00:31:483
789628512,58sleep17905-21diskstats00:01:381
277572858,14sleep20-21swapper/221:36:492
249232856,10sleep20-21swapper/221:27:372
41902839,23sleep30-21swapper/322:06:303
140122836,13sleep20-21swapper/222:36:462
1288228317,48sleep012487-21/usr/sbin/munin00:16:430
1025728318,48sleep110323-21/usr/sbin/munin19:01:471
311782826,13sleep20-21swapper/220:06:312
1104928220,47sleep211114-21sh19:06:202
602428112,56sleep16125-21sed23:56:211
3062028022,31sleep00-21swapper/023:31:180
2550128018,26sleep00-21swapper/021:31:340
190582806,13sleep20-21swapper/221:11:382
106682807,41sleep210674-21munin-run00:11:202
1758727916,41sleep017635-21ntp_offset19:21:530
348727815,43sleep33494-21/usr/sbin/munin20:21:383
1426827813,47sleep314413-21unixbench_singl22:36:513
2558427713,35sleep30-21swapper/321:31:373
3172627614,47sleep331794-21switchtime20:06:483
3037327613,45sleep330421-21if_err_eth021:46:433
2363227615,42sleep023764-21latency_hist21:26:210
2002227621,28sleep30-21swapper/319:31:203
1843127613,46sleep318475-21irqstats22:51:413
1263527619,35sleep112661-21munin-run19:11:191
1699427515,41sleep316670-21apt-get19:21:233
1537227523,27sleep10-21swapper/100:26:211
1730227412,47sleep117426-21timerwakeupswit22:46:501
1690927418,41sleep316939-21sed00:31:193
2879827325,26sleep00-21swapper/021:41:400
105572736,12sleep20-21swapper/220:42:032
1004227315,40sleep110087-21ntp_states20:41:461
2386227213,43sleep123899-21ps19:41:481
2004827211,41sleep020196-21grep19:31:210
1780027219,30sleep00-21swapper/021:06:440
162027213,43sleep31764-21cat20:16:213
863427112,45sleep38700-21latency_hist22:21:203
2152927123,31sleep121559-21munin-run19:36:191
98312705,10sleep20-21swapper/222:22:282
759027010,28sleep00-21swapper/000:01:210
56727016,20sleep00-21swapper/023:36:510
2775327016,37sleep327802-21tr19:56:193
1508427010,44sleep315103-21apt-get22:41:313
625926921,29sleep36256-21apt-get23:56:323
442826913,36sleep011-21ksoftirqd/023:51:200
2757526911,36sleep00-21swapper/023:21:210
157126919,35sleep01592-21awk21:57:020
1410526915,26sleep30-21swapper/320:56:213
533526818,31sleep30-21swapper/320:26:423
1745226822,24sleep10-21swapper/119:21:501
1291626816,29sleep10-21swapper/100:16:441
832226714,23sleep10-21swapper/120:36:411
62326724,24sleep30-21swapper/321:56:213
1796826715,23sleep00-21swapper/022:51:210
1510826716,24sleep10-21swapper/120:56:521
1373626717,28sleep00-21swapper/022:36:390
1325526720,32sleep312648-21apt-get19:11:253
1325526720,32sleep312648-21apt-get19:11:253
1051326717,24sleep10-21swapper/122:26:391
3270426620,27sleep132706-21/usr/sbin/munin20:11:311
306852660,8sleep30-21swapper/323:31:203
2337026613,22sleep10-21swapper/123:06:401
1007326614,18sleep00-21swapper/019:01:420
2604226518,24sleep00-21swapper/023:16:200
21302656,14sleep30-21swapper/320:16:413
1925426523,18sleep30-21swapper/321:11:433
1633926520,26sleep10-21swapper/121:01:481
134226514,19sleep30-21swapper/323:41:223
1319426523,20sleep30-21swapper/300:16:503
699326414,26sleep10-21swapper/120:31:491
3249226411,38sleep10-21swapper/123:36:351
3145526411,36sleep131521-21/usr/sbin/munin23:31:471
2635126416,29sleep126356-21/usr/sbin/munin23:16:371
230822645,10sleep10-21swapper/119:40:591
230822645,10sleep10-21swapper/119:40:591
2157226418,27sleep121568-21apt-get23:01:321
1752126414,34sleep317525-21diskstats21:06:383
1587926415,16sleep30-21swapper/319:16:553
1543826420,24sleep10-21swapper/119:16:371
1489026415,31sleep114921-21munin-run19:16:201
1489026415,31sleep114921-21munin-run19:16:201
898226313,30sleep08995-21/usr/sbin/munin20:36:530
808526324,21sleep30-21swapper/320:36:353
550426311,22sleep10-21swapper/122:11:211
3094926318,25sleep10-21swapper/120:06:221
3093226310,39sleep030948-21latency_hist20:06:210
3012626315,30sleep330205-21/usr/sbin/munin20:01:483
271726313,36sleep12726-21diskstats22:01:361
261526318,26sleep32634-21apt-get22:01:303
1146126311,39sleep311474-21fschecks_count19:06:373
85212626,15sleep10-21swapper/122:20:291
574526223,23sleep30-21swapper/322:11:353
387626219,18sleep10-21swapper/120:21:481
161502629,35sleep016242-21/usr/sbin/munin21:01:430
1146026222,22sleep00-21swapper/020:46:410
2601426115,32sleep326037-21latency_hist23:16:193
2392226110,23sleep00-21swapper/021:26:290
2065526122,20sleep10-21swapper/121:16:371
2039126121,23sleep30-21swapper/322:56:493
1035126116,27sleep310495-21unixbench_singl20:41:523
67652606,36sleep10-21swapper/122:13:191
340926017,26sleep03403-21apt-get20:21:350
2352626015,26sleep323578-21munin-run21:26:203
2212126011,36sleep022156-21latency19:36:420
2212126011,36sleep022156-21latency19:36:420
1588126018,18sleep10-21swapper/100:26:411
2884225916,24sleep128895-21irqstats21:41:411
2589025918,26sleep125928-21/usr/sbin/munin21:31:431
2531625922,21sleep30-21swapper/319:46:423
223025915,30sleep12276-21awk20:16:431
131825917,18sleep10-21swapper/121:56:551
1018825923,17sleep00-21swapper/022:26:220
2843625813,25sleep028560-21cat23:21:490
272132585,11sleep30-21swapper/323:17:203
256725814,21sleep00-21swapper/020:16:500
1869925811,32sleep118700-21/usr/sbin/munin19:26:301
178925814,19sleep10-21swapper/123:41:401
897125715,21sleep10-21swapper/122:21:381
2859925717,25sleep328619-21apt-get21:41:313
233172578,13sleep00-21swapper/023:06:390
105325710,32sleep3542-21apt-get21:56:293
3118125613,25sleep031175-21apt-get20:06:330
2939925613,18sleep10-21swapper/120:01:211
286425617,23sleep00-21swapper/023:46:190
2507825618,21sleep10-21swapper/123:11:451
2365925614,22sleep30-21swapper/319:41:423
1744325618,19sleep10-21swapper/121:06:361
1694225612,18sleep00-21swapper/000:31:200
3234925522,18sleep00-21swapper/021:51:490
2817425513,22sleep10-21swapper/123:21:421
1715725514,20sleep00-21swapper/000:31:320
1405925515,25sleep014055-21apt-get00:21:310
1137825516,25sleep00-21swapper/019:06:340
602325414,23sleep00-21swapper/023:56:200
2948425411,21sleep10-21swapper/123:26:381
2914725417,16sleep30-21swapper/323:26:203
2880525414,19sleep30-21swapper/319:56:513
2689225418,19sleep00-21swapper/019:51:440
1197825414,24sleep111973-21apt-get22:31:321
961525313,15sleep30-21swapper/300:06:413
936825312,23sleep09362-21apt-get00:06:340
934325314,16sleep10-21swapper/100:06:311
275412539,17sleep30-21swapper/321:36:463
1945625316,20sleep10-21swapper/121:11:491
1126225312,25sleep311268-21df_inode20:46:363
1103225317,18sleep00-21swapper/022:26:500
1049825311,17sleep00-21swapper/020:41:530
163832528,35sleep00-21swapper/019:17:570
1283725211,18sleep10-21swapper/120:51:391
1142725212,22sleep111519-21mailstats00:11:471
372525115,21sleep33803-21find23:46:473
2084725114,24sleep013441-21snmpd22:58:400
2071525111,16sleep00-21swapper/019:31:460
1611925112,23sleep316148-21irqstats21:01:423
1083325113,22sleep310878-21ntp_states22:26:453
790525013,19sleep00-21swapper/022:16:490
653325013,18sleep10-21swapper/123:56:421
466325012,24sleep04695-21/usr/sbin/munin22:06:520
2938225015,17sleep00-21swapper/020:01:200
2077825012,19sleep30-21swapper/321:16:403
2056825013,22sleep020561-21apt-get21:16:320
1409325013,19sleep10-21swapper/122:36:491
1439724911,23sleep314404-21diskstats20:56:373
1323924910,20sleep00-21swapper/022:36:190
931524812,21sleep39326-21ntp_states22:21:453
781924811,21sleep07815-21apt-get00:01:330
503224810,20sleep05041-21diskstats20:26:350
2714824812,21sleep027154-21df_inode21:36:350
1907224816,15sleep00-21swapper/021:11:400
314124613,19sleep0281-21dbus-daemon22:01:470
3018124612,17sleep10-21swapper/121:46:371
2715724611,20sleep127162-21diskstats21:36:361
250292419,16sleep10-21swapper/119:46:361
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional