You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-23 - 13:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Tue Dec 23, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
114482441013,4385sleep132236-21kworker/1:222:56:331
114482441013,4385sleep132236-21kworker/1:222:56:331
24172270833,623sleep2369-21snmpd23:39:272
5235267529,601sleep25362-21latency_hist19:14:272
5235267529,601sleep25362-21latency_hist19:14:272
30745263817,605sleep330751-21/usr/sbin/munin22:15:493
810124066,387sleep28147-21ntpq00:29:512
6634239617,364sleep16671-21ntpq20:59:501
6634239617,364sleep16671-21ntpq20:59:501
4076232724,270sleep2369-21snmpd20:50:222
9290229315,255sleep39327-21awk21:09:253
8342287114,90sleep20-21swapper/222:24:282
1242225084,59sleep20-21swapper/200:09:272
30382224619,204sleep030443-21fschecks_time22:15:420
16023224452,24sleep20-21swapper/219:44:572
906822419,219sleep19074-21df19:24:441
18347223435,49sleep20-21swapper/221:34:572
16624223444,23sleep20-21swapper/223:14:442
24273222647,51sleep20-21swapper/220:14:292
14838222014,41sleep214886-21latency_hist23:09:262
22318221212,169sleep20-21swapper/221:49:472
7457220755,44sleep20-21swapper/200:29:262
6418220481,53sleep20-21swapper/219:14:562
2730220434,129sleep20-21swapper/200:14:242
1868022046,160sleep218715-21ntpq19:54:582
29758220134,126sleep229879-21latency_hist22:14:262
26265220123,155sleep126284-21apt-get20:19:371
1126921998,154sleep211407-21unixbench_singl19:29:582
28295219642,34sleep20-21swapper/220:24:552
20242218948,25sleep20-21swapper/223:24:532
2698021687,122sleep227068-21tune2fs22:04:452
10393216816,136sleep310401-21/usr/sbin/munin22:54:373
10393216816,136sleep310401-21/usr/sbin/munin22:54:373
25024216652,33sleep225027-21cron21:59:242
32529216413,110sleep232667-21grep22:20:002
32529216413,110sleep232667-21grep22:20:002
5166216311,116sleep25240-21switchtime00:19:572
5166216311,116sleep25240-21switchtime00:19:572
28329215621,74sleep20-21swapper/223:49:572
13356215111,126sleep013392-21ntpq21:19:520
3249721499,18sleep20-21swapper/220:39:402
25738214816,23sleep225797-21http20:19:252
8045214512,116sleep28097-21cpuspeed21:04:442
9912214311,91sleep210044-21cat00:35:002
18284214310,120sleep318177-21/usr/sbin/munin23:19:453
5045213916,67sleep20-21swapper/220:54:512
29687213718,99sleep029696-21kworker/u8:322:14:240
22501213719,100sleep122564-21latency21:49:501
31830213318,35sleep20-21swapper/200:00:552
31830213318,35sleep20-21swapper/200:00:552
30580213331,68sleep230595-21munin-run23:59:262
30580213331,68sleep230595-21munin-run23:59:262
2250421338,37sleep20-21swapper/220:07:342
1603721337,71sleep20-21swapper/221:29:402
1288021337,14sleep20-21swapper/219:34:592
1288021337,14sleep20-21swapper/219:34:592
10989213318,11sleep20-21swapper/221:13:122
10989213318,11sleep20-21swapper/221:13:122
27615213219,35sleep00-21swapper/020:24:280
2627621326,11sleep20-21swapper/222:00:172
4220213155,34sleep20-21swapper/222:34:382
9796213013,101sleep09311-21apt-get21:09:340
9796213013,101sleep09311-21apt-get21:09:340
8605213018,35sleep20-21swapper/219:23:042
21803213015,12sleep20-21swapper/221:47:372
21803213015,12sleep20-21swapper/221:47:372
3378212925,38sleep20-21swapper/222:29:572
27427212912,99sleep21-21systemd23:49:242
31976212716,12sleep20-21swapper/220:36:502
22298212715,11sleep20-21swapper/223:31:232
530321250,8sleep10-21swapper/120:54:571
29642212552,55sleep229656-21kernelversion20:29:472
16378212253,29sleep216360-21latency_hist23:14:272
30211211525,65sleep129702-21apt-get22:14:381
7343211411,86sleep07406-21/usr/sbin/munin22:44:450
23478211420,71sleep323486-21modprobe21:54:243
23100211410,87sleep123159-21basename20:09:471
3156121128,60sleep20-21swapper/222:19:272
30513211121,70sleep130569-40ogrotate23:59:251
30513211121,70sleep130569-40ogrotate23:59:251
772221109,66sleep27716-21munin-run21:04:272
772221109,66sleep27716-21munin-run21:04:272
6518211012,75sleep16657-21unixbench_singl22:39:581
3293210931,43sleep23329-21cat00:14:482
75121080,7sleep30-21swapper/300:05:033
10242210826,68sleep210129-21cron22:54:282
10242210826,68sleep210129-21cron22:54:282
2167821068,13sleep00-21swapper/020:04:500
2988121050,7sleep30-21swapper/322:14:263
7728210418,73sleep37762-21wc19:19:503
3157210416,71sleep03234-21fw_packets00:14:460
2730210418,12sleep20-21swapper/220:48:002
10569210325,26sleep10-21swapper/100:39:271
12630210228,25sleep20-21swapper/222:59:572
32175210126,52sleep332199-21cron20:39:253
327929910,73sleep13377-21timerandwakeup22:29:561
73022976,12sleep20-21swapper/222:44:312
2115529714,61sleep321132-21latency_hist20:04:283
1546129721,23sleep00-21swapper/019:44:460
1020129717,63sleep310221-21cut21:11:153
1020129717,63sleep310221-21cut21:11:153
235429621,39sleep10-21swapper/119:04:421
93382958,74sleep00-21swapper/022:49:540
2478529411,71sleep124823-21cut23:40:111
1295329312,38sleep20-21swapper/221:19:382
275902929,44sleep227599-21munin-run20:24:262
191742925,10sleep20-21swapper/223:20:492
1404129218,20sleep00-21swapper/023:04:560
89892909,44sleep28991-21apt-get19:24:372
816529019,55sleep18229-21ps00:29:541
689329021,39sleep20-21swapper/222:42:292
2563229012,63sleep025713-21/usr/sbin/munin21:59:510
2264529021,54sleep322646-21cron20:09:253
96028910,61sleep00-21swapper/020:40:030
524128920,54sleep35364-21expr00:19:583
524128920,54sleep35364-21expr00:19:583
243228917,48sleep02540-21latency_hist22:29:270
1637028917,56sleep10-21swapper/123:14:261
1400028921,50sleep314037-21ps23:04:553
99062888,41sleep20-21swapper/222:51:342
2114428820,31sleep20-21swapper/220:04:272
1380828811,60sleep113891-21sed23:04:491
2186328617,46sleep021256-21/usr/sbin/munin23:29:560
213722869,29sleep10-21swapper/120:04:381
1215428611,60sleep312207-21idleruntime22:59:463
1174528621,30sleep20-21swapper/221:14:542
1174528621,30sleep20-21swapper/221:14:542
1973928522,30sleep00-21swapper/023:24:390
2843828413,53sleep128505-21df22:09:411
279552846,57sleep10-21swapper/123:49:481
150132847,39sleep215092-21systemctl21:25:072
246382838,33sleep20-21swapper/223:39:432
1401728314,52sleep314028-21/usr/sbin/munin19:39:493
68128225,30sleep10-21swapper/122:24:271
68128225,30sleep10-21swapper/122:24:271
283328230,25sleep20-21swapper/219:04:552
2259828221,44sleep322752-21latency_hist23:34:273
1719028212,26sleep00-21swapper/023:14:580
801428113,50sleep08066-21memory00:29:490
466428127,22sleep00-21swapper/000:19:430
466428127,22sleep00-21swapper/000:19:430
2510328121,43sleep125042-21latency_hist21:59:271
39332809,51sleep10-21swapper/119:09:381
39332809,51sleep10-21swapper/119:09:381
327028017,43sleep03340-21/usr/sbin/munin20:49:480
326262800,6sleep10-21swapper/100:04:491
2039628016,46sleep120423-21latency_hist21:44:251
2039628016,46sleep120423-21latency_hist21:44:251
191432808,32sleep20-21swapper/221:39:372
191432808,32sleep20-21swapper/221:39:372
216827919,20sleep00-21swapper/020:44:550
2328127813,28sleep00-21swapper/023:34:530
974327714,48sleep09817-21sendmail_mailtr00:34:580
555327725,20sleep30-21swapper/322:39:273
44727720,28sleep00-21swapper/000:05:010
98302767,12sleep30-21swapper/322:50:343
2110127620,42sleep021110-21modprobe20:04:250
59327520,36sleep3619-21munin-run22:24:253
59327520,36sleep3619-21munin-run22:24:253
2975227520,26sleep30-21swapper/323:54:593
2975227520,26sleep30-21swapper/323:54:593
2780527516,34sleep327899-21fschecks_time23:49:463
936127425,26sleep10-21swapper/122:49:551
472227420,38sleep14798-21awk22:34:531
282502747,12sleep10-21swapper/120:24:541
172227419,29sleep30-21swapper/322:25:103
1184827424,25sleep30-21swapper/319:34:283
1890927316,22sleep00-21swapper/023:19:590
122132730,8sleep00-21swapper/022:59:460
750727218,33sleep37522-21apt-get21:04:243
750727218,33sleep37522-21apt-get21:04:243
289827223,30sleep30-21swapper/319:04:563
1766327215,29sleep10-21swapper/121:34:441
1718727213,45sleep0369-21snmpd21:33:050
134727223,25sleep10-21swapper/120:44:281
3110627118,35sleep031159-21/usr/sbin/munin20:34:460
937827020,27sleep10-21swapper/121:09:271
57242708,50sleep00-21swapper/020:56:420
2778927017,28sleep30-21swapper/322:04:583
176292707,43sleep30-21swapper/321:34:413
1081227018,28sleep10-21swapper/119:29:491
3226426917,33sleep032326-21latency_hist00:04:290
3226426917,33sleep032326-21latency_hist00:04:290
2830326922,25sleep30-21swapper/320:24:563
1724026923,24sleep00-21swapper/019:49:550
1307726912,39sleep313163-21perl21:19:473
1216626914,38sleep112176-21/usr/sbin/munin19:34:461
1216626914,38sleep112176-21/usr/sbin/munin19:34:461
2984826822,25sleep00-21swapper/020:29:530
894626715,34sleep38983-21basename00:34:243
676926713,21sleep10-21swapper/100:24:571
2456326722,27sleep124594-21tune2fs20:14:441
1960626720,28sleep019645-21wc21:39:510
1960626720,28sleep019645-21wc21:39:510
1641326712,36sleep316566-21latency_hist23:14:283
1370426721,25sleep00-21swapper/019:39:420
1263426719,23sleep10-21swapper/121:19:251
1263426719,23sleep10-21swapper/121:19:251
536426621,20sleep30-21swapper/319:14:283
1999826623,30sleep319714-21/usr/sbin/munin23:24:463
1700826618,14sleep20-21swapper/219:49:492
151662668,12sleep10-21swapper/119:44:351
946026516,23sleep00-21swapper/019:24:560
633926523,21sleep30-21swapper/300:24:473
366026514,25sleep10-21swapper/120:49:571
2946526515,26sleep30-21swapper/320:29:443
2580126524,24sleep30-21swapper/320:19:263
2114326513,35sleep121149-21ntp_states21:44:531
2114326513,35sleep121149-21ntp_states21:44:531
1361926517,22sleep10-21swapper/121:20:011
1040226522,25sleep10-21swapper/121:11:211
1040226522,25sleep10-21swapper/121:11:211
875826420,22sleep00-21swapper/019:24:270
565926416,33sleep15804-21df19:14:351
500426416,34sleep15041-21awk00:19:521
500426416,34sleep15041-21awk00:19:521
311026418,30sleep33150-21sort00:14:453
1477926418,29sleep314811-21/usr/sbin/munin21:25:003
1459426417,19sleep10-21swapper/121:24:541
933326317,28sleep09345-21apt-key21:09:260
648626317,22sleep30-21swapper/319:14:573
518326314,32sleep35184-21cron19:14:243
3099726315,22sleep10-21swapper/120:34:431
3067426312,17sleep10-21swapper/120:34:261
2414126312,34sleep024171-21/usr/sbin/munin21:54:500
2098926315,31sleep121007-21modprobe23:29:241
1568726312,19sleep00-21swapper/023:09:570
1333126312,35sleep00-21swapper/019:39:250
1333126312,35sleep00-21swapper/019:39:250
124662635,45sleep00-21swapper/021:17:140
124662635,45sleep00-21swapper/021:17:140
423626218,25sleep04303-21memory19:09:490
423626218,25sleep04303-21memory19:09:490
3126526215,33sleep131303-21ntpq00:00:141
3126526215,33sleep131303-21ntpq00:00:141
3115726216,23sleep30-21swapper/300:00:123
3115726216,23sleep30-21swapper/300:00:123
238462626,11sleep30-21swapper/323:36:293
218402628,42sleep30-21swapper/321:48:063
218402628,42sleep30-21swapper/321:48:063
1838326215,27sleep118436-21/usr/sbin/munin23:19:471
168052628,12sleep30-21swapper/323:14:493
1429326224,21sleep20-21swapper/219:39:562
1227026216,25sleep30-21swapper/319:34:483
1227026216,25sleep30-21swapper/319:34:483
313126113,33sleep33138-21wc20:49:433
303052616,11sleep00-21swapper/023:56:410
303052616,11sleep00-21swapper/023:56:410
2644426112,21sleep30-21swapper/320:20:073
2643726117,24sleep30-21swapper/323:44:493
262126121,19sleep00-21swapper/019:04:490
2579226120,21sleep30-21swapper/321:59:543
2506426118,20sleep30-21swapper/323:40:173
2493526116,28sleep00-21swapper/023:40:140
1795926116,24sleep10-21swapper/123:19:271
154362615,17sleep00-21swapper/021:25:220
355726018,18sleep10-21swapper/100:14:561
180072599,34sleep00-21swapper/021:34:510
1495325914,26sleep10-21swapper/119:44:271
561825813,27sleep05658-21sshd19:14:340
53052587,40sleep00-21swapper/022:37:140
473225813,31sleep34756-21df_inode20:54:443
2492425817,25sleep024931-21ntp_states20:14:510
2302325815,22sleep00-21swapper/020:09:460
1960425815,20sleep30-21swapper/319:59:273
1174825818,18sleep00-21swapper/022:59:270
1174825818,18sleep00-21swapper/022:59:270
745625716,25sleep17453-21apt-get19:19:401
640625716,27sleep06516-21timerandwakeup22:39:570
52925713,16sleep30-21swapper/320:39:583
2859025714,24sleep028621-21users23:49:590
2619025716,25sleep126251-21users22:00:041
170332577,40sleep30-21swapper/321:31:063
670225618,19sleep00-21swapper/000:24:560
2696825615,26sleep026974-21df_inode22:04:430
2663825613,27sleep10-21swapper/123:44:551
2413925615,26sleep124160-21munin-run23:39:261
24092565,15sleep30-21swapper/300:10:133
203925616,25sleep12045-21ntp_states20:44:521
1837025613,22sleep00-21swapper/019:54:520
126825617,18sleep10-21swapper/122:25:001
742525517,18sleep30-21swapper/322:44:483
2791325515,24sleep027919-21df_inode20:24:440
2223125514,24sleep022226-21apt-get21:49:410
185125519,18sleep10-21swapper/100:09:511
773825411,17sleep10-21swapper/122:44:541
208482548,11sleep10-21swapper/120:01:071
1736825415,24sleep117514-21latency_hist21:34:271
661625315,19sleep00-21swapper/020:59:490
661625315,19sleep00-21swapper/020:59:490
2814525315,15sleep10-21swapper/122:09:251
2617325312,25sleep026168-21apt-get23:44:380
2094625317,20sleep00-21swapper/021:44:480
2094625317,20sleep00-21swapper/021:44:480
1673125312,24sleep316729-21apt-get19:49:383
1526425315,22sleep315271-21df_inode19:44:413
1158425316,19sleep30-21swapper/321:14:503
1158425316,19sleep30-21swapper/321:14:503
621425211,24sleep36246-21ntp_states22:39:523
2320525217,16sleep30-21swapper/320:09:493
2173525213,23sleep121741-21ntp_states23:29:531
192662526,34sleep10-21swapper/119:56:021
165325212,23sleep31660-21df_inode20:44:423
1061725213,23sleep010623-21diskstats19:29:440
1029825215,22sleep30-21swapper/319:29:273
308825111,25sleep03124-21ntp_states22:29:510
3187825014,21sleep031884-21diskstats22:19:460
3187825014,21sleep031884-21diskstats22:19:460
284524917,17sleep30-21swapper/322:29:463
2374724914,20sleep323766-21apt-get21:54:363
148624811,21sleep01553-21df00:09:420
1290024813,21sleep10-21swapper/123:00:011
1914424612,17sleep30-21swapper/321:39:383
1914424612,17sleep30-21swapper/321:39:383
1681624510,17sleep10-21swapper/119:49:441
92562416,12sleep30-21swapper/300:34:443
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional