You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-10 - 17:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Apr 10, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
108982370515,3670sleep14156-21kworker/u8:308:56:451
133692274911,2722sleep124961-1kworker/1:0H07:22:191
6090283816,805sleep36094-21diskstats08:41:533
6090283816,805sleep36094-21diskstats08:41:533
20697280816,776sleep320760-21/usr/sbin/munin09:26:563
7230266014,630sleep17240-21/usr/sbin/munin12:11:361
7230266014,630sleep17240-21/usr/sbin/munin12:11:361
32404235322,314sleep332473-21/usr/sbin/munin11:47:023
82062291123,33sleep28345-21grep12:12:052
82062291123,33sleep28345-21grep12:12:052
19416228887,179sleep218909-21apt-get11:06:472
7744228596,129sleep27821-21latency_hist07:06:372
142662259138,19sleep20-21swapper/210:51:362
30084225668,150sleep230196-21latency_hist11:41:362
22527223659,29sleep20-21swapper/211:16:442
212442227107,25sleep20-21swapper/207:46:552
5448222014,191sleep35464-21kernelversion10:21:573
5448222014,191sleep35464-21kernelversion10:21:573
32752222027,171sleep2373-21mailstats08:22:052
32752222027,171sleep2373-21mailstats08:22:052
19327222029,179sleep219338-21uname09:21:582
19327222029,179sleep219338-21uname09:21:582
27986221311,118sleep228017-21awk09:51:352
27986221311,118sleep228017-21awk09:51:352
26504221359,19sleep20-21swapper/208:02:142
28524220510,95sleep20-21swapper/211:36:352
7315220349,24sleep20-21swapper/208:46:372
7315220349,24sleep20-21swapper/208:46:372
3118422007,180sleep231267-21mailstats08:17:042
29609220031,22sleep20-21swapper/209:56:352
29609220031,22sleep20-21swapper/209:56:352
23628220017,170sleep223635-21cpu09:36:492
438221987,178sleep24356-21/usr/sbin/munin12:01:482
9319219315,39sleep29453-21latency_hist07:11:382
9257218511,149sleep20-21swapper/212:17:022
9257218511,149sleep20-21swapper/212:17:022
26117218558,91sleep226157-21uname11:27:022
18009218410,157sleep218046-21ntp_kernel_err11:02:022
14077218319,136sleep314224-21latency_hist09:06:373
1371921837,161sleep213692-21latency_hist12:31:362
1529421809,160sleep215319-21ntpq07:27:002
463121777,160sleep213441-21snmpd10:18:122
1935217010,146sleep02014-21find08:27:040
1935217010,146sleep02014-21find08:27:040
17131216813,106sleep20-21swapper/210:58:472
11217216617,132sleep111304-21find12:22:061
274521648,142sleep22764-21apt-get11:56:432
9199215919,86sleep20-21swapper/208:51:542
9912215516,102sleep20-21swapper/210:36:502
13362215111,64sleep20-21swapper/207:22:162
12225215110,95sleep20-21swapper/207:17:072
23383215010,83sleep20-21swapper/209:36:362
23383215010,83sleep20-21swapper/209:36:362
4505214411,115sleep34591-21fw_packets12:01:533
3317214212,93sleep23320-21cron10:16:352
19947213911,71sleep20-21swapper/207:41:592
14041213911,84sleep20-21swapper/209:06:362
1023121350,8sleep10-21swapper/107:11:561
7188213426,90sleep37215-21grep12:11:343
7188213426,90sleep37215-21grep12:11:343
18090213426,91sleep318154-21cut07:36:553
12379213414,101sleep311900-21apt-get12:26:443
87221328,89sleep2908-21ntpq10:06:582
17420213031,15sleep20-21swapper/209:16:492
6536212725,45sleep20-21swapper/212:07:052
6536212725,45sleep20-21swapper/212:07:052
16131212611,100sleep316156-21df10:56:503
2278212518,27sleep20-21swapper/208:27:302
2278212518,27sleep20-21swapper/208:27:302
18611212116,17sleep20-21swapper/207:37:062
1569721215,12sleep20-21swapper/210:55:442
31380212022,14sleep20-21swapper/210:01:442
25349212015,44sleep20-21swapper/211:26:262
13321212023,77sleep312811-21apt-get07:21:483
1120121206,28sleep20-21swapper/207:16:182
8057211910,94sleep28203-21latency_hist10:31:372
2417211919,82sleep12450-21/usr/sbin/munin10:11:571
1229211911,96sleep2934-21/usr/sbin/munin11:51:542
1229211911,96sleep2934-21/usr/sbin/munin11:51:542
29082211819,80sleep028594-21apt-get08:11:450
29082211819,80sleep028594-21apt-get08:11:450
27330211820,14sleep20-21swapper/208:06:502
11303211710,73sleep211328-21awk08:57:032
1059921178,51sleep210605-21cat12:21:472
12715211516,83sleep112208-21apt-get09:01:461
3106521145,26sleep20-21swapper/210:00:522
20608211310,17sleep20-21swapper/209:26:552
31581211220,75sleep331589-21modprobe11:46:343
2465621108,39sleep20-21swapper/207:57:012
2465621108,39sleep20-21swapper/207:57:012
41021095,92sleep00-21swapper/011:48:130
3846210910,81sleep20-21swapper/208:32:472
3200821099,40sleep20-21swapper/211:46:532
2471421094,10sleep30-21swapper/309:39:203
1559121098,13sleep20-21swapper/209:11:362
2907921087,16sleep20-21swapper/208:11:432
2907921087,16sleep20-21swapper/208:11:432
9455210718,66sleep39362-21munin-node07:11:393
2346321079,35sleep20-21swapper/207:52:082
14546210715,74sleep114638-21/usr/sbin/munin12:32:031
26117210619,10sleep20-21swapper/209:42:262
1396021067,57sleep20-21swapper/212:31:482
32293210514,77sleep132321-21df08:21:521
32293210514,77sleep132321-21df08:21:521
1636121058,37sleep20-21swapper/209:12:052
755721037,36sleep20-21swapper/208:46:492
755721037,36sleep20-21swapper/208:46:492
31633210311,76sleep131612-21latency_hist11:46:361
2383210320,64sleep02550-21rm11:56:370
2383210320,64sleep02550-21rm11:56:370
10386210111,70sleep00-21swapper/012:21:360
10386210111,70sleep00-21swapper/012:21:360
925829913,73sleep09298-21sh07:11:360
3051729970,14sleep00-21swapper/008:16:380
212292995,29sleep20-21swapper/211:11:562
1878029916,66sleep118924-21modprobe09:21:371
1878029916,66sleep118924-21modprobe09:21:371
1789729920,65sleep017916-21uname11:01:560
994729818,19sleep00-21swapper/010:36:520
56792989,32sleep20-21swapper/210:22:042
56792989,32sleep20-21swapper/210:22:042
2708829811,74sleep127137-21latency_hist08:06:371
168802986,31sleep20-21swapper/207:32:012
168802986,31sleep20-21swapper/207:32:012
2626729720,50sleep30-21swapper/308:02:113
255129616,63sleep12561-21apt-key11:56:381
2329529622,51sleep323438-21head11:17:063
303242958,51sleep20-21swapper/211:41:482
237032957,11sleep00-21swapper/009:36:510
3158029318,54sleep331381-21/usr/sbin/munin10:01:543
194332936,73sleep319413-21apt-get11:06:563
1690329221,53sleep016968-21sendmail_mailqu07:32:030
1690329221,53sleep016968-21sendmail_mailqu07:32:030
515629110,65sleep05173-21apt10:21:460
515629110,65sleep05173-21apt10:21:460
48632908,45sleep20-21swapper/208:36:592
349429011,64sleep03616-21timerwakeupswit11:57:500
2836529013,64sleep128410-21fschecks_count09:51:541
2836529013,64sleep128410-21fschecks_count09:51:541
249729011,62sleep32507-21sh10:11:593
3070028814,59sleep130744-21/usr/sbin/munin08:16:511
2696028712,62sleep027017-21cat11:31:370
1296028725,45sleep312953-21apt-get10:46:493
797228620,46sleep38044-21mailstats12:12:023
797228620,46sleep38044-21mailstats12:12:023
124612866,30sleep20-21swapper/210:43:532
1208928616,55sleep012205-21timerwakeupswit10:42:050
15932855,19sleep00-21swapper/010:09:260
136328512,56sleep11408-21/usr/sbin/munin11:51:571
136328512,56sleep11408-21/usr/sbin/munin11:51:571
794328414,52sleep07987-21ntp_kernel_pll_08:46:590
794328414,52sleep07987-21ntp_kernel_pll_08:46:590
668628425,38sleep36744-21/usr/sbin/munin07:01:583
2803728415,46sleep028120-21latency_hist09:51:370
2803728415,46sleep028120-21latency_hist09:51:370
1305428412,56sleep113271-21latency_hist07:21:381
3031128312,53sleep330317-21ntp_states09:57:013
1802928330,26sleep018144-21timerandwakeup09:17:040
1298128323,33sleep00-21swapper/009:02:300
1101128315,40sleep00-21swapper/008:56:520
767328235,29sleep07683-21modprobe07:06:350
1750428217,48sleep117584-21tune2fs09:16:531
1202028224,28sleep30-21swapper/312:26:363
954428018,42sleep39572-21munin-run10:36:353
1422428013,51sleep114265-21unin-run10:51:351
1177728010,56sleep111807-21awk10:41:581
639927918,16sleep00-21swapper/012:07:030
639927918,16sleep00-21swapper/012:07:030
1130227913,49sleep011358-21sed07:16:360
925927837,30sleep20-21swapper/210:32:542
1032327842,19sleep10-21swapper/110:37:011
1914027729,32sleep119346-21latency_hist11:06:381
951227612,48sleep09549-21ntpq08:52:000
414027617,41sleep14155-21modprobe08:36:341
488527522,37sleep04964-21systemctl12:02:020
416927528,28sleep30-21swapper/312:01:363
2855227522,34sleep328475-21latency_hist11:36:363
288582740,9sleep00-21swapper/009:52:040
288582740,9sleep00-21swapper/009:52:040
2482627413,44sleep024966-21unixbench_singl11:22:080
2097327410,49sleep021040-21df11:11:500
2500827321,27sleep00-21swapper/007:57:070
2500827321,27sleep00-21swapper/007:57:070
3224827211,45sleep032292-21cpu08:21:510
3224827211,45sleep032292-21cpu08:21:510
2929827221,29sleep30-21swapper/308:11:563
2929827221,29sleep30-21swapper/308:11:563
2493427214,45sleep124970-21sed09:41:361
2437627214,28sleep30-21swapper/311:21:583
618327116,32sleep10-21swapper/108:41:551
618327116,32sleep10-21swapper/108:41:551
2673027125,26sleep10-21swapper/109:46:491
2673027125,26sleep10-21swapper/109:46:491
2871527016,27sleep00-21swapper/008:11:370
2458127013,39sleep124639-21/usr/sbin/munin11:22:051
2231627015,24sleep30-21swapper/311:16:373
2267126916,27sleep30-21swapper/309:32:053
2267126916,27sleep30-21swapper/309:32:053
2150426918,31sleep321549-21/usr/sbin/munin07:47:013
1554326918,22sleep00-21swapper/007:27:070
67252688,42sleep10-21swapper/110:26:471
514426810,16sleep00-21swapper/008:37:060
2938526814,21sleep10-21swapper/108:11:581
2938526814,21sleep10-21swapper/108:11:581
2728526821,20sleep00-21swapper/011:31:530
1111926813,12sleep30-21swapper/308:57:003
985826717,32sleep39854-21apt-get10:36:473
782326717,35sleep37961-21modprobe07:06:383
669826711,42sleep06700-21acpi10:26:420
159572670,6sleep30-21swapper/310:56:373
768726633,20sleep10-21swapper/107:06:361
3021926614,38sleep030228-21/usr/sbin/munin09:56:580
236052667,15sleep00-21swapper/011:18:520
2091426617,22sleep00-21swapper/007:46:370
849026513,32sleep00-21swapper/010:31:550
648126510,15sleep10-21swapper/110:26:351
648126510,15sleep10-21swapper/110:26:351
22182655,11sleep30-21swapper/311:56:323
22182655,11sleep30-21swapper/311:56:323
2117826510,27sleep10-21swapper/111:11:551
1954926522,18sleep10-21swapper/107:41:501
2791726414,32sleep127302-21/usr/sbin/munin08:07:061
272302646,11sleep30-21swapper/309:47:033
272302646,11sleep30-21swapper/309:47:033
2180726422,25sleep10-21swapper/107:47:071
1709626420,27sleep317129-21modprobe09:16:343
1629426414,23sleep00-21swapper/009:12:030
1583026419,20sleep10-21swapper/109:11:481
1522126410,38sleep015240-21apt-get12:36:350
1459126414,20sleep00-21swapper/010:51:550
1240126418,29sleep012398-21apt-get12:26:450
962226315,24sleep30-21swapper/312:17:113
962226315,24sleep30-21swapper/312:17:113
812626318,26sleep18205-21switchtime08:47:041
812626318,26sleep18205-21switchtime08:47:041
71026319,25sleep3768-21irqstats10:06:543
2862726320,26sleep328651-21munin-run08:11:363
2291626314,22sleep00-21swapper/007:51:580
465026216,30sleep14670-21sed12:01:551
312692620,7sleep30-21swapper/308:17:043
2085426216,28sleep320908-21latency_hist07:46:363
1739626220,21sleep00-21swapper/011:01:360
1652126216,25sleep30-21swapper/307:31:543
1652126216,25sleep30-21swapper/307:31:543
422026111,27sleep34311-21/usr/sbin/munin10:17:043
363526116,28sleep03627-21apt-get10:16:490
339226117,22sleep00-21swapper/008:32:050
2834626111,25sleep30-21swapper/309:51:523
2834626111,25sleep30-21swapper/309:51:523
253792616,11sleep10-21swapper/108:00:111
253792616,11sleep10-21swapper/108:00:111
1983126115,16sleep00-21swapper/007:41:570
1385426117,26sleep00-21swapper/007:22:340
1181226118,21sleep00-21swapper/007:16:570
454426017,23sleep30-21swapper/308:36:523
4312606,42sleep10-21swapper/111:48:301
227912608,14sleep10-21swapper/111:16:561
215932605,10sleep10-21swapper/109:29:201
184226011,25sleep10-21swapper/110:11:371
2923725923,17sleep00-21swapper/011:37:020
261232597,39sleep30-21swapper/309:42:313
1930925920,18sleep00-21swapper/007:41:360
1763225916,26sleep317629-21apt-get11:01:473
1342625912,20sleep00-21swapper/010:47:020
1284525914,31sleep312929-21gpgv07:21:363
90792586,13sleep10-21swapper/107:09:181
2767925820,19sleep10-21swapper/111:32:021
2273225817,19sleep10-21swapper/107:51:551
1477425819,22sleep10-21swapper/109:07:031
1442425815,23sleep30-21swapper/312:32:013
35122575,12sleep10-21swapper/110:16:381
2518025714,27sleep125171-21apt-get09:41:481
2488925721,16sleep30-21swapper/307:57:063
2488925721,16sleep30-21swapper/307:57:063
2287525714,25sleep322914-21irqstats07:51:573
213325713,28sleep02165-21tune2fs10:11:520
184225717,18sleep30-21swapper/308:27:033
184225717,18sleep30-21swapper/308:27:033
1272125712,18sleep30-21swapper/310:46:363
991525613,25sleep310013-21users08:52:073
422025617,19sleep30-21swapper/308:36:363
263342569,20sleep00-21swapper/008:02:120
2615925617,18sleep30-21swapper/311:27:033
1954925616,25sleep119556-21df_inode11:07:311
1608325614,25sleep116075-21apt-get10:56:481
1487425616,24sleep114868-21apt-get07:26:471
1271925618,21sleep10-21swapper/112:27:231
1083625616,23sleep00-21swapper/012:21:570
3022925518,20sleep10-21swapper/109:56:591
2251925518,16sleep00-21swapper/009:32:020
2251925518,16sleep00-21swapper/009:32:020
1800125514,23sleep117995-21apt-get07:36:491
1618725514,21sleep30-21swapper/307:31:383
1437725514,23sleep014470-21perl09:06:540
2026125412,24sleep320270-21munin-run09:26:363
2026125412,24sleep320270-21munin-run09:26:363
447125313,23sleep14534-21df08:36:501
907725214,22sleep19144-21df12:16:491
907725214,22sleep19144-21df12:16:491
791725214,23sleep07941-21ntp_states12:11:590
791725214,23sleep07941-21ntp_states12:11:590
747625213,23sleep17470-21apt-get12:11:481
747625213,23sleep17470-21apt-get12:11:481
3140525213,23sleep131399-21apt-get10:01:471
2578625217,20sleep00-21swapper/009:42:040
2051325212,23sleep020503-21apt-get09:26:490
201725213,23sleep12154-21awk08:27:051
201725213,23sleep12154-21awk08:27:051
1947325213,23sleep019413-21apt-get11:07:260
3269025118,19sleep00-21swapper/010:06:350
2922925114,22sleep329235-21ntp_states11:37:013
2244825115,17sleep10-21swapper/109:31:591
2244825115,17sleep10-21swapper/109:31:591
669025012,22sleep06826-21awk08:42:060
669025012,22sleep06826-21awk08:42:060
2027225013,17sleep00-21swapper/009:26:370
2027225013,17sleep00-21swapper/009:26:370
1650325013,22sleep116509-21diskstats07:31:521
1650325013,22sleep116509-21diskstats07:31:521
144802509,16sleep10-21swapper/110:51:471
1060525015,20sleep110735-21gpgv08:56:371
3039224913,20sleep030400-21df_inode11:41:500
2573624915,17sleep10-21swapper/111:26:531
715324812,18sleep30-21swapper/310:26:583
792724715,15sleep30-21swapper/308:46:583
792724715,15sleep30-21swapper/308:46:583
871324513,16sleep10-21swapper/110:32:001
326882448,13sleep30-21swapper/308:22:023
326882448,13sleep30-21swapper/308:22:023
288212447,14sleep10-21swapper/111:36:511
142572448,16sleep30-21swapper/309:06:453
1765024212,15sleep30-21swapper/309:16:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional