You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-11 - 20:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Mon May 11, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18570273416,698sleep018627-21/usr/sbin/munin08:54:570
19849241610,313sleep119988-21apt-key08:59:381
10159239872,142sleep110285-21latency_hist11:54:381
27551228761,133sleep127698-21unixbench_singl09:20:181
26733226484,107sleep126876-21unixbench_singl07:35:091
8192225963,181sleep17274-21/usr/sbin/munin11:45:091
8192225963,181sleep17274-21/usr/sbin/munin11:45:091
30462225671,120sleep130607-21latency_hist07:49:391
15055223612,210sleep015071-21awk12:09:470
21620223557,95sleep10-21swapper/109:04:391
17443223440,109sleep117585-21latency_hist10:34:381
23006222846,167sleep123020-21fschecks_count07:24:551
3208222279,196sleep132210-21timerwakeupswit09:35:081
29651222111,189sleep129765-21latency_hist09:29:391
1786222038,118sleep11-21systemd09:44:121
13619221711,135sleep1363-21snmpd12:04:561
13619221711,135sleep1363-21snmpd12:04:561
12452221615,187sleep012512-21sendmail_mailqu12:00:060
12452221615,187sleep012512-21sendmail_mailqu12:00:060
22824221531,170sleep122861-21idleruntime-cro12:34:361
10170221323,42sleep10-21swapper/110:10:001
10170221323,42sleep10-21swapper/110:10:001
9249220834,109sleep19357-21cut08:24:571
18046220817,178sleep218103-21awk07:09:582
14761220514,130sleep114800-21idleruntime10:24:571
23503220491,92sleep10-21swapper/109:09:561
22128219711,173sleep122197-21sort09:04:591
8110219514,107sleep18258-21latency_hist10:04:391
22684218892,52sleep122797-21latency_hist07:24:391
21332218613,83sleep10-21swapper/112:29:391
6484218188,72sleep16522-21processes08:15:031
1040217671,24sleep10-21swapper/107:59:361
10490217515,139sleep310634-21cat08:29:393
7284217325,127sleep27343-21sendmail-msp08:19:362
19033216030,86sleep119153-21latency_hist10:39:391
11657215014,121sleep211689-21idleruntime-cro11:59:352
16017214947,28sleep10-21swapper/108:45:081
30467214811,65sleep10-21swapper/111:14:561
23055214560,16sleep10-21swapper/110:50:091
8858214454,76sleep18874-21idleruntime-cro08:24:361
6016214027,94sleep16079-21latency11:40:001
6016214027,94sleep16079-21latency11:40:001
22841213911,115sleep222901-21processes10:50:062
2112321378,47sleep10-21swapper/110:45:001
2698213458,15sleep10-21swapper/111:29:541
2698213458,15sleep10-21swapper/111:29:541
26331213118,28sleep20-21swapper/209:19:382
26331213118,28sleep20-21swapper/209:19:382
19435212748,65sleep10-21swapper/107:14:411
13586212616,84sleep10-21swapper/108:39:391
1252212628,86sleep11290-21ls11:24:571
2358621249,101sleep023643-21processes12:35:080
1116521247,104sleep111202-21ps08:30:061
18923212314,93sleep318925-21ntp_states12:20:043
2606212116,78sleep10-21swapper/109:45:051
4972212030,75sleep15052-21mailstats08:10:061
19733212014,48sleep10-21swapper/110:40:031
1078721200,37sleep310791-21unixbench_singl10:10:093
1078721200,37sleep310791-21unixbench_singl10:10:093
27536211932,65sleep10-21swapper/111:05:061
11930211937,15sleep10-21swapper/110:15:041
26272211829,74sleep126326-21systemd-run09:19:371
26272211829,74sleep126326-21systemd-run09:19:371
3663211732,50sleep13701-21cpu09:49:531
5604211529,18sleep10-21swapper/109:55:021
19362211524,21sleep10-21swapper/112:20:101
5429211412,85sleep15457-21modprobe11:39:351
5429211412,85sleep15457-21modprobe11:39:351
2338621145,100sleep00-21swapper/010:51:250
1841211419,12sleep10-21swapper/108:00:051
23922211313,74sleep10-21swapper/110:54:531
12923211329,47sleep112928-21run-parts12:00:351
12923211329,47sleep112928-21run-parts12:00:351
12017211330,62sleep212090-21switchtime10:15:052
17352211224,73sleep10-21swapper/112:15:221
29250211129,20sleep10-21swapper/111:10:111
14335211030,20sleep114344-21unin-run10:24:371
28809210916,75sleep228323-21apt-get11:09:482
26737210727,22sleep1363-21snmpd11:04:181
22922210718,68sleep222976-21cpuspeed07:24:522
29888210518,18sleep10-21swapper/109:29:511
4853210318,64sleep34995-21grep11:35:093
4853210318,64sleep34995-21grep11:35:093
3381210327,40sleep10-21swapper/108:05:061
13824210318,17sleep10-21swapper/108:39:511
18298210225,18sleep118332-21ps07:10:051
19992210017,66sleep220184-21date08:59:392
15082210022,15sleep10-21swapper/112:09:521
3114629811,46sleep10-21swapper/107:50:031
1371429713,68sleep213751-21idleruntime12:04:572
1371429713,68sleep213751-21idleruntime12:04:572
2210429623,51sleep022241-21head07:20:100
1696929618,54sleep016982-21grep12:15:080
1651929612,71sleep216520-21sshd08:48:122
3106229535,42sleep330669-21/usr/sbin/munin07:50:013
254302945,26sleep10-21swapper/107:30:451
1978429428,42sleep219780-21apt-get07:14:502
1837129442,27sleep20-21swapper/210:35:082
3265029313,64sleep032763-21sed11:20:080
2028829225,48sleep319768-21apt-get08:59:483
2471729114,59sleep224729-21systemd09:14:372
243042915,29sleep10-21swapper/112:38:441
1414429110,64sleep014247-21grep12:05:070
1414429110,64sleep014247-21grep12:05:070
888129030,23sleep18952-21sensors_temp10:05:071
447229011,65sleep20-21swapper/208:09:512
2677928937,23sleep00-21swapper/009:19:480
2728028828,40sleep027291-21modprobe07:39:360
2287828816,18sleep00-21swapper/012:34:390
1152928813,61sleep311552-21fschecks_count10:14:543
112992885,39sleep10-21swapper/111:55:101
884728713,59sleep38895-21grep11:49:513
2253928721,41sleep222685-21unixbench_multi09:05:082
422828622,28sleep20-21swapper/208:09:392
3237628615,52sleep232421-21/usr/sbin/munin11:20:022
2318128520,40sleep00-21swapper/009:09:390
2057828518,49sleep20-21swapper/212:25:082
1035928518,32sleep20-21swapper/210:10:052
1035928518,32sleep20-21swapper/210:10:052
862528416,19sleep20-21swapper/211:49:382
862528416,19sleep20-21swapper/211:49:382
1973928420,43sleep319770-21idleruntime-cro08:59:363
687928315,45sleep36945-21fschecks_time09:59:563
289002837,65sleep128906-21munin-run07:44:381
1670928340,25sleep20-21swapper/207:05:062
1670928340,25sleep20-21swapper/207:05:062
1435528321,42sleep014403-21/usr/sbin/munin08:40:060
2998528116,41sleep230092-21grep09:29:562
2914828111,14sleep00-21swapper/007:44:530
2767528115,29sleep30-21swapper/307:39:553
2595528111,53sleep325977-21/usr/sbin/munin11:00:043
248732810,6sleep00-21swapper/009:14:380
2419128112,50sleep024250-21/usr/sbin/munin10:55:000
3085028015,42sleep330364-21/usr/sbin/munin11:15:063
2569728017,46sleep025840-21sed09:15:080
2569728017,46sleep025840-21sed09:15:080
247728016,46sleep02526-21sed09:45:000
2445027915,19sleep20-21swapper/212:39:382
2107327915,49sleep221220-21awk09:00:282
1573127911,53sleep015732-21sshd10:28:110
1096627913,21sleep30-21swapper/308:30:003
843627814,45sleep38496-21/usr/sbin/munin10:04:573
610427822,38sleep06168-21sort08:14:540
324412788,33sleep10-21swapper/111:20:051
1991027812,50sleep020023-21timerwakeupswit10:40:060
175127816,40sleep01915-21grep11:25:090
173412776,31sleep10-21swapper/108:50:011
138192779,15sleep30-21swapper/308:39:493
2473027623,28sleep30-21swapper/309:14:383
1685027628,22sleep00-21swapper/007:05:080
1685027628,22sleep00-21swapper/007:05:080
267527527,27sleep20-21swapper/208:04:382
1814327521,37sleep318175-21basename12:19:363
3047427424,29sleep20-21swapper/211:14:572
2860927418,42sleep228655-21sed09:24:592
2842627423,27sleep30-21swapper/309:24:563
2770327413,29sleep20-21swapper/209:20:192
268022746,41sleep30-21swapper/309:20:003
2447127419,34sleep324614-21latency_hist12:39:393
2439827422,32sleep20-21swapper/210:55:062
1647727420,36sleep216660-21apt-config12:14:382
143512749,15sleep30-21swapper/310:24:393
425327314,42sleep24285-21fschecks_count11:34:562
425327314,42sleep24285-21fschecks_count11:34:562
299627314,17sleep00-21swapper/011:30:010
299627314,17sleep00-21swapper/011:30:010
1115527319,27sleep00-21swapper/008:30:050
820927219,26sleep00-21swapper/008:20:070
2830927216,40sleep328350-21http11:09:363
1675227224,19sleep30-21swapper/307:05:073
1675227224,19sleep30-21swapper/307:05:073
530927014,38sleep05397-21/usr/sbin/munin09:54:570
3270127019,27sleep20-21swapper/207:55:052
3152027018,28sleep00-21swapper/009:34:540
2577127013,41sleep225781-21munin-run07:34:372
2058027017,24sleep30-21swapper/310:44:383
2015327016,23sleep00-21swapper/007:15:010
1846627013,27sleep00-21swapper/012:19:530
1754727016,29sleep217679-21wc08:50:062
1854426914,40sleep218548-21diskstats12:19:552
679626816,36sleep06866-21/usr/sbin/munin09:59:530
34942689,44sleep33583-21cat08:05:083
2645426818,21sleep30-21swapper/309:19:393
2645426818,21sleep30-21swapper/309:19:393
1107826819,30sleep30-21swapper/311:55:083
960826719,30sleep20-21swapper/210:09:362
493026714,40sleep34938-21ntpq08:10:043
45832676,11sleep00-21swapper/009:50:130
387726720,25sleep20-21swapper/211:34:352
387726720,25sleep20-21swapper/211:34:352
319002677,13sleep20-21swapper/209:35:052
1710426710,27sleep00-21swapper/008:49:560
1652026628,23sleep10-21swapper/107:05:011
1652026628,23sleep10-21swapper/107:05:011
163882665,10sleep00-21swapper/008:46:280
133426619,29sleep01353-21apt-get07:59:470
311012656,12sleep00-21swapper/007:50:020
2082426515,24sleep20-21swapper/210:44:532
135772657,41sleep20-21swapper/210:20:062
947126419,21sleep20-21swapper/211:50:062
584226414,32sleep35963-21timerwakeupswit09:55:083
3244826412,38sleep332505-21awk07:54:583
3114926416,24sleep30-21swapper/309:34:373
2735926418,26sleep20-21swapper/207:39:382
2213526413,34sleep322255-21date10:49:383
1450226420,19sleep20-21swapper/208:40:082
3059726317,27sleep00-21swapper/011:14:590
2741826314,36sleep027444-21netstat11:05:010
2198726319,24sleep30-21swapper/307:20:093
1805926316,20sleep30-21swapper/310:35:013
1640226310,28sleep20-21swapper/210:29:592
407226214,29sleep34114-21/usr/sbin/munin09:50:033
2796726219,21sleep20-21swapper/207:40:032
2043926213,30sleep320561-21wc07:15:083
1792326215,21sleep00-21swapper/007:09:560
102026217,20sleep00-21swapper/009:40:030
706226117,23sleep20-21swapper/209:59:592
62726112,34sleep2634-21df_inode09:39:532
289732616,14sleep00-21swapper/011:10:060
229782617,13sleep30-21swapper/307:24:533
1964926118,23sleep20-21swapper/210:40:002
1173826116,21sleep00-21swapper/011:59:370
82826019,20sleep30-21swapper/309:39:583
705826017,29sleep27118-21latency_hist11:44:372
705826017,29sleep27118-21latency_hist11:44:372
347826014,27sleep30-21swapper/309:49:393
2796226010,37sleep2369-21sshd11:05:122
2630026010,35sleep326384-21memory07:34:593
2031126013,25sleep00-21swapper/009:00:010
173842608,11sleep20-21swapper/212:15:232
169652607,12sleep00-21swapper/010:30:090
163126017,22sleep30-21swapper/307:59:593
1195126018,25sleep311970-21apt-get11:59:493
1195126018,25sleep311970-21apt-get11:59:493
1120526014,23sleep20-21swapper/208:30:072
103522605,17sleep00-21swapper/008:28:560
358925917,23sleep00-21swapper/008:05:090
2805125918,25sleep028058-21ntp_states07:40:050
2686025913,31sleep227006-21latency_hist11:04:392
2222725915,19sleep30-21swapper/309:05:013
2183825916,19sleep20-21swapper/212:30:012
1350425914,30sleep313534-21munin-run08:39:373
1206125920,22sleep20-21swapper/211:59:562
1206125920,22sleep20-21swapper/211:59:562
497625815,20sleep20-21swapper/209:54:392
2896025814,19sleep30-21swapper/307:44:403
1791425815,28sleep317921-21/usr/sbin/munin07:09:553
1490125818,18sleep30-21swapper/312:09:383
126702589,34sleep00-21swapper/008:35:050
1144625810,12sleep00-21swapper/010:14:510
845325711,27sleep28506-21users08:20:092
3115425713,19sleep20-21swapper/207:50:042
2179525715,25sleep221827-21ntp_states07:20:042
1376425716,19sleep00-21swapper/010:20:080
103222578,33sleep010351-21ntp_states10:10:030
103222578,33sleep010351-21ntp_states10:10:030
1014825711,31sleep010158-21munin-run11:54:370
890925615,19sleep30-21swapper/308:24:383
655525612,30sleep26635-21latency_hist09:59:392
3183925615,19sleep30-21swapper/309:35:023
1621925615,26sleep316290-21cut10:29:563
157142566,17sleep30-21swapper/310:27:583
3044325516,16sleep00-21swapper/009:30:060
2047925510,25sleep00-21swapper/012:25:060
106825512,26sleep31061-21apt-get11:24:513
727325414,24sleep07292-21apt-get11:44:470
727325414,24sleep07292-21apt-get11:44:470
446825414,23sleep04464-21apt-get08:09:490
3205525414,24sleep332059-21diskstats11:19:543
263525415,16sleep30-21swapper/311:29:523
263525415,16sleep30-21swapper/311:29:523
2490225411,26sleep024926-21ntp_states07:30:030
2415725413,25sleep324189-21irqstats10:54:593
2065625413,21sleep30-21swapper/312:25:093
954825316,17sleep20-21swapper/208:25:022
871925316,15sleep00-21swapper/010:05:020
609825314,23sleep36102-21diskstats08:14:533
574325312,25sleep35740-21apt-get11:39:503
574325312,25sleep35740-21apt-get11:39:503
2833225315,23sleep028335-21apt-get09:24:490
1691925316,19sleep30-21swapper/308:49:513
256132525,11sleep30-21swapper/307:33:093
1498225215,17sleep20-21swapper/210:25:032
2549225115,22sleep00-21swapper/010:59:490
241212515,37sleep00-21swapper/007:28:540
22092515,10sleep20-21swapper/211:28:052
628125014,19sleep00-21swapper/011:40:070
628125014,19sleep00-21swapper/011:40:070
342025012,21sleep20-21swapper/209:49:382
212602509,17sleep00-21swapper/010:45:030
135302505,24sleep30-21swapper/312:04:493
135302505,24sleep30-21swapper/312:04:493
2157024911,22sleep321637-21df12:29:543
1586024910,16sleep30-21swapper/308:45:063
729724813,18sleep30-21swapper/311:44:503
729724813,18sleep30-21swapper/311:44:503
3267024710,20sleep032692-21ntp_states07:55:030
2912524712,20sleep329218-21memory11:10:093
2244824715,16sleep30-21swapper/310:49:543
1298324712,19sleep20-21swapper/208:35:122
3013524612,17sleep20-21swapper/207:46:132
464024410,20sleep04678-21processes11:35:060
464024410,20sleep04678-21processes11:35:060
176882429,18sleep017683-21apt-get10:34:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional