You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 08:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Dec 10, 2025 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1993426517,631sleep319980-21ntpq23:03:063
170625068,483sleep01750-21ntpq23:48:050
170625068,483sleep01750-21ntpq23:48:050
24677248631,437sleep224756-21/usr/sbin/munin19:53:332
16792230921,195sleep316919-21timerwakeupswit22:53:093
1532422709,246sleep315342-21latency_hist21:07:393
17250226916,235sleep117330-21tune2fs21:12:581
3584224121,202sleep13726-21unixbench_singl20:28:161
3584224121,202sleep13726-21unixbench_singl20:28:161
14309224017,206sleep114358-21unin-run22:47:381
13537223640,178sleep3302-21systemd-logind00:27:373
13537223640,178sleep3302-21systemd-logind00:27:373
8458221321,89sleep38614-21unixbench_multi20:43:123
3467220125,160sleep13531-21/usr/sbin/munin22:12:531
1726621965,179sleep00-21swapper/019:31:260
26133219023,67sleep325736-21/usr/sbin/munin23:23:053
26133219023,67sleep325736-21/usr/sbin/munin23:23:053
236321906,173sleep35666-21snmpd22:08:073
1692621859,160sleep00-21swapper/022:53:100
26440218218,51sleep30-21swapper/319:58:303
7076217895,68sleep37161-21mailstats22:23:093
7076217895,68sleep37161-21mailstats22:23:093
32654217862,104sleep332685-21proc_pri20:18:133
32654217862,104sleep332685-21proc_pri20:18:133
6791217442,108sleep36960-21sed18:58:113
12183217318,140sleep112321-21grep19:13:101
22383217132,124sleep322529-21latency_hist19:47:403
12551217114,69sleep312647-21fschecks_time20:57:583
31768216712,81sleep331850-21apt-config23:42:393
3007421647,122sleep330081-21modprobe20:09:373
1658921639,142sleep00-21swapper/021:09:300
12807215411,77sleep312923-21sed22:42:393
3388215325,30sleep30-21swapper/320:28:143
3388215325,30sleep30-21swapper/320:28:143
13159214819,19sleep20-21swapper/222:42:582
18739214711,121sleep218741-21acpi21:17:512
4994214310,117sleep15069-21latency20:33:531
4994214310,117sleep15069-21latency20:33:531
830214045,19sleep30-21swapper/322:03:063
830214045,19sleep30-21swapper/322:03:063
988721399,114sleep30-21swapper/300:13:113
988721399,114sleep30-21swapper/300:13:113
8935213938,86sleep30-21swapper/300:12:393
22425213915,111sleep322493-21latency_hist23:12:393
18107213917,24sleep30-21swapper/321:13:203
23898213512,106sleep023908-21unin-run19:52:380
16924213553,64sleep30-21swapper/321:12:393
1312021337,73sleep313124-21df_inode22:42:553
9888213252,19sleep30-21swapper/322:32:583
5968213168,46sleep30-21swapper/320:37:393
5968213168,46sleep30-21swapper/320:37:393
9026212912,100sleep30-21swapper/322:28:143
9026212912,100sleep30-21swapper/322:28:143
712321266,11sleep30-21swapper/320:38:153
19662212617,84sleep00-21swapper/021:18:140
19224212621,14sleep30-21swapper/319:37:393
10728212614,98sleep210730-21cpu00:17:542
10728212614,98sleep210730-21cpu00:17:542
7330212514,18sleep30-21swapper/300:07:383
2291821245,10sleep00-21swapper/021:29:560
2398521235,13sleep30-21swapper/319:52:413
19842212119,85sleep00-21swapper/019:38:040
3238021199,98sleep332393-21grep23:43:443
16099211914,24sleep30-21swapper/319:27:543
127221190,8sleep00-21swapper/023:47:390
24843211713,83sleep024963-21sort19:53:350
785721157,65sleep30-21swapper/319:02:433
27071211418,79sleep027111-21sed23:27:380
27071211418,79sleep027111-21sed23:27:380
31034211315,84sleep331079-21ntp_offset20:13:043
31034211315,84sleep331079-21ntp_offset20:13:043
27928211328,21sleep328045-21cat23:28:093
27579211330,67sleep30-21swapper/320:02:593
8603211116,78sleep18745-21unixbench_multi19:03:291
10894211122,16sleep30-21swapper/320:52:533
18838211018,18sleep30-21swapper/322:58:503
11207211026,15sleep30-21swapper/319:12:403
11207211026,15sleep30-21swapper/319:12:403
3248721097,88sleep332507-21munin-run22:02:383
3248721097,88sleep332507-21munin-run22:02:383
24291210921,16sleep30-21swapper/323:17:563
24291210921,16sleep30-21swapper/323:17:563
28336210823,14sleep30-21swapper/321:48:003
1481521075,12sleep30-21swapper/300:29:103
22636210521,21sleep322356-1kworker/3:0H23:12:473
22636210521,21sleep322356-1kworker/3:0H23:12:473
21017210514,19sleep30-21swapper/319:42:523
24743210420,14sleep30-21swapper/321:37:393
24743210420,14sleep30-21swapper/321:37:393
1552421047,88sleep0444-1kworker/0:2H22:48:210
11079210216,72sleep111132-21if_err_eth020:52:591
2418021019,74sleep30-21swapper/319:53:183
23169210014,35sleep223249-21awk19:48:082
1913221008,18sleep30-21swapper/321:18:043
154172997,15sleep30-21swapper/322:48:123
618029710,72sleep26198-21apt20:37:502
235832976,77sleep00-21swapper/023:13:170
235832976,77sleep00-21swapper/023:13:170
1450929711,72sleep214572-21mailstats21:03:102
120352979,74sleep112109-21cat00:22:381
120352979,74sleep112109-21cat00:22:381
199322968,17sleep30-21swapper/319:38:083
96952957,27sleep39739-21ntpq20:48:053
46812955,56sleep30-21swapper/322:17:033
1113329513,65sleep011181-21ntp_states00:18:050
1113329513,65sleep011181-21ntp_states00:18:050
255229410,18sleep30-21swapper/320:27:373
2186329427,34sleep20-21swapper/221:27:532
2179529417,60sleep221937-21sed23:08:102
1076429410,69sleep05666-21snmpd22:35:010
3159729313,64sleep031628-21nfs_client21:58:030
3159729313,64sleep031628-21nfs_client21:58:030
227029211,61sleep22305-21ntp_kernel_pll_22:08:022
1378229213,63sleep113906-21latency_hist21:02:401
868529115,58sleep08756-21switchtime22:28:110
868529115,58sleep08756-21switchtime22:28:110
2935529156,15sleep30-21swapper/323:33:063
785829014,56sleep07555-21/usr/sbin/munin00:07:590
254428915,55sleep22550-21modprobe20:27:362
1299928911,59sleep10-21swapper/119:17:551
716528813,61sleep07235-21switchtime22:23:100
716528813,61sleep07235-21switchtime22:23:100
36982880,8sleep00-21swapper/023:53:130
142552887,18sleep314282-21munin-run19:22:373
289128711,56sleep30-21swapper/323:52:483
57372866,68sleep35768-21munin-run00:02:373
3221928620,45sleep131718-21apt-get23:42:491
179228510,59sleep30-21swapper/323:48:083
179228510,59sleep30-21swapper/323:48:083
1171828518,28sleep00-21swapper/019:13:010
116822856,13sleep30-21swapper/319:13:003
3189328425,39sleep031915-21munin-run20:17:370
3189328425,39sleep031915-21munin-run20:17:370
2234428419,44sleep122364-21apt23:12:371
203132848,60sleep30-21swapper/321:22:533
732828332,32sleep27362-21munin-run19:02:382
459828314,21sleep00-21swapper/020:32:430
459828314,21sleep00-21swapper/020:32:430
50152828,59sleep30-21swapper/322:17:523
2786928221,39sleep127892-21netstat20:03:051
1790628224,34sleep117903-21apt-get19:32:491
736128125,29sleep00-21swapper/000:07:390
3170628115,51sleep231767-21apt-config23:42:382
3020428119,39sleep030328-21latency_hist23:37:390
2499728113,50sleep225115-21users23:18:112
2499728113,50sleep225115-21users23:18:112
1585628110,55sleep00-21swapper/019:27:390
1422428113,49sleep214258-21/usr/sbin/munin00:28:042
2633928013,54sleep126418-21sort23:23:111
2633928013,54sleep126418-21sort23:23:111
1777328023,26sleep20-21swapper/222:57:402
472427916,22sleep20-21swapper/220:33:462
472427916,22sleep20-21swapper/220:33:462
307042795,11sleep30-21swapper/321:54:383
230992796,12sleep30-21swapper/321:32:183
157022796,12sleep30-21swapper/319:26:163
117002798,12sleep30-21swapper/300:19:013
117002798,12sleep30-21swapper/300:19:013
875827815,21sleep10-21swapper/122:28:121
875827815,21sleep10-21swapper/122:28:121
1308427818,41sleep213204-21timerwakeupswit20:58:092
2914027714,28sleep10-21swapper/120:08:011
276942776,12sleep30-21swapper/321:46:153
160982777,25sleep20-21swapper/219:27:532
40127612,47sleep0407-21diskstats22:02:550
40127612,47sleep0407-21diskstats22:02:550
259262765,10sleep30-21swapper/321:38:273
2352727614,18sleep00-21swapper/021:32:550
2352727614,18sleep00-21swapper/021:32:550
1663427617,37sleep116074-21/usr/sbin/munin19:28:081
1528327618,43sleep115287-21modprobe21:07:371
1445927612,47sleep014582-21sort00:28:100
940327523,32sleep10-21swapper/120:47:581
349727517,44sleep23566-21switchtime23:53:122
2868827523,27sleep10-21swapper/123:32:391
2948027417,25sleep20-21swapper/221:52:392
796327317,28sleep20-21swapper/200:08:012
1465227218,22sleep00-21swapper/021:03:120
1112727224,29sleep10-21swapper/119:12:381
1112727224,29sleep10-21swapper/119:12:381
1043627215,44sleep110478-21systemd00:17:371
1043627215,44sleep110478-21systemd00:17:371
926327114,35sleep29320-21fschecks_count00:12:582
926327114,35sleep29320-21fschecks_count00:12:582
593127123,24sleep00-21swapper/022:18:130
299022718,13sleep20-21swapper/220:08:152
1361427116,23sleep10-21swapper/100:27:391
1361427116,23sleep10-21swapper/100:27:391
1273827119,34sleep012775-21ps00:23:050
1273827119,34sleep012775-21ps00:23:050
907727020,23sleep20-21swapper/220:47:392
477527017,35sleep24820-21latency_hist22:17:392
2396627012,36sleep223981-21cat23:17:382
2396627012,36sleep223981-21cat23:17:382
1585927019,21sleep10-21swapper/119:27:401
423926924,25sleep14298-21cat23:57:391
2585126917,28sleep20-21swapper/223:23:002
2585126917,28sleep20-21swapper/223:23:002
185126915,39sleep01865-21acpi22:07:480
225026816,35sleep21311-21/usr/sbin/munin23:48:152
225026816,35sleep21311-21/usr/sbin/munin23:48:152
994626721,25sleep20-21swapper/219:07:592
994626721,25sleep20-21swapper/219:07:592
788726716,24sleep10-21swapper/122:27:391
788726716,24sleep10-21swapper/122:27:391
578926718,28sleep10-21swapper/100:02:391
3185226715,27sleep00-21swapper/023:42:400
3170226715,31sleep131703-21cron23:42:371
3131526717,29sleep10-21swapper/120:13:101
3131526717,29sleep10-21swapper/120:13:101
2626626713,36sleep10-21swapper/121:42:371
965326618,22sleep00-21swapper/000:13:080
965326618,22sleep00-21swapper/000:13:080
955026613,24sleep00-21swapper/019:07:440
955026613,24sleep00-21swapper/019:07:440
93602665,11sleep20-21swapper/222:32:162
93602665,11sleep20-21swapper/222:32:162
291492667,11sleep10-21swapper/121:50:001
275722660,7sleep20-21swapper/220:02:572
1515026614,34sleep215223-21/usr/sbin/munin19:23:092
1350226617,27sleep213591-21mailstats19:18:082
1190426621,21sleep10-21swapper/122:38:331
746726517,29sleep07485-21munin-run20:42:360
1974126515,35sleep219822-21latency19:38:022
931926417,26sleep20-21swapper/220:47:552
885526421,26sleep10-21swapper/100:12:361
3024426420,28sleep230313-21/usr/sbin/munin21:53:082
2591626417,23sleep10-21swapper/119:57:481
2108926414,34sleep221098-21df_inode19:42:552
2108926414,34sleep221098-21df_inode19:42:552
1347826415,32sleep013501-21ntp_states22:43:040
1028826417,31sleep210386-21/usr/sbin/munin22:33:132
3149626320,22sleep231522-21cut21:58:002
3149626320,22sleep231522-21cut21:58:002
2785226311,16sleep20-21swapper/221:47:392
2712626315,24sleep00-21swapper/021:43:090
208392639,39sleep020986-21sed23:07:390
1487026315,28sleep10-21swapper/122:48:021
1087026314,20sleep00-21swapper/020:52:500
793626218,21sleep20-21swapper/222:27:402
73212625,17sleep00-21swapper/019:02:310
3194426220,25sleep20-21swapper/220:17:392
3194426220,25sleep20-21swapper/220:17:392
2448626213,23sleep00-21swapper/023:18:010
2448626213,23sleep00-21swapper/023:18:010
2758726113,25sleep00-21swapper/020:03:000
1773726115,32sleep017818-21switchtime21:13:090
539526013,28sleep24456-21/usr/sbin/munin23:58:162
418326014,28sleep04187-21munin-run20:32:390
418326014,28sleep04187-21munin-run20:32:390
2741026014,25sleep00-21swapper/023:27:580
2467526014,32sleep124679-21ntpq23:18:061
2467526014,32sleep124679-21ntpq23:18:061
2383826016,26sleep223877-21grep21:33:012
2383826016,26sleep223877-21grep21:33:012
208012608,12sleep00-21swapper/021:23:100
1931826015,26sleep10-21swapper/121:18:111
1798126020,20sleep20-21swapper/219:33:422
1788326016,27sleep017380-21apt-get19:32:480
1654826016,21sleep10-21swapper/122:53:031
1202226010,30sleep2284-21dbus-daemon22:38:382
2739225914,26sleep227398-21diskstats23:27:562
265925917,25sleep12678-21munin-run23:52:381
265925917,25sleep12678-21munin-run23:52:381
260842596,12sleep00-21swapper/019:58:110
2101925919,23sleep00-21swapper/019:42:530
2101925919,23sleep00-21swapper/019:42:530
1829925921,19sleep10-21swapper/122:58:401
1762525918,20sleep20-21swapper/221:13:072
3276625814,27sleep1387-21ls20:18:151
3276625814,27sleep1387-21ls20:18:151
2966925815,25sleep129666-21apt-get21:52:511
1672325817,20sleep20-21swapper/222:53:082
136742589,13sleep00-21swapper/019:18:100
813725712,23sleep00-21swapper/020:43:050
2935325719,18sleep00-21swapper/021:52:370
2720525714,28sleep227232-21munin-run20:02:372
2307625716,27sleep123168-21mailstats19:48:071
967925622,18sleep10-21swapper/122:32:481
2293825615,21sleep10-21swapper/123:13:011
2293825615,21sleep10-21swapper/123:13:011
1468725618,17sleep20-21swapper/222:47:592
798725513,24sleep18036-21irqstats20:43:021
632925520,21sleep20-21swapper/222:22:392
602725514,24sleep26021-21apt-get00:02:522
2934825514,16sleep20-21swapper/223:33:052
2664025511,31sleep126665-21expr21:42:581
2354025513,24sleep123648-21perl21:32:561
2354025513,24sleep123648-21perl21:32:561
2039425515,22sleep220398-21diskstats21:22:562
2021425515,24sleep120340-21unixbench_multi23:03:121
126842554,11sleep20-21swapper/219:17:192
1146525516,23sleep010980-21apt-get22:37:470
915925416,20sleep20-21swapper/219:07:382
48512548,30sleep04456-21/usr/sbin/munin23:58:060
3216225415,23sleep232181-21apt-get20:17:512
3216225415,23sleep232181-21apt-get20:17:512
2536025420,19sleep20-21swapper/221:38:022
1625025413,24sleep216371-21sed21:08:102
1557525413,23sleep115569-21apt-get21:07:531
1113325417,19sleep20-21swapper/220:53:002
3254225314,18sleep10-21swapper/122:02:401
3254225314,18sleep10-21swapper/122:02:401
3228325314,23sleep232221-21apt-get23:43:312
2625125319,19sleep00-21swapper/021:42:360
158825312,19sleep10-21swapper/122:07:361
158825312,19sleep10-21swapper/122:07:361
129725314,23sleep01278-21apt-get20:23:020
1012925312,15sleep00-21swapper/020:48:140
241572528,32sleep10-21swapper/119:52:591
2052425212,22sleep120581-21irqstats21:22:591
656525113,22sleep26566-21apt-get22:22:522
627125112,23sleep16300-21irqstats00:03:001
2905325112,22sleep029047-21apt-get20:07:560
1796925113,24sleep017460-21apt-get22:57:500
1227425111,24sleep212270-21apt-get00:22:512
1227425111,24sleep212270-21apt-get00:22:512
837625012,24sleep05666-21snmpd19:03:240
1990025013,19sleep20-21swapper/223:03:052
187825014,21sleep11946-21df22:07:531
1245325012,23sleep012449-21apt-get20:57:510
2964724910,18sleep00-21swapper/021:52:500
2889224911,23sleep128887-21apt-get23:32:521
214942499,15sleep00-21swapper/023:08:040
195482499,16sleep10-21swapper/119:37:571
225782488,17sleep10-21swapper/121:28:121
1419624813,15sleep10-21swapper/100:28:031
282824710,22sleep22829-21apt-get20:27:492
282824710,22sleep22829-21apt-get20:27:492
206624713,16sleep10-21swapper/120:23:241
66022468,21sleep16674-21/usr/sbin/munin18:58:091
499224611,22sleep14994-21acpi22:17:501
40782459,15sleep00-21swapper/022:13:090
322772457,18sleep00-21swapper/020:18:020
322772457,18sleep00-21swapper/020:18:020
2113924513,17sleep10-21swapper/119:42:581
2113924513,17sleep10-21swapper/119:42:581
66502426,13sleep10-21swapper/120:38:051
4482419,15sleep20-21swapper/222:02:572
4482419,15sleep20-21swapper/222:02:572
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional