You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 09:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Dec 08, 2025 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1767927449,717sleep317797-21apt-config21:22:463
78192628101,464sleep37833-21fschecks_count19:08:053
164712505123,280sleep316608-21unixbench_multi19:33:173
7421241919,389sleep37435-21awk20:48:173
113792348136,157sleep30-21swapper/322:43:073
113792348136,157sleep30-21swapper/322:43:073
14217231331,72sleep30-21swapper/322:52:573
20575229321,256sleep120590-21fschecks_count19:48:101
3809228270,194sleep33576-21/usr/sbin/munin20:38:073
20624222724,99sleep320630-21fschecks_count23:13:163
20624222724,99sleep320630-21fschecks_count23:13:163
15062222553,112sleep315126-21cat21:13:093
8073221615,182sleep38087-21latency_hist00:17:453
10091220325,139sleep311903-1kworker/3:2H20:58:023
6678219925,157sleep26689-21/usr/sbin/munin19:03:242
6678219925,157sleep26689-21/usr/sbin/munin19:03:242
12682219758,72sleep312685-21cron00:32:433
108821979,149sleep31188-21wc20:28:183
6012219559,123sleep36082-21swap20:44:093
3121721957,177sleep331258-21ntpq22:03:093
3121721957,177sleep331258-21ntpq22:03:093
9870218910,111sleep30-21swapper/300:22:583
20304218517,112sleep320431-21timerwakeupswit21:28:153
4982218420,143sleep15122-21cmp20:42:461
4895218146,18sleep30-21swapper/322:23:083
18703218043,79sleep30-21swapper/323:07:473
8899217767,97sleep38926-21awk20:53:103
24886216335,112sleep324910-21latency_hist20:02:453
8178216122,58sleep37979-21/usr/sbin/munin22:33:423
8178216122,58sleep37979-21/usr/sbin/munin22:33:423
7277215912,87sleep37333-21ntpq19:07:493
7277215912,87sleep37333-21ntpq19:07:493
25711215830,86sleep325859-21latency_hist21:47:463
32052215630,72sleep332191-21grep23:48:173
32052215630,72sleep332191-21grep23:48:173
18112215629,88sleep318126-21apt-get21:22:583
18112215629,88sleep318126-21apt-get21:22:583
17819215630,88sleep317879-21find19:38:143
17819215630,88sleep317879-21find19:38:143
14040215520,77sleep30-21swapper/319:27:483
3384214849,52sleep30-21swapper/320:37:473
3384214849,52sleep30-21swapper/320:37:473
25545214849,37sleep30-21swapper/320:03:183
13657214727,75sleep313658-21ntp_states21:08:123
3076214513,54sleep30-21swapper/322:17:593
13750214548,18sleep30-21swapper/322:50:343
13750214548,18sleep30-21swapper/322:50:343
25232214318,107sleep125271-21cut20:03:121
370721419,53sleep30-21swapper/300:03:033
10948214023,66sleep30-21swapper/319:18:073
10948214023,66sleep30-21swapper/319:18:073
23341213610,68sleep323350-21munin-run23:22:453
23341213610,68sleep323350-21munin-run23:22:453
21705213611,50sleep30-21swapper/319:52:443
2333213411,15sleep30-21swapper/323:58:073
5362213188,20sleep00-21swapper/020:42:570
26687213042,44sleep30-21swapper/320:08:033
27979212942,73sleep327989-21modprobe23:37:433
29560212731,77sleep30-21swapper/321:58:083
7449212620,90sleep27531-21apt-key22:32:442
7449212620,90sleep27531-21apt-key22:32:442
32765212548,57sleep30-21swapper/322:08:103
30094212424,48sleep30-21swapper/323:43:083
27998212346,62sleep028029-21unin-run23:37:440
2858212214,51sleep30-21swapper/322:17:473
12668212220,85sleep112738-21df22:48:021
12668212220,85sleep112738-21df22:48:021
11131212223,77sleep011161-21grep00:27:440
10911212156,49sleep311063-21latency_hist22:42:473
30287211916,80sleep10-21swapper/120:18:141
30287211916,80sleep10-21swapper/120:18:141
2368021179,72sleep323568-21/usr/sbin/munin23:23:053
2368021179,72sleep323568-21/usr/sbin/munin23:23:053
5740211510,39sleep30-21swapper/300:08:163
5740211510,39sleep30-21swapper/300:08:163
20036211410,92sleep00-21swapper/023:10:590
20036211410,92sleep00-21swapper/023:10:590
2433521086,93sleep03713-1kworker/0:1H21:42:540
28882210712,79sleep228958-21/usr/sbin/munin20:13:192
28882210712,79sleep228958-21/usr/sbin/munin20:13:192
12247210714,27sleep112248-21sendmail-mta22:46:081
12247210714,27sleep112248-21sendmail-mta22:46:081
28397210627,62sleep328459-21perl20:13:073
28397210627,62sleep328459-21perl20:13:073
9001210422,64sleep2311-21in:imuxsock19:12:452
8357210410,78sleep08487-21cat20:52:500
9596210315,72sleep19690-21memory19:13:121
9596210315,72sleep19690-21memory19:13:121
4177210212,68sleep04250-21switchtime20:38:170
23156210211,77sleep223158-21sshd19:55:582
26480210113,71sleep226630-21date23:32:462
18117210115,73sleep018246-21awk23:03:160
18117210115,73sleep018246-21awk23:03:160
12764210119,20sleep20-21swapper/200:32:462
24123210024,16sleep30-21swapper/321:42:453
24123210024,16sleep30-21swapper/321:42:453
832529922,14sleep30-21swapper/300:18:013
2751029910,74sleep327681-21apt-key21:52:473
1889729922,16sleep30-21swapper/319:43:033
1889729922,16sleep30-21swapper/319:43:033
797829823,56sleep17485-21apt-get22:32:561
797829823,56sleep17485-21apt-get22:32:561
1641529824,13sleep30-21swapper/322:58:223
1641529824,13sleep30-21swapper/322:58:223
184342975,34sleep30-21swapper/323:05:333
1095929733,49sleep010843-21/usr/sbin/munin19:18:080
1095929733,49sleep010843-21/usr/sbin/munin19:18:080
92832955,34sleep30-21swapper/319:12:503
92832955,34sleep30-21swapper/319:12:503
49972956,13sleep10-21swapper/122:23:101
2233629512,67sleep022394-21memory19:53:090
1816029516,64sleep118207-21df21:23:041
1816029516,64sleep118207-21df21:23:041
44542946,11sleep20-21swapper/218:58:072
3183929413,59sleep031904-21awk23:48:140
3183929413,59sleep031904-21awk23:48:140
2982229423,17sleep30-21swapper/320:18:003
2982229423,17sleep30-21swapper/320:18:003
2645729410,19sleep30-21swapper/321:48:153
1469929411,68sleep014719-21/usr/sbin/munin19:28:300
75322939,23sleep37539-21munin-run22:32:453
75322939,23sleep37539-21munin-run22:32:453
400529111,25sleep00-21swapper/000:03:080
1454929116,62sleep014592-21latency_hist21:12:450
1134829131,40sleep011367-21idleruntime-cro21:02:440
1087029110,63sleep210909-21cat22:42:462
1392129021,53sleep2225-21systemd-journal22:52:432
1392129021,53sleep2225-21systemd-journal22:52:432
619428917,56sleep16337-21awk20:44:111
2257228912,62sleep122683-21cat21:37:471
2257228912,62sleep122683-21cat21:37:471
1576228815,56sleep215770-21/usr/sbin/munin19:32:562
1417928819,51sleep113696-21apt-get19:27:541
866728711,56sleep27979-21/usr/sbin/munin22:33:542
866728711,56sleep27979-21/usr/sbin/munin22:33:542
837528711,63sleep18401-21df00:18:031
313742877,17sleep30-21swapper/320:23:013
313742877,17sleep30-21swapper/320:23:013
1136928715,54sleep11-21systemd21:02:451
594728617,53sleep06037-21latency_hist22:27:460
1760128616,44sleep117627-21munin-run21:22:441
161528413,46sleep11655-21sort22:13:041
1173728414,56sleep011740-21entropy21:03:040
578228318,51sleep15838-21cpuspeed19:03:061
578228318,51sleep15838-21cpuspeed19:03:061
2253828326,45sleep322547-21munin-run21:37:443
2253828326,45sleep322547-21munin-run21:37:443
538428222,40sleep25381-21apt-get20:42:592
318282827,55sleep20-21swapper/220:23:122
318282827,55sleep20-21swapper/220:23:122
2329428229,42sleep323340-21systemctl19:57:453
2053728219,46sleep220549-21/usr/sbin/munin19:48:072
1869428231,24sleep00-21swapper/019:42:470
1869428231,24sleep00-21swapper/019:42:470
336028121,39sleep23381-21latency_hist20:37:462
336028121,39sleep23381-21latency_hist20:37:462
3275528115,48sleep232764-21/usr/sbin/munin22:08:092
442028017,45sleep14453-21irqstats18:58:061
723027911,51sleep27295-21latency20:48:122
2953027922,44sleep327-21ksoftirqd/323:42:443
2929427816,19sleep10-21swapper/121:58:021
2288427818,27sleep20-21swapper/223:18:242
2288427818,27sleep20-21swapper/223:18:242
2087527812,23sleep00-21swapper/023:13:210
2087527812,23sleep00-21swapper/023:13:210
1400227816,44sleep014144-21latency_hist22:52:460
1400227816,44sleep014144-21latency_hist22:52:460
58027725,13sleep30-21swapper/323:53:063
436127729,27sleep00-21swapper/018:58:050
220127713,46sleep12039-21/usr/sbin/munin23:58:041
1192927721,27sleep10-21swapper/100:28:131
483827616,40sleep24604-21/usr/sbin/munin22:23:072
2407127624,34sleep124074-21cron21:42:431
2407127624,34sleep124074-21cron21:42:431
47627514,46sleep0544-21/usr/sbin/munin20:28:030
953627417,29sleep10-21swapper/122:37:571
3263227417,35sleep132634-21cron20:27:431
3263227417,35sleep132634-21cron20:27:431
2789427417,27sleep20-21swapper/221:53:182
201542745,12sleep30-21swapper/323:12:313
201542745,12sleep30-21swapper/323:12:313
1963627419,38sleep219778-21sed23:08:222
1963627419,38sleep219778-21sed23:08:222
127712735,10sleep30-21swapper/321:05:543
1040327316,29sleep10-21swapper/100:23:121
1864427216,22sleep00-21swapper/021:23:240
1864427216,22sleep00-21swapper/021:23:240
3230027117,24sleep10-21swapper/122:07:471
3230027117,24sleep10-21swapper/122:07:471
2521227120,13sleep30-21swapper/323:28:023
1950727124,24sleep10-21swapper/119:43:211
1950727124,24sleep10-21swapper/119:43:211
158622716,22sleep10-21swapper/121:15:211
155192718,49sleep30-21swapper/319:32:453
1217527124,28sleep10-21swapper/119:22:451
1217527124,28sleep10-21swapper/119:22:451
737027024,26sleep10-21swapper/100:13:151
10162709,14sleep20-21swapper/220:28:172
516726917,31sleep25163-21apt-get00:07:592
516726917,31sleep25163-21apt-get00:07:592
3144326916,30sleep0369-21sshd20:23:020
3144326916,30sleep0369-21sshd20:23:020
2982326924,24sleep00-21swapper/023:43:010
179026921,22sleep00-21swapper/022:13:080
1484826914,28sleep20-21swapper/222:53:162
1057626919,32sleep010578-21cron19:17:430
1057626919,32sleep010578-21cron19:17:430
8402687,42sleep10-21swapper/123:53:121
3263126824,28sleep132655-21munin-run23:52:431
3263126824,28sleep132655-21munin-run23:52:431
3030026813,39sleep230353-21/usr/sbin/munin20:18:152
3030026813,39sleep230353-21/usr/sbin/munin20:18:152
215612685,51sleep00-21swapper/019:50:510
1643026819,26sleep20-21swapper/221:18:052
1550826821,26sleep10-21swapper/122:57:431
1154626816,22sleep00-21swapper/000:28:040
271372677,20sleep20-21swapper/221:52:172
262262675,11sleep10-21swapper/123:30:301
535226612,14sleep10-21swapper/100:08:061
535226612,14sleep10-21swapper/100:08:061
344626614,35sleep13477-21/usr/sbin/munin22:18:091
3164126614,34sleep231700-21/usr/sbin/munin23:48:082
3164126614,34sleep231700-21/usr/sbin/munin23:48:082
2756426623,22sleep20-21swapper/223:33:182
2599826612,37sleep026008-21/usr/sbin/munin21:48:030
2484426621,26sleep20-21swapper/220:02:442
1904526617,25sleep20-21swapper/219:43:102
1904526617,25sleep20-21swapper/219:43:102
1866126618,25sleep20-21swapper/223:07:462
1866126618,25sleep20-21swapper/223:07:462
1390226616,27sleep214039-21latency_hist19:27:472
1233826618,29sleep20-21swapper/222:47:182
1233826618,29sleep20-21swapper/222:47:182
1062726616,36sleep210690-21sed19:17:452
1062726616,36sleep210690-21sed19:17:452
1202326517,23sleep20-21swapper/221:03:102
357126414,24sleep20-21swapper/222:18:152
2904826416,32sleep129073-21latency_hist21:57:461
2396326420,21sleep00-21swapper/023:23:110
2396326420,21sleep00-21swapper/023:23:110
2174926416,32sleep221784-21idleruntime-cro23:17:432
2174926416,32sleep221784-21idleruntime-cro23:17:432
95372636,18sleep00-21swapper/020:54:240
2286726317,27sleep122928-21/usr/sbin/munin19:53:191
227526318,26sleep22332-21irqstats23:58:062
215126317,20sleep00-21swapper/023:58:030
199026317,27sleep20-21swapper/222:13:142
182526319,22sleep20-21swapper/223:57:452
1795226315,29sleep118053-21timerandwakeup19:38:161
1795226315,29sleep118053-21timerandwakeup19:38:161
880126214,17sleep10-21swapper/120:53:081
814126221,18sleep00-21swapper/022:33:390
814126221,18sleep00-21swapper/022:33:390
55426218,24sleep10-21swapper/122:08:161
2975226214,34sleep029760-21awk21:58:130
2826726217,29sleep028263-21apt-get23:37:560
246202626,17sleep10-21swapper/123:24:491
246202626,17sleep10-21swapper/123:24:491
1547126218,29sleep215500-21munin-run19:32:442
1479426217,28sleep014787-21apt-get21:13:020
1081926212,36sleep010861-21systemd-run22:42:440
95426123,20sleep10-21swapper/120:28:161
241212618,11sleep10-21swapper/119:58:151
2327326117,22sleep00-21swapper/021:38:140
2327326117,22sleep00-21swapper/021:38:140
2007226115,22sleep10-21swapper/121:28:091
809026019,24sleep00-21swapper/000:17:460
783826021,20sleep00-21swapper/019:08:060
4862608,13sleep00-21swapper/022:08:150
2874826010,12sleep00-21swapper/020:13:170
2874826010,12sleep00-21swapper/020:13:170
2559426014,29sleep025596-21ntp_states23:28:110
1651926015,18sleep00-21swapper/022:58:230
1651926015,18sleep00-21swapper/022:58:230
1616226016,24sleep00-21swapper/019:33:100
93292596,41sleep00-21swapper/000:19:320
3158125920,20sleep20-21swapper/222:03:162
3158125920,20sleep20-21swapper/222:03:162
3020025919,21sleep10-21swapper/123:43:101
2803025915,29sleep128096-21date23:37:451
2717225915,27sleep027236-21mailstats20:08:160
582025815,24sleep00-21swapper/000:08:170
582025815,24sleep00-21swapper/000:08:170
2961325817,15sleep10-21swapper/123:42:461
280972586,11sleep20-21swapper/223:37:462
2514725816,24sleep225215-21df20:03:092
215882586,40sleep10-21swapper/123:15:401
215882586,40sleep10-21swapper/123:15:401
1968025817,25sleep219676-21apt-get21:27:582
1896725816,21sleep00-21swapper/019:43:040
1896725816,21sleep00-21swapper/019:43:040
426925720,20sleep30-21swapper/318:58:043
37302577,31sleep00-21swapper/022:18:170
361125714,25sleep13605-21apt-get00:02:571
3035325719,16sleep20-21swapper/222:00:172
278825713,22sleep10-21swapper/120:33:211
278825713,22sleep10-21swapper/120:33:211
2711725715,26sleep127141-21ntp_states20:08:131
2508625718,21sleep20-21swapper/221:43:172
2491225718,16sleep00-21swapper/020:02:470
2338825716,18sleep10-21swapper/123:22:471
2338825716,18sleep10-21swapper/123:22:471
1859925718,22sleep218605-21ntp_states21:23:222
1859925718,22sleep218605-21ntp_states21:23:222
1381325715,19sleep10-21swapper/119:27:461
695125615,22sleep07087-21sshd22:28:180
695125615,22sleep07087-21sshd22:28:180
641225618,17sleep00-21swapper/019:03:220
641225618,17sleep00-21swapper/019:03:220
627225614,20sleep10-21swapper/122:28:031
627225614,20sleep10-21swapper/122:28:031
3091525613,27sleep030924-21/usr/sbin/munin22:03:020
3091525613,27sleep030924-21/usr/sbin/munin22:03:020
221872565,40sleep00-21swapper/021:33:290
221872565,40sleep00-21swapper/021:33:290
177102567,13sleep20-21swapper/219:38:092
177102567,13sleep20-21swapper/219:38:092
1607525613,22sleep216134-21/usr/sbin/munin22:58:132
1607525613,22sleep216134-21/usr/sbin/munin22:58:132
151625613,26sleep01517-21sshd23:54:280
1111125613,26sleep011104-21apt-get22:43:000
1111125613,26sleep011104-21apt-get22:43:000
722525518,17sleep20-21swapper/200:13:122
3005925515,24sleep230093-21irqstats23:43:072
2836125521,19sleep10-21swapper/120:13:051
2836125521,19sleep10-21swapper/120:13:051
2511125513,25sleep025130-21apt20:02:590
2252025517,24sleep222534-21idleruntime-cro21:37:432
2252025517,24sleep222534-21idleruntime-cro21:37:432
1641725517,15sleep00-21swapper/021:18:030
1359525518,21sleep10-21swapper/121:08:101
1242425514,24sleep012420-21apt-get19:23:000
267125417,17sleep00-21swapper/020:33:200
267125417,17sleep00-21swapper/020:33:200
2489925413,20sleep20-21swapper/223:27:452
2489925413,20sleep20-21swapper/223:27:452
1620725412,26sleep116243-21ntp_states19:33:111
1442725416,24sleep014474-21basename22:53:060
1174125412,21sleep10-21swapper/121:03:051
2664325313,18sleep10-21swapper/121:48:171
2463425314,22sleep124702-21cat21:43:081
1009025312,24sleep210086-21apt-get20:58:012
832425214,22sleep28320-21apt-get00:18:002
2024925222,16sleep10-21swapper/123:12:461
2024925222,16sleep10-21swapper/123:12:461
2769525117,17sleep00-21swapper/021:52:520
2305425113,23sleep223082-21irqstats21:38:082
2305425113,23sleep223082-21irqstats21:38:082
1320325111,23sleep213199-21apt-get21:07:592
1032325112,22sleep210329-21ntp_states00:23:092
3113425013,21sleep031236-21latency_hist20:22:460
3113425013,21sleep031236-21latency_hist20:22:460
47482494,33sleep20-21swapper/220:41:172
773224814,17sleep10-21swapper/119:07:591
178332489,19sleep10-21swapper/123:03:131
178332489,19sleep10-21swapper/123:03:131
1588024812,20sleep115886-21df_inode22:58:081
1588024812,20sleep115886-21df_inode22:58:081
1486724811,21sleep214873-21df_inode21:13:042
108632479,22sleep210861-21apt-get19:17:582
108632479,22sleep210861-21apt-get19:17:582
266862467,20sleep20-21swapper/220:08:022
2043924610,14sleep00-21swapper/021:28:160
881524517,14sleep20-21swapper/220:53:092
71832456,14sleep10-21swapper/120:48:111
406724310,16sleep20-21swapper/200:03:102
2232624110,14sleep00-21swapper/023:18:140
2232624110,14sleep00-21swapper/023:18:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional