You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-07 - 16:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sun Jun 07, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1447723848,362sleep114538-21if_eth009:13:221
16584238312,354sleep016601-21cron07:38:000
28486237820,277sleep128527-21grep08:13:301
28486237820,277sleep128527-21grep08:13:301
507723547,310sleep15113-21ntpq12:09:001
1044623300,28sleep16066-21cyclictest08:58:311
8121230890,159sleep18177-21fschecks_count08:53:191
3056229799,149sleep13127-21true08:38:011
7370228414,255sleep27404-21/usr/sbin/munin10:33:252
29160227988,171sleep129250-21fschecks_time11:43:201
1727022767,258sleep117308-21ntpq07:38:251
7478225748,21sleep10-21swapper/108:49:361
7478225748,21sleep10-21swapper/108:49:361
24019223343,140sleep124060-21ntpq09:43:251
15962222950,162sleep116071-21fschecks_time09:18:181
18846222246,54sleep10-21swapper/107:43:261
552622199,195sleep35537-21awk10:28:173
18186220911,77sleep118322-21unixbench_singl09:23:391
18186220911,77sleep118322-21unixbench_singl09:23:391
1168220750,33sleep10-21swapper/110:13:251
3107220316,133sleep13129-21kernelversion12:03:231
21475220016,123sleep10-21swapper/111:18:211
21475220016,123sleep10-21swapper/111:18:211
980021987,181sleep1363-21snmpd10:39:251
2682321977,178sleep126848-21grep09:53:181
19425219711,171sleep019433-21ntpq09:28:260
2122621960,7sleep00-21swapper/011:18:030
2122621960,7sleep00-21swapper/011:18:030
31915219417,125sleep10-21swapper/111:53:011
21775219422,46sleep1284-21dbus-daemon09:38:001
1282121798,103sleep112846-21fschecks_count09:08:211
5430217855,22sleep10-21swapper/108:43:301
5430217855,22sleep10-21swapper/108:43:301
13637217637,78sleep10-21swapper/107:28:321
26034216616,27sleep20-21swapper/211:33:202
26034216616,27sleep20-21swapper/211:33:202
1862216646,44sleep10-21swapper/111:58:301
17912216616,130sleep217980-21sshd09:23:362
17912216616,130sleep217980-21sshd09:23:362
11581216361,58sleep10-21swapper/112:28:301
323921567,137sleep13249-21vmstat10:18:341
26844215612,131sleep126890-21sed08:08:251
32411214619,48sleep132439-21munin-run08:28:001
20912214650,75sleep10-21swapper/107:48:351
15727214616,69sleep10-21swapper/110:58:311
12188214211,113sleep212253-21grep10:48:222
6372214190,16sleep15866-21apt-get07:08:051
22806214131,52sleep122818-21modprobe07:57:591
22806214131,52sleep122818-21modprobe07:57:591
853213911,54sleep20-21swapper/208:28:292
24357213914,106sleep10-21swapper/108:02:361
15453213815,54sleep115477-21grep07:33:251
5946213711,86sleep16023-21cat10:28:271
3926213731,18sleep10-21swapper/108:38:291
20167213718,51sleep10-21swapper/111:13:281
20167213718,51sleep10-21swapper/111:13:281
29289213523,96sleep329326-21idleruntime-cro08:18:003
29289213523,96sleep329326-21idleruntime-cro08:18:003
2580321359,74sleep125909-21ls09:48:301
5844213340,64sleep15926-21apt-config07:08:011
12069213213,17sleep10-21swapper/107:24:041
12069213213,17sleep10-21swapper/107:24:041
2339421318,84sleep123416-21latency_hist09:43:011
13880213141,36sleep10-21swapper/110:53:261
29580213011,104sleep129583-21cron10:02:591
5323212514,96sleep35479-21uptime07:03:313
27784212320,81sleep30-21swapper/311:38:213
3113021219,60sleep131191-21systemd-run10:08:001
4492212015,84sleep00-21swapper/010:23:290
4492212015,84sleep00-21swapper/010:23:290
2864121206,77sleep10-21swapper/111:41:181
24893212011,51sleep124912-21apt-get08:03:121
6449211838,50sleep26402-21apt-get07:08:422
1908721188,11sleep10-21swapper/111:08:481
1994121167,11sleep10-21swapper/109:29:201
946721157,11sleep10-21swapper/107:15:481
2968621148,94sleep129581-21/usr/sbin/munin08:18:191
2968621148,94sleep129581-21/usr/sbin/munin08:18:191
567821125,94sleep00-21swapper/007:05:540
14081211013,81sleep214192-21sed10:53:312
779210937,23sleep10-21swapper/108:28:281
2793321090,6sleep30-21swapper/308:13:033
8337210845,51sleep18419-21switchtime12:18:331
6726210721,70sleep36796-21/usr/sbin/munin12:13:303
30978210617,73sleep031002-21netstat11:48:240
10227210629,56sleep10-21swapper/107:18:221
10227210629,56sleep10-21swapper/107:18:221
9585210539,44sleep10-21swapper/112:23:211
32243210516,67sleep032339-21fschecks_time11:53:200
32243210516,67sleep032339-21fschecks_time11:53:200
809521047,85sleep10-21swapper/110:35:541
1232521049,48sleep1363-21snmpd09:06:431
1232521049,48sleep1363-21snmpd09:06:431
7960210213,73sleep07752-21/usr/sbin/munin12:18:230
2341210122,47sleep02337-21apt-get10:18:130
10669210116,66sleep210813-21/usr/sbin/munin07:18:322
10669210116,66sleep210813-21/usr/sbin/munin07:18:322
29691210021,55sleep229705-21grep08:18:202
29691210021,55sleep229705-21grep08:18:202
3165329916,66sleep031156-21apt-get10:08:110
3165329916,66sleep031156-21apt-get10:08:110
48982976,81sleep10-21swapper/110:24:441
48982976,81sleep10-21swapper/110:24:441
2724229722,56sleep327335-21mailstats09:53:293
2719329713,62sleep1284-21dbus-daemon11:37:591
2719329713,62sleep1284-21dbus-daemon11:37:591
786829518,62sleep37957-21fschecks_time12:18:223
3263629544,12sleep10-21swapper/111:53:301
3263629544,12sleep10-21swapper/111:53:301
1714529513,65sleep30-21swapper/309:23:003
1173129528,48sleep211229-21apt-get07:23:122
1173129528,48sleep211229-21apt-get07:23:122
1011329515,58sleep39933-21/usr/sbin/munin07:18:203
1011329515,58sleep39933-21/usr/sbin/munin07:18:203
316222947,34sleep10-21swapper/110:08:041
316222947,34sleep10-21swapper/110:08:041
627529315,55sleep10-21swapper/112:13:201
2438629329,48sleep10-21swapper/111:28:141
228262930,17sleep221268-21kworker/u8:207:57:592
228262930,17sleep221268-21kworker/u8:207:57:592
1118729340,13sleep10-21swapper/110:43:331
1322929217,24sleep10-21swapper/110:53:021
3182929017,52sleep231630-21/usr/sbin/munin10:09:232
3182929017,52sleep231630-21/usr/sbin/munin10:09:232
2283229022,51sleep10-21swapper/111:23:131
2987128916,21sleep00-21swapper/010:03:150
73762874,11sleep30-21swapper/307:09:123
3175628718,48sleep031837-21cat08:23:300
2778528715,55sleep227932-21cut08:13:032
592728631,33sleep26028-21apt-config07:08:022
544328516,53sleep05501-21cpuspeed10:28:140
429028517,52sleep04507-21sed12:08:020
292328511,57sleep22965-21fschecks_count12:03:192
753828324,33sleep20-21swapper/212:18:012
2048628311,56sleep020568-21mailstats07:48:300
1311028320,43sleep013251-21apt-key07:28:020
1311028320,43sleep013251-21apt-key07:28:020
1293328310,21sleep10-21swapper/112:33:221
1062328319,51sleep010598-21apt-get12:28:000
307392820,9sleep20-21swapper/210:03:322
2870028225,28sleep00-21swapper/008:13:330
2870028225,28sleep00-21swapper/008:13:330
254828221,40sleep22574-21latency_hist12:02:592
1045728212,55sleep010506-21grep10:43:200
850128121,33sleep20-21swapper/208:53:282
803828118,49sleep28099-21cpu07:13:072
881527914,47sleep38282-21apt-get10:38:143
2970827915,45sleep029834-21/usr/sbin/munin11:43:310
1492827917,39sleep315134-21gpgv07:33:023
1138927914,28sleep00-21swapper/012:28:250
2535627821,42sleep225398-21awk08:03:502
1977527817,25sleep20-21swapper/211:13:172
1977527817,25sleep20-21swapper/211:13:172
1631127818,47sleep216338-21munin-plugin-st11:03:002
85127716,26sleep30-21swapper/310:13:193
417327719,40sleep34285-21apt-config12:08:013
2591127722,33sleep226049-21unixbench_singl09:48:312
252162777,57sleep30-21swapper/308:03:473
2036627713,46sleep320396-21/usr/sbin/munin07:48:253
1554227715,30sleep30-21swapper/307:33:273
1428327740,19sleep00-21swapper/009:13:160
1399127722,30sleep00-21swapper/009:13:000
1354227739,20sleep00-21swapper/007:28:190
679827633,26sleep00-21swapper/012:13:310
313627618,38sleep33259-21latency_hist08:38:033
2469727622,39sleep024836-21cmp08:03:030
1669027612,44sleep20-21swapper/211:03:202
734127515,42sleep37388-21users08:48:333
734127515,42sleep37388-21users08:48:333
2236227519,37sleep222421-21latency09:38:222
204852756,11sleep00-21swapper/009:33:070
1555227512,42sleep315643-21mailstats10:58:293
1472527512,17sleep30-21swapper/310:58:003
98992745,17sleep00-21swapper/010:40:410
767327419,34sleep07692-21munin-run07:13:000
3276727420,35sleep3416-21timerandwakeup11:53:323
3276727420,35sleep3416-21timerandwakeup11:53:323
2807727420,36sleep028202-21cat09:58:020
2649327426,26sleep026524-21cut09:53:010
850727313,43sleep08688-21sshd10:38:030
1778927319,43sleep10-21swapper/111:07:071
1652527322,24sleep00-21swapper/009:18:290
2288527214,17sleep30-21swapper/307:58:013
2288527214,17sleep30-21swapper/307:58:013
2141127216,23sleep20-21swapper/209:33:332
843627118,33sleep30-21swapper/308:53:253
3036027110,46sleep20-21swapper/211:48:012
501027025,28sleep35043-21grep08:43:203
501027025,28sleep35043-21grep08:43:203
2647227014,21sleep00-21swapper/008:08:160
2388327017,35sleep223912-21irqstats09:43:222
1007027015,40sleep010077-21ntpq08:58:270
480226917,28sleep00-21swapper/012:08:550
3127826919,20sleep30-21swapper/310:08:023
2758526922,32sleep227594-21diskstats11:38:162
2292226916,28sleep20-21swapper/211:23:192
2256926920,29sleep00-21swapper/009:38:280
102926918,23sleep20-21swapper/210:13:222
221792689,45sleep322245-21fschecks_time09:38:193
1698826821,25sleep00-21swapper/007:38:200
2840026715,23sleep00-21swapper/009:58:180
2787926723,24sleep00-21swapper/011:38:220
2320826714,37sleep023210-21diskstats07:58:190
175326721,28sleep30-21swapper/311:58:283
3215426617,31sleep232150-21apt-get11:53:142
3215426617,31sleep232150-21apt-get11:53:142
291232665,51sleep20-21swapper/208:15:532
291232665,51sleep20-21swapper/208:15:532
243026618,20sleep30-21swapper/310:18:203
2107826619,27sleep321224-21latency_hist11:18:033
2107826619,27sleep321224-21latency_hist11:18:033
1923626614,38sleep319267-21/usr/sbin/munin09:28:213
1854726622,18sleep00-21swapper/007:43:210
1318126618,25sleep00-21swapper/010:53:010
2358926515,20sleep30-21swapper/307:58:303
1700526514,22sleep00-21swapper/011:03:260
1320226514,38sleep213248-21processes09:08:302
1143726520,24sleep30-21swapper/309:03:223
1143726520,24sleep30-21swapper/309:03:223
17742645,11sleep00-21swapper/010:14:540
1580026415,30sleep215864-21ps07:33:342
841926313,29sleep30-21swapper/310:38:023
246422637,13sleep00-21swapper/009:44:420
2398626315,28sleep00-21swapper/011:26:360
1798126323,23sleep30-21swapper/309:23:373
1798126323,23sleep30-21swapper/309:23:373
1383926318,29sleep013878-21ntpq10:53:250
1312426320,21sleep20-21swapper/212:33:272
863726226,20sleep00-21swapper/008:53:300
847126222,23sleep00-21swapper/007:13:210
474926213,20sleep20-21swapper/212:08:412
3186426220,26sleep331870-21modprobe11:52:593
2727226220,19sleep30-21swapper/311:38:013
2727226220,19sleep30-21swapper/311:38:013
216326213,25sleep00-21swapper/008:33:240
1660626215,27sleep316674-21df11:03:173
1485526218,28sleep214927-21wc09:13:312
1304126215,34sleep313079-21munin-run07:28:013
1304126215,34sleep313079-21munin-run07:28:013
1149426215,29sleep211535-21/usr/sbin/munin12:28:272
882726113,33sleep38833-21ntp_states07:13:283
34572616,13sleep20-21swapper/208:38:182
1712726116,29sleep217144-21modprobe09:22:592
929826021,20sleep20-21swapper/212:23:102
842126022,20sleep20-21swapper/212:18:342
263052605,12sleep30-21swapper/309:51:103
1890626015,31sleep318912-21ntp_states07:43:283
1184926017,23sleep011976-21unixbench_multi09:03:310
1184926017,23sleep011976-21unixbench_multi09:03:310
439925916,25sleep34490-21cut10:23:283
439925916,25sleep34490-21cut10:23:283
2377125916,20sleep20-21swapper/207:58:332
2364025918,25sleep323635-21apt-get09:43:153
1852725918,25sleep218534-21diskstats07:43:192
181532597,11sleep30-21swapper/307:42:563
1317625913,32sleep313178-21cron10:53:003
1266225918,26sleep012657-21apt-get12:33:130
1087925917,24sleep20-21swapper/210:43:302
882825812,27sleep28810-21apt-get10:38:212
628925816,23sleep20-21swapper/208:48:032
628925816,23sleep20-21swapper/208:48:032
2905925815,26sleep329055-21apt-get11:43:143
2884725816,22sleep30-21swapper/309:58:293
1805925818,16sleep00-21swapper/009:23:380
1805925818,16sleep00-21swapper/009:23:380
95525717,25sleep30-21swapper/308:28:303
779625717,18sleep00-21swapper/008:53:020
779625717,18sleep00-21swapper/008:53:020
355625715,27sleep03617-21cat08:38:200
2436125714,25sleep224380-21apt-get11:28:112
223125716,19sleep20-21swapper/208:33:262
2162325712,17sleep00-21swapper/007:53:180
2162325712,17sleep00-21swapper/007:53:180
1504725715,25sleep015040-21apt-get10:58:130
1473625715,25sleep314742-21ntp_states09:13:283
1125825715,25sleep011286-21munin-run07:23:020
1125825715,25sleep011286-21munin-run07:23:020
1058425715,20sleep30-21swapper/312:27:593
3060525614,25sleep230601-21apt-get11:48:132
2997325612,29sleep330002-21/usr/sbin/munin08:18:253
2997325612,29sleep330002-21/usr/sbin/munin08:18:253
217092569,32sleep221744-21ntp_states11:18:262
217092569,32sleep221744-21ntp_states11:18:262
1743725615,20sleep20-21swapper/207:38:302
63622555,38sleep20-21swapper/210:28:322
2174525517,22sleep321751-21ntp_states11:18:273
2174525517,22sleep321751-21ntp_states11:18:273
1392325515,24sleep313995-21systemd-run12:38:003
531125411,23sleep30-21swapper/312:09:063
242525414,24sleep22429-21diskstats10:18:192
514925312,21sleep20-21swapper/210:28:002
514925312,21sleep20-21swapper/210:28:002
334625313,25sleep33365-21apt-get08:38:113
2008425317,16sleep20-21swapper/207:48:202
1825225317,15sleep20-21swapper/211:08:212
127925314,22sleep01274-21apt-get11:58:130
1123925315,22sleep211248-21df_inode09:03:172
1123925315,22sleep211248-21df_inode09:03:172
106892538,27sleep30-21swapper/309:00:363
61025212,23sleep0650-21ntpq08:28:230
3011125219,19sleep00-21swapper/008:18:300
3011125219,19sleep00-21swapper/008:18:300
2059325212,23sleep320599-21diskstats09:33:183
492725112,23sleep04995-21df08:43:170
492725112,23sleep04995-21df08:43:170
3244725116,18sleep20-21swapper/208:28:012
2637525111,22sleep026391-21ntp_states11:33:260
2637525111,22sleep026391-21ntp_states11:33:260
1111425113,22sleep311107-21apt-get12:28:153
681525012,22sleep26854-21memory08:48:232
681525012,22sleep26854-21memory08:48:232
2569525014,17sleep00-21swapper/011:33:010
1896425014,21sleep218960-21apt-get09:28:122
1815425014,22sleep018149-21apt-get11:08:150
3157924911,17sleep30-21swapper/308:23:253
2831124914,21sleep228308-21apt-get09:58:122
2684924910,16sleep20-21swapper/209:53:192
238892495,35sleep30-21swapper/311:25:213
2203424916,14sleep00-21swapper/011:18:320
2203424916,14sleep00-21swapper/011:18:320
1418924811,22sleep214318-21sed07:28:452
2487224713,19sleep30-21swapper/311:28:283
1835624616,16sleep30-21swapper/311:08:223
1036924512,16sleep30-21swapper/310:43:153
1001324512,14sleep20-21swapper/208:58:252
1313424311,19sleep313179-21ntpq09:08:273
5542419,13sleep30-21swapper/310:13:033
5542419,13sleep30-21swapper/310:13:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional