You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 04:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Dec 30, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
227302775412,7729sleep021067-21kworker/0:100:12:330
253232297810,2955sleep111197-21kworker/1:200:19:321
273452229312,2269sleep021067-21kworker/0:119:18:330
28179264515,613sleep228212-21/usr/sbin/munin00:29:272
3269324957,455sleep332729-21ntpq22:59:263
21802245610,428sleep021851-21unin-run20:44:020
15746245642,396sleep315836-21fschecks_time20:24:223
24280242639,363sleep324342-21fschecks_time19:09:233
24280242639,363sleep324342-21fschecks_time19:09:233
2742923859,361sleep327572-21latency_hist22:44:033
2742923859,361sleep327572-21latency_hist22:44:033
18165231020,272sleep118214-40ogrotate23:59:011
8431228113,194sleep30-21swapper/319:59:313
8431228113,194sleep30-21swapper/319:59:313
30462225332,97sleep330494-21munin-plugin-st22:54:013
30462225332,97sleep330494-21munin-plugin-st22:54:013
139142248116,21sleep30-21swapper/323:44:213
139142248116,21sleep30-21swapper/323:44:213
258842227102,54sleep30-21swapper/322:39:033
6710220914,136sleep30-21swapper/323:19:333
6710220914,136sleep30-21swapper/323:19:333
13147220535,61sleep30-21swapper/321:59:133
14985220414,97sleep314700-21/usr/sbin/munin22:04:263
30839219811,142sleep330932-21grep22:54:213
29748219616,167sleep229820-21awk19:24:322
29748219616,167sleep229820-21awk19:24:322
11671219322,145sleep111803-21unixbench_multi20:09:381
11671219322,145sleep111803-21unixbench_multi20:09:381
30881218911,121sleep330986-21fschecks_time19:29:223
207521778,133sleep32219-21latency_hist21:24:033
21297217331,117sleep30-21swapper/300:09:003
21297217331,117sleep30-21swapper/300:09:003
19440217290,60sleep30-21swapper/322:19:203
19266216932,124sleep319303-21latency20:34:403
13641216421,123sleep232012-21kworker/u8:223:44:042
13641216421,123sleep232012-21kworker/u8:223:44:042
28918216311,65sleep30-21swapper/322:49:003
21122216127,119sleep321192-21switchtime20:39:333
26116215626,78sleep30-21swapper/319:14:163
22761215610,133sleep322773-21grep22:29:173
22761215610,133sleep322773-21grep22:29:173
23155215524,106sleep00-21swapper/019:04:320
928821548,127sleep00-21swapper/021:44:320
325215314,30sleep30-21swapper/319:34:313
2967215314,123sleep02975-21cpu23:09:160
12234214917,113sleep312252-21awk20:14:003
12234214917,113sleep312252-21awk20:14:003
10475214818,108sleep30-21swapper/323:34:033
15804214747,15sleep30-21swapper/323:49:303
12095214792,17sleep30-21swapper/321:54:363
12798214613,112sleep30-21swapper/320:14:293
26976214531,102sleep326980-21find20:59:163
1237021458,102sleep312388-21fschecks_count23:39:203
5890214418,108sleep16040-21latency_hist23:19:031
17766214490,35sleep30-21swapper/320:29:313
2894721429,122sleep028948-21sshd00:32:320
486021408,73sleep34952-21tune2fs19:49:213
486021408,73sleep34952-21tune2fs19:49:213
26267214011,110sleep326257-21apt-get00:24:193
25185214029,99sleep325262-21cut20:54:033
25185214029,99sleep325262-21cut20:54:033
14476214015,71sleep30-21swapper/320:19:263
25088213711,27sleep30-21swapper/300:19:263
9821213629,73sleep39820-21munin-run21:49:033
23922213533,16sleep30-21swapper/300:14:383
6317213411,28sleep30-21swapper/319:54:183
6317213411,28sleep30-21swapper/319:54:183
16914213119,96sleep30-21swapper/323:54:183
16728213111,90sleep30-21swapper/322:09:303
1498212917,90sleep30-21swapper/323:04:193
9439212822,81sleep19475-21cpu20:04:171
9439212822,81sleep19475-21cpu20:04:171
743821268,80sleep37475-21ps21:39:303
1845721248,58sleep318579-21cat22:14:313
1845721248,58sleep318579-21cat22:14:313
3598212322,83sleep13635-21ntp_kernel_err19:44:261
3498212111,16sleep30-21swapper/319:44:243
28021212121,15sleep30-21swapper/300:29:243
9941211933,38sleep310017-21sendmail_mailtr20:04:313
9941211933,38sleep310017-21sendmail_mailtr20:04:313
21336211810,13sleep00-21swapper/000:09:010
21336211810,13sleep00-21swapper/000:09:010
1677211714,87sleep11146-21apt-get19:39:131
11364211712,21sleep30-21swapper/321:54:043
9177211513,84sleep09245-21df23:29:200
1818211522,53sleep30-21swapper/319:39:563
30655211449,26sleep330747-21apt-key00:39:013
2159021148,16sleep30-21swapper/300:09:173
29240211019,59sleep229384-21tail21:04:402
2902621098,43sleep30-21swapper/321:04:373
23257210720,11sleep30-21swapper/320:47:533
14811210713,79sleep014827-21uniq22:04:230
10809210724,28sleep10-21swapper/123:34:211
5127210512,75sleep15253-21/usr/sbin/munin23:14:341
32338210511,14sleep00-21swapper/019:34:200
23484210414,75sleep023683-21latency_hist20:49:030
517210332,17sleep3526-21munin-run21:19:023
16358210210,11sleep20-21swapper/222:09:212
2660521019,81sleep00-21swapper/020:58:040
22688210116,68sleep122694-21diskstats19:04:211
12164210117,71sleep012240-21sed21:54:370
841210013,68sleep2845-21/usr/sbin/munin21:19:182
16019210012,70sleep016127-21timerwakeupswit23:49:350
13225210015,67sleep113250-21df_inode21:59:191
280732987,29sleep30-21swapper/319:19:293
280732987,29sleep30-21swapper/319:19:293
3240929717,66sleep132427-21fschecks_count22:59:201
2924129720,14sleep30-21swapper/319:24:173
2924129720,14sleep30-21swapper/319:24:173
385729620,14sleep30-21swapper/321:29:163
47562957,47sleep30-21swapper/323:14:243
47562957,47sleep30-21swapper/323:14:243
2246129520,59sleep10-21swapper/120:44:281
304929417,13sleep30-21swapper/323:09:193
607729319,48sleep06162-21cat19:54:020
607729319,48sleep06162-21cat19:54:020
297242936,12sleep00-21swapper/000:34:300
3203629217,39sleep20-21swapper/219:34:012
307629210,17sleep30-21swapper/321:24:363
344929018,54sleep13511-21sendmail_mailst23:09:301
267528922,48sleep02693-21idleruntime-cro23:09:000
263742890,7sleep00-21swapper/000:24:230
1574228916,58sleep115765-21/usr/sbin/munin23:49:281
307832888,64sleep330914-21grep21:09:343
2789728720,47sleep127952-21fschecks_count00:29:221
2764128720,49sleep00-21swapper/022:44:130
2417928715,53sleep024216-21grep20:49:440
2417928715,53sleep024216-21grep20:49:440
822428617,49sleep18238-21apt21:44:001
2344628514,50sleep023601-21grep22:29:350
2344628514,50sleep023601-21grep22:29:350
3208628315,45sleep00-21swapper/022:59:030
256322838,18sleep325684-21if_eth020:54:233
2544328330,24sleep10-21swapper/119:14:011
2544328330,24sleep10-21swapper/119:14:011
665428217,45sleep26680-21modprobe21:39:002
665428217,45sleep26680-21modprobe21:39:002
2283528219,27sleep10-21swapper/122:29:191
2283528219,27sleep10-21swapper/122:29:191
76628014,15sleep00-21swapper/021:19:160
3083628016,28sleep00-21swapper/019:29:190
2962428016,47sleep229660-21cat00:34:282
2768028011,19sleep20-21swapper/219:19:152
2768028011,19sleep20-21swapper/219:19:152
1952428028,22sleep20-21swapper/222:19:232
189992808,57sleep319090-21mailstats23:59:313
759827916,41sleep30-21swapper/323:24:103
43427922,26sleep10-21swapper/119:34:331
2272427920,44sleep322818-21fw_packets19:04:233
191927911,48sleep21995-21/usr/sbin/munin23:04:302
1807427818,43sleep218122-21latency22:14:222
1807427818,43sleep218122-21latency22:14:222
298827727,25sleep20-21swapper/219:44:032
3232427616,41sleep232458-21sed21:14:332
3098927614,45sleep131023-21/usr/sbin/munin22:54:231
1403727617,25sleep10-21swapper/123:44:231
1403727617,25sleep10-21swapper/123:44:231
1207227611,28sleep20-21swapper/221:54:352
1074327615,43sleep210752-21latency_hist20:09:032
1074327615,43sleep210752-21latency_hist20:09:032
504427520,27sleep20-21swapper/219:49:242
504427520,27sleep20-21swapper/219:49:242
391127515,47sleep23935-21df_inode21:29:192
2897227518,27sleep20-21swapper/222:49:032
351227413,44sleep23586-21wc23:09:312
2639427421,29sleep10-21swapper/122:39:251
2639427421,29sleep10-21swapper/122:39:251
2307027431,24sleep20-21swapper/219:04:302
2052727413,46sleep120596-21readlink00:04:301
2052727413,46sleep120596-21readlink00:04:301
1992427419,15sleep00-21swapper/022:19:330
1858027420,33sleep118601-21fschecks_count23:59:211
1851127413,19sleep20-21swapper/220:34:032
1372627415,44sleep213787-21/usr/sbin/munin21:59:322
764927315,17sleep20-21swapper/221:39:332
1691627325,29sleep00-21swapper/023:54:190
1613627314,44sleep016203-21run-parts20:24:320
1454427214,42sleep114563-21ntp_states20:19:281
117322725,10sleep30-21swapper/323:35:573
506227119,21sleep00-21swapper/023:14:330
506227119,21sleep00-21swapper/023:14:330
223372715,55sleep30-21swapper/322:27:153
223372715,55sleep30-21swapper/322:27:153
1330527113,24sleep20-21swapper/220:14:402
27462699,14sleep10-21swapper/121:24:301
2679026915,23sleep20-21swapper/222:39:342
2679026915,23sleep20-21swapper/222:39:342
1684026919,23sleep10-21swapper/122:09:321
1344826916,22sleep00-21swapper/020:14:420
120072699,12sleep00-21swapper/020:11:030
120072699,12sleep00-21swapper/020:11:030
84526810,50sleep00-21swapper/023:00:200
2970826820,30sleep129747-21/usr/sbin/munin19:24:311
2970826820,30sleep129747-21/usr/sbin/munin19:24:311
2939326819,24sleep10-21swapper/122:49:221
2939326819,24sleep10-21swapper/122:49:221
266242688,13sleep00-21swapper/022:39:320
266242688,13sleep00-21swapper/022:39:320
189142686,10sleep10-21swapper/122:16:231
189142686,10sleep10-21swapper/122:16:231
1666326813,41sleep016698-21ntp_kernel_pll_22:09:270
1267726821,24sleep10-21swapper/123:39:261
1196826818,32sleep011997-21munin-run23:39:010
769526727,20sleep00-21swapper/019:59:030
65312675,51sleep30-21swapper/321:37:243
65312675,51sleep30-21swapper/321:37:243
1620426722,27sleep10-21swapper/120:24:331
3056926611,39sleep030633-21ps21:09:300
2967126617,25sleep00-21swapper/022:49:300
2967126617,25sleep00-21swapper/022:49:300
1817426619,22sleep00-21swapper/022:14:240
1817426619,22sleep00-21swapper/022:14:240
900726520,24sleep10-21swapper/121:44:291
287522655,48sleep00-21swapper/019:21:230
287522655,48sleep00-21swapper/019:21:230
172226516,26sleep00-21swapper/019:39:450
1061526514,37sleep210696-21switchtime21:49:302
1005926534,17sleep00-21swapper/021:49:160
80092646,17sleep00-21swapper/021:41:130
2756226422,23sleep20-21swapper/200:29:022
2648626416,26sleep20-21swapper/219:14:272
2647526424,21sleep10-21swapper/100:24:251
2505026420,27sleep00-21swapper/022:34:400
2395926415,34sleep123962-21diskstats20:49:201
2395926415,34sleep123962-21diskstats20:49:201
1989626414,31sleep219909-21awk20:34:512
1746926414,28sleep20-21swapper/223:54:342
102622644,47sleep10-21swapper/123:31:551
337526313,37sleep23376-21sshd21:26:332
1989026317,26sleep10-21swapper/122:19:301
1719526312,33sleep017265-21df20:29:160
1108326314,33sleep211127-21ntpq23:34:262
691726218,28sleep16991-21/usr/sbin/munin19:54:321
691726218,28sleep16991-21/usr/sbin/munin19:54:321
3037026212,35sleep230384-21kernelversion21:09:242
2985326217,20sleep00-21swapper/021:09:020
2784926220,21sleep10-21swapper/122:44:221
2169426216,24sleep10-21swapper/122:24:391
2169426216,24sleep10-21swapper/122:24:391
2609926114,31sleep126205-21ls20:54:331
2185226121,21sleep10-21swapper/120:44:031
1930626115,27sleep019365-21memory20:34:410
1844726117,19sleep10-21swapper/120:34:021
1299626116,29sleep10-21swapper/120:14:351
124526114,29sleep11311-21wc21:19:291
2322226016,23sleep20-21swapper/200:14:242
217372605,16sleep20-21swapper/220:43:112
1224126020,23sleep10-21swapper/121:54:381
807725912,30sleep28083-21ntp_states23:24:252
273732595,12sleep10-21swapper/119:18:541
2607425917,18sleep00-21swapper/000:24:040
2135125917,19sleep20-21swapper/200:09:032
2135125917,19sleep20-21swapper/200:09:032
135382599,25sleep00-21swapper/023:44:020
640125823,19sleep20-21swapper/219:54:212
640125823,19sleep20-21swapper/219:54:212
448325814,27sleep04514-21munin-run19:49:020
356825819,21sleep00-21swapper/021:29:010
2448225814,29sleep024493-21latency_hist00:19:030
2419725812,28sleep024267-21df19:09:200
2419725812,28sleep024267-21df19:09:200
2303025822,19sleep20-21swapper/222:29:242
2303025822,19sleep20-21swapper/222:29:242
2121925815,18sleep00-21swapper/022:24:180
2121925815,18sleep00-21swapper/022:24:180
1479525815,27sleep214801-21df_inode22:04:202
1145825818,25sleep20-21swapper/220:09:352
1145825818,25sleep20-21swapper/220:09:352
769525715,25sleep17704-21df_inode23:24:161
70622578,10sleep20-21swapper/223:20:252
70622578,10sleep20-21swapper/223:20:252
531725720,18sleep10-21swapper/119:49:311
531725720,18sleep10-21swapper/119:49:311
3241125712,19sleep20-21swapper/219:34:222
2928725717,24sleep229293-21diskstats22:49:192
2928725717,24sleep229293-21diskstats22:49:192
2921725713,30sleep0369-21snmpd19:24:140
2921725713,30sleep0369-21snmpd19:24:140
2580525713,29sleep125829-21munin-run22:39:011
1408325712,20sleep20-21swapper/220:19:122
71932565,12sleep00-21swapper/023:22:050
71932565,12sleep00-21swapper/023:22:050
567525617,25sleep05706-21latency21:34:260
567525617,25sleep05706-21latency21:34:260
2542625612,17sleep20-21swapper/220:54:182
2158925613,25sleep221585-21apt-get00:09:162
1418025615,22sleep014186-21df_inode20:19:190
1054325619,19sleep10-21swapper/121:49:291
682525518,15sleep00-21swapper/019:54:310
453225517,24sleep04655-21sed21:29:330
3224625517,19sleep10-21swapper/121:14:321
2748825517,19sleep10-21swapper/120:59:341
2449625514,24sleep124639-21cut00:19:041
2209025517,22sleep222087-21apt-get20:44:162
149772559,34sleep00-21swapper/023:47:370
149772559,34sleep00-21swapper/023:47:370
876625421,18sleep20-21swapper/219:59:342
876625421,18sleep20-21swapper/219:59:342
56712546,11sleep20-21swapper/223:17:422
56712546,11sleep20-21swapper/223:17:422
465625414,24sleep14761-21users21:29:341
3078325412,25sleep130778-21apt-get19:29:161
3017325414,25sleep130179-21df_inode21:09:191
2900725417,16sleep10-21swapper/121:04:351
2437025419,19sleep20-21swapper/220:49:502
2437025419,19sleep20-21swapper/220:49:502
168742546,36sleep20-21swapper/220:28:372
834225313,24sleep08348-21ntp_states19:59:280
834225313,24sleep08348-21ntp_states19:59:280
762725313,24sleep07694-21df23:24:150
586625311,28sleep15880-21awk21:34:311
586625311,28sleep15880-21awk21:34:311
3204325317,19sleep00-21swapper/021:14:270
2956625315,16sleep10-21swapper/100:34:271
2472125313,23sleep224714-21apt-get00:19:172
198062539,20sleep20-21swapper/200:04:032
1875625316,21sleep00-21swapper/023:59:240
1586825314,18sleep20-21swapper/223:49:332
943825213,23sleep09432-21apt-get20:04:160
943825213,23sleep09432-21apt-get20:04:160
862425214,16sleep10-21swapper/119:59:331
862425214,16sleep10-21swapper/119:59:331
2767725215,21sleep10-21swapper/119:19:141
2767725215,21sleep10-21swapper/119:19:141
2480225213,17sleep20-21swapper/219:09:342
2480225213,17sleep20-21swapper/219:09:342
2175725213,26sleep121815-21basename00:09:231
200662525,10sleep10-21swapper/120:36:521
1391725221,16sleep00-21swapper/021:59:340
742425115,18sleep10-21swapper/123:24:041
742425115,18sleep10-21swapper/123:24:041
317502516,10sleep20-21swapper/222:55:412
2708025113,24sleep227086-21diskstats20:59:232
2451725112,23sleep224039-21apt-get22:34:092
2700425019,18sleep00-21swapper/020:59:210
908924913,16sleep20-21swapper/221:44:302
1482824911,13sleep10-21swapper/122:04:241
944124814,18sleep20-21swapper/223:29:262
141124811,21sleep11414-21apt-get23:04:131
3243024714,14sleep20-21swapper/222:59:212
376024611,18sleep20-21swapper/219:44:312
1283924413,17sleep20-21swapper/223:39:312
2140124310,18sleep20-21swapper/222:24:322
2140124310,18sleep20-21swapper/222:24:322
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional