You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-13 - 20:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Mon Apr 13, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
885125688,550sleep313441-21snmpd09:07:443
885125688,550sleep313441-21snmpd09:07:443
11932252611,500sleep311934-21apt-get11:01:363
29321245394,242sleep329464-21unixbench_singl10:11:573
2447823917,372sleep324492-21grep11:41:423
2447823917,372sleep324492-21grep11:41:423
1901123880,15sleep31648-21cyclictest07:57:243
2440238243,321sleep32472-21fschecks_count10:31:443
318423737,312sleep33265-21tune2fs12:16:443
10028234017,307sleep310093-21cpuspeed07:31:403
15654226038,201sleep315740-21latency_hist09:31:263
15654226038,201sleep315740-21latency_hist09:31:263
19436225759,142sleep319460-21modprobe11:26:243
129202238106,18sleep30-21swapper/309:21:493
5039223123,196sleep35075-21ntpq12:21:493
21971222210,158sleep322045-21cat08:06:543
21971222210,158sleep322045-21cat08:06:543
2780222013,192sleep32819-21systemctl12:16:243
30243220912,145sleep330257-21fschecks_count10:16:423
2065220711,117sleep32103-21modprobe10:31:243
11694220734,132sleep311857-21grep09:16:573
11694220734,132sleep311857-21grep09:16:573
12069220612,139sleep311578-21/usr/sbin/munin07:36:533
18993220528,46sleep30-21swapper/309:41:363
4019219622,155sleep30-21swapper/310:36:453
2227221910,47sleep222277-21unixbench_singl08:06:562
2227221910,47sleep222277-21unixbench_singl08:06:562
1260721869,164sleep012630-21sh09:21:420
2342521847,62sleep323471-21switchtime11:36:583
13584217717,56sleep30-21swapper/311:06:453
13584217717,56sleep30-21swapper/311:06:453
31670217388,26sleep30-21swapper/308:37:513
2647721739,17sleep30-21swapper/311:47:043
1669721728,34sleep30-21swapper/311:16:393
561216685,23sleep3649-21mailstats08:41:513
561216685,23sleep3649-21mailstats08:41:513
7539216525,108sleep37630-21sed07:21:553
8392215890,47sleep30-21swapper/312:31:593
8392215890,47sleep30-21swapper/312:31:593
29878215724,116sleep329952-21switchtime11:56:533
29878215724,116sleep329952-21switchtime11:56:533
32320214813,105sleep30-21swapper/310:21:553
2823214816,117sleep02861-21processes10:31:530
626521478,31sleep30-21swapper/312:26:413
626521478,31sleep30-21swapper/312:26:413
1626221455,31sleep30-21swapper/311:15:463
1626221455,31sleep30-21swapper/311:15:463
1840121449,28sleep30-21swapper/309:37:123
30955214312,27sleep331045-21fschecks_time12:01:423
974421407,32sleep30-21swapper/310:55:183
1955214012,113sleep21964-21sh12:11:502
32495213630,94sleep332530-21fschecks_count12:06:423
24503213617,45sleep30-21swapper/308:16:353
14935213530,92sleep314926-21apt-get07:46:383
6289213221,36sleep30-21swapper/307:16:583
6271213233,15sleep30-21swapper/310:41:583
3464213036,13sleep30-21swapper/308:51:493
23024212811,102sleep323119-21timerwakeupswit09:51:583
20645212818,21sleep20-21swapper/211:27:042
9615212719,54sleep30-21swapper/307:29:503
653921278,104sleep328860-1kworker/3:2H09:01:463
1586212518,87sleep11636-21latencyplot07:06:261
14753212516,92sleep114793-21modprobe11:11:241
14753212516,92sleep114793-21modprobe11:11:241
1009321257,81sleep310216-21cat09:11:563
1009321257,81sleep310216-21cat09:11:563
26164212329,82sleep326302-21grep10:01:563
27754212210,96sleep327896-21awk10:06:563
27754212210,96sleep327896-21awk10:06:563
23362212025,14sleep30-21swapper/308:11:503
1101321196,13sleep00-21swapper/009:16:410
1101321196,13sleep00-21swapper/009:16:410
7031211813,88sleep30-21swapper/310:46:423
10646211811,91sleep310724-21mailstats10:56:523
19859211710,21sleep30-21swapper/308:01:453
13502211712,87sleep013565-21/usr/sbin/munin11:06:420
13502211712,87sleep013565-21/usr/sbin/munin11:06:420
25851211618,22sleep00-21swapper/010:01:480
13219211621,36sleep30-21swapper/307:41:423
5945211520,13sleep30-21swapper/309:00:213
1558321150,6sleep20-21swapper/211:11:522
1558321150,6sleep20-21swapper/211:11:522
2644721138,93sleep326052-21/usr/sbin/munin08:21:483
2763221118,16sleep30-21swapper/308:26:413
27842210916,15sleep30-21swapper/311:51:443
1081321080,6sleep20-21swapper/209:16:262
1081321080,6sleep20-21swapper/209:16:262
26756210756,34sleep226881-21/usr/sbin/munin08:21:552
309210614,77sleep0341-21nfs_client12:06:470
1970221068,14sleep30-21swapper/311:26:363
18082210623,48sleep218163-21cat09:36:592
8243210519,27sleep20-21swapper/207:26:262
3780210412,76sleep13801-21apt-get07:11:351
22317210420,65sleep021846-21apt-get09:51:350
2134721046,14sleep30-21swapper/309:46:573
1244721049,81sleep012454-21ntpq11:02:220
30525210217,63sleep330625-21latency_hist08:36:263
30525210217,63sleep330625-21latency_hist08:36:263
2570210121,51sleep22713-21grep07:06:562
2570210121,51sleep22713-21grep07:06:562
16756210013,70sleep216836-21fw_packets11:16:422
401429914,31sleep30-21swapper/307:11:453
2630129918,58sleep026391-21grep11:47:010
249542996,11sleep00-21swapper/009:58:210
2367229924,48sleep123815-21unixbench_singl08:11:571
2225129911,74sleep10-21swapper/111:32:481
757529818,57sleep07578-21cron09:06:230
690229813,66sleep17046-21/usr/sbin/munin12:26:561
690229813,66sleep17046-21/usr/sbin/munin12:26:561
296832987,76sleep00-21swapper/010:14:200
225252985,11sleep20-21swapper/211:36:202
1619429832,46sleep016220-21grep07:51:240
2784329715,66sleep227901-21irqstats08:26:482
1993429718,62sleep019703-21/usr/sbin/munin11:26:450
1154629710,71sleep111615-21sensors_temp09:16:551
1154629710,71sleep111615-21sensors_temp09:16:551
857629618,64sleep18642-21sort10:51:411
828529618,60sleep08311-21modprobe10:51:240
295429613,64sleep22331-21/usr/sbin/munin10:31:552
1154529630,29sleep10-21swapper/111:01:261
717529515,63sleep27222-21users09:01:572
557229315,65sleep05574-21sshd12:22:100
241942929,47sleep324267-21sed09:56:473
169182927,29sleep30-21swapper/309:33:243
3057329116,58sleep130575-21cron12:01:231
3057329116,58sleep130575-21cron12:01:231
1762229011,65sleep017688-21fw_packets09:36:490
163202907,29sleep30-21swapper/307:51:273
285762898,61sleep20-21swapper/210:11:402
30522885,30sleep30-21swapper/307:10:443
30522885,30sleep30-21swapper/307:10:443
28022886,29sleep30-21swapper/308:50:113
2114128814,61sleep121225-21cat09:46:551
1462828821,43sleep014422-21apt-get07:46:270
3271528713,17sleep30-21swapper/307:01:363
2617728739,29sleep126174-21apt-get11:46:371
2535828719,49sleep025432-21/usr/sbin/munin08:16:560
224312876,29sleep30-21swapper/311:35:083
3062828612,58sleep030831-21cat08:36:270
3062828612,58sleep030831-21cat08:36:270
1905028610,14sleep00-21swapper/007:57:540
608028514,18sleep00-21swapper/009:01:270
3262228517,48sleep132637-21cut08:41:411
3262228517,48sleep132637-21cut08:41:411
2799328412,55sleep128008-21uname11:51:461
254332840,7sleep10-21swapper/108:16:561
1051228413,54sleep010594-21mailstats07:31:530
298628313,49sleep12912-21latency_hist08:51:261
2833928325,27sleep10-21swapper/110:11:271
2833928325,27sleep10-21swapper/110:11:271
918428212,50sleep215044-1kworker/2:0H09:11:262
918428212,50sleep215044-1kworker/2:0H09:11:262
680028213,20sleep00-21swapper/012:26:550
680028213,20sleep00-21swapper/012:26:550
656028214,53sleep26604-21ntpq12:26:492
656028214,53sleep26604-21ntpq12:26:492
3043528222,34sleep20-21swapper/208:36:252
3043528222,34sleep20-21swapper/208:36:252
2116128226,29sleep00-21swapper/008:06:260
3096428121,37sleep030475-21apt-get08:36:350
2586828119,38sleep226018-21latency_hist08:21:272
2331628117,22sleep20-21swapper/208:11:492
1982328117,43sleep219334-21apt-get08:01:362
201628014,50sleep12111-21mailstats12:11:531
1948828028,29sleep119461-21latency_hist11:26:261
163728021,27sleep00-21swapper/008:46:410
1168928024,25sleep20-21swapper/207:36:442
96312798,60sleep013441-21snmpd10:53:500
3168127912,51sleep031700-21apt-get10:21:350
314372790,9sleep10-21swapper/112:01:511
440727720,29sleep00-21swapper/012:21:250
2106127711,45sleep021140-21/usr/sbin/munin09:46:540
1245527719,44sleep112492-21ps11:02:231
2417527616,38sleep024320-21latency_hist11:41:270
2984427516,42sleep129864-21modprobe10:16:231
2772327517,37sleep027777-21grep08:26:460
556527418,29sleep20-21swapper/210:41:442
3264027426,24sleep20-21swapper/208:41:422
3264027426,24sleep20-21swapper/208:41:422
3207527418,41sleep132120-21ntp_kernel_err10:21:481
1689627423,21sleep00-21swapper/011:16:440
1418427416,30sleep10-21swapper/109:26:281
1418427416,30sleep10-21swapper/109:26:281
1952527317,29sleep20-21swapper/208:01:272
679127219,30sleep00-21swapper/010:46:270
450427214,26sleep20-21swapper/210:36:562
336827221,35sleep23406-21latency12:16:472
160382728,14sleep00-21swapper/009:31:440
664727118,34sleep16662-21apt07:21:241
1965527125,26sleep20-21swapper/209:41:552
216127015,26sleep20-21swapper/208:46:552
1921026911,37sleep019269-21if_eth009:41:450
617326819,20sleep20-21swapper/207:16:572
46462679,13sleep20-21swapper/212:21:402
3113226719,32sleep231226-21memory12:01:452
2828726713,29sleep20-21swapper/210:11:232
2828726713,29sleep20-21swapper/210:11:232
2321726716,26sleep00-21swapper/008:11:470
1648626718,25sleep10-21swapper/107:51:411
1575726720,21sleep00-21swapper/011:11:550
1575726720,21sleep00-21swapper/011:11:550
1440826713,41sleep114450-21apt-get07:46:241
1128226720,25sleep20-21swapper/209:16:472
1128226720,25sleep20-21swapper/209:16:472
961626621,25sleep10-21swapper/109:11:461
961626621,25sleep10-21swapper/109:11:461
406826613,21sleep00-21swapper/010:36:460
2740926623,29sleep00-21swapper/010:06:490
2740926623,29sleep00-21swapper/010:06:490
2274026618,18sleep00-21swapper/008:11:270
2274026618,18sleep00-21swapper/008:11:270
2022626616,20sleep10-21swapper/111:26:541
130626630,19sleep00-21swapper/010:26:560
673726519,30sleep16771-21awk10:46:241
2810026515,23sleep10-21swapper/108:26:551
158902657,39sleep10-21swapper/109:31:371
1381326514,31sleep113837-21munin-run09:26:231
79432646,14sleep00-21swapper/007:23:130
3160026415,28sleep231669-21/usr/sbin/munin08:37:502
895126318,25sleep10-21swapper/107:26:531
2701726310,39sleep113441-21snmpd08:22:181
239326317,23sleep00-21swapper/007:06:540
239326317,23sleep00-21swapper/007:06:540
2122626324,20sleep20-21swapper/209:46:562
1985726319,23sleep10-21swapper/108:01:431
916526220,18sleep00-21swapper/007:26:570
2770526213,33sleep227724-21apt-get11:51:352
2531826215,22sleep20-21swapper/211:42:062
2531826215,22sleep20-21swapper/211:42:062
2212926218,27sleep122271-21sed08:06:561
2212926218,27sleep122271-21sed08:06:561
1445126216,27sleep214537-21apt-config07:46:252
3107026116,31sleep131137-21df08:37:371
3055926115,25sleep10-21swapper/110:16:481
2588526117,26sleep125934-21/usr/sbin/munin10:01:491
979126010,36sleep29893-21latency_hist07:31:272
783726019,20sleep00-21swapper/012:31:470
783726019,20sleep00-21swapper/012:31:470
515226020,17sleep00-21swapper/007:16:270
2930426017,26sleep229301-21apt-get11:56:362
2930426017,26sleep229301-21apt-get11:56:362
211726020,18sleep10-21swapper/110:31:261
1493826015,33sleep11533-21kworker/u8:007:46:401
1317226020,20sleep20-21swapper/209:21:562
367925918,25sleep13765-21mailstats08:51:551
3248725912,29sleep232494-21/usr/sbin/munin12:06:412
30652594,11sleep10-21swapper/107:10:541
30652594,11sleep10-21swapper/107:10:541
2793925918,20sleep00-21swapper/011:51:450
2679325913,24sleep20-21swapper/210:06:272
2615225915,27sleep026159-21df_inode08:21:410
2476425917,20sleep10-21swapper/111:41:491
2476425917,20sleep10-21swapper/111:41:491
2293325916,24sleep122957-21fschecks_count11:36:441
1782125916,26sleep217762-21latency_hist07:56:272
1754625917,24sleep117555-21df_inode09:36:451
1723025918,16sleep20-21swapper/209:36:262
46182588,13sleep10-21swapper/108:56:271
1552125817,23sleep115582-21wc11:11:521
1552125817,23sleep115582-21wc11:11:521
1170125822,15sleep20-21swapper/211:01:272
882025720,22sleep01-21systemd09:07:200
882025720,22sleep01-21systemd09:07:200
52625714,26sleep0609-21sed07:01:450
286525717,18sleep10-21swapper/110:31:541
2750625716,18sleep20-21swapper/211:51:272
197525711,21sleep00-21swapper/012:11:520
1817625716,23sleep218170-21apt-get11:21:382
94982565,17sleep20-21swapper/210:52:062
939825615,24sleep09417-21apt-get09:11:360
939825615,24sleep09417-21apt-get09:11:360
40925617,26sleep2467-21idleruntime07:01:432
3105025615,19sleep00-21swapper/012:01:430
2926325615,24sleep129298-21expr08:31:431
2926325615,24sleep129298-21expr08:31:431
1901625615,25sleep119011-21apt-get09:41:381
1442525618,14sleep00-21swapper/009:26:430
1442525618,14sleep00-21swapper/009:26:430
822925512,16sleep10-21swapper/112:31:561
822925512,16sleep10-21swapper/112:31:561
728025516,23sleep17301-21irqstats10:46:481
433725520,22sleep10-21swapper/110:36:511
3062625516,24sleep00-21swapper/012:01:260
3062625516,24sleep00-21swapper/012:01:260
3026225515,22sleep030365-21perl10:16:430
2392325513,24sleep223919-21apt-get09:56:372
1730825518,17sleep10-21swapper/111:16:531
1376025518,23sleep20-21swapper/211:06:482
1376025518,23sleep20-21swapper/211:06:482
126525515,23sleep21264-21ntp_states10:26:542
833525414,19sleep20-21swapper/210:51:262
460625415,22sleep24704-21users07:11:562
1835925410,29sleep018411-21cat11:21:440
1213125412,20sleep00-21swapper/007:36:540
817125315,22sleep28195-21ntp_states12:31:532
817125315,22sleep28195-21ntp_states12:31:532
55625314,17sleep20-21swapper/210:26:262
1564425315,23sleep215653-21munin-run09:31:252
1564425315,23sleep215653-21munin-run09:31:252
1490325313,24sleep214422-21apt-get07:46:332
134325311,18sleep00-21swapper/008:46:240
134325311,18sleep00-21swapper/008:46:240
1187225314,18sleep10-21swapper/107:36:471
1029825317,20sleep210390-21load07:31:472
819525218,16sleep10-21swapper/107:26:251
2565625214,19sleep00-21swapper/011:46:240
2565625214,19sleep00-21swapper/011:46:240
2479725214,23sleep124830-21users09:56:571
2293025215,21sleep20-21swapper/209:51:572
2285525212,25sleep122929-21/usr/sbin/munin09:51:561
794925114,23sleep17854-21/usr/sbin/munin09:06:411
794925114,23sleep17854-21/usr/sbin/munin09:06:411
318925112,22sleep23185-21apt-get08:51:392
2263025114,16sleep00-21swapper/011:36:260
1657025114,15sleep00-21swapper/007:51:440
1014625111,25sleep010141-21apt-get10:56:370
1002625116,17sleep10-21swapper/107:31:381
452625013,16sleep00-21swapper/008:56:260
309625012,20sleep13091-21apt-get12:16:381
2321525012,15sleep20-21swapper/211:36:492
538924911,22sleep05382-21apt-get07:16:390
662324811,20sleep16689-21memory09:01:481
3248024712,20sleep132486-21df_inode12:06:401
509824612,17sleep10-21swapper/112:21:511
514624513,16sleep20-21swapper/208:56:482
1012224510,17sleep20-21swapper/210:56:352
1813324011,18sleep10-21swapper/107:56:421
3146723911,15sleep20-21swapper/210:21:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional