You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 14:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Aug 07, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
138272458815,4561sleep34366-21kworker/3:120:08:013
513429209,894sleep35116-21apt-get19:42:383
13783258919,553sleep313828-21/usr/sbin/munin21:52:453
5889239754,269sleep16029-21grep19:43:171
17579233464,213sleep117639-21fschecks_time23:47:411
31374232028,251sleep131520-21latency_hist21:07:221
31374232028,251sleep131520-21latency_hist21:07:221
159312265126,115sleep10-21swapper/123:42:401
12551226410,239sleep012620-21df23:32:340
1061225551,91sleep11105-21ntp_kernel_err21:12:431
16514221496,59sleep116599-21latency_hist22:02:201
30902221127,147sleep131040-21sed19:22:211
12434220812,177sleep112515-21threads21:47:511
12689220717,171sleep012841-21latency_hist20:07:220
23404220434,136sleep123450-21/usr/sbin/munin00:07:191
1913219928,134sleep11-21systemd22:57:471
603218615,154sleep3742-21latency_hist00:37:203
9142218213,69sleep19181-21ntpq21:37:501
12121217713,148sleep312255-21uptime20:02:533
2380521738,129sleep123837-21grep20:42:351
10138217144,106sleep110302-21grep23:22:541
396216865,90sleep1426-21idleruntime-cro21:12:181
8541216745,56sleep10-21swapper/121:37:211
21177216618,129sleep121253-21switchtime23:58:021
27369216214,131sleep227425-21sort19:12:212
27369216214,131sleep227425-21sort19:12:212
3356216112,132sleep13394-21ntpq23:02:441
2170215820,112sleep02321-21head22:57:500
5836215720,120sleep15872-21latency_hist23:12:201
17978215529,105sleep118014-21ntpq20:22:451
14284215313,124sleep114344-21idleruntime23:37:401
26455215218,113sleep10-21swapper/122:32:441
25865215222,112sleep226013-21wc22:32:212
2225215013,98sleep10-21swapper/119:32:471
2225215013,98sleep10-21swapper/119:32:471
26955214915,107sleep00-21swapper/000:17:390
26955214915,107sleep00-21swapper/000:17:390
1837221458,123sleep0364-21snmpd22:07:360
12516214515,114sleep212628-21timerwakeupswit21:47:522
11044214519,105sleep10-21swapper/120:02:021
1474214418,75sleep1680-21cron19:32:181
6171214212,66sleep16252-21fschecks_time23:12:411
6171214212,66sleep16252-21fschecks_time23:12:411
19811214114,104sleep219960-21tail20:27:512
15307213810,36sleep10-21swapper/120:12:521
27852213629,63sleep10-21swapper/119:12:281
28580213511,46sleep10-21swapper/122:37:571
11543213552,28sleep111554-21modprobe21:47:181
14692213412,19sleep10-21swapper/121:57:201
9208213315,105sleep08714-21/usr/sbin/munin21:37:530
20119213346,22sleep10-21swapper/122:12:421
32252213249,24sleep10-21swapper/100:32:501
20439213233,61sleep120505-21latency_hist20:32:221
18717212816,91sleep30-21swapper/323:52:193
2088921277,12sleep10-21swapper/120:32:421
2088921277,12sleep10-21swapper/120:32:421
4176212613,98sleep24245-21users19:37:532
29256212659,41sleep329295-21sendmail19:17:193
28991212610,23sleep10-21swapper/120:57:491
22324212517,61sleep10-21swapper/120:37:341
1402221255,12sleep10-21swapper/120:10:301
1981212421,86sleep22049-21wc22:57:482
16034212423,18sleep10-21swapper/120:17:321
15167212413,96sleep214660-21apt-get21:57:292
19132212320,19sleep30-21swapper/320:27:353
8040212264,43sleep125809-21cyclictest23:17:451
20654212214,36sleep320656-21/usr/sbin/munin20:32:323
20654212214,36sleep320656-21/usr/sbin/munin20:32:323
24900212118,64sleep124937-21sed22:27:441
25724212045,48sleep025759-21munin-run19:07:190
17680211815,84sleep017783-21fschecks_time20:22:400
31376211615,84sleep330738-21/usr/sbin/munin22:47:523
3551211410,66sleep13560-21munin-run21:22:191
28468211414,22sleep30-21swapper/300:22:393
11918211445,53sleep111335-21/usr/sbin/munin20:02:511
4065211317,78sleep23548-21apt-get21:22:372
4065211317,78sleep23548-21apt-get21:22:372
9990211111,82sleep310059-21sensors_temp23:22:523
21432211122,17sleep10-21swapper/122:17:341
1112211144,54sleep11114-21cron22:57:191
32379211022,66sleep232406-21awk19:27:192
32110211017,72sleep032256-21latency_hist22:52:210
14641210918,74sleep014669-21munin-run21:57:190
3073210714,76sleep03161-21tune2fs23:02:390
17227210722,56sleep20-21swapper/223:47:212
6339210623,60sleep36480-21grep21:27:513
31125210631,48sleep331118-21apt-get19:22:363
11656210611,81sleep011798-21modprobe21:47:210
16520210413,74sleep016596-21tr20:17:470
24304210319,66sleep224406-21cat00:07:482
2298321038,51sleep10-21swapper/122:22:351
1857021028,57sleep10-21swapper/122:07:411
1968829919,53sleep019835-21grep23:52:530
2118929718,55sleep321221-21latency_hist22:17:203
333729626,15sleep10-21swapper/119:37:311
2838629527,18sleep00-21swapper/000:22:360
275829517,63sleep22822-21find21:17:522
3076129423,15sleep10-21swapper/122:47:341
503629222,35sleep10-21swapper/121:23:371
503629222,35sleep10-21swapper/121:23:371
2683429221,15sleep10-21swapper/100:17:321
2683429221,15sleep10-21swapper/100:17:321
1071229214,63sleep210759-21systemctl23:27:192
258072916,46sleep10-21swapper/120:47:441
258072916,46sleep10-21swapper/120:47:441
2198229116,56sleep022051-21grep22:17:490
1396129111,67sleep013964-21sshd20:09:440
116492916,11sleep20-21swapper/220:02:432
2735228926,51sleep127360-21munin-run19:12:201
2735228926,51sleep127360-21munin-run19:12:201
2506128916,51sleep025067-21modprobe20:47:180
1200628823,16sleep1364-21snmpd23:29:041
668128712,17sleep10-21swapper/119:47:371
2106528717,52sleep320564-21/usr/sbin/munin23:58:003
587428618,47sleep26021-21latency_hist23:12:212
302122869,17sleep10-21swapper/122:44:031
2918628625,15sleep10-21swapper/100:22:531
3080428511,60sleep030923-21timerwakeupswit00:28:010
1479228520,39sleep214931-21gpgv21:57:212
147828514,58sleep21523-21unin-run19:32:192
3243428411,21sleep10-21swapper/119:27:221
2512228411,12sleep30-21swapper/319:02:503
87382838,22sleep10-21swapper/119:52:491
2429428317,49sleep324371-21latency_hist22:27:213
199272829,14sleep10-21swapper/123:53:021
1110728120,46sleep31-21systemd20:02:203
480428017,42sleep34970-21latency_hist19:42:223
2907928028,35sleep229162-21find20:57:502
2663928011,51sleep226787-21cat00:17:212
2233928026,24sleep30-21swapper/320:37:363
309727915,48sleep33127-21cat19:37:213
309727915,48sleep33127-21cat19:37:213
1868627915,28sleep30-21swapper/322:07:433
179862795,12sleep10-21swapper/122:07:011
179862795,12sleep10-21swapper/122:07:011
543527722,34sleep05499-21cat19:43:070
3149227713,50sleep031516-21netstat19:22:450
2742627718,38sleep327325-21latency_hist19:12:223
1889327725,29sleep20-21swapper/220:27:222
2701027613,49sleep327011-21sshd22:33:023
208332768,55sleep019293-21kworker/0:122:13:500
170027615,43sleep31792-21/usr/sbin/munin22:57:413
501527526,24sleep00-21swapper/023:07:470
1692827512,48sleep316961-21munin-run23:47:183
138472748,51sleep113884-21munin-run23:37:191
1231227411,48sleep312290-21/usr/sbin/munin23:32:213
831327324,27sleep30-21swapper/319:52:393
724727319,28sleep20-21swapper/221:32:382
483127315,38sleep04980-21head21:23:110
483127315,38sleep04980-21head21:23:110
3078127320,26sleep30-21swapper/321:02:523
3078127320,26sleep30-21swapper/321:02:523
2835427311,40sleep028363-21/usr/sbin/munin19:12:440
2483427325,25sleep10-21swapper/119:02:441
734827221,36sleep27381-21munin-run23:17:182
734827221,36sleep27381-21munin-run23:17:182
532827225,27sleep20-21swapper/219:43:052
2846027214,23sleep30-21swapper/320:57:313
273792728,29sleep10-21swapper/120:52:481
273792728,29sleep10-21swapper/120:52:481
2058927216,25sleep20-21swapper/222:12:512
1262927212,42sleep212633-21diskstats23:32:362
264772716,11sleep10-21swapper/100:16:481
249912719,48sleep10-21swapper/100:12:191
955127019,28sleep09600-21fschecks_count23:22:410
1252327020,29sleep30-21swapper/323:32:293
311542695,11sleep10-21swapper/100:28:511
2979826918,25sleep00-21swapper/021:02:200
2252326920,31sleep222581-21latency20:37:402
2815226811,44sleep028234-21grep22:37:520
2737126812,39sleep027378-21ntpq20:52:470
2737126812,39sleep027378-21ntpq20:52:470
2511326822,22sleep20-21swapper/220:47:202
191282688,44sleep00-21swapper/020:27:320
1007626814,40sleep010122-21sed19:57:450
2593426616,26sleep00-21swapper/020:47:470
2593426616,26sleep00-21swapper/020:47:470
2094226618,25sleep20-21swapper/220:32:432
2094226618,25sleep20-21swapper/220:32:432
2026226617,30sleep220307-21systemctl23:57:192
1734326621,21sleep20-21swapper/220:22:212
1190426613,37sleep311911-21df_inode21:47:373
1146426615,37sleep211496-21proc_pri23:27:482
793626513,33sleep37971-21idleruntime-cro19:52:193
635426514,33sleep06423-21latency23:12:440
635426514,33sleep06423-21latency23:12:440
3087826521,22sleep00-21swapper/019:22:200
3001726510,39sleep230028-21ntp_states19:17:462
2504226523,22sleep20-21swapper/219:02:492
1766726519,30sleep30-21swapper/320:22:393
144682657,12sleep00-21swapper/023:37:430
66472645,11sleep10-21swapper/121:29:261
353526417,28sleep03548-21apt21:22:180
3136126421,23sleep00-21swapper/021:07:210
3136126421,23sleep00-21swapper/021:07:210
2676226417,26sleep00-21swapper/022:32:510
2561226414,24sleep30-21swapper/300:12:463
3066626312,15sleep20-21swapper/200:27:592
2576926311,38sleep025771-21sshd22:32:020
2575026324,23sleep30-21swapper/320:47:423
2575026324,23sleep30-21swapper/320:47:423
2349026317,26sleep30-21swapper/300:07:213
1518826313,20sleep30-21swapper/321:57:303
11362636,46sleep00-21swapper/019:27:580
1089126321,22sleep20-21swapper/221:42:512
470726215,25sleep30-21swapper/321:23:103
470726215,25sleep30-21swapper/321:23:103
322226216,21sleep20-21swapper/223:02:412
3002926215,29sleep330038-21ntpq19:17:473
211452627,16sleep00-21swapper/020:32:490
211452627,16sleep00-21swapper/020:32:490
1909226221,22sleep30-21swapper/323:52:393
162512627,41sleep00-21swapper/021:59:370
766926116,30sleep37663-21apt-get23:17:353
44426119,24sleep00-21swapper/022:52:500
3196726119,19sleep20-21swapper/221:07:432
268102615,39sleep30-21swapper/300:17:303
268102615,39sleep30-21swapper/300:17:303
194826115,26sleep20-21swapper/219:32:402
194826115,26sleep20-21swapper/219:32:402
19002615,16sleep00-21swapper/021:16:330
797426022,18sleep00-21swapper/019:52:200
668226021,23sleep20-21swapper/219:47:382
326782605,46sleep00-21swapper/000:33:300
3167026013,28sleep331677-21df_inode21:07:363
2989326014,28sleep030021-21/usr/sbin/munin22:42:530
2513926021,21sleep30-21swapper/322:27:503
2435726019,19sleep30-21swapper/320:42:493
2059026016,25sleep20-21swapper/223:57:352
1617126014,19sleep00-21swapper/023:42:470
1037326013,27sleep30-21swapper/321:42:403
66125916,26sleep3679-21apt21:12:283
622525910,26sleep20-21swapper/221:27:502
3076225916,26sleep230758-21apt-get22:47:352
3044325913,21sleep30-21swapper/300:27:523
2831025910,25sleep20-21swapper/222:37:542
2743925910,21sleep20-21swapper/220:52:492
2743925910,21sleep20-21swapper/220:52:492
2306025913,24sleep023066-21diskstats22:22:380
1803425915,27sleep018111-21ls23:47:520
1499825913,26sleep215066-21/usr/sbin/munin20:12:492
2803325823,16sleep30-21swapper/322:37:473
2440925818,21sleep30-21swapper/300:07:493
2214825811,29sleep20-21swapper/200:02:352
1453725818,23sleep314547-21forks20:12:383
143525817,18sleep20-21swapper/221:12:522
1231925817,22sleep00-21swapper/023:32:220
10402586,11sleep30-21swapper/322:56:243
803025713,25sleep08039-21/usr/sbin/munin23:17:440
791625713,29sleep27939-21irqstats23:17:422
67825716,23sleep30-21swapper/319:27:483
579825714,18sleep00-21swapper/021:27:390
2699425719,20sleep30-21swapper/320:52:383
2699425719,20sleep30-21swapper/320:52:383
1723425713,21sleep30-21swapper/322:02:473
1723425713,21sleep30-21swapper/322:02:473
1346125720,20sleep20-21swapper/220:07:562
764525614,26sleep37707-21sendmail_mailst21:32:473
2709725623,17sleep20-21swapper/200:17:412
2709725623,17sleep20-21swapper/200:17:412
2615625613,16sleep20-21swapper/220:47:492
2615625613,16sleep20-21swapper/220:47:492
2602625615,25sleep326044-21apt19:07:303
237025619,21sleep00-21swapper/019:32:500
237025619,21sleep00-21swapper/019:32:500
1955825613,24sleep219581-21munin-run22:12:182
882325513,25sleep28828-21diskstats21:37:432
3004025514,26sleep230032-21apt-get21:02:342
3004025514,26sleep230032-21apt-get21:02:342
2921325513,25sleep329208-21apt-get22:42:353
2306925515,18sleep20-21swapper/222:22:402
2192425518,22sleep30-21swapper/322:17:483
2143325513,16sleep20-21swapper/222:17:352
1612625514,26sleep316217-21/usr/sbin/munin20:17:383
358925417,16sleep30-21swapper/323:02:503
3199025416,20sleep20-21swapper/200:32:432
269825412,23sleep32734-21ntp_states21:17:493
2651825415,18sleep00-21swapper/000:17:190
1749225416,17sleep20-21swapper/222:02:502
1749225416,17sleep20-21swapper/222:02:502
1466325416,19sleep30-21swapper/323:37:503
1455025415,17sleep00-21swapper/020:12:390
882925316,16sleep30-21swapper/321:37:443
867125314,20sleep20-21swapper/219:52:462
71602536,32sleep00-21swapper/021:32:320
2962325314,21sleep229665-21ntp_states22:42:462
1567025312,23sleep215663-21apt-get23:42:332
869625220,19sleep00-21swapper/019:52:480
63132528,17sleep30-21swapper/323:12:433
63132528,17sleep30-21swapper/323:12:433
3241625212,24sleep232513-21perl22:52:392
2310325212,21sleep023155-21users20:37:500
2252525213,16sleep00-21swapper/000:02:460
459625111,27sleep04632-21modprobe19:42:190
2648525111,22sleep20-21swapper/222:32:452
1075425115,20sleep00-21swapper/021:42:490
46392508,16sleep30-21swapper/323:07:383
2823125012,22sleep228263-21irqstats19:12:422
2614925011,16sleep00-21swapper/019:07:390
2614925011,16sleep00-21swapper/019:07:390
2081525011,15sleep00-21swapper/023:57:530
1060325011,21sleep210706-21users19:57:552
2862624812,20sleep228652-21irqstats00:22:422
1675124510,20sleep016746-21apt-get22:02:310
1675124510,20sleep016746-21apt-get22:02:310
149872459,20sleep215025-21users23:37:532
2387524312,15sleep00-21swapper/000:07:380
3167924010,16sleep00-21swapper/021:07:370
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional