You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-20 - 03:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sun Sep 20, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
296342105677,10484sleep216995-21kworker/2:020:50:582
162562603619,6003sleep112452-21kworker/1:223:32:571
1104424588,438sleep211111-21systemctl21:33:092
3033235514,226sleep23197-21latency_hist22:52:402
30540235314,322sleep330624-21grep20:53:093
2914230716,195sleep23033-21timerandwakeup21:08:102
48292277137,23sleep20-21swapper/222:57:412
48292277137,23sleep20-21swapper/222:57:412
27034225670,90sleep227050-21fschecks_count20:42:582
761422469,219sleep07645-21/usr/sbin/munin21:23:220
1003224671,64sleep20-21swapper/219:18:112
182842219103,22sleep212-21rcu_sched20:13:102
1172220950,115sleep21317-21latency_hist00:27:412
8751220211,173sleep28791-21uname19:43:102
2945222027,11sleep20-21swapper/219:08:052
691921718,104sleep20-21swapper/219:38:012
14779216955,91sleep20-21swapper/220:03:052
722121670,62sleep20-21swapper/223:03:092
31059216613,104sleep20-21swapper/222:37:542
31849215731,21sleep20-21swapper/200:20:592
27956215442,93sleep20-21swapper/200:08:102
18426215414,124sleep118532-21timerwakeupswit23:38:091
2690215032,103sleep22818-21sort19:23:112
17442214572,55sleep20-21swapper/223:37:382
6479214213,21sleep20-21swapper/221:19:572
431021408,119sleep04316-21ntpq21:13:090
431021408,119sleep04316-21ntpq21:13:090
23024214014,26sleep20-21swapper/223:53:042
10172214073,51sleep210351-21apt-config21:32:412
9249213866,57sleep09308-21/usr/sbin/munin21:28:110
1598221388,26sleep20-21swapper/223:32:412
20707213725,42sleep30-21swapper/320:22:483
18129213726,86sleep20-21swapper/223:38:022
12024213617,100sleep212160-21grep23:18:122
17670213511,21sleep20-21swapper/221:53:122
32090213420,36sleep20-21swapper/220:58:082
23273213311,109sleep023363-21cat22:13:010
8964213221,89sleep19101-21unixbench_multi19:43:131
29752213215,18sleep20-21swapper/200:13:112
26755213299,16sleep20-21swapper/222:23:132
26755213299,16sleep20-21swapper/222:23:132
10196213124,86sleep20-21swapper/219:48:032
10196213124,86sleep20-21swapper/219:48:032
31696213036,19sleep231777-21cat19:13:102
10363213011,100sleep310044-21apt-get21:32:423
291921299,107sleep02942-21grep00:32:560
28324212719,93sleep028453-21sed22:28:110
1080621245,11sleep20-21swapper/223:15:212
1080621245,11sleep20-21swapper/223:15:212
17870212220,68sleep10-21swapper/120:13:011
12701212268,17sleep212710-21munin-run19:57:402
513821208,94sleep20-21swapper/222:57:572
513821208,94sleep20-21swapper/222:57:572
25488212026,18sleep20-21swapper/220:37:572
22760212049,23sleep20-21swapper/220:28:082
29112211932,58sleep329131-21apt-get00:12:523
1997211821,78sleep31999-21cron21:07:383
5341211735,21sleep20-21swapper/219:33:002
2516121175,11sleep20-21swapper/223:58:212
2516121175,11sleep20-21swapper/223:58:212
20807211710,87sleep120837-21latency_hist23:47:381
1311421166,96sleep213078-21apt-get23:23:122
797521155,11sleep20-21swapper/221:23:322
23891211510,83sleep20-21swapper/222:13:112
20511211511,86sleep020657-21latency_hist20:22:410
2031121155,11sleep20-21swapper/220:20:532
2555021135,13sleep20-21swapper/222:18:532
2555021135,13sleep20-21swapper/222:18:532
22443211315,69sleep30-21swapper/323:52:413
2684421125,10sleep20-21swapper/200:04:592
1579211114,80sleep11629-21if_eth000:28:001
1579211114,80sleep11629-21if_eth000:28:001
14915211112,84sleep215003-21cat23:28:052
5243210812,80sleep05250-21/usr/sbin/munin19:32:580
30936210812,80sleep030997-21switchtime19:12:470
27987210834,44sleep228031-21ntpq19:03:052
7193210715,80sleep17196-21acpi21:22:511
6691210714,78sleep16732-21gpgv21:22:381
27891210717,66sleep127314-21/usr/sbin/munin00:08:091
7289210621,66sleep37363-21switchtime19:38:103
489021065,45sleep20-21swapper/221:14:482
489021065,45sleep20-21swapper/221:14:482
2484621068,85sleep2367-21snmpd20:34:342
947210515,74sleep1997-21/usr/sbin/munin21:03:001
16831210510,78sleep316953-21users20:08:103
14155210512,78sleep214215-21find21:43:072
418421038,16sleep20-21swapper/200:37:382
3005521030,7sleep00-21swapper/022:33:110
3036210126,19sleep30-21swapper/321:08:113
2810021017,18sleep20-21swapper/222:28:052
21512210112,15sleep20-21swapper/223:48:042
11064210116,73sleep329249-1kworker/3:1H23:17:403
11064210116,73sleep329249-1kworker/3:1H23:17:403
573029914,73sleep35804-21switchtime19:33:103
220342998,13sleep20-21swapper/220:27:402
9982975,12sleep20-21swapper/221:03:012
82542977,14sleep20-21swapper/223:07:582
3198029714,68sleep032008-21munin-run00:22:380
293242976,13sleep20-21swapper/222:32:582
3015629513,50sleep20-21swapper/220:53:002
152182956,13sleep20-21swapper/221:47:512
3236529415,64sleep332444-21latency_hist22:42:403
160532948,48sleep20-21swapper/220:07:532
2880129333,33sleep00-21swapper/000:12:410
218302936,11sleep20-21swapper/222:08:062
5242925,11sleep20-21swapper/222:43:032
730429128,36sleep30-21swapper/323:03:113
2767929121,50sleep127774-21perl19:03:001
1954529110,66sleep20-21swapper/223:42:552
1736229110,69sleep00-21swapper/023:36:350
120032915,12sleep20-21swapper/221:37:422
1751129018,50sleep117667-21grep21:53:111
1627229011,66sleep316290-21fschecks_count23:32:593
889128916,46sleep39038-21grep23:08:123
208242895,11sleep20-21swapper/222:03:182
140402895,12sleep20-21swapper/220:00:222
22882885,11sleep20-21swapper/200:28:182
22882885,11sleep20-21swapper/200:28:182
215328815,42sleep32240-21fschecks_time19:23:003
193792885,10sleep20-21swapper/221:59:382
1910428815,53sleep019252-21date23:42:400
980028719,46sleep19873-21fschecks_time23:12:581
980028719,46sleep19873-21fschecks_time23:12:581
46512875,10sleep20-21swapper/219:29:542
353028710,14sleep00-21swapper/021:12:160
2390328717,50sleep023928-21munin-run23:57:380
3184728614,17sleep00-21swapper/020:58:020
611528322,41sleep36253-21unixbench_multi21:18:123
312212838,63sleep0367-21snmpd20:56:290
2301028311,56sleep123023-21/usr/sbin/munin23:53:031
2609328214,30sleep30-21swapper/300:03:033
1271228216,15sleep30-21swapper/319:57:413
543828123,31sleep10-21swapper/122:58:041
543828123,31sleep10-21swapper/122:58:041
1311728115,34sleep00-21swapper/023:23:140
153282808,17sleep00-21swapper/020:03:230
1281628011,51sleep012955-21tail21:38:120
2629627917,29sleep00-21swapper/020:38:120
140932797,17sleep30-21swapper/321:43:043
1008527913,50sleep110171-21apt-config21:32:401
2860027816,20sleep30-21swapper/322:28:143
2005027814,27sleep00-21swapper/023:43:090
1736327720,38sleep117490-21latency_hist20:12:411
2687527618,39sleep327024-21grep22:23:143
2687527618,39sleep327024-21grep22:23:143
3269227514,41sleep132752-21/usr/sbin/munin22:42:581
2157327524,32sleep10-21swapper/123:48:071
1459627516,46sleep314669-21fschecks_time20:03:023
338127414,44sleep13405-21proc_pri00:33:091
1843027415,42sleep118442-21/usr/sbin/munin21:57:561
2736627210,15sleep00-21swapper/022:27:140
2736627210,15sleep00-21swapper/022:27:140
1519927218,22sleep10-21swapper/121:47:501
147727220,24sleep30-21swapper/322:47:403
120027219,34sleep11266-21timerwakeupswit19:18:141
1189327218,28sleep10-21swapper/121:37:411
86202710,7sleep30-21swapper/321:27:403
2975727111,20sleep10-21swapper/122:33:091
2346627112,12sleep30-21swapper/320:31:403
1961427112,30sleep30-21swapper/322:02:383
1860927112,42sleep018622-21/usr/sbin/munin21:57:590
1255927117,31sleep012582-21munin-run23:22:380
579727023,25sleep00-21swapper/022:58:110
579727023,25sleep00-21swapper/022:58:110
1147627020,26sleep00-21swapper/019:53:010
349226926,26sleep30-21swapper/300:33:113
3162526922,24sleep10-21swapper/119:13:091
298042699,43sleep30-21swapper/320:52:403
2915626914,27sleep30-21swapper/319:07:543
145626926,21sleep30-21swapper/300:27:583
145626926,21sleep30-21swapper/300:27:583
38726820,28sleep00-21swapper/000:23:110
3231526821,26sleep10-21swapper/122:42:381
2933126822,24sleep30-21swapper/322:32:593
2118226817,25sleep00-21swapper/023:47:570
79622676,13sleep00-21swapper/019:42:150
3048626711,40sleep030632-21latency_hist00:17:410
3040326618,31sleep130441-21sendmail00:17:381
2908126621,23sleep00-21swapper/020:48:090
1654826618,22sleep00-21swapper/020:08:070
987926518,30sleep39903-21/usr/sbin/munin19:47:553
987926518,30sleep39903-21/usr/sbin/munin19:47:553
388526515,26sleep30-21swapper/322:53:063
388526515,26sleep30-21swapper/322:53:063
376726514,29sleep13861-21/usr/sbin/munin22:53:031
376726514,29sleep13861-21/usr/sbin/munin22:53:031
2867526515,27sleep328794-21apt-config00:12:403
2819526524,20sleep30-21swapper/319:03:103
2710126518,26sleep00-21swapper/000:07:400
2515626519,20sleep10-21swapper/120:37:401
2449626516,20sleep00-21swapper/020:33:110
220926521,15sleep00-21swapper/022:48:100
12892659,47sleep00-21swapper/022:46:000
1010126520,27sleep110195-21/usr/sbin/munin19:48:021
1010126520,27sleep110195-21/usr/sbin/munin19:48:021
51526416,28sleep3563-21awk19:17:553
421926411,28sleep10-21swapper/121:13:051
421926411,28sleep10-21swapper/121:13:051
673626321,19sleep10-21swapper/119:37:561
278702639,12sleep10-21swapper/122:28:001
2469526316,24sleep10-21swapper/123:58:081
2469526316,24sleep10-21swapper/123:58:081
191552638,12sleep30-21swapper/320:17:433
1677226316,25sleep30-21swapper/321:52:533
1344126316,25sleep10-21swapper/121:42:411
1108526311,36sleep011227-21wc23:17:410
1108526311,36sleep011227-21wc23:17:410
1064526313,34sleep010650-21/usr/sbin/munin21:32:590
834326216,19sleep00-21swapper/023:08:000
590326215,20sleep00-21swapper/021:18:090
49522629,23sleep00-21swapper/019:32:420
32792627,11sleep10-21swapper/119:27:061
2799626214,22sleep30-21swapper/320:44:573
2407926217,30sleep324102-21irqstats20:33:023
2051226217,28sleep30-21swapper/322:03:113
1919426210,38sleep119196-21grep20:17:581
712926116,26sleep10-21swapper/123:03:091
340026112,32sleep03549-21date19:27:410
3102626118,24sleep00-21swapper/022:37:480
48402607,10sleep30-21swapper/319:32:173
2458726013,33sleep124591-21grep22:17:551
2458726013,33sleep124591-21grep22:17:551
2450226020,23sleep00-21swapper/023:58:030
2450226020,23sleep00-21swapper/023:58:030
2693925910,11sleep10-21swapper/100:06:121
1529525912,31sleep315304-21df_inode21:47:563
1331625919,22sleep30-21swapper/323:23:423
110725915,19sleep00-21swapper/021:03:030
3081925815,24sleep130906-21perl00:17:591
2463925814,25sleep30-21swapper/323:58:063
2463925814,25sleep30-21swapper/323:58:063
2404525816,26sleep324181-21cat23:57:413
217125817,24sleep32203-21proc_pri22:48:093
1581125813,31sleep115840-21cut20:07:401
1113525820,22sleep00-21swapper/019:52:410
1113525820,22sleep00-21swapper/019:52:410
877425716,24sleep38765-21apt-get21:27:583
300025715,25sleep13020-21munin-run22:52:391
2831725713,21sleep10-21swapper/120:47:411
2759025713,26sleep027584-21apt-get19:02:550
241202577,16sleep30-21swapper/322:15:433
1439325713,20sleep10-21swapper/123:27:401
50925615,25sleep0505-21apt-get19:17:520
384825614,26sleep33843-21apt-get21:12:553
384825614,26sleep33843-21apt-get21:12:553
3113825614,26sleep331133-21apt-get19:12:543
254052565,11sleep30-21swapper/320:37:543
2426925621,21sleep124299-21cron22:17:381
2276525614,17sleep00-21swapper/022:12:410
2130625619,20sleep10-21swapper/120:23:061
1463925614,25sleep314634-21apt-get23:27:543
1463925614,25sleep314634-21apt-get23:27:543
879325516,18sleep30-21swapper/319:43:113
838525515,23sleep18441-21irqstats23:08:011
672425512,18sleep00-21swapper/023:03:000
50412555,11sleep30-21swapper/322:57:503
50412555,11sleep30-21swapper/322:57:503
3235025514,17sleep30-21swapper/320:58:143
3013725512,28sleep130154-21fschecks_count20:52:581
231982555,11sleep00-21swapper/020:28:140
2125425513,16sleep10-21swapper/122:07:411
1964625510,25sleep30-21swapper/323:43:003
149425514,25sleep01578-21perl00:27:590
149425514,25sleep01578-21perl00:27:590
1137525514,25sleep111372-21apt-get19:52:541
112725517,23sleep11171-21latency_hist00:27:401
406825412,27sleep04148-21ls22:53:110
406825412,27sleep04148-21ls22:53:110
280542547,36sleep10-21swapper/120:45:401
2512425418,16sleep30-21swapper/322:18:093
2512425418,16sleep30-21swapper/322:18:093
1493425416,24sleep114967-21munin-run21:47:381
1221525415,24sleep312220-21diskstats21:37:593
515825315,24sleep10-21swapper/119:32:551
406825311,18sleep10-21swapper/119:28:071
2744625314,20sleep10-21swapper/122:27:401
2744625314,20sleep10-21swapper/122:27:401
2458625313,24sleep024580-21apt-get22:17:540
2458625313,24sleep024580-21apt-get22:17:540
1292525316,15sleep10-21swapper/119:57:511
1157425314,19sleep10-21swapper/123:18:031
3226825213,23sleep332264-21apt-get00:22:533
3003025213,24sleep030088-21users19:08:150
2802525227,15sleep00-21swapper/020:45:180
2582225213,22sleep125858-21ntp_states20:38:051
224852525,11sleep30-21swapper/322:09:453
1826025215,17sleep30-21swapper/323:38:073
1807025216,17sleep30-21swapper/320:13:073
1216425213,21sleep312205-21users23:18:133
828325115,18sleep00-21swapper/019:42:560
219212519,18sleep10-21swapper/122:08:091
1709625112,22sleep017143-21memory21:53:020
157302515,38sleep10-21swapper/123:30:301
157302515,38sleep10-21swapper/123:30:301
1376125111,24sleep013762-21diskstats21:42:570
307212508,21sleep30-21swapper/300:17:533
252525012,22sleep12570-21irqstats21:08:011
233362504,35sleep10-21swapper/120:30:011
3190524914,22sleep131939-21wc20:58:031
218982496,36sleep00-21swapper/020:26:250
1549624915,19sleep00-21swapper/021:48:010
2918024810,21sleep129177-21apt-get19:07:561
1294824812,19sleep312944-21apt-get19:57:533
129502469,19sleep00-21swapper/019:57:540
1989824310,19sleep00-21swapper/022:02:510
176002439,16sleep00-21swapper/020:12:520
148232408,14sleep00-21swapper/023:28:030
148232408,14sleep00-21swapper/023:28:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional