You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 22:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sat Mar 21, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
200127877,772sleep218126-1kworker/2:0H08:28:182
870627648,744sleep08791-21perl12:13:020
26491254616,513sleep226500-21/usr/sbin/munin08:03:022
29536247219,438sleep229581-21/usr/sbin/munin09:53:112
1600924698,446sleep116053-21ntpq10:53:111
1600924698,446sleep116053-21ntpq10:53:111
1603324030,11sleep10-21swapper/109:12:461
1387239432,348sleep01422-21latency_hist10:07:470
25829239130,322sleep025946-21cat09:42:470
20197238821,347sleep320246-21/usr/sbin/munin07:43:003
23796237525,336sleep023871-21run-parts07:53:150
25136230526,267sleep025204-21ls07:58:080
20865228439,183sleep020868-21cron09:27:440
1563422659,220sleep015671-21grep10:53:020
1563422659,220sleep015671-21grep10:53:020
30423226312,157sleep01-21systemd09:57:450
18523225912,164sleep00-21swapper/011:02:470
8305225448,188sleep0222-21systemd-journal12:12:380
21903225166,144sleep00-21swapper/011:13:010
12360223849,65sleep00-21swapper/012:23:160
12360223849,65sleep00-21swapper/012:23:160
12617223619,203sleep312664-21fschecks_count10:43:033
8745223144,172sleep08637-21/usr/sbin/munin07:08:040
4555223047,140sleep04464-21latency_hist10:17:480
11725223037,137sleep011866-21unixbench_singl10:38:160
11725223037,137sleep011866-21unixbench_singl10:38:160
312002202144,14sleep00-21swapper/009:58:140
312002202144,14sleep00-21swapper/009:58:140
675022019,149sleep00-21swapper/008:43:060
15498219646,132sleep00-21swapper/007:28:070
15498219646,132sleep00-21swapper/007:28:070
4785219457,93sleep00-21swapper/008:37:450
9229218723,30sleep10-21swapper/110:32:481
2411421745,31sleep00-21swapper/009:36:370
5104217341,88sleep00-21swapper/008:38:010
10083217228,131sleep010209-21cat10:33:150
26907217032,82sleep00-21swapper/011:28:120
2039621706,107sleep020458-21fschecks_time11:08:040
2039621706,107sleep020458-21fschecks_time11:08:040
13269217021,57sleep00-21swapper/012:27:560
1868321646,147sleep319808-1kworker/3:2H07:38:243
337216341,45sleep00-21swapper/011:47:460
484521598,137sleep04877-21tune2fs10:18:050
14168215935,79sleep014189-21munin-run09:07:450
14168215935,79sleep014189-21munin-run09:07:450
13627215624,11sleep00-21swapper/010:45:390
30784215533,42sleep00-21swapper/008:17:460
2940214828,109sleep03044-21latency_hist10:12:480
26665214816,94sleep026042-21/usr/sbin/munin09:43:160
11789214729,63sleep011831-21grep08:58:120
11789214729,63sleep011831-21grep08:58:120
5220214634,98sleep05223-21cron12:02:440
1514621468,103sleep015177-21proc_pri09:08:140
23513214239,49sleep00-21swapper/011:18:020
2109214126,93sleep12125-21fschecks_count08:28:371
802213822,34sleep00-21swapper/011:48:070
29598213818,99sleep129629-21fschecks_count08:13:021
13397213818,96sleep113500-21fschecks_time12:28:051
28592213742,61sleep013441-21snmpd09:49:050
17856213711,91sleep00-21swapper/007:34:410
2542021357,37sleep00-21swapper/011:23:110
16184213432,80sleep216250-21cat10:53:162
16184213432,80sleep216250-21cat10:53:162
16893213233,40sleep00-21swapper/009:13:150
16893213233,40sleep00-21swapper/009:13:150
23046213130,79sleep00-21swapper/007:52:470
5904212830,58sleep05940-21ntpq12:03:090
3092521265,31sleep00-21swapper/011:39:060
3092521265,31sleep00-21swapper/011:39:060
25032212515,95sleep225086-21idleruntime07:58:052
2505212320,19sleep10-21swapper/111:53:081
2505212320,19sleep10-21swapper/111:53:081
1090221225,28sleep00-21swapper/008:56:180
1317421216,28sleep00-21swapper/007:19:110
7294211912,28sleep17328-21irqstats12:08:061
1732211922,71sleep10-21swapper/110:08:041
32219211821,80sleep232363-21latency_hist10:02:482
32219211821,80sleep232363-21latency_hist10:02:482
2613221170,12sleep30-21swapper/309:43:013
12742211516,26sleep10-21swapper/110:43:051
27095211314,50sleep00-21swapper/008:03:160
8423211230,47sleep08498-21sed10:28:140
18134211228,42sleep00-21swapper/009:18:190
3978211124,50sleep20-21swapper/211:58:182
20019211125,45sleep00-21swapper/011:07:440
20019211125,45sleep00-21swapper/011:07:440
17515211116,78sleep217642-21unixbench_multi07:33:142
25552211021,59sleep125711-21grep07:58:171
14548210810,82sleep00-21swapper/010:48:130
28791210712,77sleep228804-21/usr/sbin/munin11:33:192
26883210719,67sleep126397-21/usr/sbin/munin08:03:131
12972210626,63sleep313003-21fschecks_count09:03:083
12972210626,63sleep313003-21fschecks_count09:03:083
5309210514,74sleep15359-21if_eth008:38:061
28865210510,45sleep00-21swapper/011:33:210
14881210518,57sleep00-21swapper/007:26:130
14881210518,57sleep00-21swapper/007:26:130
20565210478,13sleep00-21swapper/007:43:100
1118321025,11sleep00-21swapper/012:19:120
1118321025,11sleep00-21swapper/012:19:120
15507210114,72sleep215630-21timerwakeupswit12:33:152
24518210024,63sleep224534-21cpu09:38:002
2265829910,53sleep022703-21systemd-run09:32:450
2966129821,29sleep10-21swapper/111:37:461
2895429622,36sleep00-21swapper/008:09:140
75622955,56sleep00-21swapper/010:27:420
1869029511,70sleep118764-21find09:18:371
1438529515,64sleep314505-21sed07:23:183
1438529515,64sleep314505-21sed07:23:183
3264329418,41sleep00-21swapper/008:22:590
148372949,45sleep00-21swapper/012:32:570
3250229313,64sleep132020-21apt-get10:02:551
2161429322,57sleep121649-21latency_hist11:12:461
2161429322,57sleep121649-21latency_hist11:12:461
452329226,44sleep30-21swapper/310:17:473
3220229215,43sleep00-21swapper/008:20:450
761729114,58sleep17701-21latency_hist10:27:471
3250329114,61sleep20-21swapper/210:02:562
294912910,11sleep20-21swapper/208:12:542
186212918,47sleep018640-21apt-get07:37:570
1290429118,53sleep213052-21head07:18:162
36092899,42sleep03767-21gpgv11:57:480
84122887,39sleep00-21swapper/008:48:060
833528833,33sleep20-21swapper/207:07:462
278128810,41sleep00-21swapper/011:53:150
2389928816,58sleep323903-21ntpq11:18:133
1106528710,38sleep00-21swapper/007:13:130
19842868,36sleep00-21swapper/008:28:030
1423128619,51sleep114307-21tr07:23:161
1423128619,51sleep114307-21tr07:23:161
2732828529,36sleep127343-21modprobe09:47:451
2321028427,29sleep20-21swapper/211:17:482
1750128420,43sleep2222-21systemd-journal09:17:452
1750128420,43sleep2222-21systemd-journal09:17:452
3146828316,47sleep131469-21/usr/sbin/munin11:42:551
2958128318,43sleep329593-21modprobe11:37:443
2300228316,47sleep3222-21systemd-journal07:52:463
2218828311,16sleep20-21swapper/207:48:122
119528314,51sleep11272-21/usr/sbin/munin11:48:161
1020128322,20sleep30-21swapper/312:18:023
1020128322,20sleep30-21swapper/312:18:023
796128215,46sleep18108-21latency_hist08:47:471
1917428129,26sleep20-21swapper/207:38:482
828428030,32sleep20-21swapper/208:48:042
2998627928,24sleep20-21swapper/211:38:032
2998627928,24sleep20-21swapper/211:38:032
262427925,26sleep20-21swapper/211:53:132
262427925,26sleep20-21swapper/211:53:132
1739227920,26sleep20-21swapper/210:58:052
722527815,46sleep27294-21/usr/sbin/munin08:43:162
67627827,35sleep2749-21grep11:48:052
3272927820,25sleep20-21swapper/208:23:052
394427732,27sleep13941-21apt-get11:57:571
2796327724,33sleep30-21swapper/311:32:473
1632827723,29sleep10-21swapper/109:13:031
1632827723,29sleep10-21swapper/109:13:031
1571627714,29sleep30-21swapper/310:53:053
1571627714,29sleep30-21swapper/310:53:053
838627627,25sleep20-21swapper/212:12:482
328627620,30sleep30-21swapper/308:32:473
3157527613,45sleep331633-21/usr/sbin/munin08:18:133
1751627616,47sleep317614-21apt-key09:17:463
1751627616,47sleep317614-21apt-key09:17:463
146512760,8sleep20-21swapper/212:32:462
912027520,25sleep20-21swapper/210:32:452
2825827517,15sleep30-21swapper/309:48:153
2667927518,42sleep326692-21irqstats08:03:073
1543227525,27sleep20-21swapper/210:52:482
1129527518,42sleep111455-21awk07:13:181
1101627512,45sleep311060-21ntp_offset07:13:123
1696227413,24sleep30-21swapper/307:33:033
756527315,46sleep37576-21sh10:27:453
2396527326,24sleep20-21swapper/207:53:172
2302627315,22sleep10-21swapper/109:33:031
1902727317,34sleep319058-21/usr/sbin/munin11:03:063
1902727317,34sleep319058-21/usr/sbin/munin11:03:063
165727313,44sleep11464-21apt-get08:27:481
153282737,15sleep10-21swapper/110:52:381
1510827319,40sleep115158-21latency12:33:061
1349927323,25sleep10-21swapper/107:22:471
1328527317,40sleep313286-21sshd07:20:393
1184027317,35sleep111939-21fschecks_time12:23:051
1184027317,35sleep111939-21fschecks_time12:23:051
841627217,37sleep00-21swapper/007:07:480
3061827217,37sleep30-21swapper/308:15:393
1654427220,33sleep116571-21munin-run07:32:441
1654427220,33sleep116571-21munin-run07:32:441
1261227212,45sleep212616-21diskstats10:43:022
1132727213,41sleep311322-21apt-get08:57:593
1132727213,41sleep311322-21apt-get08:57:593
912427114,25sleep10-21swapper/107:08:131
2998127119,34sleep129985-21diskstats11:38:021
2998127119,34sleep129985-21diskstats11:38:021
188727122,23sleep30-21swapper/311:52:463
839127022,23sleep30-21swapper/307:07:473
1846927021,32sleep118485-21idleruntime-cro11:02:441
1465427022,24sleep20-21swapper/209:07:552
1305627020,34sleep213164-21sed12:27:462
1305627020,34sleep213164-21sed12:27:462
2087526924,23sleep10-21swapper/109:27:451
1594426914,35sleep315971-21/usr/sbin/munin09:12:453
1430826920,25sleep20-21swapper/207:23:172
1430826920,25sleep20-21swapper/207:23:172
2150826814,17sleep10-21swapper/107:47:471
947826715,36sleep29487-21df_inode10:33:012
727826723,27sleep17330-21/usr/sbin/munin07:03:051
2680726725,22sleep20-21swapper/211:28:102
2365526717,22sleep20-21swapper/209:33:162
2183126713,20sleep30-21swapper/311:12:563
1537526715,35sleep215265-21/usr/sbin/munin07:28:052
1537526715,35sleep215265-21/usr/sbin/munin07:28:052
80592667,12sleep30-21swapper/310:28:053
673626618,19sleep30-21swapper/308:43:053
2270426610,43sleep122762-21latency_hist09:32:461
2046726620,19sleep20-21swapper/207:43:082
1146826623,22sleep10-21swapper/110:38:131
1146826623,22sleep10-21swapper/110:38:131
2817626518,23sleep10-21swapper/108:08:061
2320026518,19sleep10-21swapper/111:17:461
1566026516,33sleep315688-21/usr/sbin/munin07:28:103
1566026516,33sleep315688-21/usr/sbin/munin07:28:103
1191026518,20sleep20-21swapper/208:58:142
1191026518,20sleep20-21swapper/208:58:142
988126417,29sleep29914-21modprobe12:17:452
348926414,28sleep30-21swapper/311:57:463
348926414,28sleep30-21swapper/311:57:463
2580626412,37sleep325808-21sshd07:58:393
253532649,12sleep20-21swapper/211:23:092
2331126423,19sleep30-21swapper/309:33:093
1922526422,19sleep10-21swapper/111:03:131
1922526422,19sleep10-21swapper/111:03:131
137912646,12sleep10-21swapper/109:03:221
137912646,12sleep10-21swapper/109:03:221
122892649,18sleep30-21swapper/312:23:153
122892649,18sleep30-21swapper/312:23:153
332126319,21sleep10-21swapper/110:13:051
2151826313,34sleep221573-21tune2fs09:28:072
1139726312,15sleep10-21swapper/108:58:011
1139726312,15sleep10-21swapper/108:58:011
1047026312,21sleep20-21swapper/212:18:092
1047026312,21sleep20-21swapper/212:18:092
681326215,32sleep16848-21sed12:07:461
486426220,20sleep30-21swapper/308:37:483
3037326217,25sleep30-21swapper/311:38:123
3037326217,25sleep30-21swapper/311:38:123
2863526218,27sleep328743-21grep08:08:163
2057826218,26sleep320637-21/usr/sbin/munin11:08:073
2057826218,26sleep320637-21/usr/sbin/munin11:08:073
107962626,12sleep20-21swapper/207:13:072
61826115,29sleep3653-21grep10:04:113
2962726115,19sleep10-21swapper/109:53:141
2624226120,21sleep10-21swapper/111:27:441
372726015,29sleep23768-21/usr/sbin/munin08:33:062
350126014,18sleep10-21swapper/108:32:571
3200426014,19sleep30-21swapper/310:02:453
3200426014,19sleep30-21swapper/310:02:453
1061626013,28sleep110653-21ntp_states12:18:121
1061626013,28sleep110653-21ntp_states12:18:121
1012226018,22sleep20-21swapper/208:53:092
914825910,34sleep29214-21systemctl07:08:142
637725912,31sleep26384-21df_inode10:23:032
55025918,20sleep30-21swapper/308:23:103
392025918,25sleep33419-21apt-get11:57:553
2265325919,22sleep30-21swapper/309:32:443
1811025917,27sleep118188-21apt-config07:37:451
982625819,18sleep30-21swapper/308:53:023
768825814,28sleep37811-21cat12:08:173
3213625814,19sleep10-21swapper/110:02:471
3213625814,19sleep10-21swapper/110:02:471
2796825816,25sleep227962-21apt-get08:07:592
2449825811,26sleep10-21swapper/109:37:551
940725717,24sleep39403-21apt-get10:32:583
860725719,22sleep18604-21apt-get12:12:551
327112579,35sleep20-21swapper/211:47:122
264625713,25sleep20-21swapper/210:09:392
1997425715,22sleep20-21swapper/209:23:102
140852575,10sleep20-21swapper/209:06:582
140852575,10sleep20-21swapper/209:06:582
1098325716,24sleep310978-21apt-get10:37:583
1098325716,24sleep310978-21apt-get10:37:583
2761525612,26sleep227611-21apt-get09:47:582
260362567,40sleep20-21swapper/209:42:532
1928025617,18sleep20-21swapper/211:03:142
1928025617,18sleep20-21swapper/211:03:142
179942567,13sleep30-21swapper/309:17:503
522125510,34sleep15258-21ps10:18:141
3170725523,19sleep10-21swapper/108:18:151
2923925520,15sleep30-21swapper/309:53:043
2395125516,15sleep10-21swapper/111:18:151
145182555,13sleep30-21swapper/312:32:193
983925416,19sleep10-21swapper/108:53:041
865925414,24sleep38655-21apt-get07:07:593
3118525412,16sleep20-21swapper/208:18:042
2629625413,18sleep30-21swapper/311:27:463
2328725413,24sleep323285-21apt-get07:52:583
1941525419,18sleep10-21swapper/107:38:511
162825414,22sleep31623-21apt-get10:07:583
849925320,12sleep10-21swapper/110:28:151
575825315,18sleep30-21swapper/308:38:163
2038325314,16sleep20-21swapper/211:08:022
2038325314,16sleep20-21swapper/211:08:022
1609825313,18sleep30-21swapper/312:37:443
475925211,25sleep34755-21apt-get10:18:003
338625215,21sleep23442-21irqstats10:13:062
2195425215,18sleep10-21swapper/107:48:071
596225014,21sleep25968-21ntp_states12:03:112
550925012,22sleep35576-21df12:03:003
517925012,22sleep25185-21diskstats08:38:032
187672508,18sleep20-21swapper/209:18:382
3170424912,23sleep331761-21irqstats11:43:053
2779724915,17sleep10-21swapper/111:32:451
2768724913,21sleep127693-21df_inode09:48:011
243282496,33sleep10-21swapper/107:54:541
1763324912,22sleep317656-21ntp_states10:58:103
819424713,19sleep38192-21apt-get08:47:573
847624611,23sleep18512-21wc08:48:071
526124610,18sleep20-21swapper/210:18:152
1228524614,16sleep20-21swapper/210:42:462
1228524614,16sleep20-21swapper/210:42:462
906324510,20sleep39069-21ntp_states12:13:093
2403824514,18sleep224116-21find11:18:162
2009224514,18sleep30-21swapper/309:23:153
1416724512,16sleep30-21swapper/310:48:043
253842387,15sleep30-21swapper/309:38:183
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional