You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 15:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sat Nov 22, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29222211621,2081sleep31933-21kworker/3:208:49:073
16711282721,786sleep116847-21/usr/sbin/munin07:49:161
16711282721,786sleep116847-21/usr/sbin/munin07:49:161
595125359,511sleep15995-21ntpq08:59:021
595125359,511sleep15995-21ntpq08:59:021
3169223706,352sleep331736-21ntpq12:04:053
6067235016,204sleep36210-21unixbench_singl12:24:123
47012322146,121sleep34846-21unixbench_multi08:54:123
453023130,6sleep10-21swapper/108:54:091
21938229840,246sleep322010-21latency_hist08:08:403
21938229840,246sleep322010-21latency_hist08:08:403
312332288112,157sleep331273-21processes10:19:053
18993228511,262sleep019040-21ntpq11:24:030
21714227415,112sleep321744-21latency_hist11:33:403
16425223654,35sleep316428-21cron09:33:373
16425223654,35sleep316428-21cron09:33:373
22388222713,202sleep322443-21interrupts08:09:003
18098221911,194sleep318188-21switchtime07:54:103
21113221815,187sleep121137-21/usr/sbin/munin08:04:221
21113221815,187sleep121137-21/usr/sbin/munin08:04:221
17129221712,104sleep30-21swapper/311:18:583
10653221438,149sleep30-21swapper/309:14:023
312422137,191sleep33210-21latency07:08:433
837522079,183sleep38436-21http12:33:373
16805220515,173sleep116938-21latency_hist11:18:391
27288220019,154sleep30-21swapper/311:49:103
11981219520,31sleep30-21swapper/307:34:113
11981219520,31sleep30-21swapper/307:34:113
10908219323,147sleep110966-21cut10:59:091
13152218918,153sleep313155-21munin-run09:23:403
13152218918,153sleep313155-21munin-run09:23:403
30236218412,143sleep30-21swapper/311:59:083
9806218021,107sleep30-21swapper/307:28:583
9806218021,107sleep30-21swapper/307:28:583
28982217519,16sleep30-21swapper/310:13:393
2607217314,20sleep30-21swapper/312:14:063
31542216615,49sleep331627-21latency_hist08:38:413
16506216551,17sleep30-21swapper/307:49:143
16506216551,17sleep30-21swapper/307:49:143
6178216314,109sleep36270-21timerandwakeup08:59:083
6178216314,109sleep36270-21timerandwakeup08:59:083
14019216237,110sleep314055-21ntp_kernel_err09:24:243
14019216237,110sleep314055-21ntp_kernel_err09:24:243
9176216156,17sleep30-21swapper/309:09:103
282021598,139sleep31-21systemd07:08:393
24616215712,109sleep3362-21snmpd09:58:573
28739215632,55sleep30-21swapper/308:28:593
17236215216,115sleep017302-21sendmail_mailqu09:34:150
17236215216,115sleep017302-21sendmail_mailqu09:34:150
3011214821,44sleep30-21swapper/310:33:553
3011214821,44sleep30-21swapper/310:33:553
19600214013,92sleep319632-21latency_hist09:43:393
19600214013,92sleep319632-21latency_hist09:43:393
25578213819,19sleep30-21swapper/308:19:023
29621213614,107sleep329669-21ntpq10:14:043
2840021359,84sleep30-21swapper/308:28:393
777221349,86sleep37865-21cut10:49:133
20985213214,97sleep221000-21/usr/sbin/munin08:04:032
20985213214,97sleep221000-21/usr/sbin/munin08:04:032
20468213125,65sleep320590-21cat09:44:153
20468213125,65sleep320590-21cat09:44:153
6708212719,88sleep16763-21fschecks_count07:18:591
629212738,72sleep30-21swapper/312:08:533
19057212796,18sleep319086-21ps11:24:073
1717421225,16sleep00-21swapper/007:53:180
1717421225,16sleep00-21swapper/007:53:180
11351212210,46sleep30-21swapper/310:59:193
6268212116,68sleep36303-21fschecks_time10:44:043
6268212116,68sleep36303-21fschecks_time10:44:043
24768212112,45sleep30-21swapper/311:43:373
19725212112,46sleep319865-21timerwakeupswit07:59:113
1427521199,20sleep30-21swapper/311:09:043
952211737,63sleep3973-21apt07:03:383
32431211714,14sleep30-21swapper/308:39:133
7548211613,16sleep30-21swapper/309:04:053
5089211615,81sleep00-21swapper/012:23:400
456621158,19sleep30-21swapper/310:38:553
2886321137,11sleep10-21swapper/108:29:021
16745211310,17sleep30-21swapper/309:33:533
16745211310,17sleep30-21swapper/309:33:533
8256211012,40sleep30-21swapper/307:23:573
26692211010,83sleep226697-21/usr/sbin/munin11:48:572
26358211039,57sleep3225-21systemd-journal11:48:383
2138021098,16sleep30-21swapper/309:48:523
2138021098,16sleep30-21swapper/309:48:523
2052210921,70sleep32177-21timerwakeupswit10:29:103
3232521078,88sleep332316-21apt-get10:23:573
3210721070,8sleep212-21rcu_sched10:23:392
20128210717,71sleep119867-21apt-get11:28:411
2655621067,16sleep30-21swapper/310:04:143
214821069,42sleep32196-21systemd-run08:48:383
10462210615,72sleep010597-21unixbench_singl07:29:110
10462210615,72sleep010597-21unixbench_singl07:29:110
28181210511,34sleep30-21swapper/311:53:513
28181210511,34sleep30-21swapper/311:53:513
9666210413,33sleep30-21swapper/310:54:103
9666210413,33sleep30-21swapper/310:54:103
14971210411,16sleep30-21swapper/307:44:133
2747210312,77sleep02832-21mailstats12:14:110
11186210315,73sleep211231-21awk10:59:142
1065210336,42sleep11200-21cmp07:03:401
4696210210,14sleep30-21swapper/312:19:193
3307210213,75sleep13308-21sshd12:16:181
1185210115,72sleep11208-21latency_hist10:28:401
23951210012,12sleep30-21swapper/308:14:013
770729818,12sleep30-21swapper/307:21:273
51482979,33sleep30-21swapper/307:14:003
1492829714,54sleep214933-21latency_hist09:28:392
1492829714,54sleep214933-21latency_hist09:28:392
1417829614,67sleep114228-21latency11:09:021
1358729617,59sleep013103-21apt-get09:23:490
1358729617,59sleep013103-21apt-get09:23:490
124242968,31sleep30-21swapper/311:03:553
998029411,65sleep29993-21irqstats07:29:012
998029411,65sleep29993-21irqstats07:29:012
59729418,56sleep232565-21apt-get12:08:482
3020429413,65sleep130212-21acpi08:33:501
2950229415,59sleep229550-21latency_hist11:58:382
2950229415,59sleep229550-21latency_hist11:58:382
222222947,31sleep30-21swapper/311:34:043
1558529411,70sleep0362-21snmpd07:47:450
138422938,29sleep30-21swapper/307:40:173
138422938,29sleep30-21swapper/307:40:173
997429124,42sleep110112-21sed10:54:161
997429124,42sleep110112-21sed10:54:161
658729115,56sleep06597-21modprobe12:28:370
1809129117,51sleep118157-21latency_hist09:38:401
1809129117,51sleep118157-21latency_hist09:38:401
240662896,30sleep30-21swapper/309:56:193
1003528913,52sleep110047-21modprobe09:13:401
98228825,34sleep01050-21unin-run07:03:390
2327228814,57sleep223389-21cat11:38:392
113042887,63sleep00-21swapper/007:33:560
113042887,63sleep00-21swapper/007:33:560
666428721,43sleep26774-21dpkg12:28:392
2011128615,55sleep220124-21irqstats09:44:062
2011128615,55sleep220124-21irqstats09:44:062
167382865,29sleep30-21swapper/311:18:253
722328510,61sleep07236-21fschecks_count09:03:580
817428410,21sleep00-21swapper/007:23:540
2223728411,14sleep00-21swapper/011:34:050
2161428414,55sleep021664-21/usr/sbin/munin09:49:050
2161428414,55sleep021664-21/usr/sbin/munin09:49:050
586728323,47sleep25846-21latency_hist10:43:392
493428215,45sleep14961-21awk10:39:061
2518128219,33sleep10-21swapper/111:43:591
2349128211,33sleep10-21swapper/111:38:551
120228219,43sleep21408-21dpkg07:03:412
844428017,49sleep18477-21latency_hist09:08:391
2930628015,46sleep129345-21fschecks_count10:13:581
1517528024,36sleep015209-21modprobe11:13:380
3266727913,50sleep1374-21latency_hist12:08:391
1459327917,41sleep214650-21if_eth007:44:042
298842780,6sleep20-21swapper/210:14:102
2897127822,39sleep228981-21unin-run10:13:382
285492785,13sleep10-21swapper/111:54:011
285492785,13sleep10-21swapper/111:54:011
1151227819,35sleep10-21swapper/107:34:011
1151227819,35sleep10-21swapper/107:34:011
277827719,30sleep10-21swapper/110:33:401
218027716,31sleep00-21swapper/007:04:240
193627715,49sleep0362-21snmpd08:45:530
869127610,53sleep00-21swapper/009:08:550
2493427517,33sleep20-21swapper/209:59:052
2073027510,13sleep10-21swapper/109:44:171
2073027510,13sleep10-21swapper/109:44:171
1338227519,26sleep10-21swapper/109:23:421
2707527411,50sleep027123-21processes11:49:070
1954427414,28sleep00-21swapper/007:59:080
1448127421,24sleep20-21swapper/211:09:112
1410127419,35sleep014120-21munin-run07:43:370
1410127419,35sleep014120-21munin-run07:43:370
844127316,38sleep08475-21munin-run12:33:380
2364127318,28sleep00-21swapper/009:54:140
1949627318,42sleep00-21swapper/009:42:510
1949627318,42sleep00-21swapper/009:42:510
1219027324,19sleep00-21swapper/011:03:400
381427221,28sleep10-21swapper/108:53:411
174882726,17sleep20-21swapper/207:53:452
713627111,34sleep20-21swapper/209:03:522
56172717,12sleep00-21swapper/010:40:580
2035327121,32sleep119867-21apt-get11:28:491
143127116,15sleep00-21swapper/010:28:550
1300027117,35sleep213064-21/usr/sbin/munin07:39:032
1300027117,35sleep213064-21/usr/sbin/munin07:39:032
1005027119,27sleep20-21swapper/209:13:412
2670927015,40sleep126598-21latency_hist08:23:401
2489327013,42sleep225011-21cat11:43:402
1358327013,41sleep10-21swapper/109:23:461
1358327013,41sleep10-21swapper/109:23:461
475326915,36sleep14775-21apt07:13:371
2616326915,23sleep00-21swapper/010:03:590
611826814,36sleep16093-21/usr/sbin/munin10:43:541
611826814,36sleep16093-21/usr/sbin/munin10:43:541
45026816,27sleep00-21swapper/010:24:150
2346526827,23sleep10-21swapper/109:54:111
2147726813,38sleep121484-21/usr/sbin/munin09:48:581
2147726813,38sleep121484-21/usr/sbin/munin09:48:581
2059426822,23sleep00-21swapper/009:44:160
2059426822,23sleep00-21swapper/009:44:160
1751026814,39sleep00-21swapper/011:19:070
1091026815,28sleep10-21swapper/109:14:091
103712687,38sleep20-21swapper/209:13:572
3116726714,39sleep031211-21ntpq10:19:020
224826718,23sleep10-21swapper/108:48:401
1549126712,41sleep015546-21sort11:13:540
135652676,15sleep00-21swapper/011:08:000
91926615,28sleep10-21swapper/112:09:161
859426617,19sleep20-21swapper/212:33:402
423726617,23sleep20-21swapper/212:19:062
34342667,14sleep10-21swapper/110:34:091
34342667,14sleep10-21swapper/110:34:091
3149026627,22sleep00-21swapper/008:38:380
2738726613,37sleep027403-21uname08:24:160
1561726612,35sleep215703-21/usr/sbin/munin11:14:002
3196926516,32sleep131756-21/usr/sbin/munin08:39:031
2566026520,22sleep00-21swapper/011:44:110
2396326512,38sleep023974-21ntpq11:39:100
2262626511,34sleep022636-21ntp_states08:09:060
1607726517,26sleep10-21swapper/109:29:191
1607726517,26sleep10-21swapper/109:29:191
1343026516,35sleep113488-21sed07:39:141
1343026516,35sleep113488-21sed07:39:141
932326417,29sleep19325-21sshd07:27:191
721926416,16sleep10-21swapper/112:29:111
691026420,20sleep20-21swapper/209:03:392
691026420,20sleep20-21swapper/209:03:392
572626410,38sleep25852-21wc07:14:152
3235726418,25sleep20-21swapper/208:39:122
3215526428,21sleep00-21swapper/008:39:060
281522649,15sleep10-21swapper/110:09:091
1573326414,21sleep20-21swapper/209:29:162
1573326414,21sleep20-21swapper/209:29:162
952226316,21sleep00-21swapper/010:54:070
952226316,21sleep00-21swapper/010:54:070
68882637,12sleep00-21swapper/007:19:020
462326313,34sleep24700-21grep08:54:112
404626316,18sleep10-21swapper/112:19:011
2989126310,43sleep00-21swapper/008:33:210
29212638,11sleep20-21swapper/210:33:412
29212638,11sleep20-21swapper/210:33:412
2381626320,19sleep20-21swapper/208:13:562
2255026316,29sleep20-21swapper/208:09:042
2174726314,33sleep021896-21sed11:33:410
1762026315,22sleep10-21swapper/107:54:001
1449826322,18sleep10-21swapper/107:44:031
3256226218,19sleep20-21swapper/206:59:052
903926117,19sleep10-21swapper/110:53:401
414126116,17sleep20-21swapper/207:09:102
291126117,28sleep23040-21/usr/sbin/munin12:14:132
2662026117,20sleep00-21swapper/008:23:390
2386526116,20sleep20-21swapper/209:54:162
1564226120,24sleep00-21swapper/009:29:140
1564226120,24sleep00-21swapper/009:29:140
639426016,20sleep20-21swapper/210:44:072
639426016,20sleep20-21swapper/210:44:072
556626017,26sleep05652-21mailstats07:14:130
385926012,32sleep03920-21users10:34:160
385926012,32sleep03920-21users10:34:160
2609026020,18sleep20-21swapper/210:03:572
2451226018,23sleep00-21swapper/009:58:500
1235626018,20sleep20-21swapper/211:03:542
115026022,19sleep20-21swapper/208:44:012
69742597,15sleep00-21swapper/010:44:180
69742597,15sleep00-21swapper/010:44:180
566125916,19sleep00-21swapper/008:58:570
566125916,19sleep00-21swapper/008:58:570
49792594,16sleep00-21swapper/008:54:410
369425916,23sleep00-21swapper/007:09:000
298825924,20sleep00-21swapper/008:49:080
2263825910,35sleep122646-21ntpq08:09:071
216182595,14sleep00-21swapper/008:05:100
216182595,14sleep00-21swapper/008:05:100
1851925913,26sleep218577-21/usr/sbin/munin09:39:102
1851925913,26sleep218577-21/usr/sbin/munin09:39:102
1309425915,26sleep213151-21dpkg09:23:392
1309425915,26sleep213151-21dpkg09:23:392
1209825913,25sleep10-21swapper/109:19:011
2884625822,19sleep00-21swapper/011:54:120
2884625822,19sleep00-21swapper/011:54:120
2552825815,27sleep225566-21processes11:44:092
2193525812,17sleep10-21swapper/111:33:501
3031825711,20sleep00-21swapper/011:59:090
2724525715,18sleep10-21swapper/108:24:131
1323825714,25sleep013374-21apt-config09:23:410
1323825714,25sleep013374-21apt-config09:23:410
1232925717,18sleep10-21swapper/111:03:491
943425614,27sleep29483-21idleruntime10:54:052
943425614,27sleep29483-21idleruntime10:54:052
659825615,25sleep11-21systemd12:28:381
253125620,20sleep20-21swapper/208:48:582
2273525619,17sleep20-21swapper/209:53:392
2273525619,17sleep20-21swapper/209:53:392
1965525613,18sleep20-21swapper/207:59:102
1912325616,23sleep119215-21users09:39:211
1912325616,23sleep119215-21users09:39:211
1731225614,23sleep20-21swapper/211:19:012
120825617,24sleep00-21swapper/012:09:230
793325515,17sleep20-21swapper/207:23:402
3129525511,25sleep231362-21df12:03:552
2972125514,23sleep00-21swapper/010:14:090
2764025513,23sleep027707-21df10:08:560
2500025510,15sleep10-21swapper/109:59:081
1985125516,23sleep219905-21systemd-run11:28:382
1070025516,22sleep010729-21ntp_states09:14:040
94142545,11sleep20-21swapper/207:28:292
2468825415,20sleep00-21swapper/008:14:190
2107625412,23sleep221200-21sed11:29:272
190625418,17sleep10-21swapper/107:04:171
176102547,14sleep00-21swapper/007:53:590
861025312,28sleep0362-21snmpd10:49:350
3035125312,18sleep20-21swapper/208:33:592
2531425312,23sleep225310-21apt-get08:18:532
565325211,28sleep15725-21wc07:14:141
2749325215,16sleep20-21swapper/208:24:182
2376925213,21sleep10-21swapper/108:13:551
1011825219,17sleep10-21swapper/107:29:041
1011825219,17sleep10-21swapper/107:29:041
3259225114,21sleep00-21swapper/012:08:380
295332518,16sleep20-21swapper/208:29:272
1913325112,19sleep10-21swapper/107:58:571
1613625117,13sleep20-21swapper/207:49:042
1613625117,13sleep20-21swapper/207:49:042
3019725013,22sleep130203-21ntp_states11:59:061
2984025017,16sleep20-21swapper/211:58:592
2712425012,15sleep10-21swapper/111:49:081
142825013,21sleep21424-21apt-get10:28:532
3232924813,21sleep232396-21df10:24:002
1725424712,17sleep10-21swapper/111:19:001
1138124711,21sleep211386-21diskstats07:33:582
1138124711,21sleep211386-21diskstats07:33:582
795524511,15sleep20-21swapper/210:49:162
260842458,18sleep10-21swapper/110:03:521
168182459,20sleep216825-21df_inode09:33:572
168182459,20sleep216825-21df_inode09:33:572
307672417,15sleep10-21swapper/110:18:501
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional