You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 21:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Dec 02, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
145452515213,5127sleep113584-21kworker/1:212:30:211
30485246813,438sleep030491-21cat08:13:210
2112245212,426sleep02052-21latency_hist11:53:050
10141229514,197sleep210289-21unixbench_singl10:33:362
300222526,234sleep23043-21ntpq08:28:292
30187224912,196sleep230332-21wc08:13:072
3627223733,143sleep23666-21proc_pri10:13:332
3627223733,143sleep23666-21proc_pri10:13:332
1143222149,25sleep20-21swapper/210:38:322
1334622088,162sleep213229-21/usr/sbin/munin07:18:232
5797218511,128sleep25709-21/usr/sbin/munin08:38:232
13912217983,70sleep20-21swapper/209:03:302
30402217813,153sleep130437-21grep08:13:181
25995217444,119sleep226010-21latency_hist09:43:042
25995217444,119sleep226010-21latency_hist09:43:042
4304217248,108sleep24377-21fschecks_time08:33:262
9947216816,135sleep010021-21switchtime10:33:340
1800921689,81sleep20-21swapper/207:33:212
1800921689,81sleep20-21swapper/207:33:212
1994621676,13sleep30-21swapper/307:38:313
1535216747,105sleep21600-21sendmail_mailst08:23:332
1535216747,105sleep21600-21sendmail_mailst08:23:332
26284216224,125sleep226362-21switchtime07:58:372
14830216246,74sleep214927-21latency_hist12:33:052
28757215511,91sleep228800-21ntpq11:33:482
28757215511,91sleep228800-21ntpq11:33:482
28757215511,91sleep228800-21ntpq11:33:482
27554215457,79sleep227645-21latency_hist09:48:052
17101215321,109sleep117234-21sed07:28:441
1690921529,128sleep216905-21apt-get09:13:162
1146621518,129sleep211528-21sed07:13:062
5461215070,61sleep20-21swapper/212:03:202
30676215030,65sleep20-21swapper/211:38:412
937221490,10sleep20-21swapper/208:48:352
10795214869,21sleep20-21swapper/208:53:292
2033214512,86sleep22036-21cron11:53:022
5218214312,111sleep25266-21latency_hist12:03:042
17274213619,105sleep017339-21uniq10:58:250
7551213439,75sleep27617-21latency08:43:272
20885213316,17sleep00-21swapper/011:08:350
7574213113,104sleep07655-21latency_hist10:28:040
1635921299,58sleep216432-21gpgv09:13:032
60021286,86sleep25666-21snmpd08:20:582
28083212711,56sleep20-21swapper/211:33:042
18820212718,86sleep318853-21fschecks_count11:03:253
28278212622,89sleep128362-21mailstats09:48:321
19172212310,63sleep20-21swapper/211:03:322
15217212348,21sleep20-21swapper/207:23:322
8512212110,60sleep20-21swapper/212:13:062
20760212140,61sleep20-21swapper/209:23:462
20760212140,61sleep20-21swapper/209:23:462
2124821198,18sleep20-21swapper/207:43:222
26850211819,21sleep10-21swapper/109:43:351
14569211814,82sleep114699-21latency_hist07:23:061
23152211742,26sleep20-21swapper/209:33:232
23152211742,26sleep20-21swapper/209:33:232
22597211717,82sleep022655-21sort07:48:250
1472821167,67sleep00-21swapper/010:48:360
23028211463,35sleep20-21swapper/207:48:352
21116211422,69sleep021182-21fschecks_time07:43:200
22963211311,87sleep1313-21rs:main1
22963211311,87sleep1313-21rs:main1
15552211315,81sleep315558-21/usr/sbin/munin09:08:323
30086211212,74sleep30-21swapper/308:13:043
23651211225,22sleep20-21swapper/211:18:182
2679210916,76sleep32686-21/usr/sbin/munin08:28:223
16531210920,72sleep016720-21cat09:13:050
24780210818,65sleep124798-21sort09:38:221
24780210818,65sleep124798-21sort09:38:221
1769221088,51sleep217783-21mailstats10:58:352
892821078,79sleep08979-21fschecks_count08:48:260
22827210711,55sleep222831-21cron09:33:022
10156210712,77sleep010106-21apt-get07:08:440
879721060,6sleep10-21swapper/108:48:061
22335210619,72sleep122395-21processes11:13:321
18027210617,69sleep018118-21fschecks_time07:33:230
18027210617,69sleep018118-21fschecks_time07:33:230
479210445,47sleep25666-21snmpd10:03:312
24562210445,44sleep224600-21ntpq07:53:282
12621210415,66sleep012720-21timerwakeupswit08:58:360
12621210415,66sleep012720-21timerwakeupswit08:58:360
10622210413,77sleep110629-21sh07:09:051
5182210343,41sleep20-21swapper/210:18:332
29640210334,48sleep20-21swapper/209:53:262
1786210317,59sleep11924-21grep08:23:361
1786210317,59sleep11924-21grep08:23:361
17784210323,55sleep317865-21awk10:58:363
730721026,10sleep20-21swapper/210:25:162
27356210215,73sleep027381-21fschecks_count08:03:230
25847210022,54sleep025933-21fschecks_time07:58:270
986029925,46sleep39192-21apt-get07:08:103
3188429932,16sleep20-21swapper/211:43:252
2579129912,21sleep225181-21/usr/sbin/munin11:23:352
2579129912,21sleep225181-21/usr/sbin/munin11:23:352
2579129912,21sleep225181-21/usr/sbin/munin11:23:352
988229817,63sleep39374-21/usr/sbin/munin10:33:333
133629822,55sleep01462-21grep10:08:050
1543629720,19sleep20-21swapper/210:53:062
129612979,72sleep012979-21open_inodes10:43:310
2902029615,65sleep129103-21timerwakeupswit11:33:551
2902029615,65sleep129103-21timerwakeupswit11:33:551
2902029615,65sleep129103-21timerwakeupswit11:33:551
2861929641,15sleep20-21swapper/208:08:062
216542967,13sleep30-21swapper/309:28:233
1455229639,46sleep21-21systemd10:48:342
539529524,51sleep15536-21sed10:18:361
2805429519,65sleep228071-21cut09:48:252
2074829521,59sleep20-21swapper/211:08:332
121262959,65sleep212181-21if_err_eth012:23:232
1010429510,72sleep210106-21cron12:18:022
1010429510,72sleep210106-21cron12:18:022
231142947,53sleep20-21swapper/211:17:532
770529313,60sleep07866-21latency_hist07:03:060
930929222,23sleep10-21swapper/108:48:351
76029227,45sleep1779-21cron08:23:031
3061229217,51sleep10-21swapper/111:38:401
1097129214,62sleep311043-21cat08:53:343
946829120,46sleep09646-21fschecks_count07:08:070
3092629136,39sleep20-21swapper/209:58:182
765229035,31sleep30-21swapper/307:03:053
2495829018,57sleep024977-21kernelversion09:38:250
2495829018,57sleep024977-21kernelversion09:38:250
290432896,69sleep00-21swapper/009:52:550
219352898,12sleep00-21swapper/009:28:280
276928820,13sleep20-21swapper/211:53:272
58192877,12sleep00-21swapper/008:38:250
3171128714,50sleep030902-21/usr/sbin/munin09:58:370
883728511,21sleep10-21swapper/112:13:171
883728511,21sleep10-21swapper/112:13:171
511728521,44sleep34690-21/usr/sbin/munin10:18:303
37942855,69sleep00-21swapper/008:32:110
1088228520,40sleep20-21swapper/212:18:352
89092846,68sleep20-21swapper/210:30:492
3009428410,61sleep030117-21apt-get11:38:150
2370628331,27sleep30-21swapper/309:33:403
2370628331,27sleep30-21swapper/309:33:403
631428230,29sleep20-21swapper/206:58:162
199342829,56sleep20-21swapper/207:38:302
1255128210,14sleep30-21swapper/308:58:353
1255128210,14sleep30-21swapper/308:58:353
698628125,28sleep10-21swapper/106:58:351
2096328122,39sleep121107-21tail11:08:361
145682816,27sleep00-21swapper/007:23:050
1230528112,52sleep112314-21/usr/sbin/munin12:23:261
2593428025,31sleep10-21swapper/107:58:281
1842328023,26sleep10-21swapper/107:33:321
1842328023,26sleep10-21swapper/107:33:321
171832807,22sleep10-21swapper/110:58:181
1232128017,47sleep312398-21grep07:13:393
119732809,16sleep20-21swapper/208:58:172
119732809,16sleep20-21swapper/208:58:172
3193727915,31sleep30-21swapper/311:43:273
2843727916,41sleep328564-21timerwakeupswit09:48:343
2195027916,24sleep30-21swapper/311:13:223
1977427911,44sleep20-21swapper/209:23:052
687927815,48sleep37005-21timerwakeupswit10:23:353
1363027816,42sleep113719-21grep09:03:251
664227723,26sleep10-21swapper/110:23:291
2601127726,29sleep30-21swapper/309:43:053
2601127726,29sleep30-21swapper/309:43:053
1475527724,35sleep014786-21sendmail12:33:020
916127616,43sleep09244-21latency_hist10:33:050
53427620,28sleep10-21swapper/111:48:051
3011027619,42sleep330257-21unixbench_singl09:53:353
1530727623,36sleep315312-21modprobe10:53:033
721827520,31sleep10-21swapper/112:08:241
887427414,27sleep00-21swapper/012:13:400
887427414,27sleep00-21swapper/012:13:400
3111227415,15sleep10-21swapper/109:58:251
248112748,53sleep00-21swapper/011:21:390
191112749,49sleep019237-21sed09:18:350
165572746,19sleep00-21swapper/010:53:530
1381327420,22sleep00-21swapper/009:03:280
117082747,49sleep20-21swapper/207:13:182
2322527319,40sleep323340-21/usr/sbin/munin07:48:403
2064427323,26sleep10-21swapper/109:23:451
2064427323,26sleep10-21swapper/109:23:451
1114127321,25sleep10-21swapper/112:18:381
84582718,18sleep20-21swapper/207:03:332
2377627119,21sleep00-21swapper/009:33:410
2377627119,21sleep00-21swapper/009:33:410
2149027114,16sleep30-21swapper/307:43:273
214427119,24sleep10-21swapper/111:53:061
167227120,34sleep20-21swapper/210:08:242
163592718,47sleep20-21swapper/207:28:202
930827019,30sleep39375-21munin-node07:08:063
897827016,31sleep30-21swapper/312:13:593
897827016,31sleep30-21swapper/312:13:593
307792709,46sleep20-21swapper/208:13:272
292562706,12sleep30-21swapper/311:33:583
292562706,12sleep30-21swapper/311:33:583
292562706,12sleep30-21swapper/311:33:583
2651827016,28sleep00-21swapper/011:28:050
2651827016,28sleep00-21swapper/011:28:050
2651827016,28sleep00-21swapper/011:28:050
1998927023,25sleep30-21swapper/309:23:143
1998927023,25sleep30-21swapper/309:23:143
394226920,31sleep13879-21latency_hist08:33:051
281926915,37sleep12883-21/usr/sbin/munin11:53:301
2569826916,38sleep325708-21find07:58:183
1846826916,28sleep10-21swapper/109:18:191
1265926913,36sleep112719-21/usr/sbin/munin10:43:241
124712697,51sleep20-21swapper/210:43:132
796926814,39sleep38098-21unixbench_multi08:43:373
599326820,28sleep00-21swapper/012:03:350
45162680,7sleep00-21swapper/010:18:060
45162680,7sleep00-21swapper/010:18:060
229912680,21sleep122568-21/usr/sbin/munin07:48:331
1228826821,21sleep30-21swapper/310:43:053
917826713,38sleep19219-21gpgv07:08:041
2816826715,25sleep00-21swapper/011:33:060
2543126716,34sleep125433-21cron07:58:031
2204726719,23sleep10-21swapper/109:28:331
1743126714,23sleep10-21swapper/109:14:071
1562526715,31sleep10-21swapper/109:08:341
948326419,28sleep19490-21/usr/sbin/munin10:33:221
2959426418,29sleep029633-21http11:38:020
2959426418,29sleep029633-21http11:38:020
2959426418,29sleep029633-21http11:38:020
292226419,28sleep33038-21latency_hist10:13:053
2223626414,36sleep00-21swapper/011:13:270
86126315,28sleep30-21swapper/311:48:233
835626319,19sleep00-21swapper/012:13:030
582926312,22sleep10-21swapper/108:38:261
418326318,22sleep30-21swapper/311:58:293
366826319,24sleep30-21swapper/310:13:343
366826319,24sleep30-21swapper/310:13:343
346426310,22sleep10-21swapper/110:13:281
346426310,22sleep10-21swapper/110:13:281
2548226322,19sleep30-21swapper/307:58:053
2365226318,29sleep323644-21apt-get11:18:193
206482635,17sleep00-21swapper/007:41:470
1184126325,20sleep10-21swapper/107:13:291
428326216,31sleep14316-21/usr/sbin/munin11:58:311
281952625,11sleep30-21swapper/308:03:433
2520526217,28sleep125199-21apt-get11:23:171
2520526217,28sleep125199-21apt-get11:23:171
2520526217,28sleep125199-21apt-get11:23:171
2038826215,21sleep00-21swapper/009:23:310
2038826215,21sleep00-21swapper/009:23:310
81426121,17sleep00-21swapper/008:23:060
483326113,34sleep14953-21timerwakeupswit08:33:371
2496826123,19sleep10-21swapper/111:23:051
1249326120,27sleep312567-21switchtime12:23:333
1244026121,19sleep10-21swapper/108:58:331
1244026121,19sleep10-21swapper/108:58:331
593026019,22sleep30-21swapper/312:03:333
457726013,20sleep30-21swapper/308:33:313
3120026013,28sleep331294-21/usr/sbin/munin09:58:273
2955226019,24sleep00-21swapper/009:53:240
132632607,12sleep30-21swapper/307:18:203
810225917,24sleep08197-21users08:43:380
299802597,43sleep00-21swapper/008:11:390
2675025920,24sleep326753-21apt-get11:28:163
2234825921,17sleep10-21swapper/107:48:061
1441925913,32sleep114025-21/usr/sbin/munin10:48:291
871925813,33sleep18728-21vmstat10:28:381
47592586,11sleep00-21swapper/008:33:360
2744825817,26sleep127586-21grep11:28:431
1913225817,24sleep119170-21ntpq11:03:311
1608925821,21sleep10-21swapper/110:53:331
1547225810,20sleep30-21swapper/307:23:413
1481825814,26sleep014804-21apt-get07:23:210
132025813,21sleep30-21swapper/308:23:253
132025813,21sleep30-21swapper/308:23:253
892225715,25sleep38926-21diskstats08:48:253
400125715,19sleep00-21swapper/011:58:260
3176225722,17sleep00-21swapper/011:43:230
2764225719,23sleep127668-21sshd08:03:281
1349725717,23sleep313564-21df12:28:213
1300725716,19sleep00-21swapper/007:18:040
782325616,24sleep37817-21apt-get10:28:203
272242566,19sleep30-21swapper/309:44:263
22672566,38sleep00-21swapper/008:27:260
22672566,38sleep00-21swapper/008:27:260
1688525613,28sleep09375-21munin-node09:13:140
1157225620,19sleep00-21swapper/010:38:340
1104825619,19sleep00-21swapper/008:53:350
828625516,26sleep38310-21munin-run12:13:023
1698925516,17sleep00-21swapper/007:28:430
1696925514,25sleep316905-21apt-get09:13:533
1570825516,23sleep315712-21diskstats10:53:233
598125413,24sleep36004-21irqstats08:38:283
3095725413,24sleep331023-21switchtime08:13:323
1636125413,25sleep316354-21apt-get07:28:213
606825314,16sleep10-21swapper/112:03:361
367025319,16sleep30-21swapper/311:58:043
2927125314,23sleep129280-21ntp_states08:08:291
1613725314,23sleep116285-21grep07:28:071
3236525215,21sleep332407-21ntp_states08:18:293
3009125211,23sleep30-21swapper/311:38:143
2960625212,24sleep129639-21irqstats09:53:251
2415925217,18sleep00-21swapper/007:53:180
1871125213,22sleep018729-21apt-get11:03:180
2624325113,21sleep026237-21apt-get09:43:180
1295325114,23sleep312959-21ntp_states10:43:303
129032515,35sleep00-21swapper/012:24:140
1170725112,15sleep00-21swapper/012:23:050
1092325111,19sleep10-21swapper/110:38:151
1047425111,15sleep30-21swapper/312:18:243
60825016,17sleep10-21swapper/110:03:341
479725014,18sleep00-21swapper/010:18:230
1041825013,15sleep10-21swapper/108:53:201
234924913,16sleep00-21swapper/011:53:170
157324913,21sleep01568-21apt-get10:08:180
1970024813,18sleep10-21swapper/107:38:251
1855324814,17sleep30-21swapper/307:33:343
1855324814,17sleep30-21swapper/307:33:343
2524424713,20sleep325277-21df11:23:193
2524424713,20sleep325277-21df11:23:193
2524424713,20sleep325277-21df11:23:193
8332466,15sleep30-21swapper/310:03:363
218424610,15sleep10-21swapper/110:08:351
1402524111,12sleep30-21swapper/309:03:353
203502399,15sleep30-21swapper/311:08:213
193802397,13sleep30-21swapper/309:18:383
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional