You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 00:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sun May 04, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
319482324515,3217sleep318341-21kworker/3:009:43:363
2434728838,863sleep224387-21ntpq11:03:282
25223256811,542sleep325262-21ntpq07:38:303
1072240912,383sleep11075-21ntpq09:48:351
25126239653,286sleep225154-21fschecks_count09:23:232
25126239653,286sleep225154-21fschecks_count09:23:232
31750229121,250sleep331874-21/usr/sbin/munin07:58:383
9996224090,105sleep20-21swapper/208:33:352
17514223546,131sleep217658-21latency_hist10:43:052
30111223124,187sleep230202-21xargs07:53:362
14563222649,118sleep20-21swapper/208:48:292
28858222225,158sleep228921-21cpuspeed11:18:192
22222221929,133sleep222266-21latency09:13:252
7710221642,117sleep20-21swapper/211:53:252
7710221642,117sleep20-21swapper/211:53:252
2728022168,46sleep227299-21apt-get11:13:142
20496221616,119sleep220568-21fschecks_time09:08:252
13036221611,35sleep213063-21munin-plugin-st07:03:022
24390221312,75sleep223473-21/usr/sbin/munin09:18:392
24390221312,75sleep223473-21/usr/sbin/munin09:18:392
7511220742,146sleep27655-21unixbench_multi10:08:352
26980220711,145sleep226591-21/usr/sbin/munin09:28:322
28553220227,95sleep228619-21ls09:33:332
13005219911,177sleep3370-21snmpd10:28:073
13005219911,177sleep3370-21snmpd10:28:073
6521219815,167sleep36588-21unin-run10:08:033
11787219613,164sleep111851-21cpuspeed06:58:211
12593219520,154sleep012587-21apt-get08:43:180
13215219128,124sleep213372-21modprobe07:03:072
338021889,165sleep23418-21grep08:13:272
4861218433,118sleep24871-21open_files11:43:322
82121808,17sleep20-21swapper/209:48:272
20564217711,127sleep220565-21cron10:53:022
17582217117,23sleep00-21swapper/012:23:360
15053216711,122sleep215106-21latency_hist12:18:042
9966216510,142sleep19993-21df10:18:191
9966216510,142sleep19993-21df10:18:191
10697216587,28sleep210729-21fschecks_count12:03:222
9178216444,75sleep29276-21users10:13:362
9178216444,75sleep29276-21users10:13:362
18797216113,30sleep20-21swapper/209:03:162
18797216113,30sleep20-21swapper/209:03:162
12288215753,68sleep212302-21fschecks_count12:08:232
16859215616,129sleep216866-21wc07:13:212
21731215128,51sleep20-21swapper/207:28:152
22596215028,59sleep222654-21if_eth010:58:232
3237321489,120sleep20-21swapper/208:03:052
26887214819,106sleep126979-21/usr/sbin/munin09:28:311
8782214518,102sleep38825-21unin-run11:58:023
8782214518,102sleep38825-21unin-run11:58:023
663621409,84sleep20-21swapper/208:23:252
2827221398,12sleep10-21swapper/111:14:141
886721385,12sleep30-21swapper/308:28:393
23817213421,29sleep20-21swapper/207:33:362
10176213437,38sleep30-21swapper/306:53:193
5574213322,26sleep30-21swapper/308:18:373
2998213310,108sleep23026-21munin-run08:13:032
3466213233,87sleep23469-21latency_hist09:58:052
1887721326,88sleep218934-21ps07:18:342
21204213110,80sleep221244-21sendmail07:28:032
13673213127,38sleep20-21swapper/210:28:512
13673213127,38sleep20-21swapper/210:28:512
10318213177,42sleep210350-21munin-run12:03:022
6519212915,41sleep20-21swapper/211:48:362
15198212616,67sleep215305-21timerandwakeup10:33:352
15198212616,67sleep215305-21timerandwakeup10:33:352
32742212431,50sleep2355-21sh09:48:032
20934212242,36sleep20-21swapper/210:53:232
2632212133,73sleep22738-21apt-key11:38:062
24941212020,81sleep20-21swapper/207:38:252
6586211825,28sleep10-21swapper/108:23:241
17991211648,56sleep218002-21kernelversion10:43:252
12724211650,50sleep212743-21sendmail10:28:022
12724211650,50sleep212743-21sendmail10:28:022
1217121166,101sleep00-21swapper/012:08:090
7433211511,90sleep10-21swapper/110:08:341
6141211527,76sleep26203-21sed08:23:052
20245211517,83sleep020265-21latency_hist09:08:060
20245211517,83sleep020265-21latency_hist09:08:060
1218321140,9sleep10-21swapper/110:23:361
1218321140,9sleep10-21swapper/110:23:361
2161211329,43sleep00-21swapper/009:53:180
4835210920,43sleep10-21swapper/108:18:191
91521060,10sleep00-21swapper/009:48:290
23474210637,36sleep10-21swapper/109:18:171
23474210637,36sleep10-21swapper/109:18:171
15402210625,54sleep315467-21fschecks_time07:08:253
15402210625,54sleep315467-21fschecks_time07:08:253
23140210419,60sleep123276-21grep10:58:351
31978210319,33sleep00-21swapper/011:28:190
3975210216,70sleep34027-21latency09:58:263
31614210214,59sleep00-21swapper/009:43:290
30311210223,36sleep20-21swapper/209:38:302
15761210229,28sleep20-21swapper/212:18:332
15585210240,24sleep215653-21latency07:08:282
15585210240,24sleep215653-21latency07:08:282
9831210117,59sleep19044-21/usr/sbin/munin11:58:371
2954210120,60sleep02985-21tune2fs11:38:330
24801210117,59sleep024869-21latency_hist09:23:050
24801210117,59sleep024869-21latency_hist09:23:050
11366210124,35sleep20-21swapper/208:38:282
4107210018,61sleep14132-21basename11:43:021
15386210019,58sleep115421-21fschecks_count12:18:241
2776229910,26sleep00-21swapper/011:14:050
1909429814,65sleep119186-21/usr/sbin/munin09:03:271
1909429814,65sleep119186-21/usr/sbin/munin09:03:271
1681229810,74sleep316816-21find12:23:143
1656929816,62sleep116622-21users08:53:371
1337429516,59sleep313485-21irqstats07:03:083
813329417,56sleep38100-21latency_hist10:13:043
470129219,30sleep20-21swapper/209:59:422
817829119,56sleep08322-21latency_hist10:13:050
2203929116,51sleep322099-21fschecks_time09:13:223
2731729013,59sleep327299-21apt-get11:13:233
846228816,20sleep10-21swapper/110:13:231
846228816,20sleep10-21swapper/110:13:231
83028814,40sleep20-21swapper/211:33:052
2729528812,63sleep211573-21cyclictest07:43:382
1975628713,54sleep019279-21/usr/sbin/munin10:48:320
2373328624,43sleep323875-21sort11:03:053
1858628613,55sleep018734-21date09:03:050
1858628613,55sleep018734-21date09:03:050
1579128622,44sleep115292-21/usr/sbin/munin07:08:351
1579128622,44sleep115292-21/usr/sbin/munin07:08:351
1392828614,54sleep314045-21latency_hist08:48:063
63992855,27sleep20-21swapper/210:06:372
2815328512,56sleep028259-21cat09:33:070
1820828525,25sleep10-21swapper/112:28:051
1735028525,36sleep20-21swapper/208:58:232
1735028525,36sleep20-21swapper/208:58:232
1683728510,35sleep20-21swapper/212:23:172
1372228511,60sleep013734-21cat12:13:140
432728422,44sleep04447-21timerwakeupswit09:58:350
347128323,31sleep30-21swapper/309:58:063
139272837,39sleep213926-21munin-run08:48:052
110928317,31sleep20-21swapper/208:04:062
1091328323,43sleep211083-21unixbench_singl06:53:382
235328219,45sleep02432-21find08:08:410
1292928220,44sleep313075-21grep12:08:363
1080828227,26sleep10-21swapper/106:53:371
2837028122,30sleep10-21swapper/109:33:161
2560428111,32sleep20-21swapper/211:08:212
192128116,47sleep31933-21sort09:53:053
250182808,16sleep30-21swapper/309:23:163
250182808,16sleep30-21swapper/309:23:163
3079827926,42sleep230834-21ntpq11:23:282
2992727817,25sleep10-21swapper/107:53:311
284372787,33sleep20-21swapper/207:48:352
501927719,24sleep00-21swapper/008:18:260
2339727723,24sleep00-21swapper/007:33:260
2087627717,47sleep320932-21processes09:08:343
1881027713,44sleep318380-21/usr/sbin/munin07:18:313
1840527739,23sleep10-21swapper/107:18:201
1571827722,37sleep20-21swapper/208:53:182
111572775,61sleep00-21swapper/006:54:120
10682776,30sleep20-21swapper/211:33:172
10682776,30sleep20-21swapper/211:33:172
749927616,30sleep00-21swapper/011:53:150
749927616,30sleep00-21swapper/011:53:150
304032768,51sleep00-21swapper/011:23:180
188062765,59sleep00-21swapper/010:45:450
2190827526,26sleep10-21swapper/107:28:431
1920027519,28sleep30-21swapper/309:03:293
1920027519,28sleep30-21swapper/309:03:293
991627419,15sleep20-21swapper/210:18:162
991627419,15sleep20-21swapper/210:18:162
1419527421,27sleep00-21swapper/007:03:330
1311627415,40sleep113020-21/usr/sbin/munin10:28:211
1311627415,40sleep113020-21/usr/sbin/munin10:28:211
2678127320,32sleep326801-21munin-run11:13:033
2648227323,22sleep00-21swapper/007:43:240
157992737,14sleep00-21swapper/010:37:290
157992737,14sleep00-21swapper/010:37:290
501027216,40sleep14972-21latency_hist10:03:051
224202729,49sleep022455-21ps09:13:320
1060827213,39sleep010764-21unixbench_multi10:18:350
1060827213,39sleep010764-21unixbench_multi10:18:350
750827114,33sleep10-21swapper/111:53:161
750827114,33sleep10-21swapper/111:53:161
318122715,13sleep30-21swapper/311:28:053
2928127121,24sleep30-21swapper/307:53:043
2796227110,45sleep10-21swapper/107:48:181
245627123,27sleep10-21swapper/111:38:051
245627123,27sleep10-21swapper/111:38:051
2377827113,39sleep123816-21processes07:33:351
2239427114,39sleep022475-21cat07:28:540
1181927117,31sleep311017-21/usr/sbin/munin08:38:373
949827011,42sleep09565-21df08:33:210
2944027018,33sleep029532-21/usr/sbin/munin11:18:330
2688227015,35sleep127032-21apt-key11:13:051
2125227013,29sleep30-21swapper/307:28:043
1584727016,36sleep11-21systemd10:38:031
1584727016,36sleep11-21systemd10:38:031
88626918,35sleep31029-21modprobe11:33:063
88626918,35sleep31029-21modprobe11:33:063
659026912,42sleep06654-21cut10:08:040
3189026919,35sleep131928-21ntpq09:43:341
2961126916,30sleep30-21swapper/307:53:253
213132697,14sleep30-21swapper/310:53:323
1816026920,23sleep10-21swapper/107:18:041
1208026917,31sleep012182-21/usr/sbin/munin10:23:360
1208026917,31sleep012182-21/usr/sbin/munin10:23:360
59772689,13sleep00-21swapper/011:48:220
2935626817,36sleep329435-21awk11:18:323
1052326813,32sleep310605-21grep10:18:343
1052326813,32sleep310605-21grep10:18:343
21172675,10sleep00-21swapper/011:35:220
21172675,10sleep00-21swapper/011:35:220
2056726711,38sleep020669-21timerandwakeup07:23:370
1639726716,19sleep00-21swapper/010:38:260
150126720,28sleep10-21swapper/108:08:051
487226619,21sleep30-21swapper/311:43:333
1969826618,31sleep019712-21latency_hist07:23:040
174826617,24sleep10-21swapper/108:08:211
1203026618,26sleep00-21swapper/012:08:050
605926517,25sleep30-21swapper/311:48:253
415826512,41sleep34138-21latency_hist11:43:043
2682126521,29sleep026881-21cmp11:13:040
2067926519,22sleep10-21swapper/109:08:281
182526517,31sleep31832-21/usr/sbin/munin08:08:233
1569626519,31sleep315744-21grep12:18:303
635226419,28sleep36353-21munin-node08:23:143
3097526413,37sleep331054-21switchtime11:23:333
307442647,38sleep00-21swapper/007:57:380
2883026413,26sleep10-21swapper/111:18:141
1840326416,23sleep30-21swapper/310:43:343
1714126410,23sleep10-21swapper/110:39:361
132472645,13sleep10-21swapper/112:10:201
93572635,14sleep00-21swapper/010:14:310
93572635,14sleep00-21swapper/010:14:310
795226311,14sleep20-21swapper/208:28:182
3002926316,20sleep10-21swapper/109:38:251
454526219,27sleep14551-21modprobe08:18:031
3270226210,21sleep30-21swapper/308:03:263
262226218,28sleep12646-21proc_pri09:53:311
3168226114,26sleep10-21swapper/111:28:031
285252615,17sleep00-21swapper/009:33:310
2558726117,18sleep10-21swapper/109:23:341
2558726117,18sleep10-21swapper/109:23:341
2493426114,32sleep024935-21sshd11:04:360
1740026120,23sleep10-21swapper/108:58:261
1740026120,23sleep10-21swapper/108:58:261
958226010,21sleep30-21swapper/308:33:243
74522605,19sleep00-21swapper/008:25:410
283622608,11sleep30-21swapper/309:33:143
2121426016,24sleep00-21swapper/010:53:290
1472026013,30sleep314796-21sed08:48:343
1463226013,24sleep10-21swapper/107:03:381
135202605,11sleep10-21swapper/108:43:481
1308026017,26sleep013089-21munin-run07:03:050
970625913,31sleep09830-21sed11:58:360
3114225915,23sleep10-21swapper/107:58:241
2546125913,25sleep10-21swapper/107:38:361
1734125916,21sleep30-21swapper/307:13:343
3098725812,34sleep0370-21snmpd09:40:220
1479925818,23sleep014912-21wc08:48:350
968625717,17sleep30-21swapper/310:18:053
968625717,17sleep30-21swapper/310:18:053
867725720,20sleep00-21swapper/008:28:360
38625715,24sleep1422-21users11:28:361
2797125710,24sleep00-21swapper/007:48:210
1793725719,19sleep10-21swapper/110:43:241
1727925715,26sleep017311-21ntp_states07:13:310
1426525715,24sleep114371-21tune2fs08:48:241
1153025714,20sleep00-21swapper/008:38:340
949725614,25sleep39520-21ntp_states11:58:313
459825617,21sleep04569-21latency_hist08:18:060
439825613,25sleep04393-21apt-get11:43:170
2222325616,23sleep10-21swapper/110:58:061
148522565,11sleep10-21swapper/112:16:061
119632568,20sleep10-21swapper/108:38:391
2647525514,24sleep326480-21diskstats07:43:233
2349625514,24sleep323492-21apt-get09:18:193
2349625514,24sleep323492-21apt-get09:18:193
1435125515,19sleep00-21swapper/010:33:050
1435125515,19sleep00-21swapper/010:33:050
376225416,19sleep30-21swapper/308:13:363
2464225413,25sleep324785-21latency_hist07:38:063
1574625417,21sleep015748-21ntp_states12:18:310
309282535,10sleep30-21swapper/309:39:363
2235625312,23sleep10-21swapper/109:13:291
1419025311,26sleep314201-21ntp_states12:13:293
1203625314,23sleep312055-21irqstats06:58:273
3262025214,18sleep00-21swapper/008:03:230
1930125211,24sleep319297-21apt-get10:48:193
1135325211,18sleep30-21swapper/312:03:363
259642515,34sleep00-21swapper/009:23:460
259642515,34sleep00-21swapper/009:23:460
2285525114,21sleep00-21swapper/010:58:300
2168925113,23sleep021711-21munin-run09:13:030
203452516,12sleep10-21swapper/110:50:101
1190225110,22sleep30-21swapper/310:23:313
1190225110,22sleep30-21swapper/310:23:313
634625016,17sleep10-21swapper/111:48:311
2173225014,16sleep30-21swapper/307:28:163
1577825016,16sleep00-21swapper/007:08:340
1577825016,16sleep00-21swapper/007:08:340
2809724912,22sleep327878-21apt-get09:33:063
1632624912,16sleep30-21swapper/308:53:353
1080324911,17sleep10-21swapper/108:38:061
3107424810,23sleep031141-21df07:58:230
2507424813,19sleep00-21swapper/007:38:270
1495624811,24sleep314565-21/usr/sbin/munin10:33:283
1495624811,24sleep314565-21/usr/sbin/munin10:33:283
1995624715,18sleep10-21swapper/107:23:211
1771324712,21sleep317733-21ntp_states08:58:323
1771324712,21sleep317733-21ntp_states08:58:323
437324611,16sleep10-21swapper/111:43:141
2402624611,20sleep324031-21diskstats11:03:213
1572124611,19sleep015791-21df08:53:200
925824514,15sleep10-21swapper/108:33:051
2773224412,17sleep127865-21cat07:48:051
1086024311,15sleep10-21swapper/112:03:251
531724210,15sleep10-21swapper/110:03:211
240752419,13sleep10-21swapper/111:03:231
171192387,15sleep30-21swapper/310:39:343
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional