You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 17:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Sep 18, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19012619619,6164sleep33545-21kworker/3:107:44:193
21901289644,814sleep223475-1kworker/2:1H07:08:052
25299247125,431sleep125375-21/usr/sbin/munin08:57:521
25262335144,169sleep22618-21fschecks_time07:47:452
16784224917,211sleep00-21swapper/010:12:550
9174224849,182sleep29194-21kernelversion08:07:472
25251224168,21sleep20-21swapper/212:22:412
25251224168,21sleep20-21swapper/212:22:412
611222517,195sleep3615-21uname09:22:533
18180222595,113sleep217757-21/usr/sbin/munin10:17:542
22436222212,34sleep222558-21latency_hist10:32:272
8029221411,19sleep20-21swapper/208:02:592
8029221411,19sleep20-21swapper/208:02:592
22463221220,177sleep322590-21timerwakeupswit07:08:313
6400220611,174sleep26498-21fschecks_time11:22:482
11676220520,158sleep20-21swapper/209:58:002
13839220315,168sleep213914-21latency_hist11:47:262
24429220173,37sleep224579-21unixbench_singl12:18:052
2223921877,144sleep222283-21ntpq12:12:482
2092621877,11sleep00-21swapper/012:07:540
2142218128,133sleep22155-21modprobe07:47:252
24479217412,111sleep224491-21kworker/u8:108:57:242
2493217317,104sleep20-21swapper/209:28:082
29730216822,16sleep20-21swapper/207:32:272
4103216616,124sleep20-21swapper/209:32:572
28879216512,140sleep028887-21acpi10:52:390
28879216512,140sleep028887-21acpi10:52:390
8256216258,19sleep20-21swapper/209:47:462
8256216258,19sleep20-21swapper/209:47:462
338216236,21sleep2359-21latency_hist07:42:242
29218216269,78sleep20-21swapper/209:12:262
3159321599,136sleep231662-21swap09:17:552
24127215752,22sleep20-21swapper/207:12:492
10263215415,82sleep20-21swapper/209:52:552
15066214224,53sleep20-21swapper/210:07:492
15066214224,53sleep20-21swapper/210:07:492
16420214123,34sleep116529-21users11:52:581
15389214127,23sleep20-21swapper/211:52:262
2896721398,67sleep229003-21ntpq12:32:552
18512213546,15sleep20-21swapper/208:37:392
30561213420,94sleep330655-21cut10:57:433
23934213425,91sleep323949-21modprobe10:37:243
2706321335,64sleep20-21swapper/212:27:582
12685213254,66sleep212820-21cat08:18:012
30236213115,24sleep20-21swapper/210:57:252
30236213115,24sleep20-21swapper/210:57:252
24522213079,40sleep224596-21cat10:37:482
31797212715,23sleep20-21swapper/211:02:262
26687212551,51sleep20-21swapper/209:02:502
8613212422,82sleep38623-21modprobe08:07:243
8613212422,82sleep38623-21modprobe08:07:243
19233212458,49sleep219298-21awk12:02:562
31778212117,88sleep031926-21sed09:17:570
30801212156,54sleep21-21systemd09:17:272
17698212018,28sleep00-21swapper/011:57:530
17698212018,28sleep00-21swapper/011:57:530
598211958,49sleep2610-21ntpq09:22:522
32429211814,86sleep132463-21/usr/sbin/munin11:02:491
27008211721,77sleep326528-21apt-get12:27:353
6380211617,84sleep325395-1kworker/3:2H09:42:263
27886211655,42sleep20-21swapper/210:47:532
27886211655,42sleep20-21swapper/210:47:532
852211518,76sleep0355-21apt-get07:42:340
700021158,95sleep1362-21snmpd08:01:331
700021158,95sleep1362-21snmpd08:01:331
3670211560,35sleep20-21swapper/211:13:202
16869211421,72sleep117002-21sed10:12:561
28184211318,81sleep328215-21latency09:07:473
22058211350,43sleep20-21swapper/208:47:522
18151211360,38sleep20-21swapper/211:58:242
18151211360,38sleep20-21swapper/211:58:242
10931211355,43sleep210996-21awk08:12:562
10739211312,70sleep30-21swapper/311:37:263
1608321116,57sleep00-21swapper/011:52:530
25692211010,83sleep025703-21ntp_states12:22:510
25692211010,83sleep025703-21ntp_states12:22:510
16405211031,57sleep116428-21munin-run08:32:251
9970210910,87sleep210061-21switchtime11:32:532
5807210921,67sleep05792-21apt-get07:57:440
5807210921,67sleep05792-21apt-get07:57:440
29263210917,76sleep129385-21cat12:33:021
22674210847,18sleep20-21swapper/210:32:392
14321210718,64sleep014459-21sed08:23:070
12173210718,66sleep112177-21fschecks_count08:17:481
27677210620,70sleep327748-21latency_hist09:07:273
3986210546,15sleep20-21swapper/207:52:412
20613210510,20sleep20-21swapper/208:42:552
19529210555,16sleep219055-21apt-get10:22:302
25805210441,16sleep20-21swapper/210:42:402
25805210441,16sleep20-21swapper/210:42:402
12092210315,70sleep012163-21sensors_temp09:58:100
6589210143,16sleep20-21swapper/209:42:332
24794210143,15sleep20-21swapper/208:57:372
20682210112,69sleep320778-21/usr/sbin/munin08:42:563
16165210141,16sleep20-21swapper/210:12:372
22329210011,71sleep122391-21/usr/sbin/munin07:08:291
2143210016,65sleep12184-21/usr/sbin/munin09:27:591
20278210043,20sleep20-21swapper/212:07:282
2662329710,63sleep20-21swapper/207:22:262
2662329710,63sleep20-21swapper/207:22:262
1852629723,58sleep118572-21cat12:02:261
1852629723,58sleep118572-21cat12:02:261
1802029723,48sleep118036-21munin-run08:37:251
7282968,77sleep2362-21snmpd11:05:572
1439529615,68sleep014431-21latency_hist10:07:260
1228929610,72sleep20-21swapper/211:42:262
2896129518,61sleep329028-21switchtime07:27:523
3122329427,45sleep331248-21munin-run07:37:253
3122329427,45sleep331248-21munin-run07:37:253
2119929417,60sleep321227-21fschecks_count10:27:443
2045129429,32sleep00-21swapper/007:03:070
2057129326,33sleep20-21swapper/207:03:092
2026129335,29sleep30-21swapper/307:03:023
2862929225,44sleep028779-21sed09:07:560
140152927,15sleep20-21swapper/208:23:012
1097529224,45sleep310994-21apt-get09:57:243
2782929122,47sleep127981-21unixbench_singl12:28:301
1095229111,67sleep110970-21apt11:37:361
2696228928,30sleep00-21swapper/007:22:480
2696228928,30sleep00-21swapper/007:22:480
78512887,14sleep20-21swapper/211:27:412
62152885,71sleep20-21swapper/209:40:562
2808028824,38sleep028226-21grep10:47:550
2808028824,38sleep028226-21grep10:47:550
2422828813,27sleep30-21swapper/310:37:373
1509928821,32sleep00-21swapper/010:07:510
1509928821,32sleep00-21swapper/010:07:510
972828713,15sleep10-21swapper/109:52:421
59272875,70sleep20-21swapper/207:58:152
59272875,70sleep20-21swapper/207:58:152
486328720,20sleep10-21swapper/109:37:281
1649228716,53sleep016550-21latency_hist08:32:280
1443228721,34sleep10-21swapper/110:07:271
58442865,70sleep20-21swapper/211:20:142
271062867,13sleep20-21swapper/207:22:502
271062867,13sleep20-21swapper/207:22:502
150152850,7sleep10-21swapper/110:07:461
150152850,7sleep10-21swapper/110:07:461
1434928515,47sleep20-21swapper/210:07:242
539528411,17sleep30-21swapper/307:57:273
1707928413,50sleep017102-21/usr/sbin/munin08:32:560
2791828313,55sleep127987-21df09:07:411
13442837,36sleep20-21swapper/211:07:352
313172829,57sleep231477-21sort07:37:282
313172829,57sleep231477-21sort07:37:282
2547528220,39sleep324772-21/usr/sbin/munin08:57:543
3163028120,28sleep10-21swapper/107:37:431
2017328130,29sleep10-21swapper/107:03:001
1956328119,38sleep019556-21apt-get10:22:400
97328012,51sleep01058-21users09:22:580
572528025,25sleep30-21swapper/311:18:463
531427920,29sleep10-21swapper/111:17:551
512827822,26sleep00-21swapper/011:17:500
2233127817,42sleep121847-21/usr/sbin/munin12:12:521
1285327821,34sleep112824-21latency_hist10:02:271
194427715,47sleep31941-21apt-get09:27:413
468727612,28sleep00-21swapper/007:53:000
36327623,35sleep3459-21sshd07:42:253
2431927623,24sleep10-21swapper/112:18:041
169542766,54sleep20-21swapper/208:32:542
137227623,33sleep11432-21processes07:43:081
369127518,37sleep1225-21systemd-journal07:52:241
2533427525,31sleep125352-21fschecks_count12:22:441
2533427525,31sleep125352-21fschecks_count12:22:441
166612745,12sleep20-21swapper/211:54:282
161412746,11sleep20-21swapper/208:28:592
1157227426,21sleep00-21swapper/008:15:240
299227310,14sleep30-21swapper/311:12:283
2968027313,46sleep01-21systemd07:32:250
2796227324,26sleep30-21swapper/310:47:543
2796227324,26sleep30-21swapper/310:47:543
2631027323,24sleep00-21swapper/010:42:540
2631027323,24sleep00-21swapper/010:42:540
1171927326,26sleep30-21swapper/309:58:013
2382827221,29sleep10-21swapper/107:12:441
286727112,43sleep02886-21ntp_states07:47:510
2230727118,38sleep022328-21awk07:08:280
1928727115,15sleep10-21swapper/108:38:221
297302707,51sleep00-21swapper/012:35:510
2664027018,36sleep326821-21cat12:27:273
2664027018,36sleep326821-21cat12:27:273
2078027014,40sleep020932-21awk08:42:580
1475827016,39sleep014911-21awk11:47:570
1173527018,39sleep111729-21cron08:17:271
1008327018,21sleep00-21swapper/009:52:490
2501026917,38sleep325049-21latency_hist12:22:263
1884826917,36sleep018852-21diskstats12:02:460
1658626930,23sleep30-21swapper/310:12:503
866526823,21sleep10-21swapper/108:07:261
866526823,21sleep10-21swapper/108:07:261
2710626820,28sleep30-21swapper/310:47:253
2710626820,28sleep30-21swapper/310:47:253
2665526824,22sleep30-21swapper/307:22:273
2753426718,34sleep127593-21cat10:47:441
2753426718,34sleep127593-21cat10:47:441
2349326723,21sleep10-21swapper/108:52:471
1960926718,29sleep30-21swapper/310:23:073
186226717,27sleep30-21swapper/311:08:083
1860226721,24sleep00-21swapper/008:37:450
166126712,21sleep10-21swapper/111:08:021
141942675,19sleep10-21swapper/111:47:451
1388126719,24sleep30-21swapper/308:22:583
76622666,11sleep00-21swapper/009:44:370
855926520,24sleep30-21swapper/311:27:583
761226518,26sleep00-21swapper/011:27:260
69572650,7sleep10-21swapper/111:22:591
3084526517,24sleep00-21swapper/010:57:480
2320526520,28sleep323233-21munin-run12:17:263
2189926521,25sleep30-21swapper/308:47:493
754726416,29sleep00-21swapper/008:02:490
754726416,29sleep00-21swapper/008:02:490
629826417,25sleep30-21swapper/307:58:243
629826417,25sleep30-21swapper/307:58:243
524026412,30sleep15266-21munin-run07:57:251
3211826420,23sleep30-21swapper/311:02:433
2984226421,24sleep00-21swapper/009:12:530
2578226421,22sleep10-21swapper/107:17:531
2578226421,22sleep10-21swapper/107:17:531
185662647,26sleep10-21swapper/110:18:011
1220026421,20sleep30-21swapper/308:17:503
2589326310,28sleep30-21swapper/307:17:553
2443526321,21sleep00-21swapper/010:37:460
2381726314,20sleep00-21swapper/007:12:430
84326213,24sleep00-21swapper/011:07:240
46642624,11sleep30-21swapper/309:36:173
366226212,36sleep13669-21sh11:13:191
937426113,21sleep00-21swapper/008:07:540
265722610,7sleep00-21swapper/009:02:460
2329426111,35sleep023303-21df_inode08:52:420
2288426116,20sleep10-21swapper/110:32:461
2214026118,22sleep00-21swapper/012:12:460
2031426115,27sleep120307-21apt-get12:07:391
1786626114,30sleep00-21swapper/010:17:480
134726120,21sleep00-21swapper/011:07:370
104542616,9sleep10-21swapper/111:34:461
942926015,25sleep10-21swapper/109:52:251
942926015,25sleep10-21swapper/109:52:251
669426016,22sleep30-21swapper/311:22:533
269162607,13sleep10-21swapper/110:45:301
269162607,13sleep10-21swapper/110:45:301
265026013,31sleep02812-21sed09:28:110
182042607,11sleep30-21swapper/311:58:573
182042607,11sleep30-21swapper/311:58:573
897725912,19sleep10-21swapper/108:07:421
3236225919,18sleep10-21swapper/109:22:261
2974425915,27sleep329839-21memory09:12:513
2190025917,22sleep00-21swapper/010:27:570
2072825917,18sleep30-21swapper/312:07:493
163522597,11sleep30-21swapper/308:31:423
135022597,10sleep30-21swapper/311:43:443
1223925913,25sleep00-21swapper/008:17:510
92052586,43sleep30-21swapper/309:49:333
92052586,43sleep30-21swapper/309:49:333
349525815,21sleep00-21swapper/009:32:430
2379525820,21sleep00-21swapper/012:17:510
153222586,12sleep30-21swapper/311:52:113
1181025817,20sleep30-21swapper/308:17:293
34825716,19sleep10-21swapper/109:22:471
3015125720,17sleep00-21swapper/007:32:450
3015125720,17sleep00-21swapper/007:32:450
2850025713,26sleep128595-21perl07:27:421
2425225714,25sleep124247-21apt-get10:37:391
2199525714,28sleep022012-21netstat08:47:500
1818625714,25sleep318394-21sshd08:37:273
1334025714,24sleep313363-21irqstats10:02:493
544925619,23sleep00-21swapper/009:37:510
2611825615,19sleep00-21swapper/009:02:270
2314925614,20sleep00-21swapper/010:32:530
825425514,18sleep10-21swapper/111:27:521
633025513,21sleep10-21swapper/109:42:241
243125515,23sleep32428-21apt-get07:47:383
2132525512,30sleep121090-21/usr/sbin/munin10:27:461
167042556,38sleep30-21swapper/311:55:023
661125413,25sleep36608-21apt-get09:42:343
3231025413,25sleep332311-21cron09:22:243
2338325416,17sleep30-21swapper/307:12:303
2007025412,24sleep120140-21df08:42:421
1913125416,20sleep30-21swapper/310:22:273
1874225415,22sleep118761-21apt-get12:02:381
1253025413,24sleep012524-21apt-get11:42:400
1152225411,30sleep311590-21awk11:37:553
2927425314,18sleep30-21swapper/310:52:543
2927425314,18sleep30-21swapper/310:52:543
2163025315,21sleep121699-21df08:47:431
1498425314,17sleep10-21swapper/108:27:281
1097725314,23sleep010972-21apt-get11:37:390
787825214,22sleep11-21systemd09:47:251
327322527,35sleep00-21swapper/007:41:290
2643825211,25sleep126442-21diskstats09:02:441
2005625212,23sleep120080-21ntp_states10:23:341
1253125213,20sleep10-21swapper/111:42:411
3071125111,18sleep10-21swapper/110:57:451
297332517,33sleep30-21swapper/312:35:543
2599025115,15sleep00-21swapper/007:17:560
2599025115,15sleep00-21swapper/007:17:560
2359525112,19sleep30-21swapper/308:52:493
976525013,15sleep00-21swapper/011:32:470
3004125011,16sleep10-21swapper/107:32:421
3004125011,16sleep10-21swapper/107:32:421
2946825012,21sleep129464-21apt-get09:12:411
2936025013,22sleep129367-21ntp_states10:52:561
2936025013,22sleep129367-21ntp_states10:52:561
2704425013,23sleep127105-21basename07:22:491
2704425013,23sleep127105-21basename07:22:491
1671225012,18sleep30-21swapper/308:32:473
857724910,14sleep00-21swapper/009:47:520
857724910,14sleep00-21swapper/009:47:520
380324913,21sleep03881-21find11:13:220
2771024918,15sleep00-21swapper/012:28:290
2006324813,21sleep31-21systemd12:07:253
832524711,17sleep00-21swapper/011:27:550
1463724710,18sleep30-21swapper/310:07:373
1463724710,18sleep30-21swapper/310:07:373
101652467,16sleep30-21swapper/308:12:253
268752388,15sleep30-21swapper/307:22:433
268752388,15sleep30-21swapper/307:22:433
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional