You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 07:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Dec 01, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19600297018,935sleep119601-21sendmail-mta19:16:271
19690292116,857sleep019015-21kworker/0:222:43:300
15322285442,797sleep015366-21ntpq22:28:360
13475281221,772sleep013565-21/usr/sbin/munin18:58:300
13475281221,772sleep013565-21/usr/sbin/munin18:58:300
11372279219,751sleep111436-21/usr/sbin/munin20:33:451
2058266717,634sleep12073-21/usr/sbin/munin20:03:401
1076626186,601sleep010790-21unin-run23:58:090
2437825737,529sleep024419-21ntpq21:13:330
10144239613,338sleep010294-21head20:28:510
1498723920,41sleep00-21swapper/000:13:270
1360237412,269sleep01426-21wc21:43:380
1360237412,269sleep01426-21wc21:43:380
32263232714,203sleep032280-21awk23:23:080
66692318166,66sleep00-21swapper/020:18:450
4410227623,195sleep04433-21idleruntime-cro20:13:080
4410227623,195sleep04433-21idleruntime-cro20:13:080
204982253111,111sleep00-21swapper/019:18:360
27164224045,172sleep027297-21apt-config23:08:100
7589223442,179sleep07095-21/usr/sbin/munin22:03:370
5996223432,31sleep00-21swapper/023:42:340
5996223432,31sleep00-21swapper/023:42:340
1143222088,185sleep011481-21latency_hist00:03:100
1909122038,98sleep019197-21timerandwakeup19:13:440
26301219810,173sleep126305-21ntpq23:03:381
3864219649,105sleep03979-21timerwakeupswit23:33:420
3864219649,105sleep03979-21timerwakeupswit23:33:420
22119219111,164sleep022124-21modprobe21:08:080
5000218723,107sleep00-21swapper/020:13:360
32550218714,41sleep20-21swapper/223:23:222
32435218415,142sleep00-21swapper/021:38:410
194521747,51sleep00-21swapper/023:28:330
1711821716,129sleep017244-21cat22:33:410
19703216910,122sleep019704-21/usr/sbin/munin00:28:220
13220216821,16sleep00-21swapper/000:08:190
9141216035,67sleep09204-21sendmail-msp23:53:080
29911215730,111sleep029950-21idleruntime-cro21:33:080
13767215767,52sleep013813-21ntpq22:23:350
18135215119,117sleep318163-21fschecks_count22:38:283
11742214667,24sleep00-21swapper/000:03:220
27580214312,111sleep027639-21sendmail_mailqu21:23:390
27580214312,111sleep027639-21sendmail_mailqu21:23:390
19711214326,96sleep00-21swapper/020:58:430
707214210,110sleep00-21swapper/023:23:400
2961421379,20sleep00-21swapper/023:13:350
2961421379,20sleep00-21swapper/023:13:350
3051213652,70sleep03175-21sed21:48:420
3051213652,70sleep03175-21sed21:48:420
2686221367,78sleep026962-21timerandwakeup19:38:390
28181213546,48sleep00-21swapper/019:43:360
23665213545,54sleep00-21swapper/021:12:570
1227721358,117sleep012311-21ps22:18:390
1494321348,69sleep00-21swapper/019:03:260
10009213419,66sleep00-21swapper/022:13:100
6843213217,29sleep10-21swapper/120:18:501
2936821326,108sleep00-21swapper/019:48:250
2936821326,108sleep00-21swapper/019:48:250
13240213241,50sleep00-21swapper/020:38:400
824121309,70sleep00-21swapper/020:23:380
4239212715,94sleep00-21swapper/021:53:310
23002212740,49sleep00-21swapper/022:53:320
23002212740,49sleep00-21swapper/022:53:320
1936021268,105sleep019413-21cat22:43:100
25259212226,69sleep325287-21grep21:18:083
24400212222,78sleep224479-21fschecks_time22:58:302
18235212014,45sleep00-21swapper/019:13:110
14283211812,92sleep114302-21apt-get20:43:221
493211714,84sleep1500-21ntp_states19:58:351
13866211521,76sleep20-21swapper/200:08:372
98021136,59sleep00-21swapper/019:59:030
25922211113,50sleep00-21swapper/021:18:340
25922211113,50sleep00-21swapper/021:18:340
19861211125,72sleep119920-21tune2fs00:28:391
18814211111,65sleep018900-21grep20:58:110
29375211016,67sleep229518-21unixbench_singl21:28:412
23796210922,47sleep00-21swapper/019:28:420
23796210922,47sleep00-21swapper/019:28:420
2062821097,16sleep20-21swapper/219:18:382
7388210620,68sleep27416-21kernelversion22:03:302
2157421068,45sleep00-21swapper/019:23:250
23338210513,76sleep223394-21/usr/sbin/munin19:28:322
23338210513,76sleep223394-21/usr/sbin/munin19:28:322
16149210317,71sleep316150-21sshd19:07:293
30602210211,74sleep130630-21/usr/sbin/munin21:33:331
14671210212,72sleep214726-21latency_hist22:28:102
11571210125,55sleep00-21swapper/022:18:100
11571210125,55sleep00-21swapper/022:18:100
3153929913,70sleep231615-21/usr/sbin/munin19:53:402
1925529933,47sleep019324-21gpgv00:28:090
1343929932,39sleep313472-21grep18:58:293
1343929932,39sleep313472-21grep18:58:293
2445429814,68sleep224460-21ntp_states21:13:352
170582987,76sleep017199-21unixbench_multi00:18:410
1064729827,50sleep110704-21http20:33:081
877229725,52sleep28878-21tune2fs22:08:312
36322975,82sleep112172-1kworker/1:0H21:52:231
36322975,82sleep112172-1kworker/1:0H21:52:231
2855129714,61sleep30-21swapper/319:43:433
1349129721,56sleep113526-21latency00:08:281
251742966,24sleep025238-21ps19:33:390
251742966,24sleep025238-21ps19:33:390
123692965,12sleep00-21swapper/020:36:500
448729511,12sleep30-21swapper/321:53:383
2608629421,33sleep10-21swapper/121:18:391
2608629421,33sleep10-21swapper/121:18:391
3030929313,64sleep330316-21diskstats21:33:273
218332936,11sleep20-21swapper/221:04:312
3235729216,60sleep332430-21switchtime21:38:403
64942916,11sleep00-21swapper/021:58:520
756928949,29sleep00-21swapper/023:47:510
291228925,29sleep30-21swapper/323:33:083
2506528917,17sleep00-21swapper/022:58:450
2114628917,35sleep30-21swapper/322:48:223
1347228815,59sleep313616-21unixbench_singl20:38:433
3031828641,23sleep00-21swapper/021:33:280
1941428612,57sleep119560-21latency_hist22:43:111
1383028518,48sleep313336-21/usr/sbin/munin22:23:383
1343228525,29sleep20-21swapper/218:58:282
1343228525,29sleep20-21swapper/218:58:282
2376128426,28sleep20-21swapper/221:13:112
1722528412,57sleep117364-21latency_hist20:53:111
2921528316,52sleep229216-21http23:13:252
2921528316,52sleep229216-21http23:13:252
2060528210,51sleep20-21swapper/200:28:522
611728125,26sleep30-21swapper/323:43:103
611728125,26sleep30-21swapper/323:43:103
3255528113,46sleep132684-21/usr/sbin/munin21:38:421
3177728134,30sleep00-21swapper/019:53:420
2073128117,25sleep30-21swapper/319:18:393
1697028117,44sleep316993-21fschecks_count19:08:303
1562828132,25sleep10-21swapper/119:03:431
3231428019,45sleep232390-21sed23:23:102
2759428033,32sleep00-21swapper/023:08:180
2046028035,24sleep320663-21apt-key21:03:103
1711028020,10sleep00-21swapper/020:52:540
2883527913,49sleep328890-21if_err_eth021:28:303
641727822,31sleep26511-21cut23:43:292
2712927815,39sleep327155-21munin-run23:08:093
2404127827,32sleep10-21swapper/122:58:091
2404127827,32sleep10-21swapper/122:58:091
2169927835,29sleep00-21swapper/022:48:390
191522787,58sleep1225-21systemd-journal22:41:091
2214427720,39sleep322208-21switchtime19:23:403
1536827717,18sleep10-21swapper/122:28:371
608527618,30sleep20-21swapper/220:18:112
487027611,48sleep24960-21perl23:38:282
487027611,48sleep24960-21perl23:38:282
305712765,12sleep10-21swapper/119:52:341
305712765,12sleep10-21swapper/119:52:341
20472768,12sleep20-21swapper/223:28:352
326732756,12sleep30-21swapper/319:58:293
3150527528,25sleep10-21swapper/123:18:491
265727518,12sleep00-21swapper/020:05:410
2657727417,38sleep126624-21memory19:38:321
1978327415,43sleep319896-21latency_hist19:18:113
505227320,30sleep14754-21/usr/sbin/munin23:38:321
505227320,30sleep14754-21/usr/sbin/munin23:38:321
29102737,12sleep10-21swapper/123:33:061
1553827318,10sleep00-21swapper/020:47:370
594127214,29sleep30-21swapper/321:58:353
453927224,21sleep30-21swapper/323:38:093
453927224,21sleep30-21swapper/323:38:093
2777027222,24sleep30-21swapper/323:08:453
688027120,24sleep20-21swapper/222:03:102
3133827120,33sleep131387-21/usr/sbin/munin19:53:351
1919327113,41sleep319216-21/usr/sbin/munin00:28:083
1279927122,32sleep30-21swapper/300:06:283
614827018,23sleep10-21swapper/121:58:411
314427013,26sleep20-21swapper/220:08:262
314427013,26sleep20-21swapper/220:08:262
2709727020,31sleep227117-21apt-get23:08:082
2497627021,27sleep20-21swapper/219:33:332
2497627021,27sleep20-21swapper/219:33:332
2321327014,25sleep20-21swapper/222:53:382
2321327014,25sleep20-21swapper/222:53:382
2021127019,34sleep30-21swapper/322:43:413
2370326921,28sleep30-21swapper/319:28:413
2370326921,28sleep30-21swapper/319:28:413
319902686,52sleep30-21swapper/323:19:403
2281526816,23sleep30-21swapper/321:08:353
2100926821,23sleep10-21swapper/121:03:531
160326819,31sleep31750-21sed21:43:413
160326819,31sleep31750-21sed21:43:413
1557826826,18sleep20-21swapper/222:28:422
1131526810,48sleep25666-21snmpd22:15:342
1131526810,48sleep25666-21snmpd22:15:342
835526723,22sleep10-21swapper/123:48:371
2933726712,14sleep30-21swapper/323:13:303
2933726712,14sleep30-21swapper/323:13:303
2932826718,30sleep329347-21apt-get19:48:203
2932826718,30sleep329347-21apt-get19:48:203
1793626720,27sleep10-21swapper/100:23:261
1775926725,26sleep317787-21munin-run22:38:083
3154226615,35sleep231672-21latency_hist21:38:102
2309126619,17sleep10-21swapper/121:08:421
1969626620,20sleep30-21swapper/300:28:213
1915326622,21sleep10-21swapper/120:58:321
916026519,23sleep30-21swapper/322:08:403
336726518,28sleep13442-21fw_packets23:33:311
336726518,28sleep13442-21fw_packets23:33:311
2792026516,27sleep20-21swapper/223:08:482
2129726515,23sleep30-21swapper/321:03:593
1634726510,21sleep30-21swapper/300:18:193
960326418,28sleep20-21swapper/220:28:372
931726417,24sleep10-21swapper/122:08:421
842026424,19sleep10-21swapper/120:23:431
320652645,46sleep30-21swapper/319:56:233
2531226427,19sleep10-21swapper/121:18:101
2241126421,22sleep10-21swapper/122:53:081
376726318,21sleep30-21swapper/321:53:103
376726318,21sleep30-21swapper/321:53:103
2911726310,28sleep20-21swapper/219:48:102
2288026319,24sleep10-21swapper/119:28:101
197426315,29sleep22013-21/usr/sbin/munin20:03:372
1709426319,25sleep10-21swapper/119:08:321
1471226328,17sleep30-21swapper/320:43:363
809726220,24sleep28146-21irqstats23:48:302
2673026213,30sleep226789-21/usr/sbin/munin19:38:372
2559726219,21sleep10-21swapper/123:03:101
1363326219,28sleep113670-21grep22:23:321
1193726224,17sleep10-21swapper/100:03:271
709426119,17sleep30-21swapper/322:03:193
651226120,21sleep30-21swapper/323:43:303
524026116,23sleep30-21swapper/320:13:443
446526120,25sleep14471-21ntp_states21:53:361
409526128,19sleep20-21swapper/221:53:292
186442615,11sleep20-21swapper/220:56:352
1744126124,20sleep20-21swapper/219:08:412
1724526121,17sleep10-21swapper/122:33:421
1637226117,27sleep216366-21apt-get00:18:222
1472926119,19sleep30-21swapper/322:28:113
1406926116,28sleep20-21swapper/220:43:102
1312026123,17sleep30-21swapper/322:23:103
1273426110,36sleep112751-21apt20:38:211
863126018,17sleep30-21swapper/320:23:453
573626012,35sleep15666-21snmpd20:14:551
2718826017,23sleep327247-21sshd21:23:303
2718826017,23sleep327247-21sshd21:23:303
2510926012,34sleep125156-21ntp_offset19:33:361
2510926012,34sleep125156-21ntp_offset19:33:361
2115026021,18sleep20-21swapper/222:48:252
1228526021,24sleep212380-21timerandwakeup00:03:362
691225918,19sleep20-21swapper/220:18:512
2759125916,14sleep30-21swapper/319:43:113
2623325925,18sleep30-21swapper/321:18:413
2623325925,18sleep30-21swapper/321:18:413
2601625911,33sleep326089-21cat19:38:103
2601625911,33sleep326089-21cat19:38:103
1840425917,26sleep20-21swapper/200:23:392
1779725919,22sleep30-21swapper/320:53:333
1223125919,23sleep112267-21ntp_states22:18:361
974925817,21sleep39843-21memory23:53:363
944125811,26sleep30-21swapper/320:28:293
762825816,25sleep10-21swapper/122:03:381
2737325816,26sleep227432-21/usr/sbin/munin21:23:332
2737325816,26sleep227432-21/usr/sbin/munin21:23:332
2001825815,27sleep120013-21apt-get19:18:211
1457425817,27sleep314647-21latency_hist00:13:103
1335225816,25sleep113349-21apt-get18:58:231
1335225816,25sleep113349-21apt-get18:58:231
1202925815,24sleep20-21swapper/220:34:102
973425711,31sleep29748-21irqstats23:53:352
947325715,24sleep19543-21df23:53:261
754425714,30sleep37588-21true20:23:083
1612525710,25sleep10-21swapper/100:18:091
1544925722,23sleep315570-21grep00:13:393
1509025726,17sleep20-21swapper/200:13:302
2921425614,26sleep129208-21apt-get23:13:241
2921425614,26sleep129208-21apt-get23:13:241
2759525610,21sleep10-21swapper/123:08:191
149525622,17sleep20-21swapper/221:43:402
149525622,17sleep20-21swapper/221:43:402
1149525616,20sleep30-21swapper/320:33:473
2970325514,23sleep10-21swapper/121:30:301
2197425518,22sleep222015-21ntpq19:23:352
5182544,23sleep30-21swapper/323:23:353
3200125413,25sleep232052-21irqstats21:38:312
2478725418,17sleep30-21swapper/322:58:403
1684325421,20sleep10-21swapper/100:18:371
1657425413,14sleep20-21swapper/222:33:272
1642825416,16sleep30-21swapper/320:48:413
1066125317,22sleep310232-21/usr/sbin/munin22:13:333
1066125317,22sleep310232-21/usr/sbin/munin22:13:333
848625219,16sleep30-21swapper/323:48:393
783725213,24sleep27833-21apt-get20:23:242
303632527,11sleep20-21swapper/219:49:562
303632527,11sleep20-21swapper/219:49:562
2732325211,18sleep10-21swapper/121:23:321
2732325211,18sleep10-21swapper/121:23:321
1226825211,18sleep20-21swapper/222:18:372
2327625113,15sleep30-21swapper/322:53:393
2327625113,15sleep30-21swapper/322:53:393
1849425114,22sleep218505-21ntp_states22:38:362
153482517,32sleep20-21swapper/220:45:162
1058125111,14sleep10-21swapper/122:13:311
1058125111,14sleep10-21swapper/122:13:311
3275925010,20sleep10-21swapper/123:23:291
2037925018,17sleep20-21swapper/221:03:092
1586725012,22sleep215864-21apt-get20:48:232
1500825010,16sleep20-21swapper/219:03:282
142725017,14sleep10-21swapper/121:43:391
142725017,14sleep10-21swapper/121:43:391
33032499,15sleep20-21swapper/223:33:282
33032499,15sleep20-21swapper/223:33:282
2391824911,22sleep124028-21users19:28:431
2391824911,22sleep124028-21users19:28:431
2086024913,20sleep220890-21munin-run22:48:092
248124811,21sleep22508-21df21:48:282
248124811,21sleep22508-21df21:48:282
678724711,21sleep16793-21ntp_states23:43:361
3246524612,15sleep20-21swapper/219:58:202
260042468,16sleep20-21swapper/221:18:362
260042468,16sleep20-21swapper/221:18:362
1847324611,20sleep218469-21apt-get19:13:252
32692457,16sleep10-21swapper/120:08:331
32692457,16sleep10-21swapper/120:08:331
177082459,22sleep10-21swapper/120:53:311
159324512,16sleep30-21swapper/320:03:263
1116824511,21sleep211169-21sshd00:00:292
1138124414,13sleep30-21swapper/300:03:093
291062436,15sleep10-21swapper/119:48:091
31402389,14sleep30-21swapper/320:08:223
31402389,14sleep30-21swapper/320:08:223
112262348,14sleep10-21swapper/100:01:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional