You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-03 - 05:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Jul 03, 2025 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29161284721,809sleep229164-21/usr/sbin/munin20:19:492
29161284721,809sleep229164-21/usr/sbin/munin20:19:492
12610235743,112sleep00-21swapper/019:24:330
577722915,274sleep00-21swapper/019:03:530
13727226516,234sleep013802-21fschecks_time22:54:530
1971226483,63sleep00-21swapper/023:59:530
10212225815,229sleep110294-21mailstats19:15:011
5060224713,217sleep25083-21netstat20:45:142
13694224295,24sleep00-21swapper/021:10:040
25359223525,163sleep125378-21apt-get21:49:431
11851223159,74sleep011826-21latency_hist22:49:320
3131822289,71sleep031350-21munin-run22:09:290
2233122277,201sleep00-21swapper/021:39:490
27060222349,84sleep027201-21unixbench_singl23:35:040
16765222228,153sleep00-21swapper/023:04:430
16765222228,153sleep00-21swapper/023:04:430
2920221732,126sleep03046-21sort22:20:080
27514221750,72sleep00-21swapper/020:14:460
953022130,64sleep09324-21gpgv20:59:320
11473220927,168sleep311511-21cat19:19:523
3196222089,160sleep032095-21sed20:29:320
19103219412,31sleep00-21swapper/019:45:050
20132219320,160sleep320187-21cat23:14:523
716721897,12sleep00-21swapper/000:15:030
15268218817,126sleep015282-21fschecks_count22:59:500
15268218817,126sleep015282-21fschecks_count22:59:500
7823218137,132sleep07867-21awk22:35:060
1613221677,106sleep00-21swapper/021:19:530
11033216738,19sleep00-21swapper/019:19:320
19317216218,131sleep019131-21/usr/sbin/munin21:29:510
11531216034,105sleep00-21swapper/021:04:510
22015215527,113sleep122079-21latency_hist21:39:331
5985215144,85sleep00-21swapper/020:49:320
14105214740,86sleep00-21swapper/019:29:300
3148521466,12sleep20-21swapper/220:25:062
7649214543,48sleep07852-21apt-config20:54:330
23007214417,111sleep223158-21latency_hist23:24:322
4788214019,104sleep24751-21latency_hist00:09:322
20545214050,72sleep020688-21unixbench_multi23:15:010
423213913,106sleep0519-21fschecks_time23:54:520
30375213918,64sleep230526-21unixbench_multi23:45:042
24300213950,76sleep024347-21processes21:45:020
2145321399,118sleep021521-21basename19:54:510
25494213818,71sleep00-21swapper/020:06:540
1060213712,24sleep00-21swapper/020:34:330
20264213337,40sleep00-21swapper/019:50:020
3928212910,69sleep04069-21cut18:59:330
3928212910,69sleep04069-21cut18:59:330
562821286,15sleep00-21swapper/022:29:490
14977212874,40sleep015040-21mailstats21:15:010
1028621259,103sleep0286-21dbus-daemon22:44:310
23119212417,94sleep123134-21kernelversion19:59:531
26583212222,81sleep326664-21awk20:10:013
14368212224,18sleep00-21swapper/019:29:420
14368212224,18sleep00-21swapper/019:29:420
8143211920,80sleep27659-21apt-get19:09:412
2608321198,97sleep026180-21latency_hist23:34:320
29565211834,72sleep029649-21grep20:19:590
29565211834,72sleep029649-21grep20:19:590
16121211818,76sleep30-21swapper/321:19:523
3191021178,95sleep15571-1kworker/1:1H20:29:291
10587211711,29sleep010736-21grep21:00:260
3257121165,29sleep00-21swapper/022:10:430
3096521169,92sleep230976-21/usr/sbin/munin22:05:072
28075211510,21sleep00-21swapper/023:39:410
555521140,6sleep10-21swapper/122:29:331
555521140,6sleep10-21swapper/122:29:331
2561211443,54sleep02570-21modprobe20:39:300
27300211242,29sleep026906-21/usr/sbin/munin21:54:550
17885211225,70sleep117889-21cron19:44:301
26352211115,80sleep126383-21nfs_client20:09:541
2626721098,86sleep025960-21/usr/sbin/munin20:09:530
19275210823,49sleep00-21swapper/023:10:410
886210738,57sleep0943-21if_err_eth022:14:500
6134210720,61sleep20-21swapper/222:30:032
2847421076,48sleep00-21swapper/021:59:440
2847421076,48sleep00-21swapper/021:59:440
948321048,44sleep09635-21sed00:24:320
8963210414,40sleep20-21swapper/222:39:492
2306121037,36sleep00-21swapper/019:59:520
18918210210,71sleep30-21swapper/321:29:343
2360121017,13sleep00-21swapper/023:24:590
14791210018,68sleep214834-21ntpq19:29:572
14791210018,68sleep214834-21ntpq19:29:572
229102988,14sleep00-21swapper/023:24:300
1669829810,32sleep00-21swapper/019:39:440
2162229721,56sleep321135-21apt-get23:19:403
1010629720,61sleep210122-21kernelversion21:00:162
86729527,46sleep1885-21fschecks_count18:49:491
86729527,46sleep1885-21fschecks_count18:49:491
67832957,41sleep06824-21ntpq19:05:110
45892955,12sleep30-21swapper/322:25:243
45892955,12sleep30-21swapper/322:25:243
452329510,67sleep34142-21apt-get20:44:343
420129415,63sleep14313-21date20:44:321
420129415,63sleep14313-21date20:44:321
1617929427,56sleep0370-21snmpd19:37:260
85662938,74sleep08146-21/usr/sbin/munin19:10:110
50232937,33sleep00-21swapper/000:09:420
413129236,40sleep00-21swapper/020:44:310
413129236,40sleep00-21swapper/020:44:310
410929221,54sleep34127-21cron20:44:303
410929221,54sleep34127-21cron20:44:303
296912927,36sleep00-21swapper/023:44:460
3229729114,59sleep10-21swapper/122:10:031
2176629118,39sleep00-21swapper/021:36:580
431629017,56sleep24522-21latency_hist20:44:332
431629017,56sleep24522-21latency_hist20:44:332
37272908,39sleep0370-21snmpd00:04:530
3255029010,22sleep00-21swapper/023:54:310
872028914,57sleep18692-21latency_hist22:39:321
273482887,53sleep10-21swapper/121:54:561
1758328818,27sleep00-21swapper/021:24:420
864828710,61sleep38722-21find20:55:163
2538428720,18sleep00-21swapper/021:49:460
1820628718,42sleep30-21swapper/319:44:483
1054128715,59sleep210602-21cpuspeed22:44:502
2946828527,34sleep329432-40mandb23:44:333
708328414,53sleep37163-21mailstats00:15:023
603428421,47sleep36247-21sed19:04:333
210928421,44sleep02152-21munin-run18:54:320
210928421,44sleep02152-21munin-run18:54:320
2096228410,22sleep20-21swapper/221:34:542
163328416,18sleep30-21swapper/323:59:313
1332028420,18sleep10-21swapper/121:09:571
347728313,31sleep30-21swapper/300:04:443
1574528317,46sleep115754-21modprobe21:19:301
1179828332,23sleep10-21swapper/122:49:291
638228212,56sleep30-21swapper/319:04:453
53042827,29sleep00-21swapper/022:29:080
53042827,29sleep00-21swapper/022:29:080
2524528117,45sleep224753-21/usr/sbin/munin23:30:012
783128023,38sleep27851-21modprobe00:19:292
778127821,34sleep27870-21latency_hist19:09:332
755827820,36sleep37646-21grep20:54:323
2427927815,46sleep124289-21ntp_states21:44:591
227322780,8sleep217892-21kworker/u8:319:59:322
193327725,26sleep30-21swapper/320:35:013
1523327721,27sleep30-21swapper/321:15:043
959327622,32sleep10-21swapper/122:40:041
945327625,23sleep30-21swapper/300:24:313
938227626,30sleep2286-21dbus-daemon00:24:302
752227611,53sleep27555-21munin-run20:54:312
351627623,35sleep23547-21munin-run22:24:302
3260427621,24sleep10-21swapper/123:54:321
127012768,54sleep012809-21timerandwakeup22:50:010
748227524,24sleep10-21swapper/120:54:301
592227521,34sleep26033-21apt-key19:04:322
3100327518,41sleep131073-21sed23:49:311
2593727515,33sleep30-21swapper/321:50:013
224662750,8sleep30-21swapper/321:39:513
203302758,16sleep10-21swapper/123:14:581
1564627518,29sleep10-21swapper/122:59:591
1564627518,29sleep10-21swapper/122:59:591
1232327513,43sleep312357-21/usr/sbin/munin22:49:523
188827415,22sleep10-21swapper/123:59:491
2136227327,19sleep30-21swapper/321:35:033
2110627322,26sleep20-21swapper/219:54:322
1796527320,29sleep30-21swapper/319:44:323
117962738,13sleep10-21swapper/121:04:561
317922726,12sleep20-21swapper/222:09:522
284227216,35sleep32900-21/usr/sbin/munin18:55:323
284227216,35sleep32900-21/usr/sbin/munin18:55:323
2412127221,33sleep224127-21modprobe20:04:302
1217427224,22sleep10-21swapper/122:49:491
99027121,24sleep30-21swapper/318:49:513
99027121,24sleep30-21swapper/318:49:513
3191627115,26sleep30-21swapper/323:50:023
2887127118,36sleep228998-21date20:19:332
2813827116,31sleep30-21swapper/320:15:053
1414127111,15sleep20-21swapper/222:55:042
851327019,35sleep18551-21processes20:55:141
2953827023,29sleep329563-21/usr/sbin/munin20:19:583
2953827023,29sleep329563-21/usr/sbin/munin20:19:583
2757627028,25sleep10-21swapper/123:39:301
2883326925,22sleep10-21swapper/120:19:321
2452526920,23sleep20-21swapper/220:04:502
207226917,21sleep20-21swapper/223:59:552
1742326921,19sleep30-21swapper/323:05:023
1742326921,19sleep30-21swapper/323:05:023
1557826913,27sleep20-21swapper/222:59:562
1557826913,27sleep20-21swapper/222:59:562
76972689,47sleep10-21swapper/100:17:471
303132689,14sleep30-21swapper/322:04:563
2944226825,28sleep229467-21sed23:44:322
3213926712,39sleep332213-21/usr/sbin/munin22:10:013
2765226725,26sleep30-21swapper/321:55:023
201126718,23sleep10-21swapper/122:19:321
1962326712,40sleep319717-21latency_hist23:14:313
831626618,23sleep30-21swapper/319:10:063
628526611,34sleep30-21swapper/300:14:303
462326612,40sleep14630-21df_inode18:59:461
2453626620,21sleep10-21swapper/123:29:321
2310526612,38sleep223152-21users21:40:032
1708626620,22sleep20-21swapper/219:39:542
1642326620,27sleep116466-21awk21:19:581
1036826611,39sleep310488-21timerwakeupswit19:15:033
934926525,23sleep30-21swapper/322:39:583
589526514,25sleep30-21swapper/322:29:563
2754426513,27sleep20-21swapper/221:55:012
2543426512,39sleep225458-21df21:49:482
2470826524,21sleep10-21swapper/120:04:541
1797426517,34sleep117986-21netstat21:24:551
1206826521,20sleep30-21swapper/321:05:023
93712646,17sleep10-21swapper/100:24:211
357126426,22sleep30-21swapper/322:24:313
3217526417,31sleep332193-21/usr/sbin/munin20:29:443
2194526422,25sleep10-21swapper/123:20:111
203562649,42sleep10-21swapper/121:33:441
2027026413,20sleep20-21swapper/223:14:552
1915626417,21sleep30-21swapper/321:29:463
1831626419,26sleep20-21swapper/223:09:462
168326418,23sleep20-21swapper/222:15:582
1252826421,27sleep112530-21cron19:24:301
876126318,22sleep20-21swapper/222:39:332
84426321,21sleep20-21swapper/223:55:022
302426317,27sleep23036-21/usr/sbin/munin18:55:352
302426317,27sleep23036-21/usr/sbin/munin18:55:352
2545826322,21sleep10-21swapper/123:30:051
2451326325,22sleep30-21swapper/321:45:053
1973226323,17sleep20-21swapper/221:30:012
1415826314,34sleep314293-21latency_hist19:29:333
1414626310,38sleep214156-21systemd19:29:322
130262637,12sleep30-21swapper/321:09:503
633726216,26sleep10-21swapper/100:14:321
2758926219,19sleep20-21swapper/223:39:312
1033026213,19sleep10-21swapper/121:00:231
692426115,31sleep17008-21apt-config22:34:311
3257226116,32sleep232608-21/usr/sbin/munin20:29:592
2331526113,26sleep30-21swapper/319:59:593
162962617,18sleep30-21swapper/323:02:323
162962617,18sleep30-21swapper/323:02:323
1570326122,19sleep10-21swapper/119:34:321
1570326122,19sleep10-21swapper/119:34:321
1284926117,27sleep112843-21apt-get19:24:471
483526024,16sleep20-21swapper/222:25:272
483526024,16sleep20-21swapper/222:25:272
3229026022,24sleep132352-21grep20:29:541
206822605,10sleep10-21swapper/119:50:111
178026015,30sleep11843-21mailstats20:34:591
1779226020,28sleep20-21swapper/221:24:522
1633726012,33sleep216369-21munin-run19:39:302
124572607,11sleep20-21swapper/222:49:552
1136626017,19sleep20-21swapper/219:19:512
738125918,18sleep30-21swapper/322:34:413
3195125917,25sleep331960-21munin-run20:29:313
2638425913,33sleep226426-21ntp_offset20:09:552
2320525915,20sleep30-21swapper/323:24:453
1734325923,19sleep20-21swapper/223:05:012
1734325923,19sleep20-21swapper/223:05:012
1060325916,28sleep310615-21df_inode22:44:513
70425825,15sleep30-21swapper/322:14:453
563825818,19sleep20-21swapper/200:10:002
2918725816,25sleep329327-21unixbench_singl22:00:033
2918725816,25sleep329327-21unixbench_singl22:00:033
2893025811,31sleep228938-21ntp_states21:59:582
2893025811,31sleep228938-21ntp_states21:59:582
34412576,10sleep10-21swapper/122:23:321
263425715,24sleep10-21swapper/118:55:061
263425715,24sleep10-21swapper/118:55:061
2130525715,25sleep121302-21apt-get19:54:441
1421725721,16sleep30-21swapper/322:55:053
13022576,11sleep10-21swapper/123:55:461
2257825613,24sleep122617-21memory21:39:541
1139425614,25sleep211388-21apt-get21:04:452
948425512,25sleep10-21swapper/119:14:331
1831525513,25sleep118309-21apt-get23:09:451
393325415,24sleep13939-21ntp_states00:04:581
3082225416,18sleep10-21swapper/122:05:061
2763725415,23sleep327719-21munin-run23:39:323
2128125413,25sleep321298-21/usr/sbin/munin19:54:423
185325416,19sleep20-21swapper/220:35:002
1736425418,22sleep217395-21cat21:24:322
1575625411,25sleep215793-21systemd-run21:19:312
2854325315,24sleep128572-21ntp_states23:40:061
2323025313,24sleep223224-21apt-get23:24:482
972225213,22sleep29718-21apt-get19:14:462
932125213,23sleep29409-21gpgv20:59:312
769325217,20sleep17778-21systemd-run19:09:321
324325217,14sleep30-21swapper/300:04:323
2950625214,15sleep10-21swapper/120:19:561
2950625214,15sleep10-21swapper/120:19:561
2889325211,17sleep10-21swapper/121:59:571
2889325211,17sleep10-21swapper/121:59:571
2528425213,23sleep325377-21grep23:30:023
1820525214,22sleep218199-21apt-get19:44:472
75725112,22sleep2776-21apt-get18:49:422
75725112,22sleep2776-21apt-get18:49:422
2527625115,17sleep30-21swapper/320:05:043
1828225114,22sleep118288-21diskstats19:44:501
1400925113,17sleep10-21swapper/122:54:581
514725013,22sleep15153-21ntp_states20:45:171
2392725011,15sleep20-21swapper/221:44:512
1988325012,21sleep319968-21perl19:49:533
3095124917,19sleep30-21swapper/323:49:293
1349724916,20sleep213561-21wc21:10:022
1637724712,20sleep316386-21munin-run19:39:313
1029924712,21sleep310305-21ntp_states21:00:213
3132224611,21sleep231326-21diskstats23:49:492
1439224410,17sleep30-21swapper/319:29:453
1439224410,17sleep30-21swapper/319:29:453
167672407,17sleep10-21swapper/123:04:441
167672407,17sleep10-21swapper/123:04:441
159142368,18sleep10-21swapper/119:34:401
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional