You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 02:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon May 12, 2025 12:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
397129678,946sleep132763-21kworker/1:211:06:101
1990427416,723sleep219940-21ntpq09:37:592
1990427416,723sleep219940-21ntpq09:37:592
5631269211,664sleep05665-21/usr/sbin/munin08:52:580
2889026536,635sleep228925-21ntpq08:22:592
46602433110,306sleep24663-21cron07:07:342
9711240620,331sleep29842-21unixbench_multi11:23:072
9711240620,331sleep29842-21unixbench_multi11:23:072
12482239029,328sleep2370-21snmpd07:30:002
1682123676,322sleep216865-21ntpq09:28:022
11513233718,301sleep311553-21sort07:27:553
2743923238,192sleep227145-21kworker/2:212:22:362
259632308120,172sleep226035-21switchtime08:13:062
4159229231,135sleep24246-21latency_hist11:07:362
17395227587,55sleep20-21swapper/207:47:372
14852224149,154sleep214917-21cpuspeed09:22:532
14852224149,154sleep214917-21cpuspeed09:22:532
20445223787,44sleep21-21systemd07:57:352
20445223787,44sleep21-21systemd07:57:352
21439223074,135sleep221483-21ntpq09:43:002
9154222512,195sleep39240-21fw_packets11:22:563
9154222512,195sleep39240-21fw_packets11:22:563
899121937,150sleep29121-21grep07:18:082
18862217629,134sleep218905-21find11:53:082
11872176114,47sleep21320-21latency_hist06:57:372
23952217529,65sleep20-21swapper/208:07:572
30000217147,24sleep20-21swapper/210:32:352
13963216710,88sleep20-21swapper/211:37:572
18241216620,11sleep20-21swapper/207:48:052
809721619,113sleep28193-21timerandwakeup11:18:062
17273215812,125sleep21200-21cyclictest11:48:032
24630215511,55sleep224720-21fschecks_time10:02:512
2169215432,60sleep22309-21grep06:58:092
2169215432,60sleep22309-21grep06:58:092
1975215041,93sleep21978-21cron08:42:342
23625214828,105sleep223743-21latency_hist08:07:362
21713214719,11sleep20-21swapper/207:59:002
21713214719,11sleep20-21swapper/207:59:002
27680214630,99sleep227745-21cpuspeed12:22:522
10205214613,16sleep20-21swapper/209:07:552
24855214414,118sleep024886-21latency12:12:570
31598213863,24sleep20-21swapper/210:42:482
5395213524,60sleep20-21swapper/208:52:522
5003213514,101sleep30-21swapper/311:08:063
7074213326,34sleep20-21swapper/208:57:532
1460921339,15sleep20-21swapper/209:22:372
20647213114,65sleep220781-21unixbench_singl11:58:042
24294213011,103sleep124310-21idleruntime-cro12:12:331
1287421269,78sleep213018-21unixbench_singl11:33:092
1287421269,78sleep213018-21unixbench_singl11:33:092
12736212619,91sleep112780-21latency_hist07:32:371
9537212522,88sleep09628-21cut11:23:050
9537212522,88sleep09628-21cut11:23:050
7128212510,42sleep20-21swapper/207:12:592
7128212510,42sleep20-21swapper/207:12:592
3065021228,56sleep230724-21date10:37:362
32764212012,17sleep00-21swapper/006:53:030
1452121179,65sleep20-21swapper/207:37:512
23013211432,16sleep20-21swapper/212:07:462
1890821130,7sleep30-21swapper/311:53:083
22454211247,33sleep20-21swapper/212:04:142
13432211110,84sleep013455-21/usr/sbin/munin07:33:010
496221095,69sleep20-21swapper/208:50:462
11885210910,53sleep20-21swapper/209:12:552
25265210819,69sleep125396-21grep12:13:061
30279210715,16sleep30-21swapper/308:27:573
30279210715,16sleep30-21swapper/308:27:573
18049210725,67sleep218128-21fschecks_time09:32:552
4845210629,51sleep15022-21cat07:07:371
962921049,73sleep10-21swapper/111:23:061
962921049,73sleep10-21swapper/111:23:061
8794210420,67sleep08893-21latency_hist11:22:360
7094210321,67sleep07192-21cut08:57:550
23035210314,74sleep123138-21latency_hist09:52:371
2576210212,75sleep32718-21cut11:02:363
183721026,68sleep20-21swapper/210:56:542
25882210126,63sleep225892-21munin-run12:17:352
11264210144,40sleep20-21swapper/211:28:082
11264210144,40sleep20-21swapper/211:28:082
4767210019,23sleep10-21swapper/111:07:591
27458210026,37sleep227530-21mailstats08:18:072
263632995,82sleep20-21swapper/210:11:072
265822987,51sleep226481-21cron10:12:362
160762989,22sleep20-21swapper/211:43:102
518429621,33sleep20-21swapper/207:07:422
229822969,49sleep222985-21cron09:52:332
1770629615,65sleep317720-21/usr/sbin/munin11:48:083
89132946,71sleep18988-21find07:18:071
3161429420,35sleep20-21swapper/208:32:492
575329110,38sleep20-21swapper/211:12:372
2581229119,47sleep00-21swapper/010:07:340
615629014,55sleep16209-21/usr/sbin/munin11:13:001
594529037,37sleep20-21swapper/211:12:532
2914728924,46sleep00-21swapper/008:23:050
1306428914,50sleep30-21swapper/309:17:373
280102868,59sleep20-21swapper/210:22:422
1948428613,61sleep019498-21acpi09:37:450
1948428613,61sleep019498-21acpi09:37:450
1894928617,53sleep119033-21latency_hist07:52:371
1771528612,61sleep317732-21latency_hist09:32:353
1369428616,40sleep313687-21apt-get11:37:503
309122855,69sleep20-21swapper/210:38:202
22662855,67sleep20-21swapper/208:42:502
144328515,55sleep01464-21cpu06:57:510
144328515,55sleep01464-21cpu06:57:510
90428318,29sleep20-21swapper/210:51:172
3088028323,15sleep20-21swapper/208:28:082
3088028323,15sleep20-21swapper/208:28:082
2830528320,42sleep128011-21/usr/sbin/munin10:22:531
900428215,17sleep00-21swapper/009:03:040
32128224,42sleep3339-21apt10:47:343
2922328226,36sleep328965-21apt-get12:27:383
447328117,46sleep14591-21sort08:48:071
2762928113,55sleep027754-21cat08:18:090
830228027,29sleep10-21swapper/109:02:361
2630528019,46sleep126317-21vmstat08:13:091
2407728018,50sleep20-21swapper/210:00:132
982727911,48sleep30-21swapper/307:22:463
2249227825,28sleep10-21swapper/108:02:591
1109627815,28sleep20-21swapper/207:27:252
1416527716,44sleep114228-21/usr/sbin/munin11:38:041
127262778,54sleep00-21swapper/007:32:360
1987227622,28sleep10-21swapper/107:53:091
1987227622,28sleep10-21swapper/107:53:091
56242759,51sleep35662-21ntpq07:08:083
1470327514,46sleep314739-21/usr/sbin/munin07:37:573
226132746,14sleep00-21swapper/012:06:100
1388127411,49sleep20-21swapper/207:33:072
376227324,25sleep30-21swapper/307:02:593
2511527319,33sleep325145-21munin-run08:12:343
801927222,31sleep18036-21awk07:17:351
801927222,31sleep18036-21awk07:17:351
408027219,36sleep04105-21munin-run11:07:340
219892727,37sleep30-21swapper/308:02:323
219892727,37sleep30-21swapper/308:02:323
84042715,29sleep20-21swapper/209:02:372
327132716,14sleep30-21swapper/306:53:023
1889827123,23sleep30-21swapper/307:52:353
1386927122,33sleep113880-21/usr/sbin/munin07:33:061
1276227121,29sleep10-21swapper/111:33:071
1276227121,29sleep10-21swapper/111:33:071
1231027119,28sleep30-21swapper/311:32:573
1231027119,28sleep30-21swapper/311:32:573
232972705,29sleep20-21swapper/209:53:222
21572706,28sleep20-21swapper/210:57:472
1577427010,12sleep30-21swapper/311:43:073
1500027020,30sleep00-21swapper/011:42:350
589026919,28sleep10-21swapper/107:08:141
288882699,51sleep00-21swapper/012:27:020
2516726917,36sleep125245-21latency_hist08:12:361
2052226923,27sleep10-21swapper/111:58:031
273762686,28sleep20-21swapper/210:17:412
2295126819,24sleep30-21swapper/308:03:093
1330426815,22sleep00-21swapper/009:17:500
985526731,20sleep00-21swapper/007:22:510
40626721,26sleep10-21swapper/110:47:361
2263226713,40sleep322633-21sshd09:48:103
1662826716,24sleep00-21swapper/011:47:360
15912678,14sleep00-21swapper/010:53:410
1536326715,37sleep115449-21/usr/sbin/munin11:42:571
652626617,26sleep00-21swapper/007:12:370
404626622,26sleep30-21swapper/308:47:563
2081626613,28sleep30-21swapper/309:42:373
2081626613,28sleep30-21swapper/309:42:373
1525426624,20sleep00-21swapper/007:38:140
1184126617,26sleep10-21swapper/109:12:541
2652726518,19sleep10-21swapper/110:12:351
760326424,22sleep30-21swapper/308:58:063
316152648,20sleep00-21swapper/010:42:510
3059926418,31sleep030616-21apt10:37:340
2937426412,36sleep329456-21date10:27:363
2875326418,30sleep30-21swapper/308:22:563
2776326421,22sleep10-21swapper/112:22:551
1625126421,23sleep00-21swapper/007:42:580
871226314,23sleep00-21swapper/007:18:020
714226313,33sleep37181-21ntp_kernel_pll_07:13:003
714226313,33sleep37181-21ntp_kernel_pll_07:13:003
351926318,30sleep13554-21awk08:47:341
2354626318,28sleep023610-21sendmail-msp08:07:340
1606826317,26sleep30-21swapper/307:42:533
2671326220,17sleep00-21swapper/008:17:360
202926218,23sleep10-21swapper/108:42:371
1774126214,23sleep00-21swapper/009:32:360
1202526214,22sleep00-21swapper/009:12:580
1118626216,28sleep111238-21latency_hist07:27:361
675026117,23sleep10-21swapper/108:57:361
587126122,19sleep30-21swapper/308:53:053
388426119,19sleep00-21swapper/007:03:040
2585826116,27sleep125891-21munin-run10:07:351
2647826011,33sleep026526-21systemd-run10:12:340
2199226015,24sleep10-21swapper/108:02:341
2199226015,24sleep10-21swapper/108:02:341
1946126015,20sleep00-21swapper/007:53:000
1946126015,20sleep00-21swapper/007:53:000
1112426012,32sleep39834-21apt-get09:08:163
1058526019,18sleep10-21swapper/109:08:061
353525917,18sleep30-21swapper/311:03:053
2850525917,27sleep00-21swapper/010:23:000
2613225916,18sleep00-21swapper/012:17:490
2430025920,19sleep30-21swapper/310:02:363
1897225917,20sleep00-21swapper/011:53:100
1427025920,23sleep314279-21unin-run07:37:363
77882588,38sleep00-21swapper/007:14:350
77882588,38sleep00-21swapper/007:14:350
30525823,19sleep10-21swapper/106:53:041
3001725815,17sleep30-21swapper/310:32:363
269325815,29sleep32711-21awk07:02:343
269325815,29sleep32711-21awk07:02:343
2674325823,19sleep30-21swapper/312:18:053
2367425817,19sleep00-21swapper/009:57:360
2115325818,18sleep10-21swapper/109:42:551
194025818,22sleep30-21swapper/310:57:353
164312588,37sleep00-21swapper/011:46:480
1378525811,22sleep10-21swapper/109:18:031
1060425815,25sleep010598-21apt-get11:27:490
1060425815,25sleep010598-21apt-get11:27:490
748825714,25sleep17483-21apt-get11:17:491
3161525715,26sleep331610-21apt-get08:32:503
2509025717,25sleep325182-21mailstats12:13:043
2501825713,20sleep30-21swapper/310:03:003
2303525715,25sleep323032-21apt-get12:07:473
226725712,31sleep32337-21df08:42:513
1823325716,24sleep10-21swapper/109:32:581
1610325719,22sleep30-21swapper/309:27:343
1610325719,22sleep30-21swapper/309:27:343
107825710,29sleep31109-21/usr/sbin/munin08:37:593
877625613,17sleep30-21swapper/307:18:053
603425612,20sleep30-21swapper/311:12:583
2695625617,17sleep10-21swapper/108:17:531
2431325612,22sleep00-21swapper/010:02:370
226425617,23sleep02259-21apt-get08:42:480
1803625613,18sleep00-21swapper/009:32:530
1081025612,24sleep30-21swapper/311:27:573
1081025612,24sleep30-21swapper/311:27:573
881525514,23sleep38909-21memory09:02:593
2783125516,17sleep10-21swapper/110:22:371
2620325513,27sleep126206-21df_inode12:17:501
22632558,34sleep00-21swapper/010:59:100
2151125514,24sleep021517-21ntp_states09:43:020
1893425515,23sleep018948-21munin-run07:52:360
1702925516,23sleep117127-21sed09:28:091
1662225513,29sleep116707-21switchtime07:43:081
1190025516,20sleep10-21swapper/111:32:351
1190025516,20sleep10-21swapper/111:32:351
113925513,16sleep00-21swapper/010:52:360
749025416,19sleep30-21swapper/311:17:513
3160825412,17sleep00-21swapper/008:32:470
3007125412,25sleep130065-21apt-get08:27:511
3007125412,25sleep130065-21apt-get08:27:511
2779925415,17sleep00-21swapper/010:22:360
122025411,26sleep01302-21systemctl08:38:040
854025313,23sleep18537-21apt-get09:02:491
2055825313,17sleep00-21swapper/007:57:370
2055825313,17sleep00-21swapper/007:57:370
2049625311,17sleep30-21swapper/307:57:363
2049625311,17sleep30-21swapper/307:57:363
1357725312,22sleep313667-21memory09:17:573
449225220,17sleep00-21swapper/011:07:540
2695325212,24sleep326948-21apt-get08:17:513
2666125216,22sleep31-21systemd08:17:353
23642525,11sleep10-21swapper/106:58:161
23642525,11sleep10-21swapper/106:58:161
2192025218,18sleep10-21swapper/112:03:041
2045425218,18sleep00-21swapper/011:58:020
1493525217,17sleep10-21swapper/109:22:561
1493525217,17sleep10-21swapper/109:22:561
1083025211,25sleep010097-21/usr/sbin/munin09:08:130
383925112,21sleep03833-21apt-get08:47:490
303652517,32sleep10-21swapper/110:34:321
2386525111,23sleep023861-21apt-get08:07:510
1836925110,15sleep10-21swapper/111:52:541
1344525113,22sleep013455-21munin-run11:37:350
1344525113,22sleep013455-21munin-run11:37:350
641025012,24sleep06450-21ps11:13:070
301482509,16sleep00-21swapper/008:27:540
301482509,16sleep00-21swapper/008:27:540
240682505,33sleep10-21swapper/110:00:081
2231025016,22sleep3227-21systemd-journal09:47:343
1993925012,23sleep319979-21df11:57:493
1812925017,19sleep30-21swapper/309:32:563
1726625013,22sleep117272-21ntp_states11:48:021
274022495,32sleep30-21swapper/310:18:033
2715524913,20sleep027167-21munin-run10:17:350
1492824913,21sleep014934-21diskstats09:22:550
1492824913,21sleep014934-21diskstats09:22:550
132924913,16sleep10-21swapper/110:52:371
3132824816,15sleep10-21swapper/110:42:391
3123824811,22sleep031270-21munin-run10:42:340
240572488,31sleep00-21swapper/012:09:290
1479024813,20sleep10-21swapper/107:37:591
2859824710,20sleep128699-21perl08:22:541
326442465,30sleep10-21swapper/108:34:381
2079224613,19sleep10-21swapper/109:42:351
2079224613,19sleep10-21swapper/109:42:351
2361424516,19sleep10-21swapper/109:57:291
2996024311,15sleep10-21swapper/110:32:341
252624210,14sleep10-21swapper/111:02:341
2203024210,13sleep30-21swapper/312:03:063
294592408,13sleep00-21swapper/010:27:370
236252389,15sleep30-21swapper/309:57:353
10462367,14sleep30-21swapper/310:52:353
2702323510,13sleep10-21swapper/110:16:231
2677323510,13sleep30-21swapper/310:13:103
230242358,13sleep00-21swapper/009:52:360
99823410,12sleep10-21swapper/110:52:291
307252334,11sleep30-21swapper/310:37:373
260992318,13sleep30-21swapper/310:07:413
303302308,12sleep30-21swapper/310:34:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional