You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 20:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Dec 07, 2025 12:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25893225349,2509sleep014365-21kworker/u8:111:58:060
25893225349,2509sleep014365-21kworker/u8:111:58:060
21990267116,638sleep222050-21/usr/sbin/munin10:03:052
20727254115,511sleep320760-21/usr/sbin/munin09:58:133
4890234910,324sleep14760-21latency_hist10:52:481
31611230914,204sleep131691-21fschecks_time07:08:081
22174229411,89sleep10-21swapper/110:03:081
3125622900,14sleep129357-21cyclictest08:48:151
24196227633,195sleep124345-21grep10:08:171
22459227063,159sleep122433-21grep08:22:471
22459227063,159sleep122433-21grep08:22:471
20231226085,95sleep10-21swapper/108:13:391
28909225451,187sleep128929-21latency_hist08:42:471
15972225140,159sleep116022-21latency_hist08:02:481
101422286,212sleep15666-21snmpd08:56:281
9147222010,192sleep19165-21modprobe09:22:451
16584221676,97sleep10-21swapper/111:28:181
28905219012,160sleep00-21swapper/012:07:590
1161721897,139sleep111673-21fschecks_count07:48:061
20108218837,23sleep10-21swapper/109:57:471
11478218642,52sleep10-21swapper/109:28:191
11478218642,52sleep10-21swapper/109:28:191
6945217310,151sleep16959-21grep10:58:091
32745217318,134sleep2444-21unixbench_multi12:18:192
20637217329,98sleep120744-21latency_hist11:42:481
20637217329,98sleep120744-21latency_hist11:42:481
15427216713,112sleep015546-21latency_hist09:42:470
10549216516,59sleep10-21swapper/111:08:191
5009216456,59sleep14487-21/usr/sbin/munin09:08:151
27537216114,131sleep027567-21idleruntime12:03:060
27537216114,131sleep027567-21idleruntime12:03:060
3753216036,65sleep10-21swapper/107:23:021
3753216036,65sleep10-21swapper/107:23:021
707221559,94sleep10-21swapper/109:13:361
3087121539,102sleep130953-21latency_hist07:07:471
3087121539,102sleep130953-21latency_hist07:07:471
23643215211,126sleep223429-21/usr/sbin/munin10:08:062
30478215062,50sleep130450-21latency_hist08:47:481
30478215062,50sleep130450-21latency_hist08:47:481
2697921489,102sleep126994-21awk08:33:341
2697921489,102sleep126994-21awk08:33:341
1249321465,130sleep00-21swapper/007:48:540
32660214544,55sleep10-21swapper/112:18:181
842821439,37sleep10-21swapper/107:38:001
4041213927,100sleep14143-21latency_hist12:32:481
32055213914,110sleep031997-21latency_hist08:52:470
1458821388,87sleep10-21swapper/109:38:161
850421379,114sleep38601-21timerandwakeup09:18:193
3096821377,95sleep131002-21ps12:13:171
29617213710,114sleep329628-21ntpq08:43:133
29617213710,114sleep329628-21ntpq08:43:133
31069213617,105sleep030973-21latency_hist10:32:480
25275213641,79sleep125292-21munin-run11:57:461
16618213635,88sleep116667-21/usr/sbin/munin08:03:131
12649213631,21sleep10-21swapper/109:33:031
12649213631,21sleep10-21swapper/109:33:031
2889021356,11sleep20-21swapper/206:58:202
3124921347,92sleep131269-21apt-get10:32:571
30804213317,104sleep230829-21grep08:48:052
11517213330,76sleep311536-21apt-get11:12:573
9979213224,58sleep10-21swapper/107:43:011
9979213224,58sleep10-21swapper/107:43:011
887721328,89sleep18911-21ps11:03:221
103521318,33sleep10-21swapper/107:13:121
103521318,33sleep10-21swapper/107:13:121
866212929,86sleep1951-21cut10:38:181
25648212716,96sleep225724-21timerandwakeup10:13:192
25648212716,96sleep225724-21timerandwakeup10:13:192
21144212718,91sleep321163-21cpu08:18:053
21144212718,91sleep321163-21cpu08:18:053
27899212630,23sleep127997-21date10:22:471
27899212630,23sleep127997-21date10:22:471
20628212526,76sleep10-21swapper/109:58:111
14722212517,87sleep314797-21threads09:38:183
1126212127,67sleep01145-21apt-get12:22:570
24960212010,23sleep10-21swapper/108:28:251
24960212010,23sleep10-21swapper/108:28:251
1434721186,103sleep00-21swapper/011:22:400
3232211628,49sleep13283-21ls10:47:471
30094211511,90sleep130130-21ntpq10:28:121
18487211512,90sleep118551-21unin-run09:52:461
2726211320,70sleep30-21swapper/309:02:483
20039211322,65sleep320183-21grep11:38:183
20039211322,65sleep320183-21grep11:38:183
12051211315,55sleep10-21swapper/111:13:161
24458211116,83sleep00-21swapper/010:09:250
19803211116,71sleep00-21swapper/011:38:150
19803211116,71sleep00-21swapper/011:38:150
9684211015,81sleep29712-21basename07:42:462
28692211031,41sleep10-21swapper/110:23:161
28692211031,41sleep10-21swapper/110:23:161
22549211022,63sleep222426-21latency_hist08:22:482
22549211022,63sleep222426-21latency_hist08:22:482
29783210921,66sleep329800-21fschecks_count10:28:063
28623210916,19sleep00-21swapper/010:23:150
28623210916,19sleep00-21swapper/010:23:150
2124821099,64sleep121284-21ntpq11:43:121
13293210810,57sleep113352-21if_eth011:18:081
11736210722,68sleep011881-21unixbench_singl09:28:220
11736210722,68sleep011881-21unixbench_singl09:28:220
2423921058,57sleep124357-21cat08:27:481
1416721056,11sleep10-21swapper/107:55:051
2887321045,11sleep10-21swapper/112:07:521
5902210317,27sleep20-21swapper/209:12:492
32424210315,69sleep332458-21ntp_kernel_err12:18:123
1902921039,80sleep119104-21dpkg08:12:461
636421025,34sleep10-21swapper/110:55:101
6962210117,61sleep26853-21/usr/sbin/munin07:33:052
1660621015,12sleep10-21swapper/109:43:411
9090210012,72sleep09214-21sed11:03:250
2270329912,74sleep222795-21/usr/sbin/munin11:48:102
1918629919,59sleep319191-21apt-key08:12:483
2581029721,36sleep10-21swapper/111:58:021
2581029721,36sleep10-21swapper/111:58:021
24612975,11sleep10-21swapper/109:00:071
32892965,10sleep10-21swapper/107:20:321
32892965,10sleep10-21swapper/107:20:321
2824329623,56sleep328305-21latency06:58:093
278042965,81sleep00-21swapper/010:22:090
278042965,81sleep00-21swapper/010:22:090
2034429616,54sleep220478-21head08:13:402
1277229616,61sleep312538-21/usr/sbin/munin09:33:053
1277229616,61sleep312538-21/usr/sbin/munin09:33:053
2808829531,18sleep10-21swapper/112:03:201
2808829531,18sleep10-21swapper/112:03:201
1722029529,42sleep317212-21apt-get09:48:043
342829446,33sleep10-21swapper/109:03:141
270429414,60sleep32780-21/usr/sbin/munin07:18:153
270429414,60sleep32780-21/usr/sbin/munin07:18:153
2551229420,54sleep025690-21latency_hist11:57:490
2289529318,60sleep222414-21apt-get08:22:572
2457129214,59sleep324642-21fschecks_time08:28:073
2457129214,59sleep324642-21fschecks_time08:28:073
2096629218,50sleep021030-21fschecks_time11:43:070
195529220,55sleep01974-21latency_hist07:17:480
195529220,55sleep01974-21latency_hist07:17:480
1903329220,51sleep319038-21modprobe11:37:453
1847429227,41sleep018475-21cron09:52:450
2711829113,62sleep227260-21latency_hist12:02:482
2711829113,62sleep227260-21latency_hist12:02:482
2368829129,49sleep11-21systemd11:52:451
1311029112,65sleep013160-21cpuspeed07:53:020
199112884,10sleep00-21swapper/009:55:480
867628714,55sleep38379-21/usr/sbin/munin11:03:163
3082028724,40sleep330834-21apt07:07:453
3082028724,40sleep330834-21apt07:07:453
2890828715,56sleep328953-21/usr/sbin/munin12:08:023
1128728720,47sleep311341-21latency_hist07:47:483
1128728720,47sleep311341-21latency_hist07:47:483
2009528619,49sleep320159-21find08:13:373
1908528626,16sleep10-21swapper/111:37:471
1625728624,18sleep00-21swapper/011:28:090
15612867,67sleep11607-21ntpq12:23:151
266242850,9sleep326594-21timerwakeupswit11:58:323
266242850,9sleep326594-21timerwakeupswit11:58:323
175628524,41sleep21834-21find12:23:202
2725528429,33sleep327377-21sort10:18:213
2725528429,33sleep327377-21sort10:18:213
584628319,51sleep05862-21open_inodes10:54:120
2465128311,54sleep224778-21/usr/sbin/munin11:53:192
1371228213,18sleep30-21swapper/307:53:173
3076728117,42sleep330779-21kernelversion12:13:113
2844028128,30sleep00-21swapper/006:58:140
95132807,32sleep10-21swapper/109:23:061
95132807,32sleep10-21swapper/109:23:061
79982809,36sleep10-21swapper/107:35:581
539728011,54sleep15496-21fschecks_time07:28:071
1475228014,51sleep314831-21fschecks_time07:58:063
463627913,49sleep04723-21fw_packets09:08:060
2527827913,21sleep10-21swapper/110:13:101
2527827913,21sleep10-21swapper/110:13:101
818227816,34sleep00-21swapper/007:37:470
579027714,49sleep25855-21cat07:28:162
2742327713,48sleep027563-21latency_hist08:37:490
2573727727,32sleep225764-21munin-run08:32:462
2573727727,32sleep225764-21munin-run08:32:462
150802777,32sleep10-21swapper/111:23:131
150802777,32sleep10-21swapper/111:23:131
1301027712,49sleep013020-21ntpq09:33:100
1301027712,49sleep013020-21ntpq09:33:100
314302760,8sleep30-21swapper/308:48:173
2717127617,47sleep227254-21switchtime10:18:202
2717127617,47sleep227254-21switchtime10:18:202
1485727622,31sleep014897-21if_eth011:23:080
1485727622,31sleep014897-21if_eth011:23:080
655827520,37sleep36559-21cron07:32:453
473127516,37sleep34770-21munin-run10:52:463
2517427517,24sleep00-21swapper/008:28:280
2517427517,24sleep00-21swapper/008:28:280
1005127512,26sleep30-21swapper/309:23:203
1005127512,26sleep30-21swapper/309:23:203
58627320,33sleep2728-21grep08:53:182
2923327318,24sleep10-21swapper/107:02:461
2013627325,27sleep20-21swapper/209:57:482
1265227311,50sleep3369-21sshd07:50:593
1046527316,42sleep00-21swapper/011:08:180
72212727,14sleep20-21swapper/210:58:142
31372728,53sleep00-21swapper/010:47:100
527627114,41sleep34751-21apt-get10:52:583
167427118,26sleep00-21swapper/008:58:090
2141027015,37sleep221482-21/usr/sbin/munin11:43:172
1648327016,40sleep216535-21latency08:03:102
1444527025,23sleep20-21swapper/207:57:482
2969126923,22sleep00-21swapper/007:03:100
2969126923,22sleep00-21swapper/007:03:100
2522526925,21sleep00-21swapper/010:13:090
2522526925,21sleep00-21swapper/010:13:090
20682690,7sleep20-21swapper/208:58:422
1463926918,27sleep20-21swapper/209:38:172
1015826918,25sleep00-21swapper/009:23:210
1015826918,25sleep00-21swapper/009:23:210
3169426825,23sleep20-21swapper/207:08:092
2090026826,20sleep00-21swapper/008:17:500
1042426814,40sleep010431-21ntp_states07:43:120
1042426814,40sleep010431-21ntp_states07:43:120
500726716,33sleep05030-21munin-run07:27:450
500726716,33sleep05030-21munin-run07:27:450
290512675,49sleep20-21swapper/210:23:212
290512675,49sleep20-21swapper/210:23:212
2623926717,23sleep20-21swapper/208:33:152
2623926717,23sleep20-21swapper/208:33:152
517526624,22sleep30-21swapper/309:08:173
2572726623,24sleep30-21swapper/310:13:203
2572726623,24sleep30-21swapper/310:13:203
3018226513,19sleep20-21swapper/212:12:462
1618626517,29sleep30-21swapper/308:02:583
1493626519,22sleep20-21swapper/207:58:092
1318226512,38sleep313192-21forks11:18:063
1232826526,21sleep20-21swapper/209:32:482
1232826526,21sleep20-21swapper/209:32:482
116626516,25sleep00-21swapper/008:57:480
800226417,23sleep00-21swapper/009:18:080
504926419,26sleep25240-21gpgv10:52:492
3248026414,26sleep30-21swapper/308:53:073
3164626416,31sleep05666-21snmpd10:33:120
1566926412,37sleep315736-21df09:43:033
971626321,22sleep20-21swapper/211:07:472
266726313,32sleep22702-21grep07:18:142
266726313,32sleep22702-21grep07:18:142
2291326315,27sleep322414-21apt-get08:22:583
2119426321,23sleep00-21swapper/009:58:230
2044526315,31sleep220522-21fschecks_time09:58:082
102226314,26sleep30-21swapper/310:38:203
593626214,30sleep06054-21timerwakeupswit07:28:180
3059226213,33sleep030610-21fschecks_count12:13:080
2198026119,22sleep00-21swapper/008:18:220
2198026119,22sleep00-21swapper/008:18:220
116872618,39sleep011518-21/usr/sbin/munin11:13:060
813626011,24sleep30-21swapper/307:37:463
412026023,21sleep30-21swapper/307:23:123
412026023,21sleep30-21swapper/307:23:123
357926011,23sleep00-21swapper/009:03:180
287202607,41sleep30-21swapper/308:40:513
80692595,10sleep30-21swapper/311:02:113
320325916,22sleep30-21swapper/309:03:083
2861425919,23sleep20-21swapper/212:07:452
2861425919,23sleep20-21swapper/212:07:452
2829125921,18sleep00-21swapper/008:38:170
269982598,41sleep00-21swapper/008:33:370
18372599,13sleep30-21swapper/312:23:213
849325815,29sleep38504-21df_inode07:38:023
3179525814,20sleep00-21swapper/007:08:110
2940925812,33sleep329456-21unin-run10:27:463
2940925812,33sleep329456-21unin-run10:27:463
2279625810,21sleep30-21swapper/311:48:113
1595425813,29sleep215993-21sed11:27:462
1595425813,29sleep215993-21sed11:27:462
1303225817,23sleep20-21swapper/209:33:122
1303225817,23sleep20-21swapper/209:33:122
1133925815,26sleep211486-21latency_hist11:12:482
80132575,11sleep00-21swapper/011:01:270
2345125714,23sleep323447-21apt-get10:07:593
179452579,33sleep018004-21basename11:33:060
1785025719,18sleep20-21swapper/208:08:062
153552577,12sleep30-21swapper/311:23:193
153552577,12sleep30-21swapper/311:23:193
86325617,22sleep00-21swapper/010:38:170
2969825614,27sleep229692-21apt-get10:28:012
270425613,28sleep32755-21/usr/sbin/munin12:28:013
253925623,18sleep30-21swapper/310:43:253
2446925618,17sleep30-21swapper/311:53:163
229522569,39sleep00-21swapper/008:23:190
127525618,17sleep30-21swapper/307:13:193
127525618,17sleep30-21swapper/307:13:193
33862555,38sleep00-21swapper/007:21:480
33862555,38sleep00-21swapper/007:21:480
3183125516,24sleep231896-21switchtime10:33:182
275725511,19sleep00-21swapper/012:28:020
973825419,18sleep00-21swapper/007:42:480
324632547,15sleep00-21swapper/012:18:130
2984625411,25sleep029972-21unixbench_multi08:43:180
2984625411,25sleep029972-21unixbench_multi08:43:180
2956825416,23sleep229558-21apt-get07:03:042
2956825416,23sleep229558-21apt-get07:03:042
2742325414,24sleep327426-21df12:03:013
2742325414,24sleep327426-21df12:03:013
1772725416,17sleep20-21swapper/209:48:192
111512545,11sleep20-21swapper/207:46:382
111512545,11sleep20-21swapper/207:46:382
110772548,12sleep30-21swapper/309:28:083
110772548,12sleep30-21swapper/309:28:083
1074925417,17sleep20-21swapper/209:27:472
1074925417,17sleep20-21swapper/209:27:472
1035825413,26sleep30-21swapper/311:08:133
641625315,22sleep36425-21diskstats09:13:063
2606225311,24sleep326055-21apt-get08:33:033
2606225311,24sleep326055-21apt-get08:33:033
1914625315,17sleep20-21swapper/211:37:482
1914625315,17sleep20-21swapper/211:37:482
1783825312,23sleep017844-21df_inode08:08:040
168525316,18sleep20-21swapper/210:42:472
158752536,12sleep00-21swapper/009:43:080
1352625314,23sleep213550-21ntp_states11:18:132
375425213,24sleep23744-21apt-get07:23:032
375425213,24sleep23744-21apt-get07:23:032
2320225212,25sleep323215-21munin-run10:07:473
1566825211,25sleep215660-21apt-get09:43:022
2801225110,22sleep20-21swapper/208:38:112
2243925117,18sleep00-21swapper/010:03:150
1689725117,19sleep00-21swapper/008:03:200
158832515,37sleep00-21swapper/008:02:150
1237525115,20sleep20-21swapper/207:48:192
630625011,22sleep05809-21apt-get09:12:550
480125014,18sleep00-21swapper/010:52:470
1777425011,23sleep217766-21apt-get11:32:592
120525015,17sleep20-21swapper/207:13:182
120525015,17sleep20-21swapper/207:13:182
1162625011,24sleep211630-21diskstats11:13:042
427724911,21sleep24375-21users10:48:212
328524913,15sleep20-21swapper/210:47:482
3204124914,21sleep332127-21mailstats07:08:183
2261524911,22sleep022674-21if_err_eth011:48:080
2096024912,23sleep320964-21diskstats11:43:063
381924811,20sleep23868-21users09:03:202
1645224813,19sleep216477-21irqstats11:28:152
653724619,15sleep20-21swapper/209:13:252
331824613,14sleep20-21swapper/212:28:162
1879224611,20sleep318786-21apt-get09:53:003
83222458,19sleep20-21swapper/209:18:142
3050424412,16sleep20-21swapper/212:13:022
3136424315,15sleep30-21swapper/310:33:073
2923224111,16sleep20-21swapper/208:43:042
2923224111,16sleep20-21swapper/208:43:042
161952376,14sleep30-21swapper/311:28:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional