You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 08:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Tue Jul 07, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2455528618,794sleep224632-21apt-config22:59:112
8650246620,424sleep08669-21apt-get23:49:230
201124556,439sleep2364-21snmpd20:03:362
25795244852,325sleep225941-21unixbench_multi22:59:442
25795244852,325sleep225941-21unixbench_multi22:59:442
209832422162,184sleep221006-21apt21:04:112
13080233254,199sleep2680-21cron00:04:102
30595231553,200sleep230656-21sendmail_mailqu19:49:402
30595231553,200sleep230656-21sendmail_mailqu19:49:402
464230410,280sleep1364-21snmpd19:59:111
2180223049,278sleep221850-21ntp_offset00:29:382
333822856,231sleep2364-21snmpd20:05:462
22050226115,67sleep221249-21/usr/sbin/munin21:04:442
10308225964,153sleep2294-21in:imuxsock22:14:102
17695224862,47sleep217826-21timerwakeupswit22:34:412
16009224847,161sleep216073-21mailstats22:29:452
16343224433,196sleep216452-21cut20:49:142
21845224037,178sleep221933-21fschecks_time22:49:312
14059224011,204sleep214214-21unixbench_multi00:04:432
19501223241,164sleep219652-21latency_hist20:59:152
1745222129,191sleep0364-21snmpd00:15:430
24971220951,116sleep20-21swapper/219:34:142
23424220420,132sleep223492-21latency_hist19:29:132
26668220345,115sleep20-21swapper/219:39:142
30977219210,124sleep231013-21ntp_kernel_err21:34:392
24865219225,152sleep224936-21tr21:14:412
2759421919,166sleep227689-21perl21:24:302
18690218717,72sleep20-21swapper/200:19:492
5349218345,21sleep20-21swapper/221:55:382
26563218017,41sleep126665-21apt-config19:39:131
8903217620,135sleep38968-21sort20:24:323
8903217620,135sleep38968-21sort20:24:323
12173217514,144sleep312184-21systemd22:19:123
10808217524,128sleep16952-21kworker/u8:222:14:221
9994217412,62sleep20-21swapper/223:54:132
12277217335,21sleep20-21swapper/220:34:362
2912721667,21sleep20-21swapper/221:29:302
2102521668,23sleep20-21swapper/219:19:402
5741215924,123sleep25771-21ntpq23:39:362
652021587,114sleep26593-21switchtime21:59:412
652021587,114sleep26593-21switchtime21:59:412
32196214754,32sleep232279-21timerandwakeup23:19:502
8198214449,27sleep28283-21latency_hist23:49:132
145421427,82sleep21517-21cat21:44:322
145421427,82sleep21517-21cat21:44:322
9983214014,113sleep19993-21unin-run23:54:121
18591213217,101sleep318602-21ntpq20:54:403
27971212840,77sleep228044-21latency_hist23:09:122
27971212840,77sleep228044-21latency_hist23:09:122
25136212813,80sleep10-21swapper/121:14:441
5142212622,26sleep00-21swapper/023:39:140
9260212464,45sleep20-21swapper/220:24:392
9260212464,45sleep20-21swapper/220:24:392
23463212428,17sleep20-21swapper/221:09:442
20574212412,29sleep020716-21sed00:24:480
12822212448,61sleep212869-21ntpq22:19:362
27032212112,95sleep027013-21apt-get19:39:320
27032212112,95sleep027013-21apt-get19:39:320
21877212112,92sleep321986-21latency_hist19:24:133
17916212127,24sleep30-21swapper/300:19:133
17614212145,59sleep3225-21systemd-journal19:09:123
4818212016,35sleep24964-21unixbench_multi20:09:462
12716212017,19sleep00-21swapper/022:19:340
2025211917,63sleep22172-21latency_hist23:29:152
26993211628,35sleep226995-21/usr/sbin/munin19:39:222
26993211628,35sleep226995-21/usr/sbin/munin19:39:222
10499211525,64sleep110648-21cp22:14:141
1356211411,90sleep01362-21cat21:44:300
1356211411,90sleep01362-21cat21:44:300
12366211323,66sleep012431-21switchtime23:59:400
23291211215,78sleep223337-21awk00:39:112
13400210912,82sleep313401-21sshd22:20:373
7352210715,73sleep37383-21fschecks_count20:19:323
7352210715,73sleep37383-21fschecks_count20:19:323
644210718,11sleep20-21swapper/221:39:512
644210718,11sleep20-21swapper/221:39:512
491921077,11sleep10-21swapper/121:54:441
393021076,22sleep30-21swapper/320:09:143
2310210711,59sleep22200-21/usr/sbin/munin23:29:312
23215210523,63sleep023254-21ntp_kernel_pll_21:09:380
7736210414,74sleep07770-21irqstats22:04:320
7736210414,74sleep07770-21irqstats22:04:320
29518210326,19sleep20-21swapper/223:14:132
27920210118,65sleep027936-21modprobe23:09:100
27920210118,65sleep027936-21modprobe23:09:100
233712997,78sleep00-21swapper/000:39:130
1935229813,34sleep219391-21ntpq19:14:362
1348929821,17sleep20-21swapper/220:39:292
184422968,63sleep00-21swapper/020:54:360
1503629614,70sleep115092-21cpuspeed20:44:281
1465029517,54sleep20-21swapper/222:24:482
1438729511,67sleep014432-21ntp_states22:24:420
1076129511,73sleep210803-21ntpq20:29:362
604029423,50sleep06129-21fschecks_time21:59:310
604029423,50sleep06129-21fschecks_time21:59:310
543029417,62sleep05440-21forks23:39:300
534729415,60sleep15396-21/usr/sbin/munin23:39:271
1190929413,66sleep111910-21/usr/sbin/munin20:34:271
1178729412,64sleep111788-21/usr/sbin/munin23:59:211
1680129333,16sleep20-21swapper/220:49:342
750129219,47sleep20-21swapper/222:04:262
750129219,47sleep20-21swapper/222:04:262
2884229216,55sleep328548-21/usr/sbin/munin19:44:323
2884229216,55sleep328548-21/usr/sbin/munin19:44:323
2576529214,55sleep025665-21latency_hist21:19:150
1474229222,49sleep014758-21idleruntime-cro20:44:110
2975929122,51sleep129812-21ls23:14:281
3121729011,63sleep131320-21timerandwakeup21:34:461
3084329020,46sleep130985-21grep19:49:431
3084329020,46sleep130985-21grep19:49:431
1446329018,57sleep114465-21sshd00:07:041
1036229014,59sleep110426-21cpuspeed20:29:271
2900028914,33sleep20-21swapper/219:44:352
2900028914,33sleep20-21swapper/219:44:352
857628833,29sleep10-21swapper/120:24:141
857628833,29sleep10-21swapper/120:24:141
2721928812,59sleep327278-21idleruntime19:40:003
2721928812,59sleep327278-21idleruntime19:40:003
2240828815,60sleep122537-21sed22:49:421
210828817,51sleep32195-21grep20:04:143
727028716,32sleep00-21swapper/020:19:290
727028716,32sleep00-21swapper/020:19:290
2375628721,37sleep10-21swapper/122:54:461
1030828736,15sleep20-21swapper/223:54:292
1030828736,15sleep20-21swapper/223:54:292
881528620,42sleep08811-21apt-get20:24:250
881528620,42sleep08811-21apt-get20:24:250
307802866,67sleep20-21swapper/223:16:112
211522867,67sleep20-21swapper/222:45:152
702628516,53sleep17127-21latency_hist20:19:131
2147028517,48sleep121375-21/usr/sbin/munin00:29:291
197628510,61sleep01981-21systemd-run23:29:120
1782728523,43sleep20-21swapper/200:19:122
2937228414,48sleep028548-21/usr/sbin/munin19:44:410
2937228414,48sleep028548-21/usr/sbin/munin19:44:410
1831828424,34sleep30-21swapper/300:19:383
1156828411,54sleep011709-21/usr/sbin/munin22:14:530
49622839,36sleep20-21swapper/223:38:192
2064528314,20sleep10-21swapper/119:19:311
1774528217,44sleep117826-21apt-config00:19:111
1148928119,44sleep311536-40ogrotate23:59:113
1148928119,44sleep311536-40ogrotate23:59:113
565528029,12sleep20-21swapper/220:14:222
316442807,15sleep20-21swapper/219:54:242
2642928033,14sleep20-21swapper/221:19:412
2446128011,52sleep324472-21/usr/sbin/munin21:14:293
1926028015,40sleep20-21swapper/222:39:422
1823328032,12sleep20-21swapper/219:13:272
1823328032,12sleep20-21swapper/219:13:272
160642805,64sleep20-21swapper/200:12:462
2193627912,50sleep321994-21if_eth022:49:323
198562798,19sleep219847-21apt-get00:24:292
1157627929,23sleep10-21swapper/123:59:131
1157627929,23sleep10-21swapper/123:59:131
881327820,18sleep10-21swapper/122:09:131
881327820,18sleep10-21swapper/122:09:131
871327812,33sleep30-21swapper/323:49:503
1907027820,38sleep119164-21fschecks_time19:14:311
117327812,32sleep30-21swapper/323:24:443
1139827833,11sleep20-21swapper/222:14:512
1556027720,41sleep115686-21timerwakeupswit00:09:431
1434827717,45sleep314384-21ntp_kernel_err22:24:413
3228227520,27sleep332437-21grep23:19:513
3101527513,23sleep30-21swapper/321:34:403
2857827515,31sleep10-21swapper/123:09:361
249227521,40sleep12519-21kernelversion23:29:341
1851027513,41sleep118117-21/usr/sbin/munin20:54:371
9522749,13sleep10-21swapper/123:24:371
815127414,42sleep18196-21gpgv23:49:121
2258427416,28sleep30-21swapper/321:09:133
2046427416,40sleep320571-21grep00:24:473
1635027421,22sleep20-21swapper/219:04:232
726727310,47sleep20-21swapper/220:19:272
726727310,47sleep20-21swapper/220:19:272
604827215,15sleep30-21swapper/320:14:353
1579427218,28sleep00-21swapper/022:29:390
107227223,25sleep10-21swapper/121:44:141
107227223,25sleep10-21swapper/121:44:141
440727120,27sleep10-21swapper/120:09:371
2960427114,37sleep329667-21/usr/sbin/munin21:29:433
290727121,25sleep00-21swapper/021:49:300
2654327121,20sleep00-21swapper/021:19:430
2094427118,25sleep00-21swapper/019:19:380
1598627113,41sleep115494-21/usr/sbin/munin22:29:441
1574227120,31sleep315790-21irqstats22:29:383
2455827017,33sleep124571-21vmstat19:29:451
2414027020,32sleep324226-21grep21:14:133
146562705,10sleep30-21swapper/320:43:053
602926924,27sleep36036-21forks21:59:303
602926924,27sleep36036-21forks21:59:303
33262698,46sleep20-21swapper/221:49:402
314922690,7sleep00-21swapper/019:54:130
314922690,7sleep00-21swapper/019:54:130
1834426919,24sleep00-21swapper/019:14:130
1834426919,24sleep00-21swapper/019:14:130
509226815,29sleep35232-21grep21:54:463
365126819,30sleep13741-21apt-config20:09:121
2966926818,26sleep00-21swapper/021:29:440
2866926816,28sleep00-21swapper/023:09:390
2732826818,22sleep10-21swapper/119:40:021
2732826818,22sleep10-21swapper/119:40:021
2152026816,24sleep00-21swapper/000:29:320
1812526811,31sleep00-21swapper/000:19:140
1614026829,21sleep10-21swapper/119:04:141
143192687,49sleep0364-21snmpd00:05:130
54326720,22sleep30-21swapper/319:59:133
3200526722,25sleep132042-21ntp_states23:19:451
2984226716,27sleep00-21swapper/023:14:310
2787526715,36sleep327898-21/usr/sbin/munin21:24:353
275126720,23sleep00-21swapper/023:29:410
2551026721,25sleep30-21swapper/319:34:353
2463426719,20sleep30-21swapper/322:59:123
787626619,20sleep30-21swapper/322:04:353
787626619,20sleep30-21swapper/322:04:353
317602668,13sleep00-21swapper/023:19:400
2636326623,28sleep30-21swapper/321:19:383
258226619,23sleep30-21swapper/321:49:133
258226619,23sleep30-21swapper/321:49:133
2366526618,18sleep00-21swapper/019:29:270
2337226621,23sleep10-21swapper/100:39:141
3225526514,34sleep032264-21/usr/sbin/munin21:39:290
3225526514,34sleep032264-21/usr/sbin/munin21:39:290
2637726523,23sleep10-21swapper/123:04:111
2637726523,23sleep10-21swapper/123:04:111
2232226525,20sleep00-21swapper/022:49:410
68662646,12sleep30-21swapper/323:44:213
430426420,27sleep34375-21/usr/sbin/munin23:34:403
326426417,28sleep30-21swapper/321:49:373
2061926415,32sleep320632-21/usr/sbin/munin20:59:483
1995726422,21sleep10-21swapper/122:44:131
1833626425,18sleep00-21swapper/022:39:110
1685126415,25sleep30-21swapper/320:49:353
1037926419,27sleep010497-21apt-key22:14:130
2029326317,20sleep00-21swapper/020:59:450
1789626323,23sleep10-21swapper/120:54:131
134822639,13sleep10-21swapper/120:39:241
726426222,19sleep10-21swapper/122:04:121
726426222,19sleep10-21swapper/122:04:121
2881326220,25sleep30-21swapper/323:09:423
2029026219,21sleep00-21swapper/022:44:320
1762026215,24sleep10-21swapper/122:34:401
115726213,19sleep10-21swapper/119:59:361
2643126116,29sleep026578-21latency_hist23:04:140
2643126116,29sleep026578-21latency_hist23:04:140
2366326114,29sleep323658-21apt-get19:29:263
117662618,41sleep00-21swapper/020:34:140
485226025,20sleep00-21swapper/021:54:430
31612605,16sleep30-21swapper/323:30:043
2952126019,21sleep10-21swapper/121:29:411
1531326016,19sleep30-21swapper/300:09:363
127326018,21sleep10-21swapper/121:44:271
127326018,21sleep10-21swapper/121:44:271
665625916,20sleep30-21swapper/323:44:133
2808325918,22sleep10-21swapper/121:24:411
2647025919,22sleep00-21swapper/019:39:120
2623925918,23sleep126332-21memory21:19:361
2409025915,28sleep124115-21awk21:14:111
2145925915,20sleep30-21swapper/322:49:113
2135825912,21sleep10-21swapper/121:04:301
159702598,10sleep30-21swapper/320:45:033
140525910,34sleep00-21swapper/019:59:430
941825810,27sleep30-21swapper/322:09:353
2260725815,20sleep00-21swapper/021:09:140
1455625816,19sleep10-21swapper/122:24:471
1064525814,25sleep010727-21memory20:29:340
455325716,16sleep00-21swapper/020:09:400
2725125721,16sleep10-21swapper/123:04:431
2725125721,16sleep10-21swapper/123:04:431
2233025718,21sleep122387-21irqstats19:24:361
1421725714,24sleep314270-21users00:04:443
1031825713,27sleep010411-21grep23:54:310
1031825713,27sleep010411-21grep23:54:310
764325611,18sleep00-21swapper/023:44:430
2790225615,18sleep00-21swapper/021:24:360
1960625616,18sleep10-21swapper/100:24:121
40112555,11sleep10-21swapper/121:53:191
225862558,12sleep00-21swapper/019:24:430
2034725513,25sleep120475-21timerwakeupswit20:59:461
1887825517,17sleep00-21swapper/019:14:210
1785125517,20sleep00-21swapper/020:54:120
1782825511,28sleep317041-21/usr/sbin/munin22:34:423
1571325515,21sleep00-21swapper/020:44:440
1278925515,23sleep10-21swapper/122:19:351
3264525421,17sleep30-21swapper/321:39:403
3264525421,17sleep30-21swapper/321:39:403
193522547,11sleep30-21swapper/322:39:433
1740725419,18sleep00-21swapper/019:06:350
1549025417,18sleep00-21swapper/000:09:420
2975625311,25sleep329752-21apt-get23:14:263
243625314,17sleep00-21swapper/020:04:310
211812534,37sleep10-21swapper/122:45:381
1244525310,17sleep30-21swapper/320:34:413
1062525319,18sleep30-21swapper/320:29:333
232125218,16sleep30-21swapper/320:04:223
1864725213,22sleep018650-21apt-get22:39:230
374725114,19sleep10-21swapper/123:34:221
2838725113,17sleep10-21swapper/119:44:141
2838725113,17sleep10-21swapper/119:44:141
1939325112,23sleep319435-21ntp_states19:14:373
609525014,18sleep00-21swapper/020:14:360
2472625014,21sleep024592-21latency_hist22:59:130
2220525017,16sleep30-21swapper/319:24:343
2034125015,16sleep10-21swapper/100:24:451
306872499,16sleep00-21swapper/021:34:330
3199724813,16sleep00-21swapper/019:54:380
3033124812,21sleep330388-21irqstats19:49:333
3033124812,21sleep330388-21irqstats19:49:333
2518524815,18sleep30-21swapper/322:59:293
2518524815,18sleep30-21swapper/322:59:293
230542485,31sleep30-21swapper/300:36:093
2140024816,20sleep30-21swapper/300:29:273
1188224712,20sleep311891-21df_inode23:59:273
276952466,28sleep30-21swapper/323:06:193
276952466,28sleep30-21swapper/323:06:193
980224214,15sleep10-21swapper/120:25:431
980224214,15sleep10-21swapper/120:25:431
183322387,16sleep30-21swapper/319:14:123
183322387,16sleep30-21swapper/319:14:123
181992325,11sleep10-21swapper/119:13:011
181992325,11sleep10-21swapper/119:13:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional