You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 20:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Mar 13, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
195372230931,2256sleep019538-21apt-get12:23:260
888625489,527sleep38919-21ntpq10:08:363
888625489,527sleep38919-21ntpq10:08:363
19348249048,426sleep219354-21df_inode07:18:282
1387245233,397sleep31474-21fschecks_time08:03:313
1387245233,397sleep31474-21fschecks_time08:03:313
32200242115,390sleep26002-21kworker/u8:007:58:292
245482321105,200sleep324672-21sort07:33:433
189602318112,36sleep318976-21modprobe07:18:133
18771226267,175sleep318903-21unixbench_multi10:38:443
16514225575,54sleep30-21swapper/310:33:273
16056225576,124sleep316118-21mailstats08:48:573
2932322429,191sleep329406-21mailstats07:48:413
2932322429,191sleep329406-21mailstats07:48:413
24974222542,164sleep325058-21timerwakeupswit10:58:433
23273222571,48sleep30-21swapper/310:53:423
20955222441,163sleep321093-21grep09:03:433
20955222441,163sleep321093-21grep09:03:433
6940222234,169sleep36945-21cpu10:03:273
31469221422,166sleep231487-21fschecks_count09:38:352
27541220520,165sleep327752-21latency_hist11:08:153
27541220520,165sleep327752-21latency_hist11:08:153
23337220010,177sleep323370-21cpu09:13:283
23337220010,177sleep323370-21cpu09:13:283
7896219923,58sleep30-21swapper/308:23:383
7896219923,58sleep30-21swapper/308:23:383
26224219830,99sleep326263-21wc07:39:013
29654219411,147sleep329420-21/usr/sbin/munin11:13:333
14767219412,168sleep314829-21idleruntime12:08:313
2959621929,138sleep30-21swapper/309:33:153
2398521888,165sleep324095-21latency_hist12:38:153
2398521888,165sleep324095-21latency_hist12:38:153
29157218525,55sleep30-21swapper/311:13:123
4395217923,39sleep30-21swapper/308:13:293
4395217923,39sleep30-21swapper/308:13:293
32224217729,112sleep313441-21snmpd07:58:473
13224217720,141sleep013253-21grep12:04:040
1438217626,111sleep31511-21latency_hist11:28:143
1337421718,127sleep313412-21wc12:04:073
241921678,101sleep30-21swapper/309:48:373
383821669,118sleep33955-21cat11:33:433
26101216314,132sleep125607-21apt-get11:03:251
26101216314,132sleep125607-21apt-get11:03:251
3225216211,136sleep03228-21grep11:33:280
1749721609,135sleep317533-21ntpq08:53:383
1749721609,135sleep317533-21ntpq08:53:383
7122215414,117sleep00-21swapper/010:03:320
2246121509,76sleep31-21systemd10:53:133
21197214365,61sleep021228-21/usr/sbin/munin07:23:390
20578214315,112sleep020600-21sort07:23:140
13832214129,100sleep313863-21tune2fs08:43:323
28018214014,105sleep328162-21unixbench_singl07:43:453
21092213918,16sleep30-21swapper/312:28:263
19514213725,22sleep30-21swapper/312:23:253
808213427,86sleep1955-21grep11:23:441
4104213311,107sleep34129-21modprobe08:13:123
31849213347,73sleep331883-21ps09:38:443
23046213339,57sleep323068-21idleruntime-cro09:13:123
916021299,78sleep30-21swapper/308:28:303
5012212713,101sleep35031-21apt-get11:38:243
5012212713,101sleep35031-21apt-get11:38:243
3939212613,97sleep14025-21fw_packets09:53:361
1954521269,97sleep30-21swapper/307:18:333
13407212656,36sleep30-21swapper/310:23:263
377321226,33sleep30-21swapper/309:53:183
12002212242,39sleep30-21swapper/310:19:143
10095212213,39sleep30-21swapper/311:53:383
19991211915,38sleep30-21swapper/309:03:153
19802211917,84sleep10-21swapper/112:24:041
10038211735,61sleep30-21swapper/310:13:293
8872211616,75sleep09016-21unixbench_singl11:48:460
875621169,71sleep38868-21cat11:48:453
5736211516,26sleep00-21swapper/008:18:140
5736211516,26sleep00-21swapper/008:18:140
1532211510,83sleep11566-21/usr/sbin/munin08:03:331
1532211510,83sleep11566-21/usr/sbin/munin08:03:331
1108521159,94sleep311122-21ps08:33:393
743211433,65sleep3782-21latency09:43:373
1483221149,15sleep00-21swapper/012:08:320
27217211312,36sleep30-21swapper/309:23:503
14441211321,74sleep014526-21memory07:03:490
14441211321,74sleep014526-21memory07:03:490
6704211128,42sleep36804-21latency_hist10:03:153
29982211132,61sleep30-21swapper/309:33:313
5466210819,71sleep15537-21fschecks_time09:58:291
16191210829,64sleep316225-21ps07:08:403
22374210713,79sleep122419-21grep07:28:281
9064210438,51sleep09160-21sed10:08:410
9064210438,51sleep09160-21sed10:08:410
702421046,34sleep30-21swapper/311:43:413
11304210416,67sleep211310-21modprobe10:18:122
2952521037,79sleep00-21swapper/011:13:300
27042210315,71sleep127083-21latency_hist07:43:141
2368221037,32sleep30-21swapper/307:33:153
19640210331,39sleep319614-21/usr/sbin/munin10:43:283
12411210216,71sleep112490-21/usr/sbin/munin10:19:241
5769210122,23sleep10-21swapper/108:18:151
5769210122,23sleep10-21swapper/108:18:151
17033210112,73sleep117064-21latency_hist07:13:141
14956210027,45sleep214953-21apt-get10:28:252
181292996,84sleep00-21swapper/008:55:210
181292996,84sleep00-21swapper/008:55:210
1507829924,53sleep115201-21latency_hist08:48:151
1284629916,59sleep112965-21latency_hist12:03:161
2391129812,73sleep123982-21switchtime09:13:431
2391129812,73sleep123982-21switchtime09:13:431
1570329819,62sleep215717-21acpi07:08:232
1069329816,65sleep010596-21/usr/sbin/munin08:33:280
1217529724,14sleep30-21swapper/308:38:243
3266229620,63sleep032666-21entropy11:23:310
637929518,64sleep26423-21ntpq08:18:362
637929518,64sleep26423-21ntpq08:18:362
728929411,56sleep00-21swapper/008:23:150
728929411,56sleep00-21swapper/008:23:150
599529413,65sleep06092-21timerandwakeup09:58:400
1777529412,68sleep11-21systemd10:38:131
721029311,58sleep26915-21/usr/sbin/munin10:03:342
605829321,15sleep30-21swapper/308:18:293
605829321,15sleep30-21swapper/308:18:293
2118429325,35sleep00-21swapper/010:48:260
1472229318,54sleep014768-21latency_hist10:28:150
2491829221,14sleep30-21swapper/309:18:253
176029266,12sleep30-21swapper/311:28:303
1455129215,60sleep014649-21users08:43:460
1190529232,41sleep30-21swapper/311:58:423
1662929114,43sleep316659-21proc_pri12:13:403
1317329116,62sleep013246-21cat10:23:150
719029010,65sleep17192-21sshd08:22:351
719029010,65sleep17192-21sshd08:22:351
586328917,58sleep25921-21sendmail_mailqu09:58:382
3138328922,42sleep00-21swapper/009:38:290
2340228922,29sleep00-21swapper/012:33:440
2340228922,29sleep00-21swapper/012:33:440
149572897,41sleep30-21swapper/310:28:273
794128829,34sleep10-21swapper/111:48:151
178128821,47sleep00-21swapper/009:48:140
1759828820,30sleep30-21swapper/312:18:163
280212876,13sleep10-21swapper/111:08:581
2621628618,51sleep226238-21fschecks_count11:03:462
2621628618,51sleep226238-21fschecks_count11:03:462
2352128511,55sleep023312-21/usr/sbin/munin09:13:340
2352128511,55sleep023312-21/usr/sbin/munin09:13:340
2143128529,27sleep20-21swapper/207:23:452
211282859,34sleep30-21swapper/307:23:383
1573228413,58sleep213441-21snmpd12:11:212
705028311,56sleep07056-21ntp_states11:43:420
514828314,52sleep05155-21/usr/sbin/munin11:38:450
514828314,52sleep05155-21/usr/sbin/munin11:38:450
2684328317,43sleep226921-21fschecks_time09:23:412
2242028312,54sleep222446-21df07:28:302
1410828326,35sleep213630-21apt-get07:03:232
1410828326,35sleep213630-21apt-get07:03:232
2802228217,51sleep028119-21latency_hist09:28:150
2796928220,45sleep127974-21modprobe09:28:121
2752128113,55sleep127259-21/usr/sbin/munin07:43:351
2247028117,26sleep20-21swapper/209:08:432
220082815,59sleep20-21swapper/212:28:452
697328014,43sleep27022-21ntpq11:43:392
2084128015,48sleep220954-21cat09:03:422
2084128015,48sleep220954-21cat09:03:422
1106528017,30sleep00-21swapper/011:58:150
303792797,32sleep30-21swapper/307:53:263
1976127915,19sleep10-21swapper/107:18:391
1835727917,48sleep118402-21latency10:38:341
1591327916,26sleep00-21swapper/007:08:330
3196227833,25sleep132044-21switchtime09:38:461
1743727821,37sleep0294-21rs:main0
16732789,16sleep10-21swapper/111:28:241
1414527819,39sleep314128-21apt-get07:03:323
1414527819,39sleep314128-21apt-get07:03:323
370327731,27sleep10-21swapper/111:33:411
270832778,57sleep00-21swapper/011:04:330
270832778,57sleep00-21swapper/011:04:330
415727624,29sleep20-21swapper/208:13:152
2281927615,44sleep122879-21fschecks_time10:53:321
2312727519,25sleep10-21swapper/112:33:411
2312727519,25sleep10-21swapper/112:33:411
1541827520,35sleep115473-21munin-run07:08:131
1541827520,35sleep115473-21munin-run07:08:131
80232747,52sleep00-21swapper/008:23:430
80232747,52sleep00-21swapper/008:23:430
354727416,24sleep13730-21cat09:53:161
3275827416,17sleep10-21swapper/107:59:171
1625127411,40sleep216141-21/usr/sbin/munin12:13:312
402627318,38sleep24083-21interrupts09:53:372
37752739,51sleep00-21swapper/009:53:190
2725827321,38sleep227260-21/usr/sbin/munin07:43:242
1848227321,23sleep10-21swapper/112:18:431
1553827323,32sleep115532-21apt-get08:48:271
309782727,29sleep30-21swapper/311:18:173
3016727218,37sleep230191-21/usr/sbin/munin09:33:342
2070627217,15sleep00-21swapper/009:03:400
2070627217,15sleep00-21swapper/009:03:400
947827114,22sleep10-21swapper/108:28:371
341127115,36sleep03538-21gpgv09:53:150
2765727117,29sleep00-21swapper/007:43:380
2643727122,25sleep20-21swapper/207:39:042
1613627114,42sleep016205-21cat08:48:590
1140027118,29sleep00-21swapper/010:18:140
3047127016,41sleep130475-21modprobe11:15:121
1659727017,32sleep216489-21/usr/sbin/munin10:33:302
2929726937,18sleep20-21swapper/207:48:402
2929726937,18sleep20-21swapper/207:48:402
1840226913,41sleep218433-21munin-run08:58:132
1840226913,41sleep218433-21munin-run08:58:132
1777026914,22sleep10-21swapper/108:53:441
1777026914,22sleep10-21swapper/108:53:441
2158226814,26sleep10-21swapper/112:28:401
1432626818,33sleep1281-21dbus-daemon12:08:131
507826723,19sleep20-21swapper/208:13:442
507826723,19sleep20-21swapper/208:13:442
328026718,29sleep23308-21munin-run09:53:132
2631226713,37sleep026466-21cat09:23:150
1812426713,21sleep20-21swapper/212:18:362
1172526711,40sleep011749-21proc_pri11:58:390
2558026616,34sleep21-21systemd11:03:142
3258226513,35sleep232651-21df11:23:292
22692656,11sleep20-21swapper/208:04:312
22692656,11sleep20-21swapper/208:04:312
1312126522,25sleep20-21swapper/210:23:132
876226412,34sleep18852-21memory10:08:341
876226412,34sleep18852-21memory10:08:341
624926416,24sleep10-21swapper/111:42:331
624926416,24sleep10-21swapper/111:42:331
3160826420,28sleep131677-21/usr/sbin/munin11:18:431
2825826412,33sleep028307-21/usr/sbin/munin09:28:280
2474226421,19sleep20-21swapper/209:18:162
1776326419,28sleep00-21swapper/007:13:290
1302726413,36sleep212544-21apt-get12:03:222
1151226426,19sleep10-21swapper/110:18:151
3216826311,20sleep00-21swapper/007:58:230
2467826322,19sleep00-21swapper/010:58:350
1972326315,29sleep219755-21tune2fs10:43:312
1802426313,18sleep00-21swapper/012:18:340
162262636,16sleep10-21swapper/110:33:131
3152926214,23sleep00-21swapper/011:18:420
2929026217,28sleep129296-21ntp_states07:48:391
2929026217,28sleep129296-21ntp_states07:48:391
292026219,22sleep20-21swapper/211:33:012
1487826220,19sleep10-21swapper/112:08:331
1374626219,27sleep213742-21apt-get08:43:272
2940826119,17sleep00-21swapper/007:48:420
2940826119,17sleep00-21swapper/007:48:420
290462615,44sleep00-21swapper/011:11:480
2264526114,20sleep00-21swapper/007:28:360
219426116,26sleep00-21swapper/008:03:450
219426116,26sleep00-21swapper/008:03:450
1748026113,20sleep10-21swapper/107:13:221
1674226114,33sleep116820-21ls12:13:421
1398226117,23sleep214058-21wc10:23:412
3164026019,20sleep20-21swapper/207:58:132
3151026015,22sleep10-21swapper/107:56:351
3148926014,30sleep231495-21ntp_states11:18:402
3004726015,26sleep030070-21irqstats09:33:320
231032609,32sleep223137-21ntp_kernel_err10:53:372
2244526019,20sleep20-21swapper/210:53:122
1864726012,18sleep10-21swapper/108:58:241
1070826019,19sleep20-21swapper/208:33:302
344325914,28sleep23494-21irqstats11:33:342
28742595,44sleep00-21swapper/011:32:270
1504325912,28sleep00-21swapper/010:28:320
983225817,25sleep29896-21/usr/sbin/munin11:53:332
966025816,24sleep19679-21apt-get11:53:231
87825818,25sleep2923-21/usr/sbin/munin09:43:402
3180525821,22sleep00-21swapper/007:58:150
2551725815,25sleep125600-21/usr/sbin/munin09:18:441
2437925818,22sleep124388-21df_inode10:58:281
1963825817,25sleep119633-21apt-get10:43:261
1935125817,27sleep019400-21unin-run10:43:130
1867225815,24sleep018666-21apt-get08:58:280
2803125716,22sleep20-21swapper/211:08:592
2415525716,24sleep224189-21irqstats07:33:342
1477125713,19sleep10-21swapper/110:28:161
40362567,37sleep00-21swapper/008:12:210
3082425619,16sleep20-21swapper/211:18:162
2980025615,24sleep129794-21apt-get09:33:251
2398425615,19sleep20-21swapper/209:13:442
2398425615,19sleep20-21swapper/209:13:442
1133925612,17sleep20-21swapper/211:58:292
979925516,22sleep29929-21sort10:13:152
979925516,22sleep29929-21sort10:13:152
953825517,21sleep00-21swapper/008:28:400
2518025512,19sleep10-21swapper/107:38:141
2109425519,16sleep00-21swapper/012:28:270
1817425516,18sleep20-21swapper/210:38:312
1777525517,17sleep20-21swapper/207:13:312
1697725526,16sleep00-21swapper/010:33:400
2626525411,22sleep00-21swapper/007:39:020
2319425415,16sleep20-21swapper/212:33:422
2319425415,16sleep20-21swapper/212:33:422
1247025412,23sleep212566-21memory08:38:352
1238825414,27sleep012449-21if_eth008:38:330
85732539,16sleep20-21swapper/211:48:402
303125315,19sleep10-21swapper/108:08:341
2230125314,18sleep00-21swapper/009:08:410
2117525313,24sleep121178-21apt-get10:48:231
2080625315,18sleep20-21swapper/212:28:132
752725216,20sleep10-21swapper/108:23:281
752725216,20sleep10-21swapper/108:23:281
554825217,21sleep25610-21mailstats11:38:552
554825217,21sleep25610-21mailstats11:38:552
1963725214,20sleep00-21swapper/010:43:250
1062125212,23sleep110615-21apt-get08:33:251
63325116,18sleep10-21swapper/109:43:351
2851425120,19sleep228233-21/usr/sbin/munin09:28:342
1867525112,15sleep20-21swapper/208:58:302
798325010,18sleep20-21swapper/208:23:412
798325010,18sleep20-21swapper/208:23:412
2389525012,19sleep023914-21apt-get07:33:240
1556625010,22sleep215532-21apt-get08:48:402
2951524914,20sleep129556-21systemd-run09:33:131
2386124915,17sleep00-21swapper/010:56:340
819124713,16sleep10-21swapper/111:48:311
736924611,20sleep17393-21ntp_states10:03:371
1037524612,17sleep20-21swapper/208:33:142
217862448,19sleep10-21swapper/109:08:261
121982448,17sleep10-21swapper/108:38:261
215332428,13sleep10-21swapper/109:08:141
215332428,13sleep10-21swapper/109:08:141
2103624210,13sleep10-21swapper/107:23:361
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional