You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 20:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Mar 10, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
161532116642,1108sleep05436-1kworker/0:0H07:33:400
17066210229,995sleep316128-21/usr/sbin/munin07:34:003
16223277315,742sleep116268-21/usr/sbin/munin10:58:481
24312243310,405sleep024348-21ntpq07:58:480
29668243026,351sleep029758-21fschecks_time09:58:430
29668243026,351sleep029758-21fschecks_time09:58:430
149342407129,259sleep014962-21sh12:38:230
2688226767,179sleep02753-21fschecks_time11:58:580
213422667,246sleep02170-21ntpq10:13:450
213422667,246sleep02170-21ntpq10:13:450
2343422516,235sleep013441-21snmpd11:21:280
2343422516,235sleep013441-21snmpd11:21:280
18867224645,189sleep018179-21/usr/sbin/munin09:23:550
26799224441,182sleep026820-21latency_hist08:08:250
5037223737,184sleep05095-21fschecks_count10:23:420
1515322289,207sleep015217-21grep09:13:420
1674122278,206sleep016814-21cut09:18:420
3751222632,177sleep03796-21ntpq10:18:460
13091222645,83sleep00-21swapper/010:48:460
21139222038,126sleep021267-21timerwakeupswit11:13:530
21139222038,126sleep021267-21timerwakeupswit11:13:530
4077221512,188sleep04088-21acpi12:03:350
17617220434,121sleep00-21swapper/011:03:440
1526219150,28sleep00-21swapper/010:13:250
23847219087,85sleep023851-21acpi11:23:340
7279218510,162sleep32117-1kworker/3:1H08:48:373
20915218346,81sleep00-21swapper/007:48:420
20915218346,81sleep00-21swapper/007:48:420
20257217561,99sleep020298-21latency_hist11:13:250
1848217421,134sleep31903-21fschecks_count10:13:403
1848217421,134sleep31903-21fschecks_count10:13:403
19513217217,79sleep219550-21latency_hist09:28:252
1986216921,131sleep11993-21kworker/u8:111:58:231
30149216611,106sleep08857-21munin-node08:18:340
27724216647,59sleep00-21swapper/009:53:260
27724216647,59sleep00-21swapper/009:53:260
26577216646,80sleep00-21swapper/009:48:460
910721587,14sleep00-21swapper/008:53:470
910721587,14sleep00-21swapper/008:53:470
18368215846,98sleep018495-21sed07:38:530
1157021576,116sleep00-21swapper/012:24:580
2466521568,104sleep00-21swapper/009:43:370
1562421559,130sleep015681-21apt-config07:33:240
29762215314,124sleep129820-21if_eth009:58:441
29762215314,124sleep129820-21if_eth009:58:441
9316215243,22sleep00-21swapper/010:38:240
1048121517,130sleep00-21swapper/008:58:400
611221488,104sleep013441-21snmpd12:08:520
26969214647,81sleep00-21swapper/011:33:390
9141214343,86sleep09245-21fschecks_count07:13:300
9141214343,86sleep09245-21fschecks_count07:13:300
26680214345,80sleep026699-21sh11:33:230
12967214020,102sleep012968-21fschecks_count07:23:590
2237221386,42sleep07256-21cyclictest09:36:020
2237221386,42sleep07256-21cyclictest09:36:020
15622213863,19sleep00-21swapper/010:58:270
12210213814,108sleep012283-21fschecks_time12:28:420
4571213714,108sleep14625-21processes12:03:521
30491213749,50sleep030535-21ntpq11:43:490
17186213614,107sleep317139-21latency_hist11:03:263
484213452,43sleep00-21swapper/011:53:250
8717213114,99sleep38789-21unin-run12:18:233
19271213148,47sleep019266-21apt-get07:43:380
19271213148,47sleep019266-21apt-get07:43:380
10967213116,48sleep211005-21/usr/sbin/munin12:23:472
17966213029,83sleep318013-21latency_hist09:23:263
5700212954,19sleep00-21swapper/008:43:460
3939212952,17sleep00-21swapper/008:38:370
1580212818,80sleep10-21swapper/110:13:261
1580212818,80sleep10-21swapper/110:13:261
19279212711,42sleep119347-21sh07:43:431
19279212711,42sleep119347-21sh07:43:431
10338212544,43sleep010472-21sed12:23:250
7868212415,96sleep17899-21ntp_kernel_pll_12:14:051
7870212238,63sleep00-21swapper/007:08:490
31609212239,45sleep031671-21/usr/sbin/munin10:03:520
16562212211,68sleep016709-21unixbench_singl10:58:550
11490212211,67sleep011532-21grep07:18:510
7676212012,83sleep00-21swapper/008:49:030
1298821170,27sleep112992-21unixbench_singl12:28:541
7413211620,77sleep26922-21apt-get12:13:342
29334211628,29sleep20-21swapper/209:58:252
29334211628,29sleep20-21swapper/209:58:252
4172211014,76sleep24180-21sshd12:03:412
3700211012,40sleep03770-21latency_hist08:38:250
3700211012,40sleep03770-21latency_hist08:38:250
4358210913,80sleep34403-21ntp_states08:38:563
7257210716,73sleep16779-21apt-get08:48:351
16339210518,62sleep116340-21cron09:18:231
8791210310,17sleep00-21swapper/012:18:240
6897210330,22sleep06939-21awk10:28:470
24428210327,63sleep024487-21date09:43:250
20146210313,22sleep00-21swapper/009:28:480
19115210212,75sleep319173-21idleruntime11:08:433
11834210216,67sleep111851-21modprobe12:28:231
1214621006,43sleep00-21swapper/009:03:440
823529924,51sleep38383-21grep07:08:563
121262995,46sleep00-21swapper/010:44:260
2304429726,15sleep00-21swapper/007:53:550
1692429613,68sleep117053-21awk07:33:581
2975029512,67sleep129752-21sshd11:42:451
2415429412,63sleep124211-21irqstats07:58:451
1866929410,65sleep10-21swapper/109:23:521
130929437,17sleep00-21swapper/008:28:520
2169929311,16sleep20-21swapper/209:33:502
2169929311,16sleep20-21swapper/209:33:502
7292928,16sleep00-21swapper/011:53:380
985229136,13sleep00-21swapper/010:38:440
2748629135,17sleep00-21swapper/008:08:490
525229017,60sleep15368-21latency_hist08:43:261
2830029016,29sleep20-21swapper/208:13:232
729928922,48sleep27274-21apt-get08:48:482
632428819,24sleep30-21swapper/312:08:553
318452887,44sleep031931-21perl08:23:470
2863028824,41sleep20-21swapper/211:38:422
2907928724,22sleep00-21swapper/011:38:520
335128628,35sleep33404-21ls10:18:373
2408128613,19sleep20-21swapper/211:23:442
2098628616,55sleep221062-21tr11:13:512
2098628616,55sleep221062-21tr11:13:512
1955128616,17sleep30-21swapper/309:28:263
1402928415,53sleep314060-21latency_hist10:53:253
2139428343,23sleep221480-21sed07:48:532
2139428343,23sleep221480-21sed07:48:532
303428215,15sleep20-21swapper/210:18:242
303428215,15sleep20-21swapper/210:18:242
2754828217,40sleep30-21swapper/308:08:523
137552815,13sleep30-21swapper/312:33:413
133328119,49sleep21428-21sed11:53:522
361928024,40sleep23623-21cron08:38:232
361928024,40sleep23623-21cron08:38:232
1863828010,53sleep30-21swapper/307:38:563
1749128019,40sleep217534-21uniq11:03:422
1429228014,28sleep20-21swapper/209:08:552
697527915,49sleep17082-21latency_hist12:13:251
291172799,54sleep30-21swapper/308:13:533
2537327915,45sleep325520-21grep09:43:533
1012827920,43sleep210258-21grep07:13:562
1012827920,43sleep210258-21grep07:13:562
991027814,49sleep30-21swapper/307:13:533
991027814,49sleep30-21swapper/307:13:533
2751027821,31sleep227538-21munin-run09:53:242
2576827821,42sleep225807-21expr08:03:472
1634427811,51sleep21-21systemd09:18:242
1233727820,35sleep20-21swapper/207:23:252
90227722,33sleep21044-21grep10:08:552
2527027716,28sleep20-21swapper/209:43:522
2307627722,36sleep323218-21grep11:19:013
2307627722,36sleep323218-21grep11:19:013
2011927713,32sleep30-21swapper/309:28:473
724327615,46sleep17195-21latency_hist07:08:261
600927629,29sleep36119-21tune2fs07:03:443
2745427619,30sleep10-21swapper/111:33:521
990227524,33sleep19967-21latency10:38:451
89327517,40sleep1897-21diskstats08:28:411
2980127522,29sleep30-21swapper/311:43:233
2550927524,32sleep325549-21fschecks_count11:28:423
600127422,35sleep25891-21/usr/sbin/munin07:03:432
58527421,24sleep20-21swapper/208:28:262
303627421,32sleep30-21swapper/308:33:563
303627421,32sleep30-21swapper/308:33:563
297922746,56sleep00-21swapper/008:17:330
2747027424,23sleep10-21swapper/109:53:231
1391527414,16sleep30-21swapper/309:08:483
2394927229,23sleep30-21swapper/311:23:413
1297227214,34sleep212995-21irqstats10:48:442
1175427215,38sleep211823-21/usr/sbin/munin10:43:552
978127111,27sleep10-21swapper/112:18:541
92012710,8sleep20-21swapper/212:18:422
2859127110,15sleep20-21swapper/208:13:402
1321127122,23sleep113297-21mailstats10:48:511
695927014,40sleep36996-21ps10:28:503
3172327012,19sleep10-21swapper/111:48:421
147727019,27sleep30-21swapper/310:13:243
647126919,28sleep20-21swapper/210:28:332
3120226915,29sleep10-21swapper/110:03:411
3017626917,38sleep230224-21fschecks_count11:43:422
1321926918,35sleep313232-21modprobe09:08:233
851526812,41sleep38552-21ps10:33:513
585926823,24sleep30-21swapper/308:43:493
1228726816,32sleep112311-21/usr/sbin/munin07:23:241
3121426712,19sleep30-21swapper/310:03:433
285472677,11sleep30-21swapper/309:54:063
285472677,11sleep30-21swapper/309:54:063
2358426725,27sleep123597-21idleruntime-cro11:23:231
2358426725,27sleep123597-21idleruntime-cro11:23:231
1904126719,24sleep20-21swapper/211:08:422
1681526714,30sleep10-21swapper/109:18:431
1343426715,29sleep30-21swapper/312:33:253
1071226720,28sleep110759-21awk07:18:231
1071226720,28sleep110759-21awk07:18:231
1049626714,37sleep210590-21perl08:58:422
2683026612,38sleep326958-21timerwakeupswit09:48:533
180872669,13sleep10-21swapper/107:38:461
1460226612,39sleep214636-21/usr/sbin/munin07:28:462
875526523,26sleep28757-21cron07:13:232
66326518,18sleep10-21swapper/110:08:501
625926520,21sleep20-21swapper/210:28:252
321782657,15sleep30-21swapper/308:23:543
3063726526,22sleep10-21swapper/108:18:521
3057126512,38sleep230614-21tr08:18:492
2402926513,29sleep30-21swapper/307:58:433
134892655,11sleep00-21swapper/009:08:290
377226417,19sleep10-21swapper/108:38:261
377226417,19sleep10-21swapper/108:38:261
3074926422,22sleep30-21swapper/308:18:543
2255726420,28sleep322565-21modprobe09:38:233
2255726420,28sleep322565-21modprobe09:38:233
214626414,34sleep32294-21latency_hist11:58:253
212982648,15sleep10-21swapper/109:33:371
212982648,15sleep10-21swapper/109:33:371
1111226419,21sleep30-21swapper/307:18:413
694026312,36sleep16951-21ntp_states10:28:481
552526318,26sleep35612-21sed10:23:543
343426322,25sleep20-21swapper/210:18:402
2841626316,31sleep128553-21modprobe08:13:261
2338826319,25sleep30-21swapper/309:39:123
1561326320,22sleep30-21swapper/310:58:263
1281426318,27sleep212808-21apt-get07:23:372
127526318,22sleep30-21swapper/308:28:513
1219526318,22sleep20-21swapper/212:28:402
1193526316,27sleep20-21swapper/209:03:342
304582625,12sleep30-21swapper/311:43:483
2370426220,26sleep223838-21cut07:58:252
612326111,34sleep06176-21cat07:03:450
2601126119,20sleep10-21swapper/109:48:271
2294826115,27sleep323042-21/usr/sbin/munin07:53:543
2241526115,17sleep20-21swapper/211:18:432
2241526115,17sleep20-21swapper/211:18:432
1779326119,19sleep20-21swapper/207:38:392
314226019,25sleep20-21swapper/211:59:092
2439626015,28sleep224403-21/usr/sbin/munin07:58:502
2278526018,23sleep222583-21apt-get09:38:262
145626013,28sleep21555-21/usr/sbin/munin08:28:542
3017725926,19sleep30-21swapper/309:58:543
3017725926,19sleep30-21swapper/309:58:543
1385725917,21sleep10-21swapper/112:33:431
674725816,24sleep26767-21/usr/sbin/munin08:48:242
2817125810,35sleep313441-21snmpd11:37:423
2712925824,19sleep10-21swapper/108:08:421
256625815,26sleep12535-21apt-get11:58:511
1427525816,25sleep114268-21apt-get10:53:391
1196025814,26sleep111954-21apt-get09:03:371
971925717,20sleep20-21swapper/210:38:422
2558525716,21sleep30-21swapper/308:03:433
1633725714,24sleep20-21swapper/207:33:472
1533825715,29sleep315406-21ls09:13:453
708325616,18sleep20-21swapper/212:13:262
2106325611,27sleep221182-21latency_hist09:33:262
1378525610,27sleep10-21swapper/107:24:411
1010825617,23sleep310172-21find10:38:513
888325513,25sleep19021-21latency_hist07:13:271
881525514,24sleep18827-21apt-get08:53:361
881525514,24sleep18827-21apt-get08:53:361
556825518,21sleep10-21swapper/108:43:431
495125512,25sleep15018-21df10:23:391
290122557,10sleep10-21swapper/109:55:171
290122557,10sleep10-21swapper/109:55:171
2255125519,16sleep10-21swapper/107:53:441
2200925512,22sleep10-21swapper/111:18:261
2200925512,22sleep10-21swapper/111:18:261
30212548,18sleep30-21swapper/311:59:063
2670325415,25sleep126729-21munin-run11:33:241
2541925413,24sleep225413-21apt-get11:28:372
2081625413,24sleep120813-21apt-get07:48:351
2081625413,24sleep120813-21apt-get07:48:351
1436425412,25sleep314359-21apt-get07:28:393
1278425413,25sleep30-21swapper/307:23:343
1182425411,26sleep311950-21timerwakeupswit10:43:563
1114925422,16sleep30-21swapper/312:23:523
1040325413,23sleep310400-21apt-get08:58:353
957725312,25sleep39644-21switchtime12:18:523
939225316,15sleep10-21swapper/110:38:251
850825314,23sleep28514-21ntp_states10:33:502
361225317,22sleep10-21swapper/110:18:431
2127325310,18sleep30-21swapper/307:48:503
2127325310,18sleep30-21swapper/307:48:503
1740725314,25sleep317474-21df11:03:393
1568225317,18sleep10-21swapper/107:33:251
891225214,23sleep38919-21diskstats08:53:423
891225214,23sleep38919-21diskstats08:53:423
968925110,16sleep20-21swapper/208:53:572
968925110,16sleep20-21swapper/208:53:572
1626925112,22sleep216305-21ntp_states10:58:492
3221125012,15sleep10-21swapper/108:23:561
2106325018,19sleep30-21swapper/311:13:523
2106325018,19sleep30-21swapper/311:13:523
1360625013,16sleep10-21swapper/109:08:421
4742499,25sleep30-21swapper/311:53:243
2682324914,16sleep10-21swapper/108:08:261
1801124911,16sleep10-21swapper/111:03:531
1795524916,19sleep20-21swapper/209:23:252
142762497,17sleep20-21swapper/210:53:402
3139724814,15sleep20-21swapper/210:03:462
2831624810,15sleep20-21swapper/211:38:262
1922424811,15sleep10-21swapper/111:08:451
1374724811,21sleep213754-21diskstats12:33:402
437224713,19sleep30-21swapper/312:03:463
3017024710,21sleep130175-21diskstats11:43:411
2018324712,20sleep120205-21ntp_states09:28:491
1166724711,18sleep10-21swapper/110:43:541
238532469,14sleep10-21swapper/111:23:351
1935724613,16sleep30-21swapper/307:43:453
1935724613,16sleep30-21swapper/307:43:453
316302449,20sleep231632-21apt-get11:48:352
2314924412,18sleep10-21swapper/109:39:021
432024314,14sleep20-21swapper/208:38:552
104062429,16sleep10-21swapper/108:58:371
192732417,18sleep20-21swapper/207:43:402
192732417,18sleep20-21swapper/207:43:402
2964524010,16sleep20-21swapper/209:58:412
2964524010,16sleep20-21swapper/209:58:412
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional