You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 15:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Nov 27, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
773274376,7419sleep331655-21kworker/3:209:49:533
15499228576,2838sleep213993-21kworker/2:212:21:532
10827251922,481sleep310833-21cat06:58:423
10827251922,481sleep310833-21cat06:58:423
18999247517,440sleep019030-21/usr/sbin/munin10:48:420
9585242312,289sleep39724-21unixbench_singl08:33:523
4461241631,301sleep34547-21cut08:19:063
4461241631,301sleep34547-21cut08:19:063
16264238812,361sleep216300-21ntpq07:13:472
303572370334,22sleep230359-21sshd09:42:532
13411235011,227sleep313513-21timerandwakeup12:13:503
309522344162,155sleep30-21swapper/309:43:433
309522344162,155sleep30-21swapper/309:43:433
85812327148,119sleep312086-21cyclictest11:58:483
85812327148,119sleep312086-21cyclictest11:58:483
25141229471,207sleep3225-21systemd-journal07:43:213
14449228981,193sleep314557-21cut12:18:393
13177226316,232sleep113221-21ntpq12:13:441
11926226224,227sleep35666-21snmpd10:27:533
6914225779,89sleep30-21swapper/311:53:433
17922225417,86sleep317942-21sh09:03:203
8972241113,20sleep30-21swapper/308:08:403
19646223821,198sleep119655-21/usr/sbin/munin10:49:071
231872234102,22sleep30-21swapper/311:03:233
14309223229,43sleep314357-21fschecks_time10:34:403
18964223119,50sleep319112-21unixbench_singl09:03:523
4554223035,130sleep34647-21grep10:03:423
6090222111,194sleep26171-21sshd08:23:532
6090222111,194sleep26171-21sshd08:23:532
24632221726,137sleep324683-21sort09:23:433
6387221570,30sleep30-21swapper/310:08:473
31872200118,15sleep30-21swapper/309:58:453
32455219948,137sleep332494-21cat11:33:203
22582219610,171sleep322635-21sendmail_mailqu07:33:523
22582219610,171sleep322635-21sendmail_mailqu07:33:523
21538218820,146sleep30-21swapper/309:13:423
307218611,118sleep30-21swapper/311:33:363
2805821799,158sleep328112-21processes09:33:533
18939217944,18sleep30-21swapper/310:48:413
24547217817,145sleep124560-21fschecks_count09:23:411
496821718,47sleep30-21swapper/311:48:343
15078217181,63sleep30-21swapper/308:53:303
1735121680,23sleep312086-21cyclictest12:28:233
1259521649,137sleep312719-21/usr/sbin/munin08:43:533
1259521649,137sleep312719-21/usr/sbin/munin08:43:533
2856216115,56sleep10-21swapper/108:13:471
17645216010,121sleep30-21swapper/310:43:483
9753215841,50sleep30-21swapper/312:03:413
30222215443,39sleep30-21swapper/311:23:503
30222215443,39sleep30-21swapper/311:23:503
16425215426,81sleep316503-21switchtime07:13:533
3822215214,123sleep23834-21vmstat09:58:562
11995215251,73sleep112008-21modprobe07:03:201
11995215251,73sleep112008-21modprobe07:03:201
2655721508,70sleep30-21swapper/309:28:483
18578215014,121sleep218462-21sh10:48:282
15343214931,97sleep015345-21cron10:38:200
9859214511,117sleep110004-21unixbench_singl10:18:501
2850621448,11sleep10-21swapper/109:33:591
1333621415,12sleep30-21swapper/310:30:563
1333621415,12sleep30-21swapper/310:30:563
2051621399,88sleep30-21swapper/307:28:353
13990213821,39sleep113505-21/usr/sbin/munin08:48:481
5624213212,91sleep30-21swapper/308:23:423
5624213212,91sleep30-21swapper/308:23:423
20240212911,83sleep320302-21uname09:08:493
768721288,38sleep30-21swapper/310:13:393
23779212824,20sleep30-21swapper/311:03:433
23779212824,20sleep30-21swapper/311:03:433
31712212720,67sleep30-21swapper/308:03:353
1839221275,12sleep30-21swapper/307:19:423
1610721257,12sleep30-21swapper/310:38:523
12870212412,44sleep30-21swapper/307:03:513
2796121235,35sleep30-21swapper/307:49:433
2796121235,35sleep30-21swapper/307:49:433
8713212118,86sleep18814-21timerandwakeup11:58:501
8713212118,86sleep18814-21timerandwakeup11:58:501
29926212117,63sleep30-21swapper/307:58:233
3993212010,80sleep30-21swapper/311:43:503
2162821207,99sleep321771-21latency_hist10:58:223
12804212016,85sleep212867-21find07:03:502
14554211931,74sleep314578-21kernelversion07:08:443
22914211836,51sleep222906-21apt-get09:18:382
2027621178,40sleep30-21swapper/307:28:223
1745321169,94sleep317445-21apt-get12:28:363
1255521168,73sleep312562-21munin-run12:13:223
7604211512,17sleep00-21swapper/010:13:360
2818621146,72sleep30-21swapper/311:18:423
20225211213,71sleep120242-21awk07:28:201
1121211217,75sleep11220-21latency_hist09:53:221
8634211120,34sleep30-21swapper/308:33:233
26680210933,65sleep30-21swapper/307:48:133
23524210731,36sleep323630-21cat09:18:553
18711210721,64sleep20-21swapper/212:33:202
14750210512,76sleep014790-21ntp_kernel_pll_12:18:440
30536210424,64sleep130583-21awk07:58:451
25701210422,65sleep025717-21cut07:43:460
601021030,11sleep10-21swapper/108:23:511
601021030,11sleep10-21swapper/108:23:511
1819210238,29sleep30-21swapper/309:53:483
21020210113,68sleep121093-21/usr/sbin/munin07:28:491
24965210019,35sleep30-21swapper/311:08:363
24965210019,35sleep30-21swapper/311:08:363
23167210014,70sleep223186-21latency_hist11:03:222
881529826,23sleep20-21swapper/211:58:512
881529826,23sleep20-21swapper/211:58:512
39229815,65sleep1399-21/usr/sbin/munin11:33:381
642029710,73sleep05996-21/usr/sbin/munin10:08:480
2682529721,34sleep30-21swapper/311:13:453
2543929713,65sleep225506-21ls07:43:402
958229623,33sleep30-21swapper/310:18:443
736529615,57sleep20-21swapper/210:13:222
1587629518,61sleep115920-21/usr/sbin/munin07:13:381
323112947,59sleep20-21swapper/209:48:382
1636929411,68sleep016417-21processes12:23:490
267729318,27sleep00-21swapper/009:58:220
696129112,60sleep00-21swapper/011:53:440
2962229013,60sleep029640-21/usr/sbin/munin11:23:310
2962229013,60sleep029640-21/usr/sbin/munin11:23:310
494528720,30sleep10-21swapper/111:48:321
2237228715,57sleep122451-21latency07:33:461
2237228715,57sleep122451-21latency07:33:461
219622868,61sleep122037-21find09:13:521
1031828613,56sleep210434-21ls12:03:522
1352128517,49sleep013656-21/usr/sbin/munin12:13:510
3024428419,31sleep30-21swapper/309:41:243
1879028421,44sleep018877-21grep12:33:220
18722847,43sleep31867-21apt-get11:38:373
1785928418,48sleep218009-21sed10:43:512
167462849,50sleep20-21swapper/208:58:382
1563128422,34sleep20-21swapper/207:13:232
891428322,25sleep10-21swapper/110:18:221
2600828312,56sleep126078-21/usr/sbin/munin09:28:341
940328214,49sleep19475-21/usr/sbin/munin08:33:501
291528223,34sleep12912-21apt-get09:58:351
26432826,13sleep30-21swapper/308:13:413
631628112,52sleep06413-21users08:23:550
631628112,52sleep06413-21users08:23:550
56322819,12sleep00-21swapper/010:07:060
2756128115,50sleep227575-21acpi09:33:322
2349828122,34sleep223513-21fschecks_count11:03:382
2349828122,34sleep223513-21fschecks_count11:03:382
1632428120,40sleep216482-21sed10:38:552
159022817,55sleep20-21swapper/212:23:352
473328013,51sleep04781-21grep11:48:230
302428014,49sleep03095-21cut09:58:420
2576928024,38sleep225778-21wc09:28:232
2312127913,15sleep10-21swapper/109:18:451
1755727922,25sleep00-21swapper/008:58:530
1051027911,52sleep210599-21grep08:38:432
1051027911,52sleep210599-21grep08:38:432
543327821,30sleep00-21swapper/008:23:310
543327821,30sleep00-21swapper/008:23:310
137762776,32sleep30-21swapper/308:48:423
116302777,15sleep00-21swapper/012:08:460
104212778,36sleep30-21swapper/308:38:363
104212778,36sleep30-21swapper/308:38:363
3024827636,28sleep30-21swapper/307:58:393
2665027612,49sleep026759-21fschecks_time11:13:420
246852768,14sleep00-21swapper/009:23:440
2363827618,39sleep023783-21unixbench_singl09:18:560
192132769,16sleep00-21swapper/007:23:430
1352227623,34sleep113564-21modprobe10:33:201
1352227623,34sleep113564-21modprobe10:33:201
864527518,42sleep08712-21/usr/sbin/munin11:58:490
864527518,42sleep08712-21/usr/sbin/munin11:58:490
2759727516,43sleep027708-21timerandwakeup07:48:510
2759727516,43sleep027708-21timerandwakeup07:48:510
1253027514,43sleep212593-21switchtime08:43:522
1253027514,43sleep212593-21switchtime08:43:522
774627414,42sleep07756-21/usr/sbin/munin08:29:530
42132747,14sleep10-21swapper/111:43:521
378727416,44sleep23831-21sed08:18:242
2971227314,17sleep00-21swapper/009:38:510
2144727314,37sleep221536-21fschecks_time09:13:412
1650527325,24sleep00-21swapper/007:13:540
178527218,26sleep20-21swapper/211:38:232
59327030,23sleep2537-21latency_hist08:08:232
2828527026,22sleep128319-21basename11:18:441
135832707,47sleep013611-21munin-run07:08:200
549226914,26sleep10-21swapper/108:23:361
549226914,26sleep10-21swapper/108:23:361
3156426918,27sleep031588-21netstat11:28:440
3142426820,23sleep00-21swapper/008:03:200
3114826817,26sleep10-21swapper/109:43:491
3114826817,26sleep10-21swapper/109:43:491
3109026817,24sleep20-21swapper/209:43:462
3109026817,24sleep20-21swapper/209:43:462
2195226811,43sleep122012-21fschecks_time10:58:401
2771026720,19sleep10-21swapper/107:48:521
2771026720,19sleep10-21swapper/107:48:521
1741326717,26sleep20-21swapper/207:18:322
1738926717,23sleep00-21swapper/010:43:410
3211226616,26sleep10-21swapper/108:03:461
2906326616,27sleep20-21swapper/207:53:452
2194726614,35sleep00-21swapper/010:58:390
1670926611,42sleep00-21swapper/010:40:590
913926519,20sleep20-21swapper/208:33:432
775726515,24sleep10-21swapper/108:29:541
189326517,21sleep10-21swapper/109:53:501
1201926516,22sleep00-21swapper/010:28:220
1184226523,19sleep10-21swapper/112:08:521
1110826519,29sleep011120-21netstat06:58:480
1110826519,29sleep011120-21netstat06:58:480
36432648,10sleep00-21swapper/008:18:060
2968226421,24sleep00-21swapper/007:55:530
2889726419,22sleep00-21swapper/009:38:230
2348926417,28sleep223559-21munin-run07:38:232
2348926417,28sleep223559-21munin-run07:38:232
2028226424,19sleep10-21swapper/110:53:311
1421026420,26sleep214215-21/usr/sbin/munin10:34:142
1375226416,19sleep00-21swapper/007:08:240
1106726419,30sleep211104-21sed10:23:442
2913726329,18sleep10-21swapper/109:38:361
2807926313,33sleep028098-21apt-get11:18:340
1720626316,25sleep10-21swapper/112:28:221
1590326315,24sleep10-21swapper/110:38:451
1443526314,21sleep10-21swapper/107:08:411
1373926316,22sleep00-21swapper/010:33:230
1373926316,22sleep00-21swapper/010:33:230
1194426320,27sleep011963-21apt-get08:43:340
1194426320,27sleep011963-21apt-get08:43:340
89226210,37sleep2896-21diskstats08:08:392
2860526217,23sleep20-21swapper/211:18:532
1768826224,19sleep00-21swapper/007:18:420
1053826215,23sleep20-21swapper/210:23:242
786826121,20sleep10-21swapper/111:58:211
586026118,26sleep25873-21users11:48:532
318526119,22sleep10-21swapper/111:43:231
29722614,11sleep00-21swapper/011:41:160
2600726116,27sleep026000-21apt-get09:28:330
222652615,42sleep00-21swapper/009:13:550
2077926127,18sleep20-21swapper/207:28:422
182222617,36sleep10-21swapper/109:03:341
1674026116,20sleep10-21swapper/108:58:371
161602615,9sleep20-21swapper/208:55:352
505426017,25sleep00-21swapper/011:48:390
433626015,18sleep20-21swapper/208:18:572
433626015,18sleep20-21swapper/208:18:572
3127126014,28sleep231280-21diskstats11:28:382
265126014,17sleep20-21swapper/211:39:002
2014026020,22sleep20-21swapper/209:08:442
1356526015,27sleep213619-21apt-config10:33:212
1356526015,27sleep213619-21apt-config10:33:212
2384925912,18sleep20-21swapper/207:38:312
2052125914,20sleep00-21swapper/007:28:360
1380125916,18sleep00-21swapper/008:48:430
1201025922,20sleep20-21swapper/207:03:212
1201025922,20sleep20-21swapper/207:03:212
1096625911,34sleep011044-21latency_hist12:08:220
2905125815,20sleep10-21swapper/107:53:441
2631425815,27sleep10-21swapper/111:13:221
2631425815,27sleep10-21swapper/111:13:221
246472586,40sleep00-21swapper/011:07:560
246472586,40sleep00-21swapper/011:07:560
2402625817,22sleep00-21swapper/007:39:220
1953625817,23sleep00-21swapper/009:08:220
1278825819,22sleep10-21swapper/107:03:491
1977425716,26sleep119768-21apt-get09:08:351
5512567,12sleep00-21swapper/009:48:530
3014525615,26sleep230221-21sendmail_mailtr11:23:492
3014525615,26sleep230221-21sendmail_mailtr11:23:492
2335325613,19sleep00-21swapper/007:38:210
2335325613,19sleep00-21swapper/007:38:210
203092569,12sleep20-21swapper/210:53:372
1965025619,19sleep20-21swapper/207:23:532
1443025615,15sleep10-21swapper/112:18:371
140882569,21sleep10-21swapper/110:33:411
134825613,20sleep10-21swapper/108:08:501
128992566,11sleep10-21swapper/108:43:591
128992566,11sleep10-21swapper/108:43:591
973025514,26sleep09770-21users08:33:530
2694825513,25sleep126528-21/usr/sbin/munin11:13:471
1790925513,25sleep117915-21ntp_states12:28:511
173912558,18sleep10-21swapper/107:18:311
156825511,30sleep05666-21snmpd11:38:090
1117925513,22sleep211198-21apt-get12:08:322
104982557,13sleep00-21swapper/008:38:410
104982557,13sleep00-21swapper/008:38:410
689825415,24sleep26913-21needreboot11:53:422
52225422,16sleep00-21swapper/008:08:210
492225414,23sleep24921-21ntp_states10:03:492
1985425412,17sleep00-21swapper/009:08:380
1717925412,26sleep117174-21apt-get10:43:341
1279825413,25sleep212794-21apt-get12:13:332
3044325314,21sleep130546-21users11:23:531
3044325314,21sleep130546-21users11:23:531
135825316,23sleep21352-21apt-get09:53:352
91192525,35sleep00-21swapper/010:18:250
646225213,23sleep16473-21ntp_states10:08:491
3147525211,20sleep20-21swapper/208:03:222
310122529,15sleep00-21swapper/009:43:440
310122529,15sleep00-21swapper/009:43:440
2784925213,23sleep227887-21users07:48:532
2784925213,23sleep227887-21users07:48:532
2698525211,22sleep227005-21ntp_states11:13:482
264252526,17sleep20-21swapper/209:28:432
2523325213,19sleep20-21swapper/211:08:432
2523325213,19sleep20-21swapper/211:08:432
2349125214,23sleep123497-21diskstats11:03:371
2349125214,23sleep123497-21diskstats11:03:371
1822325214,21sleep218217-21apt-get09:03:352
1519025220,17sleep00-21swapper/008:53:400
1103825214,22sleep111135-21/usr/sbin/munin08:38:551
1103825214,22sleep111135-21/usr/sbin/munin08:38:551
107852529,20sleep10-21swapper/110:23:391
1008625212,23sleep110109-21ntp_states12:03:471
238792515,34sleep10-21swapper/107:38:341
793225015,17sleep20-21swapper/208:29:592
768125014,21sleep27686-21diskstats10:13:382
341825013,21sleep23415-21apt-get11:43:322
3124625012,22sleep131270-21df11:28:371
2552425016,16sleep10-21swapper/107:43:431
2138625018,18sleep00-21swapper/010:55:530
2576624911,18sleep00-21swapper/011:08:530
2576624911,18sleep00-21swapper/011:08:530
211724913,21sleep02159-21munin-run08:13:210
1547224814,16sleep10-21swapper/108:53:451
1400724812,20sleep20-21swapper/208:48:492
545924712,18sleep10-21swapper/110:04:531
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional