You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 16:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Tue Nov 25, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21767243430,29sleep00-21swapper/010:48:490
7451232252,252sleep07570-21wc08:19:000
30523227174,178sleep030647-21cat07:49:000
11069224112,213sleep211118-21ntpq10:13:522
32304224012,128sleep032313-21munin-run11:23:280
13998222310,202sleep014045-21latency_hist12:08:320
1739322198,199sleep00-21swapper/010:33:560
2578422099,124sleep025852-21ls09:18:510
1808521839,162sleep1362-21snmpd07:12:071
7522217666,30sleep07290-21/usr/sbin/munin10:03:490
23727217620,83sleep023816-21latency_hist09:13:300
223122171128,25sleep00-21swapper/007:23:570
223122171128,25sleep00-21swapper/007:23:570
2352321709,110sleep023619-21grep07:29:010
2352321709,110sleep023619-21grep07:29:010
22310216827,117sleep322460-21grep10:49:003
1013821659,13sleep20-21swapper/208:28:512
3944215961,83sleep04075-21grep11:34:090
3944215961,83sleep04075-21grep11:34:090
1872721599,97sleep018738-21apt-get07:13:390
5862215233,99sleep05970-21fschecks_time09:58:480
22141214811,120sleep022222-21latency_hist09:08:290
1310821478,12sleep10-21swapper/108:38:471
28699214315,35sleep30-21swapper/309:28:473
2752821438,121sleep027549-21latency_hist11:08:260
2752821438,121sleep027549-21latency_hist11:08:260
27736213963,61sleep00-21swapper/007:39:510
12009213938,23sleep00-21swapper/010:18:300
70421387,116sleep0838-21apt-config09:43:300
20372213854,60sleep00-21swapper/012:28:300
6783213644,23sleep00-21swapper/011:43:580
1759213412,108sleep21860-21timerwakeupswit09:44:252
14876213420,88sleep215026-21unixbench_singl12:09:102
14710213343,16sleep00-21swapper/007:03:290
14710213343,16sleep00-21swapper/007:03:290
14400213345,32sleep014487-21sort10:23:590
21343213113,91sleep00-21swapper/010:48:290
21343213113,91sleep00-21swapper/010:48:290
3033221308,109sleep1225-21systemd-journal07:48:571
10860212641,24sleep010852-21apt-get11:58:430
10860212641,24sleep010852-21apt-get11:58:430
16306212514,95sleep015932-21/usr/sbin/munin12:14:530
15126212516,96sleep315181-21idleruntime07:03:493
3328212410,100sleep33335-21forks11:33:563
3328212410,100sleep33335-21forks11:33:563
2146721237,105sleep0362-21snmpd07:22:150
2146721237,105sleep0362-21snmpd07:22:150
23544212127,25sleep00-21swapper/010:53:540
27901212013,79sleep00-21swapper/011:08:470
12449212045,63sleep012465-21cpu12:03:430
5156211827,67sleep00-21swapper/011:38:580
29230211821,72sleep229362-21timerwakeupswit09:28:582
26030211711,25sleep30-21swapper/311:03:293
1868821168,93sleep00-21swapper/010:38:480
1868821168,93sleep00-21swapper/010:38:480
18243211613,88sleep218213-21latency_hist10:38:302
29804211513,87sleep229847-21ntp_offset11:13:532
25223211514,24sleep00-21swapper/010:58:580
10257211544,19sleep010284-21awk08:28:530
30461211418,84sleep130544-21sed09:33:501
14664211343,18sleep014780-21perl08:43:470
16612211112,79sleep00-21swapper/010:33:270
16100210928,30sleep00-21swapper/008:48:390
9063210639,17sleep08594-21apt-get10:08:330
16314210610,27sleep016448-21latency_hist07:08:300
3139621059,85sleep0362-21snmpd09:37:440
5527210312,75sleep25670-21date09:58:302
29950210310,23sleep00-21swapper/009:33:280
52121027,11sleep00-21swapper/007:58:330
29582210217,70sleep018933-1kworker/0:1H07:48:270
1599210153,34sleep01665-21mailstats09:44:230
31210210014,70sleep231350-21latency_hist07:53:302
13502210030,46sleep113524-21munin-plugin-st10:23:271
31022998,66sleep00-21swapper/009:48:550
64029817,62sleep3680-21sort07:58:483
95642976,11sleep00-21swapper/008:28:000
34802978,74sleep0362-21snmpd08:08:300
2318129713,14sleep20-21swapper/209:09:002
10912975,12sleep00-21swapper/011:24:570
584929611,70sleep35859-21forks09:58:473
289929620,60sleep22937-21wc09:48:492
28802965,11sleep00-21swapper/011:33:000
1747529610,75sleep00-21swapper/012:18:330
187862955,11sleep00-21swapper/008:56:020
1759429512,15sleep30-21swapper/312:18:483
575129419,17sleep00-21swapper/008:13:590
1884929418,60sleep118957-21fschecks_time07:13:521
53782935,11sleep00-21swapper/009:57:100
3271429325,20sleep30-21swapper/307:58:273
3271429325,20sleep30-21swapper/307:58:273
3252529311,54sleep20-21swapper/211:23:382
30742935,11sleep00-21swapper/008:04:420
280462935,76sleep00-21swapper/009:24:530
2011728913,58sleep220153-21fschecks_count10:43:462
2011728913,58sleep220153-21fschecks_count10:43:462
2472828817,48sleep124851-21head09:13:591
152152885,72sleep00-21swapper/012:11:060
1363328860,14sleep013769-21ls08:38:580
907328715,53sleep38594-21apt-get10:08:403
597228714,36sleep10-21swapper/109:58:491
2212928720,47sleep10-21swapper/110:48:581
124752876,34sleep00-21swapper/008:35:110
1960128617,51sleep319669-21/usr/sbin/munin12:23:573
169252869,22sleep016918-21apt-get07:08:460
153222869,11sleep20-21swapper/207:03:522
3071728549,23sleep0302-21systemd-logind11:18:270
291428418,46sleep22939-21munin-run11:33:272
221882849,58sleep022377-21cat12:33:300
2687028215,50sleep326974-21/usr/sbin/munin11:03:573
2687028215,50sleep326974-21/usr/sbin/munin11:03:573
204528224,44sleep32090-21awk11:28:523
2362028116,49sleep123681-21if_eth007:29:021
2362028116,49sleep123681-21if_eth007:29:021
128572819,57sleep012973-21timerandwakeup10:18:580
2987228016,44sleep129383-21/usr/sbin/munin11:13:561
2730228026,30sleep30-21swapper/307:39:463
1194228016,52sleep311989-21processes08:34:003
2160327922,40sleep1313-21rs:main1
2160327922,40sleep1313-21rs:main1
1272827922,33sleep212760-21cut08:38:282
804727821,37sleep38062-21ls08:23:273
361527816,42sleep13208-21/usr/sbin/munin11:34:021
361527816,42sleep13208-21/usr/sbin/munin11:34:021
3200727826,31sleep20-21swapper/207:54:022
3200727826,31sleep20-21swapper/207:54:022
250292788,15sleep00-21swapper/009:15:560
2246427825,28sleep10-21swapper/109:08:471
1584827821,27sleep20-21swapper/210:28:572
1355427816,29sleep30-21swapper/310:23:293
59927726,24sleep30-21swapper/309:43:293
54927720,38sleep2597-21systemd-run09:43:282
1702127721,25sleep30-21swapper/307:08:543
1294527721,34sleep20-21swapper/212:03:582
83927621,20sleep10-21swapper/109:43:311
2624327620,29sleep20-21swapper/207:38:282
2624327620,29sleep20-21swapper/207:38:282
1151127626,23sleep20-21swapper/211:59:092
973727521,24sleep30-21swapper/311:53:543
973727521,24sleep30-21swapper/311:53:543
89082755,51sleep00-21swapper/011:50:240
89082755,51sleep00-21swapper/011:50:240
3171927519,27sleep10-21swapper/107:53:531
3171927519,27sleep10-21swapper/107:53:531
2119427517,41sleep321217-21/usr/sbin/munin09:03:523
1701427517,29sleep30-21swapper/310:33:473
2590827417,40sleep324948-21/usr/sbin/munin07:34:093
2590827417,40sleep324948-21/usr/sbin/munin07:34:093
218827420,35sleep12205-21fschecks_count08:03:461
1463527420,32sleep214674-21munin-run07:03:272
1463527420,32sleep214674-21munin-run07:03:272
2642827318,29sleep10-21swapper/111:03:471
2642827318,29sleep10-21swapper/111:03:471
2324627316,36sleep223262-21fschecks_count10:53:472
192602737,13sleep00-21swapper/008:58:400
1588527319,24sleep30-21swapper/308:48:303
3139927220,35sleep131436-21sed11:18:531
2539627218,24sleep20-21swapper/210:59:002
1650527224,27sleep20-21swapper/212:15:002
1306227217,31sleep313146-21grep12:04:033
1195527222,33sleep1225-21systemd-journal10:18:271
277402719,44sleep30-21swapper/309:23:593
1090227120,32sleep10-21swapper/111:58:441
1090227120,32sleep10-21swapper/111:58:441
946627020,24sleep20-21swapper/211:53:452
946627020,24sleep20-21swapper/211:53:452
252572706,11sleep00-21swapper/007:33:540
252572706,11sleep00-21swapper/007:33:540
1665527012,25sleep30-21swapper/308:48:583
256932699,14sleep20-21swapper/209:18:492
2258626921,25sleep30-21swapper/309:08:493
1581826920,26sleep10-21swapper/110:28:561
449226820,22sleep20-21swapper/211:38:272
449226820,22sleep20-21swapper/211:38:272
3263426816,28sleep10-21swapper/111:23:451
322132686,11sleep00-21swapper/007:54:040
322132686,11sleep00-21swapper/007:54:040
2038326811,12sleep30-21swapper/307:18:473
2038326811,12sleep30-21swapper/307:18:473
1946826820,25sleep30-21swapper/307:14:153
181682687,13sleep20-21swapper/208:53:582
1539826816,30sleep30-21swapper/312:13:273
82362676,49sleep30-21swapper/310:04:083
800926712,40sleep17981-21fschecks_count11:48:511
800926712,40sleep17981-21fschecks_count11:48:511
505426721,20sleep20-21swapper/211:38:562
2916726723,22sleep20-21swapper/211:13:292
964726618,29sleep10-21swapper/108:28:291
2068226622,23sleep30-21swapper/312:28:453
1953326618,29sleep219597-21cat08:58:502
1893526613,38sleep218943-21sh10:38:542
1893526613,38sleep218943-21sh10:38:542
473326523,20sleep30-21swapper/311:38:443
234126521,27sleep12380-21modprobe09:48:271
2031926519,26sleep120326-21modprobe12:28:271
1253326512,41sleep112534-21sshd08:35:561
1007426512,20sleep10-21swapper/108:28:501
570726417,26sleep30-21swapper/308:13:583
2922226420,21sleep10-21swapper/107:44:001
2576226419,23sleep10-21swapper/107:34:071
2576226419,23sleep10-21swapper/107:34:071
2140226419,23sleep10-21swapper/109:03:581
2087626411,38sleep220967-21cat07:18:582
2087626411,38sleep220967-21cat07:18:582
2039926420,21sleep30-21swapper/310:43:523
2039926420,21sleep30-21swapper/310:43:523
1779026417,25sleep10-21swapper/108:53:491
164152647,42sleep30-21swapper/310:30:563
1625026412,35sleep315932-21/usr/sbin/munin12:14:523
260726321,20sleep30-21swapper/308:03:573
1468726327,20sleep30-21swapper/307:03:283
1468726327,20sleep30-21swapper/307:03:283
583926221,26sleep15924-21awk08:14:001
3066726215,25sleep330731-21find09:33:563
2999526221,17sleep10-21swapper/109:33:291
246632625,15sleep20-21swapper/207:33:272
246632625,15sleep20-21swapper/207:33:272
23262627,12sleep10-21swapper/111:28:581
1937926222,23sleep20-21swapper/207:14:142
15762628,12sleep10-21swapper/108:00:231
147826214,27sleep30-21swapper/309:44:173
1392926216,12sleep30-21swapper/308:39:013
904926112,28sleep30-21swapper/308:23:593
634126120,19sleep20-21swapper/209:58:582
2246326120,19sleep30-21swapper/307:24:003
2246326120,19sleep30-21swapper/307:24:003
2208926117,28sleep222109-21modprobe09:08:272
779926017,18sleep20-21swapper/211:48:402
757126014,28sleep17693-21/usr/sbin/munin08:19:011
1876826017,20sleep20-21swapper/212:23:272
1720126016,24sleep20-21swapper/207:08:572
136426013,27sleep30-21swapper/311:28:273
1232026015,31sleep112341-21ls12:03:281
927125913,22sleep20-21swapper/210:09:202
3027825919,22sleep20-21swapper/209:33:472
2984825917,21sleep30-21swapper/311:13:543
1892625911,32sleep118934-21ntp_states10:38:531
1892625911,32sleep118934-21ntp_states10:38:531
1155525918,18sleep30-21swapper/311:59:103
86302589,17sleep10-21swapper/110:08:301
476225821,18sleep30-21swapper/309:53:583
47562589,11sleep10-21swapper/108:11:171
2963125819,16sleep20-21swapper/207:48:292
2781825820,20sleep10-21swapper/111:08:441
2061025820,24sleep120621-21grep12:28:431
1998225816,20sleep30-21swapper/308:59:003
1800925813,26sleep118071-21/usr/sbin/munin12:19:101
173082586,10sleep20-21swapper/208:52:282
158922586,40sleep10-21swapper/107:04:111
1288525819,21sleep10-21swapper/112:03:571
1055225815,20sleep30-21swapper/308:29:003
63342577,12sleep10-21swapper/111:43:431
500225721,16sleep30-21swapper/308:13:293
2837325712,15sleep10-21swapper/109:28:291
241472578,18sleep20-21swapper/209:13:482
2057625717,18sleep10-21swapper/110:43:581
2057625717,18sleep10-21swapper/110:43:581
2004625710,36sleep20-21swapper/212:24:572
178872577,37sleep10-21swapper/110:34:331
1596125711,30sleep115950-21apt-get12:13:451
457525618,23sleep14183-21/usr/sbin/munin09:53:521
2597525620,18sleep10-21swapper/109:18:561
1440425615,26sleep114498-21/usr/sbin/munin12:09:011
1376725616,21sleep10-21swapper/110:23:401
826325514,29sleep28264-21sshd10:04:272
820925515,23sleep28258-21memory11:48:562
820925515,23sleep28258-21memory11:48:562
809625514,20sleep30-21swapper/311:48:533
809625514,20sleep30-21swapper/311:48:533
49052557,35sleep10-21swapper/111:38:501
316425514,27sleep13235-21sensors_temp09:48:561
2967425515,17sleep30-21swapper/307:48:303
2692725515,17sleep20-21swapper/207:39:372
2079125519,21sleep20-21swapper/212:28:482
1770225513,26sleep317699-21apt-get08:53:433
1400225513,24sleep314059-21irqstats10:23:503
1230725517,22sleep212318-21df_inode10:18:442
524525413,20sleep20-21swapper/208:13:452
290162545,37sleep30-21swapper/311:12:333
2679425413,24sleep126760-21apt-get07:38:591
2320525413,16sleep20-21swapper/207:28:312
2320525413,16sleep20-21swapper/207:28:312
1461625412,26sleep114642-21/usr/sbin/munin08:43:431
1427525413,16sleep20-21swapper/208:43:282
499225315,24sleep25001-21munin-run08:13:282
267002537,11sleep30-21swapper/309:22:293
942125213,22sleep19515-21memory10:09:231
3143725212,24sleep231456-21ntp_states11:18:542
2316025212,23sleep123155-21apt-get10:53:421
1092825211,22sleep310962-21irqstats10:13:493
2451925114,18sleep30-21swapper/309:13:573
2219825114,22sleep222309-21sed10:48:592
3164325011,22sleep30-21swapper/311:18:593
149122505,10sleep20-21swapper/210:26:312
258042496,32sleep10-21swapper/111:01:131
2369924910,22sleep323793-21memory07:29:043
2369924910,22sleep323793-21memory07:29:043
1927824911,21sleep319383-21users10:38:593
1927824911,21sleep319383-21users10:38:593
1640924914,19sleep10-21swapper/108:48:521
700224813,18sleep20-21swapper/208:18:492
3044524815,17sleep30-21swapper/307:48:593
2647624812,21sleep226532-21irqstats11:03:482
2647624812,21sleep226532-21irqstats11:03:482
1500724811,21sleep215026-21ntp_states08:43:532
631924710,19sleep30-21swapper/311:43:413
87922469,15sleep20-21swapper/208:23:542
293824611,20sleep32997-21memory09:48:503
2352724613,16sleep30-21swapper/310:53:523
1344624613,13sleep10-21swapper/106:58:471
1344624613,13sleep10-21swapper/106:58:471
2316624515,15sleep10-21swapper/107:28:301
2316624515,15sleep10-21swapper/107:28:301
10022459,14sleep20-21swapper/207:58:552
210524415,13sleep20-21swapper/208:03:432
196792448,15sleep10-21swapper/108:58:531
2751524011,15sleep20-21swapper/209:23:542
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional