You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 13:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot4s.osadl.org (updated Sat Nov 22, 2025 00:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
172221821950,3sleep3586-21kworker/3:2-events20:45:003
172221821950,3sleep3586-21kworker/3:2-events20:45:003
18627302600,1sleep3586-21kworker/3:2-events23:06:183
16425222570,4chrt586-21kworker/3:2-mm_percpu_wq19:21:333
1626132995646,7cyclictest1766841-21latency_hist21:30:000
1626132995445,7cyclictest1767153-21/usr/sbin/munin21:31:300
1626132995445,7cyclictest1767153-21/usr/sbin/munin21:31:300
1626137995349,3cyclictest1480634-21kworker/u8:0+events_unbound19:15:031
1626137995146,4cyclictest1672700-21kworker/u8:3+flush-8:020:51:261
1626137995146,4cyclictest1480634-21kworker/u8:0+flush-8:021:36:311
1626137995146,4cyclictest1480634-21kworker/u8:0+flush-8:021:36:311
1626137995144,6cyclictest1480634-21kworker/u8:0+flush-8:021:16:321
1626137995046,3cyclictest1672700-21kworker/u8:3+flush-8:021:21:291
1626132995042,6cyclictest1672700-21kworker/u8:3+flush-8:021:41:230
1626137994941,6cyclictest1875559-21kworker/u8:3+events_unbound23:41:261
1626137994941,6cyclictest1830734-21ntpq22:31:231
1626137994842,4cyclictest1480634-21kworker/u8:0+flush-8:020:16:301
1626137994742,4cyclictest1672700-21kworker/u8:3+events_unbound22:41:241
1626137994740,5cyclictest1886278-21kworker/u8:1+flush-8:023:36:251
1626137994739,5cyclictest1935717-21latency_hist00:20:011
1626137994736,9cyclictest1905209-21ntpq23:46:231
1626137994641,4cyclictest1480634-21kworker/u8:0+events_unbound20:06:291
1626137994640,5cyclictest1906104-21kworker/u8:2+flush-8:000:11:251
1626137994639,5cyclictest1480634-21kworker/u8:0+events_unbound20:36:261
1626137994639,5cyclictest1480634-21kworker/u8:0+events_unbound20:36:261
1626132994639,5cyclictest1697731-21/usr/sbin/munin20:21:320
1626132994637,7cyclictest282-21systemd-journal20:45:000
1626132994637,7cyclictest282-21systemd-journal20:45:000
1626137994540,4cyclictest1672700-21kworker/u8:3+flush-8:022:21:231
1626137994540,3cyclictest1791272-21kworker/u8:1+flush-8:022:16:291
1626132994540,3cyclictest1886278-21kworker/u8:1+flush-8:000:16:320
1626137994439,4cyclictest1672700-21kworker/u8:3+flush-8:021:11:291
1626137994438,5cyclictest1480634-21kworker/u8:0+flush-8:022:36:271
1626137994435,7cyclictest1732312-21/usr/sbin/munin20:56:291
1626132994433,8cyclictest282-21systemd-journal21:05:010
1626132994432,9cyclictest1643434-21diskstats19:26:170
1626137994339,3cyclictest1866199-21kworker/u8:2+flush-8:023:16:251
1626137994337,4cyclictest1480634-21kworker/u8:0+flush-8:019:11:241
1626137994236,5cyclictest1791272-21kworker/u8:1+flush-8:022:11:291
1626137994236,5cyclictest1791272-21kworker/u8:1+flush-8:022:11:291
1626137994236,4cyclictest1672700-21kworker/u8:3+events_unbound21:56:281
1626137994134,5cyclictest1480634-21kworker/u8:0+events_unbound20:41:271
1626137994134,5cyclictest1480634-21kworker/u8:0+events_unbound20:41:271
1626137994132,6cyclictest1771806-21latency_hist21:35:011
1626137994132,6cyclictest1771806-21latency_hist21:35:011
1626132994134,5cyclictest1866199-21kworker/u8:2+flush-8:023:16:300
1626137994033,5cyclictest1691676-21ntpq20:11:251
1626138993932,5cyclictest1801795-21cp22:05:012
1626138993932,5cyclictest1801795-21cp22:05:012
1626137993934,4cyclictest1841230-21kworker/u8:1+flush-8:023:11:201
1626137993933,4cyclictest1672700-21kworker/u8:3+events_unbound23:01:151
1626137993931,6cyclictest1886278-21kworker/u8:1+flush-8:000:06:261
1626132993932,5cyclictest1876702-21apt-get23:21:100
1626132993932,5cyclictest1672700-21kworker/u8:3+flush-8:021:11:310
1626132993930,6cyclictest1667554-21latency_hist19:50:010
1626137993833,4cyclictest1886278-21kworker/u8:1+flush-8:023:51:151
1626137993833,4cyclictest1480634-21kworker/u8:0+flush-8:020:21:311
1626137993833,3cyclictest1626003-21kworker/u8:1+events_unbound19:56:291
1626137993832,4cyclictest1672700-21kworker/u8:3+flush-8:022:51:221
1626137993832,4cyclictest1672700-21kworker/u8:3+flush-8:022:26:301
1626137993831,6cyclictest1875559-21kworker/u8:3+flush-8:023:31:291
1626137993827,8cyclictest282-21systemd-journal21:26:411
1626137993732,3cyclictest1712200-21latency_hist20:35:001
1626137993731,4cyclictest1866199-21kworker/u8:2+events_unbound23:26:191
1626132993732,4cyclictest1626003-21kworker/u8:1+flush-8:020:31:130
1626132993731,4cyclictest1626003-21kworker/u8:1+flush-8:019:31:290
1626132993730,5cyclictest1771733-21kworker/u8:2+flush-8:021:46:130
1626137993631,4cyclictest1480634-21kworker/u8:0+flush-8:020:26:181
1626137993631,4cyclictest1480634-21kworker/u8:0+events_unbound19:21:261
1626137993630,5cyclictest1626003-21kworker/u8:1+flush-8:019:31:271
1626137993629,5cyclictest1841230-21kworker/u8:1+flush-8:022:56:261
1626137993629,5cyclictest1672700-21kworker/u8:3+flush-8:020:46:301
1626137993627,6cyclictest282-21systemd-journal20:04:421
1626137993626,8cyclictest1920107-21ntpq00:01:241
1626132993631,3cyclictest1866199-21kworker/u8:2+events_unbound23:26:300
1626132993628,6cyclictest1926213-21apt-get00:11:100
1626132993628,5cyclictest282-21systemd-journal23:34:440
16259362364,10sleep10-21swapper/119:09:521
1626137993530,4cyclictest1866199-21kworker/u8:2+flush-8:023:21:151
1626137993529,4cyclictest1915191-21kworker/u8:3+flush-8:000:21:251
1626137993529,4cyclictest1915191-21kworker/u8:3+events_unbound00:36:321
1626137993529,4cyclictest1915191-21kworker/u8:3+events_unbound00:36:321
1626137993529,4cyclictest1886278-21kworker/u8:1+flush-8:000:26:241
1626137993529,4cyclictest1480634-21kworker/u8:0+events_unbound19:36:251
1626137993528,5cyclictest1626003-21kworker/u8:1+flush-8:019:41:291
1626132993529,4cyclictest1480634-21kworker/u8:0+events_unbound20:00:000
1626132993526,6cyclictest1940669-21latency_hist00:25:010
1626138993430,2cyclictest1752210-21aten2_r5power_v21:16:162
1626138993427,5cyclictest1915999-40logrotate23:59:592
1626137993429,4cyclictest1886278-21kworker/u8:1+events_unbound00:31:241
1626137993428,4cyclictest1672700-21kworker/u8:3+flush-8:023:06:291
1626137993428,4cyclictest1480634-21kworker/u8:0+events_unbound22:06:281
1626137993428,4cyclictest1480634-21kworker/u8:0+events_unbound21:06:201
1626137993427,5cyclictest1915989-21latency_hist00:00:011
1626137993424,8cyclictest1780898-21ntpq21:41:251
1626132993430,3cyclictest1791272-21kworker/u8:1+flush-8:022:11:120
1626132993430,3cyclictest1791272-21kworker/u8:1+flush-8:022:11:110
1626132993427,5cyclictest1836180-21grep22:36:320
1626132993425,6cyclictest1950743-21latency_hist00:35:010
1626138993327,4cyclictest1876149-21journal-offline23:17:382
1626138993324,7cyclictest282-21systemd-journal21:01:422
1626138993323,8cyclictest282-21systemd-journal23:45:002
1626137993324,7cyclictest1741058-21ntpq21:01:251
1626132993327,4cyclictest1915191-21kworker/u8:3+flush-8:000:05:000
1626132993327,4cyclictest1480634-21kworker/u8:0+flush-8:020:06:280
1626138993226,4cyclictest831-21mta-sts-daemon22:09:592
1626137993226,4cyclictest1480634-21kworker/u8:0+events_unbound22:46:271
1626138993127,3cyclictest405-21dbus-daemon20:40:002
1626138993127,3cyclictest405-21dbus-daemon20:40:002
1626138993125,4cyclictest1480634-21kworker/u8:0+flush-8:020:21:342
1626138993124,5cyclictest1796545-21apt-config22:00:012
1626137993124,5cyclictest1771733-21kworker/u8:2+flush-8:021:46:281
1626132993126,4cyclictest1672700-21kworker/u8:3+flush-8:023:01:160
1626132993126,3cyclictest1830693-21needreboot22:31:220
1626132993125,4cyclictest1626003-21kworker/u8:1+flush-8:020:11:310
1626132993122,7cyclictest1746873-21cat21:10:010
1626138993027,2cyclictest1726739-21unixbench_singl20:46:322
1626132993025,2cyclictest0-21swapper/023:40:000
1626132993024,4cyclictest1637896-21turbostat.cron19:20:000
1626132993021,6cyclictest1657626-21latency_hist19:40:010
1626138992924,4cyclictest1480634-21kworker/u8:0+flush-8:019:20:002
1626138992923,4cyclictest1771733-21kworker/u8:2+flush-8:021:51:292
1626132992925,3cyclictest1480634-21kworker/u8:0+flush-8:020:26:270
1626132992925,2cyclictest1796540-21systemd-run22:00:010
1626132992923,4cyclictest1791272-21kworker/u8:1+flush-8:022:16:150
18005062280,3sleep00-21swapper/022:01:220
18005062280,3sleep00-21swapper/022:01:220
1626138992821,5cyclictest1841230-21kworker/u8:1+flush-8:022:46:442
1626137992822,4cyclictest1626003-21kworker/u8:1+events_unbound19:26:311
1626137992820,6cyclictest1662963-21apt-get19:46:101
1626132992824,2cyclictest1867727-21kthreadcore23:11:240
1626132992823,4cyclictest1886278-21kworker/u8:1+flush-8:023:51:280
1626132992823,4cyclictest1480634-21kworker/u8:0+flush-8:020:16:150
1626138992721,4cyclictest1920028-21needreboot00:01:222
1626138992721,4cyclictest1663254-21fschecks_count19:46:142
1626137992723,3cyclictest1480634-21kworker/u8:0+events_unbound19:51:221
1626137992722,4cyclictest1771733-21kworker/u8:2+flush-8:021:51:181
1626132992723,3cyclictest1886278-21kworker/u8:1+flush-8:000:26:140
1626132992723,3cyclictest1672700-21kworker/u8:3+flush-8:023:06:130
1626132992722,3cyclictest1821703-21cron22:25:010
1626132992722,3cyclictest1678712-21kthreadcore20:01:210
1626132992722,3cyclictest1638365-21aten2_r5power_v19:21:150
1626132992721,4cyclictest0-21swapper/022:45:010
1626132992721,3cyclictest0-21swapper/022:56:120
1626132992719,6cyclictest1891437-21kworker/u8:0+flush-8:023:48:350
1626138992622,3cyclictest1881293-21idleruntime-cro23:25:002
1626138992622,3cyclictest1677441-21munin-run20:00:002
1626138992621,3cyclictest1751429-21unixbench_singl21:11:312
1626138992621,3cyclictest0-21swapper/222:25:012
1626138992620,4cyclictest1681447-21needreboot20:01:252
162613799261,9cyclictest18005112kthreadcore22:01:221
162613799261,9cyclictest18005112kthreadcore22:01:221
1626132992622,3cyclictest1672700-21kworker/u8:3+flush-8:020:56:420
1626132992622,2cyclictest1842302-21tr22:46:170
1626132992621,4cyclictest1875559-21kworker/u8:3+events_unbound23:41:300
1626138992521,2cyclictest1827235-21cpuspeed_turbos22:31:132
1626138992521,2cyclictest1702953-21fschecks_count20:26:142
1626138992521,2cyclictest0-21swapper/221:40:002
1626138992521,2cyclictest0-21swapper/221:40:002
1626138992517,6cyclictest282-21systemd-journal23:30:002
162613299253,18cyclictest1657686-21kworker/0:1+events19:40:060
1626132992516,6cyclictest1825843-21ntpq22:26:250
1626138992420,2cyclictest1890810-21unixbench_multi23:31:312
1626138992420,2cyclictest1732627-21fschecks_count20:56:152
162613299245,17cyclictest1806909-21kworker/0:1+events23:59:420
162613299244,19cyclictest1657686-21kworker/0:1+events19:52:500
162613299244,18cyclictest1746857-21kworker/0:2+events21:19:180
162613299242,21cyclictest0-21swapper/020:46:300
1626132992420,2cyclictest0-21swapper/021:25:000
1626132992419,3cyclictest1791235-21unixbench_singl21:51:320
1626138992320,2cyclictest0-21swapper/220:10:012
1626138992318,4cyclictest1811439-21unixbench_multi22:11:312
1626138992318,4cyclictest1811439-21unixbench_multi22:11:312
1626138992318,3cyclictest1856965-21aten2_r5power_c23:01:142
1626138992317,4cyclictest1822134-21aten2_r5power_p22:26:132
162613299239,13cyclictest1657686-21kworker/0:1+events20:52:050
162613299234,18cyclictest1806909-21kworker/0:1+events00:36:380
162613299234,18cyclictest1806909-21kworker/0:1+events00:36:380
162613299233,19cyclictest1806909-21kworker/0:1+events00:06:180
1626132992318,4cyclictest1630653-21unixbench_multi19:11:290
1626132992318,4cyclictest1480634-21kworker/u8:0+flush-8:022:06:310
1626132992316,6cyclictest1480634-21kworker/u8:0+flush-8:020:36:130
1626132992316,6cyclictest1480634-21kworker/u8:0+flush-8:020:36:130
18331182220,8sleep3586-21kworker/3:2+mm_percpu_wq22:36:193
1626138992219,2cyclictest1832210-21fschecks_count22:36:172
1626138992218,2cyclictest0-21swapper/219:51:262
1626138992216,4cyclictest1630398-21mailstats19:11:272
1626138992213,7cyclictest1647485-21sshd19:26:422
1626132992213,7cyclictest1775948-21ntpq21:36:280
1626132992213,7cyclictest1775948-21ntpq21:36:280
162613299220,21cyclictest1806909-21kworker/0:1+events22:52:540
17674082210,3sleep3586-21kworker/3:2-mm_percpu_wq21:31:153
17674082210,3sleep3586-21kworker/3:2-mm_percpu_wq21:31:153
1626138992117,3cyclictest1946392-21cpuspeed_turbos00:31:152
1626138992117,3cyclictest1816763-21sendmail22:19:592
1626138992117,2cyclictest0-21swapper/221:25:002
1626138992116,3cyclictest1906507-21aten2_r5power_p23:51:122
1626138992115,5cyclictest1892458-21kthreadcore23:36:192
1626138992113,6cyclictest1940338-21unixbench_singl00:21:312
1626138992113,6cyclictest1766458-21unixbench_multi21:26:332
162613899202,12cyclictest1648098-21apt-get19:31:142
1626138992015,3cyclictest0-21swapper/223:00:002
1626138992014,4cyclictest1717704-21cpuspeed_turbos20:41:122
1626138992014,4cyclictest1717704-21cpuspeed_turbos20:41:122
1626138992012,6cyclictest1730676-21kthreadcore20:51:202
190713221910,5sleep30-21swapper/323:51:183
16842952190,3sleep3471rcuc/320:06:203
162613899194,13cyclictest1902223-21sed23:46:182
1626138991916,2cyclictest1767493-21fschecks_time21:31:172
1626138991916,2cyclictest1767493-21fschecks_time21:31:162
1626138991915,2cyclictest1658103-21aten2_r5power_v19:41:182
1626138991914,3cyclictest0-21swapper/222:45:012
1626138991914,3cyclictest0-21swapper/200:20:002
1626138991911,6cyclictest1861866-21apt-get23:06:092
162613899191,13cyclictest1951082-21apt-get00:36:142
162613899191,13cyclictest1951082-21apt-get00:36:142
16258492193,12sleep20-21swapper/219:08:452
162613899182,12cyclictest1941001-21apt-get00:26:112
1626138991814,2cyclictest1926528-21fschecks_time00:11:162
1626138991811,5cyclictest1782038-21acpi21:46:092
1626138991811,5cyclictest1654131-21tr19:36:182
162613899181,12cyclictest1847048-21apt-get22:51:162
1626138991810,6cyclictest1923168-21cut00:06:212
16249522183,8sleep30-21swapper/319:06:213
16481012172,9sleep30-21swapper/319:31:133
162613899178,7cyclictest1745784-21ls21:06:222
1626138991713,2cyclictest0-21swapper/223:11:262
1626138991710,5cyclictest1691524-21cut20:11:212
162613899170,8cyclictest16949001kthreadcore20:16:192
19492372160,2chrt481ktimers/300:31:223
18569372162,9sleep30-21swapper/323:01:143
162613899162,13cyclictest0-21swapper/220:32:042
1626138991613,2cyclictest0-21swapper/219:23:002
16216712163,6sleep00-21swapper/019:06:140
17721222152,8sleep30-21swapper/321:36:143
17721222152,8sleep30-21swapper/321:36:143
16282632156,5sleep30-21swapper/319:11:193
1626138991510,3cyclictest0-21swapper/221:41:132
19409752133,6sleep30-21swapper/300:26:093
19163512133,6sleep30-21swapper/300:01:133
17901762133,7sleep30-21swapper/321:51:233
17434642132,7sleep30-21swapper/321:06:203
16974412132,6sleep30-21swapper/320:20:013
16678612136,4sleep30-21swapper/319:51:123
16617652132,7sleep30-21swapper/319:41:293
19511972122,6sleep30-21swapper/300:36:163
19511972122,6sleep30-21swapper/300:36:163
19360452123,5sleep30-21swapper/300:21:123
19309242122,5sleep30-21swapper/300:15:013
19271252122,6sleep30-21swapper/300:11:203
19217062122,6sleep30-21swapper/300:06:193
19113872122,6sleep30-21swapper/323:56:113
19057462122,6sleep30-21swapper/323:46:323
18914722122,6sleep30-21swapper/323:35:453
18909172122,6sleep30-21swapper/323:31:393
18824742122,6sleep30-21swapper/323:26:213
18767062123,6sleep30-21swapper/323:21:123
18706412122,6sleep30-21swapper/323:11:333
18565622122,5sleep30-21swapper/323:00:013
18514962122,6sleep30-21swapper/322:53:193
18421552122,6sleep30-21swapper/322:46:173
18370342123,6sleep30-21swapper/322:41:153
18282252122,6sleep30-21swapper/322:31:203
18215062122,6sleep30-21swapper/322:22:333
18124022122,6sleep30-21swapper/322:16:153
17970832122,6sleep30-21swapper/322:01:143
17970832122,6sleep30-21swapper/322:01:143
17923812122,7sleep30-21swapper/321:56:213
17521012122,6sleep30-21swapper/321:16:163
17468352122,5sleep30-21swapper/321:10:013
17372232122,6sleep30-21swapper/321:00:423
17327142122,6sleep30-21swapper/320:56:173
17274052122,6sleep30-21swapper/320:51:113
17129062122,6sleep30-21swapper/320:36:193
17129062122,6sleep30-21swapper/320:36:183
17065082122,6sleep30-21swapper/320:26:273
16947922121,7sleep30-21swapper/320:16:203
16877982122,6sleep30-21swapper/320:10:423
16781622122,6sleep30-21swapper/320:01:193
16732872122,6sleep30-21swapper/319:56:183
16335152122,6sleep30-21swapper/319:16:173
19002552112,5sleep30-21swapper/323:41:273
18716962112,5sleep30-21swapper/323:16:103
18225582112,6sleep30-21swapper/322:26:183
18069052112,4sleep30-21swapper/322:10:013
18069052112,4sleep30-21swapper/322:10:013
18026132112,5sleep30-21swapper/322:06:163
17817292112,5sleep30-21swapper/321:45:013
17784332112,5sleep30-21swapper/321:41:193
17620372112,5sleep30-21swapper/321:25:023
17609162112,5sleep30-21swapper/321:21:293
17222182112,5sleep30-21swapper/320:45:013
17076032112,5sleep30-21swapper/320:31:133
16629002112,5sleep30-21swapper/319:45:413
16530012112,6sleep30-21swapper/319:35:513
16436542112,5sleep30-21swapper/319:26:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional