You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 21:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Thu Nov 20, 2025 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1712673993935,3cyclictest1837521-21kworker/u8:3+flush-259:009:45:212
1712673993935,3cyclictest1837521-21kworker/u8:3+flush-259:009:45:202
1712669993933,4cyclictest2698871-21systemd-journal10:40:001
1712669993831,5cyclictest1791010-21latency_hist08:25:011
1712677993733,3cyclictest1780749-21kworker/u8:0+flush-259:009:25:003
1712673993734,2cyclictest1654334-21kworker/u8:3+flush-259:007:40:192
1712677993327,4cyclictest2698871-21systemd-journal11:24:593
1712673993229,2cyclictest1682306-21kworker/u8:4+flush-259:007:30:242
1712673993228,3cyclictest2009341-21kworker/u8:1+flush-259:012:05:212
1712673993228,3cyclictest1728477-21kworker/u8:0+flush-259:007:50:152
1712673993226,4cyclictest1743891-21latency_hist07:40:002
1712673993026,3cyclictest1946304-21kworker/u8:1+flush-259:011:05:202
1712673992922,5cyclictest1822030-21latency_hist08:55:012
1712666992926,2cyclictest1883580-21kworker/u8:0+flush-259:010:00:000
1712673992824,3cyclictest1993502-21kworker/u8:3+events_unbound12:15:012
1712673992824,3cyclictest1780749-21kworker/u8:0+events_unbound09:10:132
1712673992822,4cyclictest1925101-21ntpq10:30:222
1712669992824,2cyclictest1837521-21kworker/u8:3+flush-259:009:50:181
1712669992824,2cyclictest1837521-21kworker/u8:3+flush-259:009:50:171
1712673992724,2cyclictest1827437-21kworker/u8:1+flush-259:009:30:202
1712669992723,3cyclictest1926541-21kworker/u8:0+flush-259:010:55:001
1712669992722,3cyclictest1827437-21kworker/u8:1+flush-259:009:30:001
1712666992723,3cyclictest1858158-21latency_hist09:30:000
1712673992622,3cyclictest1883580-21kworker/u8:0+events_unbound09:55:242
1712673992622,3cyclictest1827437-21kworker/u8:1+events_unbound09:25:182
1712669992622,3cyclictest1946304-21kworker/u8:1+flush-259:011:10:151
1712666992621,3cyclictest2045417-21/usr/sbin/munin12:30:150
1712673992522,2cyclictest2019432-21kworker/u8:0+flush-259:012:10:242
1712673992521,3cyclictest1926541-21kworker/u8:0+events_unbound10:55:192
1712673992519,4cyclictest2004210-21/usr/sbin/munin11:50:162
1712673992519,4cyclictest1832351-21latency_hist09:05:012
1712669992519,4cyclictest1780749-21kworker/u8:0+flush-259:008:30:171
1712666992522,2cyclictest1780749-21kworker/u8:0+flush-259:009:20:240
1712673992421,2cyclictest1998719-21kworker/u8:2+events_unbound11:45:172
1712673992421,2cyclictest1654334-21kworker/u8:3+flush-259:007:25:002
1712673992420,2cyclictest1837521-21kworker/u8:3+events_unbound09:39:392
1712669992419,3cyclictest1905153-21cp10:15:071
1712677992319,3cyclictest1915656-21gpgv10:25:013
1712673992320,2cyclictest1837521-21kworker/u8:3+events_unbound10:25:232
1712669992319,3cyclictest2019432-21kworker/u8:0+events_unbound12:05:011
1712669992318,4cyclictest3000063-21rs:main1
1712666992319,3cyclictest2698871-21systemd-journal09:45:280
1712666992319,3cyclictest2698871-21systemd-journal09:45:270
1712673992219,2cyclictest1951524-21kworker/u8:3+events_unbound11:13:412
1712673992219,2cyclictest1780749-21kworker/u8:0+flush-259:008:30:222
1712673992219,2cyclictest1763990-21kworker/u8:1+flush-259:008:40:182
1712673992218,3cyclictest2019432-21kworker/u8:0+flush-259:012:30:212
1712673992218,3cyclictest1926541-21kworker/u8:0+flush-259:010:35:182
1712669992217,4cyclictest1972683-21apt-get11:20:001
1712677992117,3cyclictest2698871-21systemd-journal07:50:003
1712677992116,4cyclictest1893339-21ntpq10:00:203
1712673992119,1cyclictest1888750-21kworker/u8:2+flush-259:010:00:152
1712673992118,2cyclictest1743909-21kworker/u8:1+events_unbound07:45:232
1712673992117,3cyclictest1993502-21kworker/u8:3+flush-259:012:04:592
1712673992116,3cyclictest2698871-21systemd-journal08:00:362
1712669992117,3cyclictest1993502-21kworker/u8:3+flush-259:012:35:231
1712669992117,3cyclictest1904561-21kworker/u8:1+flush-259:010:30:251
1712669992117,3cyclictest1837521-21kworker/u8:3+events_unbound09:35:221
1712669992117,3cyclictest1780749-21kworker/u8:0+flush-259:008:50:181
1712669992117,3cyclictest1770137-21base6408:05:011
1712669992116,4cyclictest1654334-21kworker/u8:3+flush-259:007:55:141
1712669992116,3cyclictest1780997-21apt-get08:15:151
1712666992118,2cyclictest2019432-21kworker/u8:0+flush-259:012:10:160
1712666992116,4cyclictest2039105-21ntpq12:20:200
1712677992018,1cyclictest0-21swapper/310:35:003
1712677992014,4cyclictest2003968-21latency_hist11:50:013
1712673992017,2cyclictest1946304-21kworker/u8:1+events_unbound11:35:222
1712673992017,2cyclictest1888750-21kworker/u8:2+events_unbound10:07:402
1712673992017,2cyclictest1654334-21kworker/u8:3+events_unbound07:15:172
1712673992016,3cyclictest1957302-21kworker/u8:0+flush-259:011:40:172
1712673992016,3cyclictest1926541-21kworker/u8:0+events_unbound10:50:192
1712673992016,3cyclictest1837480-21sh09:09:592
1712673992014,4cyclictest2698871-21systemd-journal08:39:372
1712673992013,5cyclictest1940567-21ntpq10:45:212
1712669992016,3cyclictest2040277-21apt-get12:25:091
1712669992016,3cyclictest1821904-21kworker/u8:2+events_unbound09:00:151
1712666992017,2cyclictest1780749-21kworker/u8:0+flush-259:008:50:170
1712677991916,2cyclictest1837521-21kworker/u8:3+flush-259:009:15:253
1712673991916,2cyclictest1957302-21kworker/u8:0+flush-259:011:20:242
1712673991916,2cyclictest1753682-21kworker/u8:2+events_unbound08:18:372
1712673991915,3cyclictest1946304-21kworker/u8:1+events_unbound11:25:172
1712673991915,3cyclictest1712874-21apt-get07:10:122
1712673991915,3cyclictest1712874-21apt-get07:10:122
1712673991915,3cyclictest1682306-21kworker/u8:4+events_unbound07:25:212
1712673991915,3cyclictest1682306-21kworker/u8:4+events_unbound07:25:202
1712669991916,2cyclictest1654334-21kworker/u8:3+flush-259:007:15:161
1712669991915,3cyclictest1888750-21kworker/u8:2+flush-259:010:05:211
1712669991915,3cyclictest1827437-21kworker/u8:1+flush-259:009:20:301
1712669991915,3cyclictest1780749-21kworker/u8:0+events_unbound09:05:131
1712669991915,2cyclictest1946304-21kworker/u8:1+flush-259:011:30:201
1712669991913,4cyclictest1733621-21latency_hist07:30:001
1712669991913,4cyclictest1733621-21latency_hist07:30:001
1712669991912,5cyclictest1764725-21/usr/sbin/munin08:00:141
1712666991916,2cyclictest1837521-21kworker/u8:3+flush-259:009:35:210
1712666991915,3cyclictest1780749-21kworker/u8:0+flush-259:008:20:260
1712666991912,5cyclictest1769806-21cp08:05:010
1712677991816,1cyclictest0-21swapper/311:00:003
1712677991815,2cyclictest1780749-21kworker/u8:0+flush-259:009:35:003
1712677991814,3cyclictest1993502-21kworker/u8:3+flush-259:012:20:163
1712673991815,2cyclictest1998719-21kworker/u8:2+flush-259:011:55:262
1712673991815,2cyclictest1926541-21kworker/u8:0+flush-259:010:40:202
1712673991814,3cyclictest1654334-21kworker/u8:3+events_unbound08:05:042
1712673991813,4cyclictest1827437-21kworker/u8:1+events_unbound09:20:142
1712669991815,2cyclictest1993502-21kworker/u8:3+flush-259:012:15:171
1712669991814,3cyclictest2029742-21kworker/u8:2+flush-259:012:25:001
1712669991814,3cyclictest1888750-21kworker/u8:2+events_unbound10:00:131
1712666991815,1cyclictest0-21swapper/009:09:590
1712666991813,4cyclictest1801287-21latency_hist08:35:010
1712677991714,2cyclictest3000061-21in:imuxsock12:30:003
1712677991714,2cyclictest1682306-21kworker/u8:4+flush-259:007:30:243
1712677991713,3cyclictest2024605-21latency_hist12:10:013
1712673991713,3cyclictest1654334-21kworker/u8:3+flush-259:007:55:192
1712673991712,3cyclictest1888750-21kworker/u8:2+flush-259:010:10:152
1712669991714,2cyclictest1888750-21kworker/u8:2+flush-259:010:45:231
1712669991713,3cyclictest2698871-21systemd-journal11:58:391
1712669991713,3cyclictest1837521-21kworker/u8:3+flush-259:009:40:171
1712669991713,3cyclictest1654334-21kworker/u8:3+events_unbound07:45:151
1712669991712,4cyclictest1957302-21kworker/u8:0+events_unbound11:40:141
1712666991715,1cyclictest2055476-21idleruntime-cro12:39:590
1712666991715,1cyclictest0-21swapper/011:34:590
1712666991712,4cyclictest1770476-21diskstats08:05:130
1712677991614,1cyclictest0-21swapper/312:34:593
1712677991613,2cyclictest1994104-21fschecks_count11:40:143
1712677991613,2cyclictest1888750-21kworker/u8:2+events_unbound10:10:213
1712677991613,2cyclictest1837521-21kworker/u8:3+flush-259:010:45:003
1712673991613,2cyclictest1993502-21kworker/u8:3+events_unbound12:35:162
171266999169,5cyclictest1952318-21cat11:00:021
1712669991612,3cyclictest1888750-21kworker/u8:2+flush-259:010:10:221
1712669991611,4cyclictest1926541-21kworker/u8:0+events_unbound10:55:181
1712669991610,4cyclictest2698871-21systemd-journal09:17:381
1712666991614,1cyclictest60650irq/133-enp0s29f1:rx-011:27:210
1712666991614,1cyclictest60650irq/133-enp0s29f1:rx-010:44:380
1712666991614,1cyclictest60650irq/133-enp0s29f1:rx-010:34:200
1712666991614,1cyclictest60650irq/133-enp0s29f1:rx-008:35:250
1712666991613,2cyclictest60650irq/133-enp0s29f1:rx-008:59:430
1712666991613,2cyclictest60650irq/133-enp0s29f1:rx-008:15:240
1712666991612,3cyclictest1682306-21kworker/u8:4+flush-259:007:35:200
1712677991513,1cyclictest0-21swapper/308:10:003
1712677991513,1cyclictest0-21swapper/307:35:193
1712677991512,2cyclictest1946304-21kworker/u8:1+flush-259:011:05:163
1712677991511,3cyclictest1972899-21gpgv11:20:013
1712677991511,3cyclictest1753682-21kworker/u8:2+flush-259:008:15:003
1712677991510,3cyclictest2698871-21systemd-journal11:15:003
1712673991512,2cyclictest1845239-21kthreadcore09:15:172
1712673991512,2cyclictest1763990-21kworker/u8:1+flush-259:008:10:162
1712673991511,3cyclictest1946304-21kworker/u8:1+events_unbound11:15:142
1712673991511,3cyclictest1801312-21kworker/u8:3+events_unbound08:48:562
1712669991512,2cyclictest1946304-21kworker/u8:1+events_unbound11:20:151
1712669991512,2cyclictest1936286-21idleruntime-cro10:45:001
1712669991512,2cyclictest1682306-21kworker/u8:4+events_unbound07:20:131
1712669991511,3cyclictest2019432-21kworker/u8:0+flush-259:012:10:241
1712669991511,3cyclictest1998719-21kworker/u8:2+events_unbound12:00:181
1712666991513,1cyclictest60650irq/133-enp0s29f1:rx-012:06:540
1712666991513,1cyclictest60650irq/133-enp0s29f1:rx-011:47:060
1712666991513,1cyclictest60650irq/133-enp0s29f1:rx-011:12:050
1712666991513,1cyclictest60650irq/133-enp0s29f1:rx-010:35:190
1712666991513,1cyclictest60650irq/133-enp0s29f1:rx-010:00:220
1712666991513,1cyclictest60650irq/133-enp0s29f1:rx-009:31:160
1712666991513,1cyclictest60650irq/133-enp0s29f1:rx-009:01:570
1712666991513,1cyclictest60650irq/133-enp0s29f1:rx-008:47:080
1712666991513,1cyclictest60650irq/133-enp0s29f1:rx-007:30:150
1712666991512,2cyclictest61450irq/134-enp0s29f1:tx-011:15:120
1712666991512,2cyclictest60650irq/133-enp0s29f1:rx-011:05:200
1712666991512,2cyclictest1718875-21irqcore07:15:190
1712666991511,3cyclictest1842949-21apt-get09:15:110
1712666991510,4cyclictest1910319-21/usr/sbin/munin10:20:190
1712677991411,2cyclictest1873914-21cpuspeed_turbos09:45:163
1712677991411,2cyclictest1873914-21cpuspeed_turbos09:45:163
1712677991411,2cyclictest1744232-21cpuspeed_turbos07:40:153
1712677991410,3cyclictest1905175-21/usr/sbin/munin10:15:223
1712677991410,3cyclictest1837521-21kworker/u8:3+flush-259:010:25:163
171267399147,5cyclictest2698871-21systemd-journal11:03:402
1712673991410,3cyclictest1905000-21gpgv10:15:052
1712673991410,3cyclictest1837521-21kworker/u8:3+flush-259:010:20:192
171266999148,4cyclictest1983407-21latency_hist11:30:011
1712669991412,1cyclictest1827213-21grep09:00:001
1712669991411,2cyclictest1763990-21kworker/u8:1+flush-259:008:30:001
1712669991410,3cyclictest1763990-21kworker/u8:1+flush-259:008:35:201
171266699148,4cyclictest1795254-21ntpq08:25:170
1712666991413,1cyclictest60650irq/133-enp0s29f1:rx-011:56:010
1712666991413,1cyclictest60650irq/133-enp0s29f1:rx-010:10:170
1712666991413,1cyclictest60650irq/133-enp0s29f1:rx-007:59:220
1712666991413,1cyclictest60650irq/133-enp0s29f1:rx-007:21:070
1712666991413,1cyclictest60650irq/133-enp0s29f1:rx-007:11:450
1712666991413,1cyclictest60650irq/133-enp0s29f1:rx-007:11:450
1712666991412,1cyclictest60650irq/133-enp0s29f1:rx-011:54:510
1712666991412,1cyclictest60650irq/133-enp0s29f1:rx-009:42:510
1712666991412,1cyclictest60650irq/133-enp0s29f1:rx-009:12:150
1712666991412,1cyclictest60650irq/133-enp0s29f1:rx-008:40:140
1712666991412,1cyclictest60650irq/133-enp0s29f1:rx-007:53:480
1712666991412,1cyclictest60650irq/133-enp0s29f1:rx-007:48:150
1712666991412,1cyclictest60650irq/133-enp0s29f1:rx-007:25:220
1712666991412,1cyclictest60650irq/133-enp0s29f1:rx-007:25:220
1712666991412,1cyclictest3000063-21rs:main0
1712666991411,2cyclictest60650irq/133-enp0s29f1:rx-007:40:170
1712666991411,2cyclictest1998833-21sh11:45:000
1712677991311,1cyclictest1868440-21sh09:39:593
1712677991310,1cyclictest0-21swapper/309:50:173
1712677991310,1cyclictest0-21swapper/309:50:173
1712677991310,1cyclictest0-21swapper/309:14:593
171267399139,3cyclictest1780749-21kworker/u8:0+events_unbound08:55:262
1712673991310,2cyclictest1993502-21kworker/u8:3+events_unbound12:20:172
1712673991310,2cyclictest1873156-21unixbench_singl09:40:272
171266999139,3cyclictest1837521-21kworker/u8:3+events_unbound10:20:161
171266999139,3cyclictest1753682-21kworker/u8:2+flush-259:008:10:241
171266999139,3cyclictest1682306-21kworker/u8:4+flush-259:007:30:251
171266999138,3cyclictest1780749-21kworker/u8:0+events_unbound08:45:201
1712669991311,1cyclictest0-21swapper/107:15:001
1712669991311,1cyclictest0-21swapper/107:15:001
1712666991312,1cyclictest60650irq/133-enp0s29f1:rx-011:02:220
1712666991312,1cyclictest60650irq/133-enp0s29f1:rx-010:27:070
1712666991312,1cyclictest60650irq/133-enp0s29f1:rx-010:08:170
1712666991311,1cyclictest60650irq/133-enp0s29f1:rx-012:17:030
1712666991311,1cyclictest60650irq/133-enp0s29f1:rx-009:54:000
1712666991311,1cyclictest60650irq/133-enp0s29f1:rx-009:53:590
1712666991310,2cyclictest1780321-21unixbench_singl08:10:250
171267799129,2cyclictest1983677-21apt-get11:30:153
171267799129,2cyclictest1929737-21kthreadcore10:35:193
171267799128,3cyclictest1801312-21kworker/u8:3+flush-259:008:50:003
1712677991210,1cyclictest2027146-21kthreadcore12:10:183
171267799120,5cyclictest18960731kthreadcore10:05:163
171267399127,1cyclictest0-21swapper/209:55:002
171267399127,1cyclictest0-21swapper/209:55:002
171266999128,3cyclictest1998719-21kworker/u8:2+flush-259:011:45:251
171266999127,3cyclictest2019432-21kworker/u8:0+flush-259:012:30:201
171266699129,2cyclictest60650irq/133-enp0s29f1:rx-010:17:280
1712666991211,1cyclictest60650irq/133-enp0s29f1:rx-012:00:130
171267799116,3cyclictest1785496-21sshd08:15:363
171266999118,2cyclictest0-21swapper/111:05:111
171266999117,3cyclictest1827437-21kworker/u8:1+events_unbound09:30:191
171266699119,2cyclictest60650irq/133-enp0s29f1:rx-011:20:150
171267799108,1cyclictest2054800-21switchtime12:35:253
171267799108,1cyclictest0-21swapper/311:50:173
171267799108,1cyclictest0-21swapper/311:35:233
171267799108,1cyclictest0-21swapper/309:40:293
171267799108,1cyclictest0-21swapper/309:05:103
171267799107,2cyclictest1654334-21kworker/u8:3+flush-259:008:00:163
171266999107,2cyclictest1989011-21fschecks_count11:35:141
17126779997,1cyclictest0-21swapper/309:30:003
17126779997,1cyclictest0-21swapper/308:35:163
17126779997,1cyclictest0-21swapper/308:20:163
17126779995,3cyclictest1821225-21smartctl08:50:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional