You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-08 - 09:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Sat Nov 08, 2025 00:44:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1520753994238,3cyclictest1655377-21kworker/u8:1+flush-259:021:35:011
1520749993630,4cyclictest1706758-21/usr/sbin/munin22:10:160
1520759993227,3cyclictest1747906-21/usr/sbin/munin22:50:163
1520749993225,5cyclictest1747665-21latency_hist22:50:000
1520753993127,3cyclictest1571380-21kworker/u8:2+flush-259:020:00:251
1520749992923,4cyclictest1655377-21kworker/u8:1+events_unbound21:20:190
1520759992824,2cyclictest1546535-21kworker/u8:0+flush-259:019:50:173
1520753992823,3cyclictest2698871-21systemd-journal22:46:271
1520753992823,3cyclictest1758201-21kworker/u8:1+flush-259:023:25:001
1520753992720,5cyclictest1835764-21latency_hist00:15:011
1520753992622,3cyclictest1571380-21kworker/u8:2+flush-259:020:00:011
1520749992620,4cyclictest2698871-21systemd-journal21:51:270
1520749992620,4cyclictest2698871-21systemd-journal21:51:260
1520749992619,5cyclictest1597233-21ntpq20:20:210
1520755992518,5cyclictest1768659-21cp23:10:022
1520749992518,5cyclictest1572315-21latency_hist20:00:000
1520755992421,2cyclictest2631560-21kworker/u8:0+events_unbound19:20:002
1520755992418,5cyclictest1643982-21ntpq21:05:192
1520753992420,3cyclictest1753094-21apt-get22:55:121
1520749992419,3cyclictest1629053-21kworker/u8:1+flush-259:021:05:090
1520759992320,2cyclictest1571380-21kworker/u8:2+events_unbound20:50:153
1520759992318,4cyclictest1732482-21/usr/sbin/munin22:35:123
1520755992320,2cyclictest1546535-21kworker/u8:0+events_unbound19:55:002
1520755992320,2cyclictest1510687-21kworker/u8:1+flush-259:019:30:182
1520755992320,2cyclictest1510687-21kworker/u8:1+flush-259:019:30:182
1520749992318,4cyclictest1675720-21kworker/u8:0+events_unbound21:45:170
1520749992318,4cyclictest1675720-21kworker/u8:0+events_unbound21:45:170
1520749992315,6cyclictest2698871-21systemd-journal23:00:010
1520759992218,3cyclictest1810094-21kworker/u8:4+flush-259:000:00:163
1520755992218,3cyclictest1665440-21kworker/u8:3+flush-259:021:40:002
1520755992216,4cyclictest1649972-21latency_hist21:15:002
1520749992217,3cyclictest1737411-21kworker/u8:4+events_unbound23:00:220
1520749992215,5cyclictest1530177-21ntpq19:15:240
1520759992118,2cyclictest1804952-21kworker/u8:3+events_unbound00:20:153
1520759992116,3cyclictest1742565-21awk22:45:003
1520755992118,2cyclictest1567424-21/usr/sbin/munin19:55:162
1520755992117,3cyclictest1571380-21kworker/u8:2+events_unbound21:55:232
1520753992114,5cyclictest1789252-21latency_hist23:30:001
1520749992118,2cyclictest1737411-21kworker/u8:4+events_unbound22:51:270
1520749992118,2cyclictest1603335-21kworker/u8:0+events_unbound20:35:240
1520749992117,3cyclictest1758201-21kworker/u8:1+events_unbound23:37:270
1520749992117,3cyclictest1758201-21kworker/u8:1+events_unbound23:15:150
1520749992117,3cyclictest1722196-21kworker/u8:3+events_unbound22:37:360
1520749992117,3cyclictest1571380-21kworker/u8:2+events_unbound20:50:020
1520749992115,4cyclictest1804952-21kworker/u8:3+events_unbound00:05:120
1520749992115,4cyclictest1804952-21kworker/u8:3+events_unbound00:05:110
1520749992114,5cyclictest1841334-21latency_hist00:20:010
1520759992017,2cyclictest1737411-21kworker/u8:4+flush-259:023:15:013
1520759992016,3cyclictest1804952-21kworker/u8:3+events_unbound00:30:023
1520759992016,3cyclictest1571380-21kworker/u8:2+events_unbound20:00:253
1520755992017,2cyclictest1675720-21kworker/u8:0+flush-259:022:45:282
1520755992016,3cyclictest2698871-21systemd-journal22:35:012
1520753992017,2cyclictest0-21swapper/100:00:001
1520749992017,2cyclictest1588037-21kworker/u8:4+flush-259:020:40:150
1520749992015,3cyclictest1768274-21sh23:10:010
1520759991916,2cyclictest1649997-21kworker/u8:0+flush-259:021:20:223
1520759991916,2cyclictest1588037-21kworker/u8:4+events_unbound20:40:253
1520759991916,2cyclictest1571380-21kworker/u8:2+flush-259:021:15:183
1520759991916,2cyclictest1546505-21idleruntime-cro19:35:013
1520759991916,2cyclictest1546505-21idleruntime-cro19:35:003
1520759991915,3cyclictest1706242-21kworker/u8:1+flush-259:022:09:573
1520759991914,4cyclictest1829731-21ntpq00:05:213
1520759991914,4cyclictest1829731-21ntpq00:05:213
1520759991914,3cyclictest3000063-21rs:main3
1520755991917,1cyclictest1634122-21turbostat.cron20:59:592
1520755991916,2cyclictest1758201-21kworker/u8:1+flush-259:023:00:152
1520755991916,2cyclictest0-21swapper/222:20:012
1520753991916,2cyclictest1542012-21irqcore19:30:171
1520753991916,2cyclictest1542012-21irqcore19:30:161
1520753991915,3cyclictest1804952-21kworker/u8:3+flush-259:000:24:591
1520753991915,3cyclictest1758201-21kworker/u8:1+flush-259:023:40:001
1520749991915,3cyclictest1804952-21kworker/u8:3+events_unbound00:25:170
1520749991915,3cyclictest1758201-21kworker/u8:1+flush-259:023:55:140
1520749991915,3cyclictest1571380-21kworker/u8:2+events_unbound20:57:260
1520749991914,4cyclictest1571380-21kworker/u8:2+events_unbound21:55:200
1520749991914,3cyclictest1758201-21kworker/u8:1+flush-259:023:10:160
1520749991912,3cyclictest1804952-21kworker/u8:3+events_unbound00:15:000
1520759991815,3cyclictest1629053-21kworker/u8:1+events_unbound21:05:133
1520759991815,2cyclictest3000061-21in:imuxsock22:15:003
1520759991814,3cyclictest1836429-21/usr/sbin/munin00:15:163
1520759991814,3cyclictest1778999-21kworker/u8:2+events_unbound23:25:253
1520759991814,3cyclictest1551890-21kworker/u8:4+events_unbound19:55:223
1520759991812,5cyclictest1820064-21apt-get23:58:273
1520755991814,3cyclictest1804952-21kworker/u8:3+flush-259:000:35:222
1520755991813,4cyclictest1577505-21latency_hist20:05:002
1520753991814,3cyclictest1810094-21kworker/u8:4+flush-259:023:50:271
1520753991812,5cyclictest1520944-21/usr/sbin/munin19:10:161
1520749991815,2cyclictest1711470-21kworker/u8:2+flush-259:022:15:180
1520749991815,2cyclictest1571380-21kworker/u8:2+flush-259:021:25:210
1520749991813,3cyclictest1571380-21kworker/u8:2+events_unbound20:10:170
1520759991715,1cyclictest1675720-21kworker/u8:0+events_unbound23:50:003
1520759991714,2cyclictest1588037-21kworker/u8:4+events_unbound20:30:253
1520759991714,2cyclictest1571380-21kworker/u8:2+flush-259:021:25:193
1520759991713,3cyclictest1562035-21turbostat19:50:013
1520755991715,1cyclictest0-21swapper/200:15:002
1520755991714,2cyclictest1835723-21kworker/u8:0+flush-259:000:15:182
1520755991714,2cyclictest1571380-21kworker/u8:2+flush-259:021:50:012
1520755991714,2cyclictest1571380-21kworker/u8:2+flush-259:021:50:002
1520755991714,2cyclictest1571380-21kworker/u8:2+flush-259:020:20:002
1520755991713,3cyclictest1675720-21kworker/u8:0+flush-259:022:15:002
1520753991713,3cyclictest1571380-21kworker/u8:2+flush-259:021:50:171
1520753991713,3cyclictest1571380-21kworker/u8:2+flush-259:021:50:171
1520749991713,3cyclictest1810094-21kworker/u8:4+events_unbound00:00:180
1520749991712,4cyclictest1706242-21kworker/u8:1+events_unbound22:20:270
1520749991712,4cyclictest1670822-21/usr/sbin/munin21:35:170
1520749991712,3cyclictest1623875-21turbostat20:50:010
1520759991613,2cyclictest1510687-21kworker/u8:1+flush-259:019:20:273
1520759991613,2cyclictest1510687-21kworker/u8:1+flush-259:019:10:153
1520759991612,3cyclictest1546752-21/usr/sbin/munin19:35:143
1520755991613,2cyclictest503-21systemd-logind00:35:002
1520755991612,3cyclictest1510687-21kworker/u8:1+events_unbound19:25:142
1520753991610,4cyclictest1598409-21apt-get20:25:111
1520749991613,2cyclictest61450irq/134-enp0s29f1:tx-019:50:180
1520749991613,2cyclictest60650irq/133-enp0s29f1:rx-020:30:130
1520749991613,2cyclictest60650irq/133-enp0s29f1:rx-020:05:240
1520749991613,2cyclictest60650irq/133-enp0s29f1:rx-019:20:170
1520749991612,3cyclictest60650irq/133-enp0s29f1:rx-019:25:170
1520749991612,3cyclictest1629053-21kworker/u8:1+events_unbound21:05:000
1520749991612,3cyclictest1571380-21kworker/u8:2+events_unbound21:15:250
1520749991612,3cyclictest1551890-21kworker/u8:4+flush-259:019:45:150
1520749991611,3cyclictest1732256-21turbostat22:35:000
1520749991611,3cyclictest1588047-21/usr/sbin/munin20:15:240
1520749991611,3cyclictest1525913-21turbostat19:15:000
1520749991610,4cyclictest2698871-21systemd-journal22:28:260
1520759991513,2cyclictest1758201-21kworker/u8:1+flush-259:023:05:153
1520759991511,3cyclictest1804952-21kworker/u8:3+flush-259:000:36:573
152075599159,4cyclictest1757975-21latency_hist23:00:012
1520755991512,2cyclictest3002-21cron23:20:002
1520755991512,2cyclictest1547190-21irqcore19:35:152
1520755991511,2cyclictest0-21swapper/200:10:002
1520755991511,2cyclictest0-21swapper/200:10:002
1520755991510,4cyclictest1799561-21latency_hist23:40:012
1520753991512,2cyclictest1623866-21sh20:50:001
1520753991510,4cyclictest1830591-21latency_hist00:10:011
1520753991510,4cyclictest1830591-21latency_hist00:10:011
1520749991514,1cyclictest60650irq/133-enp0s29f1:rx-000:36:110
1520749991513,2cyclictest61450irq/134-enp0s29f1:tx-022:40:150
1520749991513,1cyclictest60650irq/133-enp0s29f1:rx-023:50:190
1520749991513,1cyclictest60650irq/133-enp0s29f1:rx-022:02:080
1520749991512,2cyclictest60650irq/133-enp0s29f1:rx-020:00:310
1520759991412,2cyclictest1675720-21kworker/u8:0+flush-259:021:55:263
1520759991412,1cyclictest1722196-21kworker/u8:3+events_unbound22:25:133
1520759991412,1cyclictest0-21swapper/320:40:003
152075999141,12cyclictest492-21dbus-daemon20:30:013
1520759991411,2cyclictest1571380-21kworker/u8:2+flush-259:021:00:213
1520755991412,1cyclictest0-21swapper/220:50:002
1520755991411,2cyclictest1698815-21kthreadcore22:00:192
1520755991410,3cyclictest1655377-21kworker/u8:1+flush-259:021:30:192
1520755991410,3cyclictest1551927-21apt-get19:40:152
1520755991410,2cyclictest1-21systemd21:55:012
1520755991410,2cyclictest1-21systemd21:55:002
152075399149,4cyclictest1571380-21kworker/u8:2+flush-259:020:20:271
152075399149,3cyclictest2698871-21systemd-journal22:14:261
152075399147,5cyclictest1793575-21ntpq23:30:231
152075399147,5cyclictest1768282-21latency_hist23:10:011
1520753991412,1cyclictest0-21swapper/123:15:011
152074999147,5cyclictest1669684-21ntpq21:30:220
1520749991412,2cyclictest60650irq/133-enp0s29f1:rx-019:37:120
1520749991412,1cyclictest60650irq/133-enp0s29f1:rx-023:40:560
1520749991412,1cyclictest60650irq/133-enp0s29f1:rx-022:05:180
1520749991412,1cyclictest60650irq/133-enp0s29f1:rx-021:10:020
1520749991412,1cyclictest60650irq/133-enp0s29f1:rx-019:30:210
1520749991412,1cyclictest60650irq/133-enp0s29f1:rx-019:30:200
1520749991411,2cyclictest60650irq/133-enp0s29f1:rx-019:44:240
1520749991411,2cyclictest60650irq/133-enp0s29f1:rx-000:21:330
1520749991410,3cyclictest1571380-21kworker/u8:2+flush-259:021:40:160
152074999140,12cyclictest1784126-21turbostat23:25:000
152075999139,3cyclictest1804952-21kworker/u8:3+events_unbound00:25:183
1520759991310,2cyclictest1799582-21kworker/u8:2+events_unbound23:45:003
1520759991310,2cyclictest1783664-21unixbench_multi23:20:253
1520759991310,2cyclictest1753354-21fschecks_time22:55:143
1520759991310,2cyclictest1683408-21kthreadcore21:45:163
1520759991310,2cyclictest1683408-21kthreadcore21:45:163
1520759991310,2cyclictest1665440-21kworker/u8:3+events_unbound21:40:143
1520759991310,2cyclictest1551890-21kworker/u8:4+flush-259:019:40:203
152075999130,4cyclictest1712876-21sed22:15:163
152075599139,3cyclictest1758201-21kworker/u8:1+events_unbound23:05:182
152075599138,3cyclictest0-21swapper/220:45:002
1520755991310,3cyclictest1675720-21kworker/u8:0+flush-259:022:35:202
1520755991310,2cyclictest1824444-21needreboot00:00:182
1520755991310,2cyclictest1804952-21kworker/u8:3+flush-259:000:20:192
1520755991310,2cyclictest1799582-21kworker/u8:2+flush-259:023:45:152
1520755991310,2cyclictest1722196-21kworker/u8:3+events_unbound22:40:162
1520755991310,2cyclictest1675720-21kworker/u8:0+flush-259:022:55:002
1520755991310,2cyclictest1588037-21kworker/u8:4+flush-259:020:20:192
1520755991310,2cyclictest1562019-21idleruntime-cro19:50:002
152075399139,3cyclictest2698871-21systemd-journal19:41:271
152075399139,3cyclictest1629053-21kworker/u8:1+flush-259:021:05:221
152075399139,3cyclictest1603335-21kworker/u8:0+flush-259:020:40:011
152075399139,2cyclictest1533611-21tr19:20:191
152075399138,3cyclictest3000063-21rs:main1
152075399136,5cyclictest1571380-21kworker/u8:2+flush-259:021:35:191
1520753991310,2cyclictest1752404-21unixbench_singl22:50:261
1520753991310,2cyclictest1657700-21kthreadcore21:20:181
1520753991310,2cyclictest1624197-21cpuspeed_turbos20:50:121
1520749991311,2cyclictest60650irq/133-enp0s29f1:rx-000:34:590
1520749991311,1cyclictest60650irq/133-enp0s29f1:rx-023:32:490
1520749991311,1cyclictest60650irq/133-enp0s29f1:rx-023:25:260
152075999129,3cyclictest1588037-21kworker/u8:4+events_unbound21:10:183
152075999129,2cyclictest1571380-21kworker/u8:2+flush-259:020:10:163
152075999129,2cyclictest1571380-21kworker/u8:2+events_unbound20:15:223
152075999128,3cyclictest1820481-21kworker/u8:2+events_unbound00:10:223
152075999128,2cyclictest1775124-21kthreadcore23:15:193
1520759991210,1cyclictest0-21swapper/322:50:013
152075599129,2cyclictest1846868-21cpuspeed_turbos00:25:142
152075599129,2cyclictest1571380-21kworker/u8:2+events_unbound20:10:252
152075599125,5cyclictest1792223-21tr23:30:202
152075399129,2cyclictest1629376-21cpuspeed_turbos20:55:111
152075399129,2cyclictest1587395-21unixbench_singl20:10:241
152075399129,1cyclictest0-21swapper/100:40:001
152075399127,3cyclictest2698871-21systemd-journal20:19:261
1520753991210,1cyclictest0-21swapper/121:50:001
1520753991210,1cyclictest0-21swapper/121:49:591
152075399120,3cyclictest1665426-21basename21:29:591
152074999129,2cyclictest1809873-21turbostat23:50:000
152075999118,2cyclictest1722196-21kworker/u8:3+flush-259:022:34:573
152075999118,2cyclictest1571380-21kworker/u8:2+flush-259:021:50:243
152075999118,2cyclictest1571380-21kworker/u8:2+flush-259:021:50:233
152075999117,3cyclictest1799582-21kworker/u8:2+events_unbound23:45:123
152075999116,3cyclictest1510687-21kworker/u8:1+flush-259:019:25:053
152075599119,1cyclictest0-21swapper/221:04:592
152075599118,1cyclictest0-21swapper/219:10:152
152075599117,3cyclictest1799582-21kworker/u8:2+events_unbound23:40:142
152075399118,2cyclictest1836522-21cpuspeed_turbos00:15:141
152075399116,4cyclictest1717091-21apt-get22:20:121
152075399115,4cyclictest1691383-21/usr/sbin/munin21:55:171
152075399110,3cyclictest1737381-21idleruntime-cro22:40:001
152075399110,2cyclictest17271272sh22:30:011
152075999107,2cyclictest1670154-21unixbench_singl21:30:253
152075999105,3cyclictest1578564-21cut20:05:183
152075599108,1cyclictest0-21swapper/222:25:202
152075599105,3cyclictest1656323-21sed21:20:182
152075399108,1cyclictest0-21swapper/122:30:211
152075399108,1cyclictest0-21swapper/119:35:151
152075399107,2cyclictest1737913-21fschecks_time22:40:161
152075399105,4cyclictest1758233-21apt-get23:00:111
15207599994,4cyclictest1571380-21kworker/u8:2+flush-259:021:35:163
15207559997,1cyclictest0-21swapper/222:20:162
15207539997,1cyclictest0-21swapper/121:40:251
15207539995,2cyclictest16035792sleep120:30:141
15207539990,7cyclictest1618734-21latency_hist20:45:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional