You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-23 - 12:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Thu Oct 23, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
161112499749742,6cyclictest1650185-21kworker/3:2+events20:31:063
161112499746742,3cyclictest1650185-21kworker/3:2+events20:35:213
161112499744740,3cyclictest1862087-21kworker/3:0+events22:39:023
161112499743739,3cyclictest1650185-21kworker/3:2+events22:03:443
161112499742739,2cyclictest1862087-21kworker/3:0+events00:30:213
161112499742736,5cyclictest1650185-21kworker/3:2+events21:48:043
161111999742738,3cyclictest1604841-21kworker/1:1+events23:06:591
161112499741736,4cyclictest1650185-21kworker/3:2+events21:50:213
161112499741736,4cyclictest1650185-21kworker/3:2+events21:20:223
161112499741735,5cyclictest1650185-21kworker/3:2+events20:00:213
161112499739732,6cyclictest1650185-21kworker/3:2+events20:11:383
161111999739733,5cyclictest1604841-21kworker/1:1+events22:33:021
161111999737733,3cyclictest1604841-21kworker/1:1+events23:11:561
161112499736731,4cyclictest1937413-21turbostat23:34:553
161112499734730,3cyclictest2648483-21kworker/3:1+events19:10:233
161112499733730,2cyclictest1862172-21turbostat22:34:553
161112499733726,5cyclictest1862087-21kworker/3:0+events22:42:323
161112499732728,3cyclictest1650185-21kworker/3:2+events22:17:353
161112499731726,4cyclictest2648483-21kworker/3:1+events19:25:203
161112499731726,4cyclictest1862087-21kworker/3:0+events23:51:273
161112499731726,4cyclictest1862087-21kworker/3:0+events22:58:513
161112499731726,4cyclictest1650185-21kworker/3:2+events21:57:563
161112499730725,4cyclictest1650185-21kworker/3:2+events19:43:473
161112499730724,4cyclictest1650185-21kworker/3:2+events20:26:523
161112499729724,4cyclictest1650185-21kworker/3:2+events22:27:203
161111999729725,3cyclictest1604841-21kworker/1:1+events21:39:141
161112499728724,3cyclictest2648483-21kworker/3:1+events19:34:443
161112499728724,3cyclictest1862087-21kworker/3:0+events22:50:223
161112499728724,3cyclictest1650185-21kworker/3:2+events20:05:203
161112499728722,5cyclictest1862087-21kworker/3:0+events23:40:223
161112499727723,3cyclictest1650185-21kworker/3:2+events19:57:343
161112499727722,4cyclictest2648483-21kworker/3:1+events19:20:323
161112499727721,5cyclictest1862087-21kworker/3:0+events22:46:473
161112499727721,5cyclictest1862087-21kworker/3:0+events00:26:583
161112499726722,3cyclictest2648483-21kworker/3:1+events19:19:423
161112499726722,3cyclictest1862087-21kworker/3:0+events00:18:283
161112499726722,3cyclictest1650185-21kworker/3:2+events19:39:203
161112499726721,4cyclictest1650185-21kworker/3:2+events22:20:213
161112499726721,4cyclictest1650185-21kworker/3:2+events22:20:203
161111999726722,3cyclictest1900108-21turbostat23:04:561
161112499725720,4cyclictest2017877-21kworker/3:2+events00:39:073
161112499725717,3cyclictest1862087-21kworker/3:0+events23:35:433
161112499725716,4cyclictest1650185-21kworker/3:2+events21:25:213
161112499723719,3cyclictest1862087-21kworker/3:0+events00:13:503
161112499722718,3cyclictest1862087-21kworker/3:0+events23:48:223
161112499721717,3cyclictest1862087-21kworker/3:0+events00:23:243
161112499721717,3cyclictest1862087-21kworker/3:0+events00:23:243
161112499721717,3cyclictest1650185-21kworker/3:2+events20:21:503
161112499720716,3cyclictest1650185-21kworker/3:2+events22:10:213
161112299720717,2cyclictest1681542-21kworker/2:1+events21:09:112
161112499719715,3cyclictest1650185-21kworker/3:2+events22:08:573
161112499719715,3cyclictest1650185-21kworker/3:2+events21:31:053
161112499719715,3cyclictest1650185-21kworker/3:2+events20:18:373
161112499719715,3cyclictest1650185-21kworker/3:2+events20:18:373
161112499718714,3cyclictest1862087-21kworker/3:0+events23:55:223
161112499718714,3cyclictest1862087-21kworker/3:0+events23:26:403
161112499718714,3cyclictest1862087-21kworker/3:0+events00:05:433
161112499717713,3cyclictest1862087-21kworker/3:0+events23:02:103
161111999717713,3cyclictest1855950-21turbostat22:29:551
161112499715711,3cyclictest1862087-21kworker/3:0+events00:04:003
161112299711705,5cyclictest1638068-21kworker/2:0+events19:50:232
161112299704701,2cyclictest1750254-21turbostat21:04:552
161112499703698,2cyclictest1768851-21turbostat21:19:563
161112299701696,4cyclictest1912532-21turbostat23:14:552
161111799698693,4cyclictest1604598-21kworker/0:3+events21:00:230
161112299697689,7cyclictest1681542-21kworker/2:1+events20:48:592
161111999689685,3cyclictest1787586-21turbostat21:34:551
161112499688685,2cyclictest1924983-21turbostat23:24:563
161112499687683,3cyclictest1799999-21turbostat21:44:543
161112299680675,4cyclictest1681542-21kworker/2:1+events21:10:212
161112299679675,3cyclictest1681542-21kworker/2:1+events21:15:432
161112299677673,3cyclictest1681542-21kworker/2:1+events20:52:512
161111799673669,3cyclictest1604598-21kworker/0:3+events20:59:290
161112299672668,3cyclictest1638068-21kworker/2:0+events19:48:492
161112499670666,3cyclictest1662940-21turbostat19:54:553
161112299667661,5cyclictest1719121-21turbostat20:39:552
161111799665661,3cyclictest1737839-21turbostat20:54:550
161112299663658,4cyclictest1793760-21turbostat21:39:552
161112299659654,4cyclictest1824891-21kworker/2:2+events23:16:372
161112299656651,4cyclictest1824891-21kworker/2:2+events23:20:232
161112299655649,5cyclictest1681542-21kworker/2:1+events20:43:452
161112299649645,3cyclictest1681542-21kworker/2:1+events21:44:182
161112299646642,3cyclictest1650514-21turbostat19:44:552
1611122992824,3cyclictest491-21avahi-daemon22:35:002
1611119992825,2cyclictest1725366-21systemctl20:40:001
1611117992421,2cyclictest0-21swapper/000:30:000
1611122992320,2cyclictest1962249-21idleruntime-cro23:50:002
1611119992316,6cyclictest1772124-21kthreadcore21:15:171
161111999231,21cyclictest1631847-21cron19:24:591
1611119992213,8cyclictest1962241-21munin-plugin-st23:50:001
1611119992213,2cyclictest1837265-21basename22:10:001
1611119992212,8cyclictest0-21swapper/119:40:171
1611119992210,11cyclictest1955616-21unixbench_singl23:40:251
1611119992111,8cyclictest0-21swapper/119:20:011
1611119992110,10cyclictest1775429-21cpuspeed_turbos21:20:161
161111999209,9cyclictest1644865-21gltestperf19:35:171
161111999208,5cyclictest3763191-21systemd-journal00:20:001
1611119992014,5cyclictest0-21swapper/122:40:011
1611117992015,3cyclictest0-21swapper/023:45:000
161111799200,19cyclictest1617296-21sh19:15:010
1611122991917,1cyclictest1625636-21cut19:20:012
1611122991914,3cyclictest0-21swapper/222:15:002
161111999198,9cyclictest0-21swapper/100:20:151
161111999198,9cyclictest0-21swapper/100:20:151
161111999198,2cyclictest0-21swapper/123:50:111
161111999198,2cyclictest0-21swapper/121:40:231
161111999198,10cyclictest1982710-21kthreadcore00:05:211
161111999197,3cyclictest1949276-21grep23:35:251
161111999196,3cyclictest2006685-21cut00:25:131
161111999191,6cyclictest1812483-21idleruntime-cro21:50:001
161111999191,6cyclictest1-21systemd22:50:001
1611119991910,8cyclictest1706431-21cron20:24:591
1611119991910,7cyclictest0-21swapper/122:50:171
161111999190,1cyclictest0-21swapper/121:00:181
1611117991916,2cyclictest0-21swapper/000:00:000
161111999189,7cyclictest0-21swapper/120:30:011
161111999188,2cyclictest1781936-21gltestperf21:25:121
161111999187,2cyclictest1877440-21kthreadcore22:40:181
161111999187,2cyclictest0-21swapper/122:25:011
161111999187,2cyclictest0-21swapper/122:25:011
161111999187,2cyclictest0-21swapper/100:10:131
161111999186,3cyclictest1824430-21grep21:55:271
161111999181,9cyclictest1938005-21gltestperf23:30:141
161111999181,6cyclictest1632210-21tr19:25:111
1611119991815,2cyclictest1744084-21idleruntime-cro20:55:011
161111999181,4cyclictest493-21dbus-daemon20:00:001
161111999181,3cyclictest1662943-21munin-run19:49:591
161111999180,2cyclictest2018688-21diskstats00:35:141
161111999180,1cyclictest0-21swapper/123:55:131
1611117991815,2cyclictest1900499-21cpuspeed_turbos23:00:120
1611117991814,3cyclictest1812042-21unixbench_singl21:45:210
161111799180,17cyclictest0-21swapper/023:45:240
1611122991714,2cyclictest1843876-21cpuspeed_turbos22:15:152
1611122991712,4cyclictest1986711-21grep00:05:282
1611122991712,3cyclictest0-21swapper/223:10:012
161111999178,2cyclictest0-21swapper/122:55:141
161111999177,2cyclictest0-21swapper/119:50:171
161111999177,1cyclictest0-21swapper/120:48:591
161111999176,3cyclictest0-21swapper/120:15:171
161111999176,3cyclictest0-21swapper/120:15:171
161111999175,4cyclictest1767942-21ntp_kernel_err21:10:221
161111999175,3cyclictest1727388-21tr20:40:171
161111999174,3cyclictest1843735-21/usr/sbin/munin22:15:131
161111999171,5cyclictest1931195-21latency_hist23:25:011
1611117991714,2cyclictest1732812-21cut20:45:160
1611117991713,3cyclictest1775711-21gltestperf21:20:170
1611117991713,3cyclictest0-21swapper/020:20:000
1611117991713,3cyclictest0-21swapper/020:19:590
1611117991710,5cyclictest3763191-21systemd-journal22:10:000
1611124991613,2cyclictest1925014-21systemd-run23:20:013
1611124991612,3cyclictest1-21systemd21:11:123
1611122991613,2cyclictest3763191-21systemd-journal00:15:002
1611122991612,3cyclictest1825386-21fschecks_time22:00:132
161111999167,2cyclictest0-21swapper/122:10:171
161111999166,2cyclictest6950irq/9-acpi19:10:191
161111999163,4cyclictest1924683-21sshd23:16:361
161111999161,7cyclictest1719119-21munin-plugin-st20:35:001
161111999161,6cyclictest3763191-21systemd-journal00:35:001
161111999161,6cyclictest1980973-21cron00:05:001
1611117991613,2cyclictest1708010-21irqrtprio20:25:190
1611117991613,2cyclictest1669603-21fschecks_time19:55:140
1611117991612,3cyclictest2018824-21gltestperf00:35:150
1611124991510,3cyclictest1738256-21grep20:50:133
161112299158,5cyclictest523-21gdbus20:30:012
1611122991512,2cyclictest1888198-21gltestperf22:50:152
1611122991512,2cyclictest1787579-21cut21:30:002
1611122991512,2cyclictest1701474-21irqrtprio20:20:162
1611122991512,2cyclictest1641327-21kthreadcore19:30:172
161111999159,1cyclictest0-21swapper/122:04:481
161111999155,2cyclictest0-21swapper/123:25:181
1611119991512,2cyclictest1749848-21unixbench_singl20:55:251
1611119991512,2cyclictest1688471-21grep20:10:181
1611119991512,2cyclictest0-21swapper/120:05:161
161111999150,7cyclictest1644275-21munin-run19:35:001
1611117991512,2cyclictest1725359-21cut20:40:000
1611117991512,2cyclictest1604598-21kworker/0:3+kacpid21:13:050
1611117991511,3cyclictest1830110-21needreboot22:00:190
1611117991510,3cyclictest0-21swapper/021:40:000
16050382153,7sleep30-21swapper/319:05:103
1611124991411,2cyclictest1800002-21munin-run21:40:003
161112299148,4cyclictest1980130-21ntpq00:00:222
161112299148,4cyclictest1899891-21sshd22:57:352
1611122991411,2cyclictest1687345-21unixbench_singl20:05:252
1611122991411,2cyclictest0-21swapper/200:00:002
1611122991410,3cyclictest1824442-21unixbench_singl21:55:262
161112299140,2cyclictest0-21swapper/219:15:012
161111999149,3cyclictest1818681-21systemd-run21:55:001
161111799149,4cyclictest3763191-21systemd-journal23:26:340
161111799149,3cyclictest1888207-21gltestperf22:50:150
1611117991411,2cyclictest1907544-21tr23:05:160
1611117991411,2cyclictest1875596-21cpuspeed_turbos22:40:140
1611117991411,2cyclictest1844062-21gltestperf22:15:160
1611117991411,2cyclictest1781700-21tr21:25:100
1611117991411,2cyclictest0-21swapper/019:35:000
1611117991411,2cyclictest0-21swapper/000:15:210
1611117991411,2cyclictest0-21swapper/000:09:590
1611117991411,2cyclictest0-21swapper/000:05:000
1611117991410,3cyclictest1819398-21sed21:55:190
1611117991410,2cyclictest2005428-21head00:20:260
1611117991410,2cyclictest2005428-21head00:20:260
1611117991410,2cyclictest0-21swapper/022:40:010
1611124991310,2cyclictest1762651-21sh21:10:013
161112299139,3cyclictest523-21gdbus20:00:002
161112299139,3cyclictest1718662-21unixbench_multi20:30:262
161112299138,4cyclictest1874038-21ntpq22:35:202
161112299135,1cyclictest1694561-21grep20:15:152
161112299135,1cyclictest1694561-21grep20:15:152
1611122991310,2cyclictest64-21kcompactd023:30:182
1611122991310,2cyclictest1962734-21fschecks_time23:50:142
1611122991310,2cyclictest1688402-21idleruntime20:10:172
161111999130,6cyclictest1678582-21kthreadcore20:00:181
161111799139,3cyclictest1649641-21ntpq19:35:220
161111799139,3cyclictest1604598-21kworker/0:3+kacpi_notify21:30:290
161111799139,3cyclictest1572438-21kworker/0:2+kacpi_notify19:45:150
161111799138,1cyclictest0-21swapper/000:30:160
161111799137,4cyclictest1921626-21tr23:15:170
1611117991311,1cyclictest0-21swapper/019:15:010
1611117991310,2cyclictest1944321-21tr23:35:170
1611117991310,2cyclictest1837885-21gltestperf22:10:130
1611117991310,2cyclictest1682244-21sed20:05:160
1611117991310,2cyclictest1604598-21kworker/0:3+kacpi_notify22:21:390
1611117991310,2cyclictest1604598-21kworker/0:3+kacpi_notify22:21:380
1611117991310,2cyclictest1604598-21kworker/0:3+kacpid20:00:180
161111799130,4cyclictest1915311-21tr23:10:220
161112499129,2cyclictest0-21swapper/323:06:423
161112499128,2cyclictest0-21swapper/321:00:013
1611124991210,1cyclictest0-21swapper/323:13:403
161112299129,2cyclictest2006164-21cpuspeed_turbos00:25:112
161112299129,2cyclictest1944144-21gltestperf23:35:152
161112299129,2cyclictest1776042-21irqcore21:20:182
161112299129,2cyclictest1744804-21sed20:55:162
161112299129,2cyclictest0-21swapper/222:20:192
161112299129,2cyclictest0-21swapper/222:20:192
161112299129,2cyclictest0-21swapper/200:35:162
161112299128,3cyclictest1675648-21cpuspeed_turbos20:00:142
161112299128,2cyclictest1993973-21grep00:15:152
161112299126,4cyclictest1905439-21ntpq23:00:202
161112299126,2cyclictest391rcuc/219:40:002
161112299124,2cyclictest191rcu_preempt23:40:142
161112299123,1cyclictest191rcu_preempt22:45:002
1611122991210,1cyclictest1808173-21kthreadcore21:45:142
1611122991210,1cyclictest1789302-21kthreadcore21:30:232
1611122991210,1cyclictest0-21swapper/221:50:152
161112299120,2cyclictest191rcu_preempt19:24:592
161112299120,1cyclictest0-21swapper/222:45:242
161111999129,2cyclictest1757069-21gltestperf21:05:141
161111799129,2cyclictest1654830-21kthreadcore19:40:190
161111799129,2cyclictest1604598-21kworker/0:3+kacpi_notify22:45:240
161111799129,2cyclictest1604598-21kworker/0:3+kacpid21:09:220
161111799129,2cyclictest0-21swapper/021:40:150
161111799129,2cyclictest0-21swapper/020:15:000
161111799128,3cyclictest1572438-21kworker/0:2+kacpi_notify19:25:190
161111799128,2cyclictest1931188-21cron23:25:010
161111799128,2cyclictest1900108-21turbostat23:00:010
161111799126,5cyclictest523-21gdbus19:55:000
161111799123,8cyclictest0-21swapper/023:50:160
1611117991210,1cyclictest1604598-21kworker/0:3+kacpid21:50:180
161111799120,4cyclictest1769571-21ps21:15:160
161111799120,11cyclictest0-21swapper/023:30:120
16106662123,2sleep20-21swapper/219:06:182
161112499119,1cyclictest0-21swapper/319:45:223
161112499118,2cyclictest0-21swapper/320:45:163
161112299119,1cyclictest0-21swapper/222:05:182
161112299117,3cyclictest2000317-21irqcore00:20:172
161112299117,3cyclictest2000317-21irqcore00:20:172
161112299117,3cyclictest0-21swapper/223:25:172
161112299117,1cyclictest191rcu_preempt00:30:212
161112299116,4cyclictest1856397-21fschecks_count22:25:132
161112299116,4cyclictest1632942-21sed19:25:142
161111799118,2cyclictest171ktimers/019:23:470
161111799118,2cyclictest0-21swapper/022:30:210
161111799118,2cyclictest0-21swapper/020:20:160
161111799115,4cyclictest523-21gdbus00:15:000
161111799110,9cyclictest1725615-21/usr/sbin/munin20:40:150
16107082112,2sleep10-21swapper/119:06:501
161112499108,1cyclictest0-21swapper/321:00:153
161112499105,4cyclictest1727560-21tr20:40:173
161111799105,4cyclictest1717243-21tr20:30:200
161111799100,3cyclictest3763191-21systemd-journal22:29:350
16064792102,5sleep01607841-21cat19:05:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional