You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 14:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Wed Sep 17, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
149588899752747,4cyclictest1502163-21kworker/3:3+events22:30:233
149588899740735,4cyclictest1502163-21kworker/3:3+events20:52:453
149588899740735,4cyclictest1502163-21kworker/3:3+events20:06:453
149588899739734,4cyclictest1754145-21kworker/3:1+events23:06:053
149588699739734,4cyclictest1697791-21kworker/2:1+events23:17:092
149588399739733,5cyclictest1653979-21kworker/1:0+events21:20:321
149588199735729,5cyclictest1572729-21turbostat20:14:560
149588899731728,2cyclictest1502163-21kworker/3:3+events22:28:343
149588899731726,4cyclictest1502163-21kworker/3:3+events22:37:093
149588899728723,4cyclictest1766761-21turbostat22:54:543
149588899727722,4cyclictest1779220-21turbostat23:04:553
149588899726723,2cyclictest1754145-21kworker/3:1+events00:02:213
149588899726719,6cyclictest1502163-21kworker/3:3+events20:11:463
149588399726722,3cyclictest1653979-21kworker/1:0+events23:33:491
149588899723719,3cyclictest1502163-21kworker/3:3+events21:51:503
149588899723718,4cyclictest939048-21kworker/3:1+events19:12:493
149588899722718,3cyclictest1502163-21kworker/3:3+events19:17:193
149588899721716,4cyclictest1660175-21turbostat21:24:553
149588899721711,8cyclictest1502163-21kworker/3:3+pm20:30:113
149588399719715,3cyclictest1653979-21kworker/1:0+events23:27:301
149588899717712,4cyclictest1754145-21kworker/3:1+events22:55:233
149588899716712,3cyclictest1754145-21kworker/3:1+events22:43:163
149588899716710,5cyclictest1502163-21kworker/3:3+events20:48:293
149588899716710,5cyclictest1502163-21kworker/3:3+events20:37:453
149588899715711,3cyclictest1754145-21kworker/3:1+events22:45:433
149588899715711,3cyclictest1502163-21kworker/3:3+events22:10:203
149588899715711,3cyclictest1502163-21kworker/3:3+events20:58:013
149588899714710,3cyclictest1754145-21kworker/3:1+events23:43:123
149588899714710,3cyclictest1502163-21kworker/3:3+events20:43:123
149588899712708,3cyclictest1502163-21kworker/3:3+events19:48:483
149588399712709,2cyclictest1653979-21kworker/1:0+events22:05:461
149588899711709,1cyclictest1502163-21kworker/3:3+events21:05:333
149588899711708,2cyclictest1502163-21kworker/3:3+events21:45:313
149588899711707,3cyclictest1502163-21kworker/3:3+events21:58:023
149588899711707,3cyclictest1502163-21kworker/3:3+events21:26:503
149588899711707,3cyclictest1502163-21kworker/3:3+events21:26:503
149588899711706,4cyclictest1754145-21kworker/3:1+events23:47:303
149588899711705,5cyclictest1502163-21kworker/3:3+events21:41:283
149588899710706,3cyclictest1754145-21kworker/3:1+events00:20:203
149588899710706,3cyclictest1502163-21kworker/3:3+events20:01:513
149588899710706,3cyclictest1502163-21kworker/3:3+events19:40:223
149588899708704,3cyclictest1754145-21kworker/3:1+events00:30:213
149588899708704,3cyclictest1502163-21kworker/3:3+events21:17:343
149588899707704,2cyclictest1502163-21kworker/3:3+events22:23:383
149588899707704,2cyclictest1502163-21kworker/3:3+events21:36:053
149588899706702,3cyclictest1754145-21kworker/3:1+events23:38:583
149588899706702,3cyclictest1754145-21kworker/3:1+events23:38:583
149588899706700,5cyclictest1502163-21kworker/3:3+events19:55:233
149588199706701,4cyclictest1528930-21kworker/0:1+events19:35:210
149588899704700,3cyclictest1754145-21kworker/3:1+events00:12:163
149588899704700,3cyclictest1502163-21kworker/3:3+events19:20:233
149588899704698,3cyclictest1754145-21kworker/3:1+events23:58:003
149588899703699,3cyclictest1754145-21kworker/3:1+events23:52:213
149588899703699,3cyclictest1754145-21kworker/3:1+events00:19:553
149588899703699,3cyclictest1502163-21kworker/3:3+events19:52:533
149588199702696,5cyclictest2541665-21kworker/0:3+events19:25:220
149588899701694,6cyclictest1502163-21kworker/3:3+events22:16:593
149588899699695,3cyclictest1754145-21kworker/3:1+events00:09:373
149588899698694,3cyclictest1754145-21kworker/3:1+events00:36:373
149588899698692,5cyclictest1754145-21kworker/3:1+events23:10:223
149588899698692,5cyclictest1754145-21kworker/3:1+events00:25:213
149588899698691,6cyclictest1502163-21kworker/3:3+events21:00:223
149588899697693,3cyclictest1502163-21kworker/3:3+events21:33:083
149588899696691,4cyclictest1502163-21kworker/3:3+events21:13:153
149588699695690,4cyclictest1791644-21turbostat23:14:552
149588399695692,2cyclictest1653979-21kworker/1:0+events22:03:461
149588699693690,2cyclictest1697791-21kworker/2:1+events23:23:152
149588199691686,4cyclictest2541665-21kworker/0:3+events20:20:220
149588399690685,4cyclictest1703957-21turbostat21:59:561
149588199690686,3cyclictest1522433-21kworker/0:0+events20:15:220
149588899682677,4cyclictest1716417-21turbostat22:09:563
149588399679676,2cyclictest1653951-21turbostat21:19:551
149588899674668,5cyclictest1591421-21turbostat20:29:563
149588199671667,3cyclictest2541665-21kworker/0:3+events20:28:160
149588199669666,2cyclictest2541665-21kworker/0:3+events19:34:160
149588899668665,2cyclictest1816553-21turbostat23:34:543
149588899643639,3cyclictest1529230-21turbostat19:39:563
149588199642639,2cyclictest1510387-21turbostat19:24:550
149588399638634,3cyclictest1804071-21turbostat23:24:541
2957983720,368rtkit-daemon939048-21kworker/3:1+pm19:07:303
149588199540,53cyclictest1598175-21/usr/sbin/munin20:30:120
16620792250,2sleep241-21ksoftirqd/221:20:182
1495886992317,4cyclictest1878867-21idleruntime-cro00:20:012
1495886992219,2cyclictest1866440-21cron00:10:002
1495886992118,2cyclictest0-21swapper/222:25:012
1495883992116,3cyclictest1667117-21irqcore21:25:211
1495883992116,3cyclictest1667117-21irqcore21:25:201
1495883992016,3cyclictest1636126-21irqcore21:00:181
1495881992016,3cyclictest493-21dbus-daemon23:20:000
1495881992015,4cyclictest1866445-21sh00:10:000
1495886991916,2cyclictest1756117-21kthreadcore22:40:162
1495886991916,2cyclictest0-21swapper/200:40:002
1495886991915,3cyclictest1502040-21cron19:15:002
1495883991916,2cyclictest499-21polkitd00:25:271
1495883991914,3cyclictest1755626-21cut22:40:161
1495881991916,2cyclictest1622999-21cron20:49:590
1495881991913,2cyclictest191rcu_preempt21:15:150
1495883991814,3cyclictest0-21swapper/123:40:001
1495883991814,3cyclictest0-21swapper/123:39:591
1495883991814,3cyclictest0-21swapper/100:20:011
1495883991813,3cyclictest1591895-21fschecks_count20:25:161
149588399181,14cyclictest1891537-21apt-get00:30:131
1495881991815,2cyclictest2873-21cron00:20:010
1495881991815,2cyclictest1610535-21ls20:39:590
1495881991815,2cyclictest1541910-21kworker/0:1+kacpi_notify19:45:430
1495886991714,2cyclictest509-21rs:main2
1495883991713,3cyclictest1585198-21idleruntime-cro20:20:011
1495883991713,2cyclictest0-21swapper/122:40:011
1495883991713,2cyclictest0-21swapper/119:20:001
149588399170,13cyclictest0-21swapper/119:35:161
1495881991715,1cyclictest1797853-21munin-run23:15:000
1495881991714,2cyclictest502-21systemd-logind21:00:000
1495881991713,2cyclictest0-21swapper/022:00:010
1495888991613,2cyclictest0-21swapper/323:20:003
149588699163,6cyclictest1541696-21munin-run19:45:012
1495886991613,2cyclictest1704542-21gltestperf21:55:152
1495886991612,2cyclictest0-21swapper/220:30:012
1495886991610,5cyclictest493-21dbus-daemon20:55:002
1495886991610,4cyclictest1722814-21grep22:10:212
149588399169,5cyclictest3763191-21systemd-journal21:00:001
149588399169,5cyclictest3763191-21systemd-journal20:55:001
1495883991613,2cyclictest1849517-21kthreadcore23:55:151
149588399161,2cyclictest1647765-21munin-run21:10:001
1495883991612,3cyclictest1736261-21fschecks_count22:25:171
1495883991612,2cyclictest1748168-21sh22:35:011
1495883991612,2cyclictest0-21swapper/119:50:161
1495883991610,4cyclictest3763191-21systemd-journal22:20:011
1495881991613,2cyclictest1885080-21munin-run00:25:010
1495881991613,2cyclictest1842191-21gltestperf23:50:170
1495881991613,2cyclictest1761320-21irqcore22:45:200
1495881991613,2cyclictest0-21swapper/000:00:000
1495881991612,3cyclictest62850irq/126-eno1:rx-023:30:150
1495881991610,4cyclictest3763191-21systemd-journal21:10:000
1495888991512,1cyclictest1584802-21head20:15:273
1495888991511,3cyclictest1516987-21cpuspeed_turbos19:25:103
1495886991512,2cyclictest1774717-21kthreadcore22:55:182
1495886991512,2cyclictest1535485-21sendmail19:40:012
1495886991511,3cyclictest1604811-21fschecks_time20:35:132
149588399159,5cyclictest3763191-21systemd-journal00:40:001
149588399158,5cyclictest493-21dbus-daemon19:30:001
1495883991512,2cyclictest1779956-21sed23:00:161
1495883991512,2cyclictest0-21swapper/121:40:001
1495883991512,2cyclictest0-21swapper/100:20:201
1495883991511,2cyclictest1766310-21grep22:45:281
1495883991511,2cyclictest0-21swapper/123:10:151
1495883991510,3cyclictest1590484-21needreboot20:20:191
1495881991512,2cyclictest499-21polkitd00:15:000
1495881991512,2cyclictest2873-21cron22:15:010
1495881991512,2cyclictest0-21swapper/022:05:000
1495881991511,3cyclictest1834484-21needreboot23:40:230
1495881991511,3cyclictest1785983-21gltestperf23:05:130
1495881991511,3cyclictest1559790-21unixbench_multi19:55:270
1495881991511,3cyclictest1504494-21cpuspeed_turbos19:15:150
149588699147,6cyclictest493-21dbus-daemon20:05:012
149588699145,8cyclictest0-21swapper/222:10:012
1495886991412,1cyclictest391rcuc/221:15:192
1495886991411,2cyclictest1891602-21cpuspeed_turbos00:30:142
1495886991411,2cyclictest1741481-21unixbench_multi22:25:262
1495886991411,2cyclictest1629533-21cpuspeed_turbos20:55:142
1495886991411,2cyclictest1585841-21gltestperf20:20:152
1495886991411,2cyclictest1526206-21kthreadcore19:30:222
1495886991410,3cyclictest1791167-21unixbench_multi23:05:212
1495886991410,3cyclictest1697339-21grep21:45:252
1495886991410,3cyclictest0-21swapper/200:05:002
149588699140,8cyclictest191rcu_preempt23:59:592
149588399149,4cyclictest311ktimers/123:45:111
149588399147,5cyclictest493-21dbus-daemon23:09:591
1495883991411,2cyclictest1561008-21sed20:00:161
1495883991410,3cyclictest1773341-21cpuspeed_turbos22:55:141
1495883991410,3cyclictest1571831-21grep20:05:231
1495883991410,3cyclictest1559785-21unixbench_multi19:55:281
1495883991410,2cyclictest16904992sleep121:40:221
1495883991410,2cyclictest0-21swapper/122:20:161
1495883991410,2cyclictest0-21swapper/120:40:231
149588199146,7cyclictest0-21swapper/019:10:200
149588199141,2cyclictest18386802kthreadcore23:45:160
1495881991412,1cyclictest1591130-21kworker/0:2+kacpid21:30:190
1495881991411,2cyclictest2873-21cron00:30:000
1495881991411,2cyclictest1878544-21kworker/0:1+kacpid00:35:170
1495881991411,2cyclictest1723265-21kworker/0:0+kacpid22:55:120
1495881991411,2cyclictest1548350-21fschecks_time19:50:140
1495881991411,2cyclictest0-21swapper/023:35:140
1495881991411,2cyclictest0-21swapper/023:35:130
1495881991411,2cyclictest0-21swapper/021:55:010
1495881991410,3cyclictest1679681-21gltestperf21:35:140
14898272143,6sleep20-21swapper/219:05:122
295798130,3rtkit-daemon2961-21sendmail-mta21:40:250
149588699139,3cyclictest493-21dbus-daemon19:20:232
149588699139,3cyclictest1691118-21unixbench_singl21:40:262
149588699139,3cyclictest0-21swapper/221:10:002
149588699137,4cyclictest523-21gdbus19:25:192
149588699136,5cyclictest523-21gdbus20:45:002
149588699133,2cyclictest191rcu_preempt23:30:002
1495886991311,1cyclictest1579718-21irqcore20:15:192
1495886991311,1cyclictest0-21swapper/223:40:182
1495886991310,2cyclictest1882150-21kthreadcore00:20:192
1495886991310,2cyclictest1838803-21kthreadcore23:45:162
1495886991310,2cyclictest1766784-21systemd-run22:50:002
1495886991310,2cyclictest1724022-21irqcore22:15:172
1495886991310,2cyclictest1666628-21/usr/sbin/munin21:25:222
1495886991310,2cyclictest1666628-21/usr/sbin/munin21:25:212
1495886991310,2cyclictest1641175-21grep21:00:272
149588699130,2cyclictest1578991-21awk20:15:012
149588399139,3cyclictest1830947-21kthreadcore23:40:201
149588399139,2cyclictest0-21swapper/122:50:151
149588399138,4cyclictest1535916-21sshd19:40:131
149588399138,3cyclictest0-21swapper/119:15:001
149588399136,6cyclictest1854578-21head00:00:151
1495883991310,2cyclictest1510748-21cpuspeed_turbos19:20:141
1495883991310,2cyclictest0-21swapper/121:10:151
1495883991310,2cyclictest0-21swapper/120:35:011
149588199139,2cyclictest0-21swapper/021:50:010
149588199138,4cyclictest3763191-21systemd-journal23:05:000
149588199133,3cyclictest1671846-21sendmail_mailst21:25:270
149588199133,3cyclictest1671846-21sendmail_mailst21:25:260
1495881991312,1cyclictest1495875-21cyclictest22:40:170
1495881991311,1cyclictest1591130-21kworker/0:2+kacpid20:50:140
1495881991310,2cyclictest1858451-21kthreadcore00:00:190
1495881991310,2cyclictest1810286-21latency_hist23:25:000
1495881991310,2cyclictest1767117-21cpuspeed_turbos22:50:130
1495881991310,2cyclictest1749886-21kthreadcore22:35:200
1495881991310,2cyclictest1742295-21cpuspeed_turbos22:30:130
1495881991310,2cyclictest1723526-21/usr/sbin/munin22:15:160
1495881991310,2cyclictest1648109-21cpuspeed_turbos21:10:140
1495881991310,2cyclictest0-21swapper/020:00:150
149588899129,2cyclictest0-21swapper/320:20:133
149588899126,4cyclictest1805767-21sshd23:20:173
149588699129,2cyclictest401ktimers/220:30:102
149588699129,2cyclictest1822284-21unixbench_multi23:30:262
149588699129,2cyclictest1679401-21cpuspeed_turbos21:35:132
149588699129,2cyclictest0-21swapper/223:40:002
149588699129,2cyclictest0-21swapper/223:40:002
149588699128,2cyclictest1767168-21tr22:50:132
149588699127,4cyclictest493-21dbus-daemon19:15:252
149588699127,4cyclictest1888779-21tr00:25:202
149588699126,4cyclictest1650058-21tr21:10:172
149588699125,5cyclictest493-21dbus-daemon21:30:002
149588699124,2cyclictest191rcu_preempt23:00:182
1495886991210,1cyclictest1622053-21needreboot20:45:192
1495886991210,1cyclictest0-21swapper/219:55:232
1495886991210,1cyclictest0-21swapper/219:50:152
1495886991210,1cyclictest0-21swapper/219:45:182
149588399127,4cyclictest1622245-21systemctl20:45:211
149588399126,5cyclictest523-21gdbus00:10:011
149588399126,5cyclictest1679065-21cut21:35:011
1495883991210,1cyclictest0-21swapper/120:11:311
149588199129,2cyclictest502-21systemd-logind22:30:010
149588199129,2cyclictest1685116-21kworker/0:3+kacpid23:26:230
149588199128,3cyclictest1566847-21cpuspeed_turbos20:05:170
149588199124,7cyclictest0-21swapper/020:40:170
1495881991210,1cyclictest1723265-21kworker/0:0+kacpid22:20:070
1495881991210,1cyclictest1685116-21kworker/0:3+kacpid22:07:540
1495881991210,1cyclictest1685116-21kworker/0:3+kacpid00:34:220
1495881991210,1cyclictest0-21swapper/019:40:200
149588899118,2cyclictest1810812-21fschecks_time23:25:153
149588899118,2cyclictest0-21swapper/319:30:203
149588699118,2cyclictest1842342-21irqcore23:50:182
149588699118,2cyclictest1714133-21kthreadcore22:00:192
149588699117,3cyclictest401ktimers/222:30:132
149588699113,2cyclictest191rcu_preempt20:05:272
149588399118,2cyclictest0-21swapper/123:15:141
149588399118,2cyclictest0-21swapper/122:10:211
149588399118,2cyclictest0-21swapper/121:50:161
149588399110,2cyclictest15234582grep19:30:171
149588399110,1cyclictest0-21swapper/121:48:411
149588399110,1cyclictest0-21swapper/120:37:141
149588399110,11cyclictest0-21swapper/100:10:161
149588399110,10cyclictest0-21swapper/123:50:291
149588199118,2cyclictest1660621-21diskstats21:20:150
149588199116,4cyclictest1635675-21apt-get21:00:140
149588899107,2cyclictest1716427-21dump-pmu-power22:05:013
149588699107,2cyclictest401ktimers/200:10:172
149588699106,3cyclictest1698335-21gltestperf21:50:152
149588399101,7cyclictest1542314-21gltestperf19:45:151
149588199108,1cyclictest2303365-21kworker/u8:2-ext4-rsv-conversion19:10:010
14956732106,2sleep10-21swapper/119:09:241
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional