You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 07:19
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8.osadl.org (updated Sat Jul 27, 2024 00:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
182531621090,1sleep244-21ksoftirqd/222:20:012
16064822940,1sleep01606486-21cut19:35:180
18625532300,1sleep30-21swapper/322:43:503
1569745992926,2cyclictest1749000-21munin-plugin-st21:30:001
16995312280,1sleep30-21swapper/320:50:243
156974599251,23cyclictest1645619-21apt-get20:10:111
1569749992421,2cyclictest1893267-21cron23:04:592
1569751992320,1cyclictest0-21swapper/322:25:003
1569751992313,8cyclictest1741139-21turbostat21:25:003
1569751992313,1cyclictest0-21swapper/323:10:183
1569751992311,1cyclictest0-21swapper/323:50:163
1569749992320,2cyclictest1984063-21turbostat.cron00:05:002
1569745992318,3cyclictest1984482-21tr00:05:111
1569751992216,4cyclictest0-21swapper/322:20:003
1569751992212,1cyclictest0-21swapper/321:50:173
1569751992210,1cyclictest0-21swapper/321:55:113
1569751992210,1cyclictest0-21swapper/320:00:163
1569751992210,1cyclictest0-21swapper/319:25:253
1569745992219,2cyclictest1915988-21latency_hist23:20:001
1569745992219,2cyclictest1700021-21apt-config20:55:011
1569743992220,1cyclictest1718384-21idleruntime-cro21:10:010
156975199219,1cyclictest0-21swapper/321:50:013
1569751992118,1cyclictest0-21swapper/300:15:013
1569751992111,1cyclictest0-21swapper/323:40:003
1569751992110,1cyclictest0-21swapper/322:50:153
1569751992110,1cyclictest0-21swapper/322:00:153
1569749992118,2cyclictest1620783-21apt-key19:50:002
1569749992118,2cyclictest1589870-21sh19:25:002
1569749992118,2cyclictest0-21swapper/200:00:002
1569745992118,2cyclictest3974914-21kworker/u8:2+events_unbound23:59:591
1569743992119,1cyclictest1809785-21basename22:10:000
1569751992017,1cyclictest0-21swapper/323:00:003
1569751992012,6cyclictest0-21swapper/321:10:003
1569751992011,1cyclictest0-21swapper/300:15:133
1569749992017,2cyclictest1639061-21/usr/sbin/munin20:05:002
1569745992018,1cyclictest0-21swapper/121:20:011
1569745992017,2cyclictest1657442-21sendmail20:20:001
1569745992014,5cyclictest0-21swapper/123:30:001
1569743992018,1cyclictest1885719-21idleruntime-cro23:00:000
1569743992017,2cyclictest1832778-21cron22:25:000
1569743992017,2cyclictest1699958-21cron20:55:000
1569743992017,2cyclictest0-21swapper/019:50:000
1569743992016,2cyclictest0-21swapper/021:50:000
156975199198,1cyclictest0-21swapper/321:45:003
1569751991914,3cyclictest0-21swapper/319:30:223
1569751991913,4cyclictest0-21swapper/323:00:273
1569751991910,1cyclictest0-21swapper/322:35:113
1569749991917,1cyclictest1885716-21cron23:00:002
1569749991916,2cyclictest1583857-21sendmail19:20:002
1569749991916,2cyclictest0-21swapper/200:20:002
1569749991915,3cyclictest1699964-21idleruntime-cro20:55:012
1569749991913,4cyclictest0-21swapper/221:45:002
1569745991916,2cyclictest1706269-21idleruntime-cro21:00:011
1569743991914,3cyclictest1785374-21grep21:50:280
1569743991912,6cyclictest1842059-21kthreadcore22:30:180
1569751991815,1cyclictest0-21swapper/323:30:203
1569751991815,1cyclictest0-21swapper/323:25:143
1569751991815,1cyclictest0-21swapper/321:25:003
1569751991815,1cyclictest0-21swapper/321:10:233
1569751991812,3cyclictest1840350-21sleep22:30:003
1569749991816,1cyclictest1602252-21idleruntime-cro19:35:002
1569749991815,2cyclictest306-21jbd2/sda2-819:55:012
1569749991814,2cyclictest0-21swapper/220:50:012
1569745991816,1cyclictest0-21swapper/122:45:011
1569745991815,2cyclictest1749978-21irqcore21:30:241
1569745991815,2cyclictest1625982-21needreboot19:50:191
1569743991815,2cyclictest851-21cron23:40:000
1569743991815,2cyclictest2030390-21cpuspeed_turbos00:35:110
1569743991815,2cyclictest1891556-21unixbench_singl23:00:270
1569743991815,2cyclictest1794696-21sendmail22:00:000
1569743991815,2cyclictest1756702-21latency_hist21:35:000
1569743991815,2cyclictest1576158-21kthreadcore19:15:010
1569743991813,4cyclictest1848460-21fschecks_count22:35:160
1569751991714,1cyclictest0-21swapper/300:30:003
1569749991714,2cyclictest2037740-21latency_hist00:40:012
1569749991714,2cyclictest1916467-21fschecks_count23:20:152
1569749991713,3cyclictest1848558-21gltestperf22:35:172
1569749991713,3cyclictest1772907-21irqcore21:45:142
1569749991713,3cyclictest1723693-21ntp_offset21:10:192
1569749991713,3cyclictest1706812-21cut21:00:152
1569749991713,3cyclictest0-21swapper/221:30:012
1569745991714,2cyclictest1871615-21irqcore22:50:131
1569745991714,2cyclictest1717660-21systemctl21:05:241
1569745991714,2cyclictest1670084-21gltestperf20:30:171
1569745991714,2cyclictest1645346-21latency_hist20:10:001
1569745991713,2cyclictest276-21plymouthd22:30:011
1569743991714,2cyclictest773-21gmain23:20:000
1569743991714,2cyclictest1876281-21unixbench-2d22:50:230
1569743991714,2cyclictest1680607-21needreboot20:35:210
1569743991714,2cyclictest1020-21rtkit-daemon00:12:000
1569743991714,2cyclictest0-21swapper/022:15:000
1569751991613,1cyclictest0-21swapper/321:35:003
1569751991613,1cyclictest0-21swapper/319:40:003
1569751991613,1cyclictest0-21swapper/319:21:313
1569749991614,1cyclictest1675548-21cron20:35:012
1569749991612,3cyclictest1909013-21gltestperf23:15:142
1569749991612,3cyclictest1764177-21systemd-run21:40:002
1569749991612,2cyclictest0-21swapper/219:40:002
1569745991614,1cyclictest1946177-21munin-plugin-st23:40:001
1569745991613,2cyclictest1978257-21kthreadcore00:00:201
1569745991613,2cyclictest1799979-21ntp_kernel_err22:00:201
1569745991613,2cyclictest1782796-21kthreadcore21:50:191
1569745991613,2cyclictest0-21swapper/123:50:011
1569745991613,2cyclictest0-21swapper/123:50:001
1569745991612,3cyclictest2027785-21grep00:30:241
1569745991612,3cyclictest1997014-21processes00:10:271
1569745991612,3cyclictest1887479-21kthreadcore23:00:181
156974399164,2cyclictest191rcu_preempt19:35:010
1569743991613,2cyclictest503-21polkitd20:15:240
1569743991613,2cyclictest2017082-21kthreadcore00:25:170
1569743991613,2cyclictest2007477-21cpuspeed_turbos00:20:110
1569743991613,2cyclictest1970169-21grep23:55:180
1569743991613,2cyclictest0-21swapper/021:10:210
1569743991613,2cyclictest0-21swapper/019:25:010
1569743991613,2cyclictest0-21swapper/000:20:000
1569743991612,2cyclictest0-21swapper/023:55:000
1569743991612,2cyclictest0-21swapper/021:25:000
1569743991611,4cyclictest0-21swapper/020:00:000
1569743991611,4cyclictest0-21swapper/020:00:000
1569743991611,4cyclictest0-21swapper/000:05:130
17076162150,5sleep016-21ksoftirqd/021:00:180
1569751991512,1cyclictest0-21swapper/323:55:213
1569751991512,1cyclictest0-21swapper/323:40:223
1569751991512,1cyclictest0-21swapper/323:21:533
1569751991512,1cyclictest0-21swapper/323:09:503
1569751991512,1cyclictest0-21swapper/322:48:023
1569751991512,1cyclictest0-21swapper/322:33:283
1569751991512,1cyclictest0-21swapper/321:15:093
1569751991512,1cyclictest0-21swapper/320:59:313
1569751991512,1cyclictest0-21swapper/320:45:203
1569751991512,1cyclictest0-21swapper/320:44:273
1569751991512,1cyclictest0-21swapper/320:35:193
1569751991512,1cyclictest0-21swapper/320:30:223
1569751991512,1cyclictest0-21swapper/320:22:453
1569751991512,1cyclictest0-21swapper/320:18:163
1569751991512,1cyclictest0-21swapper/320:05:163
1569751991512,1cyclictest0-21swapper/319:48:413
1569751991512,1cyclictest0-21swapper/300:36:293
1569751991512,1cyclictest0-21swapper/300:30:253
1569751991512,1cyclictest0-21swapper/300:21:133
1569751991512,1cyclictest0-21swapper/300:07:093
1569751991512,1cyclictest0-21swapper/300:01:283
1569749991512,2cyclictest2012609-21timerandwakeup00:20:242
1569749991512,2cyclictest1951903-21unixbench-2d23:40:242
1569749991512,2cyclictest1733687-21latency_hist21:20:002
1569749991512,2cyclictest1680701-21ntp_kernel_pll_20:35:222
1569749991512,2cyclictest1632990-21sendmail20:00:002
1569749991512,2cyclictest1632990-21sendmail20:00:002
1569749991511,3cyclictest1938690-21latency_hist23:35:002
1569749991511,3cyclictest1906591-21unixbench_multi23:10:252
1569749991511,3cyclictest1787947-21irqcore21:55:162
1569749991511,3cyclictest1609091-21cut19:40:152
1569749991510,3cyclictest0-21swapper/222:10:002
1569745991513,1cyclictest1880930-21kthreadcore22:55:221
1569745991512,2cyclictest2016310-21kthreadcore00:25:171
1569745991512,2cyclictest1898427-21missed_timers23:05:191
1569745991512,2cyclictest1863922-21irqcore22:45:151
1569745991512,2cyclictest1849632-21kernelversion22:35:201
1569745991512,2cyclictest1706699-21cpuspeed_turbos21:00:131
1569745991512,1cyclictest0-21swapper/121:45:001
1569745991511,3cyclictest1938607-21idleruntime-cro23:35:001
1569745991511,3cyclictest1724153-21unixbench_singl21:10:231
156974599150,14cyclictest1922609-21ssh23:22:351
156974599150,14cyclictest0-21swapper/122:05:221
1569743991512,2cyclictest1947741-21irqrtprio23:40:170
1569743991512,2cyclictest1838288-21switchtime22:25:270
1569743991512,2cyclictest1756989-21cpuspeed_turbos21:35:120
1569743991512,2cyclictest1675552-21turbostat20:35:000
1569743991512,2cyclictest1658051-21gltestperf20:20:140
1569743991512,2cyclictest1642920-21kthreadcore20:05:180
1569743991512,2cyclictest0-21swapper/021:25:340
1569743991510,4cyclictest1-21systemd20:55:320
1569751991412,1cyclictest0-21swapper/323:45:423
1569751991412,1cyclictest0-21swapper/323:45:423
1569751991412,1cyclictest0-21swapper/323:15:133
1569751991412,1cyclictest0-21swapper/322:05:383
1569751991412,1cyclictest0-21swapper/319:50:113
1569751991411,1cyclictest0-21swapper/322:13:153
1569751991411,1cyclictest0-21swapper/320:25:223
1569751991411,1cyclictest0-21swapper/320:13:023
1569751991411,1cyclictest0-21swapper/319:14:423
1569749991412,1cyclictest1798001-21kthreadcore22:00:172
1569749991411,2cyclictest1718383-21diskmemload22:20:202
1569749991411,2cyclictest0-21swapper/222:30:002
1569749991411,2cyclictest0-21swapper/221:25:002
1569749991410,3cyclictest2015239-21grep00:25:142
1569749991410,3cyclictest1861248-21unixbench_singl22:40:262
156974599149,4cyclictest1845654-21ntpq22:30:221
156974599148,5cyclictest1907296-21ssh23:12:091
1569745991412,1cyclictest0-21swapper/119:50:011
1569745991412,1cyclictest0-21swapper/119:35:011
1569745991411,2cyclictest1830942-21unixbench-2d22:20:271
1569745991411,2cyclictest1823362-21tail22:15:251
1569745991411,2cyclictest1734394-21sed21:20:131
1569745991410,3cyclictest1814533-21kthreadcore22:10:171
156974599140,2cyclictest276-21plymouthd23:52:241
156974399147,6cyclictest361-21systemd-journal22:05:010
156974399141,2cyclictest15925001kthreadcore19:25:200
1569743991411,2cyclictest757-21systemctl20:45:010
1569743991411,2cyclictest698-21snmpd23:26:180
1569743991411,2cyclictest1956218-21kthreadcore23:45:150
1569743991411,2cyclictest1956218-21kthreadcore23:45:150
1569743991411,2cyclictest1938637-21systemd-run23:35:010
1569743991411,2cyclictest1916613-21gltestperf23:20:160
1569743991411,2cyclictest1895181-21kthreadcore23:05:150
1569743991411,2cyclictest1718383-21diskmemload23:10:250
1569743991411,2cyclictest1718383-21diskmemload22:15:100
1569743991411,2cyclictest1688374-21irqcore20:45:150
1569743991411,2cyclictest1650420-21needreboot20:10:210
1569743991411,2cyclictest0-21swapper/021:45:000
1569743991411,2cyclictest0-21swapper/019:50:240
1569743991411,2cyclictest0-21swapper/000:00:120
1569751991311,1cyclictest0-21swapper/321:35:283
1569751991311,1cyclictest0-21swapper/321:00:223
1569751991311,1cyclictest0-21swapper/319:59:383
1569751991311,1cyclictest0-21swapper/319:59:373
1569751991311,1cyclictest0-21swapper/319:16:483
1569751991310,1cyclictest0-21swapper/319:40:233
156974999139,3cyclictest431ktimers/219:25:132
156974999138,4cyclictest431ktimers/220:40:142
1569749991311,1cyclictest1718383-21diskmemload21:33:352
1569749991311,1cyclictest1706516-21latency_hist21:00:012
1569749991311,1cyclictest0-21swapper/221:55:002
1569749991310,2cyclictest1929178-21timerandwakeup23:25:232
1569749991310,2cyclictest1866909-21kthreadcore22:45:182
1569749991310,2cyclictest1846133-21unixbench_singl22:30:272
1569749991310,2cyclictest1810648-21gltestperf22:10:112
1569749991310,2cyclictest1718383-21diskmemload23:35:202
1569749991310,2cyclictest1575315-21unixbench_singl19:10:262
1569749991310,2cyclictest0-21swapper/220:05:212
156974599137,4cyclictest1595473-21python319:25:261
1569745991311,1cyclictest2035042-21kthreadcore00:35:181
1569745991310,2cyclictest2007715-21gltestperf00:20:131
1569745991310,2cyclictest1946629-21cpuspeed_turbos23:40:121
1569745991310,2cyclictest1718651-21kworker/u8:3-nfsiod21:55:251
1569745991310,2cyclictest1638626-21unixbench_singl20:00:271
1569745991310,2cyclictest0-21swapper/119:15:001
156974599130,3cyclictest0-21swapper/100:16:101
156974599130,1cyclictest0-21swapper/119:20:181
1569743991311,1cyclictest1855504-21kworker/0:0+kacpid22:49:340
1569743991311,1cyclictest0-21swapper/020:05:000
1569743991310,2cyclictest2024491-21kthreadcore00:30:150
1569743991310,2cyclictest1855825-21cpu22:40:110
1569743991310,2cyclictest1727467-21irqrtprio21:15:180
1569743991310,2cyclictest1570842-21irqrtprio19:10:160
1569743991310,2cyclictest0-21swapper/020:30:000
156974999129,2cyclictest507-21systemd-logind00:10:002
156974999129,2cyclictest2022640-21sh00:30:122
156974999129,2cyclictest1959561-21unixbench_singl23:45:252
156974999129,2cyclictest1959561-21unixbench_singl23:45:242
156974999128,3cyclictest1963107-21kthreadcore23:50:152
156974999128,2cyclictest431ktimers/220:25:102
156974999126,4cyclictest361-21systemd-journal21:10:002
1569749991210,1cyclictest1718383-21diskmemload22:54:252
156974599129,2cyclictest1689076-21irqrtprio20:45:161
156974599129,2cyclictest1676500-21irqcore20:35:151
156974599129,2cyclictest1664080-21gltestperf20:25:131
156974599128,2cyclictest0-21swapper/120:45:011
156974599125,6cyclictest1583843-21munin-run19:20:001
156974599122,4cyclictest1608568-21munin-plugin-st19:40:001
1569745991210,1cyclictest1762290-21grep21:35:241
1569745991210,1cyclictest1662922-21timerwakeupswit20:20:241
1569743991210,1cyclictest1613520-21latency19:40:200
156974999118,2cyclictest1658013-21gltestperf20:20:132
156974999118,2cyclictest0-21swapper/223:05:172
156974999118,2cyclictest0-21swapper/220:20:002
156974999116,4cyclictest1651553-21sed20:15:012
156974599119,1cyclictest1765115-21irqcore21:40:151
156974599118,1cyclictest0-21swapper/119:55:181
156974599118,1cyclictest0-21swapper/119:55:181
156974599110,10cyclictest0-21swapper/119:40:101
156974999107,2cyclictest1718383-21diskmemload00:10:372
156974599105,3cyclictest1569754-21cat19:10:001
15697439996,2cyclictest0-21swapper/019:10:000
1563839280,3sleep20-21swapper/219:05:122
15697519974,1cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional