You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-11 - 22:29
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8.osadl.org (updated Sun May 11, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8519402360,1chrt851939-21ssh12:29:190
374409992926,2cyclictest769-21runrttasks11:55:002
374403992624,1cyclictest0-21swapper/009:40:000
4454232250,1chrt0-21swapper/308:05:163
374407992522,2cyclictest541617-21sendmail09:20:011
374412992320,1cyclictest0-21swapper/312:00:003
374412992320,1cyclictest0-21swapper/311:00:013
374412992219,1cyclictest0-21swapper/311:55:003
374412992219,1cyclictest0-21swapper/309:40:013
374412992219,1cyclictest0-21swapper/308:40:003
374409992218,2cyclictest0-21swapper/207:20:002
374407992218,2cyclictest0-21swapper/110:00:001
8266822210,1sleep00-21swapper/012:12:370
374409992118,2cyclictest599261-21cron09:55:002
374409992118,2cyclictest599261-21cron09:55:002
37440999210,4cyclictest519043-21munin-run09:05:002
374407992118,2cyclictest0-21swapper/107:35:011
374412992017,1cyclictest0-21swapper/307:15:003
374412992016,1cyclictest0-21swapper/312:20:003
374412992016,1cyclictest0-21swapper/310:30:003
374407992017,2cyclictest738295-21sh11:19:591
374403992018,1cyclictest0-21swapper/010:30:000
374412991916,1cyclictest0-21swapper/310:40:013
374412991916,1cyclictest0-21swapper/307:55:013
374409991916,2cyclictest844087-21latency_hist12:25:012
374409991916,2cyclictest829126-21irqrtprio12:15:182
374409991915,3cyclictest631899-21/usr/sbin/munin10:15:002
374409991915,2cyclictest607444-21sendmail10:00:002
374407991916,2cyclictest500-21avahi-daemon10:55:011
374407991916,2cyclictest3564018-21systemctl11:35:011
374403991917,1cyclictest837224-21sed12:20:180
374403991917,1cyclictest609204-21kthreadcore10:00:200
374403991917,1cyclictest374398-21cyclictest12:30:200
374403991916,2cyclictest803486-21latency_hist12:00:000
4266942180,1sleep20-21swapper/207:50:202
374412991815,1cyclictest0-21swapper/308:45:013
374412991815,1cyclictest0-21swapper/307:45:173
374412991814,1cyclictest0-21swapper/312:22:193
374412991812,3cyclictest7140502sleep311:04:563
374409991815,2cyclictest547-21gdbus09:45:282
374409991815,2cyclictest508-21polkitd08:20:262
374409991815,2cyclictest500-21avahi-daemon10:45:252
374409991815,2cyclictest457201-21systemd-run08:15:002
374409991814,3cyclictest714936-21sed11:05:182
374409991814,2cyclictest721853-21kworker/u8:1-nfsiod11:15:182
374409991812,4cyclictest488098-21kworker/u8:0-nfsiod11:10:192
374407991815,2cyclictest7550irq/9-acpi12:30:141
374407991815,2cyclictest650392-21kthreadcore10:25:241
374407991815,2cyclictest431557-21ntp_kernel_pll_07:50:271
374403991816,1cyclictest779012-21sh11:45:000
374403991815,2cyclictest761-21cups-browsed09:45:000
5730722170,0sleep1573080-21ssh09:36:581
374412991714,1cyclictest0-21swapper/310:55:003
374412991714,1cyclictest0-21swapper/310:10:003
374412991714,1cyclictest0-21swapper/309:45:213
374412991714,1cyclictest0-21swapper/308:15:003
374412991714,1cyclictest0-21swapper/307:40:003
374409991714,2cyclictest528597-21kthreadcore09:10:232
374409991713,3cyclictest563610-21ntp_kernel_err09:30:202
374409991712,3cyclictest0-21swapper/210:20:012
374407991714,2cyclictest812186-21fschecks_time12:05:161
374407991714,2cyclictest703374-21mailstats10:55:251
374403991714,2cyclictest559861-21irqstats09:30:170
374403991713,2cyclictest0-21swapper/012:35:180
374403991710,6cyclictest0-21swapper/009:25:220
374412991613,1cyclictest0-21swapper/310:15:243
374412991613,1cyclictest0-21swapper/308:50:013
374412991613,1cyclictest0-21swapper/307:45:003
374412991613,1cyclictest0-21swapper/307:20:173
37440999164,6cyclictest582947-21munin-run09:45:002
374409991613,2cyclictest569696-21kthreadcore09:35:212
374409991613,2cyclictest533698-21/usr/sbin/munin09:15:212
374409991612,3cyclictest406857-21mailstats07:30:222
374409991612,2cyclictest0-21swapper/212:40:002
374409991612,2cyclictest0-21swapper/210:10:002
37440799167,8cyclictest0-21swapper/107:40:001
374407991614,1cyclictest591065-21cron09:50:001
374407991613,2cyclictest769-21runrttasks08:45:241
374407991613,2cyclictest547-21gdbus09:25:291
374407991613,2cyclictest0-21swapper/109:25:001
374407991612,3cyclictest469135-21unixbench_singl08:20:291
374407991612,3cyclictest331ktimers/107:55:141
374407991612,3cyclictest331ktimers/107:55:141
374407991610,5cyclictest665840-21irqcore10:35:201
37440399169,6cyclictest487910-21kworker/0:0+kacpid09:23:360
37440399167,3cyclictest746470-21idleruntime-cro11:24:590
374403991613,2cyclictest625841-21kthreadcore10:10:180
374403991613,2cyclictest515-21wpa_supplicant10:43:290
374403991612,3cyclictest763372-21gltestperf11:35:150
374403991612,3cyclictest643800-21kthreadcore10:20:230
374403991612,3cyclictest3561480-21systemd-journal11:19:180
374412991512,1cyclictest0-21swapper/312:35:273
374412991512,1cyclictest0-21swapper/312:28:033
374412991512,1cyclictest0-21swapper/312:10:163
374412991512,1cyclictest0-21swapper/312:00:183
374412991512,1cyclictest0-21swapper/311:45:183
374412991512,1cyclictest0-21swapper/311:45:173
374412991512,1cyclictest0-21swapper/311:30:263
374412991512,1cyclictest0-21swapper/311:25:113
374412991512,1cyclictest0-21swapper/310:45:223
374412991512,1cyclictest0-21swapper/310:30:253
374412991512,1cyclictest0-21swapper/310:20:273
374412991512,1cyclictest0-21swapper/309:55:213
374412991512,1cyclictest0-21swapper/309:20:213
374412991512,1cyclictest0-21swapper/308:51:563
374412991512,1cyclictest0-21swapper/308:25:003
374412991512,1cyclictest0-21swapper/308:15:203
37440999158,4cyclictest395556-21sshd07:25:132
374409991512,2cyclictest852726-21cut12:30:162
374409991512,2cyclictest817621-21sh12:05:272
374409991512,2cyclictest795578-21runrttasks11:55:082
374409991512,2cyclictest779781-21idleruntime11:45:172
374409991512,2cyclictest779781-21idleruntime11:45:172
374409991512,2cyclictest757910-21kthreadcore11:30:222
374409991512,2cyclictest707741-21kthreadcore11:00:182
374409991512,2cyclictest543203-21irqrtprio09:20:212
374409991512,2cyclictest475091-21threads08:25:262
374409991512,2cyclictest443841-21needreboot08:00:222
374409991512,2cyclictest413886-21cron07:40:002
374409991511,3cyclictest807626-21kthreadcore12:00:212
374409991511,3cyclictest419612-21unixbench-2d07:40:352
374409991511,2cyclictest0-21swapper/210:25:312
374409991511,2cyclictest0-21swapper/207:45:142
37440799159,4cyclictest1-21systemd10:35:001
374407991513,1cyclictest794628-21ssh11:53:371
374407991513,1cyclictest727561-21needreboot11:10:221
374407991512,2cyclictest864546-21kthreadcore12:35:231
374407991512,2cyclictest763985-21irqrtprio11:35:171
374407991512,2cyclictest686880-21ntp_kernel_pll_10:45:231
374407991512,2cyclictest562076-21kthreadcore09:30:181
374407991512,2cyclictest530690-21netstat09:10:251
374407991512,2cyclictest524-21rs:main1
374407991512,2cyclictest481416-21unixbench_multi08:30:321
374407991512,2cyclictest457288-21latency_hist08:15:011
374407991512,2cyclictest399530-21kthreadcore07:25:191
374407991512,2cyclictest0-21swapper/110:15:001
374407991512,2cyclictest0-21swapper/110:05:001
374407991512,1cyclictest0-21swapper/112:30:001
374407991511,3cyclictest841451-21proc_pri12:20:251
374407991511,3cyclictest7550irq/9-acpi08:40:201
37440799151,13cyclictest714455-21apt-get11:05:141
374407991511,3cyclictest637751-21timerandwakeup10:15:301
374407991511,3cyclictest620613-21kthreadcore10:05:231
374407991511,2cyclictest583103-21latency_hist09:45:001
374407991511,2cyclictest475317-21unixbench_singl08:25:271
374407991511,2cyclictest415998-21kthreadcore07:40:271
37440799150,3cyclictest445416-21gltestperf08:05:151
37440399159,4cyclictest426244-21cron07:50:000
37440399156,8cyclictest0-21swapper/012:00:210
374403991512,2cyclictest715410-21tr11:05:180
374403991512,2cyclictest689754-21latency_hist10:50:010
374403991512,2cyclictest640272-21latency_hist10:20:010
374403991512,2cyclictest588374-21needreboot09:45:240
374403991512,2cyclictest482511-21gltestperf08:35:190
374403991512,2cyclictest439347-21irqcore08:00:170
374403991512,2cyclictest408197-21gltestperf07:35:170
374403991511,3cyclictest779477-21cpuspeed_turbos11:45:150
374403991511,3cyclictest779477-21cpuspeed_turbos11:45:140
374403991511,3cyclictest475277-21grep08:25:260
374403991511,3cyclictest390751-21kthreadcore07:20:190
374403991511,2cyclictest0-21swapper/010:35:180
5182612140,1chrt0-21swapper/309:00:273
374412991411,1cyclictest0-21swapper/312:05:163
374412991411,1cyclictest0-21swapper/311:35:243
374412991411,1cyclictest0-21swapper/311:16:323
374412991411,1cyclictest0-21swapper/311:11:123
374412991411,1cyclictest0-21swapper/311:08:093
374412991411,1cyclictest0-21swapper/310:40:433
374412991411,1cyclictest0-21swapper/310:00:213
374412991411,1cyclictest0-21swapper/309:40:153
374412991411,1cyclictest0-21swapper/309:25:203
374412991411,1cyclictest0-21swapper/309:16:233
374412991411,1cyclictest0-21swapper/309:13:453
374412991411,1cyclictest0-21swapper/308:58:313
374412991411,1cyclictest0-21swapper/308:30:223
374412991411,1cyclictest0-21swapper/307:55:163
374412991411,1cyclictest0-21swapper/307:55:153
374412991411,1cyclictest0-21swapper/307:35:003
374412991411,1cyclictest0-21swapper/307:27:183
374412991411,1cyclictest0-21swapper/307:19:083
37440999149,4cyclictest476922-21irqrtprio08:30:242
37440999147,5cyclictest547-21gdbus08:00:012
37440999147,5cyclictest547-21gdbus08:00:002
374409991411,2cyclictest834258-21kworker/u8:0-nfsiod12:25:202
374409991411,2cyclictest609206-21ps10:00:192
374409991411,2cyclictest488559-21uniq08:40:222
374409991411,2cyclictest445797-21irqcore08:05:162
374409991411,2cyclictest0-21swapper/211:25:012
374409991411,2cyclictest0-21swapper/210:30:142
374409991410,3cyclictest820339-21gltestperf12:10:162
374409991410,3cyclictest700515-21kthreadcore10:55:202
374409991410,3cyclictest645649-21kthreadcore10:20:242
374409991410,3cyclictest485853-21kthreadcore08:35:232
374409991410,3cyclictest1-21systemd11:35:182
37440799148,4cyclictest3561480-21systemd-journal07:15:031
374407991411,2cyclictest804057-21fschecks_time12:00:181
374407991411,2cyclictest592911-21kthreadcore09:50:211
374407991411,2cyclictest592911-21kthreadcore09:50:211
374407991411,2cyclictest525201-21diskmemload11:40:161
374407991411,2cyclictest524481-21mailstats09:05:281
374407991411,2cyclictest513430-21gltestperf09:00:191
374407991411,2cyclictest506157-21unixbench-2d08:50:291
374407991410,3cyclictest796011-21gltestperf11:55:151
37440799140,2cyclictest510936-21cut08:55:221
37440799140,13cyclictest825160-21ntpq12:10:221
37440799140,13cyclictest0-21swapper/108:00:131
374403991412,1cyclictest556674-21kworker/0:1+kacpid10:50:500
374403991412,1cyclictest487910-21kworker/0:0+kacpid09:13:020
374403991412,1cyclictest487910-21kworker/0:0+kacpid08:45:030
374403991411,2cyclictest812261-21grep12:05:160
374403991411,2cyclictest787734-21fschecks_count11:50:220
374403991411,2cyclictest746914-21tr11:25:150
374403991411,2cyclictest699113-21irqrtprio10:55:190
374403991411,2cyclictest618877-21kthreadcore10:05:220
374403991411,2cyclictest599617-21/usr/sbin/munin09:55:220
374403991411,2cyclictest512313-21timerwakeupswit08:55:330
374403991411,2cyclictest446450-21kernelversion08:05:170
374403991411,2cyclictest430008-21kthreadcore07:50:240
374403991411,2cyclictest388822-21latency_hist07:20:000
374403991411,2cyclictest0-21swapper/010:30:260
37440399141,11cyclictest171ktimers/011:30:150
374403991410,3cyclictest534070-21gltestperf09:15:170
374403991410,2cyclictest828663-21irqcore12:15:180
374403991410,2cyclictest457332-21ls08:15:010
374403991410,2cyclictest0-21swapper/008:50:210
37440399140,2cyclictest5141881irqrtprio09:00:200
374412991311,1cyclictest0-21swapper/312:30:173
374412991311,1cyclictest0-21swapper/309:05:013
374412991310,1cyclictest0-21swapper/311:40:093
374412991310,1cyclictest0-21swapper/311:20:253
374412991310,1cyclictest0-21swapper/309:50:183
374412991310,1cyclictest0-21swapper/309:50:173
374412991310,1cyclictest0-21swapper/309:30:173
374412991310,1cyclictest0-21swapper/308:30:013
374412991310,1cyclictest0-21swapper/308:00:173
37440999139,3cyclictest747133-21gltestperf11:25:172
37440999139,3cyclictest374943-21gltestperf07:10:172
37440999139,2cyclictest0-21swapper/210:55:002
37440999136,5cyclictest547-21gdbus07:20:002
37440999135,6cyclictest1-21systemd09:30:012
37440999131,2cyclictest471rcuog/208:50:182
374409991310,2cyclictest524783-21unixbench_multi09:05:292
374409991310,2cyclictest500473-21systemd-run08:50:002
374409991310,2cyclictest0-21swapper/210:45:002
374407991310,2cyclictest714212-21latency_hist11:05:011
374407991310,2cyclictest641136-21gltestperf10:20:201
374407991310,2cyclictest485120-21tr08:35:221
374407991310,2cyclictest380093-21unixbench_multi07:10:261
37440799130,1cyclictest0-21swapper/110:42:461
37440399139,3cyclictest457767-21gltestperf08:15:180
37440399138,3cyclictest712893-21ssh11:02:240
374403991311,1cyclictest556674-21kworker/0:1+kacpid11:10:410
374403991310,2cyclictest469569-21true08:25:010
374403991310,2cyclictest444743-21kworker/0:1+kacpid08:32:030
374403991310,2cyclictest413540-21kworker/0:2+kacpid07:57:280
374403991310,2cyclictest413540-21kworker/0:2+kacpid07:57:280
374403991310,2cyclictest413540-21kworker/0:2+kacpid07:40:260
374403991310,2cyclictest407813-21latency_hist07:35:010
374403991310,1cyclictest0-21swapper/009:50:170
374403991310,1cyclictest0-21swapper/009:50:170
37441299128,1cyclictest0-21swapper/310:10:133
37440999129,2cyclictest774673-21kthreadcore11:40:212
37440999129,2cyclictest0-21swapper/210:35:262
37440999128,3cyclictest1-21systemd09:00:012
37440999128,2cyclictest0-21swapper/208:20:002
37440799129,2cyclictest390817-21kthreadcore07:20:191
37440799128,3cyclictest833662-21unixbench-2d12:15:261
37440799126,1cyclictest191rcu_preempt11:20:241
37440799120,4cyclictest0-21swapper/111:25:191
37440799120,1cyclictest0-21swapper/108:15:281
37440399129,2cyclictest521329-21kthreadcore09:05:240
374403991210,1cyclictest395585-21cpu07:25:140
37440799112,3cyclictest782557-21kthreadcore11:45:191
37440799112,3cyclictest782557-21kthreadcore11:45:191
37440399118,2cyclictest488501-21grep08:40:200
37440399116,3cyclictest374788-21diskstats07:10:160
37441299108,1cyclictest0-21swapper/307:10:023
37440799108,1cyclictest0-21swapper/107:10:011
37440399107,1cyclictest0-21swapper/007:10:010
3744099995,2cyclictest374326-21latency_hist07:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional