You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-22 - 03:47
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8.osadl.org (updated Sun Jul 21, 2024 12:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
413295421440,1sleep30-21swapper/308:50:183
1878992960,1sleep10-21swapper/111:35:291
773662320,1sleep30-21swapper/310:25:173
2816722300,1chrt0-21swapper/312:40:003
40142562290,1sleep00-21swapper/007:15:010
4006793992723,3cyclictest59828-21sh10:15:002
4006789992521,2cyclictest0-21swapper/112:30:001
1500612250,1chrt0-21swapper/011:10:390
4006795992420,1cyclictest0-21swapper/307:40:003
4006793992422,1cyclictest281711-21sendmail12:40:002
4006786992420,3cyclictest514-21in:imuxsock08:40:000
4006793992320,2cyclictest113059-21idleruntime-cro10:50:002
4006795992218,1cyclictest0-21swapper/310:20:003
4006789992219,2cyclictest0-21swapper/109:15:011
4006789992217,3cyclictest152166-21grep11:15:121
4006795992118,1cyclictest0-21swapper/311:05:003
4006795992118,1cyclictest0-21swapper/310:10:003
4006786992118,2cyclictest0-21swapper/009:25:000
4006786992116,3cyclictest4012839-21latency_hist07:15:000
4006795992017,1cyclictest0-21swapper/311:35:003
4006795992017,1cyclictest0-21swapper/310:05:003
4006793992014,4cyclictest220626-21ssh12:00:012
4006789992017,2cyclictest497-21dbus-daemon12:15:001
4006789992017,2cyclictest0-21swapper/110:25:001
4006789992015,4cyclictest113089-21http10:50:001
4006786992016,3cyclictest496-21avahi-daemon08:25:000
4006786992015,3cyclictest27456-21kworker/u8:1+events_unbound10:55:020
4006795991916,1cyclictest0-21swapper/310:40:013
4006793991916,2cyclictest182356-21apt-key11:35:002
4006793991915,2cyclictest0-21swapper/210:00:222
4006793991915,2cyclictest0-21swapper/208:55:012
4006789991916,2cyclictest4099851-21latency_hist08:25:011
4006789991916,2cyclictest0-21swapper/111:00:001
4006786991915,2cyclictest30797-21irqrtprio09:55:220
4006786991911,3cyclictest581rcuop/311:45:000
102198190,3rtkit-daemon722-21ntpd12:21:590
102198190,1rtkit-daemon0-21swapper/012:05:570
4006793991815,2cyclictest516-21rs:main2
4006793991815,2cyclictest236533-21gltestperf12:10:122
4006793991815,2cyclictest0-21swapper/209:00:012
400679399180,2cyclictest4148020-21munin-run09:05:012
4006789991815,2cyclictest773-21gmain10:15:021
4006789991815,2cyclictest51948-21sh10:10:011
4006789991815,2cyclictest331ktimers/110:34:001
4006789991815,2cyclictest251192-21latency_hist12:20:001
4006786991816,1cyclictest274058-21cron12:35:010
4006786991815,2cyclictest729-21sshd09:03:280
4006786991814,3cyclictest4047000-21kthreadcore07:40:160
102198180,2rtkit-daemon722-21ntpd11:47:250
4006795991714,1cyclictest0-21swapper/312:15:003
4006795991714,1cyclictest0-21swapper/310:25:003
4006795991714,1cyclictest0-21swapper/309:20:003
4006795991714,1cyclictest0-21swapper/309:00:023
4006795991714,1cyclictest0-21swapper/308:00:003
400679399172,2cyclictest41526542tr09:05:182
4006793991714,2cyclictest496-21avahi-daemon10:30:012
4006793991714,2cyclictest0-21swapper/210:20:022
4006793991713,3cyclictest514-21in:imuxsock12:10:012
4006793991713,3cyclictest4045812-21fschecks_count07:40:132
4006793991713,3cyclictest144430-21/usr/sbin/munin11:10:252
4006793991713,3cyclictest0-21swapper/208:25:002
4006793991713,2cyclictest4069477-21sendmail08:00:002
4006789991714,2cyclictest496-21avahi-daemon07:15:291
4006789991713,2cyclictest177769-21kthreadcore11:30:161
400678699178,8cyclictest0-21swapper/009:15:230
4006786991714,2cyclictest236674-21irqcore12:10:130
4006786991714,2cyclictest137152-21gltestperf11:05:130
4006786991713,3cyclictest4039085-21latency_hist07:35:010
4006786991712,3cyclictest51949-21latency_hist10:10:010
4006795991613,1cyclictest0-21swapper/312:05:173
4006795991613,1cyclictest0-21swapper/311:55:163
4006795991613,1cyclictest0-21swapper/310:55:153
4006795991613,1cyclictest0-21swapper/310:30:533
4006795991613,1cyclictest0-21swapper/310:15:003
4006795991613,1cyclictest0-21swapper/309:55:283
4006795991613,1cyclictest0-21swapper/309:55:003
4006795991613,1cyclictest0-21swapper/309:40:003
4006795991613,1cyclictest0-21swapper/308:35:003
4006795991613,1cyclictest0-21swapper/308:20:253
4006795991612,1cyclictest0-21swapper/312:30:003
400679399166,3cyclictest205299-21awk11:50:002
4006793991613,2cyclictest4093854-21latency_hist08:20:012
4006793991613,2cyclictest4032883-21cut07:30:002
4006793991613,2cyclictest136518-21sh11:05:002
4006793991613,2cyclictest11514-21latency09:40:172
4006793991613,2cyclictest0-21swapper/208:10:002
4006793991612,3cyclictest516-21rs:main2
4006793991612,3cyclictest4154986-21gltestperf09:10:132
4006793991612,3cyclictest103662-21unixbench_singl10:40:252
4006793991612,2cyclictest167098-21latency_hist11:25:002
4006793991612,2cyclictest0-21swapper/208:15:002
4006793991611,3cyclictest22143-21sh09:50:152
4006793991610,5cyclictest46107-21kthreadcore10:05:192
4006789991613,2cyclictest729-21sshd12:22:271
4006789991613,2cyclictest4076010-21fschecks_count08:05:141
4006789991610,5cyclictest50340-21kworker/u8:0-rpciod10:25:231
400678699162,12cyclictest4185395-21ssh09:30:110
4006786991613,2cyclictest851-21cron08:45:000
4006786991613,2cyclictest752-21runrttasks10:00:170
4006786991613,2cyclictest75058-21idleruntime-cro10:25:000
4006786991613,2cyclictest4069697-21latency_hist08:00:010
4006786991613,2cyclictest220637-21systemd-run12:00:000
4006786991613,2cyclictest105544-21latency_hist10:45:010
4006786991612,3cyclictest182450-21apt-key11:35:010
4006786991612,2cyclictest0-21swapper/010:15:170
4006786991612,2cyclictest0-21swapper/009:55:000
4006786991612,2cyclictest0-21swapper/008:50:010
400679599152,10cyclictest190016-21turbostat11:44:553
4006795991513,1cyclictest0-21swapper/312:00:113
4006795991512,1cyclictest0-21swapper/311:50:233
4006795991512,1cyclictest0-21swapper/311:35:223
4006795991512,1cyclictest0-21swapper/311:10:593
4006795991512,1cyclictest0-21swapper/311:05:153
4006795991512,1cyclictest0-21swapper/309:47:453
4006795991512,1cyclictest0-21swapper/309:32:353
4006795991512,1cyclictest0-21swapper/309:05:013
4006795991512,1cyclictest0-21swapper/308:40:163
4006795991512,1cyclictest0-21swapper/307:45:003
4006795991512,1cyclictest0-21swapper/307:13:123
4006795991511,2cyclictest0-21swapper/311:45:243
4006793991513,1cyclictest4039314-21latency_hist07:35:012
4006793991512,2cyclictest4100336-21hddtemp_smartct08:25:142
4006793991512,2cyclictest244033-21fschecks_time12:15:122
4006793991512,2cyclictest113552-21gdbus10:50:012
4006793991511,3cyclictest90842-21fschecks_time10:35:132
4006793991511,3cyclictest735-21cups-browsed07:40:002
4006793991511,3cyclictest729-21sshd08:35:282
4006793991511,3cyclictest155282-21kthreadcore11:15:172
4006793991511,3cyclictest137526-21irqcore11:05:142
4006793991510,4cyclictest4111630-21sshd08:33:272
400678999159,5cyclictest4154350-21diskmemload11:10:161
4006789991513,1cyclictest0-21swapper/107:30:001
4006789991512,2cyclictest98741-21cut10:40:151
4006789991512,2cyclictest773-21gmain11:41:341
4006789991512,2cyclictest4082767-21tr08:10:171
4006789991512,2cyclictest4074556-21needreboot08:00:191
4006789991512,2cyclictest4038464-21unixbench_multi07:30:261
4006789991512,2cyclictest11695-21ntp_kernel_err09:40:181
4006789991512,2cyclictest0-21swapper/111:55:151
4006789991511,3cyclictest7550irq/9-acpi09:25:091
4006789991511,3cyclictest4190340-21ntp_kernel_err09:30:201
4006789991511,3cyclictest4129071-21proc_pri08:45:221
4006789991511,3cyclictest4068469-21missed_timers07:55:171
4006789991511,3cyclictest4045032-21file07:37:191
4006789991511,2cyclictest4093649-21idleruntime-cro08:20:001
4006789991511,2cyclictest29547-21acpi09:55:151
400678699159,4cyclictest361-21systemd-journal09:44:270
4006786991512,2cyclictest4060555-21kthreadcore07:50:170
4006786991512,2cyclictest4042816-21kthreadcore07:35:190
4006786991512,2cyclictest0-21swapper/009:50:000
4006786991512,2cyclictest0-21swapper/008:55:010
4006786991511,3cyclictest176254-21kthreadcore11:30:160
4006786991510,4cyclictest210477-21wc11:50:190
4006786991510,3cyclictest174528-21turbostat11:30:000
4006795991412,1cyclictest0-21swapper/311:15:123
4006795991412,1cyclictest0-21swapper/307:15:183
4006795991411,1cyclictest0-21swapper/312:30:163
4006795991411,1cyclictest0-21swapper/312:20:123
4006795991411,1cyclictest0-21swapper/311:25:173
4006795991411,1cyclictest0-21swapper/309:40:163
4006795991411,1cyclictest0-21swapper/309:20:163
4006795991411,1cyclictest0-21swapper/308:35:183
4006795991411,1cyclictest0-21swapper/308:15:173
4006795991411,1cyclictest0-21swapper/308:00:163
4006795991411,1cyclictest0-21swapper/307:25:143
4006793991411,2cyclictest514-21in:imuxsock07:55:012
4006793991411,2cyclictest266742-21/usr/sbin/munin12:30:152
4006793991411,2cyclictest221768-21irqcore12:00:152
4006793991411,2cyclictest0-21swapper/210:55:172
4006793991411,2cyclictest0-21swapper/209:20:142
4006793991410,3cyclictest68459-21irqcore10:20:172
4006793991410,3cyclictest4040-21needreboot09:35:192
400678999148,4cyclictest210682-21ntpq11:50:211
4006789991412,2cyclictest331ktimers/110:02:001
4006789991412,1cyclictest0-21swapper/108:25:131
4006789991411,2cyclictest4177536-21sh09:25:011
4006789991411,2cyclictest4007136-21fschecks_count07:10:131
4006789991411,2cyclictest19575-21threads09:45:231
4006789991411,2cyclictest167541-21fschecks_time11:25:141
4006789991411,2cyclictest0-21swapper/112:05:001
4006789991411,2cyclictest0-21swapper/109:35:141
4006789991411,1cyclictest50340-21kworker/u8:0-rpciod10:50:231
4006789991411,1cyclictest0-21swapper/107:20:131
4006789991410,3cyclictest165060-21unixbench-2d11:20:251
4006789991410,2cyclictest0-21swapper/109:15:111
400678999140,12cyclictest205472-21latency_hist11:50:011
400678699147,6cyclictest1-21systemd10:25:230
400678699145,8cyclictest0-21swapper/011:19:010
4006786991412,1cyclictest4032711-21kworker/0:2+kacpid09:06:460
4006786991412,1cyclictest4032711-21kworker/0:2+kacpid08:15:520
4006786991412,1cyclictest219927-21kworker/0:0+kacpid12:00:190
4006786991412,1cyclictest132555-21kworker/0:0+kacpid11:21:230
4006786991411,2cyclictest57483-21ntp_kernel_pll_10:10:190
4006786991411,2cyclictest496-21avahi-daemon07:25:000
4006786991411,2cyclictest4180372-21kthreadcore09:25:160
4006786991411,2cyclictest4154658-21/usr/sbin/munin09:10:180
4006786991411,2cyclictest4142169-21latency_hist09:00:010
4006786991411,2cyclictest3969796-21kworker/0:1+kacpid07:25:100
4006786991411,2cyclictest110395-21kthreadcore10:45:180
4006786991411,2cyclictest0-21swapper/012:25:220
400678699141,11cyclictest90832-21ssh10:35:130
4006786991410,3cyclictest83219-21tune2fs10:30:120
4006786991410,3cyclictest4110946-21proc_pri08:30:220
4006786991410,2cyclictest4075128-21unixbench_singl08:00:250
400679599139,1cyclictest0-21swapper/307:50:003
4006795991310,1cyclictest0-21swapper/312:19:563
4006795991310,1cyclictest0-21swapper/311:25:013
4006795991310,1cyclictest0-21swapper/310:45:503
4006795991310,1cyclictest0-21swapper/310:40:203
4006795991310,1cyclictest0-21swapper/309:25:143
4006795991310,1cyclictest0-21swapper/308:45:173
4006795991310,1cyclictest0-21swapper/307:25:003
400679399139,3cyclictest194528-21kthreadcore11:40:192
400679399138,4cyclictest172283-21ntpq11:25:212
400679399138,4cyclictest0-21swapper/210:30:102
400679399137,4cyclictest361-21systemd-journal07:20:002
400679399137,4cyclictest258800-21cron12:25:002
4006793991310,2cyclictest4022556-21kthreadcore07:20:202
4006793991310,2cyclictest30952-21kernelversion09:55:222
4006793991310,2cyclictest262968-21kthreadcore12:25:232
400678999139,3cyclictest4110884-21ntp_kernel_pll_08:30:211
400678999138,3cyclictest4147718-21sshd09:01:271
400678999138,3cyclictest4056563-21ntpq07:45:211
400678999136,5cyclictest57813-21sed10:10:231
4006789991311,1cyclictest0-21swapper/112:36:031
4006789991310,2cyclictest4045588-21/usr/sbin/munin07:40:171
4006789991310,2cyclictest267478-21irqcore12:30:201
4006789991310,2cyclictest0-21swapper/110:35:161
400678999130,12cyclictest4060805-21kthreadcore07:50:161
400678999130,12cyclictest129217-21apt-get11:00:181
400678699139,3cyclictest4105137-21timerandwakeup08:25:240
400678699139,3cyclictest277313-21kthreadcore12:35:170
400678699131,1cyclictest41935221irqcore09:35:150
4006786991311,1cyclictest4032711-21kworker/0:2+kacpid07:45:140
4006786991311,1cyclictest21387-21kworker/0:1+kacpid10:55:020
4006786991310,2cyclictest4032711-21kworker/0:2+kacpid08:10:210
4006786991310,2cyclictest0-21swapper/012:15:100
400678699130,3cyclictest1358992ssh11:03:330
400679599129,1cyclictest0-21swapper/309:11:423
400679599129,1cyclictest0-21swapper/308:05:273
4006795991210,1cyclictest0-21swapper/309:10:003
4006795991210,1cyclictest0-21swapper/307:35:003
400679399129,2cyclictest0-21swapper/209:30:142
400679399129,2cyclictest0-21swapper/208:45:172
400679399127,3cyclictest4179636-21tr09:25:162
400679399127,3cyclictest184828-21tr11:35:232
400679399121,8cyclictest205278-21cron11:50:002
400679399120,4cyclictest2144250ssh09:49:342
400678999129,2cyclictest4152645-21runrttasks09:05:181
400678999129,2cyclictest4136287-21cpuspeed_turbos08:55:121
400678999129,2cyclictest331ktimers/107:10:001
400678999129,2cyclictest141773-21needreboot11:05:191
400678999128,2cyclictest0-21swapper/108:35:271
4006789991210,1cyclictest0-21swapper/112:07:141
4006786991210,2cyclictest231rcuc/008:05:190
40755912110,1sleep20-21swapper/208:05:012
400679599119,1cyclictest0-21swapper/310:52:473
400679599118,1cyclictest0-21swapper/308:25:143
400679599118,1cyclictest0-21swapper/307:50:143
400679599110,3cyclictest40818102sleep308:10:113
400679399110,2cyclictest4057414-21munin-run07:50:002
400678999118,2cyclictest7550irq/9-acpi09:50:141
400678999118,2cyclictest0-21swapper/108:40:141
400678999110,1cyclictest0-21swapper/108:50:061
400679399105,4cyclictest4121876-21cut08:40:182
4000902291,4sleep30-21swapper/307:05:123
40067869981,5cyclictest4006634-21latency_hist07:10:000
10219880,4rtkit-daemon3970083-21munin-node07:07:372
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional