You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-20 - 20:13
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8.osadl.org (updated Mon Oct 20, 2025 12:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9405212300,1sleep1940525-21cut10:50:201
626249992819,8cyclictest1011379-21sh11:35:011
7245332270,1chrt0-21swapper/108:25:171
626249992714,7cyclictest0-21swapper/109:40:011
626249992420,2cyclictest922474-21/usr/sbin/munin10:40:001
626253992320,2cyclictest954665-21idleruntime-cro11:00:002
62624999235,13cyclictest769-21runrttasks11:16:581
626249992319,2cyclictest671801-21munin-run07:45:001
626249992316,6cyclictest849581-21sh09:55:001
62624999230,12cyclictest1019908-21df11:40:131
626253992219,2cyclictest776872-21cut09:10:002
626253992219,2cyclictest764720-21latency_hist09:00:002
626253992218,3cyclictest0-21swapper/210:19:592
626249992218,2cyclictest776856-21diskmemload11:50:011
62624999221,11cyclictest776856-21diskmemload09:43:241
62624999221,10cyclictest776856-21diskmemload10:27:371
626249992210,11cyclictest501-21dbus-daemon07:55:001
62624999220,17cyclictest993850-21chrt11:21:501
62624999220,11cyclictest1043068-21ssh11:53:161
626249992118,2cyclictest1060022-21sh12:04:591
626249992114,6cyclictest0-21swapper/112:20:001
626249992113,7cyclictest969103-21unixbench-2d11:05:591
62624999211,15cyclictest937405-21sh10:47:311
62624999211,15cyclictest937405-21sh10:47:301
62624999211,10cyclictest971748-21hddtemp_smartct11:10:161
62624999210,12cyclictest961823-21taskset11:03:041
62624999210,11cyclictest923267-21interrupts10:40:161
62624999210,11cyclictest856451-21runrttasks09:57:271
62624999210,11cyclictest816506-21taskset09:33:241
62624999210,10cyclictest912859-21taskset10:31:441
62624999210,10cyclictest864506-21taskset10:02:201
62624999210,10cyclictest752495-21/usr/sbin/munin08:50:271
62624699219,2cyclictest653304-21cpuspeed_turbos07:30:120
626246992118,2cyclictest1061328-21irqrtprio12:05:170
626253992016,3cyclictest876375-21kthreadcore10:10:192
626249992012,7cyclictest833848-21cpuspeed_turbos09:45:151
626249992012,7cyclictest1085804-21irqcore12:20:151
626249992012,6cyclictest7550irq/9-acpi07:10:141
626249992011,8cyclictest789745-21kthreadcore09:15:201
62624999201,14cyclictest764223-21taskset08:56:021
626249992010,9cyclictest500-21avahi-daemon10:09:591
62624999200,9cyclictest714143-21needreboot08:15:211
62624999200,15cyclictest1106318-21ssh12:30:251
62624999200,10cyclictest891914-21uname10:20:171
62624999200,10cyclictest1050510-21sh11:56:541
626246992017,2cyclictest522-21in:imuxsock11:10:000
626253991917,1cyclictest1051928-21sendmail12:00:002
626253991915,3cyclictest790887-21unixbench-2d09:15:252
62624999199,9cyclictest501-21dbus-daemon07:40:011
62624999198,9cyclictest501-21dbus-daemon12:39:591
626249991916,2cyclictest677930-21cron07:50:011
626249991915,3cyclictest1017393-21sshd11:35:321
626249991911,7cyclictest745184-21needreboot08:40:181
626249991910,8cyclictest947297-21gltestperf10:55:151
62624999190,9cyclictest783211-21taskset09:11:061
62624999190,9cyclictest769-21runrttasks12:05:171
62624999190,9cyclictest645967-21/usr/sbin/munin07:20:221
62624999190,10cyclictest889748-21taskset10:19:101
62624999190,10cyclictest776856-21diskmemload09:28:211
626246991916,2cyclictest842346-21irqcore09:50:190
626246991916,2cyclictest0-21swapper/012:04:590
62625399183,2cyclictest191rcu_preempt07:35:242
626253991815,2cyclictest995272-21/usr/sbin/munin11:25:012
626253991815,2cyclictest761-21cups-browsed09:40:012
626253991815,2cyclictest0-21swapper/208:29:592
626253991814,3cyclictest985012-21unixbench_multi11:15:252
626253991813,3cyclictest635178-21grep07:15:142
62624999186,10cyclictest1000473-21cut11:25:191
626249991810,7cyclictest0-21swapper/108:35:241
62624999180,9cyclictest690430-21taskset07:59:221
62624999180,14cyclictest752231-21chrt08:49:541
62624999180,11cyclictest776509-21sshd09:05:401
626246991815,2cyclictest0-21swapper/011:20:000
626246991815,2cyclictest0-21swapper/008:50:010
626246991814,3cyclictest1009214-21unixbench_singl11:30:230
626246991812,2cyclictest191rcu_preempt11:35:170
626246991810,1cyclictest191rcu_preempt09:20:240
626255991714,1cyclictest0-21swapper/312:23:303
62625399174,3cyclictest191rcu_preempt08:40:002
626253991715,1cyclictest0-21swapper/208:00:002
626253991714,2cyclictest906335-21sh10:29:592
626253991714,2cyclictest845733-21kthreadcore09:50:202
626253991714,2cyclictest524-21rs:main2
626253991714,2cyclictest0-21swapper/209:45:002
626253991711,2cyclictest581rcuop/310:05:162
62624999179,7cyclictest708918-21sshd08:14:271
62624999179,7cyclictest0-21swapper/107:35:001
62624999175,10cyclictest701433-21cut08:05:191
62624999173,6cyclictest471rcuog/207:17:311
626249991713,3cyclictest770179-21grep09:00:231
62624999170,9cyclictest728208-21taskset08:30:161
62624699178,1cyclictest471rcuog/210:49:550
62624699178,1cyclictest471rcuog/210:49:540
626246991714,2cyclictest959990-21needreboot11:00:200
626246991714,2cyclictest817947-21gltestperf09:35:150
626246991714,2cyclictest809482-21/usr/sbin/munin09:30:140
626246991714,2cyclictest703025-21latency_hist08:10:000
626246991714,2cyclictest0-21swapper/011:50:010
626246991714,2cyclictest0-21swapper/008:30:000
626246991713,3cyclictest1077152-21gltestperf12:15:150
626246991710,1cyclictest191rcu_preempt07:55:020
626255991613,1cyclictest0-21swapper/311:55:013
626255991613,1cyclictest0-21swapper/311:50:003
626255991613,1cyclictest0-21swapper/309:35:133
626253991614,1cyclictest0-21swapper/207:30:012
626253991613,2cyclictest1081817-21missed_timers12:15:202
626253991612,3cyclictest1073493-21kthreadcore12:10:182
626253991612,2cyclictest0-21swapper/211:05:012
62624999169,6cyclictest0-21swapper/107:29:561
62624999164,10cyclictest691227-21cat08:00:171
626249991610,5cyclictest878804-21kthreadcore10:10:201
626246991613,2cyclictest973411-21kthreadcore11:10:180
626246991613,2cyclictest863099-21ntp_kernel_pll_10:00:200
626246991613,2cyclictest760053-21sed08:55:180
626246991613,2cyclictest732698-21latency08:30:210
626246991613,2cyclictest691090-21gltestperf08:00:160
626246991613,2cyclictest665636-21sendmail07:40:010
626246991613,2cyclictest626319-21latency_hist07:10:010
626246991613,2cyclictest500-21avahi-daemon08:35:220
626246991613,2cyclictest1044462-21fschecks_time11:55:170
626246991613,2cyclictest1036452-21gltestperf11:50:160
626255991512,1cyclictest0-21swapper/312:35:003
626255991512,1cyclictest0-21swapper/309:10:193
626255991512,1cyclictest0-21swapper/307:30:213
62625399159,4cyclictest3818356-21systemd-journal08:15:002
62625399154,3cyclictest0-21swapper/210:40:002
62625399154,2cyclictest191rcu_preempt12:25:122
626253991512,2cyclictest968979-21timerandwakeup11:05:582
626253991512,2cyclictest854779-21latency09:55:182
626253991512,2cyclictest776856-21diskmemload09:10:212
626253991512,2cyclictest714660-21unixbench_multi08:15:252
626253991512,2cyclictest1006643-21kthreadcore11:30:182
626253991512,2cyclictest0-21swapper/212:24:592
626253991512,2cyclictest0-21swapper/210:35:002
626253991512,2cyclictest0-21swapper/207:50:012
626253991510,4cyclictest807079-21sshd09:25:272
62624999158,5cyclictest1098659-21sshd12:25:321
62624999154,10cyclictest1071160-21tr12:10:161
62624999151,11cyclictest793665-21cut09:20:161
62624699155,5cyclictest1027474-21munin-run11:45:000
62624699153,2cyclictest9405252cut10:50:200
626246991513,1cyclictest801663-21tr09:25:160
626246991513,1cyclictest776856-21diskmemload12:32:290
626246991513,1cyclictest709363-21tr08:15:140
626246991513,1cyclictest0-21swapper/009:50:000
62624699151,2cyclictest191rcu_preempt10:35:000
626246991512,2cyclictest947249-21cut10:55:150
626246991512,2cyclictest901602-21sort10:25:170
626246991512,2cyclictest866516-21grep10:05:150
626246991512,2cyclictest780239-21kthreadcore09:10:200
626246991512,2cyclictest776856-21diskmemload10:13:580
626246991512,2cyclictest750-21sshd11:26:090
626246991512,2cyclictest646743-21cron07:25:000
626246991512,2cyclictest639618-21needreboot07:15:180
626246991512,2cyclictest1112387-21kthreadcore12:35:170
626246991512,2cyclictest1098522-21unixbench_multi12:25:230
626246991510,2cyclictest471rcuog/209:55:140
62625599141,2cyclictest8579852sleep310:00:113
626255991411,1cyclictest0-21swapper/312:05:153
626255991411,1cyclictest0-21swapper/310:25:173
626255991411,1cyclictest0-21swapper/310:13:113
626255991411,1cyclictest0-21swapper/309:58:123
626255991411,1cyclictest0-21swapper/309:25:153
626255991411,1cyclictest0-21swapper/309:21:513
626255991411,1cyclictest0-21swapper/308:10:193
626255991411,1cyclictest0-21swapper/307:25:193
62625599140,1cyclictest0-21swapper/310:50:003
62625599140,1cyclictest0-21swapper/310:49:593
62625399144,2cyclictest191rcu_preempt11:45:012
62625399141,2cyclictest191rcu_preempt10:25:012
626253991412,1cyclictest857978-21runrttasks10:00:092
626253991412,1cyclictest0-21swapper/210:40:172
626253991411,2cyclictest931966-21irqrtprio10:45:172
626253991411,2cyclictest931966-21irqrtprio10:45:162
626253991411,2cyclictest757908-21unixbench-2d08:50:252
626253991411,2cyclictest644340-21kthreadcore07:20:192
626253991411,2cyclictest1-21systemd08:48:252
626253991411,2cyclictest1111835-21kthreadcore12:35:172
626253991411,2cyclictest1068109-21apt-key12:10:002
626253991411,2cyclictest1057807-21unixbench-2d12:00:262
626253991411,2cyclictest0-21swapper/207:45:002
62624999144,8cyclictest716671-21cut08:20:181
62624699149,1cyclictest191rcu_preempt09:15:140
62624699148,4cyclictest752688-21diskstats08:50:140
626246991412,1cyclictest776856-21diskmemload11:24:380
626246991412,1cyclictest1068660-21/usr/sbin/munin12:10:170
626246991411,2cyclictest923302-21irqcore10:40:150
626246991411,2cyclictest887244-21kthreadcore10:15:190
626246991411,2cyclictest776856-21diskmemload12:20:190
626246991411,2cyclictest666140-21cut07:40:150
626246991411,2cyclictest649973-21kthreadcore07:25:200
626246991410,3cyclictest775962-21ntp_kernel_pll_09:05:210
6203052140,2sleep30-21swapper/307:05:133
626255991311,1cyclictest0-21swapper/312:11:083
626255991310,1cyclictest0-21swapper/312:26:163
626255991310,1cyclictest0-21swapper/312:15:223
626255991310,1cyclictest0-21swapper/312:00:433
626255991310,1cyclictest0-21swapper/311:59:313
626255991310,1cyclictest0-21swapper/311:42:043
626255991310,1cyclictest0-21swapper/311:37:053
626255991310,1cyclictest0-21swapper/311:30:333
626255991310,1cyclictest0-21swapper/311:25:363
626255991310,1cyclictest0-21swapper/311:23:193
626255991310,1cyclictest0-21swapper/311:17:433
626255991310,1cyclictest0-21swapper/311:12:313
626255991310,1cyclictest0-21swapper/311:01:453
626255991310,1cyclictest0-21swapper/310:55:323
626255991310,1cyclictest0-21swapper/310:36:023
626255991310,1cyclictest0-21swapper/310:30:213
626255991310,1cyclictest0-21swapper/310:20:193
626255991310,1cyclictest0-21swapper/310:15:203
626255991310,1cyclictest0-21swapper/309:50:393
626255991310,1cyclictest0-21swapper/309:46:483
626255991310,1cyclictest0-21swapper/309:44:283
626255991310,1cyclictest0-21swapper/309:15:233
626255991310,1cyclictest0-21swapper/308:55:483
626255991310,1cyclictest0-21swapper/308:50:253
626255991310,1cyclictest0-21swapper/308:46:453
626255991310,1cyclictest0-21swapper/308:37:113
626255991310,1cyclictest0-21swapper/308:27:263
626255991310,1cyclictest0-21swapper/308:23:263
626255991310,1cyclictest0-21swapper/308:18:353
626255991310,1cyclictest0-21swapper/308:05:103
626255991310,1cyclictest0-21swapper/307:56:113
626255991310,1cyclictest0-21swapper/307:52:443
626255991310,1cyclictest0-21swapper/307:48:443
626255991310,1cyclictest0-21swapper/307:42:063
626255991310,1cyclictest0-21swapper/307:35:113
626255991310,1cyclictest0-21swapper/307:18:333
626255991310,1cyclictest0-21swapper/307:15:013
62625399138,4cyclictest836242-21tr09:45:202
62625399138,4cyclictest1041009-21grep11:50:212
62625399133,2cyclictest191rcu_preempt11:25:132
62625399132,3cyclictest0-21swapper/209:31:082
62625399131,2cyclictest793663-21tune2fs09:20:162
626253991311,1cyclictest699622-21kthreadcore08:05:192
626253991311,1cyclictest658201-21open_files07:30:202
626253991311,1cyclictest0-21swapper/209:00:202
626253991310,2cyclictest695290-21kthreadcore08:00:192
626253991310,2cyclictest679766-21kthreadcore07:50:152
626253991310,2cyclictest547-21gdbus12:32:022
626253991310,2cyclictest0-21swapper/211:10:162
626253991310,2cyclictest0-21swapper/208:40:112
62624999130,11cyclictest626275-21date07:10:011
62624699137,4cyclictest720892-21users08:20:270
626246991311,1cyclictest739853-21kworker/0:2+kacpid09:00:120
626246991311,1cyclictest0-21swapper/007:15:000
626246991310,2cyclictest776856-21diskmemload10:35:190
626246991310,2cyclictest743638-21kthreadcore08:40:160
626246991310,2cyclictest687778-21kthreadcore07:55:180
626246991310,2cyclictest673060-21irqrtprio07:45:180
62624699130,2cyclictest191rcu_preempt09:40:260
62625599129,1cyclictest0-21swapper/311:05:103
62625599129,1cyclictest0-21swapper/310:50:073
62625599129,1cyclictest0-21swapper/310:40:183
62625599129,1cyclictest0-21swapper/309:00:073
62625599129,1cyclictest0-21swapper/308:30:033
62625599129,1cyclictest0-21swapper/308:00:093
62625599121,2cyclictest7709562sleep309:05:123
626255991210,1cyclictest0-21swapper/312:36:243
626255991210,1cyclictest0-21swapper/308:41:053
62625399129,2cyclictest940331-21kthreadcore10:50:202
62625399129,2cyclictest1032945-21ssh11:45:222
62625399129,2cyclictest1014264-21kthreadcore11:35:182
62625399129,2cyclictest0-21swapper/207:10:002
62624699129,2cyclictest776856-21diskmemload10:20:070
62625599119,1cyclictest0-21swapper/310:05:013
62625599119,1cyclictest0-21swapper/307:20:033
62625599118,1cyclictest0-21swapper/309:30:013
62625399118,2cyclictest729853-21kthreadcore08:30:192
62625399110,2cyclictest191rcu_preempt08:20:142
62624699118,2cyclictest0-21swapper/008:10:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional