You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 04:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8s.osadl.org (updated Tue Jul 01, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
392948799746742,3cyclictest3859352-21kworker/2:1+events23:59:212
392949099743738,4cyclictest99024-21kworker/3:2+events00:30:223
392949099743738,4cyclictest4193515-21turbostat22:44:553
392948799742737,4cyclictest3859352-21kworker/2:1+events21:10:212
392949099739735,3cyclictest3891585-21kworker/3:3+events19:36:053
392949099738733,4cyclictest4075050-21kworker/3:2+events21:41:573
392949099736725,5cyclictest4050014-21kworker/3:1+events20:50:203
392948799735730,4cyclictest3859352-21kworker/2:1+events00:15:232
392948799735727,7cyclictest3859352-21kworker/2:1+events20:58:462
392949099734730,3cyclictest4075050-21kworker/3:2+events22:52:333
392949099733728,4cyclictest4050014-21kworker/3:1+events21:09:163
392949099732728,3cyclictest99024-21kworker/3:2+events00:26:253
392948799732728,3cyclictest112037-21kworker/2:2+events00:21:482
392948599732728,3cyclictest4068643-21kworker/1:2+events21:49:201
392948199729723,5cyclictest4161582-21kworker/0:0+events22:11:260
392949099727723,3cyclictest4018666-21kworker/3:2+events20:27:513
392949099727721,5cyclictest4075050-21kworker/3:2+events21:31:473
392948799727718,4cyclictest3859352-21kworker/2:1+events00:04:532
392949099726723,2cyclictest4075050-21kworker/3:2+events23:02:513
392948799726720,5cyclictest3859352-21kworker/2:1+events00:07:562
392949099725719,4cyclictest61675-21kworker/3:1+events23:36:583
392949099724721,2cyclictest4075050-21kworker/3:2+events23:09:513
392949099724721,2cyclictest4075050-21kworker/3:2+events23:09:513
392949099724720,3cyclictest4075050-21kworker/3:2+events22:58:033
392949099724720,3cyclictest4006148-21kworker/3:0+events20:16:063
392948799724718,5cyclictest3859352-21kworker/2:1+events20:35:222
392948199724717,5cyclictest74001-21kworker/0:0+events23:45:220
392949099723719,3cyclictest4075050-21kworker/3:2+events22:16:353
392949099723719,3cyclictest3891585-21kworker/3:3+events19:24:123
392948799723719,3cyclictest3859352-21kworker/2:1+events21:26:262
392949099722719,2cyclictest3891585-21kworker/3:3+events20:03:383
392949099722718,3cyclictest61675-21kworker/3:1+events23:33:453
392949099722718,3cyclictest4006148-21kworker/3:0+events20:21:553
392949099722718,3cyclictest3891585-21kworker/3:3+events19:26:503
392949099722717,4cyclictest4006456-21turbostat20:14:553
392949099722717,4cyclictest4006148-21kworker/3:0+events20:09:283
392948799722717,4cyclictest3859352-21kworker/2:1+events21:01:572
392949099721718,2cyclictest4075050-21kworker/3:2+events22:46:053
392949099721718,2cyclictest4075050-21kworker/3:2+events22:24:213
392949099720716,3cyclictest4075050-21kworker/3:2+events23:21:513
392949099720714,5cyclictest3891585-21kworker/3:3+events19:33:413
392949099718715,2cyclictest3891585-21kworker/3:3+events19:11:083
392949099718713,4cyclictest3975391-21turbostat19:49:563
392949099717713,3cyclictest3891585-21kworker/3:3+events19:19:473
392949099717710,6cyclictest4075050-21kworker/3:2+events23:28:123
392948799717713,3cyclictest3859352-21kworker/2:1+events00:10:322
392949099716713,2cyclictest4075050-21kworker/3:2+events22:28:503
392948799716712,3cyclictest3859352-21kworker/2:1+events21:18:032
392949099715712,2cyclictest4050014-21kworker/3:1+events20:47:333
392949099714711,2cyclictest4018666-21kworker/3:2+events20:31:383
392949099714710,3cyclictest99024-21kworker/3:2+events00:39:313
392949099714710,3cyclictest3891585-21kworker/3:3+events19:53:153
392949099713710,2cyclictest4075050-21kworker/3:2+events23:13:243
392949099713709,3cyclictest3891585-21kworker/3:3+events19:57:303
392949099713709,3cyclictest3891585-21kworker/3:3+events19:40:423
392949099713706,4cyclictest4075050-21kworker/3:2+events23:18:353
392948799713708,4cyclictest3859352-21kworker/2:1+events21:20:222
392948799713707,5cyclictest3859352-21kworker/2:1+events22:37:102
392949099711705,4cyclictest4075050-21kworker/3:2+events22:32:513
392948199711707,3cyclictest67934-21turbostat23:39:550
392948199710704,5cyclictest4130861-21kworker/0:1+events23:40:200
392948799708703,4cyclictest3859352-21kworker/2:1+events20:41:042
392949099705702,2cyclictest4068599-21turbostat21:04:553
392949099705699,3cyclictest4075050-21kworker/3:2+events21:36:593
392948799699695,3cyclictest4031375-21turbostat20:34:552
392948199696687,8cyclictest3993801-21kworker/0:3+events22:00:210
392948199690684,5cyclictest74001-21kworker/0:0+events23:52:530
392948199682679,2cyclictest4149258-21kworker/0:2+events22:07:570
392949099681677,3cyclictest4187293-21turbostat22:39:553
392948599681676,4cyclictest4118549-21turbostat21:44:561
392948199681678,2cyclictest3993801-21kworker/0:3+events21:57:090
392948799676671,4cyclictest4074808-21turbostat21:09:552
392948199676673,2cyclictest4031344-21kworker/0:0+events21:54:370
392948199670664,5cyclictest4124755-21turbostat21:49:550
392948799665660,4cyclictest86525-21turbostat23:54:552
392949099623619,3cyclictest4099934-21turbostat21:29:563
392949099611607,3cyclictest4043797-21turbostat20:44:553
392949099606603,2cyclictest4155716-21turbostat22:14:553
392948799600596,3cyclictest4056192-21turbostat20:54:552
392949099598595,2cyclictest118546-21turbostat00:24:543
392948799598594,3cyclictest4181072-21turbostat22:34:552
3929481994500,447cyclictest3963618-21grep19:35:220
3929487991500,148cyclictest4168191-21taskset22:17:402
41682790,1sleep20-21swapper/222:40:202
22952550,2chrt0-21swapper/122:40:191
388242530,1chrt0-21swapper/223:10:182
39943222320,1sleep0211rcuc/020:00:110
3929485992723,3cyclictest527-21gmain21:10:001
3929490992420,3cyclictest98998-21idleruntime-cro00:00:003
3929481992419,3cyclictest4118574-21sendmail21:40:000
3929481992419,3cyclictest0-21swapper/022:40:000
3929485992213,8cyclictest0-21swapper/121:25:001
3929481992118,2cyclictest118547-21idleruntime-cro00:19:590
3929481992117,2cyclictest0-21swapper/021:10:000
3929490992017,2cyclictest4143329-21sendmail22:00:003
3929487992016,2cyclictest137176-21idleruntime-cro00:35:002
3929485992017,2cyclictest32466-21kthreadcore23:05:201
3929485992017,2cyclictest32466-21kthreadcore23:05:201
3929481992012,2cyclictest4012660-21turbostat20:15:000
3929487991915,2cyclictest0-21swapper/222:15:002
3929485991916,2cyclictest105276-21systemd-run00:05:001
3929485991916,2cyclictest0-21swapper/100:19:591
3929481991917,1cyclictest4087508-21latency_hist21:15:000
3929481991915,3cyclictest0-21swapper/021:20:010
3929481991915,2cyclictest3976107-21irqcore19:45:150
3929481991912,5cyclictest3614154-21systemd-journal23:00:000
448472180,1sleep044854-21sort23:15:170
3929485991815,2cyclictest4065647-21kthreadcore20:55:181
3929485991814,3cyclictest4056195-21sh20:50:001
3929481991815,2cyclictest491-21avahi-daemon20:00:020
3929481991814,2cyclictest12073-21sh22:50:000
3929487991713,2cyclictest0-21swapper/220:25:012
3929487991711,4cyclictest3956026-21grep19:25:262
3929485991714,2cyclictest499-21polkitd21:40:001
3929485991712,3cyclictest4193490-21cron22:40:011
3929485991712,3cyclictest4082045-21irqcore21:10:171
3929481991713,3cyclictest0-21swapper/021:25:000
39421422160,1sleep241-21ksoftirqd/219:15:172
3929490991613,2cyclictest87265-21irqcore23:50:173
392948799167,1cyclictest4130967-21munin-run21:50:002
3929487991611,4cyclictest80310-21cron23:45:002
3929487991611,3cyclictest49325-21cron23:20:002
3929485991612,3cyclictest4175377-21fschecks_time22:25:181
392948199169,5cyclictest3614154-21systemd-journal22:45:000
3929481991613,2cyclictest3975386-21sh19:45:010
3929481991612,3cyclictest4100478-21gltestperf21:25:160
3929481991612,3cyclictest131527-21gltestperf00:30:170
3929481991612,2cyclictest3987818-21turbostat19:55:010
39233872163,7sleep30-21swapper/319:05:133
3929490991512,2cyclictest4125285-21fschecks_time21:45:203
3929490991512,2cyclictest100220-21irqrtprio00:00:173
3929487991512,2cyclictest4030727-21switchtime20:25:252
3929487991511,2cyclictest0-21swapper/200:35:152
392948599153,2cyclictest41486222netstat22:00:201
3929485991512,2cyclictest93457-21irqcore23:55:161
3929485991512,2cyclictest4138870-21kthreadcore21:55:211
3929485991512,2cyclictest3956754-21latency_hist19:30:011
3929485991512,2cyclictest3929475-21cyclictest20:15:001
3929485991512,2cyclictest11609-21unixbench_multi22:45:281
3929485991512,2cyclictest0-21swapper/123:45:171
3929485991512,2cyclictest0-21swapper/100:10:011
3929485991511,3cyclictest56089-21fschecks_time23:25:161
392948199158,5cyclictest3614154-21systemd-journal22:55:010
3929481991512,2cyclictest752418-21kworker/0:1+kacpid19:31:480
3929481991512,2cyclictest4074077-21systemctl21:00:240
3929481991512,2cyclictest30727-21cron23:05:000
3929481991511,3cyclictest4167941-21unixbench_multi22:15:270
39289982155,2sleep20-21swapper/219:05:522
392949099149,3cyclictest0-21swapper/322:10:013
392949099148,4cyclictest493-21dbus-daemon21:00:003
3929490991411,2cyclictest4087093-21unixbench_singl21:10:253
392948799149,3cyclictest3935262-21grep19:10:252
3929487991411,2cyclictest4007183-21sed20:10:182
3929487991410,3cyclictest4118933-21cpuspeed_turbos21:40:152
3929487991410,2cyclictest12685-21tail22:50:162
3929487991410,2cyclictest0-21swapper/222:25:012
3929487991410,2cyclictest0-21swapper/221:40:002
392948599142,7cyclictest753-21runrttasks00:31:421
3929485991411,2cyclictest4131689-21irqcore21:50:171
3929485991411,2cyclictest4049593-21unixbench_singl20:40:241
3929485991411,2cyclictest4006446-21cron20:10:001
3929485991411,2cyclictest3981151-21unixbench_multi19:45:241
3929485991411,2cyclictest0-21swapper/120:05:001
3929485991411,2cyclictest0-21swapper/119:10:151
392948199147,5cyclictest3614154-21systemd-journal19:25:000
392948199147,4cyclictest4118847-21apt-get21:40:130
3929481991412,1cyclictest74001-21kworker/0:0+kacpid00:02:510
3929481991411,2cyclictest4130861-21kworker/0:1+kacpid22:30:190
3929481991411,2cyclictest3993801-21kworker/0:3+kacpid20:15:220
3929481991411,2cyclictest118913-21cpuspeed_turbos00:20:150
3929481991411,2cyclictest0-21swapper/023:10:000
3929481991411,2cyclictest0-21swapper/023:10:000
3929481991411,1cyclictest4177273-21kthreadcore22:25:210
3929481991410,3cyclictest4031719-21cpuspeed_turbos20:30:160
392948199140,13cyclictest3951581-21cut19:25:170
3929490991310,2cyclictest4094086-21cpuspeed_turbos21:20:163
3929490991310,2cyclictest0-21swapper/323:40:163
3929490991310,2cyclictest0-21swapper/321:15:163
392948799139,3cyclictest68275-21cpuspeed_turbos23:35:142
392948799139,3cyclictest0-21swapper/223:00:002
392948799139,2cyclictest4149546-21cron22:05:002
392948799139,2cyclictest0-21swapper/219:45:012
392948799134,2cyclictest3614154-21systemd-journal23:25:002
3929487991310,2cyclictest3999266-21needreboot20:00:192
3929487991310,2cyclictest0-21swapper/219:25:002
392948599139,3cyclictest71022-21cut23:35:181
392948599139,2cyclictest0-21swapper/120:20:001
392948599138,4cyclictest3938473-21irqcore19:15:141
392948599136,5cyclictest67588-21sshd23:30:471
3929485991310,2cyclictest4182302-21cut22:30:191
3929485991310,2cyclictest4107864-21tr21:30:191
3929485991310,2cyclictest0-21swapper/119:35:191
392948599130,2cyclictest49084-21cron23:16:591
392948599130,12cyclictest4167611-21sendmail_mailqu22:15:251
392948199139,3cyclictest491-21avahi-daemon21:00:000
392948199139,3cyclictest0-21swapper/000:15:000
392948199139,2cyclictest4049997-21turbostat20:45:000
392948199138,4cyclictest0-21swapper/020:10:000
392948199137,4cyclictest3614154-21systemd-journal00:40:000
3929481991311,1cyclictest4031344-21kworker/0:0+kacpid20:35:250
3929481991310,2cyclictest92454-21kworker/0:3+kacpid00:05:170
3929481991310,2cyclictest4062398-21systemd-run20:55:000
3929481991310,2cyclictest37189-21/usr/sbin/munin23:10:190
3929481991310,2cyclictest0-21swapper/023:35:000
392948199130,5cyclictest39421462tr19:15:180
392949099129,2cyclictest491ktimers/300:10:013
392949099128,2cyclictest0-21swapper/300:10:103
392948799129,2cyclictest4175472-21gltestperf22:25:182
392948799129,2cyclictest4006047-21head20:05:242
392948799129,2cyclictest131171-21sort00:30:002
392948799129,2cyclictest0-21swapper/219:55:142
392948799127,3cyclictest0-21swapper/219:55:012
392948799126,5cyclictest3980764-21ntpq19:45:212
392948599129,2cyclictest86926-21cpuspeed_turbos23:50:161
392948599129,2cyclictest4056783-21gltestperf20:50:151
392948599129,2cyclictest4030918-21unixbench_multi20:25:261
392948599129,2cyclictest4020176-21tr20:20:211
392948599129,2cyclictest30265-21unixbench_multi23:00:251
392948599129,2cyclictest0-21swapper/123:10:171
392948599129,2cyclictest0-21swapper/122:55:171
392948599129,2cyclictest0-21swapper/122:25:001
392948599129,2cyclictest0-21swapper/100:15:001
392948599128,3cyclictest3969702-21gltestperf19:40:211
392948599128,2cyclictest0-21swapper/123:20:181
392948599126,5cyclictest493-21dbus-daemon20:00:011
392948599120,11cyclictest0-21swapper/120:30:151
392948599120,11cyclictest0-21swapper/100:20:211
392948199129,2cyclictest49606-21/usr/sbin/munin23:20:200
392948199129,2cyclictest130980-21turbostat00:29:590
392948199122,9cyclictest60471-21cut23:25:210
392948199121,9cyclictest99001-40updatedb.mlocat00:00:000
392948199121,9cyclictest4019219-21apt-get20:20:160
392948199121,2cyclictest4050265-21apt-get20:45:130
392948199120,10cyclictest4173762-21ntp_states22:20:250
392949099118,2cyclictest4131303-21cpuspeed_turbos21:50:143
392949099118,2cyclictest0-21swapper/320:35:253
392949099118,2cyclictest0-21swapper/300:19:593
392949099116,4cyclictest84816-21tr23:45:203
392948799118,2cyclictest4051207-21irqrtprio20:45:182
392948799118,2cyclictest4013070-21cpuspeed_turbos20:15:172
392948799118,2cyclictest31308-21gltestperf23:05:182
392948799118,2cyclictest31308-21gltestperf23:05:182
392948799118,2cyclictest0-21swapper/223:25:172
392948799118,2cyclictest0-21swapper/219:35:312
392948799117,2cyclictest0-21swapper/222:05:182
392948799116,4cyclictest83646-21cut23:45:192
392948799116,4cyclictest4138802-21cut21:55:192
392948799116,4cyclictest4136685-21grep21:50:252
392948799115,4cyclictest4110394-21tr21:30:202
392948799115,4cyclictest28918-21tr23:00:202
392948799110,3cyclictest0-21swapper/223:30:412
392948599119,1cyclictest0-21swapper/119:20:211
392948599118,2cyclictest6950irq/9-acpi21:04:521
392948599118,2cyclictest4156444-21irqcore22:10:211
392948599117,3cyclictest3984051-21tr19:50:181
392948599116,4cyclictest141878-21tr00:35:171
392948599116,4cyclictest127066-21cut00:25:181
3929485991110,1cyclictest0-21swapper/119:30:231
392948599110,1cyclictest0-21swapper/121:18:511
392948599110,10cyclictest0-21swapper/122:50:331
392948199119,1cyclictest3929475-21cyclictest19:10:160
392948199118,2cyclictest0-21swapper/020:25:170
392948199113,7cyclictest211rcuc/021:30:170
39290542117,2sleep10-21swapper/119:06:351
392949099105,4cyclictest4143779-21df_inode22:00:133
392948799107,2cyclictest401ktimers/219:30:202
392948799104,4cyclictest11186-21/usr/sbin/munin22:45:242
392948599107,2cyclictest6950irq/9-acpi20:35:401
392948599107,2cyclictest311ktimers/123:40:171
392948599107,2cyclictest0-21swapper/121:25:191
392948599105,4cyclictest4151764-21cut22:05:201
39290482106,2sleep00-21swapper/019:06:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional