You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-19 - 07:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8s.osadl.org (updated Fri Sep 19, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
33477199755751,3cyclictest461683-21kworker/3:2+events23:18:293
33476999752748,3cyclictest506449-21kworker/2:1+events00:29:512
33477199748744,3cyclictest593369-21turbostat22:44:553
33476999747743,3cyclictest2408932-21kworker/2:2+events21:06:562
33477199746742,3cyclictest461683-21kworker/3:2+events21:42:543
33477199745738,3cyclictest191rcu_preempt21:25:233
33477199744740,3cyclictest730584-21turbostat00:34:553
33476999744739,4cyclictest506449-21kworker/2:1+events00:32:282
33476599743739,3cyclictest1364500-21kworker/1:0+events19:23:541
33477199742739,2cyclictest618185-21turbostat23:04:553
33477199740735,3cyclictest461683-21kworker/3:2+events22:35:223
33477199738734,3cyclictest461683-21kworker/3:2+events22:24:393
33477199738732,5cyclictest461683-21kworker/3:2+events22:00:323
33476599735731,3cyclictest655371-21kworker/1:0+events23:48:071
33476599734730,3cyclictest655371-21kworker/1:0+events23:40:201
33476599734730,3cyclictest349454-21kworker/1:2+events20:47:501
33476599733727,5cyclictest655371-21kworker/1:0+events00:20:201
33477199732729,2cyclictest461683-21kworker/3:2+events21:31:363
33477199731728,2cyclictest340656-21kworker/3:1+events19:18:463
33477199731728,2cyclictest340656-21kworker/3:1+events19:18:463
33476599730727,2cyclictest655371-21kworker/1:0+events23:38:561
33476599729726,2cyclictest655371-21kworker/1:0+events00:03:361
33476599729726,2cyclictest349454-21kworker/1:2+events19:30:221
33477199727723,3cyclictest423393-21kworker/3:1+events20:30:223
33476599727723,3cyclictest349454-21kworker/1:2+events19:48:271
33476599725721,3cyclictest349454-21kworker/1:2+events23:26:051
33476599723715,4cyclictest349454-21kworker/1:2+events20:38:481
33477199721718,2cyclictest461555-21turbostat20:54:553
33476599720716,3cyclictest349454-21kworker/1:2+events19:36:481
33477199719713,5cyclictest461683-21kworker/3:2+events21:21:133
33477199719713,5cyclictest461683-21kworker/3:2+events21:21:133
33476599718715,2cyclictest655371-21kworker/1:0+events23:30:211
33476599718715,2cyclictest349454-21kworker/1:2+events20:41:381
33476599718714,3cyclictest349454-21kworker/1:2+events19:41:371
33477199716713,2cyclictest630645-21turbostat23:14:543
33477199716712,3cyclictest461683-21kworker/3:2+events00:11:493
33476599716713,2cyclictest349454-21kworker/1:2+events23:23:331
33476599716713,2cyclictest349454-21kworker/1:2+events19:27:191
33476599716712,3cyclictest349454-21kworker/1:2+events19:58:261
33477199715707,6cyclictest367835-21kworker/3:2+events20:10:213
33476599715712,2cyclictest349454-21kworker/1:2+events20:00:231
33477199714711,2cyclictest367835-21kworker/3:2+events20:09:113
33477199712708,3cyclictest674341-21turbostat23:49:553
33477199711708,2cyclictest461683-21kworker/3:2+events22:33:243
33476599711708,2cyclictest349454-21kworker/1:2+events19:50:211
33476599709705,3cyclictest711691-21turbostat00:19:561
33477199707700,5cyclictest461683-21kworker/3:2+events23:57:133
33477199706701,4cyclictest461683-21kworker/3:2+events21:10:223
33477199706701,4cyclictest461683-21kworker/3:2+events21:10:223
33477199704701,2cyclictest429959-21turbostat20:29:563
33477199704700,3cyclictest692960-21turbostat00:04:553
33477199703699,3cyclictest461683-21kworker/3:2+events00:05:233
33477199701690,7cyclictest461683-21kworker/3:2+events22:25:223
33476999697693,3cyclictest718178-21turbostat00:24:562
33477199696692,3cyclictest398810-21turbostat20:04:553
33477199693688,4cyclictest461683-21kworker/3:2+events21:45:233
33477199693688,4cyclictest455384-21turbostat20:49:553
33477199692688,3cyclictest410878-21kworker/3:0+events20:17:313
33476999692689,2cyclictest468922-21turbostat21:04:552
33477199688684,3cyclictest461683-21kworker/3:2+events22:57:413
33477199688683,4cyclictest461683-21kworker/3:2+events22:15:213
33477199688683,4cyclictest461683-21kworker/3:2+events21:35:223
33477199687681,5cyclictest461683-21kworker/3:2+events23:05:213
33477199685681,3cyclictest2409586-21kworker/3:2+events19:11:493
33477199684679,4cyclictest475116-21turbostat21:09:553
33477199683679,3cyclictest461683-21kworker/3:2+events21:55:223
33477199678676,1cyclictest461683-21kworker/3:2+events00:19:163
33477199677674,2cyclictest531252-21turbostat21:54:543
33477199677673,3cyclictest599598-21turbostat22:49:553
33477199676672,3cyclictest461683-21kworker/3:2+events22:50:233
33477199675671,3cyclictest461683-21kworker/3:2+events20:57:333
33477199674670,3cyclictest461683-21kworker/3:2+events22:12:103
33476599674670,3cyclictest636850-21turbostat23:19:551
33477199673670,2cyclictest410878-21kworker/3:0+events20:20:233
33477199673669,3cyclictest461683-21kworker/3:2+events21:02:433
33477199667664,2cyclictest461683-21kworker/3:2+events23:50:213
33476599661658,2cyclictest436432-21turbostat20:34:551
33477199659655,3cyclictest461683-21kworker/3:2+events21:15:233
33476999659656,2cyclictest736816-21turbostat00:39:552
33477199658654,3cyclictest549897-21turbostat22:09:563
33476599656653,2cyclictest686736-21turbostat23:59:551
33477199648644,3cyclictest730607-21kworker/3:0+events00:39:273
33476599601598,2cyclictest340905-21turbostat19:19:551
33476599601598,2cyclictest340905-21turbostat19:19:551
40729621170,1sleep10-21swapper/120:05:201
3491592360,2chrt211rcuc/019:19:460
3491592360,2chrt211rcuc/019:19:450
334769992219,2cyclictest449146-21sendmail20:40:002
334769992218,3cyclictest693089-21latency_hist00:00:012
334769992217,3cyclictest0-21swapper/221:29:592
334769992117,2cyclictest0-21swapper/200:15:012
334769992016,2cyclictest0-21swapper/220:00:012
334762992016,3cyclictest0-21swapper/020:30:010
3705322190,1sleep30-21swapper/319:35:203
334765991916,2cyclictest574701-21cron22:25:001
334765991915,2cyclictest0-21swapper/122:55:001
334769991816,1cyclictest0-21swapper/219:20:002
334769991816,1cyclictest0-21swapper/219:20:002
334769991814,2cyclictest0-21swapper/222:45:002
334769991813,3cyclictest509-21rs:main2
33476999181,15cyclictest699855-21gltestperf00:05:162
334762991815,2cyclictest632372-21kthreadcore23:10:190
334762991814,2cyclictest0-21swapper/020:00:010
334769991714,2cyclictest0-21swapper/220:50:002
334769991712,3cyclictest374362-21cpuspeed_turbos19:40:162
334765991714,2cyclictest494591-21gltestperf21:20:151
334765991714,2cyclictest494591-21gltestperf21:20:151
334762991714,2cyclictest0-21swapper/023:30:000
334771991613,2cyclictest356650-21tr19:25:173
334771991612,3cyclictest662250-21cpuspeed_turbos23:35:163
334771991611,4cyclictest393817-21tr19:55:183
33476999163,2cyclictest191rcu_preempt19:20:152
334769991613,2cyclictest643328-21latency_hist23:20:012
334769991613,2cyclictest618574-21cpuspeed_turbos23:00:162
334769991612,3cyclictest567420-21latency22:15:202
334769991611,4cyclictest656949-21ps23:30:162
334769991610,4cyclictest543631-21cron22:00:002
33476599169,5cyclictest605775-21ls22:50:001
33476599169,5cyclictest523-21gdbus00:10:011
334765991613,2cyclictest636455-21unixbench_singl23:10:281
334762991613,2cyclictest491-21avahi-daemon21:20:000
334762991613,2cyclictest0-21swapper/020:20:010
334762991612,3cyclictest711235-21unixbench_multi00:10:250
334762991612,3cyclictest555724-21unixbench_singl22:05:290
334762991611,4cyclictest580925-21munin-run22:30:010
334762991611,3cyclictest531241-21cron21:50:000
334762991611,3cyclictest405759-21irqcore20:05:190
334762991610,5cyclictest0-21swapper/019:40:150
334762991610,4cyclictest3763191-21systemd-journal23:17:550
334769991512,2cyclictest712475-21fschecks_time00:15:152
334769991512,2cyclictest525068-21munin-run21:45:002
334769991512,2cyclictest399182-21cpuspeed_turbos20:00:142
334769991512,2cyclictest0-21swapper/223:10:002
334769991512,2cyclictest0-21swapper/221:30:252
334769991511,3cyclictest362832-21irqrtprio19:30:182
334769991511,2cyclictest0-21swapper/223:15:002
334765991512,2cyclictest0-21swapper/122:35:001
33476599150,3cyclictest630793-21cut23:10:001
33476299156,8cyclictest661906-21munin-plugin-st23:35:000
334762991513,1cyclictest2401543-21kworker/0:0+kacpid19:30:160
334762991512,2cyclictest499-21polkitd23:55:000
334762991511,3cyclictest630227-21unixbench_singl23:05:270
33477199149,4cyclictest3763191-21systemd-journal19:33:303
334771991411,2cyclictest656450-21cut23:30:163
334771991411,2cyclictest374611-21gltestperf19:40:183
334771991410,3cyclictest386727-21cpuspeed_turbos19:50:123
334771991410,2cyclictest0-21swapper/323:25:203
33476999149,4cyclictest411228-21cron20:10:012
33476999149,3cyclictest693321-21cpuspeed_turbos00:00:132
33476999146,6cyclictest523-21gdbus19:50:012
334769991411,2cyclictest419219-21tr20:15:162
334769991411,2cyclictest0-21swapper/223:25:002
334769991410,3cyclictest502-21systemd-logind21:15:002
334769991410,3cyclictest502-21systemd-logind21:15:002
334769991410,3cyclictest361126-21unixbench_multi19:25:252
334769991410,2cyclictest0-21swapper/220:10:172
334765991411,2cyclictest0-21swapper/100:30:161
334765991410,3cyclictest623436-21needreboot23:00:221
334765991410,2cyclictest0-21swapper/121:05:191
33476599140,13cyclictest544719-21sed22:00:161
33476299147,5cyclictest523-21gdbus00:30:000
334762991411,2cyclictest617773-21unixbench_singl22:55:240
334762991411,2cyclictest513258-21gltestperf21:35:160
334762991411,2cyclictest482067-21tr21:10:170
334762991411,2cyclictest482067-21tr21:10:170
334762991411,2cyclictest475589-21fschecks_count21:05:150
334762991411,2cyclictest368418-21gltestperf19:35:160
334762991410,2cyclictest0-21swapper/021:40:160
334762991410,2cyclictest0-21swapper/020:49:590
334762991410,1cyclictest0-21swapper/000:35:210
3287072142,7sleep00-21swapper/019:05:120
33477199137,5cyclictest719217-21sed00:20:203
33477199137,4cyclictest349491-21apt-get19:20:153
334771991310,2cyclictest385958-21unixbench_singl19:45:273
334771991310,2cyclictest0-21swapper/323:45:013
334771991310,2cyclictest0-21swapper/323:20:183
334771991310,2cyclictest0-21swapper/320:35:213
33476999139,3cyclictest462052-21/usr/sbin/munin20:50:222
33476999139,2cyclictest0-21swapper/223:40:002
33476999138,3cyclictest550156-21find22:05:162
33476999137,4cyclictest655558-21sshd23:28:042
33476999137,4cyclictest611768-21sshd22:52:312
33476999133,3cyclictest2956-21rtkit-daemon22:30:012
334769991310,2cyclictest617763-21unixbench_singl22:55:242
334769991310,2cyclictest600178-21gltestperf22:45:152
334769991310,2cyclictest531671-21cpuspeed_turbos21:50:142
33476599139,3cyclictest493549-21unixbench_singl21:15:271
334765991311,1cyclictest524502-21timerwakeupswit21:40:231
334765991310,2cyclictest557342-21irqrtprio22:10:181
334765991310,2cyclictest430403-21apt-key20:25:011
334765991310,2cyclictest0-21swapper/122:40:001
334765991310,2cyclictest0-21swapper/122:20:011
334765991310,2cyclictest0-21swapper/121:15:001
334765991310,2cyclictest0-21swapper/121:15:001
33476299139,3cyclictest512415-21kworker/0:0+kacpi_notify21:50:190
33476299137,4cyclictest493-21dbus-daemon20:55:000
33476299137,4cyclictest454505-21ntpq20:40:230
33476299136,6cyclictest523-21gdbus23:40:000
33476299136,5cyclictest1-21systemd19:15:010
334762991311,1cyclictest692792-21kworker/0:0+kacpid00:04:290
334762991311,1cyclictest543507-21kworker/0:3+kacpid22:40:190
334762991310,2cyclictest687013-21apt-get23:55:150
334762991310,2cyclictest587473-21cpuspeed_turbos22:35:140
334762991310,2cyclictest500578-21cpuspeed_turbos21:25:160
334762991310,2cyclictest499-21polkitd23:00:280
334762991310,2cyclictest367498-21kworker/0:1+kacpid22:00:180
334762991310,2cyclictest171ktimers/000:20:160
334762991310,2cyclictest0-21swapper/019:45:170
334762991310,2cyclictest0-21swapper/000:15:170
3345432135,2sleep10-21swapper/119:09:181
33477199129,2cyclictest449755-21gltestperf20:40:183
33477199126,1cyclictest191rcu_preempt00:25:183
33476999129,2cyclictest0-21swapper/220:40:272
33476999129,2cyclictest0-21swapper/220:30:142
33476999128,3cyclictest488499-21sed21:15:182
33476999127,4cyclictest561467-21ntp_states22:10:232
33476999127,3cyclictest0-21swapper/223:50:172
33476599129,2cyclictest523-21gdbus22:45:001
33476599129,2cyclictest0-21swapper/122:25:151
33476599129,2cyclictest0-21swapper/120:10:241
33476599127,4cyclictest3763191-21systemd-journal00:40:001
33476599127,3cyclictest556138-21systemd-run22:10:011
33476599126,4cyclictest724967-21head00:25:181
33476599124,6cyclictest493-21dbus-daemon21:30:241
334765991210,1cyclictest0-21swapper/121:50:011
33476599120,1cyclictest0-21swapper/120:24:241
33476299129,2cyclictest702541-21kthreadcore00:05:200
33476299129,2cyclictest543507-21kworker/0:3+kacpi_notify22:20:160
33476299129,2cyclictest429957-21idleruntime-cro20:25:010
33476299128,3cyclictest62850irq/126-eno1:rx-019:50:210
33476299128,3cyclictest506697-21apt21:30:140
33476299128,3cyclictest171ktimers/023:20:260
33476299128,2cyclictest606050-21apt-get22:50:130
334762991210,1cyclictest0-21swapper/021:20:240
334762991210,1cyclictest0-21swapper/021:20:230
334762991210,1cyclictest0-21swapper/019:20:260
33476999119,1cyclictest0-21swapper/221:40:012
33476999119,1cyclictest0-21swapper/220:30:012
33476999118,2cyclictest0-21swapper/222:38:312
33476999118,2cyclictest0-21swapper/219:40:002
33476999118,1cyclictest0-21swapper/222:25:012
33476999116,4cyclictest496717-21tr21:20:182
33476999116,4cyclictest496717-21tr21:20:182
33476999116,3cyclictest493-21dbus-daemon21:50:012
33476999116,3cyclictest391682-21ntp_kernel_err19:50:202
33476999110,2cyclictest3398442sed19:10:192
33476999110,11cyclictest0-21swapper/222:30:322
33476599118,2cyclictest537032-21unixbench_singl21:50:261
33476599118,2cyclictest505435-21/usr/sbin/munin21:25:231
33476599118,2cyclictest0-21swapper/120:50:241
33476599117,3cyclictest0-21swapper/120:15:141
33476599116,4cyclictest710320-21tr00:10:181
33476599115,4cyclictest3763191-21systemd-journal20:59:551
33476599110,1cyclictest0-21swapper/123:50:211
33476599110,1cyclictest0-21swapper/122:56:541
33476599110,1cyclictest0-21swapper/121:03:121
33476599110,10cyclictest0-21swapper/121:36:261
33476299119,1cyclictest0-21swapper/020:55:010
33476299118,2cyclictest562723-21cpuspeed_turbos22:15:140
33476299118,2cyclictest0-21swapper/022:45:150
33476299117,2cyclictest736816-21turbostat00:35:000
33476299116,4cyclictest493-21dbus-daemon22:00:010
33476299116,4cyclictest439486-21tr20:30:190
33476299110,3cyclictest416592-21ntp_states20:10:220
33476299110,11cyclictest0-21swapper/022:14:320
33476299110,11cyclictest0-21swapper/021:02:130
33476299110,10cyclictest403508-21tr20:00:200
33476299110,10cyclictest0-21swapper/020:38:090
3345862118,2sleep20-21swapper/219:09:512
3345672118,2sleep30-21swapper/319:09:363
33476999107,2cyclictest0-21swapper/223:45:272
33476999106,3cyclictest468973-21wc21:00:002
33476599107,3cyclictest6950irq/9-acpi19:12:571
33476599106,3cyclictest537961-21uniq21:55:151
33476299108,1cyclictest0-21swapper/022:30:190
33476299107,2cyclictest0-21swapper/023:45:190
33476299105,4cyclictest668464-21ls23:40:130
33476299105,4cyclictest523-21gdbus19:25:230
3347699996,2cyclictest0-21swapper/222:00:172
29579890,1rtkit-daemon0-21swapper/223:40:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional