You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-19 - 01:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8s.osadl.org (updated Thu Sep 18, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
59656599758752,5cyclictest729062-21kworker/3:2+events10:04:293
59656399747744,2cyclictest1651680-21kworker/2:3+events07:39:442
59656599746743,2cyclictest729062-21kworker/3:2+events11:11:463
59656599746742,3cyclictest990755-21kworker/3:0+events12:28:433
59656599746742,3cyclictest589941-21kworker/3:0+events08:26:373
59656599744740,3cyclictest729062-21kworker/3:2+events10:57:513
59656599744739,4cyclictest729062-21kworker/3:2+events09:15:423
59656599742739,2cyclictest589941-21kworker/3:0+events08:16:473
59656599742738,3cyclictest729062-21kworker/3:2+events11:26:033
59656399741738,2cyclictest1651680-21kworker/2:3+events08:43:222
59655999741730,5cyclictest545638-21kworker/1:0+events10:30:201
59656399740734,5cyclictest1651680-21kworker/2:3+events08:54:272
59656599739736,2cyclictest729062-21kworker/3:2+events09:06:173
59656599739735,3cyclictest791526-21turbostat09:49:553
59656599739735,3cyclictest729062-21kworker/3:2+events11:02:213
59655699739735,3cyclictest648131-21kworker/0:0+events07:55:220
59656399736730,5cyclictest1651680-21kworker/2:3+events08:48:042
59656599735730,4cyclictest729062-21kworker/3:2+events10:20:203
59656399731723,7cyclictest1651680-21kworker/2:3+events07:30:202
59656599729725,3cyclictest729062-21kworker/3:2+events12:08:223
59656599728724,3cyclictest990755-21kworker/3:0+events12:30:303
59655999726720,5cyclictest545638-21kworker/1:0+events07:10:411
59656599725721,3cyclictest729062-21kworker/3:2+events10:37:513
59656399724720,3cyclictest1651680-21kworker/2:3+events10:25:522
59656399723718,4cyclictest1651680-21kworker/2:3+events07:28:052
59655999721716,4cyclictest545638-21kworker/1:0+events11:30:211
59656599720717,2cyclictest589941-21kworker/3:0+events08:56:483
59656399720716,3cyclictest1651680-21kworker/2:3+events07:17:422
59655699719714,4cyclictest642177-21turbostat07:49:550
59656399718714,3cyclictest1651680-21kworker/2:3+events07:23:362
59656599714711,2cyclictest760432-21turbostat09:24:553
59656599714711,2cyclictest729062-21kworker/3:2+events12:01:303
59656599713710,2cyclictest729062-21kworker/3:2+events12:16:133
59655999713710,2cyclictest545638-21kworker/1:0+events11:36:501
59655999712709,2cyclictest545638-21kworker/1:0+events09:30:221
59655999711709,1cyclictest545638-21kworker/1:0+events09:49:061
59655699711707,3cyclictest760260-21kworker/0:2+events09:21:540
59656599709704,4cyclictest589941-21kworker/3:0+events08:10:213
59656599708703,4cyclictest735259-21turbostat09:04:553
59655999707704,2cyclictest545638-21kworker/1:0+events11:42:541
59655999705701,3cyclictest916313-21turbostat11:29:551
59655999705701,3cyclictest545638-21kworker/1:0+events12:36:451
59655699702692,5cyclictest648131-21kworker/0:0+events07:51:230
59655999700696,3cyclictest545638-21kworker/1:0+events09:44:121
59656599697692,4cyclictest729062-21kworker/3:2+events09:55:223
59656599696693,2cyclictest589941-21kworker/3:0+events08:30:443
59655999696692,3cyclictest545638-21kworker/1:0+events09:36:291
59656599694691,2cyclictest589941-21kworker/3:0+events08:05:233
59656599693690,2cyclictest847658-21turbostat10:34:563
59656599693690,2cyclictest729062-21kworker/3:2+events10:08:483
59656599690686,3cyclictest729062-21kworker/3:2+events11:09:223
59656599690684,5cyclictest589941-21kworker/3:0+events07:45:203
59656599686674,3cyclictest191rcu_preempt11:15:213
59655699684679,4cyclictest648131-21kworker/0:0+events08:01:130
59656599683679,3cyclictest953556-21turbostat11:59:553
59656599682676,5cyclictest729062-21kworker/3:2+events12:20:223
59656599680677,2cyclictest729062-21kworker/3:2+events11:49:513
59656599680675,4cyclictest729062-21kworker/3:2+events10:45:213
59656599680675,4cyclictest660766-21turbostat08:04:553
59656599679676,2cyclictest729062-21kworker/3:2+events09:50:453
59656599678675,2cyclictest729062-21kworker/3:2+events10:16:143
59656599677674,2cyclictest589941-21kworker/3:0+events08:24:413
59656599676673,2cyclictest972246-21turbostat12:14:553
59656599672668,3cyclictest1003842-21turbostat12:39:543
59656599670667,2cyclictest729062-21kworker/3:2+events10:10:203
59656599670667,2cyclictest589941-21kworker/3:0+events07:40:223
59655699670667,2cyclictest754222-21turbostat09:19:560
59656599669666,2cyclictest729062-21kworker/3:2+events11:53:083
59656599669665,3cyclictest729062-21kworker/3:2+events10:50:233
59656599669665,3cyclictest729062-21kworker/3:2+events10:50:223
59656599667663,3cyclictest589941-21kworker/3:0+events08:37:083
59656599666662,3cyclictest934955-21turbostat11:44:553
59656599665660,4cyclictest729062-21kworker/3:2+events09:10:223
59656599662658,3cyclictest729062-21kworker/3:2+events10:42:553
59656399656652,3cyclictest704225-21turbostat08:39:552
59655999655652,2cyclictest841446-21turbostat10:29:551
59656599653650,2cyclictest729062-21kworker/3:2+events09:29:233
59656599652650,1cyclictest722855-21turbostat08:54:553
59656399651648,2cyclictest596424-21turbostat07:14:562
59656399650645,4cyclictest835232-21turbostat10:24:552
59656599649647,1cyclictest629641-21turbostat07:39:553
59656599645642,2cyclictest729062-21kworker/3:2+events11:20:553
59655999625621,3cyclictest766646-21turbostat09:29:561
59655999622620,1cyclictest997373-21turbostat12:34:551
596565992421,2cyclictest785271-21sendmail09:40:003
596556992320,2cyclictest499-21polkitd12:20:000
596565992118,2cyclictest509-21rs:main3
59656399217,7cyclictest866342-21munin-run10:45:002
596559992117,2cyclictest630473-21irqcore07:35:151
596565992016,3cyclictest617256-21idleruntime-cro07:25:003
596563992016,3cyclictest934970-21sendmail11:40:012
596565991916,2cyclictest618988-21kthreadcore07:25:213
59656399196,7cyclictest829005-21idleruntime-cro10:14:592
596563991916,2cyclictest860076-21cron10:40:002
596563991916,2cyclictest747959-21cron09:10:002
59655999195,2cyclictest810117-21turbostat10:00:001
596559991913,4cyclictest872794-21cron10:50:011
596556991915,3cyclictest729559-21fschecks_time08:55:140
596559991811,6cyclictest897649-21cron11:10:001
6846592170,3sleep20-21swapper/208:15:192
596563991713,3cyclictest502-21systemd-logind12:35:002
596559991713,2cyclictest0-21swapper/112:20:191
596559991712,3cyclictest0-21swapper/109:00:001
59656399168,6cyclictest523-21gdbus09:50:002
596563991613,1cyclictest0-21swapper/207:45:152
596563991612,3cyclictest754721-21fschecks_time09:15:172
596563991612,3cyclictest0-21swapper/208:14:592
596563991612,2cyclictest829556-21gltestperf10:15:152
596563991612,2cyclictest655187-21gltestperf07:55:182
596563991612,2cyclictest0-21swapper/212:40:002
59655999168,2cyclictest716672-21awk08:45:001
59655999161,5cyclictest905874-21tr11:15:171
59655999161,2cyclictest823050-21apt-get10:10:091
596559991612,3cyclictest737245-21kthreadcore09:00:161
59655999160,4cyclictest629031-21timerandwakeup07:30:201
596556991613,2cyclictest726113-21tr08:50:180
596556991613,2cyclictest710447-21turbostat08:40:010
596556991613,2cyclictest672435-21systemctl08:05:240
596556991613,1cyclictest0-21swapper/010:20:000
596556991612,3cyclictest773427-21gltestperf09:30:190
5904912163,7sleep10-21swapper/107:05:131
596565991512,2cyclictest597024-21gltestperf07:10:153
596565991511,3cyclictest655782-21irqcore07:55:193
596563991512,2cyclictest984729-21sh12:20:002
596563991512,2cyclictest778060-21needreboot09:30:222
596563991511,3cyclictest990402-21grep12:20:272
59655999159,5cyclictest1-21systemd09:20:011
596559991513,1cyclictest0-21swapper/112:00:001
596559991512,2cyclictest822077-21unixbench_multi10:05:271
596559991511,3cyclictest891010-21unixbench_multi11:00:221
596559991511,3cyclictest0-21swapper/111:55:011
596556991513,1cyclictest2669093-21kworker/0:2+kacpid07:35:190
596556991512,2cyclictest885064-21kworker/0:1+kacpid12:30:170
596556991512,2cyclictest866132-21kworker/0:0+kacpid11:25:170
596556991511,3cyclictest966538-21fschecks_time12:05:170
596556991511,3cyclictest941593-21fschecks_count11:45:150
596556991511,3cyclictest873253-21fschecks_count10:50:120
596556991511,3cyclictest873253-21fschecks_count10:50:120
596556991511,2cyclictest0-21swapper/011:55:020
596556991510,4cyclictest3763191-21systemd-journal10:24:550
59656599140,5cyclictest934940-21idleruntime-cro11:40:003
59656399149,3cyclictest0-21swapper/211:50:202
59656399147,5cyclictest912880-21tr11:20:192
596563991411,2cyclictest799456-21kthreadcore09:50:172
596563991410,3cyclictest786021-21sed09:40:182
596563991410,2cyclictest853437-21head10:30:252
596563991410,2cyclictest0-21swapper/210:50:172
596563991410,2cyclictest0-21swapper/210:50:172
59655999149,4cyclictest879025-21cron10:55:001
59655999149,4cyclictest879025-21cron10:55:001
59655999149,3cyclictest704228-21idleruntime-cro08:35:001
59655999149,3cyclictest636582-21sed07:40:141
596559991411,2cyclictest710428-21cron08:40:011
596559991411,2cyclictest667546-21gltestperf08:05:171
596559991410,3cyclictest647924-21unixbench_multi07:45:231
59655699144,8cyclictest860085-21cron10:40:010
596556991412,1cyclictest654339-21kworker/0:1+kacpid08:23:270
596556991411,2cyclictest935515-21gltestperf11:40:150
596556991411,2cyclictest904205-21cpuspeed_turbos11:15:130
596556991411,2cyclictest886960-21kthreadcore11:00:160
596556991411,2cyclictest885064-21kworker/0:1+kacpid11:09:480
596556991411,2cyclictest654339-21kworker/0:1+kacpid08:30:120
596556991411,2cyclictest605321-21uniq07:15:140
596556991411,2cyclictest386-21systemd-udevd07:33:590
596556991411,2cyclictest0-21swapper/010:35:000
596556991410,3cyclictest847197-21unixbench_multi10:25:230
596556991410,3cyclictest678386-21needreboot08:10:200
596556991410,3cyclictest636313-21cpuspeed_turbos07:40:120
59656599139,3cyclictest0-21swapper/308:50:013
59656599139,2cyclictest0-21swapper/308:40:163
59656599137,4cyclictest0-21swapper/309:35:003
596565991311,1cyclictest0-21swapper/309:40:203
596565991310,2cyclictest0-21swapper/307:35:003
59656399139,3cyclictest954029-21fschecks_count11:55:152
59656399139,3cyclictest748536-21gltestperf09:10:152
59656399139,2cyclictest0-21swapper/209:55:152
59656399138,4cyclictest996509-21ntpq12:25:222
59656399138,4cyclictest523-21gdbus07:50:232
59656399138,3cyclictest885979-21irqcore11:00:142
59656399138,3cyclictest822838-21ls10:10:012
59656399136,5cyclictest3763191-21systemd-journal12:14:592
59656399132,2cyclictest784057-21grep09:35:202
596563991310,2cyclictest0-21swapper/211:35:002
59655999139,3cyclictest879381-21cpuspeed_turbos10:55:171
59655999139,3cyclictest854431-21gltestperf10:35:161
59655999139,3cyclictest0-21swapper/108:20:191
59655999137,4cyclictest765884-21systemctl09:20:221
59655999131,4cyclictest3763191-21systemd-journal07:25:001
596559991310,2cyclictest961021-21cut12:00:161
596559991310,2cyclictest946949-21unixbench_singl11:45:251
596559991310,2cyclictest815863-21unixbench_multi10:00:241
596559991310,2cyclictest620673-21kthreadcore07:25:221
59655699139,3cyclictest965544-21grep12:00:230
59655699138,4cyclictest3763191-21systemd-journal11:40:010
59655699137,4cyclictest523-21gdbus08:45:000
596556991311,1cyclictest885064-21kworker/0:1+kacpid12:25:010
596556991311,1cyclictest760260-21kworker/0:2+kacpid10:10:180
596556991310,2cyclictest866939-21cpuspeed_turbos10:45:130
596556991310,2cyclictest847366-21kworker/0:1+kacpid10:40:180
596556991310,2cyclictest583624-21kworker/0:3+kacpi_notify07:20:150
596556991310,2cyclictest0-21swapper/010:04:590
596556991310,2cyclictest0-21swapper/009:00:140
5962392133,2sleep20-21swapper/207:07:552
7399232120,1sleep20-21swapper/209:00:172
59656599124,2cyclictest191rcu_preempt07:50:183
59656599120,3cyclictest0-21swapper/310:25:243
59656399129,2cyclictest692397-21gltestperf08:25:112
59656399129,2cyclictest0-21swapper/211:45:262
59656399129,2cyclictest0-21swapper/211:25:182
59656399128,3cyclictest810461-21cpuspeed_turbos10:00:132
59656399127,4cyclictest872233-21sed10:45:252
59656399127,4cyclictest734901-21sshd08:55:462
59656399127,3cyclictest904024-21sort11:15:002
59656399126,4cyclictest493-21dbus-daemon12:05:242
59655999129,2cyclictest910850-21irqcore11:20:171
59655999129,2cyclictest742001-21/usr/sbin/munin09:05:141
59655999129,2cyclictest692356-21gltestperf08:25:111
59655999129,2cyclictest311ktimers/107:15:041
59655999129,2cyclictest0-21swapper/112:10:131
59655999129,1cyclictest0-21swapper/108:15:211
59655999128,3cyclictest6950irq/9-acpi10:15:191
59655999128,3cyclictest493-21dbus-daemon12:25:231
59655999125,5cyclictest3763191-21systemd-journal10:45:001
59655999121,2cyclictest8019622cut09:50:181
59655999120,11cyclictest0-21swapper/112:06:311
59655699129,2cyclictest922866-21cpuspeed_turbos11:30:120
59655699129,2cyclictest804278-21cpuspeed_turbos09:55:140
59655699129,2cyclictest692442-21gltestperf08:25:110
59655699129,2cyclictest685174-21unixbench_singl08:15:230
59655699129,2cyclictest502-21systemd-logind12:13:580
59655699129,2cyclictest21750irq/159-ahci[0000:00:17.0]12:00:000
59655699127,4cyclictest3763191-21systemd-journal09:39:550
59655699126,4cyclictest991220-21sort12:25:000
59655699126,4cyclictest803087-21ntpq09:50:210
59655699121,9cyclictest772841-21apt09:30:000
596556991211,1cyclictest583624-21kworker/0:3+kacpid07:10:040
596556991210,1cyclictest866132-21kworker/0:0+kacpid10:58:100
59655699120,4cyclictest822730-21cat10:10:000
59656399119,1cyclictest0-21swapper/208:03:182
59656399118,2cyclictest766988-21cpuspeed_turbos09:25:142
59656399118,2cyclictest0-21swapper/208:05:262
59656399117,3cyclictest909619-21unixbench_multi11:15:232
59656399116,4cyclictest895096-21tr11:05:192
59656399116,4cyclictest761718-21cut09:20:162
59656399115,4cyclictest493-21dbus-daemon08:34:472
59656399111,8cyclictest745-21cups-browsed11:45:002
596563991110,1cyclictest0-21swapper/207:43:462
59656399110,11cyclictest0-21swapper/210:57:112
59655999119,1cyclictest0-21swapper/108:00:161
59655999118,2cyclictest6950irq/9-acpi09:10:191
59655999118,2cyclictest6950irq/9-acpi07:56:101
59655999117,3cyclictest654137-21unixbench_multi07:50:241
59655999117,3cyclictest493-21dbus-daemon08:45:241
59655999116,4cyclictest980921-21tr12:15:171
596559991110,1cyclictest0-21swapper/110:20:031
59655999110,9cyclictest903873-21munin-run11:15:011
59655699118,2cyclictest898218-21gltestperf11:10:160
59655699118,2cyclictest502-21systemd-logind11:21:580
59655699118,2cyclictest0-21swapper/009:45:010
59655699116,4cyclictest716938-21apt08:45:120
59655699116,1cyclictest0-21swapper/009:45:080
596556991110,1cyclictest0-21swapper/012:39:490
59655699110,1cyclictest0-21swapper/009:10:020
59655699110,1cyclictest0-21swapper/007:27:090
295798110,3rtkit-daemon0-21swapper/007:08:090
59656599105,4cyclictest603007-21sort07:15:013
59656399107,2cyclictest0-21swapper/208:20:172
59656399105,4cyclictest965149-21ntpq12:00:212
59655999107,2cyclictest723203-21cpuspeed_turbos08:50:131
59655999105,4cyclictest673806-21hddtemp_smartct08:10:151
59655699100,2cyclictest742977-21ps09:05:170
5962072107,2sleep30-21swapper/307:07:323
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional