You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 18:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Wed Sep 17, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2987370993631,3cyclictest3101986-21gpgv09:00:003
2987363993630,4cyclictest3170083-21latency_hist10:05:012
2987370993330,2cyclictest3185761-21kworker/u8:4+flush-259:010:35:193
2987370993128,2cyclictest3236997-21kworker/u8:3+flush-259:011:50:193
2987363993126,3cyclictest3117786-21latency_hist09:15:012
2987354993125,5cyclictest2987091-21latency_hist07:10:010
2987358993026,3cyclictest3226929-21kworker/u8:2+flush-259:011:25:001
2987354993025,4cyclictest2987098-21apt-get07:10:020
2987358992925,3cyclictest3236997-21kworker/u8:3+flush-259:011:15:261
2987358992923,5cyclictest3086200-21apt-get08:42:551
2987370992825,2cyclictest3216423-21kworker/u8:1+flush-259:011:00:253
2987370992824,3cyclictest3293972-21kworker/u8:3+flush-259:012:25:003
2987370992823,3cyclictest3247493-21/usr/sbin/munin11:20:143
2987358992823,3cyclictest3159406-21kworker/u8:1+flush-259:010:05:191
2987354992824,3cyclictest3262718-21kworker/u8:0+flush-259:012:30:250
2987370992621,4cyclictest3055021-21latency_hist08:15:013
2987370992620,4cyclictest3190686-21latency_hist10:25:013
2987354992623,2cyclictest3273000-21kworker/u8:1+flush-259:011:45:220
2987354992620,5cyclictest3189835-21ntpq10:20:240
2987358992518,5cyclictest2698871-21systemd-journal10:15:001
2987354992518,6cyclictest3106589-21ntpq09:00:240
2987354992518,5cyclictest3158927-21ntpq09:50:230
2987354992518,4cyclictest3024262-21/usr/sbin/munin07:45:200
2987363992420,3cyclictest3196047-21/usr/sbin/munin10:30:172
2987358992420,3cyclictest3236997-21kworker/u8:3+flush-259:011:45:001
2987354992421,2cyclictest3262718-21kworker/u8:0+events_unbound12:10:230
2987354992420,3cyclictest3081285-21kworker/u8:1+flush-259:008:50:170
2987370992319,3cyclictest3262718-21kworker/u8:0+flush-259:012:05:193
2987370992319,3cyclictest3262718-21kworker/u8:0+flush-259:012:05:183
2987358992320,2cyclictest3201198-21kworker/u8:3+flush-259:010:49:561
2987358992317,4cyclictest3190930-21/usr/sbin/munin10:25:161
2987358992317,4cyclictest3111760-21ntpq09:05:251
2987354992317,4cyclictest2987769-21kworker/u8:3+events_unbound08:30:030
2987354992316,5cyclictest2698871-21systemd-journal08:36:260
2987370992217,4cyclictest2992939-21/usr/sbin/munin07:15:163
2987370992217,4cyclictest2992939-21/usr/sbin/munin07:15:153
2987370992215,5cyclictest2698871-21systemd-journal09:50:003
2987358992219,2cyclictest3070172-21kworker/u8:2+flush-259:008:35:271
2987358992217,3cyclictest3086663-21/usr/sbin/munin08:45:141
2987358992216,4cyclictest3029175-21awk07:50:011
2987354992218,2cyclictest3117766-21cron09:14:590
2987354992216,4cyclictest3278959-21diskstats11:50:170
2987370992117,3cyclictest3096724-21kworker/u8:2+events_unbound09:05:273
2987370992117,3cyclictest3000063-21rs:main3
2987370992115,4cyclictest3096716-21latency_hist08:54:593
2987358992116,4cyclictest2971825-21kworker/u8:2+flush-259:007:30:281
2987354992118,2cyclictest3288788-21cron12:00:000
2987354992117,3cyclictest3236997-21kworker/u8:3+flush-259:011:35:200
2987354992116,3cyclictest3236997-21kworker/u8:3+events_unbound11:20:130
2987354992116,3cyclictest3060386-21kworker/u8:0+events_unbound08:22:250
2987370992017,2cyclictest0-21swapper/308:30:013
2987370992016,3cyclictest2977109-21kworker/u8:0+events_unbound07:20:163
2987370992015,4cyclictest3324922-21turbostat12:39:553
2987370992014,4cyclictest3102560-21apt-get09:00:143
2987363992017,2cyclictest3216423-21kworker/u8:1+flush-259:011:00:172
2987363992016,3cyclictest3236997-21kworker/u8:3+flush-259:011:35:202
2987363992016,3cyclictest3159406-21kworker/u8:1+flush-259:009:55:242
2987363992016,3cyclictest3060146-21idleruntime-cro08:20:012
2987358992014,4cyclictest2698871-21systemd-journal09:53:251
2987354992016,3cyclictest3273000-21kworker/u8:1+flush-259:012:00:260
2987354992016,3cyclictest3159406-21kworker/u8:1+events_unbound10:25:270
2987354992016,3cyclictest3076035-21kworker/u8:0+events_unbound10:00:170
2987354992015,4cyclictest3013752-21kworker/u8:0+flush-259:007:44:310
2987354992015,3cyclictest3076035-21kworker/u8:0+flush-259:010:30:200
2987354992013,5cyclictest2698871-21systemd-journal10:19:590
2987370991916,2cyclictest3076035-21kworker/u8:0+flush-259:008:38:553
2987370991915,3cyclictest3201198-21kworker/u8:3+events_unbound10:45:213
2987370991915,3cyclictest3076035-21kworker/u8:0+flush-259:009:55:173
2987363991914,4cyclictest3048904-21ntpq08:05:202
2987358991916,2cyclictest3283252-21kworker/u8:2+flush-259:011:55:251
2987358991915,3cyclictest3262718-21kworker/u8:0+flush-259:012:05:271
2987358991915,3cyclictest3262718-21kworker/u8:0+flush-259:012:05:261
2987358991914,4cyclictest3266986-21ntpq11:35:231
2987358991914,4cyclictest2698871-21systemd-journal09:02:261
2987354991915,3cyclictest3159406-21kworker/u8:1+events_unbound10:15:000
2987354991915,3cyclictest3096724-21kworker/u8:2+flush-259:009:05:230
2987354991915,3cyclictest3081285-21kworker/u8:1+flush-259:009:20:000
2987354991915,3cyclictest3076035-21kworker/u8:0+flush-259:009:20:220
2987354991913,4cyclictest3132714-21ntpq09:25:210
2987354991913,4cyclictest3029449-21apt-get07:50:140
2987370991815,2cyclictest3252441-21kworker/u8:1+events_unbound11:30:003
2987370991815,2cyclictest3159406-21kworker/u8:1+events_unbound10:40:183
2987370991815,2cyclictest3076035-21kworker/u8:0+flush-259:010:30:193
2987370991814,3cyclictest3273000-21kworker/u8:1+flush-259:011:45:203
2987370991814,3cyclictest3081285-21kworker/u8:1+flush-259:009:25:223
2987370991814,3cyclictest2971825-21kworker/u8:2+flush-259:007:30:003
2987363991811,5cyclictest3324054-21ntpq12:30:232
2987358991813,4cyclictest3096724-21kworker/u8:2+events_unbound08:55:191
2987358991812,4cyclictest3283623-21latency_hist11:55:001
2987358991812,4cyclictest3143915-21apt-get09:40:001
2987354991814,3cyclictest3314831-21kworker/u8:4+events_unbound12:35:270
2987354991814,3cyclictest3236997-21kworker/u8:3+events_unbound11:30:180
2987354991814,3cyclictest3185761-21kworker/u8:4+events_unbound10:49:270
2987354991814,3cyclictest2987769-21kworker/u8:3+flush-259:007:25:160
2987354991814,2cyclictest3170762-21irqcore10:05:170
2987354991813,3cyclictest3226702-21turbostat11:00:000
2987354991813,3cyclictest2987769-21kworker/u8:3+events_unbound08:15:170
2987354991813,3cyclictest0-21swapper/007:20:000
2987354991813,3cyclictest0-21swapper/007:20:000
2987370991713,3cyclictest3262718-21kworker/u8:0+flush-259:012:10:193
2987363991714,2cyclictest3319054-21switchtime12:25:232
2987363991713,3cyclictest3175249-21kworker/u8:3+flush-259:010:20:002
2987363991712,4cyclictest3128285-21gpgv09:25:012
2987363991712,3cyclictest3325190-21apt-get12:35:132
2987363991711,4cyclictest2698871-21systemd-journal07:20:252
2987358991714,2cyclictest3076035-21kworker/u8:0+flush-259:009:55:171
2987358991714,2cyclictest3055700-21irqcore08:15:171
2987358991712,4cyclictest3024021-21latency_hist07:45:011
2987358991712,4cyclictest2698871-21systemd-journal12:02:251
2987358991711,4cyclictest3231861-21latency_hist11:05:011
2987354991713,3cyclictest3013752-21kworker/u8:0+flush-259:007:36:300
2987354991711,4cyclictest2698871-21systemd-journal09:44:250
2987370991613,2cyclictest3272975-21apt-get11:45:003
2987370991613,2cyclictest3076035-21kworker/u8:0+flush-259:008:40:323
2987370991613,2cyclictest3060386-21kworker/u8:0+flush-259:008:20:203
2987370991613,2cyclictest3023054-21grep07:40:193
2987370991612,3cyclictest2971825-21kworker/u8:2+flush-259:007:39:303
2987363991613,2cyclictest3081931-21irqcore08:40:212
2987363991613,2cyclictest3044661-21kworker/u8:0+flush-259:008:10:182
2987363991610,4cyclictest3106530-21ntpq09:00:212
2987363991610,4cyclictest2971825-21kworker/u8:2+flush-259:007:25:182
2987354991614,1cyclictest60650irq/133-enp0s29f1:rx-009:32:070
2987354991613,2cyclictest60650irq/133-enp0s29f1:rx-012:25:210
2987354991613,2cyclictest60650irq/133-enp0s29f1:rx-012:20:130
2987354991613,2cyclictest60650irq/133-enp0s29f1:rx-009:45:240
2987354991613,2cyclictest3226929-21kworker/u8:2+flush-259:011:25:270
2987354991612,3cyclictest3236997-21kworker/u8:3+events_unbound11:15:190
2987354991612,3cyclictest3185761-21kworker/u8:4+flush-259:010:35:220
2987354991612,3cyclictest3081285-21kworker/u8:1+events_unbound08:46:260
2987354991612,3cyclictest3076035-21kworker/u8:0+events_unbound09:55:150
2987354991612,3cyclictest3023641-21kworker/u8:1+events_unbound07:59:340
2987354991610,4cyclictest2698871-21systemd-journal11:15:000
298735499160,14cyclictest1-21systemd07:33:240
2987370991512,2cyclictest3216423-21kworker/u8:1+events_unbound11:10:233
2987370991512,2cyclictest3076035-21kworker/u8:0+flush-259:009:20:303
2987370991512,2cyclictest3076035-21kworker/u8:0+events_unbound10:00:233
2987370991512,2cyclictest3035022-21irqcore07:55:173
2987370991512,2cyclictest3024252-21kworker/u8:4+events_unbound07:50:243
298737099151,13cyclictest492-21dbus-daemon10:15:003
2987370991511,3cyclictest3096724-21kworker/u8:2+events_unbound09:10:243
2987370991510,4cyclictest3304525-21kworker/u8:1+events_unbound12:15:173
2987370991510,4cyclictest3122947-21kworker/u8:3+flush-259:009:40:053
2987370991510,3cyclictest3000063-21rs:main3
2987363991511,3cyclictest3023641-21kworker/u8:1+flush-259:008:30:112
2987363991510,3cyclictest3236997-21kworker/u8:3+flush-259:011:20:162
298735899159,4cyclictest2698871-21systemd-journal09:31:251
2987358991512,2cyclictest3081285-21kworker/u8:1+flush-259:008:50:261
2987358991511,3cyclictest2987769-21kworker/u8:3+flush-259:008:25:271
2987358991510,4cyclictest3262692-21latency_hist11:35:001
2987358991510,3cyclictest2698871-21systemd-journal11:27:251
2987354991513,1cyclictest60650irq/133-enp0s29f1:rx-011:42:570
2987354991513,1cyclictest60650irq/133-enp0s29f1:rx-011:07:410
2987354991513,1cyclictest60650irq/133-enp0s29f1:rx-008:40:280
2987354991512,2cyclictest60650irq/133-enp0s29f1:rx-007:23:300
2987354991512,2cyclictest3262718-21kworker/u8:0+events_unbound12:05:270
2987354991512,2cyclictest3262718-21kworker/u8:0+events_unbound12:05:270
2987354991511,3cyclictest3054636-21kworker/u8:2+events_unbound08:12:260
2987370991411,2cyclictest3043774-21ntp_kernel_pll_08:00:233
2987370991410,3cyclictest3024252-21kworker/u8:4+flush-259:007:50:013
2987363991411,2cyclictest3159406-21kworker/u8:1+events_unbound10:40:202
2987363991411,2cyclictest0-21swapper/209:40:002
2987363991411,1cyclictest0-21swapper/209:00:002
2987363991410,3cyclictest3273000-21kworker/u8:1+flush-259:012:05:252
2987363991410,3cyclictest3273000-21kworker/u8:1+flush-259:012:05:252
298735899149,3cyclictest3003676-21apt-get07:25:141
2987358991411,2cyclictest3049327-21unixbench-2d08:05:251
2987358991411,2cyclictest3030463-21kthreadcore07:50:181
298735499149,3cyclictest2987769-21kworker/u8:3+flush-259:008:25:200
2987354991413,1cyclictest60650irq/133-enp0s29f1:rx-011:00:170
2987354991412,1cyclictest60650irq/133-enp0s29f1:rx-010:50:150
2987354991412,1cyclictest60650irq/133-enp0s29f1:rx-010:43:540
298737099139,2cyclictest0-21swapper/309:35:003
2987370991311,2cyclictest2987769-21kworker/u8:3+flush-259:008:15:183
2987370991311,1cyclictest3222254-21irqcore10:55:183
2987370991310,2cyclictest835-21runrttasks12:30:003
2987370991310,2cyclictest3201198-21kworker/u8:3+events_unbound10:50:173
2987370991310,2cyclictest3149716-21kworker/u8:2+flush-259:009:50:243
298736399139,3cyclictest3262718-21kworker/u8:0+flush-259:012:00:172
2987363991310,2cyclictest2997328-21unixbench_multi07:15:262
2987363991310,2cyclictest2997328-21unixbench_multi07:15:252
298735899137,4cyclictest3275848-21kthreadcore11:45:171
2987358991310,2cyclictest3262718-21kworker/u8:0+flush-259:012:10:161
2987358991310,2cyclictest3180756-21cpuspeed_turbos10:15:141
2987358991310,2cyclictest0-21swapper/108:35:001
298735899130,5cyclictest32017772sed10:35:191
2987354991310,2cyclictest3081285-21kworker/u8:1+events_unbound09:35:210
298737099129,2cyclictest3236997-21kworker/u8:3+flush-259:011:15:203
298737099129,2cyclictest3175249-21kworker/u8:3+events_unbound10:15:233
298737099129,2cyclictest3008965-21cpuspeed_turbos07:30:153
298737099128,3cyclictest3000063-21rs:main3
298736399129,2cyclictest516-21gdbus11:50:002
298736399129,2cyclictest503-21systemd-logind08:05:012
298736399129,2cyclictest3076766-21irqcore08:35:192
298736399129,2cyclictest0-21swapper/209:44:592
298736399128,3cyclictest3226929-21kworker/u8:2+events_unbound11:25:202
298736399128,3cyclictest3023641-21kworker/u8:1+flush-259:008:30:012
2987363991210,1cyclictest0-21swapper/212:25:002
298736399120,11cyclictest503-21systemd-logind11:15:002
298735899129,2cyclictest3132820-21cut09:25:221
298735899128,3cyclictest3076035-21kworker/u8:0+events_unbound09:15:181
298735499121,4cyclictest3044885-21/usr/sbin/munin08:05:210
298737099118,2cyclictest3091141-21unixbench_multi08:45:253
298737099117,3cyclictest3262718-21kworker/u8:0+flush-259:012:00:253
298736399119,1cyclictest0-21swapper/211:00:002
298736399118,2cyclictest508-21tpm2-abrmd10:55:012
298736399118,2cyclictest3283252-21kworker/u8:2+flush-259:011:55:172
298736399118,2cyclictest3185919-21cpuspeed_turbos10:20:152
298736399118,2cyclictest2987932-21cpuspeed_turbos07:10:142
298736399117,3cyclictest3060386-21kworker/u8:0+flush-259:008:24:562
298736399115,4cyclictest3029180-21latency_hist07:50:012
298735899118,1cyclictest0-21swapper/108:10:181
298735899118,1cyclictest0-21swapper/107:55:141
298735899117,3cyclictest3262718-21kworker/u8:0+flush-259:012:30:101
298735899117,3cyclictest3216423-21kworker/u8:1+flush-259:011:10:261
298735899117,3cyclictest3149716-21kworker/u8:2+events_unbound09:45:181
298735899117,3cyclictest3112884-21apt-get09:10:111
298737099107,2cyclictest3120332-21kthreadcore09:15:173
298736399106,3cyclictest3226929-21kworker/u8:2+flush-259:011:15:162
298736399106,3cyclictest3081285-21kworker/u8:1+flush-259:008:50:192
298736399104,5cyclictest3012854-21ntpq07:30:242
298735899107,2cyclictest2993031-21cpuspeed_turbos07:15:141
298735899107,2cyclictest2993031-21cpuspeed_turbos07:15:141
298735899106,3cyclictest3159406-21kworker/u8:1+flush-259:010:30:051
298735899101,8cyclictest3149493-21awk09:45:001
29873639997,1cyclictest3257852-21sshd11:30:142
29873639996,2cyclictest3304662-21cut12:15:132
29873639996,2cyclictest3137902-21ntp_kernel_pll_09:30:232
29873639993,4cyclictest3268120-21apt-get11:40:142
29873589996,2cyclictest3123390-21fschecks_count09:20:181
29873589995,3cyclictest3189814-21ntpq10:20:231
29873589995,3cyclictest311ktimers/110:50:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional