You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 00:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Wed Nov 19, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2756029996560,4cyclictest2776606-21apt-get07:30:023
2756016993832,4cyclictest2698871-21systemd-journal07:15:000
2756016993630,4cyclictest3093659-21latency_hist12:35:010
2756016993630,4cyclictest3057132-21latency_hist12:00:010
2756016993329,3cyclictest3010704-21kworker/u8:0+flush-259:011:14:590
2756016993325,5cyclictest3073056-21latency_hist12:15:000
2756016993228,3cyclictest2870041-21apt-get08:59:590
2756029993128,2cyclictest2854740-21apt-get08:45:113
2756016993024,4cyclictest2896308-21latency_hist09:25:000
2756016993024,4cyclictest2698871-21systemd-journal08:00:010
2756021992924,4cyclictest2865099-21/usr/sbin/munin08:55:161
2756021992621,3cyclictest2813316-21kworker/u8:3+flush-259:009:15:001
2756021992620,4cyclictest2823447-21latency_hist08:15:021
2756016992622,3cyclictest2906855-21kworker/u8:0+flush-259:009:50:000
2756021992519,4cyclictest2698871-21systemd-journal08:54:371
2756029992421,2cyclictest2740666-21kworker/u8:1+flush-259:007:45:003
2756029992418,4cyclictest3098820-21latency_hist12:40:013
2756029992319,3cyclictest3062930-21kworker/u8:4+flush-259:012:20:253
2756024992318,4cyclictest2740666-21kworker/u8:1+flush-259:007:10:172
2756024992318,3cyclictest3083594-21/usr/sbin/munin12:25:252
2756016992319,3cyclictest2942485-21kworker/u8:0+flush-259:010:50:190
2756016992319,3cyclictest2740666-21kworker/u8:1+flush-259:007:30:190
2756029992218,3cyclictest3010707-21kworker/u8:3+flush-259:011:55:193
2756029992217,3cyclictest3026439-21/usr/sbin/munin11:30:143
2756021992220,1cyclictest2844179-21latency_hist08:34:591
2756021992218,3cyclictest2713491-21kworker/u8:0+flush-259:007:15:201
2756016992218,2cyclictest2698871-21systemd-journal11:30:000
2756016992217,4cyclictest2698871-21systemd-journal08:10:010
2756029992115,4cyclictest3068291-21diskstats12:10:163
2756024992116,4cyclictest2813316-21kworker/u8:3+flush-259:009:25:172
2756024992113,7cyclictest2698871-21systemd-journal12:25:002
2756021992117,3cyclictest2766526-21kworker/u8:2+flush-259:007:30:001
2756021992115,4cyclictest2782393-21kworker/u8:0+events_unbound08:25:151
2756016992115,4cyclictest2776605-21latency_hist07:30:000
2756029992017,2cyclictest2698871-21systemd-journal09:04:543
2756024992016,3cyclictest2886018-21kworker/u8:2+flush-259:009:59:372
2756024992016,3cyclictest2777290-21apt-get07:30:142
2756024992015,3cyclictest3088738-21apt-get12:30:132
275602499200,3cyclictest3016613-21irqcore11:20:202
2756021992016,3cyclictest3000063-21rs:main1
2756021992015,4cyclictest3031362-21kworker/u8:1+flush-259:012:10:011
2756016992015,3cyclictest0-21swapper/012:30:000
2756016992014,5cyclictest2796690-21ntpq07:45:210
2756029991915,3cyclictest3062930-21kworker/u8:4+flush-259:012:30:013
2756029991914,4cyclictest2922154-21kworker/u8:1+flush-259:009:55:013
2756029991913,4cyclictest3045912-21ntpq11:45:243
2756024991914,3cyclictest2974453-21apt-get10:40:142
2756024991911,7cyclictest3057812-21sed12:00:182
275602499190,6cyclictest2921426-21/usr/sbin/munin09:45:272
2756021991916,2cyclictest2995027-21kworker/u8:2+events_unbound11:10:171
2756021991915,3cyclictest2813316-21kworker/u8:3+flush-259:009:10:001
2756021991915,3cyclictest2740666-21kworker/u8:1+flush-259:007:35:261
2756016991916,2cyclictest3031362-21kworker/u8:1+flush-259:012:05:170
2756016991913,4cyclictest2948554-21cat10:15:010
2756029991812,5cyclictest2848412-21ntpq08:35:213
275602499188,8cyclictest2942858-21latency_hist10:10:002
2756024991815,2cyclictest2740666-21kworker/u8:1+flush-259:007:25:262
2756024991815,2cyclictest0-21swapper/208:00:002
2756021991812,4cyclictest2829080-21diskstats08:20:151
2756016991814,3cyclictest0-21swapper/012:25:010
2756016991812,5cyclictest2698871-21systemd-journal07:40:000
2756029991715,1cyclictest0-21swapper/310:45:013
275602499179,7cyclictest0-21swapper/208:40:292
2756024991714,2cyclictest3015858-21cron11:19:592
2756024991714,2cyclictest2813316-21kworker/u8:3-ext4-rsv-conversion08:20:002
2756024991713,3cyclictest2984497-21turbostat10:50:002
2756024991713,3cyclictest2865124-21apt-get08:55:132
275602499171,2cyclictest3035684-21ntpq11:35:232
275602499171,2cyclictest3035684-21ntpq11:35:222
275602499171,2cyclictest2791601-21ntpq07:40:212
2756024991712,3cyclictest2906891-21apt-get09:35:142
2756024991710,6cyclictest0-21swapper/208:05:012
2756021991714,2cyclictest2813316-21kworker/u8:3+events_unbound10:10:001
2756021991713,3cyclictest3062930-21kworker/u8:4+events_unbound12:10:181
2756021991710,5cyclictest2777115-21base6407:30:021
2756016991714,2cyclictest0-21swapper/010:30:010
2756029991613,2cyclictest2997356-21kthreadcore11:00:203
2756029991613,2cyclictest2874738-21kworker/u8:0+flush-259:009:05:173
2756029991612,3cyclictest2875822-21kworker/u8:1+events_unbound09:15:183
2756029991611,4cyclictest2905757-21ntpq09:30:233
275602499162,12cyclictest0-21swapper/212:10:002
2756024991613,2cyclictest3002-21cron08:54:592
2756024991613,1cyclictest0-21swapper/212:40:012
275602499161,2cyclictest2801797-21ntpq07:50:202
275602499161,1cyclictest3072162-21ntpq12:10:232
275602499160,5cyclictest2875600-21grep09:04:592
275602499160,3cyclictest2766317-21/usr/sbin/munin07:20:002
275602499160,2cyclictest3041778-21cat11:45:012
275602499160,2cyclictest2855428-21irqcore08:45:162
275602499160,14cyclictest2818305-21munin-run08:10:002
275602199169,5cyclictest2916116-21ntpq09:40:241
2756021991614,1cyclictest0-21swapper/112:40:011
2756021991612,3cyclictest2922154-21kworker/u8:1+events_unbound10:00:201
2756021991610,4cyclictest2813316-21kworker/u8:3+events_unbound09:20:161
2756016991614,1cyclictest60650irq/133-enp0s29f1:rx-008:35:320
2756016991614,1cyclictest0-21swapper/010:00:010
2756016991613,2cyclictest60650irq/133-enp0s29f1:rx-011:00:020
2756016991613,2cyclictest60650irq/133-enp0s29f1:rx-010:04:470
2756016991613,2cyclictest60650irq/133-enp0s29f1:rx-008:10:150
2756016991612,3cyclictest2813316-21kworker/u8:3+flush-259:009:31:070
2756016991612,3cyclictest2698871-21systemd-journal07:55:000
2756016991610,4cyclictest2983574-21ntpq10:45:210
275602999159,4cyclictest2698871-21systemd-journal07:57:363
275602999158,5cyclictest2937700-21latency_hist10:05:013
275602999158,5cyclictest2932515-21latency_hist10:00:003
2756029991513,1cyclictest0-21swapper/312:35:013
2756029991512,2cyclictest3057543-21cpuspeed_turbos12:00:143
2756029991512,2cyclictest2813316-21kworker/u8:3+flush-259:008:10:173
2756029991511,3cyclictest2865087-21kworker/u8:2+flush-259:008:55:193
275602499159,4cyclictest2740666-21kworker/u8:1+flush-259:007:45:182
275602499153,10cyclictest2952801-21ldconfig10:15:232
275602499151,5cyclictest2787311-21cron07:40:002
275602499151,4cyclictest3031344-21apt11:35:002
275602499151,4cyclictest2936125-21grep10:00:202
275602499151,3cyclictest3057138-21sh12:00:002
275602499151,3cyclictest2994802-21awk11:00:002
275602499151,2cyclictest2916262-21/usr/sbin/munin09:40:262
275602499151,2cyclictest2849304-21cron08:39:592
275602499151,2cyclictest2819026-21sed08:10:172
275602499151,2cyclictest2770578-21ntpq07:20:232
2756024991512,2cyclictest3002-21cron09:19:592
2756024991510,4cyclictest2813316-21kworker/u8:3+flush-259:009:35:002
275602499150,5cyclictest3021278-21apt11:25:122
275602499150,5cyclictest2943491-21/usr/sbin/munin10:10:182
275602499150,5cyclictest2881346-21/usr/sbin/munin09:10:172
275602499150,4cyclictest2999945-21munin-plugin-st11:04:592
275602499150,4cyclictest2970403-21grep10:35:212
275602499150,4cyclictest2891518-21cut09:20:162
275602499150,3cyclictest3073566-21tune2fs12:15:162
275602499150,3cyclictest3045950-21ntpq11:45:242
275602499150,3cyclictest2923423-21ps09:50:182
275602499150,3cyclictest2878161-21ps09:05:202
275602499150,3cyclictest2834191-21tr08:25:142
275602499150,2cyclictest3047049-21/usr/sbin/munin11:50:172
275602499150,2cyclictest3010674-21awk11:15:002
275602499150,2cyclictest2985181-21grep10:50:182
275602499150,2cyclictest2954307-21sed10:20:172
275602499150,2cyclictest2831213-21kthreadcore08:20:192
275602499150,1cyclictest2968178-21ntpq10:30:242
2756021991512,2cyclictest3036556-21sh11:40:001
2756021991512,2cyclictest3036556-21sh11:39:591
2756021991512,2cyclictest2787556-21kworker/u8:4+flush-259:008:15:181
2756021991511,3cyclictest3062930-21kworker/u8:4+events_unbound12:20:251
2756021991511,3cyclictest2995065-21apt-get11:00:141
2756021991511,3cyclictest2969539-21fschecks_time10:35:181
2756021991511,3cyclictest2838994-21kworker/u8:1+flush-259:008:35:171
2756021991511,3cyclictest2782393-21kworker/u8:0+flush-259:007:50:001
2756021991511,3cyclictest2740666-21kworker/u8:1+flush-259:007:50:161
2756021991510,3cyclictest2870548-21cat09:00:001
2756021991510,3cyclictest2766537-21/usr/sbin/munin07:20:161
275602199150,13cyclictest3041675-21munin-run11:45:001
2756016991514,1cyclictest60650irq/133-enp0s29f1:rx-011:31:510
2756016991514,1cyclictest60650irq/133-enp0s29f1:rx-010:40:290
2756016991513,1cyclictest60650irq/133-enp0s29f1:rx-011:45:180
2756016991513,1cyclictest60650irq/133-enp0s29f1:rx-010:31:300
2756016991513,1cyclictest60650irq/133-enp0s29f1:rx-010:06:070
2756016991513,1cyclictest60650irq/133-enp0s29f1:rx-009:00:160
2756016991513,1cyclictest60650irq/133-enp0s29f1:rx-008:25:240
2756016991513,1cyclictest60650irq/133-enp0s29f1:rx-008:17:070
2756016991511,3cyclictest3057401-21apt-get12:00:130
275601699150,13cyclictest1-21systemd09:30:000
275602999149,4cyclictest3035627-21ntpq11:35:213
275602999149,4cyclictest3035627-21ntpq11:35:213
275602999149,3cyclictest2943092-21/usr/sbin/munin10:10:153
275602999148,4cyclictest2813325-21/usr/sbin/munin08:05:143
275602999148,4cyclictest2698871-21systemd-journal08:26:363
2756029991411,2cyclictest0-21swapper/307:25:013
2756029991410,3cyclictest2698871-21systemd-journal09:15:013
275602499141,1cyclictest3004649-21ntpq11:05:252
275602499140,3cyclictest2839391-21expr08:30:162
275602499140,2cyclictest2959430-21ps10:25:212
275602199149,3cyclictest3077929-21kworker/u8:1+flush-259:012:25:281
275602199149,3cyclictest2984490-21sh10:50:001
275602199149,3cyclictest2922154-21kworker/u8:1+flush-259:010:15:011
2756021991411,2cyclictest2813316-21kworker/u8:3+events_unbound09:35:181
2756021991410,3cyclictest3062930-21kworker/u8:4+flush-259:012:15:151
2756021991410,3cyclictest2813316-21kworker/u8:3+flush-259:009:30:271
2756016991413,1cyclictest60650irq/133-enp0s29f1:rx-011:24:550
2756016991413,1cyclictest60650irq/133-enp0s29f1:rx-011:15:180
2756016991413,1cyclictest60650irq/133-enp0s29f1:rx-010:37:350
2756016991413,1cyclictest60650irq/133-enp0s29f1:rx-009:39:020
2756016991412,2cyclictest60650irq/133-enp0s29f1:rx-011:51:400
2756016991412,2cyclictest60650irq/133-enp0s29f1:rx-009:06:350
2756016991412,1cyclictest61450irq/134-enp0s29f1:tx-011:35:170
2756016991412,1cyclictest61450irq/134-enp0s29f1:tx-011:35:160
2756016991412,1cyclictest60650irq/133-enp0s29f1:rx-011:41:200
2756016991412,1cyclictest60650irq/133-enp0s29f1:rx-010:55:170
2756016991412,1cyclictest60650irq/133-enp0s29f1:rx-009:10:330
2756016991412,1cyclictest60650irq/133-enp0s29f1:rx-008:00:290
2756016991411,2cyclictest60650irq/133-enp0s29f1:rx-008:52:230
275602999137,4cyclictest2813086-21latency_hist08:05:023
2756029991310,2cyclictest3006224-21irqcore11:10:193
275602199139,3cyclictest2838994-21kworker/u8:1+flush-259:008:49:591
275602199138,4cyclictest3010707-21kworker/u8:3+flush-259:011:55:201
2756021991310,2cyclictest2954116-21fschecks_time10:20:151
2756021991310,2cyclictest0-21swapper/112:04:591
2756021991310,1cyclictest0-21swapper/111:15:261
2756016991312,1cyclictest60650irq/133-enp0s29f1:rx-012:35:290
2756016991312,1cyclictest60650irq/133-enp0s29f1:rx-010:20:250
2756016991312,1cyclictest60650irq/133-enp0s29f1:rx-009:15:440
2756016991312,1cyclictest60650irq/133-enp0s29f1:rx-008:40:520
2756016991311,1cyclictest60650irq/133-enp0s29f1:rx-009:50:160
2756016991311,1cyclictest60650irq/133-enp0s29f1:rx-007:40:170
2756016991311,1cyclictest60650irq/133-enp0s29f1:rx-007:20:170
2756016991310,2cyclictest60650irq/133-enp0s29f1:rx-012:15:220
2756016991310,2cyclictest60650irq/133-enp0s29f1:rx-008:48:580
2756016991310,2cyclictest60650irq/133-enp0s29f1:rx-008:20:180
275602999129,2cyclictest3000703-21cpuspeed_turbos11:05:163
275602999129,2cyclictest2942369-21unixbench_multi10:05:263
275602999128,3cyclictest3046832-21kworker/u8:2+flush-259:012:16:403
275602999127,3cyclictest2797467-21sshd07:48:363
275602999126,4cyclictest2989898-21/usr/sbin/munin10:55:223
275602999121,2cyclictest2969040-21awk10:35:013
275602999121,2cyclictest140-1kworker/3:1H+kblockd08:45:003
275602199129,1cyclictest0-21swapper/111:45:151
275602199126,4cyclictest2948724-21apt-get10:15:121
2756021991210,1cyclictest2740666-21kworker/u8:1+events_unbound07:10:191
275602199120,2cyclictest3051960-21munin-run11:55:001
275601699129,2cyclictest2999967-21kworker/u8:4+flush-259:011:05:210
275601699128,3cyclictest0-21swapper/009:45:010
275601699128,2cyclictest2947552-21unixbench_multi10:10:270
275602999118,2cyclictest3025743-21unixbench_multi11:25:263
275602999118,2cyclictest0-21swapper/311:25:003
275602999117,3cyclictest2906855-21kworker/u8:0+flush-259:009:40:103
275602999117,2cyclictest0-21swapper/308:15:283
275602999117,2cyclictest0-21swapper/307:30:003
275602199118,2cyclictest0-21swapper/111:25:001
275602199118,1cyclictest0-21swapper/110:40:151
275602199117,3cyclictest3021037-21kworker/u8:0+events_unbound11:30:171
275602199117,3cyclictest2942485-21kworker/u8:0+events_unbound10:50:181
275602199117,2cyclictest2900627-21ntpq09:25:221
275602199116,3cyclictest2797627-21kworker/u8:2+flush-259:008:00:201
275601699119,1cyclictest60650irq/133-enp0s29f1:rx-008:30:220
275602999108,1cyclictest2891817-21sed09:20:193
275602999105,4cyclictest2901490-21kworker/u8:1+flush-259:009:35:273
275602999103,5cyclictest2952725-21ntpq10:15:223
27560299997,1cyclictest2953958-21cpuspeed_turbos10:20:143
27560299996,1cyclictest0-21swapper/309:45:173
27560299995,3cyclictest491ktimers/307:15:143
27560219997,1cyclictest0-21swapper/108:05:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional