You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 10:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Sat Nov 15, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2989778993831,4cyclictest3120978-21cp21:15:002
2989778993531,2cyclictest3162610-21kworker/u8:3+flush-259:022:25:162
2989773993329,3cyclictest3136817-21kworker/u8:3+flush-259:021:30:201
2989782993226,4cyclictest3000402-21cp19:20:003
2989765993227,4cyclictest3120906-21gpgv21:15:010
2989773993127,3cyclictest3078806-21kworker/u8:3+flush-259:021:10:181
2989773993127,3cyclictest3068666-21kworker/u8:2+flush-259:020:45:001
2989765993025,4cyclictest3115343-21apt-get21:10:000
2989778992926,2cyclictest3266445-21kworker/u8:3+flush-259:023:35:272
2989782992720,5cyclictest3188523-21latency_hist22:20:013
2989778992723,3cyclictest3168013-21kworker/u8:0+flush-259:000:25:202
2989778992720,5cyclictest3224571-21latency_hist22:55:012
2989778992720,5cyclictest3094721-21latency_hist20:50:002
2989773992723,3cyclictest3251038-21apt-get23:20:121
2989765992724,2cyclictest3136817-21kworker/u8:3+flush-259:021:40:000
2989778992624,1cyclictest3209144-21latency_hist22:40:012
2989778992623,2cyclictest3168013-21kworker/u8:0+events_unbound22:50:002
2989773992620,4cyclictest2698871-21systemd-journal22:34:441
2989778992521,3cyclictest3068666-21kworker/u8:2+events_unbound21:05:222
2989778992519,5cyclictest3292153-21kworker/u8:1+flush-259:000:35:152
2989773992521,3cyclictest2698871-21systemd-journal21:20:001
2989765992521,3cyclictest3147545-21diskstats21:40:140
2989765992520,3cyclictest2698871-21systemd-journal21:35:000
2989778992420,3cyclictest3042042-21kworker/u8:0+events_unbound20:35:162
2989773992420,3cyclictest3047130-21gpgv20:05:011
2989773992418,4cyclictest2698871-21systemd-journal19:50:011
2989773992417,5cyclictest2698871-21systemd-journal23:09:591
2989765992418,4cyclictest3126494-21latency_hist21:20:000
2989782992318,3cyclictest3214289-21kworker/u8:1+flush-259:022:45:173
2989778992320,2cyclictest3068666-21kworker/u8:2+events_unbound21:00:162
2989778992319,3cyclictest3168014-21kworker/u8:2+flush-259:022:00:162
2989778992319,3cyclictest2974428-21kworker/u8:0+events_unbound19:20:182
2989765992318,4cyclictest3089282-21cat20:45:000
2989765992317,5cyclictest3317696-21latency_hist00:25:000
2989782992218,3cyclictest3245405-21kworker/u8:2+flush-259:023:35:013
2989778992219,2cyclictest3276489-21kworker/u8:2+flush-259:023:50:262
2989778992219,2cyclictest3042042-21kworker/u8:0+events_unbound20:05:192
2989778992219,2cyclictest3021212-21kworker/u8:1+flush-259:020:00:212
2989778992219,2cyclictest2989706-21kworker/u8:1+events_unbound19:16:362
2989778992219,2cyclictest2974428-21kworker/u8:0+events_unbound19:30:452
2989778992218,3cyclictest3230369-21kworker/u8:4+flush-259:023:10:012
2989778992218,3cyclictest3168013-21kworker/u8:0+flush-259:000:15:202
2989778992218,3cyclictest3110204-21kworker/u8:1+flush-259:021:50:212
2989778992215,5cyclictest2698871-21systemd-journal00:30:022
2989773992218,3cyclictest3328760-21apt-get00:35:111
2989765992217,4cyclictest3021212-21kworker/u8:1+flush-259:020:20:170
2989765992217,3cyclictest2995142-21apt-get19:15:120
2989782992116,3cyclictest3276739-21apt-get23:45:123
2989778992118,2cyclictest3214289-21kworker/u8:1+events_unbound23:15:282
2989778992118,2cyclictest3021212-21kworker/u8:1+events_unbound19:50:182
2989778992117,3cyclictest3286770-21kworker/u8:3+flush-259:000:10:262
2989778992117,3cyclictest3168013-21kworker/u8:0+events_unbound23:55:162
2989778992117,3cyclictest3168013-21kworker/u8:0+events_unbound22:30:142
2989778992115,4cyclictest3214289-21kworker/u8:1+flush-259:023:25:192
2989778992115,4cyclictest3214289-21kworker/u8:1+flush-259:023:25:182
2989773992118,2cyclictest3167760-21cron22:00:001
2989773992117,3cyclictest3089136-21apt-get20:45:021
2989773992114,6cyclictest0-21swapper/119:35:001
2989782992017,2cyclictest4114158-21kworker/u8:2+events_unbound19:25:143
2989782992016,2cyclictest3168013-21kworker/u8:0+flush-259:000:25:183
2989778992017,2cyclictest4114158-21kworker/u8:2+events_unbound19:10:152
2989778992016,3cyclictest3302221-21kworker/u8:2+events_unbound00:24:452
2989778992014,4cyclictest3193665-21latency_hist22:25:012
2989778992013,5cyclictest3072695-21ntpq20:25:222
2989773992017,2cyclictest3162610-21kworker/u8:3+flush-259:022:25:011
2989773992016,3cyclictest3021212-21kworker/u8:1+flush-259:020:20:141
2989765992017,2cyclictest3224559-21cron22:54:590
2989765992017,2cyclictest3168014-21kworker/u8:2+flush-259:022:30:190
2989778991916,2cyclictest3286770-21kworker/u8:3+flush-259:000:10:012
2989778991916,2cyclictest3168013-21kworker/u8:0+events_unbound23:10:272
2989778991916,2cyclictest3136817-21kworker/u8:3+events_unbound21:35:162
2989778991916,2cyclictest3000682-21kworker/u8:3+flush-259:019:55:162
2989778991915,3cyclictest3021212-21kworker/u8:1+events_unbound19:45:222
2989765991916,2cyclictest3000063-21rs:main0
2989765991915,3cyclictest0-21swapper/021:05:000
2989782991816,1cyclictest3120903-21systemd-run21:15:003
2989782991815,2cyclictest3110204-21kworker/u8:1+events_unbound21:55:183
2989782991813,4cyclictest3036871-21/usr/sbin/munin19:55:143
2989778991815,2cyclictest3119910-21kworker/u8:0+flush-259:021:40:212
2989778991814,3cyclictest4114158-21kworker/u8:2+events_unbound19:25:192
2989778991814,3cyclictest3266445-21kworker/u8:3+events_unbound23:40:202
2989773991816,1cyclictest0-21swapper/121:05:001
2989773991814,3cyclictest3312807-21apt-get00:20:121
2989765991814,3cyclictest4114158-21kworker/u8:2+flush-259:019:10:160
2989765991814,3cyclictest3162612-21kworker/u8:4+flush-259:022:00:000
2989782991714,2cyclictest3323518-21kworker/u8:5+flush-259:000:30:173
2989782991714,2cyclictest3068002-21idleruntime-cro20:25:003
2989782991713,3cyclictest3168013-21kworker/u8:0+flush-259:000:15:173
2989778991714,2cyclictest3021212-21kworker/u8:1+events_unbound20:20:202
2989778991713,3cyclictest3214289-21kworker/u8:1+events_unbound23:45:022
2989778991713,2cyclictest0-21swapper/222:59:592
2989778991710,5cyclictest3297062-21latency_hist00:05:002
2989773991714,2cyclictest3110204-21kworker/u8:1+flush-259:021:55:001
2989765991714,2cyclictest3057988-21apt-get20:15:120
2989765991714,2cyclictest3042502-21irqcore20:00:180
2989765991713,3cyclictest3026589-21apt-get19:45:160
2989765991713,2cyclictest2698871-21systemd-journal22:58:440
2989765991710,5cyclictest2698871-21systemd-journal22:44:590
2989782991614,1cyclictest0-21swapper/323:25:013
2989782991614,1cyclictest0-21swapper/322:50:173
2989782991612,3cyclictest2698871-21systemd-journal20:50:003
298977899168,7cyclictest3062880-21turbostat20:20:002
2989778991613,2cyclictest3068666-21kworker/u8:2+events_unbound20:42:452
2989778991612,3cyclictest3168014-21kworker/u8:2+flush-259:022:15:222
2989778991612,3cyclictest3119910-21kworker/u8:0+events_unbound21:25:192
2989778991612,3cyclictest3068666-21kworker/u8:2+events_unbound20:51:462
2989778991612,3cyclictest3021212-21kworker/u8:1+events_unbound20:30:132
2989778991612,3cyclictest3000682-21kworker/u8:3+flush-259:019:40:242
2989773991613,2cyclictest3209147-21sh22:40:001
2989765991614,1cyclictest61450irq/134-enp0s29f1:tx-000:15:160
2989765991613,2cyclictest60650irq/133-enp0s29f1:rx-020:05:240
2989765991613,2cyclictest3322913-21unin-run00:30:010
2989765991613,2cyclictest3168013-21kworker/u8:0+flush-259:022:15:270
298976599160,15cyclictest1-21systemd00:34:440
2989782991513,1cyclictest3211699-21kthreadcore22:40:213
2989782991512,2cyclictest3192730-21ntp_kernel_err22:20:193
2989782991512,2cyclictest3089117-21cron20:45:003
2989782991512,2cyclictest3078806-21kworker/u8:3+flush-259:021:00:263
2989778991511,3cyclictest3168014-21kworker/u8:2+events_unbound22:41:462
298977399159,4cyclictest3307804-21diskstats00:15:151
2989773991512,2cyclictest3168014-21kworker/u8:2+flush-259:022:15:161
2989773991512,2cyclictest3131293-21kworker/u8:2+flush-259:021:45:261
2989773991512,2cyclictest3073596-21turbostat20:30:001
2989773991511,3cyclictest3000682-21kworker/u8:3+flush-259:019:54:591
2989765991514,1cyclictest60650irq/133-enp0s29f1:rx-022:10:250
2989765991513,2cyclictest60650irq/133-enp0s29f1:rx-019:20:230
2989765991513,1cyclictest60650irq/133-enp0s29f1:rx-023:50:360
2989765991513,1cyclictest60650irq/133-enp0s29f1:rx-020:25:250
2989765991513,1cyclictest60650irq/133-enp0s29f1:rx-019:50:140
2989765991510,4cyclictest3249901-21ntpq23:15:230
298978299149,4cyclictest3015079-21ntpq19:30:223
298978299148,4cyclictest3051641-21ntpq20:05:243
2989782991412,1cyclictest3042490-21irqcore20:00:183
2989782991412,1cyclictest0-21swapper/321:55:003
2989782991411,2cyclictest3316755-21ntp_kernel_err00:20:213
2989782991411,2cyclictest3120380-21kworker/u8:4+flush-259:021:20:213
298977899149,4cyclictest3136817-21kworker/u8:3+flush-259:021:30:262
2989778991412,2cyclictest3245405-21kworker/u8:2+events_unbound23:20:292
2989778991410,3cyclictest3163026-21fschecks_count21:55:152
2989778991410,3cyclictest3078806-21kworker/u8:3+flush-259:021:00:002
2989773991412,1cyclictest0-21swapper/121:10:001
2989773991411,2cyclictest3245405-21kworker/u8:2+flush-259:023:30:011
2989773991411,2cyclictest3245405-21kworker/u8:2+flush-259:023:30:001
2989765991413,1cyclictest60650irq/133-enp0s29f1:rx-023:00:520
2989765991413,1cyclictest60650irq/133-enp0s29f1:rx-022:23:270
2989765991413,1cyclictest60650irq/133-enp0s29f1:rx-020:35:180
2989765991413,1cyclictest60650irq/133-enp0s29f1:rx-019:37:020
2989765991412,1cyclictest60650irq/133-enp0s29f1:rx-023:55:150
2989765991412,1cyclictest60650irq/133-enp0s29f1:rx-023:40:270
2989765991412,1cyclictest60650irq/133-enp0s29f1:rx-022:45:250
2989765991412,1cyclictest60650irq/133-enp0s29f1:rx-022:35:210
2989765991412,1cyclictest60650irq/133-enp0s29f1:rx-022:09:370
2989765991412,1cyclictest60650irq/133-enp0s29f1:rx-022:00:190
2989765991412,1cyclictest60650irq/133-enp0s29f1:rx-020:50:250
2989765991412,1cyclictest60650irq/133-enp0s29f1:rx-020:30:170
2989765991411,2cyclictest60650irq/133-enp0s29f1:rx-020:58:440
2989765991411,2cyclictest3286770-21kworker/u8:3+flush-259:000:10:170
2989765991411,2cyclictest3036964-21cpuspeed_turbos19:55:140
2989765991410,3cyclictest2698871-21systemd-journal20:11:440
298978299136,5cyclictest492-21dbus-daemon21:10:003
2989782991311,1cyclictest0-21swapper/323:00:173
2989782991310,2cyclictest3273894-21kthreadcore23:40:203
2989782991310,2cyclictest3214289-21kworker/u8:1+flush-259:022:55:173
2989782991310,2cyclictest0-21swapper/323:50:163
2989778991310,2cyclictest4114158-21kworker/u8:2+events_unbound19:38:452
2989778991310,2cyclictest3230369-21kworker/u8:4+flush-259:023:00:252
2989778991310,2cyclictest3131293-21kworker/u8:2+events_unbound21:45:152
298977399139,3cyclictest3302469-21apt-get00:10:121
298977399139,2cyclictest0-21swapper/123:15:211
2989773991310,2cyclictest3162610-21kworker/u8:3+flush-259:022:40:191
2989773991310,2cyclictest2974428-21kworker/u8:0+flush-259:019:23:501
2989773991310,1cyclictest0-21swapper/120:35:191
298976599138,1cyclictest60650irq/133-enp0s29f1:rx-019:30:240
2989765991312,1cyclictest60650irq/133-enp0s29f1:rx-021:45:060
2989765991311,1cyclictest60650irq/133-enp0s29f1:rx-022:26:520
2989765991311,1cyclictest60650irq/133-enp0s29f1:rx-021:25:250
2989765991310,2cyclictest60650irq/133-enp0s29f1:rx-023:10:220
2989765991310,2cyclictest3328827-21cpuspeed_turbos00:35:130
298978299129,2cyclictest3297752-21irqcore00:05:193
298978299129,2cyclictest3266550-21cpuspeed_turbos23:35:163
298978299129,2cyclictest3235645-21cpuspeed_turbos23:05:173
298978299129,2cyclictest3168108-21cpuspeed_turbos22:00:133
298978299128,3cyclictest3168013-21kworker/u8:0+flush-259:023:55:263
298978299128,3cyclictest3168013-21kworker/u8:0+flush-259:022:10:263
298978299127,4cyclictest2698871-21systemd-journal19:55:003
298978299121,3cyclictest3057731-21latency_hist20:15:013
2989782991210,1cyclictest0-21swapper/323:14:593
2989782991210,1cyclictest0-21swapper/321:40:003
298977899128,3cyclictest3168013-21kworker/u8:0+events_unbound22:05:122
298977399129,2cyclictest3261762-21irqcore23:30:181
298977399128,3cyclictest3068666-21kworker/u8:2+flush-259:020:55:171
298976599128,3cyclictest2974428-21kworker/u8:0+flush-259:019:30:000
2989765991210,1cyclictest60650irq/133-enp0s29f1:rx-023:45:140
2989765991210,1cyclictest60650irq/133-enp0s29f1:rx-020:45:190
2989765991210,1cyclictest3286770-21kworker/u8:3-flush-259:000:00:170
298978299119,1cyclictest0-21swapper/323:15:143
298978299117,3cyclictest3042042-21kworker/u8:0+flush-259:020:35:253
298978299117,3cyclictest2698871-21systemd-journal21:32:013
298978299117,2cyclictest3026308-21cron19:45:003
298978299116,4cyclictest3026752-21diskstats19:45:183
298978299111,2cyclictest3203966-21latency_hist22:35:013
298977899119,1cyclictest0-21swapper/223:30:262
298977399119,1cyclictest0-21swapper/121:30:001
298977399118,2cyclictest3276008-21unixbench_multi23:40:261
298977399118,2cyclictest3224952-21cpuspeed_turbos22:55:141
298977399118,2cyclictest3147805-21irqcore21:40:181
298977399117,3cyclictest503-21systemd-logind20:15:011
298977399117,2cyclictest0-21swapper/120:15:251
298977399110,2cyclictest3052544-21awk20:10:001
298976599119,1cyclictest60650irq/133-enp0s29f1:rx-023:25:270
298976599119,1cyclictest60650irq/133-enp0s29f1:rx-023:25:260
298976599118,2cyclictest60650irq/133-enp0s29f1:rx-021:50:120
298976599117,3cyclictest3235691-21cut23:05:160
298976599116,4cyclictest3297455-21grep00:05:140
298978299108,1cyclictest0-21swapper/320:30:183
298978299107,2cyclictest3121666-21cpuspeed_turbos21:15:163
298978299107,2cyclictest3058056-21cpuspeed_turbos20:15:133
298978299106,2cyclictest3260104-21head23:25:213
298978299106,2cyclictest3260104-21head23:25:213
298977399108,1cyclictest3182808-21timerwakeupswit22:10:251
298977399107,2cyclictest3221641-21kthreadcore22:50:191
298977399107,2cyclictest3126875-21cpuspeed_turbos21:20:171
298977399107,2cyclictest301rcuc/123:10:191
298977399106,3cyclictest3214289-21kworker/u8:1+flush-259:023:05:011
29897739997,1cyclictest0-21swapper/123:45:201
29897739995,3cyclictest3136817-21kworker/u8:3+flush-259:021:35:261
29897739994,4cyclictest3322899-21dpkg00:30:001
29897739994,3cyclictest3327625-21ntp_kernel_pll_00:30:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional