You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-01 - 10:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Wed Jul 01, 2026 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2875726994943,4cyclictest2979188-21latency_hist20:50:012
2875719994439,3cyclictest335557-21systemd-journal22:20:090
2875726994137,3cyclictest3083021-21gpgv22:30:012
2875723994135,4cyclictest3145709-21latency_hist23:30:011
2875719994024,14cyclictest3214577-21latency_hist00:40:010
2875730993734,2cyclictest2891090-21kworker/u8:3+flush-259:020:00:013
2875730993734,2cyclictest2880346-21kworker/u8:2+flush-259:019:40:003
2875719993428,5cyclictest2989784-21kworker/u8:0+flush-259:021:55:260
2875726993023,5cyclictest3192548-21ntpq00:15:212
2875730992822,4cyclictest2900558-21ntpq19:30:223
2875726992822,4cyclictest2989558-21latency_hist21:00:002
2875723992815,3cyclictest335557-21systemd-journal23:45:101
2875723992813,5cyclictest3004248-21ntpq21:10:221
2875719992823,4cyclictest3156352-21diskstats23:40:130
2875723992721,5cyclictest3209152-21cp00:35:011
2875719992722,4cyclictest2937884-21latency_hist20:10:010
2875730992622,3cyclictest2891090-21kworker/u8:3+flush-259:019:25:183
2875726992623,2cyclictest2901422-21kworker/u8:0+flush-259:019:40:182
2875719992621,4cyclictest2967978-21ntpq20:35:200
2875719992612,3cyclictest3204053-21apt-get00:30:120
2875730992519,4cyclictest2958518-21latency_hist20:30:003
2875730992419,4cyclictest3161272-21cp23:45:003
2875730992418,4cyclictest3135460-21latency_hist23:20:003
2875726992420,3cyclictest2921974-21apt-get19:55:012
2875726992418,4cyclictest2927560-21latency_hist20:00:012
2875719992418,4cyclictest335557-21systemd-journal19:56:100
2875730992316,5cyclictest335557-21systemd-journal20:44:593
2875726992317,4cyclictest3093761-21latency_hist22:40:012
2875723992318,3cyclictest2958947-21diskstats20:30:141
2875719992317,4cyclictest3140836-21apt-get23:25:120
2875719992317,4cyclictest3140836-21apt-get23:25:120
2875719992317,4cyclictest2995030-21apt-get21:05:130
2875730992217,4cyclictest2906775-21/usr/sbin/munin19:40:153
2875730992213,6cyclictest3124916-21cp23:10:013
2875726992219,2cyclictest2891364-21apt-get19:25:112
2875726992217,4cyclictest3025747-21apt-get21:35:012
2875726992217,3cyclictest3125420-21apt-get23:10:112
2875726992217,3cyclictest2911698-21kworker/u8:1+flush-259:019:45:162
2875723992216,4cyclictest3155087-21ntpq23:35:221
2875719992217,4cyclictest3161091-21apt-get23:45:000
2875719992216,4cyclictest335557-21systemd-journal22:05:000
287571999221,19cyclictest1-21systemd21:15:010
2875730992116,3cyclictest3098753-21kworker/u8:4+flush-259:022:55:163
2875730992115,4cyclictest3140609-21kworker/u8:3+flush-259:000:30:173
2875730992115,4cyclictest2995005-21/usr/sbin/munin21:05:253
2875726992116,3cyclictest2891098-21latency_hist19:25:012
2875723992117,3cyclictest3124846-21gpgv23:10:001
2875719992115,4cyclictest3198629-21latency_hist00:25:010
2875730992016,3cyclictest2943075-21kworker/u8:0+flush-259:020:35:173
2875730992015,3cyclictest3140609-21kworker/u8:3+flush-259:000:35:183
2875730992015,3cyclictest2989784-21kworker/u8:0+flush-259:021:55:113
2875730992014,4cyclictest3098930-21latency_hist22:45:013
2875726992018,1cyclictest3140609-21kworker/u8:3+events_unbound23:40:182
2875726992016,3cyclictest2932737-21kworker/u8:1+flush-259:020:20:002
2875726992014,4cyclictest3067549-21latency_hist22:15:012
2875723992017,2cyclictest3140609-21kworker/u8:3+flush-259:023:50:161
2875723992016,3cyclictest3083253-21store22:30:011
2875723992014,1cyclictest2979193-21munin-run20:50:001
2875723992013,5cyclictest335557-21systemd-journal21:20:011
287571999201,2cyclictest3072656-21idleruntime-cro22:20:000
2875730991916,2cyclictest500-21systemd-logind22:05:013
2875730991913,4cyclictest509-21rs:main3
287572699194,4cyclictest3202948-21ntpq00:25:222
2875723991915,3cyclictest3083505-21base6422:30:011
2875723991913,5cyclictest3108343-21ntpq22:50:231
2875719991916,2cyclictest3083683-21apt-get22:30:110
2875730991815,2cyclictest2933385-21irqcore20:05:163
2875730991813,4cyclictest3025192-21kworker/u8:1+flush-259:022:20:013
2875730991811,4cyclictest2963705-21latency_hist20:35:003
287572699183,5cyclictest3205901-21cut00:30:182
2875726991811,5cyclictest3010251-21latency_hist21:20:012
2875723991816,1cyclictest2885946-21cron19:20:001
2875723991815,2cyclictest2891090-21kworker/u8:3+flush-259:019:55:151
2875723991814,3cyclictest2973159-21ntpq20:40:221
2875723991812,4cyclictest3088847-21apt-get22:35:121
2875719991814,3cyclictest2880346-21kworker/u8:2+flush-259:019:30:260
2875726991714,2cyclictest2964378-21irqcore20:35:172
287572399179,7cyclictest3188311-21idleruntime-cro00:15:001
2875723991713,3cyclictest2880346-21kworker/u8:2+flush-259:019:50:271
287572399170,15cyclictest3057248-21true22:05:001
2875719991714,2cyclictest60350irq/132-enp0s29f1:rx-021:00:010
2875719991714,2cyclictest2954032-21irqcore20:25:160
287573099169,5cyclictest509-21rs:main3
2875730991613,2cyclictest2989557-21sh20:59:593
2875730991612,3cyclictest3183186-21kworker/u8:2+flush-259:000:25:253
2875730991612,3cyclictest3015663-21kworker/u8:4+flush-259:021:30:113
287572699162,4cyclictest3045995-21ld21:50:222
2875726991612,3cyclictest3214689-21gpgv00:40:012
2875726991612,3cyclictest3005316-21kworker/u8:2+flush-259:021:35:202
2875723991611,3cyclictest490-21dbus-daemon21:05:011
2875723991611,3cyclictest3135723-21apt-get23:20:121
2875719991614,1cyclictest60350irq/132-enp0s29f1:rx-022:35:240
2875719991614,1cyclictest0-21swapper/019:20:000
2875719991613,2cyclictest60350irq/132-enp0s29f1:rx-023:50:130
2875719991613,2cyclictest60350irq/132-enp0s29f1:rx-019:25:230
2875719991613,2cyclictest3150829-21latency_hist23:34:590
2875719991613,2cyclictest3052037-21kworker/u8:2+flush-259:022:15:000
2875719991613,2cyclictest2989784-21kworker/u8:0+flush-259:021:20:160
2875719991612,3cyclictest3025744-21latency_hist21:35:000
2875719991610,4cyclictest3036533-21latency_hist21:45:010
287573099159,4cyclictest3183580-21diskstats00:10:093
2875730991513,1cyclictest148494-21kworker/3:1+events_power_efficient20:13:563
2875730991511,3cyclictest3115120-21irqcore23:00:173
2875726991512,2cyclictest3058512-21kthreadcore22:05:172
2875726991511,2cyclictest0-21swapper/221:09:592
2875726991510,4cyclictest3087720-21ntpq22:30:202
2875723991511,3cyclictest3209634-21/usr/sbin/munin00:35:171
2875723991511,3cyclictest2943293-21/usr/sbin/munin20:15:201
2875723991511,3cyclictest2891090-21kworker/u8:3+flush-259:020:00:161
287571999159,5cyclictest3182512-21latency_hist00:05:000
2875719991514,1cyclictest60350irq/132-enp0s29f1:rx-020:49:500
2875719991514,1cyclictest60350irq/132-enp0s29f1:rx-019:38:440
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-023:55:130
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-023:22:070
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-023:07:280
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-022:25:160
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-021:46:380
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-021:00:240
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-020:10:130
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-020:00:160
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-019:45:150
2875719991513,1cyclictest60350irq/132-enp0s29f1:rx-000:10:060
2875719991512,2cyclictest60350irq/132-enp0s29f1:rx-019:54:470
2875719991512,2cyclictest0-21swapper/020:45:000
2875719991512,1cyclictest3062401-21turbostat22:10:010
2875719991511,2cyclictest60350irq/132-enp0s29f1:rx-020:20:120
2875719991510,4cyclictest3202965-21ntpq00:25:220
287573099149,3cyclictest3025192-21kworker/u8:1+flush-259:021:50:263
2875730991411,2cyclictest3020577-21sh21:30:003
2875730991410,3cyclictest0-21swapper/322:54:593
287572699144,1cyclictest0-21swapper/221:00:132
287572699143,10cyclictest3177354-21awk00:00:002
2875726991411,1cyclictest3019675-21ntpq21:25:182
2875726991410,3cyclictest2875948-21apt-get19:10:132
287572699140,4cyclictest3056899-21sshd22:01:112
2875723991412,1cyclictest0-21swapper/123:29:591
2875723991411,2cyclictest2891090-21kworker/u8:3+flush-259:019:25:151
2875719991413,1cyclictest60350irq/132-enp0s29f1:rx-023:15:410
2875719991413,1cyclictest60350irq/132-enp0s29f1:rx-022:48:090
2875719991412,1cyclictest60350irq/132-enp0s29f1:rx-022:40:160
2875719991412,1cyclictest3119573-21turbostat23:05:000
2875719991412,1cyclictest2875714-21cyclictest00:15:080
2875719991411,2cyclictest60350irq/132-enp0s29f1:rx-022:55:210
287573099139,3cyclictest2979204-21kworker/u8:3+flush-259:021:00:233
287573099139,3cyclictest2911698-21kworker/u8:1+flush-259:019:50:163
287573099139,2cyclictest0-21swapper/320:25:003
287573099138,4cyclictest3140609-21kworker/u8:3+flush-259:000:00:253
287573099138,4cyclictest3005597-21cut21:15:153
287573099131,9cyclictest2948218-21awk20:20:003
287573099131,4cyclictest2916841-21munin-run19:50:003
2875730991310,2cyclictest3057719-21fschecks_time22:05:143
2875730991310,2cyclictest2979204-21kworker/u8:3+events_unbound21:10:183
2875730991310,2cyclictest2876013-21cpuspeed_turbos19:10:143
287572699133,7cyclictest0-21swapper/221:20:252
287572699133,2cyclictest3036313-21runrttasks21:42:222
287572699132,3cyclictest3077658-21runrttasks22:23:052
287572699131,2cyclictest3187410-21ntpq00:10:162
287572399139,3cyclictest3140609-21kworker/u8:3+flush-259:000:23:401
287572399139,3cyclictest3005316-21kworker/u8:2+flush-259:021:25:151
287572399139,3cyclictest2958747-21kworker/u8:2+flush-259:020:50:241
2875723991311,1cyclictest0-21swapper/122:25:011
2875723991310,2cyclictest3057497-21apt-get22:05:121
287571999139,3cyclictest60350irq/132-enp0s29f1:rx-019:20:220
2875719991311,1cyclictest60350irq/132-enp0s29f1:rx-020:30:260
2875719991311,1cyclictest60350irq/132-enp0s29f1:rx-020:15:190
2875719991310,2cyclictest2911676-21apt19:45:000
287573099129,2cyclictest3193845-21cpuspeed_turbos00:20:153
287573099129,2cyclictest3136131-21irqcore23:20:163
287573099129,2cyclictest3087569-21needreboot22:30:193
287573099129,1cyclictest0-21swapper/319:15:243
287573099128,3cyclictest0-21swapper/322:25:173
287573099126,4cyclictest3010493-21/usr/sbin/munin21:20:213
287573099121,6cyclictest3104117-21awk22:50:003
2875730991210,1cyclictest0-21swapper/323:50:233
287572699129,2cyclictest2963214-21unixbench_multi20:30:252
287572699126,4cyclictest3109241-21apt22:55:002
287572699123,6cyclictest0-21swapper/223:30:232
287572699122,9cyclictest0-21swapper/222:40:152
287572699122,2cyclictest0-21swapper/223:26:092
287572699122,2cyclictest0-21swapper/223:26:092
287572699121,3cyclictest2973009-21cat20:40:202
287572699121,2cyclictest3135875-21df_inode23:20:132
287572699121,10cyclictest3168012-21irqcore23:50:162
287572399129,2cyclictest3113960-21unixbench_multi22:55:251
287572399129,2cyclictest2896964-21irqcore19:30:161
2875723991210,1cyclictest0-21swapper/100:20:011
287571999129,2cyclictest3130521-21kworker/u8:1+flush-259:000:06:400
2875719991210,1cyclictest60350irq/132-enp0s29f1:rx-021:35:240
2875719991210,1cyclictest60350irq/132-enp0s29f1:rx-019:10:150
287573099118,2cyclictest2974549-21fschecks_time20:45:143
287573099118,2cyclictest0-21swapper/320:50:253
287573099116,3cyclictest3140811-21/usr/sbin/munin23:25:143
287573099116,3cyclictest3140811-21/usr/sbin/munin23:25:133
287573099111,8cyclictest3183166-21awk00:10:003
287573099111,8cyclictest3036539-21awk21:45:003
287573099111,2cyclictest3150830-21awk23:35:003
287573099110,6cyclictest3067559-21apt22:15:013
287572699116,3cyclictest3004181-21ntpq21:10:202
287572699115,5cyclictest2906124-21awk19:35:242
287572699115,4cyclictest3124716-21cron23:10:002
287572699115,1cyclictest191rcu_preempt00:00:202
287572699114,5cyclictest3151310-21fschecks_time23:35:142
287572699114,5cyclictest3036893-21tr21:45:142
287572699113,5cyclictest0-21swapper/223:00:192
287572699112,5cyclictest0-21swapper/222:15:122
287572699111,3cyclictest41-21ksoftirqd/220:05:152
287572699111,2cyclictest3194038-21hddtemp_smartct00:20:172
287572399119,1cyclictest509-21rs:main1
287572399118,2cyclictest2890690-21unixbench_singl19:20:261
287572399118,2cyclictest2876013-21cpuspeed_turbos19:10:141
287572399118,1cyclictest0-21swapper/123:35:011
287572399116,3cyclictest335557-21systemd-journal20:05:501
287572399115,4cyclictest3130291-21latency_hist23:15:001
287571999119,2cyclictest61150irq/133-enp0s29f1:tx-021:50:190
287571999119,1cyclictest60350irq/132-enp0s29f1:rx-023:35:170
287571999119,1cyclictest60350irq/132-enp0s29f1:rx-021:15:110
2875719991110,1cyclictest60350irq/132-enp0s29f1:rx-022:50:160
287573099108,1cyclictest0-21swapper/323:55:103
287573099107,2cyclictest3155000-21ntp_kernel_err23:35:223
287573099107,2cyclictest2931734-21needreboot20:00:193
287573099106,3cyclictest3093770-21turbostat22:39:593
287572699103,4cyclictest0-21swapper/223:15:032
287572399107,2cyclictest0-21swapper/121:50:141
287572399106,3cyclictest3140609-21kworker/u8:3+flush-259:000:09:391
28757269992,4cyclictest0-21swapper/200:05:142
28757239996,2cyclictest3099610-21irqcore22:45:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional