You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 11:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot4s.osadl.org (updated Mon Mar 30, 2026 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2828073993532,2cyclictest2874600-21kworker/u8:3+events_unbound20:15:252
2828073993527,3cyclictest2962413-21kworker/u8:2+events_unbound21:25:202
2828070993430,3cyclictest2874600-21kworker/u8:3+events_unbound20:10:231
2828073993328,4cyclictest3030130-21/usr/sbin/munin22:25:212
2828070993330,2cyclictest2874600-21kworker/u8:3+flush-259:020:30:171
2828073993229,2cyclictest3134080-21kworker/u8:0+events_unbound00:05:232
2828073993229,2cyclictest3045335-21kworker/u8:0+events_unbound23:35:182
2828073993229,2cyclictest2864106-21kworker/u8:1+flush-259:019:55:232
2828070993229,2cyclictest753709-21kworker/u8:1+flush-259:019:10:241
2828073993129,1cyclictest2874388-21kworker/u8:2+flush-259:020:05:252
2828073993128,2cyclictest2795863-21kworker/u8:0+events_unbound19:45:222
2828070993128,2cyclictest2899765-21kworker/u8:1+flush-259:020:20:011
2828070993127,3cyclictest3134080-21kworker/u8:0+flush-259:000:10:011
2828070993127,3cyclictest3024755-21kworker/u8:1+events_unbound00:40:011
2828070993127,3cyclictest2916092-21kworker/u8:2+flush-259:020:35:161
2828070993028,1cyclictest2874600-21kworker/u8:3+flush-259:021:05:231
2828070993027,2cyclictest3035036-21kworker/u8:3+flush-259:022:55:231
2828070993027,2cyclictest2874388-21kworker/u8:2+flush-259:020:00:241
2828070993027,2cyclictest2838348-21kworker/u8:4+events_unbound19:25:151
2828070993027,2cyclictest2795863-21kworker/u8:0+flush-259:019:20:121
2828070993026,3cyclictest3117886-21kworker/u8:2+flush-259:023:55:231
2828073992926,2cyclictest3024755-21kworker/u8:1+events_unbound00:15:222
2828073992926,2cyclictest2936684-21kworker/u8:1+flush-259:021:00:262
2828073992924,4cyclictest3024994-21apt-get22:20:112
2828073992922,2cyclictest2977932-21kworker/u8:2+events_unbound21:35:182
2828070992926,2cyclictest3134080-21kworker/u8:0+events_unbound00:10:141
2828070992925,3cyclictest3035036-21kworker/u8:3+flush-259:022:45:261
2828068992925,3cyclictest3066640-21apt-get23:00:090
2828068992924,3cyclictest2957291-21latency_hist21:15:000
2828078992824,3cyclictest2978170-21apt-get21:35:103
2828073992826,2cyclictest3024755-21kworker/u8:1+flush-259:023:15:132
2828073992825,2cyclictest3117886-21kworker/u8:2+events_unbound23:55:172
2828073992825,2cyclictest2977932-21kworker/u8:2+flush-259:021:45:162
2828073992824,3cyclictest3045335-21kworker/u8:0+events_unbound23:40:162
2828073992824,3cyclictest3035036-21kworker/u8:3+events_unbound22:45:242
2828073992724,2cyclictest3076855-21gpgconf23:10:012
2828073992724,2cyclictest2977932-21kworker/u8:2+events_unbound22:00:152
2828073992724,2cyclictest2952151-21kworker/u8:0+flush-259:021:15:202
2828073992722,4cyclictest2890096-21apt-get20:10:122
2828070992725,2cyclictest2977932-21kworker/u8:2+flush-259:022:10:221
2828070992725,1cyclictest2977932-21kworker/u8:2+events_unbound21:50:241
2828070992724,2cyclictest2977932-21kworker/u8:2+events_unbound22:05:131
2828070992724,2cyclictest2795863-21kworker/u8:0+events_unbound19:30:191
2828070992723,3cyclictest2869466-21/usr/sbin/munin19:50:131
2828073992623,2cyclictest2916092-21kworker/u8:2+flush-259:020:45:212
2828073992621,4cyclictest2988231-21latency_hist21:45:002
2828070992621,2cyclictest3045314-21latency_hist22:40:011
2828078992519,4cyclictest3108082-21latency_hist23:40:013
2828073992522,2cyclictest2911174-21apt-get20:30:132
2828070992522,2cyclictest3045335-21kworker/u8:0+events_unbound23:40:201
2828070992522,2cyclictest2874600-21kworker/u8:3+flush-259:021:10:251
2828068992521,3cyclictest3150020-21apt-get00:20:100
2828078992421,2cyclictest3123559-21sh23:55:003
2828078992419,4cyclictest509-21rs:main3
2828078992418,5cyclictest3096948-21ntpq23:25:213
2828073992421,2cyclictest3082267-21kworker/u8:2+flush-259:023:30:152
2828073992421,2cyclictest3024755-21kworker/u8:1+flush-259:023:00:162
2828073992421,2cyclictest2977932-21kworker/u8:2+events_unbound22:36:462
2828073992421,2cyclictest2905767-21kworker/u8:0+events_unbound20:25:112
2828073992420,3cyclictest2838348-21kworker/u8:4+events_unbound19:35:182
2828073992417,5cyclictest3035249-21/usr/sbin/munin22:30:142
2828070992421,2cyclictest3024755-21kworker/u8:1+flush-259:022:40:221
2828070992420,3cyclictest3154925-21kworker/u8:2+events_unbound00:30:291
2828070992420,3cyclictest2884955-21apt-get20:05:101
2828070992420,3cyclictest2795863-21kworker/u8:0+flush-259:019:40:161
2828070992416,3cyclictest2874600-21kworker/u8:3+events_unbound20:15:191
2828068992420,3cyclictest3024755-21kworker/u8:1+flush-259:023:44:590
2828078992316,5cyclictest2900651-21cat20:20:013
2828073992320,2cyclictest3082267-21kworker/u8:2+flush-259:023:20:212
2828073992320,2cyclictest3024755-21kworker/u8:1+flush-259:023:45:202
2828073992320,2cyclictest2916092-21kworker/u8:2+events_unbound20:40:232
2828070992320,2cyclictest3082267-21kworker/u8:2+flush-259:023:25:201
2828070992320,2cyclictest3035036-21kworker/u8:3+flush-259:022:50:171
2828070992320,2cyclictest3019018-21kworker/u8:0+events_unbound22:20:201
2828070992319,3cyclictest3024755-21kworker/u8:1+flush-259:000:00:261
2828068992318,4cyclictest2921211-21latency_hist20:40:020
282807899225,9cyclictest2896115-21sed20:15:183
2828078992217,3cyclictest753709-21kworker/u8:1+flush-259:019:10:163
2828073992221,1cyclictest3035036-21kworker/u8:3+flush-259:022:50:132
2828073992220,2cyclictest3117886-21kworker/u8:2+flush-259:023:50:212
2828073992219,2cyclictest3035036-21kworker/u8:3+flush-259:023:10:232
2828070992219,2cyclictest3035036-21kworker/u8:3+flush-259:023:00:211
2828070992219,2cyclictest2977932-21kworker/u8:2+flush-259:022:20:001
2828073992118,2cyclictest3024755-21kworker/u8:1+events_unbound22:55:212
2828073992118,2cyclictest3024755-21kworker/u8:1+events_unbound22:40:242
2828073992118,2cyclictest2988256-21kworker/u8:1+events_unbound21:55:242
2828073992118,2cyclictest2795863-21kworker/u8:0+events_unbound19:15:202
2828073992118,2cyclictest2795863-21kworker/u8:0+events_unbound19:15:202
2828070992118,2cyclictest3134080-21kworker/u8:0+flush-259:000:15:191
2828068992117,3cyclictest3160335-21apt-get00:30:120
2828068992116,4cyclictest252-21systemd-journal19:20:000
2828068992116,4cyclictest252-21systemd-journal19:20:000
2828078992013,5cyclictest2946985-21latency_hist21:05:003
2828073992017,2cyclictest3154925-21kworker/u8:2+events_unbound00:25:112
2828073992010,2cyclictest3154925-21kworker/u8:2+events_unbound00:30:242
2828070992018,2cyclictest2972560-21kworker/u8:0+events_unbound21:30:221
2828070992018,2cyclictest2972560-21kworker/u8:0+events_unbound21:30:221
2828070992017,2cyclictest3134080-21kworker/u8:0+events_unbound00:24:581
2828070992017,2cyclictest3024755-21kworker/u8:1+flush-259:023:15:201
2828070992016,3cyclictest3076653-21apt-get23:10:011
282806899203,5cyclictest3039433-21ps22:30:230
282807899195,4cyclictest3169520-21ld00:35:223
282807899194,8cyclictest2884343-21chrt20:00:543
282807899193,8cyclictest3003294-21taskset21:55:343
2828073991917,2cyclictest2795863-21kworker/u8:0+events_unbound19:30:372
2828073991916,2cyclictest3024755-21kworker/u8:1+flush-259:000:20:162
2828073991916,2cyclictest2936684-21kworker/u8:1+flush-259:020:55:182
2828073991916,2cyclictest2795863-21kworker/u8:0+events_unbound19:41:122
2828073991915,3cyclictest2936684-21kworker/u8:1+events_unbound21:20:282
2828070991918,1cyclictest2936684-21kworker/u8:1+events_unbound21:20:161
2828070991917,2cyclictest3045335-21kworker/u8:0+events_unbound23:05:221
2828070991916,3cyclictest2795863-21kworker/u8:0+events_unbound19:47:371
2828070991915,3cyclictest2916092-21kworker/u8:2+events_unbound20:51:251
2828068991915,3cyclictest3019830-21apt-get22:15:120
2828068991913,4cyclictest3102941-21latency_hist23:35:010
282807899185,4cyclictest0-21swapper/323:15:003
282807899184,7cyclictest2947410-21diskstats21:05:143
282807899184,5cyclictest2961159-21cut21:15:203
2828078991816,1cyclictest2967593-21munin-plugin-st21:25:003
2828073991816,2cyclictest3134080-21kworker/u8:0+events_unbound00:35:252
2828073991816,1cyclictest3024755-21kworker/u8:1+flush-259:000:15:002
2828073991815,2cyclictest3019018-21kworker/u8:0+flush-259:022:15:162
2828073991815,2cyclictest2977932-21kworker/u8:2+flush-259:022:05:262
2828073991815,2cyclictest2936684-21kworker/u8:1+events_unbound21:10:152
2828073991815,2cyclictest2874600-21kworker/u8:3+events_unbound20:20:212
2828073991815,2cyclictest2838348-21kworker/u8:4+events_unbound19:50:172
2828073991815,2cyclictest2795863-21kworker/u8:0+events_unbound19:25:292
2828073991814,3cyclictest2916092-21kworker/u8:2+events_unbound20:35:202
2828073991810,2cyclictest2936684-21kworker/u8:1+events_unbound21:30:152
2828073991810,2cyclictest2936684-21kworker/u8:1+events_unbound21:30:142
2828070991815,2cyclictest3082267-21kworker/u8:2+events_unbound23:30:281
2828070991814,3cyclictest3024755-21kworker/u8:1+flush-259:023:45:211
2828070991812,4cyclictest2930665-21ntpq20:45:201
2828070991812,4cyclictest2838324-21latency_hist19:20:001
2828070991812,4cyclictest2838324-21latency_hist19:20:001
2828070991812,2cyclictest2977932-21kworker/u8:2+events_unbound21:45:131
2828068991813,3cyclictest507-21in:imuxsock21:20:010
2828070991714,2cyclictest2988256-21kworker/u8:1+flush-259:021:55:231
2828070991713,3cyclictest3045335-21kworker/u8:0+events_unbound23:35:221
2828070991710,5cyclictest2879579-21latency_hist20:00:011
282806899171,6cyclictest2881384-21grep20:00:180
2828068991711,4cyclictest2843507-21latency_hist19:25:010
282807899166,3cyclictest0-21swapper/300:30:173
282807899165,3cyclictest3014564-21hddtemp_smartct22:10:143
282807899164,3cyclictest3138382-21memory00:05:203
282807899163,8cyclictest2910553-21chrt20:25:513
282807899163,6cyclictest832-21runrttasks22:29:163
282807899163,4cyclictest3044596-21sensors22:35:233
282807899163,3cyclictest2842634-21collect219:20:193
2828078991613,2cyclictest3112422-21ntp_offset23:40:223
2828078991613,2cyclictest3024755-21kworker/u8:1+flush-259:022:35:013
2828070991614,1cyclictest2967852-21apt-get21:25:151
2828070991613,2cyclictest2936684-21kworker/u8:1+flush-259:020:55:151
2828070991613,2cyclictest2838348-21kworker/u8:4+events_unbound19:35:381
2828070991612,3cyclictest3082267-21kworker/u8:2+events_unbound23:20:191
282806899163,3cyclictest2846011-21cat19:25:190
2828068991614,1cyclictest60350irq/132-enp0s29f1:rx-020:45:150
2828068991614,1cyclictest60350irq/132-enp0s29f1:rx-019:35:240
2828068991613,2cyclictest60350irq/132-enp0s29f1:rx-020:30:140
2828068991612,3cyclictest2941834-21turbostat20:59:590
2828068991612,2cyclictest2983099-21turbostat21:40:000
2828068991611,1cyclictest191rcu_preempt23:14:100
2828068991611,1cyclictest191rcu_preempt19:40:580
282807899159,4cyclictest3013985-21cron22:10:003
282807899159,4cyclictest2998504-21idleruntime-cro21:55:003
282807899159,4cyclictest2988232-21cron21:45:003
282807899159,4cyclictest2916033-21cron20:35:003
282807899158,5cyclictest3091674-21ntpq23:20:203
282807899158,5cyclictest2993356-21cron21:50:013
282807899158,4cyclictest3160079-21basename00:30:013
282807899155,8cyclictest2977912-21latency_hist21:35:003
282807899155,8cyclictest2977912-21latency_hist21:35:003
2828078991511,3cyclictest3061311-21base6422:55:023
2828078991510,4cyclictest3134343-21apt-config00:05:003
2828078991510,3cyclictest3050479-21apt22:45:003
2828078991510,3cyclictest2795863-21kworker/u8:0+flush-259:019:30:103
282807399157,3cyclictest2977932-21kworker/u8:2+events_unbound22:10:212
2828073991512,2cyclictest2977932-21kworker/u8:2+events_unbound21:54:362
2828070991512,2cyclictest3024755-21kworker/u8:1+events_unbound22:31:371
2828070991512,2cyclictest3019018-21kworker/u8:0+events_unbound22:25:151
2828070991512,2cyclictest2936684-21kworker/u8:1+events_unbound21:00:281
2828070991511,3cyclictest2905767-21kworker/u8:0+events_unbound20:26:381
282806899153,7cyclictest3007967-21ntpq22:00:210
282806899153,3cyclictest2853557-21modprobe19:32:370
282806899151,5cyclictest2936745-21latency_hist20:55:000
282806899151,5cyclictest2869944-21sed19:50:150
2828068991513,1cyclictest60350irq/132-enp0s29f1:rx-022:25:190
2828068991513,1cyclictest60350irq/132-enp0s29f1:rx-022:20:150
2828068991513,1cyclictest60350irq/132-enp0s29f1:rx-019:11:370
2828068991513,1cyclictest60350irq/132-enp0s29f1:rx-000:05:240
2828068991512,2cyclictest60350irq/132-enp0s29f1:rx-021:45:120
2828068991512,2cyclictest60350irq/132-enp0s29f1:rx-020:05:200
2828068991511,3cyclictest252-21systemd-journal00:38:410
282806899150,5cyclictest3114681-21cat23:45:170
282807899149,3cyclictest3102031-21ntpq23:30:203
282807899148,4cyclictest3154027-21ntpq00:20:203
282807899148,4cyclictest2936187-21unixbench_multi20:50:243
282807899148,3cyclictest2869634-21grep19:50:123
282807899147,6cyclictest2926376-21awk20:44:593
282807899147,5cyclictest3148954-21nvmesmart_nvme000:15:233
282807899147,5cyclictest3128920-21latency_hist00:00:013
282807899147,4cyclictest3025120-21tr22:20:123
282807899146,6cyclictest3021534-21kthreadcore22:15:193
282807899146,6cyclictest2857075-21kthreadcore19:35:183
282807899144,3cyclictest3118150-21runrttasks23:47:183
282807899144,1cyclictest2885239-21fw_conntrack20:05:133
282807899143,3cyclictest3066929-21fw_forwarded_lo23:00:123
282807899143,2cyclictest3139878-21df_inode00:10:143
282807899143,2cyclictest3082664-21df_inode23:15:133
282807899143,2cyclictest2874790-21df_inode19:55:123
282807899142,4cyclictest2931225-21sshd20:46:223
2828078991410,3cyclictest2952390-21apt-get21:10:123
2828073991411,2cyclictest2874388-21kworker/u8:2+events_unbound20:00:152
2828073991411,2cyclictest2795863-21kworker/u8:0+events_unbound19:20:122
2828070991412,1cyclictest3004359-21irqcore22:00:161
2828070991411,2cyclictest3156192-21kthreadcore00:25:161
282806899149,4cyclictest252-21systemd-journal23:09:550
282806899144,1cyclictest3159153-21ntpq00:25:200
282806899143,2cyclictest3061482-21apt-get22:55:090
282806899142,3cyclictest16-21ksoftirqd/023:35:200
282806899142,2cyclictest3060747-21sshd22:54:380
2828068991413,1cyclictest60350irq/132-enp0s29f1:rx-023:25:050
2828068991413,1cyclictest60350irq/132-enp0s29f1:rx-022:14:330
2828068991412,2cyclictest61150irq/133-enp0s29f1:tx-022:45:180
2828068991412,1cyclictest60350irq/132-enp0s29f1:rx-023:55:180
2828068991412,1cyclictest60350irq/132-enp0s29f1:rx-023:50:150
2828068991412,1cyclictest60350irq/132-enp0s29f1:rx-022:05:150
2828068991412,1cyclictest60350irq/132-enp0s29f1:rx-021:05:140
2828068991412,1cyclictest60350irq/132-enp0s29f1:rx-020:17:150
2828068991412,1cyclictest60350irq/132-enp0s29f1:rx-020:10:260
2828068991411,2cyclictest60350irq/132-enp0s29f1:rx-023:15:130
2828068991411,2cyclictest3134080-21kworker/u8:0+events_unbound00:16:560
2828068991411,2cyclictest3088079-21grep23:20:170
282806899140,4cyclictest2910857-21modprobe20:29:360
282806899140,4cyclictest2865633-21grep19:45:180
282807899139,3cyclictest2972707-21cron21:30:003
282807899139,3cyclictest2838324-21latency_hist19:20:003
282807899139,3cyclictest2838324-21latency_hist19:20:003
282807899138,3cyclictest2920283-21ntpq20:35:203
282807899138,3cyclictest2864051-21cron19:45:003
282807899138,3cyclictest0-21swapper/322:00:123
282807899137,4cyclictest2869222-21awk19:50:003
282807899136,5cyclictest3050713-21munin-node22:45:143
282807899136,5cyclictest2939699-21cut20:55:183
282807899136,5cyclictest2894995-21systemd-run20:15:013
282807899136,4cyclictest3055850-21acpi22:50:093
2828078991310,1cyclictest0-21swapper/319:25:173
2828073991311,2cyclictest3082267-21kworker/u8:2+flush-259:023:25:152
2828073991311,1cyclictest0-21swapper/221:10:012
2828073991310,2cyclictest753709-21kworker/u8:1+events_unbound19:11:082
282807099139,3cyclictest2977932-21kworker/u8:2+flush-259:021:35:101
2828070991310,2cyclictest2952151-21kworker/u8:0+events_unbound21:15:151
282806899136,5cyclictest2926369-21latency_hist20:45:000
282806899135,1cyclictest191rcu_preempt00:10:210
282806899133,2cyclictest2983560-21fschecks_count21:40:140
282806899131,7cyclictest3045537-21runrttasks22:40:020
282806899131,6cyclictest2962818-21df_inode21:20:130
2828068991311,1cyclictest60350irq/132-enp0s29f1:rx-021:30:260
2828068991311,1cyclictest60350irq/132-enp0s29f1:rx-021:30:250
2828068991311,1cyclictest60350irq/132-enp0s29f1:rx-021:00:170
2828068991311,1cyclictest60350irq/132-enp0s29f1:rx-000:00:190
2828068991310,2cyclictest60350irq/132-enp0s29f1:rx-020:20:120
2828068991310,2cyclictest3040524-21cpuspeed_turbos22:35:130
2828068991310,2cyclictest2999035-21fschecks_time21:55:160
282807399129,2cyclictest3024755-21kworker/u8:1+events_unbound00:00:112
282807099129,2cyclictest3117886-21kworker/u8:2+events_unbound23:54:231
282807099128,3cyclictest2874600-21kworker/u8:3+events_unbound21:41:381
282806899127,3cyclictest2997631-21ntpq21:50:210
282806899125,5cyclictest2967994-21grep21:25:160
282807099118,2cyclictest2874600-21kworker/u8:3+events_unbound20:40:101
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional