You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 10:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Wed Jan 07, 2026 00:44:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3948880993831,5cyclictest4167374-21/usr/sbin/munin22:40:131
3948883993630,4cyclictest4052367-21diskstats20:50:132
3948878993427,5cyclictest4104019-21latency_hist21:40:010
3948878993427,5cyclictest4000402-21latency_hist20:00:000
3948883993327,4cyclictest4062697-21latency_hist21:00:002
3948883993026,3cyclictest61893-21kworker/u8:3+flush-259:000:05:002
3948883993024,4cyclictest4109357-21/usr/sbin/munin21:45:122
3948878992925,3cyclictest3990343-21apt-get19:50:110
3948888992621,3cyclictest4099093-21/usr/sbin/munin21:35:143
3948883992520,3cyclictest4088532-21latency_hist21:25:002
3948878992519,4cyclictest4161446-21kworker/u8:1+flush-259:022:50:160
3948878992519,4cyclictest4134911-21latency_hist22:10:010
3948888992420,2cyclictest0-21swapper/319:14:593
3948880992419,3cyclictest3949065-21/usr/sbin/munin19:10:151
3948880992418,4cyclictest4177060-21ntpq22:45:221
3948878992417,5cyclictest4171395-21ntpq22:40:190
3948878992417,5cyclictest4082417-21ntpq21:15:200
3948888992317,4cyclictest4031576-21/usr/sbin/munin20:30:143
3948878992319,3cyclictest4129779-21kworker/u8:2+flush-259:022:40:000
3948878992317,4cyclictest88588-21latency_hist00:30:000
3948878992317,4cyclictest4021033-21latency_hist20:20:010
394887899231,20cyclictest1-21systemd00:33:390
3948880992216,4cyclictest4188427-21/usr/sbin/munin23:00:151
3948878992217,3cyclictest4067785-21kworker/u8:1+flush-259:021:40:170
3948878992216,4cyclictest3964033-21kworker/u8:3+flush-259:019:30:000
3948878992215,5cyclictest9748-21apt-get23:15:010
3948883992116,3cyclictest252-21systemd-journal21:35:002
3948880992111,9cyclictest4187269-21ntp_kernel_err22:55:221
3948878992117,3cyclictest4145617-21kworker/u8:1+events_unbound22:25:000
3948878992117,3cyclictest31235-21kworker/u8:2+flush-259:023:40:000
3948878992116,3cyclictest3895120-21kworker/u8:0+events_unbound19:35:180
394888099208,11cyclictest3958116-21needreboot19:15:191
3948880992015,4cyclictest4005857-21apt-get20:05:131
3948878992017,2cyclictest4098840-21cron21:35:000
3948878992016,3cyclictest4005614-21kworker/u8:0+flush-259:020:20:150
3948888991915,3cyclictest4161446-21kworker/u8:1+flush-259:023:35:003
3948888991915,3cyclictest4041424-21kworker/u8:2+flush-259:020:45:163
3948888991915,2cyclictest0-21swapper/300:15:003
3948888991915,2cyclictest0-21swapper/300:15:003
3948883991917,1cyclictest0-21swapper/222:40:012
3948883991916,1cyclictest0-21swapper/221:20:002
3948883991915,3cyclictest4188069-21kworker/u8:3+flush-259:023:35:232
3948883991913,4cyclictest25227-21ntpq23:25:212
394888099197,11cyclictest0-21swapper/121:55:001
394888099197,11cyclictest0-21swapper/121:55:001
3948880991916,1cyclictest0-21swapper/122:40:001
3948880991914,4cyclictest4151497-21kworker/u8:0+flush-259:022:25:171
3948878991915,3cyclictest62505-21kworker/u8:4+flush-259:000:15:270
3948878991915,3cyclictest4108158-21kworker/u8:0+flush-259:022:15:140
3948878991915,3cyclictest4067785-21kworker/u8:1+flush-259:022:00:180
3948878991914,4cyclictest4088554-21kworker/u8:0+flush-259:021:25:240
3948878991912,5cyclictest252-21systemd-journal21:52:400
3948878991912,5cyclictest252-21systemd-journal21:52:400
3948883991816,1cyclictest3990085-21sh19:49:592
3948883991815,1cyclictest0-21swapper/222:35:002
3948883991811,5cyclictest252-21systemd-journal20:10:002
394888099188,1cyclictest0-21swapper/120:40:191
394888099187,1cyclictest0-21swapper/100:35:011
3948880991813,4cyclictest93964-21/usr/sbin/munin00:35:221
394888099181,2cyclictest252-21systemd-journal20:20:011
3948878991815,2cyclictest61893-21kworker/u8:3+events_unbound00:10:180
3948878991815,2cyclictest61893-21kworker/u8:3+events_unbound00:10:180
3948878991815,2cyclictest3953965-21idleruntime-cro19:15:000
3948878991814,3cyclictest3900547-21kworker/u8:1+flush-259:019:40:210
3948878991811,5cyclictest61395-21ntpq00:00:210
3948888991715,1cyclictest0-21swapper/320:40:013
3948888991714,2cyclictest4083352-21sh21:20:003
3948888991714,2cyclictest4067860-21idleruntime-cro21:05:003
3948888991711,4cyclictest252-21systemd-journal23:30:003
3948883991714,1cyclictest0-21swapper/223:55:002
3948883991713,3cyclictest4156422-21idleruntime-cro22:30:002
394888099177,1cyclictest0-21swapper/119:40:131
3948878991714,2cyclictest60350irq/132-enp0s29f1:rx-020:00:180
3948878991713,3cyclictest4161446-21kworker/u8:1+flush-259:023:40:230
3948878991712,3cyclictest4161446-21kworker/u8:1+flush-259:000:35:190
3948878991712,3cyclictest4057338-21gpgv20:55:000
3948878991711,5cyclictest4167368-21kworker/u8:3+flush-259:022:50:010
3948878991711,5cyclictest4151497-21kworker/u8:0+events_unbound23:20:190
3948878991711,4cyclictest57135-21latency_hist00:00:000
3948888991612,3cyclictest4193371-21latency_hist23:05:013
3948888991610,4cyclictest4135544-21cp22:10:013
394888899160,14cyclictest3959151-21true19:19:593
3948883991612,3cyclictest4161446-21kworker/u8:1+flush-259:022:50:252
394888099166,1cyclictest0-21swapper/119:35:251
394888099164,4cyclictest15624-21apt-get23:20:131
394888099163,3cyclictest10409-21/usr/sbin/munin23:15:141
3948880991614,1cyclictest4073652-21mii-tool21:10:171
3948880991614,1cyclictest0-21swapper/119:30:001
3948880991612,3cyclictest4067785-21kworker/u8:1+flush-259:021:20:141
3948878991614,1cyclictest60350irq/132-enp0s29f1:rx-021:05:220
3948878991613,2cyclictest61150irq/133-enp0s29f1:tx-022:55:160
3948878991613,2cyclictest60350irq/132-enp0s29f1:rx-022:30:190
3948878991613,2cyclictest4026419-21apt-get20:25:150
3948878991612,3cyclictest60350irq/132-enp0s29f1:rx-019:15:120
3948878991612,3cyclictest4188069-21kworker/u8:3+events_unbound23:15:200
3948878991612,3cyclictest4052173-21kworker/u8:3+flush-259:021:00:280
3948878991611,4cyclictest60350irq/132-enp0s29f1:rx-020:35:210
3948888991513,1cyclictest0-21swapper/322:40:013
3948888991512,2cyclictest3989705-21kworker/u8:2+flush-259:020:00:163
3948888991512,2cyclictest3964033-21kworker/u8:3+flush-259:019:45:133
3948888991511,3cyclictest4031362-21kworker/u8:1+flush-259:020:55:013
3948883991512,2cyclictest4005614-21kworker/u8:0+flush-259:020:30:122
3948883991511,3cyclictest62505-21kworker/u8:4+flush-259:000:05:172
3948883991511,3cyclictest62505-21kworker/u8:4+flush-259:000:05:172
3948883991511,3cyclictest4151497-21kworker/u8:0+flush-259:023:15:012
3948883991511,3cyclictest3964033-21kworker/u8:3+flush-259:019:30:202
3948883991511,3cyclictest3964033-21kworker/u8:3+flush-259:019:30:192
3948883991510,4cyclictest252-21systemd-journal00:39:392
3948880991513,1cyclictest0-21swapper/120:15:001
3948880991512,2cyclictest0-21swapper/122:05:001
3948878991513,1cyclictest60350irq/132-enp0s29f1:rx-021:20:270
3948878991513,1cyclictest60350irq/132-enp0s29f1:rx-020:55:250
3948878991513,1cyclictest0-21swapper/022:00:000
3948878991512,2cyclictest60350irq/132-enp0s29f1:rx-023:50:140
3948878991512,2cyclictest60350irq/132-enp0s29f1:rx-021:10:130
3948878991510,3cyclictest4151497-21kworker/u8:0+flush-259:023:00:170
3948888991412,1cyclictest4151958-21irqcore22:25:173
3948888991412,1cyclictest0-21swapper/323:45:003
3948888991412,1cyclictest0-21swapper/322:10:003
3948888991411,2cyclictest52197-21kworker/u8:0+flush-259:000:15:153
3948888991411,2cyclictest4598-21idleruntime-cro23:10:003
3948888991411,2cyclictest4072537-21unixbench_multi21:05:253
3948888991411,2cyclictest1-21systemd23:25:003
3948888991410,3cyclictest3964274-21latency_hist19:25:013
394888399149,4cyclictest3984922-21latency_hist19:45:012
3948883991412,1cyclictest3964964-21irqcore19:25:202
3948883991412,1cyclictest0-21swapper/200:20:002
3948883991411,2cyclictest4145617-21kworker/u8:1+flush-259:022:20:502
3948883991411,2cyclictest4092707-21needreboot21:25:202
394888099148,4cyclictest3985155-21/usr/sbin/munin19:45:131
3948880991411,2cyclictest4145652-21idleruntime-cro22:20:001
3948880991410,3cyclictest79496-21tr00:20:171
3948878991413,1cyclictest60350irq/132-enp0s29f1:rx-023:49:220
3948878991412,1cyclictest61150irq/133-enp0s29f1:tx-000:20:200
3948878991412,1cyclictest60350irq/132-enp0s29f1:rx-023:05:190
3948878991412,1cyclictest60350irq/132-enp0s29f1:rx-021:45:090
3948878991412,1cyclictest60350irq/132-enp0s29f1:rx-019:30:580
3948878991412,1cyclictest60350irq/132-enp0s29f1:rx-019:30:570
3948878991412,1cyclictest60350irq/132-enp0s29f1:rx-000:07:310
3948878991412,1cyclictest60350irq/132-enp0s29f1:rx-000:07:300
3948878991411,2cyclictest60350irq/132-enp0s29f1:rx-020:45:110
3948878991411,2cyclictest60350irq/132-enp0s29f1:rx-020:14:230
3948878991411,2cyclictest60350irq/132-enp0s29f1:rx-020:05:220
3948878991411,2cyclictest4151497-21kworker/u8:0+events_unbound22:25:150
394888899139,3cyclictest4172332-21apt-get22:45:003
394888899139,3cyclictest4161446-21kworker/u8:1+flush-259:022:50:263
394888899138,4cyclictest4161177-21sshd22:30:393
394888899137,4cyclictest3964699-21diskstats19:25:163
3948888991311,1cyclictest3990754-21irqcore19:50:153
3948888991311,1cyclictest0-21swapper/322:25:013
3948888991310,2cyclictest42900-21kthreadcore23:45:173
3948888991310,2cyclictest4113797-21unixbench_multi21:45:233
3948888991310,2cyclictest3975126-21fschecks_time19:35:133
394888399139,3cyclictest4188069-21kworker/u8:3+flush-259:023:30:252
394888399137,4cyclictest78338-21cp00:20:022
394888099139,3cyclictest4062532-21kworker/u8:2+flush-259:021:05:161
394888099137,4cyclictest4005614-21kworker/u8:0+flush-259:020:20:191
3948880991310,2cyclictest68718-21kthreadcore00:10:171
3948880991310,2cyclictest68718-21kthreadcore00:10:171
3948880991310,2cyclictest4093224-21unixbench_singl21:25:251
3948880991310,2cyclictest3963381-21ntp_kernel_pll_19:20:221
3948880991310,1cyclictest0-21swapper/123:45:161
394887899139,3cyclictest3895120-21kworker/u8:0+flush-259:019:20:220
394887899137,4cyclictest3990065-21cron19:50:000
3948878991311,1cyclictest60350irq/132-enp0s29f1:rx-020:30:250
3948878991310,2cyclictest61150irq/133-enp0s29f1:tx-023:30:150
394888899129,2cyclictest94113-21cpuspeed_turbos00:35:153
394888899129,2cyclictest56167-21cut23:55:203
394888899129,2cyclictest4173432-21sed22:45:163
394888899129,2cyclictest0-21swapper/323:15:013
394888899128,3cyclictest4187112-21memory22:55:203
394888399129,2cyclictest4140485-21idleruntime-cro22:15:002
394888399129,2cyclictest0-21swapper/200:25:182
394888399121,9cyclictest93716-21awk00:35:002
394888399121,9cyclictest490-21dbus-daemon20:15:012
394888099129,2cyclictest83682-21cpuspeed_turbos00:25:121
394888099129,2cyclictest3970124-21irqcore19:30:171
394888099129,2cyclictest3970124-21irqcore19:30:161
394888099129,1cyclictest4052320-21cpuspeed_turbos20:50:121
394888099129,1cyclictest0-21swapper/122:05:251
394888099129,1cyclictest0-21swapper/121:05:001
394888099127,4cyclictest4036763-21apt-get20:35:121
3948880991210,1cyclictest0-21swapper/122:25:001
3948880991210,1cyclictest0-21swapper/119:55:131
394887899125,5cyclictest252-21systemd-journal20:45:010
394888899119,1cyclictest0-21swapper/321:29:593
394888899119,1cyclictest0-21swapper/321:10:203
394888899119,1cyclictest0-21swapper/320:10:003
394888899119,1cyclictest0-21swapper/300:00:113
394888899118,2cyclictest89269-21irqcore00:30:173
394888899117,3cyclictest4025194-21kworker/u8:2+flush-259:020:25:283
394888899117,3cyclictest3983790-21sed19:40:193
394888399119,1cyclictest0-21swapper/221:05:002
394888399118,2cyclictest56699-21unixbench_singl23:55:262
394888399118,2cyclictest4118621-21ntp_offset21:50:232
394888399118,2cyclictest4118621-21ntp_offset21:50:222
394888399118,2cyclictest4051513-21unixbench_singl20:45:252
394888399118,2cyclictest4037785-21tr20:35:172
394888399118,2cyclictest3979324-21unixbench_multi19:35:242
394888399118,2cyclictest0-21swapper/222:10:002
394888399117,3cyclictest3989705-21kworker/u8:2+flush-259:019:50:162
394888099118,2cyclictest4108111-21needreboot21:40:201
394888099117,3cyclictest4119470-21kworker/u8:3+flush-259:022:12:061
394888099116,3cyclictest252-21systemd-journal23:39:551
394888899108,1cyclictest0-21swapper/323:15:133
394888899108,1cyclictest0-21swapper/322:05:003
394888899108,1cyclictest0-21swapper/320:55:153
394888899107,2cyclictest4108158-21kworker/u8:0+flush-259:021:55:263
394888899107,2cyclictest4010972-21/usr/sbin/munin20:10:233
394888899107,2cyclictest3996527-21tr19:55:173
394888899106,3cyclictest4084003-21cat21:20:183
394888399108,1cyclictest10657-21fschecks_time23:15:142
394888399108,1cyclictest0-21swapper/223:05:242
394888399107,1cyclictest4046806-21latency_hist20:45:002
394888399106,3cyclictest4026486-21cpuspeed_turbos20:25:152
394888399100,8cyclictest3953975-21awk19:14:592
394888399100,2cyclictest3959132-21latency_hist19:20:012
394888399100,2cyclictest20502-21latency_hist23:25:002
394888099106,3cyclictest4035601-21ntp_kernel_pll_20:30:211
394888099100,8cyclictest77771-21cron00:20:001
39488889997,1cyclictest4104703-21irqcore21:40:173
39488889997,1cyclictest0-21swapper/320:25:013
39488889997,1cyclictest0-21swapper/320:15:183
39488889997,1cyclictest0-21swapper/300:05:143
39488889997,1cyclictest0-21swapper/300:05:143
39488889996,2cyclictest78884-21grep00:20:173
39488839997,1cyclictest4022588-21kthreadcore20:20:182
39488839997,1cyclictest0-21swapper/221:55:242
39488839997,1cyclictest0-21swapper/221:05:182
39488839996,2cyclictest4104347-21cpuspeed_turbos21:40:142
39488839995,3cyclictest4104190-21sed21:40:002
39488839995,3cyclictest4074875-21cut21:10:172
39488839994,4cyclictest46808-21latency_hist23:50:002
39488839994,4cyclictest4002056-21cut20:00:172
39488839991,6cyclictest72625-21turbostat.cron00:15:012
39488839991,6cyclictest72625-21turbostat.cron00:15:002
39488809995,3cyclictest4119711-21apt-get21:55:111
39488839984,3cyclictest401ktimers/223:40:122
39488839983,4cyclictest4168288-21sed22:40:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional