You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 09:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Mon Dec 01, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3860250994238,3cyclictest4020849-21kworker/u8:2+flush-259:022:30:002
3860250993834,3cyclictest3959272-21kworker/u8:1+events_unbound21:09:592
3860250993532,2cyclictest3896689-21kworker/u8:3+flush-259:020:00:002
3860250993532,2cyclictest3865053-21kworker/u8:1+flush-259:019:20:202
3860250993530,4cyclictest4020849-21kworker/u8:2+flush-259:022:40:002
3860250993430,3cyclictest4026642-21kworker/u8:3+flush-259:023:00:162
3860246993430,3cyclictest4020849-21kworker/u8:2+flush-259:022:30:181
3860246993326,5cyclictest4005354-21latency_hist21:30:001
3860250993228,3cyclictest3959272-21kworker/u8:1+flush-259:021:00:162
3860250993128,2cyclictest3968222-21kworker/u8:2+events_unbound21:15:002
3860250993128,2cyclictest3837257-21kworker/u8:0+flush-259:019:14:592
3860243993126,4cyclictest4020849-21kworker/u8:2+flush-259:022:20:000
3860253993026,3cyclictest4062570-21kworker/u8:0+flush-259:022:35:003
3860253993023,5cyclictest3969306-21latency_hist20:55:013
3860250993027,2cyclictest4129477-21kworker/u8:4+flush-259:023:55:152
3860246993026,3cyclictest3870213-21kworker/u8:3+flush-259:019:34:591
3860250992925,3cyclictest3896689-21kworker/u8:3+flush-259:020:14:592
3860250992924,3cyclictest4026642-21kworker/u8:3+events_unbound23:40:132
3860243992925,3cyclictest4192452-21kworker/u8:3+flush-259:000:35:290
3860243992925,3cyclictest3896769-21gpgv19:45:000
3860250992825,2cyclictest4026642-21kworker/u8:3+events_unbound22:15:192
3860250992824,3cyclictest3975050-21kworker/u8:0+events_unbound21:40:002
3860250992724,2cyclictest4020849-21kworker/u8:2+flush-259:022:20:222
3860250992623,2cyclictest3912824-21kworker/u8:2+flush-259:020:20:162
3860250992622,3cyclictest3837257-21kworker/u8:0+events_unbound19:45:002
3860246992622,2cyclictest3959272-21kworker/u8:1+flush-259:021:00:171
3860243992620,4cyclictest3865568-21/usr/sbin/munin19:15:250
3860253992522,2cyclictest4004828-21kworker/u8:4+flush-259:021:40:153
3860253992521,3cyclictest4166197-21kworker/u8:3+flush-259:000:15:003
3860253992521,3cyclictest3896689-21kworker/u8:3+flush-259:019:50:163
3860253992520,3cyclictest3837257-21kworker/u8:0+events_unbound19:45:163
3860253992519,4cyclictest2698871-21systemd-journal23:05:243
3860250992523,1cyclictest3912824-21kworker/u8:2+events_unbound20:35:252
3860250992522,2cyclictest4099101-21kworker/u8:0+events_unbound23:35:182
3860250992522,2cyclictest3968222-21kworker/u8:2+events_unbound21:25:172
3860250992518,2cyclictest3865053-21kworker/u8:1+events_unbound19:35:152
3860243992521,3cyclictest3870213-21kworker/u8:3+flush-259:019:30:000
3860243992521,3cyclictest3870213-21kworker/u8:3+flush-259:019:29:590
3860253992419,4cyclictest4099101-21kworker/u8:0+events_unbound23:05:003
3860250992421,2cyclictest4177186-21kworker/u8:2+flush-259:000:25:252
3860250992417,5cyclictest3886362-21latency_hist19:35:002
3860246992418,4cyclictest4165964-21latency_hist00:05:011
3860243992418,4cyclictest2698871-21systemd-journal20:20:010
3860253992318,4cyclictest2698871-21systemd-journal22:15:003
3860253992316,5cyclictest3880803-21latency_hist19:30:013
3860253992316,5cyclictest3880803-21latency_hist19:30:003
3860246992318,4cyclictest4129475-21kworker/u8:1+flush-259:000:10:131
3860243992316,5cyclictest4150125-21latency_hist23:50:010
386024399230,3cyclictest3928153-21awk20:15:000
3860253992218,3cyclictest3881311-21cat19:30:013
3860250992219,2cyclictest4099101-21kworker/u8:0+events_unbound23:30:222
3860250992219,2cyclictest3975050-21kworker/u8:0+events_unbound21:45:002
3860250992218,3cyclictest4026642-21kworker/u8:3+events_unbound23:11:252
3860246992218,3cyclictest4114001-21idleruntime-cro23:15:001
3860246992218,2cyclictest4026642-21kworker/u8:3+flush-259:022:25:181
3860243992220,1cyclictest3875640-21kworker/u8:4+events_unbound19:55:210
3860243992218,3cyclictest4177219-21apt-get00:20:110
3860243992216,4cyclictest3974676-21/usr/sbin/munin21:00:150
3860243992216,4cyclictest2698871-21systemd-journal22:50:000
3860243992216,4cyclictest2698871-21systemd-journal22:10:000
3860253992115,4cyclictest3865053-21kworker/u8:1+flush-259:019:45:003
3860250992118,2cyclictest3875640-21kworker/u8:4+flush-259:019:25:212
3860250992118,2cyclictest3875640-21kworker/u8:4+flush-259:019:25:212
3860250992117,3cyclictest4129477-21kworker/u8:4+flush-259:000:15:162
3860250992117,3cyclictest4129477-21kworker/u8:4+flush-259:000:15:152
3860250992117,3cyclictest4129475-21kworker/u8:1+events_unbound00:13:492
3860250992117,2cyclictest0-21swapper/223:25:002
3860246992115,4cyclictest4187515-21/usr/sbin/munin00:30:161
3860246992115,4cyclictest4118308-21ntpq23:15:231
3860243992115,4cyclictest3958178-21ntpq20:40:230
3860253992017,2cyclictest4036696-21cron22:00:003
3860253992017,2cyclictest4026642-21kworker/u8:3+flush-259:022:55:253
3860253992017,2cyclictest4020849-21kworker/u8:2+events_unbound23:20:243
3860253992017,2cyclictest3975050-21kworker/u8:0+events_unbound21:05:263
3860253992014,4cyclictest4156091-21diskstats23:55:143
3860250992016,3cyclictest3975050-21kworker/u8:0+events_unbound21:34:232
3860250992016,3cyclictest3875640-21kworker/u8:4+events_unbound20:55:172
3860246992018,1cyclictest0-21swapper/122:50:001
3860246992014,4cyclictest4015689-21latency_hist21:40:001
3860253991915,3cyclictest4026642-21kworker/u8:3+events_unbound22:50:253
3860253991915,3cyclictest3949040-21apt-get20:35:143
3860250991916,3cyclictest3912824-21kworker/u8:2+flush-259:020:05:162
3860250991916,2cyclictest4052227-21sh22:15:002
3860250991916,2cyclictest4026642-21kworker/u8:3+events_unbound21:50:212
3860250991916,2cyclictest4020849-21kworker/u8:2+flush-259:021:55:222
3860250991915,3cyclictest4088490-21apt-get22:50:142
3860250991913,4cyclictest3896650-21latency_hist19:45:012
3860250991911,6cyclictest2698871-21systemd-journal22:45:012
3860243991914,4cyclictest4031816-21/usr/sbin/munin21:55:180
3860243991914,3cyclictest492-21dbus-daemon23:05:000
3860253991816,1cyclictest0-21swapper/322:05:003
3860253991814,3cyclictest3875640-21kworker/u8:4+flush-259:019:40:003
3860250991815,2cyclictest4176698-21kworker/u8:0+events_unbound00:35:242
3860250991815,2cyclictest4099101-21kworker/u8:0+events_unbound23:50:002
3860250991814,3cyclictest4099101-21kworker/u8:0+events_unbound23:50:142
3860246991814,3cyclictest3912824-21kworker/u8:2+flush-259:020:05:271
3860246991813,4cyclictest3928410-21apt-get20:15:131
3860246991813,4cyclictest2698871-21systemd-journal23:23:001
3860253991714,2cyclictest3923638-21irqcore20:10:193
3860253991714,2cyclictest3912573-21cron20:00:003
3860253991711,4cyclictest2698871-21systemd-journal20:10:003
3860250991715,2cyclictest4160846-21kworker/u8:0+flush-259:000:00:212
3860250991715,2cyclictest4020849-21kworker/u8:2+flush-259:022:30:162
3860250991714,2cyclictest508-21tpm2-abrmd19:20:002
3860250991714,2cyclictest4026642-21kworker/u8:3+events_unbound23:05:202
3860250991714,2cyclictest3912824-21kworker/u8:2+events_unbound20:15:162
3860250991714,2cyclictest3875640-21kworker/u8:4+events_unbound20:45:222
3860246991714,2cyclictest0-21swapper/100:25:011
3860246991713,3cyclictest3937584-21ntp_kernel_pll_20:20:241
3860246991713,2cyclictest3912824-21kworker/u8:2+flush-259:020:25:181
3860243991714,2cyclictest60650irq/133-enp0s29f1:rx-000:30:220
3860243991713,3cyclictest4073072-21apt-get22:35:140
3860243991713,3cyclictest3975050-21kworker/u8:0+flush-259:021:45:170
3860243991713,3cyclictest3881487-21apt-get19:30:130
3860253991613,2cyclictest4027098-21sed21:50:173
3860253991612,3cyclictest4129477-21kworker/u8:4+events_unbound00:00:193
3860253991612,3cyclictest4057378-21idleruntime-cro22:20:003
3860253991611,3cyclictest4150366-21/usr/sbin/munin23:50:243
3860250991613,2cyclictest4177186-21kworker/u8:2+events_unbound00:20:142
3860250991612,3cyclictest4099101-21kworker/u8:0+flush-259:023:25:062
3860246991613,2cyclictest4129477-21kworker/u8:4+flush-259:023:55:191
3860246991612,3cyclictest4176698-21kworker/u8:0+flush-259:000:35:281
3860246991612,3cyclictest4026642-21kworker/u8:3+flush-259:023:43:281
3860246991611,4cyclictest3896689-21kworker/u8:3+flush-259:020:10:181
386024399169,5cyclictest4031581-21latency_hist21:55:000
3860243991614,1cyclictest61450irq/134-enp0s29f1:tx-020:50:140
3860243991614,1cyclictest60650irq/133-enp0s29f1:rx-023:55:240
3860243991614,1cyclictest60650irq/133-enp0s29f1:rx-021:40:160
3860243991613,2cyclictest60650irq/133-enp0s29f1:rx-023:30:210
3860243991613,2cyclictest60650irq/133-enp0s29f1:rx-021:20:120
3860243991613,2cyclictest60650irq/133-enp0s29f1:rx-021:05:160
3860243991613,2cyclictest60650irq/133-enp0s29f1:rx-020:34:050
3860243991613,2cyclictest60650irq/133-enp0s29f1:rx-019:53:560
3860243991613,2cyclictest60650irq/133-enp0s29f1:rx-000:26:320
3860243991613,2cyclictest4176940-21cron00:20:000
3860243991613,2cyclictest4176940-21cron00:20:000
3860243991613,2cyclictest3959272-21kworker/u8:1+flush-259:021:10:170
3860243991613,2cyclictest3912824-21kworker/u8:2+flush-259:020:30:010
3860243991613,2cyclictest0-21swapper/023:20:000
3860253991512,2cyclictest4182831-21irqcore00:25:173
3860253991512,2cyclictest4058672-21kthreadcore22:20:163
3860253991511,3cyclictest4046609-21unixbench_multi22:05:263
3860253991511,3cyclictest3968222-21kworker/u8:2+flush-259:021:25:153
3860253991511,3cyclictest3968222-21kworker/u8:2+flush-259:021:15:003
3860253991511,3cyclictest3865316-21sh19:14:593
386025099158,2cyclictest4026642-21kworker/u8:3+events_unbound22:05:262
3860250991512,2cyclictest3938469-21kworker/u8:0+events_unbound20:30:152
3860250991512,2cyclictest3912824-21kworker/u8:2+events_unbound20:00:292
3860250991511,3cyclictest4177186-21kworker/u8:2+events_unbound00:30:012
386024699159,4cyclictest4155434-21cp23:55:001
3860246991513,1cyclictest0-21swapper/121:50:001
3860243991513,1cyclictest60650irq/133-enp0s29f1:rx-023:50:150
3860243991513,1cyclictest60650irq/133-enp0s29f1:rx-022:25:170
3860243991513,1cyclictest60650irq/133-enp0s29f1:rx-022:20:250
3860243991513,1cyclictest60650irq/133-enp0s29f1:rx-022:13:490
3860243991513,1cyclictest60650irq/133-enp0s29f1:rx-021:30:150
3860243991513,1cyclictest60650irq/133-enp0s29f1:rx-020:05:110
3860243991512,2cyclictest60650irq/133-enp0s29f1:rx-022:50:180
3860243991512,2cyclictest60650irq/133-enp0s29f1:rx-021:35:210
3860243991512,2cyclictest60650irq/133-enp0s29f1:rx-021:25:520
3860243991512,2cyclictest60650irq/133-enp0s29f1:rx-019:35:390
3860243991510,4cyclictest4166866-21apt-get00:10:090
386025399148,4cyclictest4129475-21kworker/u8:1+events_unbound23:45:133
3860253991411,2cyclictest4129477-21kworker/u8:4+events_unbound23:35:253
3860253991410,3cyclictest4020849-21kworker/u8:2+flush-259:022:35:163
3860253991410,3cyclictest3968222-21kworker/u8:2+flush-259:021:00:003
3860250991411,3cyclictest3968222-21kworker/u8:2+events_unbound21:21:222
3860250991410,3cyclictest3938469-21kworker/u8:0+events_unbound20:25:142
386024699149,3cyclictest2698871-21systemd-journal23:29:541
3860246991411,2cyclictest4035849-21ntp_kernel_pll_21:55:251
3860246991411,2cyclictest3995752-21sed21:20:161
3860246991411,2cyclictest3959272-21kworker/u8:1+flush-259:020:50:181
3860246991411,2cyclictest3915161-21kthreadcore20:00:191
3860246991411,2cyclictest3878189-21kthreadcore19:25:171
3860246991411,2cyclictest3878189-21kthreadcore19:25:171
3860243991413,1cyclictest60650irq/133-enp0s29f1:rx-020:58:540
3860243991412,2cyclictest60650irq/133-enp0s29f1:rx-020:02:040
3860243991412,1cyclictest60650irq/133-enp0s29f1:rx-023:25:240
3860243991412,1cyclictest60650irq/133-enp0s29f1:rx-023:20:160
3860243991412,1cyclictest60650irq/133-enp0s29f1:rx-023:10:180
3860243991412,1cyclictest60650irq/133-enp0s29f1:rx-022:40:260
3860243991412,1cyclictest60650irq/133-enp0s29f1:rx-019:48:520
3860243991411,2cyclictest60650irq/133-enp0s29f1:rx-022:57:440
3860243991411,2cyclictest60650irq/133-enp0s29f1:rx-022:30:220
3860243991411,2cyclictest60650irq/133-enp0s29f1:rx-000:05:310
3860243991411,2cyclictest60650irq/133-enp0s29f1:rx-000:04:020
386025399139,3cyclictest4078211-21apt-get22:40:133
386025399138,4cyclictest4114014-21latency_hist23:15:023
386025399136,5cyclictest492-21dbus-daemon21:50:003
3860253991311,1cyclictest0-21swapper/321:25:003
3860253991310,2cyclictest4144949-21munin-plugin-st23:44:593
3860253991310,2cyclictest3994640-21unixbench_singl21:15:263
386025099139,3cyclictest3959272-21kworker/u8:1+flush-259:021:16:432
3860250991310,2cyclictest3000063-21rs:main2
3860250991310,1cyclictest0-21swapper/222:00:232
3860246991310,2cyclictest4176957-21sh00:20:001
3860246991310,2cyclictest4176957-21sh00:20:001
3860243991311,2cyclictest60650irq/133-enp0s29f1:rx-019:14:030
3860243991311,1cyclictest60650irq/133-enp0s29f1:rx-023:44:500
386025399129,2cyclictest3974767-21cpuspeed_turbos21:00:133
386025399128,3cyclictest4026642-21kworker/u8:3+flush-259:023:30:243
386025399127,4cyclictest3896689-21kworker/u8:3+flush-259:020:00:163
386025099129,2cyclictest4025128-21ntp_kernel_pll_21:45:222
386025099129,2cyclictest0-21swapper/222:55:262
386024699129,2cyclictest503-21systemd-logind23:10:011
386024699129,2cyclictest4145378-21fschecks_count23:45:151
386024699129,2cyclictest4056960-21unixbench_singl22:15:271
386024699129,2cyclictest0-21swapper/122:50:271
386024699128,3cyclictest4129477-21kworker/u8:4+flush-259:023:35:261
386024699128,3cyclictest3896689-21kworker/u8:3+flush-259:019:50:281
386024699128,2cyclictest0-21swapper/121:20:001
386024699127,4cyclictest4182420-21apt-get00:25:111
386024699126,5cyclictest4078437-21uniq22:40:151
386024699125,5cyclictest3953879-21latency_hist20:40:001
386024399129,2cyclictest60650irq/133-enp0s29f1:rx-022:00:180
386024399129,2cyclictest4104376-21irqcore23:05:180
386024399129,1cyclictest0-21swapper/019:20:200
386024399128,2cyclictest3938442-21awk20:24:590
386025399118,2cyclictest4087772-21unixbench_multi22:45:263
386025399118,2cyclictest3875640-21kworker/u8:4+flush-259:020:50:013
386025399118,2cyclictest0-21swapper/320:35:003
386025399117,3cyclictest4099101-21kworker/u8:0+events_unbound23:15:153
386025099117,3cyclictest4166197-21kworker/u8:3+events_unbound00:05:192
386024699119,1cyclictest3870488-21idleruntime-cro19:20:001
386024699118,2cyclictest4020395-21unixbench_singl21:40:251
386024699118,2cyclictest3948336-21unixbench_singl20:30:261
386024699118,1cyclictest0-21swapper/119:20:191
386024699116,4cyclictest4025996-21apt-get21:50:011
386024399119,1cyclictest60650irq/133-enp0s29f1:rx-021:15:220
386024399119,1cyclictest60650irq/133-enp0s29f1:rx-020:35:160
386024399117,2cyclictest3964193-21turbostat20:50:010
386025399107,2cyclictest0-21swapper/322:30:013
386025399106,3cyclictest3959272-21kworker/u8:1+events_unbound21:35:163
386024699106,2cyclictest0-21swapper/122:40:011
38602539997,1cyclictest0-21swapper/300:15:153
38602539997,1cyclictest0-21swapper/300:15:153
38602539984,2cyclictest835-21runrttasks00:06:583
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional