You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-28 - 07:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot4s.osadl.org (updated Tue Oct 28, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4105286994037,2cyclictest103564-21kworker/u8:2+flush-259:022:45:002
4105279993936,2cyclictest103564-21kworker/u8:2+events_unbound22:25:200
4105292993830,5cyclictest77398-21latency_hist21:50:003
4105283993832,4cyclictest202493-21latency_hist23:50:011
4105292993730,5cyclictest4167874-21latency_hist20:10:013
4105292993429,4cyclictest108928-21kworker/u8:1+flush-259:022:35:173
4105292993429,3cyclictest4162755-21kworker/u8:1+flush-259:020:10:173
4105279993430,3cyclictest51259-21kworker/u8:0+flush-259:021:35:000
4105292993329,3cyclictest218026-21kworker/u8:0+flush-259:000:30:193
4105279993329,3cyclictest41141-21kworker/u8:4+events_unbound21:25:190
4105279993327,4cyclictest154604-21ntpq23:00:230
4105279993326,5cyclictest2698871-21systemd-journal19:25:010
4105279993229,2cyclictest218026-21kworker/u8:0+events_unbound00:35:270
4105279993229,2cyclictest140073-21kworker/u8:0+events_unbound22:55:210
4105292993124,5cyclictest4146772-21kworker/u8:3+flush-259:019:55:183
4105279993128,2cyclictest103564-21kworker/u8:2+events_unbound22:20:290
4105279993127,3cyclictest218026-21kworker/u8:0+events_unbound00:05:230
4105292993024,4cyclictest108928-21kworker/u8:1+flush-259:022:20:183
4105286993025,3cyclictest108928-21kworker/u8:1+events_unbound23:40:152
4105279993027,2cyclictest4842-21kworker/u8:3+events_unbound21:15:260
4105286992926,2cyclictest4146772-21kworker/u8:3+flush-259:019:50:262
4105286992926,2cyclictest41141-21kworker/u8:4+events_unbound21:30:272
4105279992926,2cyclictest77421-21kworker/u8:0+events_unbound22:05:260
4105279992926,2cyclictest4100159-21kworker/u8:3+events_unbound19:15:160
4105279992925,3cyclictest36035-21apt-get21:10:130
4105292992826,1cyclictest4188453-21kworker/3:2+pm22:06:563
4105292992822,4cyclictest4121110-21apt-get19:25:003
4105279992826,2cyclictest155554-21kworker/u8:4+events_unbound23:45:280
4105279992825,2cyclictest140073-21kworker/u8:0+events_unbound22:50:290
4105279992825,2cyclictest108928-21kworker/u8:1+events_unbound23:40:210
4105292992724,2cyclictest5051-21kworker/u8:4+flush-259:021:00:003
4105292992722,3cyclictest4121778-21/usr/sbin/munin19:25:183
4105292992720,5cyclictest218006-21latency_hist00:05:003
4105292992719,6cyclictest4187547-21ntpq20:25:233
4105286992724,2cyclictest4077873-21kworker/u8:0+events_unbound19:15:232
4105286992720,5cyclictest191318-21ntpq23:35:262
4105279992724,2cyclictest212452-21kworker/u8:2+events_unbound00:13:300
4105283992622,3cyclictest4110477-21gpgv19:15:011
4105279992623,2cyclictest218026-21kworker/u8:0+events_unbound00:30:180
4105279992623,2cyclictest155554-21kworker/u8:4+events_unbound00:15:280
4105279992623,2cyclictest103564-21kworker/u8:2+events_unbound22:45:170
4105279992620,5cyclictest4109499-21ntpq19:10:220
4105292992519,4cyclictest4151579-21ntpq19:50:233
4105286992522,2cyclictest4120066-21kworker/u8:4+events_unbound19:35:002
4105279992523,2cyclictest4146772-21kworker/u8:3+events_unbound19:53:310
4105279992522,2cyclictest77421-21kworker/u8:0+events_unbound21:55:290
4105279992522,2cyclictest4162755-21kworker/u8:1+events_unbound20:32:320
4105279992522,2cyclictest155554-21kworker/u8:4+events_unbound23:50:210
4105279992520,4cyclictest93264-21latency_hist22:05:000
4105279992519,4cyclictest4146402-21ntpq19:45:220
4105292992417,5cyclictest243720-21latency_hist00:30:003
4105286992419,3cyclictest2698871-21systemd-journal22:36:302
4105283992419,4cyclictest40912-21latency_hist21:15:011
4105279992421,2cyclictest4162755-21kworker/u8:1+events_unbound20:14:550
4105279992421,2cyclictest4120066-21kworker/u8:4+events_unbound20:15:300
4105279992421,2cyclictest41141-21kworker/u8:4+events_unbound22:18:200
4105279992421,2cyclictest212452-21kworker/u8:2+events_unbound00:00:290
4105292992318,3cyclictest2698871-21systemd-journal21:03:013
4105286992320,2cyclictest218026-21kworker/u8:0+events_unbound00:35:172
4105286992319,3cyclictest41141-21kworker/u8:4+flush-259:022:10:202
4105286992319,3cyclictest103564-21kworker/u8:2+flush-259:022:25:262
4105286992317,4cyclictest2698871-21systemd-journal23:30:002
4105283992320,2cyclictest3000063-21rs:main1
4105279992321,1cyclictest5051-21kworker/u8:4+events_unbound20:50:130
4105279992320,2cyclictest5051-21kworker/u8:4+events_unbound20:49:560
4105279992320,2cyclictest4162755-21kworker/u8:1+events_unbound20:20:300
4105279992320,2cyclictest4120066-21kworker/u8:4+events_unbound19:30:260
4105279992320,2cyclictest212452-21kworker/u8:2+events_unbound23:58:410
4105279992320,2cyclictest108928-21kworker/u8:1+events_unbound23:15:170
4105279992320,2cyclictest108928-21kworker/u8:1+events_unbound23:10:310
4105279992319,3cyclictest4842-21kworker/u8:3+events_unbound21:23:040
4105292992218,3cyclictest4173026-21kworker/u8:2+flush-259:020:19:593
4105292992216,4cyclictest2698871-21systemd-journal20:34:593
4105286992218,3cyclictest4100159-21kworker/u8:3+flush-259:019:10:162
4105286992218,3cyclictest218026-21kworker/u8:0+events_unbound00:30:172
4105283992218,3cyclictest4152723-21apt-get19:55:131
4105279992220,2cyclictest4077873-21kworker/u8:0+events_unbound19:44:550
4105279992219,2cyclictest4162755-21kworker/u8:1+events_unbound20:27:310
4105279992216,5cyclictest242853-21ntpq00:25:260
4105292992116,4cyclictest4188446-21kworker/u8:2+flush-259:020:35:193
4105292992116,4cyclictest140264-21kworker/u8:3+flush-259:023:05:253
4105292992115,4cyclictest67360-21/usr/sbin/munin21:40:153
4105286992118,2cyclictest4178381-21kworker/u8:3+flush-259:020:20:312
4105286992118,2cyclictest4178381-21kworker/u8:3+events_unbound20:25:162
4105286992117,3cyclictest4152481-21kworker/u8:0+flush-259:021:05:192
4105286992117,3cyclictest4120066-21kworker/u8:4+events_unbound19:55:222
4105286992117,3cyclictest140073-21kworker/u8:0+flush-259:022:50:252
4105283992118,2cyclictest0-21swapper/123:25:011
4105283992116,3cyclictest61989-21latency_hist21:35:011
4105279992119,2cyclictest103564-21kworker/u8:2+events_unbound22:30:250
4105279992118,2cyclictest140073-21kworker/u8:0+events_unbound23:35:140
4105279992118,2cyclictest140073-21kworker/u8:0+events_unbound23:27:100
4105292992016,3cyclictest4120066-21kworker/u8:4+flush-259:019:50:003
4105292992016,3cyclictest103564-21kworker/u8:2+flush-259:022:30:163
4105292992015,4cyclictest135307-21diskstats22:45:183
4105292992014,4cyclictest4110387-21gpgv19:15:013
4105292992014,4cyclictest2698871-21systemd-journal23:45:013
4105292992012,6cyclictest2698871-21systemd-journal21:30:003
4105286992017,2cyclictest140073-21kworker/u8:0+flush-259:023:00:222
4105286992016,3cyclictest77421-21kworker/u8:0+flush-259:022:00:262
4105286992016,3cyclictest4152481-21kworker/u8:0+flush-259:021:05:002
4105286992016,3cyclictest155554-21kworker/u8:4+events_unbound00:25:022
4105286992016,3cyclictest140264-21kworker/u8:3+flush-259:023:13:342
4105286992015,3cyclictest212452-21kworker/u8:2+flush-259:000:00:172
410528399203,5cyclictest50347-21ntpq21:20:211
410528399203,10cyclictest170702-21ntpq23:15:251
410528399202,6cyclictest88180-21apt-config22:00:011
4105279992017,2cyclictest4842-21kworker/u8:3+events_unbound20:58:020
4105279992017,2cyclictest4188446-21kworker/u8:2+events_unbound20:35:170
4105279992017,2cyclictest4152481-21kworker/u8:0+events_unbound20:05:170
4105279992017,2cyclictest4152481-21kworker/u8:0+events_unbound20:00:260
4105279992017,2cyclictest41141-21kworker/u8:4+events_unbound21:40:280
4105279992017,2cyclictest41141-21kworker/u8:4+events_unbound21:38:150
410529299198,4cyclictest2698871-21systemd-journal20:50:003
4105292991915,3cyclictest4116163-21kworker/u8:2+flush-259:019:39:593
4105286991916,3cyclictest41141-21kworker/u8:4+events_unbound21:40:242
4105286991916,2cyclictest4188446-21kworker/u8:2+flush-259:020:30:242
4105286991916,2cyclictest4120066-21kworker/u8:4+events_unbound20:15:252
4105286991916,2cyclictest192187-21kworker/u8:2+events_unbound23:45:172
4105286991915,3cyclictest77421-21kworker/u8:0+events_unbound22:05:142
4105286991915,3cyclictest103564-21kworker/u8:2+events_unbound22:45:272
410528399194,5cyclictest86839-21collect221:55:241
410528399192,6cyclictest45243-21ntpq21:15:221
4105283991916,2cyclictest4126680-21cron19:30:001
4105283991914,3cyclictest2698871-21systemd-journal23:10:001
4105279991916,2cyclictest4152481-21kworker/u8:0+events_unbound20:41:320
4105279991916,2cyclictest4120066-21kworker/u8:4+events_unbound19:57:440
4105279991916,2cyclictest4120066-21kworker/u8:4+events_unbound19:36:310
4105279991916,2cyclictest140073-21kworker/u8:0+events_unbound23:22:250
4105279991916,2cyclictest129932-21kworker/u8:4+events_unbound22:41:050
4105279991913,4cyclictest34840-21ntpq21:05:210
4105292991814,3cyclictest140073-21kworker/u8:0+flush-259:023:35:283
4105292991814,2cyclictest57514-21irqcore21:30:193
4105292991813,4cyclictest228507-21kworker/u8:3+flush-259:000:19:593
4105292991812,4cyclictest2698871-21systemd-journal21:54:563
4105292991812,4cyclictest2698871-21systemd-journal21:24:553
4105286991815,2cyclictest218026-21kworker/u8:0+flush-259:000:10:152
4105286991815,2cyclictest155554-21kworker/u8:4+events_unbound23:23:292
4105286991815,2cyclictest108928-21kworker/u8:1+events_unbound23:30:282
4105286991815,2cyclictest108928-21kworker/u8:1+events_unbound23:15:252
4105286991814,3cyclictest4842-21kworker/u8:3+events_unbound20:58:002
4105286991814,3cyclictest4120066-21kworker/u8:4+events_unbound20:09:312
4105286991814,3cyclictest218026-21kworker/u8:0+events_unbound00:20:142
410528399184,8cyclictest217113-21ntpq00:00:231
410528399182,7cyclictest150644-21runrttasks23:00:151
410528399182,5cyclictest66205-21ntpq21:35:211
4105283991816,1cyclictest192170-21turbostat.cron23:40:011
410527999187,2cyclictest0-21swapper/022:35:180
4105279991815,2cyclictest4152481-21kworker/u8:0+events_unbound21:04:310
4105279991815,2cyclictest41141-21kworker/u8:4+events_unbound22:12:180
4105279991815,2cyclictest155554-21kworker/u8:4+events_unbound23:31:420
4105279991815,2cyclictest155554-21kworker/u8:4+events_unbound00:20:010
4105279991815,2cyclictest140264-21kworker/u8:3+events_unbound23:07:040
4105292991713,2cyclictest236010-21kthreadcore00:20:203
410529299171,14cyclictest46118-21true21:20:003
4105286991714,2cyclictest4842-21kworker/u8:3+flush-259:021:15:162
4105286991714,2cyclictest155554-21kworker/u8:4+events_unbound23:50:282
4105286991713,3cyclictest4152481-21kworker/u8:0+events_unbound20:40:312
4105286991713,3cyclictest4120066-21kworker/u8:4+events_unbound20:12:322
4105286991713,3cyclictest4116163-21kworker/u8:2+flush-259:019:40:272
4105286991713,3cyclictest140073-21kworker/u8:0+flush-259:022:55:362
4105286991713,3cyclictest140073-21kworker/u8:0+events_unbound23:05:042
4105286991713,3cyclictest108928-21kworker/u8:1+flush-259:022:25:002
4105286991713,2cyclictest0-21swapper/223:59:592
4105286991712,3cyclictest124122-21latency_hist22:35:012
4105286991711,4cyclictest3000063-21rs:main2
410528699171,14cyclictest2698871-21systemd-journal20:46:302
410528399174,3cyclictest4167004-21ntpq20:05:231
410528399173,5cyclictest133119-21tr22:40:201
410528399173,4cyclictest233452-21/usr/sbin/munin00:20:001
410528399173,3cyclictest186150-21ntpq23:30:221
410528399172,5cyclictest144492-21/usr/sbin/munin22:50:251
410528399172,4cyclictest4140615-21grep19:40:191
4105283991712,4cyclictest4162755-21kworker/u8:1+flush-259:020:35:001
410529299169,5cyclictest4130925-21ntpq19:30:223
4105292991612,3cyclictest4110350-21apt-get19:15:033
4105292991612,3cyclictest155554-21kworker/u8:4+events_unbound23:20:213
4105292991610,4cyclictest82963-21diskstats21:55:163
410528699169,5cyclictest4121277-21base6419:25:002
4105286991613,2cyclictest4188446-21kworker/u8:2+flush-259:020:35:272
4105286991612,3cyclictest87707-21gpgv22:00:002
4105286991612,3cyclictest4152481-21kworker/u8:0+events_unbound20:00:172
4105286991612,3cyclictest4146772-21kworker/u8:3+flush-259:019:45:252
4105286991612,3cyclictest4120066-21kworker/u8:4+events_unbound19:37:442
4105286991612,3cyclictest41141-21kworker/u8:4+events_unbound21:45:222
4105286991612,3cyclictest41141-21kworker/u8:4+events_unbound21:36:312
4105286991612,3cyclictest155554-21kworker/u8:4+events_unbound00:18:322
410528399165,2cyclictest139127-21ntpq22:45:241
410528399164,1cyclictest4161826-21ntpq20:00:231
410528399162,4cyclictest34878-21ntpq21:05:211
410528399161,4cyclictest227375-21ld00:10:211
410527999167,1cyclictest0-21swapper/021:45:190
4105292991512,2cyclictest155528-21sh23:05:003
4105292991511,3cyclictest228277-21basename00:14:593
4105286991513,2cyclictest41141-21kworker/u8:4+events_unbound21:25:252
4105286991511,3cyclictest77421-21kworker/u8:0+events_unbound22:17:322
4105286991511,3cyclictest41141-21kworker/u8:4+events_unbound21:23:322
410528399159,4cyclictest72500-21/usr/sbin/munin21:45:161
410528399155,7cyclictest181037-21ntpq23:25:221
410528399153,4cyclictest196508-21ntpq23:40:221
410528399153,3cyclictest81612-21ntpq21:50:221
410528399151,5cyclictest113856-21munin-run22:25:001
4105283991511,3cyclictest3000063-21rs:main1
4105279991514,1cyclictest60650irq/133-enp0s29f1:rx-021:50:180
410529299148,4cyclictest41141-21kworker/u8:4+flush-259:022:00:053
4105292991411,2cyclictest212824-21cron23:59:593
4105292991410,3cyclictest108928-21kworker/u8:1+flush-259:023:15:283
4105286991411,2cyclictest77421-21kworker/u8:0+flush-259:021:50:262
4105286991411,2cyclictest218026-21kworker/u8:0+events_unbound00:05:222
410528399148,4cyclictest212826-21latency_hist00:00:001
410528399146,6cyclictest2698871-21systemd-journal20:40:001
410528399144,2cyclictest4151590-21ntpq19:50:231
410528399143,3cyclictest102694-21ntpq22:10:241
410528399142,2cyclictest9116-21ntpq20:40:241
410528399141,4cyclictest24650-21gcc20:55:211
4105283991411,2cyclictest503-21systemd-logind21:45:001
4105283991410,2cyclictest77421-21kworker/u8:0+flush-259:022:15:181
410527999146,1cyclictest191rcu_preempt19:25:200
410529299138,4cyclictest218270-21apt-get00:05:113
4105292991310,2cyclictest198036-21irqcore23:45:183
4105286991310,2cyclictest15652-21fschecks_count20:50:152
410528399138,4cyclictest4120066-21kworker/u8:4+flush-259:019:20:271
410528399134,2cyclictest206781-21ntpq23:50:221
410528399133,2cyclictest93810-21fw_conntrack22:05:161
410528399133,2cyclictest4182425-21ntpq20:20:231
410528399133,1cyclictest0-21swapper/119:30:141
410528399131,2cyclictest4168551-21ps20:10:171
410529299129,2cyclictest108285-21unixbench_singl22:15:263
410529299128,3cyclictest4152481-21kworker/u8:0+flush-259:021:05:173
410529299128,3cyclictest103564-21kworker/u8:2+flush-259:022:30:003
410529299125,5cyclictest2698871-21systemd-journal20:25:003
410529299121,9cyclictest187031-21awk23:35:003
410529299121,6cyclictest40906-21idleruntime-cro21:15:003
410528399129,2cyclictest246286-21kthreadcore00:30:201
410528399129,2cyclictest0-21swapper/122:55:171
410528399128,3cyclictest249370-21fschecks_count00:35:171
410528399126,5cyclictest124117-21awk22:35:001
410528399126,4cyclictest237639-21grep00:20:201
410528399125,5cyclictest239016-21cut00:25:161
410528399122,2cyclictest4132384-21fw_forwarded_lo19:35:161
410528399122,2cyclictest32-21ksoftirqd/119:15:031
410528399121,9cyclictest4147267-21awk19:49:591
410528399121,3cyclictest55568-21ntpq21:25:211
410528399121,3cyclictest166357-21latency_hist23:15:011
410528399121,3cyclictest115096-21irqstats22:25:201
410527999122,2cyclictest171ktimers/019:10:010
410529299118,2cyclictest0-21swapper/323:15:013
410529299113,5cyclictest150374-21latency_hist23:00:003
410529299110,3cyclictest20354-21latency_hist20:55:013
410528399118,2cyclictest15554-21cpuspeed_turbos20:50:141
410528399113,6cyclictest218968-21sed00:05:151
410528399111,2cyclictest837-21snmpd20:15:151
410529299106,3cyclictest10032-21latency_hist20:45:013
410529299106,2cyclictest180944-21ntpq23:25:223
410529299106,2cyclictest0-21swapper/323:55:003
410529299105,4cyclictest140300-21apt-get22:50:143
41052929996,2cyclictest62652-21irqcore21:35:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional