You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 04:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System (updated Wed Dec 03, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1883012993832,2cyclictest1971586-21kworker/u8:2+events_unbound21:00:173
1883012993734,2cyclictest2112058-21kworker/u8:1+events_unbound23:00:183
1883012993633,2cyclictest2142731-21kworker/u8:1+events_unbound00:35:183
1883012993431,2cyclictest2142731-21kworker/u8:1+events_unbound23:45:233
1883008993429,4cyclictest2210048-21/usr/sbin/munin00:25:162
1883002993429,3cyclictest1924133-21kworker/u8:1+flush-259:020:00:180
1883002993428,4cyclictest2158411-21latency_hist23:35:010
1883012993330,2cyclictest2017860-21kworker/u8:0+events_unbound21:25:213
1883002993329,3cyclictest2096603-21kworker/u8:0+flush-259:022:55:170
1883012993229,2cyclictest2075184-21kworker/u8:1+flush-259:022:30:153
1883012993229,2cyclictest1913805-21kworker/u8:0+events_unbound20:30:193
1883012993229,2cyclictest1913805-21kworker/u8:0+events_unbound20:30:183
1883012993129,2cyclictest2012535-21kworker/u8:3+events_unbound22:25:153
1883012993128,2cyclictest2106385-21kworker/u8:2+flush-259:022:45:193
1883012993128,2cyclictest2028139-21kworker/u8:4+flush-259:021:30:163
1883012993128,2cyclictest1971586-21kworker/u8:2+flush-259:021:45:183
1883012993026,3cyclictest2142731-21kworker/u8:1+events_unbound00:20:143
1883008993027,2cyclictest1888336-21kworker/u8:3+flush-259:020:20:212
1883004993026,3cyclictest2012535-21kworker/u8:3+flush-259:021:25:011
1883004993024,4cyclictest2179175-21/usr/sbin/munin23:55:151
1883004993023,5cyclictest2049426-21diskstats21:50:161
1883002993024,4cyclictest1898421-21latency_hist19:25:010
1883012992926,2cyclictest2132892-21kworker/u8:2+flush-259:023:20:223
1883012992926,2cyclictest1971586-21kworker/u8:2+events_unbound20:50:173
1883008992925,3cyclictest1882978-21kworker/u8:2+events_unbound19:35:242
1883004992923,4cyclictest2132892-21kworker/u8:2+flush-259:023:20:161
1883004992919,9cyclictest1882978-21kworker/u8:2+flush-259:019:20:001
1883012992825,2cyclictest2012535-21kworker/u8:3+events_unbound22:20:133
1883012992825,2cyclictest1971586-21kworker/u8:2+flush-259:021:15:253
1883012992825,2cyclictest1888336-21kworker/u8:3+events_unbound20:10:213
1883002992822,4cyclictest2033067-21latency_hist21:35:010
1883012992724,2cyclictest2142731-21kworker/u8:1+flush-259:023:40:193
1883012992722,3cyclictest2698871-21systemd-journal19:20:183
1883012992721,4cyclictest2054315-21cat21:55:013
1883008992724,2cyclictest2121389-21kworker/u8:3+flush-259:023:10:002
1883008992723,3cyclictest2132892-21kworker/u8:2+flush-259:023:35:002
1883004992720,5cyclictest2698871-21systemd-journal19:55:001
1883004992719,6cyclictest2043423-21latency_hist21:45:011
1883002992723,3cyclictest1971622-21apt-get20:35:140
1883012992623,2cyclictest2132892-21kworker/u8:2+events_unbound23:30:243
1883012992623,2cyclictest2121389-21kworker/u8:3+flush-259:023:05:153
1883012992623,2cyclictest2096603-21kworker/u8:0+events_unbound22:50:153
1883012992623,2cyclictest2096603-21kworker/u8:0+events_unbound22:50:153
1883012992623,2cyclictest2012535-21kworker/u8:3+events_unbound22:00:233
1883008992623,2cyclictest1971586-21kworker/u8:2+events_unbound22:05:242
1883008992622,3cyclictest2059725-21kworker/u8:5+flush-259:022:50:002
1883008992621,4cyclictest2028139-21kworker/u8:4+flush-259:021:40:012
1883008992621,3cyclictest1888336-21kworker/u8:3+events_unbound19:25:202
1883002992622,3cyclictest2142731-21kworker/u8:1+flush-259:000:05:000
1883002992622,3cyclictest2017860-21kworker/u8:0+flush-259:021:25:230
1883002992622,3cyclictest1834313-21kworker/u8:5+flush-259:019:10:240
1883002992621,3cyclictest1882978-21kworker/u8:2+events_unbound19:30:150
1883002992620,4cyclictest1913784-21latency_hist19:40:020
1883012992522,2cyclictest2096603-21kworker/u8:0+events_unbound22:35:223
1883012992522,2cyclictest1913805-21kworker/u8:0+events_unbound21:05:203
1883012992522,2cyclictest1882978-21kworker/u8:2+events_unbound20:05:153
1883008992522,2cyclictest2012535-21kworker/u8:3+flush-259:021:20:262
1883008992522,2cyclictest1971586-21kworker/u8:2+events_unbound22:00:012
1883008992521,3cyclictest1882978-21kworker/u8:2+events_unbound20:05:002
1883002992519,5cyclictest2054801-21/usr/sbin/munin21:55:140
1883012992422,2cyclictest1834313-21kworker/u8:5+events_unbound19:10:143
1883012992421,2cyclictest2163538-21kworker/u8:0+events_unbound00:00:143
1883012992421,2cyclictest1882978-21kworker/u8:2+events_unbound19:15:233
1883008992421,2cyclictest2017860-21kworker/u8:0+events_unbound21:40:302
1883008992420,3cyclictest2163538-21kworker/u8:0+events_unbound00:30:122
1883004992420,3cyclictest2054827-21apt-get21:55:111
1883004992419,3cyclictest2132892-21kworker/u8:2+flush-259:023:10:171
1883004992418,4cyclictest1929493-21apt-get19:55:121
1883002992420,3cyclictest503-21systemd-logind23:25:000
1883002992420,3cyclictest2075184-21kworker/u8:1+flush-259:022:15:150
1883002992418,4cyclictest2698871-21systemd-journal20:35:000
1883002992418,4cyclictest2698871-21systemd-journal20:34:590
1883012992320,2cyclictest2194412-21kworker/u8:2+events_unbound00:10:233
1883012992320,2cyclictest2012535-21kworker/u8:3+events_unbound22:10:243
1883012992320,2cyclictest1913805-21kworker/u8:0+events_unbound20:45:203
1883008992320,2cyclictest1913805-21kworker/u8:0+events_unbound20:55:202
1883004992319,3cyclictest2163538-21kworker/u8:0+flush-259:000:35:011
1883004992318,4cyclictest2075184-21kworker/u8:1+flush-259:022:15:181
1883004992317,4cyclictest2158646-21/usr/sbin/munin23:35:161
1883004992317,4cyclictest2142731-21kworker/u8:1+flush-259:023:20:001
1883002992320,2cyclictest2096603-21kworker/u8:0+events_unbound23:15:010
1883012992219,2cyclictest1997074-21kworker/u8:1+flush-259:021:10:253
1883012992218,3cyclictest2163519-21latency_hist23:40:013
1883012992218,3cyclictest1888336-21kworker/u8:3+events_unbound19:25:143
1883012992216,4cyclictest1924094-21latency_hist19:50:003
1883008992219,2cyclictest2017860-21kworker/u8:0+events_unbound21:45:012
1883008992218,3cyclictest2075184-21kworker/u8:1+flush-259:022:35:202
1883008992218,3cyclictest1997074-21kworker/u8:1+events_unbound21:09:212
1883008992218,3cyclictest1913805-21kworker/u8:0+events_unbound20:14:182
1883008992217,2cyclictest2142731-21kworker/u8:1+events_unbound00:00:182
1883004992215,6cyclictest2142731-21kworker/u8:1+flush-259:023:45:001
1883012992119,2cyclictest2075184-21kworker/u8:1+events_unbound22:18:203
1883012992118,2cyclictest2132892-21kworker/u8:2+flush-259:023:10:153
1883012992118,2cyclictest2017860-21kworker/u8:0+flush-259:021:20:263
1883012992117,3cyclictest1971586-21kworker/u8:2+flush-259:020:44:593
1883012992116,4cyclictest1934393-21apt-get20:00:003
1883012992115,4cyclictest2075184-21kworker/u8:1+flush-259:022:40:183
1883008992118,2cyclictest2163538-21kworker/u8:0+flush-259:000:15:162
1883008992118,2cyclictest2121389-21kworker/u8:3+events_unbound23:20:152
1883008992118,2cyclictest1971586-21kworker/u8:2+flush-259:021:15:202
1883008992118,2cyclictest1913805-21kworker/u8:0+flush-259:020:25:202
1883008992118,2cyclictest1888336-21kworker/u8:3+flush-259:020:15:232
1883008992117,3cyclictest3000061-21in:imuxsock20:00:002
1883008992117,3cyclictest2012535-21kworker/u8:3+events_unbound22:30:112
1883008992117,3cyclictest1882978-21kworker/u8:2+events_unbound20:06:192
1883004992115,4cyclictest2002236-21latency_hist21:05:001
1883004992112,6cyclictest1966222-21latency_hist20:30:001
1883002992117,2cyclictest0-21swapper/022:15:000
1883002992115,4cyclictest2163760-21/usr/sbin/munin23:40:140
1883002992115,4cyclictest2117178-21latency_hist22:55:010
1883002992115,4cyclictest2117178-21latency_hist22:55:010
1883012992017,2cyclictest1888336-21kworker/u8:3+events_unbound20:15:243
1883008992018,1cyclictest0-21swapper/220:54:592
1883008992017,2cyclictest1913805-21kworker/u8:0+flush-259:019:45:202
1883008992017,2cyclictest1888336-21kworker/u8:3+events_unbound19:21:192
1883008992017,2cyclictest1834313-21kworker/u8:5+flush-259:019:10:262
1883008992016,2cyclictest2085860-21latency_hist22:25:002
1883004992015,4cyclictest2112305-21apt-get22:50:121
1883004992015,4cyclictest2112305-21apt-get22:50:121
1883004992012,6cyclictest2698871-21systemd-journal21:00:001
1883002992016,3cyclictest2189507-21apt-get00:05:130
1883002992016,3cyclictest2142731-21kworker/u8:1+events_unbound23:35:240
1883002992016,3cyclictest2075184-21kworker/u8:1+events_unbound22:35:180
1883002992016,3cyclictest1913805-21kworker/u8:0+flush-259:020:55:240
1883012991916,2cyclictest2012535-21kworker/u8:3+events_unbound22:06:103
1883012991913,4cyclictest1908679-21latency_hist19:35:023
1883008991916,2cyclictest2142731-21kworker/u8:1+flush-259:000:20:242
1883008991916,2cyclictest2142731-21kworker/u8:1+events_unbound00:10:192
1883008991916,2cyclictest2132892-21kworker/u8:2+flush-259:023:15:242
1883008991916,2cyclictest2121389-21kworker/u8:3+events_unbound23:00:272
1883008991916,2cyclictest2069601-21kworker/u8:0+events_unbound22:10:112
1883008991916,2cyclictest2028139-21kworker/u8:4+events_unbound21:30:202
1883008991916,2cyclictest1913805-21kworker/u8:0+events_unbound20:30:252
1883008991916,2cyclictest1913805-21kworker/u8:0+events_unbound20:30:252
1883008991915,3cyclictest1882978-21kworker/u8:2+flush-259:019:50:152
1883004991915,3cyclictest1913805-21kworker/u8:0+flush-259:020:35:181
1883004991912,5cyclictest2698871-21systemd-journal22:00:201
1883002991915,3cyclictest2022756-21cron21:25:000
1883012991815,2cyclictest2163538-21kworker/u8:0+events_unbound00:34:443
1883012991815,2cyclictest2142731-21kworker/u8:1+events_unbound23:56:303
1883012991815,2cyclictest1913805-21kworker/u8:0+flush-259:020:55:243
1883008991815,2cyclictest2142731-21kworker/u8:1+events_unbound23:50:252
1883008991815,2cyclictest2132892-21kworker/u8:2+flush-259:023:10:242
1883008991814,3cyclictest2121389-21kworker/u8:3+flush-259:023:25:052
1883008991814,3cyclictest2075184-21kworker/u8:1+flush-259:022:15:262
1883004991812,4cyclictest2044103-21apt-get21:45:121
1883004991812,4cyclictest1834313-21kworker/u8:5+flush-259:019:30:231
1883004991811,5cyclictest2698871-21systemd-journal20:10:171
1883004991811,5cyclictest1981655-21latency_hist20:45:011
1883002991814,3cyclictest60650irq/133-enp0s29f1:rx-022:40:220
1883002991814,3cyclictest60650irq/133-enp0s29f1:rx-000:10:200
1883002991814,3cyclictest2142731-21kworker/u8:1+events_unbound23:45:130
1883002991813,3cyclictest2219161-21ntpq00:30:200
1883002991811,5cyclictest2698871-21systemd-journal00:39:590
1883002991810,1cyclictest1961089-21turbostat20:24:590
1883012991715,2cyclictest2012535-21kworker/u8:3+flush-259:021:35:213
1883012991713,3cyclictest2117419-21/usr/sbin/munin22:55:153
1883008991715,1cyclictest0-21swapper/219:35:002
1883008991714,2cyclictest2189927-21irqcore00:05:182
1883008991714,2cyclictest2012535-21kworker/u8:3+events_unbound21:51:212
1883008991714,2cyclictest1913805-21kworker/u8:0+flush-259:019:40:162
1883008991714,2cyclictest1913805-21kworker/u8:0+events_unbound20:35:222
1883008991710,3cyclictest2012535-21kworker/u8:3+events_unbound22:00:172
1883004991714,2cyclictest2189928-21irqcore00:05:191
1883004991714,2cyclictest1971357-21idleruntime-cro20:35:001
1883004991714,2cyclictest1971357-21idleruntime-cro20:35:001
1883004991712,3cyclictest2142731-21kworker/u8:1+flush-259:000:00:251
1883002991713,3cyclictest1913805-21kworker/u8:0+events_unbound20:40:170
1883002991712,4cyclictest2001312-21ntpq21:00:210
1883012991614,2cyclictest2012535-21kworker/u8:3+events_unbound21:55:193
1883012991614,1cyclictest2038256-21kworker/u8:1+flush-259:021:40:163
1883012991613,2cyclictest2210481-21irqcore00:25:173
1883012991613,2cyclictest2153452-21kworker/u8:4+events_unbound23:53:493
1883012991613,2cyclictest2142731-21kworker/u8:1+events_unbound23:28:113
1883012991612,3cyclictest1888336-21kworker/u8:3+events_unbound19:43:183
1883008991613,2cyclictest2142731-21kworker/u8:1+events_unbound23:55:202
1883008991613,2cyclictest2086527-21irqcore22:25:152
1883008991613,2cyclictest1913805-21kworker/u8:0+flush-259:020:40:212
1883008991612,3cyclictest2075184-21kworker/u8:1+events_unbound22:40:162
1883004991612,3cyclictest3000063-21rs:main1
1883004991612,2cyclictest0-21swapper/122:30:001
1883004991610,3cyclictest2698871-21systemd-journal21:37:201
1883002991613,2cyclictest60650irq/133-enp0s29f1:rx-023:05:190
1883002991613,2cyclictest60650irq/133-enp0s29f1:rx-020:25:180
1883002991613,2cyclictest60650irq/133-enp0s29f1:rx-019:59:530
1883002991613,2cyclictest60650irq/133-enp0s29f1:rx-019:50:180
1883002991613,2cyclictest2142731-21kworker/u8:1+flush-259:000:20:210
1883002991612,2cyclictest1893266-21cron19:20:000
1883002991611,4cyclictest1986767-21latency_hist20:50:000
1883012991512,2cyclictest1913805-21kworker/u8:0+events_unbound20:39:573
1883012991512,2cyclictest1888336-21kworker/u8:3+events_unbound20:26:153
1883012991512,2cyclictest1882978-21kworker/u8:2+events_unbound20:04:213
1883012991512,2cyclictest1882978-21kworker/u8:2+events_unbound19:35:173
1883012991511,3cyclictest2163538-21kworker/u8:0+events_unbound00:17:383
1883004991512,2cyclictest2163538-21kworker/u8:0+flush-259:000:15:261
1883004991511,2cyclictest2698871-21systemd-journal22:50:001
1883002991513,1cyclictest60650irq/133-enp0s29f1:rx-023:52:580
1883002991513,1cyclictest60650irq/133-enp0s29f1:rx-023:15:240
1883002991513,1cyclictest60650irq/133-enp0s29f1:rx-023:00:250
1883002991513,1cyclictest60650irq/133-enp0s29f1:rx-020:19:460
1883002991512,2cyclictest60650irq/133-enp0s29f1:rx-023:58:280
1883002991512,2cyclictest60650irq/133-enp0s29f1:rx-022:45:180
1883002991512,2cyclictest60650irq/133-enp0s29f1:rx-020:07:100
1883002991512,2cyclictest60650irq/133-enp0s29f1:rx-019:27:330
1883002991512,2cyclictest60650irq/133-enp0s29f1:rx-000:15:130
1883002991512,2cyclictest2096578-21sh22:35:000
1883002991512,2cyclictest2012535-21kworker/u8:3+flush-259:022:20:240
1883002991512,2cyclictest1971586-21kworker/u8:2+events_unbound22:05:130
1883002991512,2cyclictest1882978-21kworker/u8:2+events_unbound19:40:240
1883002991510,3cyclictest2002471-21/usr/sbin/munin21:05:150
1883012991410,3cyclictest1913805-21kworker/u8:0+events_unbound20:20:283
1883008991411,2cyclictest2153452-21kworker/u8:4+events_unbound23:46:422
1883008991411,2cyclictest2096603-21kworker/u8:0+events_unbound22:58:052
1883008991411,2cyclictest2017860-21kworker/u8:0+events_unbound21:25:192
1883008991411,2cyclictest1882978-21kworker/u8:2+events_unbound19:15:022
188300499149,4cyclictest2117445-21apt-get22:55:121
188300499149,4cyclictest1950571-21/usr/sbin/munin20:15:221
188300499148,4cyclictest2002498-21apt-get21:05:121
1883004991410,3cyclictest1903532-21cron19:30:011
1883004991410,2cyclictest0-21swapper/100:40:001
1883002991413,1cyclictest60650irq/133-enp0s29f1:rx-022:25:150
1883002991413,1cyclictest60650irq/133-enp0s29f1:rx-021:16:500
1883002991412,1cyclictest60650irq/133-enp0s29f1:rx-023:25:450
1883002991412,1cyclictest60650irq/133-enp0s29f1:rx-021:10:170
1883002991410,3cyclictest1971586-21kworker/u8:2+events_unbound20:50:200
188301299139,3cyclictest2163538-21kworker/u8:0+events_unbound00:06:273
188300899139,3cyclictest1913805-21kworker/u8:0+events_unbound20:48:212
188300499139,3cyclictest1940412-21fschecks_count20:05:151
188300499131,3cyclictest2017623-21awk21:19:591
1883002991312,1cyclictest60650irq/133-enp0s29f1:rx-022:01:180
1883002991312,1cyclictest60650irq/133-enp0s29f1:rx-021:45:230
1883002991312,1cyclictest60650irq/133-enp0s29f1:rx-021:40:060
1883002991312,1cyclictest60650irq/133-enp0s29f1:rx-020:11:550
1883002991310,2cyclictest60650irq/133-enp0s29f1:rx-021:55:010
1883002991310,2cyclictest60650irq/133-enp0s29f1:rx-000:29:440
1883012991210,2cyclictest1888336-21kworker/u8:3+events_unbound19:52:443
188300899129,2cyclictest2142731-21kworker/u8:1+events_unbound23:36:192
188300899128,3cyclictest2096603-21kworker/u8:0+events_unbound22:50:572
188300899128,3cyclictest2096603-21kworker/u8:0+events_unbound22:50:572
188300499129,2cyclictest2132223-21unixbench_singl23:05:261
188300499129,2cyclictest2092136-21irqcore22:30:161
188300499129,2cyclictest1935251-21fschecks_count20:00:151
188300499126,4cyclictest492-21dbus-daemon22:45:001
1883004991210,1cyclictest0-21swapper/120:25:001
188300499120,10cyclictest1986770-21sh20:50:001
1883002991210,1cyclictest2038346-21latency_hist21:40:010
188300899118,2cyclictest1971586-21kworker/u8:2+events_unbound21:11:222
188300499119,1cyclictest0-21swapper/122:10:171
188300499119,1cyclictest0-21swapper/100:10:161
188300499118,2cyclictest2210485-21irqcore00:25:171
188300499118,2cyclictest1987162-21cpuspeed_turbos20:50:151
188300499118,2cyclictest1911219-21tr19:35:211
188300499111,3cyclictest1918947-21latency_hist19:45:011
188300499110,9cyclictest1888117-21awk19:15:011
188300499108,1cyclictest0-21swapper/121:30:141
188300499106,3cyclictest1919188-21/usr/sbin/munin19:45:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional