You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 12:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Thu Nov 20, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2253581993934,4cyclictest2370442-21kworker/u8:1+flush-259:022:20:231
2253581993734,2cyclictest2516018-21kworker/u8:4+events_unbound00:30:001
2253581993632,3cyclictest2355197-21kworker/u8:4+events_unbound22:30:151
2253581993532,2cyclictest2516018-21kworker/u8:4+flush-259:000:10:241
2253579993430,3cyclictest2355197-21kworker/u8:4+flush-259:022:15:170
2253581993326,5cyclictest2453309-21latency_hist23:00:011
2253581993326,5cyclictest2448170-21latency_hist22:55:011
2253585993229,2cyclictest2499945-21kworker/u8:3+flush-259:023:55:172
2253585993229,2cyclictest2243821-21kworker/u8:3+flush-259:022:10:152
2253581993229,2cyclictest2489972-21kworker/u8:0+flush-259:023:35:131
2253579993226,5cyclictest2462709-21ntpq23:05:200
2253581993128,2cyclictest2426639-21kworker/u8:1+flush-259:022:40:231
2253581993128,2cyclictest2426639-21kworker/u8:1+flush-259:022:40:231
2253581993128,2cyclictest2258648-21kworker/u8:2+events_unbound20:40:171
2253581993027,2cyclictest2489972-21kworker/u8:0+flush-259:023:50:191
2253581993027,2cyclictest2258648-21kworker/u8:2+flush-259:019:55:111
2253581993025,4cyclictest2355197-21kworker/u8:4+events_unbound21:55:121
2253581993025,3cyclictest2499945-21kworker/u8:3+events_unbound23:55:171
2253581993023,5cyclictest2437655-21/usr/sbin/munin22:45:201
2253579993024,5cyclictest2421142-21ntpq22:25:200
2253587992925,3cyclictest2355197-21kworker/u8:4+flush-259:022:00:183
2253585992924,3cyclictest2323500-21/usr/sbin/munin20:55:212
2253579992925,3cyclictest2489972-21kworker/u8:0+flush-259:023:45:220
2253585992825,2cyclictest2370442-21kworker/u8:1+events_unbound21:50:002
2253581992825,2cyclictest2426639-21kworker/u8:1+flush-259:023:15:321
2253581992825,2cyclictest2426639-21kworker/u8:1+flush-259:023:10:131
2253581992825,2cyclictest2302140-21kworker/u8:0+events_unbound21:10:231
2253581992724,2cyclictest2258648-21kworker/u8:2+events_unbound21:05:231
2253581992724,2cyclictest2243821-21kworker/u8:3+events_unbound20:04:591
2253581992723,3cyclictest2463635-21kworker/u8:2+flush-259:023:20:231
2253581992721,2cyclictest2355197-21kworker/u8:4+events_unbound21:25:181
2253579992724,2cyclictest2258648-21kworker/u8:2+flush-259:020:40:180
2253587992621,4cyclictest2505972-21apt-get23:50:153
2253585992623,2cyclictest2243821-21kworker/u8:3+flush-259:021:20:002
2253581992623,2cyclictest2426639-21kworker/u8:1+events_unbound23:25:251
2253581992623,2cyclictest2416849-21kworker/u8:0+flush-259:022:35:191
2253581992623,2cyclictest2302140-21kworker/u8:0+events_unbound20:45:141
2253581992623,2cyclictest2243821-21kworker/u8:3+events_unbound20:10:161
2253581992622,2cyclictest2302140-21kworker/u8:0+events_unbound21:35:151
2253587992519,4cyclictest2698871-21systemd-journal21:15:003
2253587992519,4cyclictest2500707-21cp23:45:013
2253581992522,2cyclictest2489972-21kworker/u8:0+flush-259:023:45:251
2253581992522,2cyclictest2426639-21kworker/u8:1+flush-259:023:30:231
2253581992522,2cyclictest2258648-21kworker/u8:2+events_unbound20:50:241
2253581992521,3cyclictest2259535-21kworker/u8:0+flush-259:020:05:261
2253581992521,3cyclictest2258648-21kworker/u8:2+flush-259:021:20:211
2253587992419,3cyclictest2526535-21/usr/sbin/munin00:10:163
2253581992421,2cyclictest2515693-21kworker/u8:2+events_unbound00:20:221
2253581992417,5cyclictest2292027-21/usr/sbin/munin20:25:161
2253579992421,2cyclictest2243821-21kworker/u8:3+flush-259:020:00:410
2253579992420,3cyclictest2365307-21latency_hist21:35:010
2253585992320,2cyclictest2505741-21kworker/u8:1+flush-259:000:40:012
2253585992320,2cyclictest2258648-21kworker/u8:2+events_unbound20:30:182
2253587992217,4cyclictest2355209-21/usr/sbin/munin21:25:233
2253587992216,4cyclictest2328623-21/usr/sbin/munin21:00:153
2253587992216,4cyclictest2323271-21latency_hist20:55:003
2253585992219,2cyclictest2302140-21kworker/u8:0+flush-259:020:45:182
2253585992217,4cyclictest2504785-21ntpq23:45:262
2253581992219,2cyclictest2489972-21kworker/u8:0+flush-259:023:40:111
2253581992219,2cyclictest2258648-21kworker/u8:2+events_unbound20:30:281
2253581992218,3cyclictest2355197-21kworker/u8:4+events_unbound22:15:241
2253581992217,4cyclictest2698871-21systemd-journal19:35:281
2253579992217,4cyclictest2380991-21/usr/sbin/munin21:50:150
2253579992216,4cyclictest2370662-21/usr/sbin/munin21:40:130
2253585992118,2cyclictest2437430-21kworker/u8:3+events_unbound23:00:232
2253585992118,2cyclictest2426639-21kworker/u8:1+events_unbound23:20:152
2253585992115,4cyclictest2266257-21/usr/sbin/munin20:00:162
2253581992118,2cyclictest2302140-21kworker/u8:0+flush-259:021:30:181
2253581992118,2cyclictest2281506-21kworker/u8:1+events_unbound20:35:121
2253581992117,3cyclictest2547217-21apt-get00:30:161
2253581992117,3cyclictest2516018-21kworker/u8:4+flush-259:000:20:001
2253581992114,5cyclictest2253987-21cat19:10:021
2253581992113,6cyclictest2698871-21systemd-journal23:05:001
2253579992117,3cyclictest2253408-21kworker/u8:2+events_unbound19:25:010
2253579992117,2cyclictest0-21swapper/023:40:010
2253587992018,1cyclictest2426639-21kworker/u8:1+events_unbound23:20:213
2253587992017,2cyclictest2378152-21kthreadcore21:45:163
2253587992017,2cyclictest2355197-21kworker/u8:4+events_unbound21:55:143
2253587992016,3cyclictest2426639-21kworker/u8:1+events_unbound22:40:203
2253587992016,3cyclictest2426639-21kworker/u8:1+events_unbound22:40:203
2253587992016,3cyclictest2343860-21apt-get21:15:033
2253587992016,3cyclictest2281506-21kworker/u8:1+events_unbound20:40:153
2253585992017,2cyclictest2416849-21kworker/u8:0+flush-259:022:40:312
2253585992017,2cyclictest2416849-21kworker/u8:0+flush-259:022:40:312
2253585992016,3cyclictest2525379-21ntpq00:05:212
2253585992015,4cyclictest2698871-21systemd-journal22:05:002
2253585992015,3cyclictest3000063-21rs:main2
2253581992018,2cyclictest2243821-21kworker/u8:3+flush-259:022:10:171
2253581992018,1cyclictest2291792-21latency_hist20:25:011
2253581992018,1cyclictest2291792-21latency_hist20:25:001
2253581992016,3cyclictest2505741-21kworker/u8:1+flush-259:000:35:201
2253581992015,3cyclictest2401301-21cron22:10:001
225358199201,17cyclictest1-21systemd22:05:001
2253579992018,1cyclictest2258648-21kworker/u8:2+events_unbound20:10:010
2253579992017,2cyclictest2505741-21kworker/u8:1+events_unbound00:15:150
2253579992017,2cyclictest2416849-21kworker/u8:0+events_unbound22:35:210
2253579992016,3cyclictest2258648-21kworker/u8:2+flush-259:020:50:310
2253585991916,2cyclictest2505741-21kworker/u8:1+events_unbound00:30:202
2253585991915,3cyclictest2253490-21apt-key19:10:002
2253585991914,3cyclictest2281506-21kworker/u8:1+flush-259:020:20:212
2253585991914,3cyclictest2281506-21kworker/u8:1+flush-259:020:20:202
2253581991916,2cyclictest2243821-21kworker/u8:3+flush-259:020:15:261
2253581991915,3cyclictest2302140-21kworker/u8:0+events_unbound21:19:561
2253581991913,4cyclictest2698871-21systemd-journal21:40:301
2253579991915,3cyclictest2698871-21systemd-journal19:34:590
2253579991915,3cyclictest2258648-21kworker/u8:2+events_unbound19:55:060
2253587991815,2cyclictest2505741-21kworker/u8:1+flush-259:000:15:203
2253587991814,3cyclictest2443094-21cat22:50:013
2253587991814,3cyclictest2426639-21kworker/u8:1+flush-259:023:10:243
2253585991816,1cyclictest2253576-21cyclictest20:00:002
2253585991815,2cyclictest2426639-21kworker/u8:1+flush-259:022:50:202
2253585991815,2cyclictest2422685-21irqcore22:30:172
2253585991814,3cyclictest2302140-21kworker/u8:0+flush-259:021:00:202
2253585991813,4cyclictest2256936-21apt-get19:35:002
2253581991815,2cyclictest2516018-21kworker/u8:4+events_unbound00:04:561
2253581991815,2cyclictest2426639-21kworker/u8:1+events_unbound23:09:341
2253581991815,2cyclictest2258648-21kworker/u8:2+events_unbound20:55:141
2253579991816,1cyclictest2468758-21cron23:15:000
2253579991814,3cyclictest2302140-21kworker/u8:0+events_unbound21:10:240
2253579991813,3cyclictest2505741-21kworker/u8:1+events_unbound00:35:180
2253579991812,4cyclictest2698871-21systemd-journal21:50:000
2253587991714,2cyclictest2412343-21irqcore22:20:163
2253587991714,2cyclictest2302140-21kworker/u8:0+events_unbound21:35:133
2253587991713,3cyclictest2468757-21apt-get23:15:023
2253587991713,3cyclictest2416849-21kworker/u8:0+flush-259:022:35:003
2253585991713,3cyclictest2516018-21kworker/u8:4+flush-259:000:30:002
225358599170,16cyclictest2281501-21true20:15:002
2253581991714,2cyclictest2505741-21kworker/u8:1+flush-259:000:05:221
2253581991713,2cyclictest0-21swapper/119:49:591
2253579991714,2cyclictest60650irq/133-enp0s29f1:rx-021:28:590
2253579991714,2cyclictest2426639-21kworker/u8:1+flush-259:023:25:230
2253587991613,2cyclictest2406709-21kworker/u8:2+events_unbound22:25:153
2253585991613,2cyclictest2431406-21ntp_kernel_pll_22:35:212
2253585991613,2cyclictest2281506-21kworker/u8:1+events_unbound20:38:312
2253585991613,2cyclictest2259535-21kworker/u8:0+events_unbound20:08:322
2253585991612,3cyclictest2355197-21kworker/u8:4+events_unbound21:40:192
2253581991613,2cyclictest2259535-21kworker/u8:0+events_unbound19:50:311
2253579991614,1cyclictest60650irq/133-enp0s29f1:rx-000:10:210
2253579991613,2cyclictest61450irq/134-enp0s29f1:tx-000:05:150
2253579991613,2cyclictest60650irq/133-enp0s29f1:rx-023:15:210
2253579991613,2cyclictest60650irq/133-enp0s29f1:rx-022:49:490
2253579991613,2cyclictest60650irq/133-enp0s29f1:rx-021:15:130
2253579991613,2cyclictest60650irq/133-enp0s29f1:rx-019:52:100
2253579991613,2cyclictest2489972-21kworker/u8:0+events_unbound23:40:140
2253579991613,2cyclictest2458455-21turbostat23:05:000
2253579991613,2cyclictest2258648-21kworker/u8:2+events_unbound21:05:190
2253579991613,2cyclictest2243821-21kworker/u8:3+flush-259:022:00:270
2253579991612,3cyclictest2292054-21apt-get20:25:130
2253579991612,3cyclictest2258648-21kworker/u8:2+events_unbound20:30:190
2253579991612,3cyclictest2256936-21apt-get19:35:010
2253579991610,4cyclictest2400472-21ntpq22:05:210
2253587991512,2cyclictest2505741-21kworker/u8:1+flush-259:000:35:163
2253587991512,2cyclictest2281747-21apt-get20:15:123
2253587991512,2cyclictest2259535-21kworker/u8:0+flush-259:020:05:163
2253587991510,4cyclictest2526299-21latency_hist00:10:003
2253587991510,3cyclictest2426639-21kworker/u8:1+flush-259:023:30:003
2253585991512,2cyclictest2505741-21kworker/u8:1+flush-259:000:15:252
2253585991512,2cyclictest2302140-21kworker/u8:0+events_unbound21:20:202
2253585991512,2cyclictest2243821-21kworker/u8:3+flush-259:021:35:162
2253585991511,3cyclictest2489972-21kworker/u8:0+flush-259:023:50:162
2253585991511,3cyclictest2426639-21kworker/u8:1+flush-259:022:45:152
2253585991511,3cyclictest2243821-21kworker/u8:3+flush-259:020:16:342
2253585991511,3cyclictest2243821-21kworker/u8:3+events_unbound21:50:142
2253581991512,2cyclictest2370442-21kworker/u8:1+events_unbound21:45:181
2253581991511,3cyclictest2417095-21apt-get22:25:101
2253581991511,3cyclictest2243821-21kworker/u8:3+events_unbound19:26:581
2253579991513,2cyclictest61450irq/134-enp0s29f1:tx-021:00:220
2253579991513,1cyclictest60650irq/133-enp0s29f1:rx-021:24:140
2253579991513,1cyclictest60650irq/133-enp0s29f1:rx-020:45:220
2253579991513,1cyclictest60650irq/133-enp0s29f1:rx-020:10:210
2253579991512,2cyclictest60650irq/133-enp0s29f1:rx-023:20:220
2253579991512,2cyclictest60650irq/133-enp0s29f1:rx-000:33:410
2253579991512,2cyclictest2499945-21kworker/u8:3+flush-259:023:55:190
2253587991412,1cyclictest0-21swapper/321:55:003
2253587991411,2cyclictest2536945-21cpuspeed_turbos00:20:123
2253587991411,2cyclictest2487162-21kthreadcore23:30:183
2253587991411,2cyclictest2426639-21kworker/u8:1+events_unbound22:45:143
2253587991411,2cyclictest2281506-21kworker/u8:1+events_unbound20:35:193
225358599149,4cyclictest2360154-21latency_hist21:30:012
225358599149,4cyclictest2355197-21kworker/u8:4+events_unbound22:21:332
2253585991411,2cyclictest2516018-21kworker/u8:4+events_unbound00:13:402
2253585991411,2cyclictest2489972-21kworker/u8:0+flush-259:023:35:232
2253585991411,2cyclictest2426639-21kworker/u8:1+flush-259:023:15:182
2253585991411,2cyclictest2360615-21fschecks_count21:30:132
2253585991411,2cyclictest2254720-21kworker/u8:4+flush-259:019:21:012
2253585991410,3cyclictest2257482-21cat19:35:002
2253581991411,2cyclictest2302140-21kworker/u8:0+events_unbound21:00:191
2253581991410,3cyclictest2254720-21kworker/u8:4+events_unbound19:18:591
2253579991412,1cyclictest60650irq/133-enp0s29f1:rx-023:50:270
2253579991412,1cyclictest60650irq/133-enp0s29f1:rx-022:55:270
2253579991412,1cyclictest60650irq/133-enp0s29f1:rx-022:50:380
2253579991412,1cyclictest60650irq/133-enp0s29f1:rx-022:30:250
2253579991412,1cyclictest60650irq/133-enp0s29f1:rx-022:20:110
2253579991412,1cyclictest60650irq/133-enp0s29f1:rx-020:20:210
2253579991412,1cyclictest60650irq/133-enp0s29f1:rx-020:20:210
2253579991411,2cyclictest60650irq/133-enp0s29f1:rx-000:28:110
2253579991411,2cyclictest2436955-21unixbench_multi22:40:280
2253579991411,2cyclictest2436955-21unixbench_multi22:40:280
2253579991411,2cyclictest2355197-21kworker/u8:4+events_unbound21:35:170
225358799138,3cyclictest2257578-21cp19:35:003
2253587991310,2cyclictest2542214-21cpuspeed_turbos00:25:143
2253587991310,2cyclictest2427538-21cpuspeed_turbos22:35:133
2253587991310,2cyclictest2301699-21unixbench_singl20:30:293
2253587991310,2cyclictest2243821-21kworker/u8:3+flush-259:020:25:203
2253587991310,1cyclictest0-21swapper/321:05:203
225358599139,3cyclictest2259535-21kworker/u8:0+events_unbound19:54:312
225358599139,3cyclictest2257592-21kworker/u8:1+events_unbound19:49:322
2253585991311,1cyclictest0-21swapper/223:40:162
2253585991310,2cyclictest2516432-21fschecks_count00:00:142
2253585991310,2cyclictest2437430-21kworker/u8:3+flush-259:022:55:232
2253585991310,2cyclictest2419392-21kthreadcore22:25:162
2253585991310,2cyclictest2355197-21kworker/u8:4+events_unbound22:05:132
225358199139,3cyclictest2254720-21kworker/u8:4+flush-259:019:20:481
2253581991310,2cyclictest2255684-21kworker/u8:0+events_unbound19:34:571
2253581991310,2cyclictest2243821-21kworker/u8:3+events_unbound21:54:361
2253579991311,1cyclictest60650irq/133-enp0s29f1:rx-021:57:510
2253579991311,1cyclictest60650irq/133-enp0s29f1:rx-020:55:220
2253579991311,1cyclictest60650irq/133-enp0s29f1:rx-000:20:140
2253579991310,2cyclictest2515693-21kworker/u8:2+events_unbound00:00:160
2253579991310,2cyclictest0-21swapper/022:10:140
225358799128,3cyclictest2255684-21kworker/u8:0+flush-259:019:30:323
225358799126,4cyclictest2494845-21cron23:40:003
2253587991210,1cyclictest0-21swapper/300:05:003
225358599129,2cyclictest0-21swapper/220:30:002
225358599128,3cyclictest2302140-21kworker/u8:0+events_unbound21:10:212
225358599128,3cyclictest2258648-21kworker/u8:2+events_unbound20:50:222
225358599128,3cyclictest2243821-21kworker/u8:3+flush-259:019:25:572
2253585991210,1cyclictest0-21swapper/223:10:002
225358199129,2cyclictest2257592-21kworker/u8:1+events_unbound19:41:271
2253579991211,1cyclictest60650irq/133-enp0s29f1:rx-023:31:180
2253579991210,1cyclictest60650irq/133-enp0s29f1:rx-020:15:250
2253579991210,1cyclictest60650irq/133-enp0s29f1:rx-019:26:140
225358799118,2cyclictest2281506-21kworker/u8:1+events_unbound20:20:153
225358799118,2cyclictest2281506-21kworker/u8:1+events_unbound20:20:153
225358799117,3cyclictest2698871-21systemd-journal20:49:553
225358799116,4cyclictest2354801-21cat21:25:013
225358599118,2cyclictest2479819-21cpuspeed_turbos23:25:172
225358599118,2cyclictest2308417-21sed20:40:182
225358599118,2cyclictest2243821-21kworker/u8:3+events_unbound19:15:292
225358599116,3cyclictest2411370-21sshd22:16:312
225357999119,1cyclictest60650irq/133-enp0s29f1:rx-019:47:570
225357999119,1cyclictest60650irq/133-enp0s29f1:rx-019:18:480
2253579991110,1cyclictest60650irq/133-enp0s29f1:rx-019:10:240
225358799107,2cyclictest2375141-21unixbench_singl21:40:243
225358799106,3cyclictest2258648-21kworker/u8:2+flush-259:019:55:363
225357999109,1cyclictest60650irq/133-enp0s29f1:rx-019:42:310
22535879997,1cyclictest0-21swapper/319:54:593
22535879994,4cyclictest2405605-21ntpq22:10:183
22535879990,7cyclictest2256335-21awk19:30:013
22535879990,2cyclictest2328382-21latency_hist21:00:013
22535859997,1cyclictest0-21swapper/219:45:002
22535879974,2cyclictest2254677-21kworker/u8:0+flush-259:019:15:263
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional