You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 18:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot4s.osadl.org (updated Fri Nov 28, 2025 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2224590995448,4cyclictest2468955-21latency_hist11:05:013
2224580994438,5cyclictest2422308-21mandb10:17:280
2224586994135,4cyclictest2348899-21latency_hist09:10:022
2224586993936,2cyclictest75-21kswapd010:17:232
2224586993832,4cyclictest2401698-21latency_hist10:00:002
2224583993734,2cyclictest75-21kswapd010:17:241
2224583993529,4cyclictest2349147-21/usr/sbin/munin09:10:121
2224590993430,3cyclictest2489629-21kworker/u8:3+flush-259:011:30:013
2224583993430,3cyclictest2531280-21kworker/u8:1+flush-259:012:25:001
2224583993430,3cyclictest2531280-21kworker/u8:1+flush-259:012:25:001
2224590993126,4cyclictest2698871-21systemd-journal10:55:013
2224583993124,5cyclictest2698871-21systemd-journal09:10:011
2224590993028,1cyclictest75-21kswapd009:55:163
2224590992925,3cyclictest3304397-21kworker/u8:2+flush-259:007:35:013
2224590992925,3cyclictest3304397-21kworker/u8:2+flush-259:007:35:003
2224590992826,1cyclictest75-21kswapd010:17:253
2224583992724,2cyclictest2417184-21kworker/u8:1+flush-259:010:20:201
2224586992620,4cyclictest2463802-21latency_hist11:00:012
2224583992623,2cyclictest2265888-21kworker/u8:4+flush-259:008:10:181
2224580992621,4cyclictest2386027-21kworker/u8:2+flush-259:010:00:170
2224590992519,4cyclictest3000063-21rs:main3
2224586992521,3cyclictest2427769-21kworker/u8:3+flush-259:010:45:182
2224583992522,2cyclictest2500366-21kworker/u8:2+events_unbound11:40:191
2224583992521,3cyclictest2301993-21kworker/u8:0+events_unbound08:45:191
2224583992422,2cyclictest2301993-21kworker/u8:0+events_unbound09:15:261
2224583992421,2cyclictest2292125-21kworker/u8:3+flush-259:009:40:231
2224583992420,3cyclictest2301993-21kworker/u8:0+events_unbound09:20:151
2224583992417,4cyclictest2307358-21latency_hist08:30:001
2224580992420,3cyclictest2235052-21kworker/u8:3+flush-259:007:45:000
2224580992419,4cyclictest2301993-21kworker/u8:0+flush-259:009:29:590
2224580992416,6cyclictest2566474-21ntpq12:35:220
2224583992319,3cyclictest2380633-21kworker/u8:1+flush-259:009:55:231
2224583992318,3cyclictest2521200-21/usr/sbin/munin11:55:221
2224583992317,4cyclictest2438011-21latency_hist10:35:011
2224590992216,4cyclictest2427991-21/usr/sbin/munin10:25:153
2224586992219,2cyclictest2354039-21sh09:15:002
2224583992219,2cyclictest2453497-21kworker/u8:2+events_unbound11:05:151
2224583992218,3cyclictest2406896-21kworker/u8:0+flush-259:011:00:201
2224583992217,4cyclictest2281166-21latency_hist08:05:001
2224583992216,4cyclictest2504608-21ntpq11:35:211
2224580992216,4cyclictest2265888-21kworker/u8:4+flush-259:008:10:140
2224580992216,4cyclictest2245502-21diskstats07:30:140
2224580992216,4cyclictest2245502-21diskstats07:30:130
2224580992215,5cyclictest2229653-21latency_hist07:15:000
2224590992117,3cyclictest2453497-21kworker/u8:2+flush-259:011:25:013
2224586992115,4cyclictest2265657-21latency_hist07:50:002
2224583992118,2cyclictest2437527-21kworker/u8:1+events_unbound11:10:231
2224583992118,2cyclictest2250470-21kworker/u8:0+flush-259:007:35:191
2224583992115,4cyclictest2483513-21ntpq11:15:201
2224583992114,5cyclictest2698871-21systemd-journal11:35:001
2224580992117,3cyclictest2380884-21apt-get09:40:100
2224580992117,2cyclictest2526144-21unin-run12:00:000
2224580992115,4cyclictest2275202-21ntpq07:55:190
2224580992114,5cyclictest2337801-21ntpq08:55:200
2224590992018,1cyclictest2265888-21kworker/u8:4+flush-259:008:25:003
2224590992016,3cyclictest2500366-21kworker/u8:2+flush-259:011:40:263
2224590992016,3cyclictest2292125-21kworker/u8:3+flush-259:010:10:003
2224590992016,3cyclictest2265888-21kworker/u8:4+flush-259:008:05:013
2224590992013,5cyclictest2239959-21latency_hist07:25:013
222458699208,8cyclictest2306546-21ntp_states08:25:222
222458699204,6cyclictest2442056-21cat10:35:162
2224586992014,4cyclictest2422789-21/usr/sbin/munin10:20:122
2224583992017,2cyclictest2333105-21kworker/u8:1+flush-259:009:25:231
2224583992016,3cyclictest3000063-21rs:main1
2224583992016,3cyclictest2500366-21kworker/u8:2+flush-259:012:00:231
2224583992016,3cyclictest2406896-21kworker/u8:0+events_unbound11:25:201
2224583992016,3cyclictest2301993-21kworker/u8:0+flush-259:008:55:241
2224583992016,3cyclictest2208722-21kworker/u8:0+flush-259:007:20:251
2224583992015,3cyclictest2698871-21systemd-journal10:49:381
2224580992016,3cyclictest2453497-21kworker/u8:2+events_unbound11:15:010
2224580992014,4cyclictest2301993-21kworker/u8:0+flush-259:009:30:170
2224580992013,5cyclictest2473269-21ntpq11:05:220
2224590991915,3cyclictest2515703-21kworker/u8:3+flush-259:011:50:003
2224590991915,3cyclictest2406896-21kworker/u8:0+flush-259:010:20:163
2224590991915,3cyclictest0-21swapper/308:10:003
2224590991913,4cyclictest2396081-21latency_hist09:55:013
2224590991912,5cyclictest2698871-21systemd-journal11:20:013
2224586991915,3cyclictest2301993-21kworker/u8:0+flush-259:009:00:182
2224583991916,2cyclictest2250470-21kworker/u8:0+events_unbound07:40:151
2224583991915,3cyclictest503-21systemd-logind10:45:011
2224583991915,3cyclictest2531280-21kworker/u8:1+flush-259:012:05:261
2224583991915,3cyclictest2500366-21kworker/u8:2+flush-259:012:30:231
2224583991915,3cyclictest2386027-21kworker/u8:2+flush-259:010:10:241
2224583991915,3cyclictest2265888-21kworker/u8:4+flush-259:008:05:161
2224583991915,2cyclictest2301993-21kworker/u8:0+flush-259:009:30:151
2224583991914,3cyclictest2552143-21apt-get12:25:101
2224580991915,3cyclictest2292125-21kworker/u8:3+flush-259:009:20:250
2224580991914,4cyclictest2427769-21kworker/u8:3+flush-259:010:30:000
2224580991914,3cyclictest2359325-21apt-key09:20:000
2224590991814,3cyclictest2437527-21kworker/u8:1+flush-259:010:35:003
2224590991814,3cyclictest2369487-21ntpq09:25:213
2224583991814,3cyclictest2406896-21kworker/u8:0+flush-259:010:25:211
2224583991814,3cyclictest2292125-21kworker/u8:3+flush-259:009:35:151
2224583991814,3cyclictest2265888-21kworker/u8:4+events_unbound07:50:191
2224580991814,3cyclictest2301993-21kworker/u8:0+flush-259:008:50:170
2224580991811,5cyclictest2321923-21ntpq08:40:200
2224580991810,6cyclictest2437161-21ntpq10:30:230
2224590991714,2cyclictest2292125-21kworker/u8:3+flush-259:008:45:013
2224590991713,3cyclictest2500366-21kworker/u8:2+flush-259:012:10:003
2224586991714,2cyclictest2302196-21sh08:25:002
2224586991711,4cyclictest2698871-21systemd-journal12:33:382
2224586991711,4cyclictest2401960-21apt-get10:00:122
2224583991714,2cyclictest2515703-21kworker/u8:3+events_unbound12:10:231
2224583991714,2cyclictest2235052-21kworker/u8:3+flush-259:007:25:231
2224583991713,3cyclictest2427769-21kworker/u8:3+events_unbound10:35:371
2224583991713,3cyclictest2245101-21kworker/u8:1+events_unbound07:55:141
2224580991715,1cyclictest2406896-21kworker/u8:0+flush-259:010:15:000
2224580991714,2cyclictest2531280-21kworker/u8:1+events_unbound12:05:190
2224580991711,5cyclictest2270822-21latency_hist07:55:010
222459099169,5cyclictest2380608-21latency_hist09:40:013
2224590991614,1cyclictest0-21swapper/311:54:593
2224586991612,3cyclictest3000063-21rs:main2
2224586991610,4cyclictest2260498-21latency_hist07:45:012
2224583991613,2cyclictest2265888-21kworker/u8:4+flush-259:008:20:161
2224583991612,3cyclictest2453497-21kworker/u8:2+events_unbound11:20:201
2224583991612,3cyclictest2292125-21kworker/u8:3+events_unbound09:04:391
2224583991612,3cyclictest2265888-21kworker/u8:4+flush-259:008:30:231
2224583991610,4cyclictest2698871-21systemd-journal10:02:381
2224580991614,1cyclictest60650irq/133-enp0s29f1:rx-011:15:130
2224580991614,1cyclictest60650irq/133-enp0s29f1:rx-010:45:260
2224580991613,2cyclictest60650irq/133-enp0s29f1:rx-010:24:280
2224580991613,2cyclictest60650irq/133-enp0s29f1:rx-008:45:140
2224580991612,3cyclictest3304397-21kworker/u8:2+events_unbound07:10:000
2224580991612,3cyclictest2500366-21kworker/u8:2+flush-259:011:40:240
2224580991612,3cyclictest2500366-21kworker/u8:2+flush-259:011:35:160
2224580991612,3cyclictest2386027-21kworker/u8:2+events_unbound09:55:130
2224580991611,4cyclictest2250480-21/usr/sbin/munin07:35:150
2224580991610,4cyclictest2234825-21latency_hist07:20:010
2224590991511,3cyclictest0-21swapper/312:25:003
2224590991511,3cyclictest0-21swapper/312:25:003
2224590991510,3cyclictest2500366-21kworker/u8:2+flush-259:012:40:003
222458699159,4cyclictest2557067-21latency_hist12:30:012
222458699155,6cyclictest2495412-21/usr/sbin/munin11:30:192
222458699151,4cyclictest2357022-21cut09:15:192
2224586991512,2cyclictest2416297-21ntp_kernel_pll_10:10:212
2224586991511,3cyclictest3304397-21kworker/u8:2+flush-259:007:30:192
2224586991511,3cyclictest3304397-21kworker/u8:2+flush-259:007:30:182
2224586991511,3cyclictest2386173-21cut09:45:132
2224586991511,3cyclictest2333072-21idleruntime-cro08:55:012
222458699150,6cyclictest2367784-21kthreadcore09:25:162
222458399159,4cyclictest3000063-21rs:main1
2224583991512,2cyclictest2536649-21kworker/u8:0+flush-259:012:15:211
2224583991512,2cyclictest2312730-21kworker/u8:2+events_unbound08:41:391
2224583991511,3cyclictest2500366-21kworker/u8:2+flush-259:011:45:271
2224583991511,3cyclictest2437527-21kworker/u8:1+flush-259:010:50:151
2224583991511,3cyclictest2250470-21kworker/u8:0+events_unbound07:45:211
2224580991513,1cyclictest60650irq/133-enp0s29f1:rx-011:50:250
2224580991513,1cyclictest60650irq/133-enp0s29f1:rx-011:31:120
2224580991513,1cyclictest60650irq/133-enp0s29f1:rx-011:02:250
2224580991513,1cyclictest60650irq/133-enp0s29f1:rx-010:09:240
2224580991513,1cyclictest60650irq/133-enp0s29f1:rx-008:35:120
2224580991513,1cyclictest60650irq/133-enp0s29f1:rx-007:49:580
2224580991512,2cyclictest60650irq/133-enp0s29f1:rx-012:00:200
2224580991512,2cyclictest60650irq/133-enp0s29f1:rx-007:21:450
2224580991512,2cyclictest2536649-21kworker/u8:0+flush-259:012:15:240
2224580991512,2cyclictest2302873-21irqcore08:25:150
2224580991512,2cyclictest2286303-21sh08:10:010
2224580991511,3cyclictest60650irq/133-enp0s29f1:rx-007:25:200
2224580991511,3cyclictest2453737-21apt-get10:50:130
2224580991510,3cyclictest2265888-21kworker/u8:4+flush-259:008:30:150
222459099148,4cyclictest2437527-21kworker/u8:1+flush-259:010:35:143
222459099141,3cyclictest2333087-21awk08:55:003
2224590991411,2cyclictest2301993-21kworker/u8:0+flush-259:008:35:133
2224590991410,3cyclictest2333779-21apt-get08:55:123
222458699142,4cyclictest2333945-21/usr/sbin/munin08:55:142
222458699141,7cyclictest2526126-21munin-run12:00:002
2224586991411,2cyclictest2514923-21ntp_kernel_pll_11:45:242
2224586991411,2cyclictest2406896-21kworker/u8:0+flush-259:010:25:172
222458699140,4cyclictest2468095-21ld11:00:242
2224583991412,1cyclictest0-21swapper/111:55:001
2224583991411,2cyclictest3304397-21kworker/u8:2+flush-259:007:30:241
2224583991411,2cyclictest3304397-21kworker/u8:2+flush-259:007:30:231
2224583991410,3cyclictest2380633-21kworker/u8:1+events_unbound09:45:121
2224580991413,1cyclictest60650irq/133-enp0s29f1:rx-010:37:500
2224580991412,2cyclictest60650irq/133-enp0s29f1:rx-011:45:260
2224580991412,1cyclictest60650irq/133-enp0s29f1:rx-012:30:260
2224580991412,1cyclictest60650irq/133-enp0s29f1:rx-011:29:280
2224580991412,1cyclictest60650irq/133-enp0s29f1:rx-009:39:130
2224580991412,1cyclictest60650irq/133-enp0s29f1:rx-008:20:160
2224580991412,1cyclictest60650irq/133-enp0s29f1:rx-008:00:240
2224580991411,2cyclictest61450irq/134-enp0s29f1:tx-008:15:230
2224580991411,2cyclictest60650irq/133-enp0s29f1:rx-012:24:140
2224580991411,2cyclictest60650irq/133-enp0s29f1:rx-012:24:130
2224580991411,2cyclictest60650irq/133-enp0s29f1:rx-011:20:110
222459099139,2cyclictest2312016-21unixbench_multi08:30:233
222459099138,4cyclictest2292125-21kworker/u8:3+flush-259:009:05:003
222459099138,3cyclictest2437527-21kworker/u8:1+flush-259:010:44:593
222459099138,3cyclictest2255724-21cut07:40:143
222459099138,3cyclictest2229662-21turbostat.cron07:14:593
2224590991311,1cyclictest2302583-21tr08:25:133
2224590991311,1cyclictest0-21swapper/310:55:153
2224590991310,2cyclictest2541907-21cpuspeed_turbos12:15:123
2224590991310,2cyclictest0-21swapper/308:10:163
2224590991310,1cyclictest0-21swapper/309:30:243
222458699139,3cyclictest2281165-21idleruntime-cro08:05:002
222458699139,2cyclictest0-21swapper/208:35:002
222458699132,7cyclictest2536838-21df_inode12:10:132
222458699132,2cyclictest2535599-21gcc12:05:242
222458699132,2cyclictest2239126-21ld07:20:232
222458699131,9cyclictest0-21swapper/209:20:212
222458699131,7cyclictest2516232-21diskstats11:50:142
222458699131,7cyclictest2453897-21diskstats10:50:152
222458699131,7cyclictest2384729-21memory09:40:192
222458699131,2cyclictest2437197-21ntpq10:30:232
222458699131,2cyclictest2411743-21modprobe10:06:392
222458699131,2cyclictest2266904-21irqstats07:50:182
2224586991311,1cyclictest391rcuc/207:38:382
2224586991310,2cyclictest0-21swapper/208:10:002
2224586991310,1cyclictest191rcu_preempt11:26:062
2224583991310,2cyclictest2386027-21kworker/u8:2+flush-259:010:05:211
2224580991312,1cyclictest60650irq/133-enp0s29f1:rx-010:59:160
2224580991311,1cyclictest60650irq/133-enp0s29f1:rx-012:26:020
2224580991310,2cyclictest60650irq/133-enp0s29f1:rx-009:06:200
222458099130,11cyclictest2405064-21sshd10:40:220
222459099129,2cyclictest2524183-21kthreadcore11:55:173
222459099129,2cyclictest2501012-21irqcore11:35:173
222459099129,2cyclictest2364759-21sh09:25:003
222459099129,2cyclictest0-21swapper/308:15:223
222459099129,1cyclictest0-21swapper/307:19:593
222458699129,1cyclictest191rcu_preempt08:41:492
222458699121,7cyclictest2526543-21diskstats12:00:142
222458699121,7cyclictest2500566-21/usr/sbin/munin11:35:192
222458699121,6cyclictest2469377-21df_inode11:05:132
222458699121,6cyclictest2323363-21fw_forwarded_lo08:45:142
222458699121,6cyclictest2271231-21df07:55:122
222458699121,2cyclictest41-21ksoftirqd/211:10:212
222458699121,2cyclictest2447389-21ntpq10:40:192
222458699121,2cyclictest2317676-21systemctl08:40:012
222458699121,2cyclictest2224966-21diskstats07:10:142
2224586991211,1cyclictest391rcuc/209:54:252
2224586991210,1cyclictest391rcuc/212:19:202
222458399129,2cyclictest2395665-21unixbench_singl09:50:261
222458099129,2cyclictest60650irq/133-enp0s29f1:rx-009:54:310
2224580991211,1cyclictest60650irq/133-enp0s29f1:rx-009:11:180
222459099119,1cyclictest0-21swapper/312:30:253
222459099119,1cyclictest0-21swapper/312:00:173
222459099119,1cyclictest0-21swapper/307:35:173
222459099118,2cyclictest2240289-21cpuspeed_turbos07:25:123
222459099117,3cyclictest2250470-21kworker/u8:0+flush-259:007:45:183
222459099115,4cyclictest2540704-21ntpq12:10:213
222459099114,5cyclictest2698871-21systemd-journal07:55:003
222459099110,9cyclictest2390948-21awk09:49:593
222458699113,6cyclictest2509806-21ntpq11:40:232
222458699111,3cyclictest41-21ksoftirqd/212:20:172
222458699111,3cyclictest41-21ksoftirqd/212:20:172
222458699111,3cyclictest41-21ksoftirqd/209:35:192
222458699111,2cyclictest41-21ksoftirqd/208:10:022
222458699110,7cyclictest2230204-21fw_conntrack07:15:132
222458699110,2cyclictest2566522-21ntpq12:35:232
222458699110,2cyclictest2372891-21ps09:30:172
222458699110,2cyclictest2241204-21irqstats07:25:162
222458099116,3cyclictest2536681-21apt-get12:10:110
222459099108,1cyclictest2552288-21cpuspeed_turbos12:25:113
222459099108,1cyclictest0-21swapper/311:10:143
222459099106,3cyclictest2437527-21kworker/u8:1+flush-259:010:45:063
222458399106,3cyclictest2265888-21kworker/u8:4+events_unbound08:15:251
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional