You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 22:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Sat Jun 28, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2184982994537,6cyclictest2320895-21latency_hist09:20:011
2184982994138,2cyclictest2315799-21kworker/u8:2+events_unbound09:40:191
2184978993933,4cyclictest2268834-21latency_hist08:30:000
2184978993833,4cyclictest1-21systemd09:00:000
2184982993633,2cyclictest2508167-21kworker/u8:0+events_unbound12:25:331
2184982993633,2cyclictest2466158-21kworker/u8:0+events_unbound11:57:341
2184978993630,4cyclictest2237666-21apt-get08:00:140
2184989993530,3cyclictest2290269-21kworker/u8:1+flush-259:008:55:193
2184982993430,3cyclictest2315799-21kworker/u8:2+events_unbound09:38:361
2184978993428,4cyclictest2279196-21latency_hist08:40:000
2184982993330,2cyclictest2305474-21kworker/u8:0+events_unbound09:30:321
2184986993122,3cyclictest2492324-21apt-get12:05:002
2184986993121,9cyclictest2279432-21/usr/sbin/munin08:40:172
2184982993128,2cyclictest2216344-21kworker/u8:3+events_unbound07:58:351
2184982993128,2cyclictest2180445-21kworker/u8:0+events_unbound07:19:561
2184978993128,2cyclictest2258113-21kworker/u8:0+flush-259:008:25:010
2184989993024,4cyclictest3754036-21systemd-journal10:35:003
2184986993026,3cyclictest2346567-21apt-get09:45:002
2184982993024,4cyclictest2326061-21latency_hist09:25:011
2184978993027,2cyclictest2180445-21kworker/u8:0+flush-259:007:15:000
2184978993024,4cyclictest2284441-21latency_hist08:45:000
2184978993023,5cyclictest2358539-21diskstats09:55:150
2184989992923,4cyclictest2528003-21ntpq12:35:243
2184982992926,2cyclictest4000128-21kworker/u8:2+events_unbound07:14:551
2184982992924,3cyclictest2236975-21apt-get08:00:031
2184982992825,2cyclictest2394179-21kworker/u8:1+events_unbound11:15:321
2184982992825,2cyclictest2378920-21kworker/u8:4+events_unbound10:16:151
2184982992825,2cyclictest2248212-21kworker/u8:1+events_unbound08:12:351
2184978992822,4cyclictest2356795-21ntpq09:50:250
2184989992722,3cyclictest2278779-21apt-get08:35:343
2184982992724,2cyclictest2216344-21kworker/u8:3+events_unbound07:40:021
2184982992721,4cyclictest2404409-21latency_hist10:40:001
2184982992720,5cyclictest2211428-21/usr/sbin/munin07:35:181
2184978992724,2cyclictest4000128-21kworker/u8:2+flush-259:008:05:210
2184978992722,4cyclictest2289717-21cp08:50:000
2184989992619,5cyclictest2377812-21ntpq10:10:233
2184982992624,2cyclictest2248212-21kworker/u8:1+events_unbound08:35:541
2184982992623,2cyclictest4000128-21kworker/u8:2+events_unbound08:16:351
2184982992623,2cyclictest2305474-21kworker/u8:0+events_unbound10:49:361
2184982992621,4cyclictest2310884-21apt-get09:10:141
2184982992621,4cyclictest2310884-21apt-get09:10:141
2184982992620,4cyclictest3754036-21systemd-journal07:20:221
2184978992623,2cyclictest2394179-21kworker/u8:1+flush-259:011:15:000
2184989992522,2cyclictest4000128-21kworker/u8:2+flush-259:008:25:193
2184986992518,5cyclictest2471320-21latency_hist11:45:002
2184982992522,2cyclictest4000128-21kworker/u8:2+events_unbound08:25:031
2184982992522,2cyclictest2508167-21kworker/u8:0+events_unbound12:39:341
2184978992521,3cyclictest2347256-21apt-get09:45:140
2184986992420,3cyclictest2290269-21kworker/u8:1+flush-259:008:55:262
2184986992416,5cyclictest2206037-21latency_hist07:30:002
2184982992421,2cyclictest2450745-21kworker/u8:3+events_unbound11:25:331
2184978992422,2cyclictest2394179-21kworker/u8:1+events_unbound11:15:220
2184978992420,3cyclictest2487432-21apt-get12:00:150
2184989992320,3cyclictest2315799-21kworker/u8:2+events_unbound09:30:023
2184989992320,2cyclictest2305474-21kworker/u8:0+events_unbound10:45:183
2184982992320,2cyclictest2471971-21kworker/u8:4+events_unbound11:53:351
2184982992319,3cyclictest2497913-21kworker/u8:2+flush-259:012:20:011
2184982992318,4cyclictest2326102-21kworker/u8:3+flush-259:010:10:181
2184982992317,4cyclictest2466372-21/usr/sbin/munin11:40:211
2184978992318,4cyclictest2409818-21apt-get10:45:140
2184978992317,4cyclictest2257663-21ntpq08:15:270
2184989992218,3cyclictest4000128-21kworker/u8:2+flush-259:007:45:203
2184982992219,2cyclictest2450745-21kworker/u8:3+events_unbound12:06:511
2184982992219,2cyclictest2326102-21kworker/u8:3+events_unbound09:46:141
2184982992219,1cyclictest0-21swapper/109:05:001
2184978992219,2cyclictest2305474-21kworker/u8:0+events_unbound09:30:220
2184978992216,4cyclictest2471317-21apt-get11:45:020
2184989992118,2cyclictest2216344-21kworker/u8:3+events_unbound07:55:013
2184986992114,5cyclictest2195289-21latency_hist07:20:012
2184982992118,2cyclictest4000128-21kworker/u8:2+events_unbound08:47:021
2184982992118,2cyclictest2487186-21kworker/u8:1+events_unbound12:11:161
2184982992118,2cyclictest2450745-21kworker/u8:3+events_unbound11:36:551
2184982992118,2cyclictest2378920-21kworker/u8:4+events_unbound10:29:201
2184982992118,2cyclictest2295406-21kworker/u8:4+events_unbound09:08:561
2184982992118,2cyclictest2258113-21kworker/u8:0+events_unbound08:42:521
2184982992118,2cyclictest2180445-21kworker/u8:0+events_unbound07:29:471
2184982992117,3cyclictest2315799-21kworker/u8:2+events_unbound09:29:161
2184978992118,2cyclictest2409547-21kworker/u8:3+events_unbound10:50:280
2184989992017,2cyclictest2394179-21kworker/u8:1+flush-259:011:15:293
2184989992016,3cyclictest4000128-21kworker/u8:2+events_unbound08:40:183
2184986992016,2cyclictest0-21swapper/210:44:592
2184986992012,6cyclictest3754036-21systemd-journal09:55:012
2184982992017,2cyclictest4000128-21kworker/u8:2+events_unbound08:06:231
2184982992017,2cyclictest2487186-21kworker/u8:1+events_unbound12:33:361
2184982992017,2cyclictest2487186-21kworker/u8:1+events_unbound12:22:261
2184982992017,2cyclictest2466158-21kworker/u8:0+events_unbound11:47:531
2184982992017,2cyclictest2430136-21kworker/u8:0+events_unbound11:12:011
2184982992017,2cyclictest2419551-21kworker/u8:2+events_unbound10:57:021
2184982992017,2cyclictest2394179-21kworker/u8:1+events_unbound10:40:071
2184982992017,2cyclictest2326102-21kworker/u8:3+events_unbound09:57:131
2184982992017,2cyclictest2290269-21kworker/u8:1+events_unbound08:57:251
2184982992017,2cyclictest2258113-21kworker/u8:0+events_unbound08:33:071
2184982992017,2cyclictest2206297-21apt-get07:30:141
2184978992017,2cyclictest2305474-21kworker/u8:0+flush-259:010:00:310
2184978992016,3cyclictest2263688-21kworker/u8:3+events_unbound08:30:230
2184989991916,2cyclictest2518779-21apt-get12:30:123
2184989991916,2cyclictest2394117-21cron10:30:003
2184989991916,2cyclictest2290269-21kworker/u8:1+events_unbound09:00:283
2184989991916,2cyclictest2180445-21kworker/u8:0+events_unbound07:10:223
2184989991915,3cyclictest2466158-21kworker/u8:0+flush-259:011:40:273
2184989991913,4cyclictest2358379-21apt-get09:55:133
218498699198,1cyclictest0-21swapper/210:10:202
2184986991916,2cyclictest2528865-21latency_hist12:40:012
2184986991916,2cyclictest2425023-21cron10:59:592
2184986991912,5cyclictest2492324-21apt-get12:05:022
218498699191,1cyclictest0-21swapper/209:05:162
2184982991916,2cyclictest2420081-21kworker/u8:4+events_unbound11:02:281
2184982991916,2cyclictest2394179-21kworker/u8:1+events_unbound11:08:231
2184982991916,2cyclictest2305474-21kworker/u8:0+events_unbound10:22:371
2184978991916,2cyclictest2466158-21kworker/u8:0+events_unbound11:40:180
2184978991915,3cyclictest2394179-21kworker/u8:1+events_unbound11:20:210
2184978991915,3cyclictest2305474-21kworker/u8:0+flush-259:010:20:190
2184978991912,5cyclictest2345643-21ntpq09:40:250
2184989991815,2cyclictest2450745-21kworker/u8:3+events_unbound12:10:203
2184989991815,2cyclictest2315799-21kworker/u8:2+flush-259:009:35:253
2184989991815,2cyclictest2180445-21kworker/u8:0+events_unbound07:15:013
218498699188,1cyclictest0-21swapper/207:50:002
2184986991814,3cyclictest2450745-21kworker/u8:3+flush-259:012:25:012
2184986991814,3cyclictest2378920-21kworker/u8:4+flush-259:010:25:002
2184986991812,4cyclictest2326302-21/usr/sbin/munin09:25:232
2184982991815,2cyclictest2430136-21kworker/u8:0+events_unbound11:22:091
2184982991815,2cyclictest2409547-21kworker/u8:3+events_unbound10:51:021
2184982991815,2cyclictest2394179-21kworker/u8:1+events_unbound11:33:161
2184982991815,2cyclictest2326102-21kworker/u8:3+events_unbound10:07:121
2184982991815,2cyclictest2216344-21kworker/u8:3+events_unbound07:51:241
2184982991815,2cyclictest2184973-21cyclictest08:55:001
2184978991816,1cyclictest0-21swapper/007:40:000
2184978991815,2cyclictest2430136-21kworker/u8:0+events_unbound11:25:190
2184978991815,2cyclictest2290269-21kworker/u8:1+flush-259:009:10:000
2184978991814,3cyclictest2508167-21kworker/u8:0+events_unbound12:30:250
2184978991814,3cyclictest2466158-21kworker/u8:0+flush-259:011:55:160
2184978991814,3cyclictest2450745-21kworker/u8:3+events_unbound11:35:170
2184989991715,1cyclictest2508167-21kworker/u8:0+flush-259:012:25:213
2184989991713,3cyclictest2450745-21kworker/u8:3+events_unbound11:30:213
2184986991714,2cyclictest2336987-21irqcore09:35:202
2184986991711,4cyclictest2284811-21cut08:45:152
2184986991710,5cyclictest2449901-21ntpq11:20:282
2184982991714,2cyclictest2357677-21cron09:55:001
2184982991714,2cyclictest2305474-21kworker/u8:0+events_unbound10:34:171
2184982991714,2cyclictest2305474-21kworker/u8:0+events_unbound10:00:571
2184982991714,2cyclictest2258113-21kworker/u8:0+events_unbound08:23:021
2184982991714,2cyclictest2216344-21kworker/u8:3+events_unbound07:45:431
2184982991711,4cyclictest3754036-21systemd-journal12:00:331
2184978991714,2cyclictest2378920-21kworker/u8:4+events_unbound10:35:170
2184978991713,3cyclictest2450745-21kworker/u8:3+events_unbound11:30:170
2184978991713,3cyclictest2315799-21kworker/u8:2+events_unbound09:25:280
2184978991713,2cyclictest2419551-21kworker/u8:2+events_unbound11:00:250
2184978991713,2cyclictest0-21swapper/009:15:000
2184978991713,2cyclictest0-21swapper/009:15:000
2184978991712,3cyclictest2216604-21/usr/sbin/munin07:40:160
2184989991614,1cyclictest2315799-21kworker/u8:2+events_unbound09:20:153
2184989991613,2cyclictest4000128-21kworker/u8:2+flush-259:008:00:003
2184989991613,2cyclictest2440478-21kworker/u8:2+events_unbound11:35:273
2184989991613,2cyclictest0-21swapper/308:25:013
2184989991612,3cyclictest2487186-21kworker/u8:1+flush-259:012:15:303
2184989991610,4cyclictest2440443-21latency_hist11:15:003
2184989991610,4cyclictest2184902-21latency_hist07:10:013
218498999160,14cyclictest1-21systemd11:30:013
218498699164,6cyclictest2503010-21cron12:15:002
2184986991611,3cyclictest2237488-21base6408:00:012
2184986991610,4cyclictest2425281-21/usr/sbin/munin11:00:272
2184978991614,1cyclictest60650irq/133-enp0s29f1:rx-008:50:180
2184978991614,1cyclictest2305474-21kworker/u8:0+flush-259:010:25:290
2184978991614,1cyclictest2200437-21awk07:25:000
2184978991613,2cyclictest60650irq/133-enp0s29f1:rx-012:15:270
2184978991613,2cyclictest2420081-21kworker/u8:4+flush-259:010:55:200
2184978991613,2cyclictest2394179-21kworker/u8:1+events_unbound11:05:210
2184978991613,2cyclictest2290269-21kworker/u8:1+flush-259:009:00:200
2184978991613,2cyclictest2180445-21kworker/u8:0+flush-259:007:30:200
2184989991512,2cyclictest2466158-21kworker/u8:0+flush-259:011:55:273
2184989991512,2cyclictest2415426-21irqcore10:50:223
2184989991512,2cyclictest2305474-21kworker/u8:0+flush-259:009:05:213
2184989991511,3cyclictest3000063-21rs:main3
2184989991511,3cyclictest2357694-21apt-get09:55:003
2184989991510,4cyclictest2497895-21latency_hist12:10:013
2184989991510,4cyclictest2352148-21latency_hist09:50:013
218498699159,4cyclictest2368654-21/usr/sbin/munin10:05:252
2184986991512,2cyclictest2399942-21irqcore10:35:202
2184986991512,2cyclictest2248184-21munin-plugin-st08:10:012
2184986991511,3cyclictest0-21swapper/208:25:002
218498699150,7cyclictest2200447-21apt07:25:002
2184978991513,1cyclictest60650irq/133-enp0s29f1:rx-010:40:270
2184978991513,1cyclictest60650irq/133-enp0s29f1:rx-010:30:160
2184978991513,1cyclictest60650irq/133-enp0s29f1:rx-010:15:240
2184978991513,1cyclictest60650irq/133-enp0s29f1:rx-007:50:350
2184978991513,1cyclictest60650irq/133-enp0s29f1:rx-007:28:370
2184978991512,2cyclictest60650irq/133-enp0s29f1:rx-012:05:250
2184978991511,3cyclictest2290269-21kworker/u8:1+flush-259:009:15:050
2184989991411,2cyclictest4000128-21kworker/u8:2+events_unbound07:40:193
2184989991411,2cyclictest2357363-21kworker/u8:2+events_unbound10:15:243
218498699148,4cyclictest3754036-21systemd-journal10:51:252
2184986991411,2cyclictest503-21systemd-logind10:50:012
2184986991411,2cyclictest2311305-21irqcore09:10:182
2184986991411,2cyclictest2311305-21irqcore09:10:182
2184986991410,3cyclictest2450745-21kworker/u8:3+flush-259:012:15:262
218498699140,5cyclictest2482072-21awk11:55:012
2184978991412,1cyclictest60650irq/133-enp0s29f1:rx-012:21:290
2184978991412,1cyclictest60650irq/133-enp0s29f1:rx-011:50:250
2184978991412,1cyclictest60650irq/133-enp0s29f1:rx-010:05:170
2184978991412,1cyclictest60650irq/133-enp0s29f1:rx-008:10:140
2184978991412,1cyclictest60650irq/133-enp0s29f1:rx-007:55:160
2184978991411,2cyclictest60650irq/133-enp0s29f1:rx-010:10:200
2184978991411,2cyclictest2315799-21kworker/u8:2+flush-259:009:20:200
218498999139,3cyclictest2237588-21base6408:00:023
218498999139,3cyclictest2180445-21kworker/u8:0+events_unbound07:25:013
2184989991311,1cyclictest0-21swapper/312:05:003
2184989991310,2cyclictest2394179-21kworker/u8:1+flush-259:011:10:003
2184989991310,2cyclictest0-21swapper/310:44:593
2184989991310,2cyclictest0-21swapper/308:15:213
218498699139,3cyclictest2186229-21kthreadcore07:10:232
218498699138,4cyclictest4000128-21kworker/u8:2+flush-259:008:28:052
218498699137,4cyclictest3754036-21systemd-journal09:45:342
218498699136,5cyclictest3754036-21systemd-journal11:30:012
2184986991310,2cyclictest2393711-21unixbench_singl10:25:302
2184978991312,1cyclictest60650irq/133-enp0s29f1:rx-007:15:260
2184978991311,1cyclictest60650irq/133-enp0s29f1:rx-012:10:220
2184978991310,2cyclictest60650irq/133-enp0s29f1:rx-007:48:440
218498999129,2cyclictest4000128-21kworker/u8:2+events_unbound08:10:243
218498999129,2cyclictest2420081-21kworker/u8:4+events_unbound10:55:223
218498999129,2cyclictest2310950-21cpuspeed_turbos09:10:153
218498999129,2cyclictest2310950-21cpuspeed_turbos09:10:153
218498999129,2cyclictest0-21swapper/310:39:593
218498999127,4cyclictest2243111-21cat08:05:013
2184989991210,1cyclictest0-21swapper/308:50:003
218498699129,2cyclictest0-21swapper/209:20:002
218498699126,4cyclictest2394179-21kworker/u8:1+flush-259:011:15:312
218498699121,7cyclictest2461026-21munin-plugin-st11:35:002
2184986991210,1cyclictest0-21swapper/208:40:002
218498999119,1cyclictest0-21swapper/310:10:013
218498999118,2cyclictest2472422-21irqcore11:45:193
218498999118,2cyclictest2429713-21unixbench_singl11:00:333
218498999118,2cyclictest2295163-21cron08:55:003
218498999117,3cyclictest2394179-21kworker/u8:1+flush-259:011:20:043
218498699118,2cyclictest2513669-21cpuspeed_turbos12:25:182
218498699118,2cyclictest2394462-21cpuspeed_turbos10:30:132
218498699118,2cyclictest2363611-21cpuspeed_turbos10:00:192
218498699118,2cyclictest2253686-21cpuspeed_turbos08:15:182
218498699118,2cyclictest0-21swapper/207:35:232
218498699117,2cyclictest0-21swapper/209:30:312
218498699113,5cyclictest2383854-21latency_hist10:20:012
218498699111,8cyclictest3754036-21systemd-journal08:50:342
218498999107,2cyclictest2315799-21kworker/u8:2+flush-259:009:25:203
218498999106,3cyclictest3754036-21systemd-journal10:05:003
218498999106,3cyclictest2180445-21kworker/u8:0+flush-259:007:30:263
218498999100,2cyclictest2513309-21latency_hist12:25:013
218498699105,3cyclictest2430957-21sed11:05:232
218498699100,2cyclictest2237296-21cat08:00:012
218497899100,2cyclictest2518212-21sshd12:26:330
21849899997,1cyclictest0-21swapper/308:30:273
21849899995,3cyclictest2180445-21kworker/u8:0+events_unbound07:25:233
21849869996,2cyclictest2248523-21cpuspeed_turbos08:10:142
21849869995,3cyclictest2482324-21apt11:55:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional