You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 18:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Thu Dec 04, 2025 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
368410994946,2cyclictest75-21kswapd011:17:581
368406994643,2cyclictest75-21kswapd011:17:580
368410994139,1cyclictest466354-21fwupd08:40:231
368406994138,2cyclictest409641-21kworker/u8:3+flush-259:008:35:000
368406993934,3cyclictest576380-21gpgv10:30:010
368422993734,2cyclictest409641-21kworker/u8:3+flush-259:009:25:173
368415993733,3cyclictest409641-21kworker/u8:3+flush-259:008:10:002
368422993633,2cyclictest75-21kswapd011:18:003
368422993532,2cyclictest602707-21kworker/u8:2+flush-259:011:40:003
368422993330,2cyclictest654606-21kworker/u8:1+flush-259:011:55:173
368422993330,2cyclictest440988-21kworker/u8:4+flush-259:008:50:253
368422993329,3cyclictest550580-21kworker/u8:4+flush-259:010:45:213
368410993329,3cyclictest550580-21kworker/u8:4+flush-259:010:50:161
368410993021,6cyclictest2698871-21systemd-journal09:25:011
368422992925,3cyclictest319678-21kworker/u8:2+flush-259:007:15:163
368415992923,4cyclictest467506-21/usr/sbin/munin08:45:142
368422992824,3cyclictest685721-21/usr/sbin/munin12:15:143
368415992822,4cyclictest582128-21/usr/sbin/munin10:35:162
368415992822,4cyclictest2698871-21systemd-journal07:35:002
368422992623,2cyclictest602707-21kworker/u8:2+events_unbound11:50:243
368410992622,2cyclictest3000061-21in:imuxsock12:40:001
368422992522,2cyclictest623285-21kworker/u8:1+events_unbound11:25:183
368422992522,2cyclictest436017-21kworker/u8:1+flush-259:008:25:223
368422992518,2cyclictest409641-21kworker/u8:3+events_unbound08:30:173
368410992521,3cyclictest535095-21kworker/u8:1+flush-259:010:00:231
368410992521,3cyclictest529710-21kworker/u8:3+events_unbound09:55:171
368406992519,4cyclictest3000063-21rs:main0
368406992518,5cyclictest653700-21ntpq11:40:210
368406992518,5cyclictest508152-21ntpq09:20:200
368422992421,2cyclictest409641-21kworker/u8:3+events_unbound08:05:193
368422992419,4cyclictest435791-21latency_hist08:15:003
368415992421,2cyclictest602707-21kworker/u8:2+flush-259:012:35:162
368415992420,3cyclictest685504-21kworker/u8:4+flush-259:012:30:262
368415992420,3cyclictest488655-21kworker/u8:0+events_unbound09:20:142
368415992419,4cyclictest3000063-21rs:main2
368410992419,3cyclictest436017-21kworker/u8:1+flush-259:008:25:241
368422992320,2cyclictest358080-21kworker/u8:1+flush-259:007:45:163
368422992319,3cyclictest575743-21kworker/u8:0+events_unbound10:35:203
368415992320,2cyclictest670070-21kworker/u8:3+flush-259:012:00:212
368415992319,3cyclictest409641-21kworker/u8:3+events_unbound09:05:252
368415992319,3cyclictest358080-21kworker/u8:1+flush-259:007:20:152
368415992317,4cyclictest2698871-21systemd-journal08:40:232
368410992319,3cyclictest409641-21kworker/u8:3+flush-259:008:10:251
368410992316,5cyclictest2698871-21systemd-journal08:50:271
368406992321,1cyclictest430533-21kworker/0:1+pm08:40:240
368422992219,2cyclictest602707-21kworker/u8:2+events_unbound11:10:153
368422992219,2cyclictest529710-21kworker/u8:3+flush-259:010:25:253
368422992218,3cyclictest436017-21kworker/u8:1+events_unbound08:20:183
368422992218,3cyclictest436017-21kworker/u8:1+events_unbound08:20:183
368422992218,2cyclictest388963-21cron07:30:003
368415992219,2cyclictest503899-21grep09:20:002
368415992218,3cyclictest602707-21kworker/u8:2+flush-259:011:00:242
368410992217,4cyclictest413897-21ntpq07:50:211
368422992118,2cyclictest529710-21kworker/u8:3+events_unbound10:30:223
368422992118,2cyclictest319678-21kworker/u8:2+flush-259:008:00:183
368422992117,3cyclictest436017-21kworker/u8:1+flush-259:008:15:253
368415992118,2cyclictest358080-21kworker/u8:1+flush-259:007:45:132
368415992117,2cyclictest550580-21kworker/u8:4+events_unbound10:40:162
368410992117,3cyclictest319678-21kworker/u8:2+flush-259:008:00:321
368410992116,3cyclictest555816-21apt-get10:10:121
368410992113,3cyclictest675235-21awk12:05:001
368406992118,2cyclictest670070-21kworker/u8:3+flush-259:012:25:190
368406992116,3cyclictest575743-21kworker/u8:0+flush-259:011:00:170
368406992113,6cyclictest560727-21latency_hist10:15:000
368422992017,2cyclictest560961-21kworker/u8:2+events_unbound10:20:203
368422992017,2cyclictest409641-21kworker/u8:3+flush-259:008:45:193
368422992016,3cyclictest654606-21kworker/u8:1+events_unbound12:10:143
368422992016,3cyclictest602707-21kworker/u8:2+events_unbound11:30:253
368422992016,3cyclictest504123-21kworker/u8:1+flush-259:009:30:193
368422992016,2cyclictest2698871-21systemd-journal09:10:003
368422992015,2cyclictest409641-21kworker/u8:3+events_unbound08:40:253
368415992017,2cyclictest670070-21kworker/u8:3+events_unbound12:27:252
368415992017,2cyclictest623285-21kworker/u8:1+events_unbound11:20:172
368415992016,3cyclictest535095-21kworker/u8:1+flush-259:010:19:592
368415992015,4cyclictest441141-21cp08:20:012
368410992014,4cyclictest2698871-21systemd-journal09:38:261
368406992016,3cyclictest519608-21kworker/u8:2+flush-259:009:45:000
368406992015,4cyclictest576928-21/usr/sbin/munin10:30:230
368406992015,4cyclictest544337-21ntpq09:55:220
368406992014,4cyclictest690932-21apt-get12:20:110
368422991916,2cyclictest602707-21kworker/u8:2+events_unbound12:25:293
368422991916,2cyclictest529710-21kworker/u8:3+events_unbound10:15:163
368422991915,3cyclictest602707-21kworker/u8:2+events_unbound12:20:253
368422991915,3cyclictest535095-21kworker/u8:1+flush-259:010:00:213
368422991915,3cyclictest358080-21kworker/u8:1+events_unbound07:40:253
368422991912,5cyclictest524531-21latency_hist09:40:013
368415991916,2cyclictest644116-21latency_hist11:35:002
368415991916,2cyclictest602707-21kworker/u8:2+flush-259:012:25:002
368415991916,2cyclictest319678-21kworker/u8:2+events_unbound08:00:222
368415991914,4cyclictest2698871-21systemd-journal11:40:002
368406991914,3cyclictest384073-21/usr/sbin/munin07:25:150
368422991815,2cyclictest440988-21kworker/u8:4+events_unbound08:57:273
368422991815,2cyclictest381244-21kthreadcore07:20:203
368422991815,2cyclictest319678-21kworker/u8:2+events_unbound07:50:223
368422991814,3cyclictest602707-21kworker/u8:2+events_unbound11:05:013
368422991814,3cyclictest550580-21kworker/u8:4+flush-259:010:50:193
368422991814,3cyclictest319678-21kworker/u8:2+flush-259:007:10:233
368422991812,4cyclictest420080-21gpgv08:00:003
368415991815,2cyclictest319678-21kworker/u8:2+events_unbound07:35:152
368415991814,3cyclictest550580-21kworker/u8:4+events_unbound10:45:242
368415991814,3cyclictest0-21swapper/210:10:002
368415991813,4cyclictest680364-21latency_hist12:10:012
368415991813,3cyclictest602906-21diskstats10:55:172
36841599181,16cyclictest550369-21true10:04:592
368410991815,2cyclictest654583-21cron11:45:011
368410991814,3cyclictest0-21swapper/112:29:591
368406991812,5cyclictest2698871-21systemd-journal11:25:010
368422991714,2cyclictest680357-21idleruntime-cro12:09:593
368422991714,2cyclictest519608-21kworker/u8:2+flush-259:009:45:213
368422991713,3cyclictest685504-21kworker/u8:4+events_unbound12:30:253
368422991713,3cyclictest612801-21kworker/u8:3+events_unbound11:09:263
368422991712,3cyclictest2698871-21systemd-journal11:43:253
368415991714,2cyclictest650749-21kthreadcore11:40:192
368415991713,3cyclictest602707-21kworker/u8:2+flush-259:011:45:202
368415991713,3cyclictest519608-21kworker/u8:2+flush-259:009:40:222
36841099176,4cyclictest519388-21grep09:35:001
36841099171,15cyclictest0-21swapper/112:20:001
368410991711,4cyclictest670041-21latency_hist12:00:011
368410991711,4cyclictest612776-21latency_hist11:05:001
368406991714,2cyclictest60650irq/133-enp0s29f1:rx-007:30:230
368406991713,3cyclictest545425-21kworker/u8:2+flush-259:010:05:000
368406991710,5cyclictest513341-21ntpq09:25:230
36842299169,5cyclictest632728-21ntpq11:20:243
368422991612,3cyclictest654606-21kworker/u8:1+flush-259:012:05:003
368422991612,3cyclictest529710-21kworker/u8:3+events_unbound09:55:193
368422991611,3cyclictest529692-21cut09:45:003
368415991613,2cyclictest638785-21idleruntime-cro11:30:012
368415991613,2cyclictest519608-21kworker/u8:2+events_unbound09:45:242
368415991612,3cyclictest670070-21kworker/u8:3+events_unbound12:15:132
368415991612,3cyclictest409641-21kworker/u8:3+flush-259:008:35:192
368415991612,3cyclictest319678-21kworker/u8:2+flush-259:008:15:002
368415991611,4cyclictest415018-21/usr/sbin/munin07:55:142
368415991610,4cyclictest372634-21ntpq07:10:212
368410991612,3cyclictest575743-21kworker/u8:0+flush-259:010:50:001
368410991612,3cyclictest2698871-21systemd-journal08:39:551
368410991611,4cyclictest535104-21/usr/sbin/munin09:50:241
368410991610,4cyclictest404437-21latency_hist07:45:001
368406991614,1cyclictest60650irq/133-enp0s29f1:rx-008:45:220
368406991613,2cyclictest706538-21cron12:35:000
368406991613,2cyclictest60650irq/133-enp0s29f1:rx-010:50:500
368406991613,2cyclictest60650irq/133-enp0s29f1:rx-009:11:560
368406991613,2cyclictest60650irq/133-enp0s29f1:rx-008:35:170
368406991613,2cyclictest60650irq/133-enp0s29f1:rx-007:53:120
368406991613,2cyclictest60650irq/133-enp0s29f1:rx-007:12:510
368406991612,3cyclictest680630-21apt-get12:10:120
368406991612,3cyclictest319678-21kworker/u8:2+flush-259:007:20:010
36842299158,5cyclictest462051-21latency_hist08:40:003
36842299158,5cyclictest2698871-21systemd-journal12:40:003
368422991512,2cyclictest503-21systemd-logind10:10:013
368422991511,3cyclictest535095-21kworker/u8:1+events_unbound09:50:253
36841599159,5cyclictest617019-21ntpq11:05:212
368415991512,2cyclictest520665-21kthreadcore09:35:202
368415991512,2cyclictest358080-21kworker/u8:1+events_unbound07:40:212
368415991511,3cyclictest654606-21kworker/u8:1+flush-259:012:10:252
368415991511,3cyclictest535095-21kworker/u8:1+flush-259:010:23:552
368415991511,3cyclictest3000063-21rs:main2
36841099159,4cyclictest644566-21/usr/sbin/munin11:35:161
368410991511,3cyclictest440538-21unixbench_singl08:15:251
368410991511,3cyclictest358080-21kworker/u8:1+flush-259:007:25:011
368406991513,2cyclictest60650irq/133-enp0s29f1:rx-010:35:550
368406991513,1cyclictest643911-21cut11:35:000
368406991513,1cyclictest60650irq/133-enp0s29f1:rx-008:55:200
368406991513,1cyclictest60650irq/133-enp0s29f1:rx-007:20:250
368406991512,2cyclictest60650irq/133-enp0s29f1:rx-010:24:310
368406991510,4cyclictest706822-21apt-get12:35:120
368422991411,2cyclictest488655-21kworker/u8:0+flush-259:009:10:203
368422991411,1cyclictest0-21swapper/307:35:133
368422991410,3cyclictest482847-21kworker/u8:5+events_unbound09:00:293
368422991410,3cyclictest409641-21kworker/u8:3+flush-259:009:15:213
368415991412,1cyclictest0-21swapper/209:05:002
368415991411,2cyclictest504123-21kworker/u8:1+flush-259:009:35:002
368415991410,3cyclictest575743-21kworker/u8:0+flush-259:010:51:252
368415991410,3cyclictest440988-21kworker/u8:4+events_unbound08:55:172
36841099148,5cyclictest2698871-21systemd-journal10:35:011
36841099148,4cyclictest451727-21kworker/u8:2+flush-259:009:00:001
368410991410,3cyclictest654606-21kworker/u8:1+flush-259:011:53:311
368410991410,3cyclictest358080-21kworker/u8:1+flush-259:007:35:001
368406991413,1cyclictest61450irq/134-enp0s29f1:tx-009:50:180
368406991413,1cyclictest60650irq/133-enp0s29f1:rx-011:07:250
368406991413,1cyclictest60650irq/133-enp0s29f1:rx-009:01:350
368406991412,2cyclictest60650irq/133-enp0s29f1:rx-010:15:450
368406991412,2cyclictest60650irq/133-enp0s29f1:rx-009:35:410
368406991412,2cyclictest60650irq/133-enp0s29f1:rx-008:07:260
368406991412,2cyclictest60650irq/133-enp0s29f1:rx-007:41:370
368406991412,1cyclictest60650irq/133-enp0s29f1:rx-012:00:240
368406991412,1cyclictest60650irq/133-enp0s29f1:rx-011:50:230
368406991411,2cyclictest60650irq/133-enp0s29f1:rx-011:26:230
368406991411,2cyclictest60650irq/133-enp0s29f1:rx-010:44:150
368406991411,2cyclictest419883-21journal-offline08:00:000
368406991410,3cyclictest408759-21ntp_kernel_pll_07:45:200
36841599139,3cyclictest654606-21kworker/u8:1+events_unbound11:55:132
36841599139,3cyclictest575743-21kworker/u8:0+events_unbound11:11:262
36841599139,3cyclictest529710-21kworker/u8:3+flush-259:010:25:202
368415991311,2cyclictest319678-21kworker/u8:2+events_unbound07:15:132
36841099139,3cyclictest654606-21kworker/u8:1+flush-259:012:10:251
36841099137,4cyclictest2698871-21systemd-journal09:48:251
36841099136,5cyclictest2698871-21systemd-journal09:15:001
368410991311,1cyclictest0-21swapper/110:40:001
368410991310,2cyclictest602707-21kworker/u8:2+flush-259:011:31:361
368410991310,2cyclictest394468-21cut07:35:131
368406991312,1cyclictest60650irq/133-enp0s29f1:rx-012:05:120
368406991312,1cyclictest60650irq/133-enp0s29f1:rx-010:05:240
368406991312,1cyclictest60650irq/133-enp0s29f1:rx-008:52:570
368406991311,2cyclictest60650irq/133-enp0s29f1:rx-010:55:280
368406991311,2cyclictest60650irq/133-enp0s29f1:rx-008:02:400
368406991311,1cyclictest60650irq/133-enp0s29f1:rx-009:15:570
368406991310,2cyclictest60650irq/133-enp0s29f1:rx-010:48:090
368406991310,2cyclictest60650irq/133-enp0s29f1:rx-008:10:160
36841599129,3cyclictest602707-21kworker/u8:2+flush-259:011:50:192
36841599129,2cyclictest409641-21kworker/u8:3+flush-259:008:25:002
36841599129,2cyclictest409641-21kworker/u8:3+flush-259:008:25:002
36841599129,1cyclictest0-21swapper/207:50:232
36841599128,3cyclictest535095-21kworker/u8:1+events_unbound09:55:132
36841099128,3cyclictest535095-21kworker/u8:1+events_unbound10:10:001
36841099128,3cyclictest504123-21kworker/u8:1+flush-259:009:25:261
36841099128,3cyclictest358080-21kworker/u8:1+flush-259:007:25:261
36840699129,2cyclictest60650irq/133-enp0s29f1:rx-011:55:190
36840699129,2cyclictest440546-21unixbench_singl08:15:250
368406991210,1cyclictest60650irq/133-enp0s29f1:rx-011:10:230
36841099118,2cyclictest529193-21unixbench_multi09:40:241
36841099118,2cyclictest0-21swapper/111:45:141
36841099118,1cyclictest0-21swapper/108:35:001
36841099117,3cyclictest319678-21kworker/u8:2+flush-259:007:55:251
36841099117,2cyclictest0-21swapper/109:05:001
36841099116,4cyclictest535095-21kworker/u8:1+flush-259:010:24:301
36841099114,5cyclictest565933-21latency_hist10:20:011
36841099111,3cyclictest592176-21latency_hist10:45:011
36841099110,5cyclictest378675-21munin-run07:20:011
36840699119,1cyclictest60650irq/133-enp0s29f1:rx-011:45:290
36840699118,2cyclictest446914-21cpuspeed_turbos08:25:120
36840699117,3cyclictest441618-21munin-node08:20:120
36840699117,3cyclictest441618-21munin-node08:20:120
368406991110,1cyclictest61450irq/134-enp0s29f1:tx-011:35:180
36841599108,1cyclictest0-21swapper/209:10:142
36841099108,1cyclictest0-21swapper/110:25:191
36841099101,5cyclictest503900-21munin-plugin-st09:20:001
36840699108,1cyclictest60650irq/133-enp0s29f1:rx-012:15:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional