You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 18:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Thu Nov 27, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3191468994437,4cyclictest3232542-21latency_hist07:50:011
3191472993734,2cyclictest3036182-21kworker/u8:0+flush-259:009:00:002
3191477993633,2cyclictest3486862-21kworker/u8:0+flush-259:012:00:003
3191468993529,4cyclictest3523234-21latency_hist12:30:021
3191472993229,2cyclictest3518320-21kworker/u8:3+events_unbound12:30:252
3191472993229,2cyclictest3403447-21kworker/u8:0+flush-259:010:35:002
3191472993228,3cyclictest3409177-21kworker/u8:4+flush-259:012:10:252
3191472993128,2cyclictest3036182-21kworker/u8:0+flush-259:008:45:272
3191472993125,4cyclictest3227387-21latency_hist07:45:002
3191468993125,4cyclictest3476542-21latency_hist11:45:001
3191477993026,3cyclictest3036182-21kworker/u8:0+events_unbound08:10:013
3191477993025,3cyclictest3399046-21/usr/sbin/munin10:30:143
3191472993027,2cyclictest3409177-21kworker/u8:4+flush-259:011:00:162
3191472993027,2cyclictest3367322-21kworker/u8:1+events_unbound10:05:182
3191468993027,2cyclictest3378160-21kworker/u8:3+events_unbound10:20:131
3191477992926,2cyclictest3310307-21kworker/u8:4+flush-259:009:34:593
3191477992923,4cyclictest3191369-21latency_hist07:10:013
3191472992921,2cyclictest3466486-21kworker/u8:5+events_unbound11:35:152
3191472992825,2cyclictest3237707-21kworker/u8:1+flush-259:008:20:172
3191477992724,2cyclictest3310304-21kworker/u8:2+flush-259:009:04:593
3191477992721,4cyclictest3372975-21latency_hist10:05:013
3191468992723,3cyclictest3383540-21/usr/sbin/munin10:15:161
3191461992721,4cyclictest3466223-21latency_hist11:35:000
3191477992623,2cyclictest3378160-21kworker/u8:3+flush-259:010:30:013
3191477992620,5cyclictest3450786-21latency_hist11:20:003
3191472992623,2cyclictest3486862-21kworker/u8:0+events_unbound11:55:202
3191472992623,2cyclictest3424375-21kworker/u8:2+events_unbound11:05:192
3191472992623,2cyclictest3036182-21kworker/u8:0+flush-259:009:10:252
3191472992622,3cyclictest3528599-21kworker/u8:4+events_unbound12:35:172
3191468992622,3cyclictest3196528-21kworker/u8:3+flush-259:007:25:121
3191468992619,5cyclictest3330370-21ntpq09:20:211
3191461992620,4cyclictest3336688-21/usr/sbin/munin09:30:250
3191461992619,5cyclictest3243090-21cat08:00:010
3191477992520,4cyclictest3429228-21ntpq10:55:203
3191472992522,2cyclictest3036182-21kworker/u8:0+events_unbound07:50:192
319146899252,21cyclictest0-21swapper/112:20:001
3191468992521,3cyclictest3310307-21kworker/u8:4+events_unbound09:45:011
3191468992521,3cyclictest3310307-21kworker/u8:4+events_unbound09:45:011
3191461992522,2cyclictest3036182-21kworker/u8:0+flush-259:008:05:230
3191477992420,3cyclictest3274431-21/usr/sbin/munin08:30:203
3191472992421,2cyclictest3036182-21kworker/u8:0+events_unbound08:30:152
3191472992420,3cyclictest3409176-21kworker/u8:1+events_unbound11:10:172
3191472992420,3cyclictest3237707-21kworker/u8:1+events_unbound09:32:352
3191472992420,3cyclictest3036182-21kworker/u8:0+events_unbound09:05:102
3191468992420,3cyclictest3456185-21apt-get11:25:151
3191468992417,5cyclictest3196511-21latency_hist07:15:001
3191461992418,4cyclictest3279379-21kworker/u8:2+flush-259:008:45:000
3191477992320,2cyclictest3310307-21kworker/u8:4+flush-259:009:40:003
3191477992320,2cyclictest3237707-21kworker/u8:1+flush-259:008:50:153
3191477992318,3cyclictest3320607-21latency_hist09:15:013
3191472992320,2cyclictest3237707-21kworker/u8:1+events_unbound08:50:232
3191472992319,3cyclictest3409177-21kworker/u8:4+flush-259:011:20:212
3191472992319,3cyclictest3237707-21kworker/u8:1+flush-259:009:40:132
3191472992319,3cyclictest3237707-21kworker/u8:1+flush-259:009:40:122
3191472992319,3cyclictest3180808-21kworker/u8:2+flush-259:007:20:252
3191468992319,3cyclictest3367322-21kworker/u8:1+flush-259:010:00:181
3191468992319,3cyclictest3196528-21kworker/u8:3+flush-259:007:30:251
3191461992318,3cyclictest3409176-21kworker/u8:1+flush-259:011:00:010
3191477992218,2cyclictest3451022-21/usr/sbin/munin11:20:243
3191477992216,4cyclictest3232791-21/usr/sbin/munin07:50:183
3191472992219,2cyclictest3476773-21kworker/u8:1+events_unbound12:05:232
3191472992219,2cyclictest3253775-21kworker/u8:4+events_unbound08:15:162
3191472992219,2cyclictest3180808-21kworker/u8:2+flush-259:007:10:242
3191472992218,3cyclictest3036182-21kworker/u8:0+flush-259:009:00:092
3191472992218,3cyclictest3036182-21kworker/u8:0+events_unbound08:09:342
3191472992215,5cyclictest3476969-21diskstats11:45:152
3191468992219,2cyclictest3378160-21kworker/u8:3+flush-259:010:40:011
3191461992216,5cyclictest2698871-21systemd-journal08:35:000
3191477992118,2cyclictest3237707-21kworker/u8:1+flush-259:009:25:213
3191472992118,2cyclictest3466486-21kworker/u8:5+events_unbound11:41:102
3191472992118,2cyclictest3352121-21kworker/u8:3+events_unbound09:55:132
3191472992118,2cyclictest3237707-21kworker/u8:1+events_unbound09:20:132
3191472992118,2cyclictest3237707-21kworker/u8:1+events_unbound08:40:102
3191472992118,2cyclictest3196528-21kworker/u8:3+flush-259:008:00:212
3191472992117,3cyclictest3403447-21kworker/u8:0+events_unbound10:35:182
3191472992117,3cyclictest3196528-21kworker/u8:3+events_unbound07:45:122
3191472992117,3cyclictest3036182-21kworker/u8:0+events_unbound07:18:142
3191472992116,4cyclictest3341629-21kworker/u8:0+flush-259:010:10:212
3191472992114,5cyclictest3413400-21ntpq10:40:202
3191472992114,3cyclictest3507833-21kworker/u8:0+events_unbound12:15:182
3191468992118,2cyclictest3476773-21kworker/u8:1+events_unbound12:05:131
3191461992116,4cyclictest3372117-21ntpq10:00:220
3191461992115,4cyclictest2698871-21systemd-journal12:01:100
3191461992115,4cyclictest2698871-21systemd-journal12:01:100
3191461992114,5cyclictest3367556-21cp10:00:010
3191477992017,2cyclictest3507833-21kworker/u8:0+events_unbound12:15:223
3191477992017,2cyclictest3476773-21kworker/u8:1+flush-259:012:00:223
3191477992017,2cyclictest3476773-21kworker/u8:1+flush-259:012:00:223
3191477992017,2cyclictest3409177-21kworker/u8:4+events_unbound10:40:153
3191477992017,2cyclictest3403447-21kworker/u8:0+flush-259:010:35:163
3191477992016,3cyclictest3378160-21kworker/u8:3+flush-259:010:20:173
3191477992015,3cyclictest2698871-21systemd-journal07:50:013
3191472992017,2cyclictest3341629-21kworker/u8:0+events_unbound10:00:142
3191468992017,2cyclictest3036182-21kworker/u8:0+flush-259:008:35:231
3191468992016,3cyclictest3476773-21kworker/u8:1+events_unbound12:00:131
3191468992016,3cyclictest3476773-21kworker/u8:1+events_unbound12:00:131
3191468992016,3cyclictest3409177-21kworker/u8:4+events_unbound12:10:151
3191468992016,3cyclictest3409176-21kworker/u8:1+events_unbound11:15:191
3191468992016,3cyclictest3036182-21kworker/u8:0+flush-259:007:40:191
3191468992015,3cyclictest3274188-21sh08:30:011
3191461992017,2cyclictest3000063-21rs:main0
3191461992014,4cyclictest3345874-21ntpq09:35:180
3191461992013,5cyclictest3331288-21latency_hist09:25:000
3191477991917,2cyclictest3341629-21kworker/u8:0+flush-259:010:15:273
3191477991917,2cyclictest3036182-21kworker/u8:0+flush-259:007:35:233
3191477991916,2cyclictest3518320-21kworker/u8:3+events_unbound12:30:213
3191477991916,2cyclictest3456148-21kworker/u8:2+flush-259:011:50:153
3191477991916,2cyclictest3409177-21kworker/u8:4+events_unbound11:35:133
3191477991914,4cyclictest2698871-21systemd-journal10:15:003
3191472991916,2cyclictest3254229-21irqcore08:10:162
3191472991916,2cyclictest3237707-21kworker/u8:1+flush-259:009:25:192
3191472991916,2cyclictest3036182-21kworker/u8:0+events_unbound08:25:152
3191472991915,3cyclictest3409176-21kworker/u8:1+events_unbound11:28:362
3191472991915,3cyclictest3409176-21kworker/u8:1+events_unbound10:50:262
3191472991915,3cyclictest3237707-21kworker/u8:1+flush-259:008:35:152
3191472991915,3cyclictest3196528-21kworker/u8:3+events_unbound07:57:332
3191468991916,2cyclictest3486862-21kworker/u8:0+events_unbound11:55:001
3191468991916,2cyclictest3352121-21kworker/u8:3+flush-259:009:50:151
3191468991915,3cyclictest3243529-21apt-get08:00:101
3191468991915,3cyclictest3036182-21kworker/u8:0+events_unbound08:30:201
3191468991915,2cyclictest0-21swapper/108:00:011
3191468991914,4cyclictest2698871-21systemd-journal11:14:591
3191461991915,3cyclictest3036182-21kworker/u8:0+flush-259:007:30:260
3191461991913,4cyclictest2698871-21systemd-journal07:54:320
3191477991816,2cyclictest3237707-21kworker/u8:1+events_unbound09:45:433
3191477991815,2cyclictest3409177-21kworker/u8:4+flush-259:012:10:353
3191477991815,2cyclictest3409177-21kworker/u8:4+flush-259:011:05:243
3191477991815,2cyclictest3409176-21kworker/u8:1+events_unbound10:45:253
3191477991815,2cyclictest3341629-21kworker/u8:0+flush-259:009:40:263
3191477991815,2cyclictest3341629-21kworker/u8:0+flush-259:009:40:253
3191477991815,2cyclictest3036182-21kworker/u8:0+events_unbound07:15:193
3191477991814,3cyclictest3321036-21cat09:15:003
319147299189,6cyclictest2698871-21systemd-journal09:19:592
3191472991814,3cyclictest3409177-21kworker/u8:4+events_unbound10:46:362
3191472991811,5cyclictest3522338-21ntpq12:25:202
3191468991815,2cyclictest3409177-21kworker/u8:4+events_unbound10:55:161
3191468991815,2cyclictest3367322-21kworker/u8:1+events_unbound10:05:161
3191468991815,2cyclictest3036182-21kworker/u8:0+flush-259:007:35:171
3191468991814,3cyclictest3409177-21kworker/u8:4+flush-259:010:40:591
3191468991814,3cyclictest3180808-21kworker/u8:2+flush-259:007:20:201
3191468991814,3cyclictest3036182-21kworker/u8:0+flush-259:009:05:121
3191468991812,4cyclictest3465337-21ntpq11:30:211
3191461991816,1cyclictest3424375-21kworker/u8:2+flush-259:011:05:260
3191461991815,2cyclictest0-21swapper/012:25:000
3191461991815,2cyclictest0-21swapper/007:45:010
3191461991814,3cyclictest3237707-21kworker/u8:1+flush-259:008:55:150
3191461991813,4cyclictest3196528-21kworker/u8:3+flush-259:007:25:150
3191461991813,3cyclictest3393650-21latency_hist10:25:010
3191461991812,4cyclictest2698871-21systemd-journal08:00:130
3191477991714,2cyclictest3409177-21kworker/u8:4+events_unbound11:12:363
3191477991714,2cyclictest3409176-21kworker/u8:1+events_unbound10:50:143
3191477991713,3cyclictest3196528-21kworker/u8:3+flush-259:008:00:353
3191477991713,3cyclictest2698871-21systemd-journal07:30:003
3191477991712,4cyclictest3471623-21apt-get11:40:123
3191472991713,3cyclictest3352121-21kworker/u8:3+events_unbound09:54:342
3191472991711,4cyclictest3211943-21latency_hist07:30:002
3191468991714,2cyclictest3387385-21kworker/u8:1+events_unbound10:25:201
3191468991713,3cyclictest3409177-21kworker/u8:4+events_unbound11:20:261
3191468991713,3cyclictest3321267-21kworker/u8:2+flush-259:010:30:211
3191468991713,3cyclictest3295106-21apt-get08:50:141
3191461991714,2cyclictest60650irq/133-enp0s29f1:rx-010:45:240
3191461991713,3cyclictest3409176-21kworker/u8:1+flush-259:011:30:000
3191461991713,3cyclictest3196783-21apt-get07:15:130
3191461991713,3cyclictest3196528-21kworker/u8:3+flush-259:007:45:150
3191477991614,2cyclictest3237707-21kworker/u8:1+events_unbound08:39:353
3191477991613,2cyclictest3409177-21kworker/u8:4+events_unbound11:45:243
3191477991613,2cyclictest3367401-21kworker/u8:4+events_unbound10:06:343
3191477991613,2cyclictest3284941-21cpuspeed_turbos08:40:133
3191477991613,2cyclictest3259402-21irqcore08:15:173
3191477991613,2cyclictest3036182-21kworker/u8:0+events_unbound08:25:213
3191477991612,3cyclictest3237707-21kworker/u8:1+events_unbound08:20:233
3191477991612,3cyclictest3036182-21kworker/u8:0+events_unbound08:45:303
3191472991613,2cyclictest3456148-21kworker/u8:2+flush-259:011:50:182
3191472991613,2cyclictest3237707-21kworker/u8:1+events_unbound09:35:022
3191468991614,1cyclictest0-21swapper/107:20:001
3191468991613,2cyclictest3237707-21kworker/u8:1+events_unbound09:25:221
3191468991612,3cyclictest3341629-21kworker/u8:0+events_unbound10:10:211
3191468991612,3cyclictest3284803-21apt-get08:40:121
3191468991612,3cyclictest3248668-21apt-get08:05:101
3191468991611,3cyclictest2698871-21systemd-journal10:00:001
3191468991610,5cyclictest3269037-21latency_hist08:25:011
3191461991613,2cyclictest61450irq/134-enp0s29f1:tx-011:20:250
3191461991613,2cyclictest60650irq/133-enp0s29f1:rx-011:40:160
3191461991612,3cyclictest3321267-21kworker/u8:2+flush-259:010:20:000
3191477991512,2cyclictest3514233-21kthreadcore12:20:163
3191477991512,2cyclictest3341629-21kworker/u8:0+flush-259:009:55:203
3191477991512,2cyclictest3170983-21kworker/u8:1+events_unbound07:35:003
3191477991510,4cyclictest3310542-21apt-get09:05:093
3191472991512,2cyclictest3393644-21cron10:25:002
3191472991512,2cyclictest3367401-21kworker/u8:4+events_unbound10:15:022
3191472991511,3cyclictest3492337-21cpuspeed_turbos12:00:112
3191472991511,3cyclictest3492337-21cpuspeed_turbos12:00:112
3191472991511,3cyclictest3409176-21kworker/u8:1+flush-259:011:15:262
3191472991511,3cyclictest3310307-21kworker/u8:4+events_unbound09:45:372
319146899159,5cyclictest3309453-21ntpq09:00:201
3191468991512,2cyclictest3341629-21kworker/u8:0+events_unbound09:35:101
319146199158,5cyclictest3325286-21ntpq09:15:200
3191461991513,1cyclictest60650irq/133-enp0s29f1:rx-012:38:310
3191461991513,1cyclictest60650irq/133-enp0s29f1:rx-010:50:540
3191461991513,1cyclictest60650irq/133-enp0s29f1:rx-009:13:240
3191461991513,1cyclictest60650irq/133-enp0s29f1:rx-008:10:130
3191461991512,2cyclictest60650irq/133-enp0s29f1:rx-011:55:120
3191461991512,2cyclictest60650irq/133-enp0s29f1:rx-009:27:130
3191461991512,2cyclictest60650irq/133-enp0s29f1:rx-009:07:250
3191461991512,2cyclictest60650irq/133-enp0s29f1:rx-008:15:200
3191461991512,2cyclictest3486849-21turbostat11:55:000
3191461991512,2cyclictest3271595-21kthreadcore08:25:170
3191461991510,3cyclictest3253775-21kworker/u8:4+events_unbound08:20:160
3191477991411,2cyclictest3518320-21kworker/u8:3+flush-259:012:25:153
3191477991411,2cyclictest3341629-21kworker/u8:0+events_unbound09:50:423
3191477991411,2cyclictest3036182-21kworker/u8:0+flush-259:007:40:213
3191472991411,2cyclictest3476773-21kworker/u8:1+events_unbound12:20:222
3191468991412,1cyclictest3237707-21kworker/u8:1+events_unbound09:45:141
319146199148,5cyclictest3449909-21ntpq11:15:220
3191461991413,1cyclictest60650irq/133-enp0s29f1:rx-011:02:530
3191461991413,1cyclictest60650irq/133-enp0s29f1:rx-010:27:020
3191461991413,1cyclictest60650irq/133-enp0s29f1:rx-009:00:140
3191461991413,1cyclictest60650irq/133-enp0s29f1:rx-008:35:120
3191461991412,2cyclictest60650irq/133-enp0s29f1:rx-012:26:230
3191461991412,1cyclictest60650irq/133-enp0s29f1:rx-010:42:320
3191461991412,1cyclictest60650irq/133-enp0s29f1:rx-010:05:170
3191461991411,2cyclictest60650irq/133-enp0s29f1:rx-012:12:370
3191461991411,2cyclictest60650irq/133-enp0s29f1:rx-011:38:330
3191477991311,1cyclictest3196496-21cron07:15:003
3191477991310,2cyclictest3533063-21unixbench_multi12:35:243
3191477991310,2cyclictest3201978-21cpuspeed_turbos07:20:143
3191477991310,2cyclictest3036182-21kworker/u8:0+flush-259:007:55:173
319147299131,10cyclictest3222259-21awk07:40:002
319146899133,1cyclictest34677722kthreadcore11:35:171
3191468991311,2cyclictest3237707-21kworker/u8:1+flush-259:009:20:001
3191468991310,2cyclictest3310307-21kworker/u8:4+flush-259:009:30:261
3191468991310,2cyclictest3036182-21kworker/u8:0+flush-259:007:50:221
3191461991311,1cyclictest60650irq/133-enp0s29f1:rx-012:05:150
3191461991311,1cyclictest60650irq/133-enp0s29f1:rx-009:46:470
3191461991311,1cyclictest3404002-21turbostat10:35:000
3191461991310,2cyclictest60650irq/133-enp0s29f1:rx-012:15:240
3191461991310,2cyclictest60650irq/133-enp0s29f1:rx-011:45:170
3191461991310,1cyclictest0-21swapper/007:35:180
319147799129,1cyclictest0-21swapper/308:55:253
319147799128,3cyclictest3461263-21kworker/u8:3+flush-259:011:30:193
319147799120,4cyclictest33302592missed_timers09:20:203
319147299129,2cyclictest0-21swapper/211:00:012
319147299121,9cyclictest3398811-21basename10:30:002
319146899129,2cyclictest3279379-21kworker/u8:2+flush-259:008:49:391
319146899128,3cyclictest3486862-21kworker/u8:0+flush-259:011:55:251
319146899128,3cyclictest3409177-21kworker/u8:4+flush-259:011:06:391
319146899128,3cyclictest3409177-21kworker/u8:4+events_unbound11:45:221
319146899128,3cyclictest3036182-21kworker/u8:0+events_unbound09:10:231
319146199129,2cyclictest60650irq/133-enp0s29f1:rx-011:10:220
319147799116,3cyclictest3409177-21kworker/u8:4+flush-259:011:25:153
319146899118,2cyclictest3497144-21kworker/u8:2+events_unbound12:20:161
319146199118,2cyclictest60650irq/133-enp0s29f1:rx-012:31:480
3191461991110,1cyclictest60650irq/133-enp0s29f1:rx-007:20:240
319146199110,4cyclictest3362251-21dump-pmu-power09:55:000
319146899108,2cyclictest3237707-21kworker/u8:1+events_unbound08:55:151
319146199108,1cyclictest60650irq/133-enp0s29f1:rx-008:45:140
319146199107,2cyclictest60650irq/133-enp0s29f1:rx-010:35:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional