You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 16:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Dec 01, 2025 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3352750996863,4cyclictest3513461-21dpkg09:42:380
3352757995753,3cyclictest75-21kswapd009:42:332
3352754995249,2cyclictest75-21kswapd009:42:381
3352762994945,3cyclictest75-21kswapd009:42:373
3352757994631,10cyclictest3529479-21latency_hist10:00:012
3352750993832,4cyclictest3622953-21latency_hist11:30:010
3352757993732,4cyclictest3424801-21latency_hist08:20:012
3352754993734,2cyclictest3664830-21kworker/u8:4+flush-259:012:20:011
3352754993631,4cyclictest3441303-21kworker/u8:2+flush-259:008:35:211
3352754993629,5cyclictest2698871-21systemd-journal08:25:001
3352754993531,3cyclictest3690628-21kworker/u8:4+flush-259:012:35:171
3352762993427,5cyclictest2698871-21systemd-journal11:25:003
3352757993329,3cyclictest3675177-21kworker/u8:1+flush-259:012:40:012
3352754993228,3cyclictest3570936-21kworker/u8:1+flush-259:010:55:171
3352750993027,2cyclictest3425358-21gpgv08:20:020
3352750992926,2cyclictest3535348-21kworker/u8:4+flush-259:010:50:170
3352750992923,4cyclictest3419676-21latency_hist08:15:010
3352750992821,5cyclictest3571320-21latency_hist10:40:010
3352750992821,5cyclictest3000063-21rs:main0
3352757992721,4cyclictest3550224-21kworker/u8:0+flush-259:010:35:192
3352754992723,2cyclictest1-21systemd11:45:001
3352754992723,2cyclictest1-21systemd11:45:001
3352754992722,3cyclictest3430075-21kworker/u8:1+events_unbound08:55:131
3352762992521,3cyclictest3393980-21kworker/u8:2+flush-259:008:05:173
3352757992521,3cyclictest3570936-21kworker/u8:1+flush-259:011:45:172
3352754992521,3cyclictest3451386-21kworker/u8:0+events_unbound09:15:181
3352754992520,3cyclictest3664830-21kworker/u8:4+events_unbound12:10:151
3352754992520,3cyclictest3535313-21kworker/u8:1+flush-259:010:05:201
3352750992522,2cyclictest3373242-21kworker/u8:4+flush-259:008:05:000
3352762992420,3cyclictest3670049-21/usr/sbin/munin12:15:213
3352757992419,3cyclictest3654136-21apt-get12:00:102
3352750992421,2cyclictest3535313-21kworker/u8:1+flush-259:010:10:170
3352750992420,3cyclictest3451380-21gpgv08:44:590
3352750992418,4cyclictest3000063-21rs:main0
3352757992316,6cyclictest3638467-21cron11:45:002
3352757992316,6cyclictest3638467-21cron11:44:592
3352754992319,3cyclictest3430075-21kworker/u8:1+events_unbound08:25:151
3352754992319,3cyclictest3393980-21kworker/u8:2+flush-259:007:55:181
3352754992318,3cyclictest3570936-21kworker/u8:1+flush-259:011:00:181
3352750992318,3cyclictest3597515-21diskstats11:05:130
335275099230,20cyclictest3000063-21rs:main0
3352762992218,3cyclictest3441303-21kworker/u8:2+flush-259:008:40:163
3352762992217,4cyclictest3628390-21apt-get11:35:113
3352757992218,3cyclictest3197135-21kworker/u8:0+flush-259:007:15:002
3352754992219,2cyclictest3685243-21cron12:29:591
3352754992218,3cyclictest3570936-21kworker/u8:1+flush-259:011:10:001
3352750992216,5cyclictest2698871-21systemd-journal08:30:000
335275799219,11cyclictest3392951-21needreboot07:45:212
335275799213,16cyclictest3404246-21munin-run08:00:002
3352754992116,3cyclictest3424812-21cut08:20:011
3352754992114,5cyclictest3357842-21latency_hist07:15:001
335275499210,18cyclictest837-21snmpd11:55:161
3352750992117,2cyclictest3498144-21kworker/u8:3+flush-259:010:05:210
3352754992017,2cyclictest3393980-21kworker/u8:2+flush-259:008:05:211
3352754992016,3cyclictest3373242-21kworker/u8:4+events_unbound08:10:151
3352754992013,5cyclictest3664595-21latency_hist12:10:011
3352754992013,5cyclictest3612835-21/usr/sbin/munin11:20:141
3352750992015,4cyclictest3493205-21apt-get09:25:120
3352750992015,4cyclictest3493205-21apt-get09:25:110
335275799198,10cyclictest0-21swapper/211:55:182
335275799197,4cyclictest3675177-21kworker/u8:1+flush-259:012:26:152
3352757991914,4cyclictest3393980-21kworker/u8:2+flush-259:007:55:002
3352754991915,3cyclictest3545740-21apt-get10:15:111
3352754991915,3cyclictest3462283-21kworker/u8:3+events_unbound09:10:121
3352750991914,4cyclictest2698871-21systemd-journal09:40:010
335275799187,10cyclictest3545463-21idleruntime-cro10:15:012
335275799186,3cyclictest3434911-21sed08:25:202
335275799181,5cyclictest3607418-21apt11:14:592
3352757991815,2cyclictest3570936-21kworker/u8:1+flush-259:011:25:152
3352757991811,5cyclictest3601897-21journal-offline11:05:392
335275799180,5cyclictest3628129-21turbostat.cron11:35:002
3352754991815,2cyclictest3368131-21idleruntime-cro07:25:001
3352754991814,3cyclictest3535313-21kworker/u8:1+flush-259:010:20:251
3352754991814,3cyclictest3451386-21kworker/u8:0+events_unbound09:05:171
3352754991814,3cyclictest3373242-21kworker/u8:4+flush-259:007:40:151
3352754991812,4cyclictest3498144-21kworker/u8:3+events_unbound09:35:181
3352750991815,2cyclictest3560952-21idleruntime-cro10:30:000
3352750991815,2cyclictest3550613-21sh10:20:010
3352750991811,5cyclictest2698871-21systemd-journal07:35:000
3352762991715,1cyclictest0-21swapper/312:05:013
335275799179,7cyclictest0-21swapper/209:30:132
335275799176,2cyclictest36648412sleep212:10:132
335275799175,3cyclictest3561138-21cat10:30:012
335275799174,2cyclictest3487839-21idleruntime-cro09:19:592
335275799171,5cyclictest3647792-21tail11:50:242
3352757991713,3cyclictest3310321-21kworker/u8:3+flush-259:007:25:182
3352757991712,4cyclictest3363250-21apt-get07:20:112
3352757991711,4cyclictest3675184-21/usr/sbin/munin12:20:142
3352754991714,2cyclictest3664830-21kworker/u8:4+flush-259:012:20:231
3352754991714,2cyclictest3451386-21kworker/u8:0+events_unbound09:45:191
3352754991713,3cyclictest3570936-21kworker/u8:1+flush-259:011:30:211
3352754991712,4cyclictest3535348-21kworker/u8:4+flush-259:010:15:001
335275499170,11cyclictest3409377-21munin-run08:05:001
3352750991714,2cyclictest60650irq/133-enp0s29f1:rx-010:30:270
3352750991714,2cyclictest60650irq/133-enp0s29f1:rx-008:31:050
3352762991613,2cyclictest3643617-21kworker/u8:2+flush-259:012:30:163
335275799169,5cyclictest2698871-21systemd-journal10:05:002
335275799166,3cyclictest401ktimers/208:30:132
335275799165,1cyclictest0-21swapper/209:55:002
335275799161,6cyclictest3497228-21ntpq09:25:202
335275799161,6cyclictest3497228-21ntpq09:25:202
3352757991612,3cyclictest2698871-21systemd-journal10:55:002
3352757991612,2cyclictest0-21swapper/208:39:592
3352757991610,4cyclictest3425458-21/usr/sbin/munin08:20:152
335275799160,5cyclictest3591899-21idleruntime-cro11:00:002
3352754991614,1cyclictest0-21swapper/108:35:011
3352754991613,2cyclictest3521651-21kthreadcore09:50:181
3352754991613,2cyclictest3430075-21kworker/u8:1+flush-259:009:20:141
3352754991612,3cyclictest3622975-21kworker/u8:0+events_unbound11:39:591
3352754991612,3cyclictest3570936-21kworker/u8:1+events_unbound11:25:161
3352754991612,3cyclictest3550842-21kworker/u8:2+flush-259:010:30:251
3352754991612,3cyclictest3503251-21cron09:35:001
3352754991611,4cyclictest3685526-21apt-get12:30:131
3352754991611,3cyclictest3607401-21cron11:14:591
3352754991610,4cyclictest3378521-21latency_hist07:35:011
335275099169,5cyclictest3372368-21ntpq07:25:220
335275099169,5cyclictest2698871-21systemd-journal11:35:000
3352750991614,1cyclictest61450irq/134-enp0s29f1:tx-008:55:180
3352750991614,1cyclictest60650irq/133-enp0s29f1:rx-009:00:160
3352750991614,1cyclictest60650irq/133-enp0s29f1:rx-008:15:120
3352750991613,2cyclictest60650irq/133-enp0s29f1:rx-007:59:540
3352750991613,2cyclictest3664830-21kworker/u8:4+flush-259:012:10:150
3352762991511,3cyclictest3514139-21apt-get09:45:143
335275799158,5cyclictest3597086-21latency_hist11:05:002
3352757991511,3cyclictest3363004-21kworker/u8:1+flush-259:007:35:002
3352754991512,1cyclictest0-21swapper/110:00:011
3352754991511,3cyclictest3535348-21kworker/u8:4+flush-259:010:28:331
3352754991511,3cyclictest3471483-21ntp_kernel_pll_09:00:211
3352754991511,3cyclictest3383656-21kworker/u8:0+flush-259:007:45:251
3352750991513,1cyclictest60650irq/133-enp0s29f1:rx-012:30:250
3352750991513,1cyclictest60650irq/133-enp0s29f1:rx-012:08:230
3352750991513,1cyclictest60650irq/133-enp0s29f1:rx-011:00:270
3352750991513,1cyclictest60650irq/133-enp0s29f1:rx-010:45:280
3352750991512,2cyclictest60650irq/133-enp0s29f1:rx-011:40:190
3352750991512,2cyclictest60650irq/133-enp0s29f1:rx-011:40:190
3352750991512,2cyclictest60650irq/133-enp0s29f1:rx-009:53:540
3352750991512,2cyclictest60650irq/133-enp0s29f1:rx-009:09:200
3352750991512,2cyclictest60650irq/133-enp0s29f1:rx-007:54:310
3352750991512,2cyclictest60650irq/133-enp0s29f1:rx-007:45:150
3352750991512,2cyclictest60650irq/133-enp0s29f1:rx-007:10:140
3352762991411,2cyclictest3591496-21unixbench_singl10:55:273
3352762991411,2cyclictest3540931-21irqcore10:10:173
3352762991411,2cyclictest3473042-21irqcore09:05:183
335275799148,4cyclictest3452204-21diskstats08:45:172
3352757991411,2cyclictest3577167-21irqcore10:45:172
3352757991411,2cyclictest3467201-21cut08:59:592
3352754991411,2cyclictest3492684-21kworker/u8:2+events_unbound09:25:181
3352754991411,2cyclictest3492684-21kworker/u8:2+events_unbound09:25:171
3352754991411,2cyclictest3458193-21kthreadcore08:50:181
3352754991411,2cyclictest3441303-21kworker/u8:2+events_unbound08:40:141
3352754991411,2cyclictest3372280-21kworker/u8:2+flush-259:007:25:241
3352754991411,2cyclictest3358531-21irqcore07:15:161
3352754991410,3cyclictest3580718-21kworker/u8:2+flush-259:011:15:161
3352754991410,3cyclictest3561197-21kworker/u8:3+events_unbound10:35:161
3352754991410,3cyclictest3529511-21kworker/u8:0+events_unbound10:00:041
335275099149,4cyclictest3430075-21kworker/u8:1+flush-259:008:50:180
335275099148,4cyclictest3452017-21/usr/sbin/munin08:45:230
335275099147,5cyclictest3477921-21diskstats09:10:130
3352750991412,2cyclictest60650irq/133-enp0s29f1:rx-009:31:090
3352750991412,1cyclictest60650irq/133-enp0s29f1:rx-009:55:230
3352750991412,1cyclictest0-21swapper/007:40:000
3352750991411,2cyclictest60650irq/133-enp0s29f1:rx-012:16:100
3352750991411,2cyclictest60650irq/133-enp0s29f1:rx-012:00:140
3352750991411,2cyclictest60650irq/133-enp0s29f1:rx-011:35:200
335276299139,3cyclictest3435942-21turbostat08:34:553
335276299139,2cyclictest3612593-21cron11:19:593
3352762991311,1cyclictest0-21swapper/307:55:203
3352762991310,2cyclictest3680499-21cpuspeed_turbos12:25:143
3352762991310,2cyclictest3483939-21kthreadcore09:15:173
3352762991310,1cyclictest0-21swapper/311:35:003
335275799139,3cyclictest3462283-21kworker/u8:3+flush-259:009:10:202
335275799135,1cyclictest3539415-21ntp_offset10:05:252
335275799134,2cyclictest35633482grep10:30:202
3352757991311,1cyclictest0-21swapper/207:40:002
3352757991310,2cyclictest3507592-21ntp_kernel_pll_09:35:212
3352757991310,2cyclictest3467523-21cpuspeed_turbos09:00:122
3352757991310,1cyclictest0-21swapper/210:20:252
3352754991310,2cyclictest0-21swapper/110:50:011
335275099137,4cyclictest3352683-21latency_hist07:10:010
335275099137,4cyclictest3352683-21latency_hist07:10:000
3352750991312,1cyclictest60650irq/133-enp0s29f1:rx-012:21:400
3352750991312,1cyclictest60650irq/133-enp0s29f1:rx-010:55:150
3352750991311,1cyclictest60650irq/133-enp0s29f1:rx-012:29:490
3352750991311,1cyclictest60650irq/133-enp0s29f1:rx-011:10:020
3352750991311,1cyclictest60650irq/133-enp0s29f1:rx-010:22:330
3352750991311,1cyclictest60650irq/133-enp0s29f1:rx-010:04:170
3352750991310,2cyclictest60650irq/133-enp0s29f1:rx-011:50:160
335276299129,2cyclictest3650044-21kthreadcore11:55:163
335276299129,2cyclictest3500680-21kthreadcore09:30:183
335276299129,2cyclictest3495520-21tr09:25:173
335276299129,2cyclictest3495520-21tr09:25:173
335276299128,3cyclictest3451863-21base6408:45:003
335276299128,3cyclictest3383615-21cron07:40:013
3352762991210,1cyclictest0-21swapper/310:00:013
3352762991210,1cyclictest0-21swapper/308:25:003
335275799129,2cyclictest3414901-21cut08:10:122
335275799126,5cyclictest3607689-21apt-get11:15:132
335275799126,4cyclictest3358067-21sshd07:15:032
335275799120,5cyclictest3685827-21hddtemp_smartct12:30:152
335275099129,2cyclictest3653396-21unixbench_multi11:55:230
3352750991211,1cyclictest60650irq/133-enp0s29f1:rx-009:15:220
3352750991210,1cyclictest60650irq/133-enp0s29f1:rx-010:40:530
3352750991210,1cyclictest60650irq/133-enp0s29f1:rx-009:45:260
3352750991210,1cyclictest60650irq/133-enp0s29f1:rx-007:24:050
335276299118,2cyclictest0-21swapper/307:10:213
335276299115,4cyclictest2698871-21systemd-journal11:45:003
335276299115,4cyclictest2698871-21systemd-journal11:44:593
335276299111,8cyclictest3540237-21cron10:10:003
335275799119,1cyclictest0-21swapper/208:45:002
335275799118,2cyclictest3412444-21kthreadcore08:05:222
335275799114,5cyclictest3492031-21ntpq09:20:212
335275499116,4cyclictest3393980-21kworker/u8:2+flush-259:007:54:081
335275499115,4cyclictest3654745-21sed12:00:151
335275499115,4cyclictest3452017-21/usr/sbin/munin08:45:181
335275099119,1cyclictest0-21swapper/011:24:590
335276299108,1cyclictest3567010-21runrttasks10:35:173
335276299108,1cyclictest0-21swapper/310:20:003
335276299108,1cyclictest0-21swapper/307:50:213
335276299107,2cyclictest3519481-21cpuspeed_turbos09:50:133
335276299106,3cyclictest3570936-21kworker/u8:1+flush-259:010:40:263
335276299105,4cyclictest3695605-21latency_hist12:40:013
335275799107,2cyclictest35496722sleep210:15:202
335275799106,3cyclictest3472595-21/usr/sbin/munin09:05:172
335275799104,4cyclictest3571563-21/usr/sbin/munin10:40:252
33527629996,2cyclictest3622953-21latency_hist11:30:003
33527629996,2cyclictest3373355-21systemctl07:30:003
33527629996,1cyclictest0-21swapper/310:30:233
33527579997,1cyclictest0-21swapper/212:05:232
33527579996,2cyclictest0-21swapper/208:00:162
33527579996,1cyclictest0-21swapper/209:45:172
33527579995,3cyclictest3616766-21needreboot11:20:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional