You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 21:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot4s.osadl.org (updated Sat May 23, 2026 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3328674993628,6cyclictest3384798-21ntpq08:00:230
3328685993530,3cyclictest3620911-21kworker/u8:1+flush-259:011:50:173
3328683993328,3cyclictest3422017-21/usr/sbin/munin08:40:152
3328674993327,4cyclictest3339182-21latency_hist07:20:010
3328685993228,3cyclictest3344341-21kworker/u8:2+flush-259:007:40:173
3328685993226,4cyclictest509-21rs:main3
3328674993228,3cyclictest3636488-21kworker/u8:2+flush-259:012:15:160
3328674993228,3cyclictest3552837-21gpgv10:45:010
3328685993127,3cyclictest3510513-21kworker/u8:2+events_unbound10:30:013
3328685993125,4cyclictest3463483-21kworker/u8:3+flush-259:009:49:593
3328674993028,1cyclictest3568876-21kworker/0:2+pm11:27:070
3328685992926,2cyclictest3531349-21kworker/u8:0+flush-259:010:40:003
3328683992918,9cyclictest335557-21systemd-journal10:30:002
3328685992823,3cyclictest3553464-21kworker/u8:1+flush-259:011:10:003
3328683992813,6cyclictest0-21swapper/207:40:002
3328685992721,4cyclictest3515875-21diskstats10:10:133
3328679992721,4cyclictest335557-21systemd-journal10:45:011
3328674992722,4cyclictest3339454-21apt-get07:20:120
3328674992721,5cyclictest3573358-21ntpq11:00:200
3328685992623,2cyclictest3579406-21kworker/u8:2+flush-259:011:11:083
3328679992613,4cyclictest3630164-21ntp_kernel_pll_11:55:211
3328685992521,3cyclictest3479160-21kworker/u8:1+flush-259:009:50:183
3328683992519,4cyclictest3401081-21latency_hist08:20:012
3328679992519,4cyclictest3569117-21latency_hist11:00:011
3328674992519,4cyclictest335557-21systemd-journal12:10:000
3328683992419,3cyclictest3531417-21cp10:25:002
3328683992414,9cyclictest500-21systemd-logind11:27:082
3328679992418,4cyclictest3536897-21latency_hist10:30:011
3328674992417,5cyclictest3353829-21ntpq07:30:220
3328685992319,3cyclictest3636488-21kworker/u8:2+flush-259:012:25:003
3328685992319,3cyclictest3426957-21kworker/u8:1+flush-259:008:55:213
3328685992318,3cyclictest3579406-21kworker/u8:2+events_unbound11:20:183
3328685992317,4cyclictest3666610-21ntpq12:30:223
3328685992317,4cyclictest3666610-21ntpq12:30:223
3328685992317,4cyclictest3553464-21kworker/u8:1+flush-259:010:55:173
332868399230,13cyclictest3467672-21ntp_kernel_err09:20:212
3328685992219,2cyclictest3508791-21kworker/u8:1+flush-259:010:00:263
3328685992219,2cyclictest3508791-21kworker/u8:1+flush-259:010:00:253
3328685992218,3cyclictest3426957-21kworker/u8:1+events_unbound08:45:233
3328685992217,4cyclictest3370023-21kworker/u8:3+events_unbound08:40:223
3328685992217,3cyclictest3463483-21kworker/u8:3+events_unbound09:25:243
3328685992216,4cyclictest3641439-21latency_hist12:10:003
3328683992219,2cyclictest507-21in:imuxsock08:15:002
3328679992218,3cyclictest3558402-21apt-get10:49:591
3328674992218,3cyclictest3390998-21apt-get08:10:130
3328674992217,4cyclictest3411719-21apt-get08:30:120
3328674992216,5cyclictest3467691-21ntpq09:20:210
3328685992118,2cyclictest3323297-21kworker/u8:2+flush-259:007:15:003
3328685992117,3cyclictest3579406-21kworker/u8:2+events_unbound11:40:223
3328685992117,3cyclictest3489417-21kworker/u8:0+events_unbound10:15:203
3328685992116,4cyclictest3457631-21kworker/u8:0+flush-259:009:20:003
3328685992116,3cyclictest3531349-21kworker/u8:0+flush-259:011:00:163
3328685992116,3cyclictest3499763-21kworker/u8:4+events_unbound12:25:133
3328683992117,3cyclictest3323297-21kworker/u8:2+flush-259:007:10:162
3328679992116,4cyclictest335557-21systemd-journal09:55:281
3328674992119,1cyclictest3370023-21kworker/u8:3+events_unbound08:45:180
3328674992118,2cyclictest3620911-21kworker/u8:1+events_unbound12:10:200
3328674992115,4cyclictest3574522-21/usr/sbin/munin11:05:140
3328674992114,4cyclictest3349457-21latency_hist07:30:000
3328685992016,3cyclictest3426957-21kworker/u8:1+events_unbound09:00:183
3328685992015,3cyclictest3344341-21kworker/u8:2+flush-259:007:35:403
3328685992014,4cyclictest3334035-21kworker/u8:1+flush-259:007:29:553
3328683992016,3cyclictest3531349-21kworker/u8:0+flush-259:010:30:182
332868399201,17cyclictest3421766-21awk08:40:002
3328674992016,3cyclictest3457631-21kworker/u8:0+flush-259:009:25:140
3328674992013,5cyclictest335557-21systemd-journal10:49:070
3328685991917,1cyclictest3334035-21kworker/u8:1+events_unbound07:55:243
3328685991916,2cyclictest3616217-21irqcore11:45:163
3328685991916,2cyclictest187307-21kworker/u8:0+flush-259:007:15:153
3328685991915,3cyclictest3531349-21kworker/u8:0+flush-259:010:50:563
3328685991915,3cyclictest3370023-21kworker/u8:3+flush-259:008:05:233
3328685991915,3cyclictest187307-21kworker/u8:0+flush-259:007:20:263
3328685991914,4cyclictest3636488-21kworker/u8:2+events_unbound12:15:123
3328685991912,5cyclictest3499763-21kworker/u8:4+flush-259:011:35:173
332868399194,6cyclictest3662240-21munin-run12:30:002
3328683991917,1cyclictest3589713-21cron11:20:002
3328683991915,3cyclictest3553373-21gpgv10:45:002
3328683991915,3cyclictest3499763-21kworker/u8:4+flush-259:012:05:002
332868399190,9cyclictest3386277-21cat08:05:172
3328679991914,3cyclictest3427181-21apt-get08:45:111
3328674991916,2cyclictest3426957-21kworker/u8:1+events_unbound08:55:230
3328674991915,3cyclictest3626149-21/usr/sbin/munin11:55:190
3328674991915,3cyclictest3432293-21kworker/u8:2+flush-259:009:40:200
3328674991915,3cyclictest187307-21kworker/u8:0+flush-259:007:55:000
3328674991914,3cyclictest3636488-21kworker/u8:2+flush-259:012:35:000
3328674991914,3cyclictest3636488-21kworker/u8:2+flush-259:012:34:590
3328674991914,3cyclictest3401310-21kworker/u8:2+flush-259:008:25:210
3328674991913,4cyclictest335557-21systemd-journal07:14:590
332867499190,18cyclictest1-21systemd10:00:290
332867499190,18cyclictest1-21systemd10:00:280
3328685991814,3cyclictest3631083-21kworker/u8:0+events_unbound12:01:033
3328685991814,3cyclictest3499763-21kworker/u8:4+events_unbound11:15:163
3328685991814,3cyclictest3370023-21kworker/u8:3+events_unbound08:23:053
3328685991813,3cyclictest3380658-21kworker/u8:4+flush-259:008:05:013
3328685991812,4cyclictest3631083-21kworker/u8:0+flush-259:012:36:103
332868399181,15cyclictest3457836-21awk09:15:012
3328683991810,6cyclictest0-21swapper/209:50:212
332868399180,17cyclictest3328682-21cat07:10:012
332868399180,10cyclictest3478740-21taskset09:32:582
3328679991813,4cyclictest3334020-21latency_hist07:15:001
3328679991812,4cyclictest3478916-21latency_hist09:35:001
3328679991811,6cyclictest3563943-21munin-run10:55:011
3328674991815,1cyclictest0-21swapper/009:35:000
3328674991814,3cyclictest3499763-21kworker/u8:4+flush-259:010:05:250
3328674991812,4cyclictest3515712-21apt-get10:10:120
3328685991714,2cyclictest3620911-21kworker/u8:1+events_unbound11:58:063
3328685991714,2cyclictest3380658-21kworker/u8:4+events_unbound08:30:383
3328685991714,2cyclictest3370023-21kworker/u8:3+flush-259:008:25:163
3328685991713,3cyclictest3344341-21kworker/u8:2+flush-259:007:45:243
3328685991713,2cyclictest3432293-21kworker/u8:2+flush-259:009:40:243
3328685991711,4cyclictest3514590-21ntpq10:05:223
332868399178,8cyclictest3583616-21needreboot11:10:302
3328683991715,1cyclictest3328669-21cyclictest12:35:252
332868399170,9cyclictest3657016-21taskset12:24:462
332868399170,9cyclictest3369866-21taskset07:47:142
332868399170,13cyclictest3662716-21tune2fs12:30:142
332868399170,13cyclictest3662716-21tune2fs12:30:142
3328679991715,1cyclictest3647374-21irqcore12:15:161
3328674991714,2cyclictest60350irq/132-enp0s29f1:rx-010:25:160
3328674991713,3cyclictest3564202-21apt-get10:55:110
3328674991713,3cyclictest3553464-21kworker/u8:1+flush-259:010:50:210
3328674991713,3cyclictest3426957-21kworker/u8:1+flush-259:009:10:210
3328674991713,3cyclictest3370023-21kworker/u8:3+events_unbound09:00:230
3328674991713,2cyclictest3499763-21kworker/u8:4+flush-259:009:55:170
3328674991713,2cyclictest0-21swapper/007:45:010
3328674991712,4cyclictest3355095-21diskstats07:35:280
3328674991712,3cyclictest335557-21systemd-journal12:24:060
3328685991612,3cyclictest3636488-21kworker/u8:2+events_unbound12:13:073
3328685991612,3cyclictest3531349-21kworker/u8:0+flush-259:010:40:393
3328685991612,3cyclictest3499763-21kworker/u8:4+events_unbound11:31:033
3328685991612,3cyclictest3479160-21kworker/u8:1+events_unbound09:39:063
3328685991612,3cyclictest3443231-21kworker/u8:0+events_unbound09:05:073
3328685991612,3cyclictest3380658-21kworker/u8:4+flush-259:008:15:003
3328685991612,2cyclictest3499763-21kworker/u8:4+events_unbound10:32:083
3328685991611,4cyclictest3525801-21kworker/u8:1+events_unbound10:20:173
332868399166,9cyclictest3344796-21uniq07:25:152
332868399165,8cyclictest3505032-21cpuspeed_turbos10:00:152
332868399165,8cyclictest3505032-21cpuspeed_turbos10:00:152
3328683991614,1cyclictest3442379-21sh09:00:012
3328683991612,2cyclictest0-21swapper/208:25:002
3328683991611,4cyclictest3542914-21diskstats10:35:142
332868399161,13cyclictest3489174-21idleruntime-cro09:45:002
332868399160,9cyclictest3344178-21taskset07:23:292
332868399160,8cyclictest3641679-21/usr/sbin/munin12:10:222
332868399160,8cyclictest3504579-21chrt09:58:442
332868399160,14cyclictest3520949-21grep10:15:012
332868399160,14cyclictest3442620-21acpi09:00:102
332868399160,13cyclictest3548100-21grep10:40:142
3328679991614,1cyclictest3531325-21idleruntime-cro10:25:001
3328679991612,3cyclictest3509865-21latency_hist10:05:011
3328679991612,3cyclictest3509865-21latency_hist10:05:011
3328679991612,3cyclictest3479160-21kworker/u8:1+flush-259:009:35:261
3328679991612,3cyclictest335557-21systemd-journal07:19:071
3328679991611,4cyclictest3604330-21ntpq11:30:201
3328679991611,3cyclictest3520636-21gpgv10:15:001
3328674991614,1cyclictest60350irq/132-enp0s29f1:rx-011:17:090
3328674991613,2cyclictest60350irq/132-enp0s29f1:rx-012:37:340
3328674991613,2cyclictest3631083-21kworker/u8:0+flush-259:012:25:210
3328674991612,3cyclictest3370023-21kworker/u8:3+flush-259:008:50:250
3328674991611,3cyclictest3483178-21ntpq09:35:220
3328685991512,2cyclictest0-21swapper/311:25:173
3328685991511,3cyclictest3426957-21kworker/u8:1+events_unbound09:11:073
3328685991511,3cyclictest3370023-21kworker/u8:3+events_unbound08:50:143
332868399159,4cyclictest3509866-21apt-get10:05:002
332868399155,8cyclictest3559350-21uniq10:50:152
332868399155,8cyclictest3338400-21open_inodes07:15:232
332868399154,8cyclictest3370561-21grep07:50:142
332868399153,10cyclictest3383063-21cut08:00:192
3328683991513,1cyclictest0-21swapper/210:15:182
3328683991512,2cyclictest335557-21systemd-journal09:30:002
332868399151,13cyclictest3610390-21cat11:40:012
332868399151,12cyclictest507-21in:imuxsock11:55:002
332868399151,12cyclictest3641589-21cat12:10:002
332868399151,12cyclictest3615073-21grep11:40:252
332868399151,12cyclictest3493552-21ntpq09:45:222
332868399151,12cyclictest335557-21systemd-journal08:00:002
332868399151,11cyclictest3447927-21tr09:05:122
332868399150,7cyclictest3364709-21taskset07:42:002
332868399150,13cyclictest3579391-21idleruntime-cro11:10:012
332868399150,13cyclictest3432507-21fschecks_count08:50:132
332868399150,13cyclictest3428198-21ps08:45:172
332868399150,13cyclictest3416762-21latency_hist08:35:012
332868399150,12cyclictest3647095-21grep12:15:132
332868399150,12cyclictest3600468-21cut11:30:132
332868399150,12cyclictest3354800-21latency_hist07:35:002
3328679991511,3cyclictest3401310-21kworker/u8:2+flush-259:008:20:101
3328679991511,3cyclictest3334035-21kworker/u8:1+flush-259:007:20:171
3328679991511,2cyclictest503-21tpm2-abrmd07:35:011
3328679991511,2cyclictest1-21systemd11:05:001
3328679991510,3cyclictest3345004-21irqcore07:25:171
332867999150,9cyclictest3396151-21seq08:15:081
332867999150,5cyclictest3588811-21ntpq11:15:191
3328674991514,1cyclictest60350irq/132-enp0s29f1:rx-008:43:230
3328674991513,2cyclictest60350irq/132-enp0s29f1:rx-009:07:340
3328674991513,1cyclictest60350irq/132-enp0s29f1:rx-011:35:250
3328674991513,1cyclictest60350irq/132-enp0s29f1:rx-011:33:170
3328674991513,1cyclictest60350irq/132-enp0s29f1:rx-011:20:230
3328674991513,1cyclictest60350irq/132-enp0s29f1:rx-011:10:330
3328674991513,1cyclictest60350irq/132-enp0s29f1:rx-009:15:150
3328674991513,1cyclictest60350irq/132-enp0s29f1:rx-008:17:210
3328674991512,2cyclictest60350irq/132-enp0s29f1:rx-009:50:200
3328674991512,2cyclictest60350irq/132-enp0s29f1:rx-007:56:200
3328674991512,2cyclictest60350irq/132-enp0s29f1:rx-007:45:160
3328674991512,2cyclictest3499763-21kworker/u8:4+events_unbound11:45:180
3328674991511,3cyclictest3620911-21kworker/u8:1+events_unbound11:50:230
332868599149,4cyclictest3458409-21gpgv09:15:013
3328685991411,2cyclictest3417299-21irqcore08:35:183
3328685991410,3cyclictest3463483-21kworker/u8:3+events_unbound09:55:083
332868399141,11cyclictest3594915-21systemd-run11:25:012
332868399140,13cyclictest3407087-21sed08:25:172
332868399140,12cyclictest3629990-21memory11:55:202
332868399140,12cyclictest3620762-21sed11:49:592
332868399140,12cyclictest3573210-21memory11:00:182
332868399140,12cyclictest3567311-21cut10:55:192
332868399140,12cyclictest3460665-21cut09:15:172
332868399140,11cyclictest3479314-21cpuspeed_turbos09:35:152
332867999143,5cyclictest0-21swapper/108:59:371
3328679991411,2cyclictest3486625-21kthreadcore09:40:171
3328679991411,2cyclictest3401310-21kworker/u8:2+flush-259:008:30:261
3328679991411,2cyclictest0-21swapper/108:00:001
332867999141,10cyclictest3631584-21cut12:00:141
332867999140,5cyclictest3542033-21taskset10:34:531
332867999140,5cyclictest3462502-21/usr/sbin/munin09:15:191
332867999140,4cyclictest3636529-21dpkg12:05:121
332867999140,4cyclictest3457743-21taskset09:13:591
332867499149,4cyclictest3416857-21/usr/sbin/munin08:35:170
332867499148,4cyclictest3542089-21gpgv10:35:000
3328674991412,2cyclictest60350irq/132-enp0s29f1:rx-008:07:070
3328674991411,2cyclictest60350irq/132-enp0s29f1:rx-010:15:320
3328674991411,2cyclictest60350irq/132-enp0s29f1:rx-009:48:470
332868599137,4cyclictest3396342-21diskstats08:15:153
332867999139,3cyclictest3610617-21apt-get11:40:121
332867999139,2cyclictest3406623-21cpuspeed_turbos08:25:141
332867999138,3cyclictest3589970-21/usr/sbin/munin11:20:191
332867999138,3cyclictest335557-21systemd-journal12:11:021
332867999130,4cyclictest3421397-21chrt08:35:291
3328674991312,1cyclictest60350irq/132-enp0s29f1:rx-010:38:010
3328674991311,1cyclictest60350irq/132-enp0s29f1:rx-012:00:310
332868599129,2cyclictest3344341-21kworker/u8:2+events_unbound07:33:063
332868599128,3cyclictest3370023-21kworker/u8:3+events_unbound07:50:233
332867999129,2cyclictest3578417-21cut11:05:201
332867999129,2cyclictest3360123-21cpuspeed_turbos07:40:131
332867999128,2cyclictest3620925-21/usr/sbin/munin11:50:141
332867999121,9cyclictest335557-21systemd-journal07:45:161
332867999120,5cyclictest3662189-21taskset12:29:431
332867999119,1cyclictest3395898-21cron08:15:011
332867999118,2cyclictest3490471-21kthreadcore09:45:181
332867999118,2cyclictest3447849-21cpuspeed_turbos09:05:131
332867999117,3cyclictest0-21swapper/109:20:251
332867999114,5cyclictest3672593-21cron12:40:001
332867499118,2cyclictest60350irq/132-enp0s29f1:rx-008:20:220
332867999106,3cyclictest335557-21systemd-journal08:44:561
332867999106,2cyclictest33705292sleep107:50:141
332867999104,5cyclictest3512685-21cut10:05:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional