You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-29 - 21:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Sun Mar 29, 2026 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3312563993834,3cyclictest3380316-21apt-get08:15:110
3312565993732,3cyclictest3348917-21apt-get07:45:141
3312565993732,3cyclictest3348917-21apt-get07:45:131
3312565993631,3cyclictest3477912-21kworker/u8:2+flush-259:011:25:171
3312573993431,2cyclictest3374583-21gpgv08:10:003
3312565993430,3cyclictest3472968-21apt-get09:45:121
3312565993428,4cyclictest3410936-21latency_hist08:45:001
3312573993330,2cyclictest3576374-21kworker/u8:1+flush-259:011:35:013
3312573993025,3cyclictest3509433-21/usr/sbin/munin10:20:133
3312565993027,2cyclictest3623231-21kworker/u8:1+flush-259:012:15:161
3312565993026,3cyclictest3354097-21apt-get07:50:101
3312563993027,2cyclictest3555759-21apt-get11:05:130
3312563993024,4cyclictest3649531-21cp12:35:000
3312573992925,3cyclictest3477912-21kworker/u8:2+flush-259:011:50:213
3312563992923,4cyclictest252-21systemd-journal12:02:350
3312565992824,3cyclictest3385480-21apt-get08:20:111
3312563992723,3cyclictest3591260-21ntpq11:35:220
3312563992723,3cyclictest3576374-21kworker/u8:1+events_unbound11:30:000
3312573992623,2cyclictest3560887-21kworker/u8:3+flush-259:011:15:013
3312569992620,4cyclictest509-21rs:main2
331256399260,24cyclictest3503693-21munin-run10:15:000
3312573992520,3cyclictest490-21dbus-daemon11:30:013
3312563992521,3cyclictest3493352-21kworker/u8:1+events_unbound10:15:130
3312573992416,6cyclictest3570074-21ntpq11:15:193
3312569992418,4cyclictest3597381-21gpgv11:45:002
3312565992419,3cyclictest509-21rs:main1
3312565992419,3cyclictest3462642-21/usr/sbin/munin09:35:161
3312563992420,3cyclictest3519289-21kworker/u8:0+flush-259:010:30:000
3312573992320,2cyclictest3416333-21apt-get08:50:123
3312565992319,3cyclictest3519289-21kworker/u8:0+flush-259:010:45:001
3312565992319,3cyclictest1241023-21kworker/u8:2+flush-259:007:20:261
3312565992318,3cyclictest3390616-21apt-get08:25:121
3312563992317,4cyclictest3333206-21latency_hist07:30:010
3312573992219,2cyclictest3405386-21kworker/u8:0+flush-259:008:40:173
3312569992216,4cyclictest3421240-21latency_hist08:55:012
3312565992217,3cyclictest3312493-21latency_hist07:10:001
3312563992219,2cyclictest3253067-21kworker/u8:3+flush-259:007:18:260
3312563992218,3cyclictest3348881-21kworker/u8:3+events_unbound09:30:140
3312563992217,3cyclictest3472572-21kworker/u8:0+flush-259:009:45:050
3312563992215,5cyclictest3323354-21diskstats07:20:120
331256399221,19cyclictest1-21systemd10:35:000
3312573992118,2cyclictest3649330-21apt-get12:35:023
3312573992118,2cyclictest3649330-21apt-get12:35:023
3312573992117,3cyclictest3628686-21kworker/u8:4+flush-259:012:25:253
331256999215,4cyclictest3591213-21ntpq11:35:212
331256999210,10cyclictest0-21swapper/208:45:102
3312563992117,3cyclictest3628686-21kworker/u8:4+events_unbound12:25:010
3312563992116,4cyclictest3405386-21kworker/u8:0+events_unbound08:55:160
331256399210,19cyclictest1-21systemd07:46:350
331256399210,19cyclictest1-21systemd07:46:340
3312573992016,3cyclictest3348881-21kworker/u8:3+flush-259:007:45:273
3312573992016,3cyclictest3348881-21kworker/u8:3+flush-259:007:45:273
331256999202,9cyclictest3555560-21cat11:05:012
3312569992013,6cyclictest490-21dbus-daemon07:50:002
3312569992013,6cyclictest490-21dbus-daemon07:49:592
3312565992016,3cyclictest3253067-21kworker/u8:3+flush-259:007:14:591
3312563992017,2cyclictest3477912-21kworker/u8:2+events_unbound12:10:140
3312563992017,2cyclictest1241023-21kworker/u8:2+flush-259:007:35:200
3312563992013,5cyclictest3649963-21cat12:35:010
3312563992013,5cyclictest3649963-21cat12:35:010
3312563992013,5cyclictest3497627-21ntpq10:05:210
3312563992013,5cyclictest3482210-21ntpq09:50:210
3312563992013,5cyclictest3447008-21latency_hist09:20:010
3312573991915,3cyclictest3493352-21kworker/u8:1+flush-259:010:25:143
3312573991913,5cyclictest3571241-21/usr/sbin/munin11:20:223
3312569991915,3cyclictest509-21rs:main2
3312569991915,3cyclictest3560887-21kworker/u8:3+flush-259:011:20:012
331256599196,5cyclictest3400670-21idleruntime-cro08:35:001
3312565991912,5cyclictest252-21systemd-journal12:05:001
3312563991916,2cyclictest3519289-21kworker/u8:0+flush-259:011:00:250
3312563991916,2cyclictest3477912-21kworker/u8:2+flush-259:011:55:000
3312563991916,2cyclictest3348881-21kworker/u8:3+flush-259:007:50:360
3312563991915,3cyclictest3623231-21kworker/u8:1+events_unbound12:19:360
3312563991915,3cyclictest3509219-21kworker/u8:3+flush-259:010:35:210
3312573991813,4cyclictest3477912-21kworker/u8:2+flush-259:011:35:223
331256999182,5cyclictest3529743-21sshd10:39:422
3312569991815,2cyclictest3540066-21idleruntime-cro10:50:002
3312569991814,3cyclictest1241023-21kworker/u8:2+flush-259:007:30:192
3312563991815,2cyclictest3333230-21kworker/u8:0+flush-259:007:40:210
3312563991814,3cyclictest3472572-21kworker/u8:0+events_unbound09:59:360
3312563991814,3cyclictest3427617-21kworker/u8:1+flush-259:009:20:220
331257399179,5cyclictest252-21systemd-journal07:14:593
3312573991714,2cyclictest3601912-21needreboot11:45:203
3312573991713,3cyclictest0-21swapper/310:14:593
331256999177,2cyclictest3323477-21fw_conntrack07:20:142
331256999175,9cyclictest3374905-21cat08:10:012
331256999173,4cyclictest3575282-21ntpq11:20:212
331256999172,4cyclictest3643777-21sshd12:25:362
3312569991714,2cyclictest509-21rs:main2
3312569991713,2cyclictest0-21swapper/211:10:002
3312565991713,2cyclictest3498463-21sh10:10:001
3312565991710,6cyclictest3348881-21kworker/u8:3+ext4-rsv-conversion09:00:001
3312563991714,2cyclictest3560887-21kworker/u8:3+events_unbound11:15:170
3312563991714,2cyclictest3348881-21kworker/u8:3+events_unbound09:10:130
3312563991713,3cyclictest3477912-21kworker/u8:2+events_unbound12:05:210
3312563991713,3cyclictest3477912-21kworker/u8:2+events_unbound11:48:370
3312563991713,3cyclictest3364145-21kworker/u8:0+events_unbound08:00:220
3312563991713,2cyclictest60350irq/132-enp0s29f1:rx-010:45:120
3312563991713,2cyclictest3514363-21turbostat10:25:000
3312563991713,2cyclictest0-21swapper/011:35:000
3312563991712,4cyclictest3493314-21cron10:05:000
3312563991710,5cyclictest3359220-21/usr/sbin/munin07:55:130
3312573991612,3cyclictest3628686-21kworker/u8:4+flush-259:012:20:253
3312573991612,3cyclictest3462423-21kworker/u8:2+events_unbound09:40:183
3312573991612,2cyclictest3348647-21sh07:44:593
3312573991611,4cyclictest3302322-21kworker/u8:1+flush-259:007:25:143
3312573991610,4cyclictest3492446-21ntpq10:00:203
331256999165,9cyclictest3464733-21cut09:35:192
3312569991613,2cyclictest3407055-21kthreadcore08:40:172
3312569991610,4cyclictest3644392-21/usr/sbin/munin12:30:202
3312565991613,2cyclictest3441875-21idleruntime-cro09:15:001
3312565991611,3cyclictest3427617-21kworker/u8:1+flush-259:009:15:161
3312565991610,5cyclictest3503860-21cp10:15:011
3312565991610,4cyclictest3623505-21latency_hist12:10:001
3312563991614,1cyclictest60350irq/132-enp0s29f1:rx-011:55:160
3312563991614,1cyclictest60350irq/132-enp0s29f1:rx-009:25:140
3312563991613,2cyclictest60350irq/132-enp0s29f1:rx-009:35:120
3312563991613,2cyclictest60350irq/132-enp0s29f1:rx-008:50:160
3312563991613,2cyclictest3477912-21kworker/u8:2+flush-259:011:10:210
3312563991613,2cyclictest1241023-21kworker/u8:2+flush-259:007:30:240
3312563991612,3cyclictest3519289-21kworker/u8:0+flush-259:010:55:000
3312563991612,3cyclictest3477912-21kworker/u8:2+events_unbound11:40:170
3312563991612,3cyclictest3405386-21kworker/u8:0+flush-259:008:45:180
3312563991612,3cyclictest3348881-21kworker/u8:3+events_unbound08:20:020
3312563991612,2cyclictest0-21swapper/007:15:000
331257399159,4cyclictest1241023-21kworker/u8:2+flush-259:007:30:253
3312573991511,2cyclictest3421910-21irqcore08:55:163
331256999155,8cyclictest3519466-21awk10:30:002
331256999153,9cyclictest3379135-21ntpq08:10:212
331256999153,4cyclictest3426608-21/usr/sbin/munin09:00:202
331256999152,4cyclictest3318173-21diskstats07:15:142
3312569991511,3cyclictest3333197-21idleruntime-cro07:29:592
331256599159,4cyclictest3431563-21latency_hist09:05:011
331256599157,6cyclictest3597247-21cron11:44:591
3312565991512,2cyclictest3332278-21ntp_kernel_err07:25:181
3312563991513,1cyclictest60350irq/132-enp0s29f1:rx-012:20:150
3312563991513,1cyclictest60350irq/132-enp0s29f1:rx-010:56:020
3312563991513,1cyclictest60350irq/132-enp0s29f1:rx-009:09:510
3312563991512,2cyclictest60350irq/132-enp0s29f1:rx-008:31:280
3312563991512,2cyclictest60350irq/132-enp0s29f1:rx-008:05:250
3312563991512,2cyclictest3530477-21irqcore10:40:170
331257399149,3cyclictest3623231-21kworker/u8:1+flush-259:012:10:243
331257399148,5cyclictest252-21systemd-journal10:35:003
331257399141,3cyclictest3633823-21awk12:20:013
3312573991410,3cyclictest3348881-21kworker/u8:3+flush-259:008:50:003
331257399140,12cyclictest3328043-21awk07:25:003
331256999149,1cyclictest0-21swapper/209:59:592
331256999147,6cyclictest3436690-21awk09:10:012
331256999146,6cyclictest3654148-21nvmesmart_nvme012:35:222
331256999146,6cyclictest3654148-21nvmesmart_nvme012:35:222
331256999146,5cyclictest3623764-21apt-get12:10:122
331256999143,2cyclictest3385147-21taskset08:19:242
331256999142,3cyclictest3442298-21df09:15:122
331256999142,2cyclictest3613621-21diskstats12:00:132
331256999142,2cyclictest3477074-21ntp_states09:45:222
3312569991411,2cyclictest0-21swapper/211:50:002
3312569991410,3cyclictest3477912-21kworker/u8:2+flush-259:012:00:002
3312563991411,2cyclictest60350irq/132-enp0s29f1:rx-011:20:200
331257399139,2cyclictest3649334-21latency_hist12:35:013
331257399137,4cyclictest3436703-21latency_hist09:10:013
331257399131,10cyclictest3488209-21awk10:00:003
3312573991310,2cyclictest3390841-21fschecks_time08:25:153
3312573991310,2cyclictest3343513-21latency_hist07:40:003
331257399130,4cyclictest35588542tr11:05:193
331256999139,3cyclictest3565804-21cron11:15:002
331256999138,3cyclictest3483038-21cron09:55:002
331256999138,3cyclictest3451244-21ntpq09:20:212
331256999137,4cyclictest3494514-21sed10:05:172
331256999137,4cyclictest3317735-21cron07:14:592
331256999136,5cyclictest3634233-21tr12:20:122
331256999136,5cyclictest3576815-21ps11:25:172
331256999136,5cyclictest3426370-21latency_hist09:00:012
331256999135,6cyclictest3508422-21grep10:15:222
331256999135,6cyclictest3398589-21tr08:30:182
331256999135,6cyclictest3391343-21sed08:25:172
331256999133,2cyclictest3540628-21fw_forwarded_lo10:50:142
331256999133,2cyclictest3364543-21df_inode08:00:142
331256999133,2cyclictest3358091-21ntpq07:50:182
331256999133,2cyclictest3344754-21irqstats07:40:152
331256999132,2cyclictest3586126-21ntp_states11:30:242
331256999132,2cyclictest3389275-21memory08:20:192
331256999132,2cyclictest3363306-21ntp_states07:55:222
3312569991310,2cyclictest3467961-21cpuspeed_turbos09:40:122
3312569991310,2cyclictest3453421-21kthreadcore09:25:172
3312569991310,1cyclictest3603548-21irqcore11:50:162
331256599138,4cyclictest3348881-21kworker/u8:3+flush-259:007:55:251
331256599138,3cyclictest252-21systemd-journal11:45:001
331256599131,11cyclictest0-21swapper/110:50:001
3312565991311,1cyclictest0-21swapper/109:35:001
3312565991310,2cyclictest3648856-21unixbench_singl12:30:251
3312565991310,2cyclictest3591636-21unixbench_multi11:35:251
3312565991310,2cyclictest3524639-21idleruntime-cro10:35:001
3312565991310,2cyclictest3482620-21unixbench_singl09:50:251
3312565991310,2cyclictest0-21swapper/108:10:011
3312565991310,2cyclictest0-21swapper/107:20:001
331256399139,3cyclictest3427617-21kworker/u8:1+events_unbound09:00:170
331256399131,2cyclictest33802771latency_hist08:15:000
3312563991311,1cyclictest60350irq/132-enp0s29f1:rx-009:43:250
3312563991311,1cyclictest60350irq/132-enp0s29f1:rx-008:41:020
331257399128,3cyclictest500-21systemd-logind08:25:003
331257399128,3cyclictest3427617-21kworker/u8:1+flush-259:009:20:023
331257399128,3cyclictest3354065-21kworker/u8:4+flush-259:009:00:153
331257399127,3cyclictest252-21systemd-journal10:20:003
331257399126,4cyclictest3322627-21sshd07:16:343
331257399125,5cyclictest3550593-21/usr/sbin/munin11:00:143
3312573991210,1cyclictest0-21swapper/311:45:003
331256999127,4cyclictest3530469-21irqcore10:40:172
331256999126,5cyclictest3339056-21ps07:35:192
331256999125,5cyclictest3622421-21/usr/sbin/munin12:05:192
331256999125,5cyclictest3499139-21ps10:10:172
331256999125,4cyclictest3488610-21grep10:00:122
331256999124,6cyclictest3457783-21grep09:30:142
331256999122,2cyclictest3629074-21df12:15:132
331256999122,2cyclictest3440959-21ntpq09:10:212
331256599128,3cyclictest3639208-21/usr/sbin/munin12:25:161
331256599128,3cyclictest3592125-21kworker/u8:0+flush-259:011:59:051
331256599128,3cyclictest252-21systemd-journal11:20:351
331256599127,4cyclictest3427617-21kworker/u8:1+flush-259:009:20:251
3312565991210,1cyclictest0-21swapper/112:40:011
3312565991210,1cyclictest0-21swapper/112:40:001
3312565991210,1cyclictest0-21swapper/110:55:201
3312565991210,1cyclictest0-21swapper/110:00:181
3312565991210,1cyclictest0-21swapper/108:40:011
331256399128,2cyclictest60350irq/132-enp0s29f1:rx-008:35:140
3312563991210,2cyclictest60350irq/132-enp0s29f1:rx-008:25:230
331257399119,1cyclictest1-21systemd12:02:353
331257399118,2cyclictest3544229-21needreboot10:50:193
331257399118,2cyclictest3437138-21fschecks_count09:10:153
331257399117,3cyclictest3348881-21kworker/u8:3+flush-259:009:34:593
331257399115,4cyclictest3400664-21cron08:35:013
331257399113,7cyclictest0-21swapper/307:54:593
331257399111,8cyclictest3364112-21awk08:00:003
331257399111,1cyclictest35310402grep10:40:183
331256599119,1cyclictest3527202-21tr10:35:171
331256599117,3cyclictest3364145-21kworker/u8:0+flush-259:008:15:011
331256599116,3cyclictest252-21systemd-journal09:43:351
331257399108,1cyclictest0-21swapper/308:10:173
331257399107,2cyclictest3610599-21kthreadcore11:55:193
331257399105,4cyclictest3481977-21memory09:50:203
331257399104,4cyclictest3456353-21ntp_09:25:203
331257399104,4cyclictest3446514-21awk09:15:233
331257399104,4cyclictest1-21systemd08:40:003
331256599107,2cyclictest0-21swapper/110:25:181
331256599106,3cyclictest3472572-21kworker/u8:0+flush-259:009:59:061
33125739997,1cyclictest0-21swapper/310:40:013
33125739996,2cyclictest0-21swapper/310:45:143
33125739994,3cyclictest252-21systemd-journal08:17:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional