You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 00:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Tue Dec 09, 2025 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3727274993929,9cyclictest3800274-21kworker/u8:3+flush-259:009:00:010
3727285993524,8cyclictest3747836-21latency_hist07:30:013
3727281993532,2cyclictest4043920-21kworker/u8:2+events_unbound12:25:152
3727274993531,3cyclictest3810788-21/usr/sbin/munin08:30:130
3727285993430,3cyclictest3909008-21kworker/u8:4+flush-259:010:25:003
3727285993425,8cyclictest3929376-21kworker/u8:0+flush-259:011:00:003
3727274993429,4cyclictest4028224-21/usr/sbin/munin12:00:230
3727285993123,5cyclictest3815703-21latency_hist08:35:013
3727281993125,4cyclictest2698871-21systemd-journal10:00:302
3727285993025,3cyclictest3758165-21kworker/u8:1+flush-259:007:50:003
3727281993028,2cyclictest3831142-21kworker/u8:0+flush-259:009:05:182
3727281993027,2cyclictest3929376-21kworker/u8:0+flush-259:010:40:312
3727281993027,2cyclictest3929376-21kworker/u8:0+flush-259:010:40:312
3727281993027,2cyclictest3862024-21kworker/u8:2+events_unbound09:50:152
3727281992926,2cyclictest3800274-21kworker/u8:3+flush-259:008:20:242
3727279992925,3cyclictest3768512-21kworker/u8:4+flush-259:007:55:251
3727274992923,5cyclictest2698871-21systemd-journal11:15:000
3727279992825,2cyclictest3747863-21kworker/u8:0+events_unbound07:45:151
3727279992825,2cyclictest3673610-21kworker/u8:4+events_unbound07:20:211
3727279992824,3cyclictest3856791-21kworker/u8:1+flush-259:009:45:251
3727274992824,3cyclictest3940367-21/usr/sbin/munin10:35:160
3727274992823,4cyclictest3778986-21base6407:59:590
3727285992714,11cyclictest2698871-21systemd-journal08:10:003
3727281992725,2cyclictest3929376-21kworker/u8:0+flush-259:012:00:222
3727281992723,3cyclictest3862024-21kworker/u8:2+flush-259:010:05:222
3727281992723,3cyclictest3821050-21kworker/u8:4+flush-259:008:50:002
3727281992721,4cyclictest3976037-21latency_hist11:10:022
3727274992722,4cyclictest4064181-21ntpq12:30:220
3727274992722,4cyclictest3867805-21kworker/u8:3+flush-259:009:30:180
3727281992623,2cyclictest3800274-21kworker/u8:3+flush-259:008:40:162
3727281992621,4cyclictest3831142-21kworker/u8:0+flush-259:008:55:162
3727274992620,4cyclictest3986344-21kworker/u8:2+flush-259:011:25:190
3727285992520,3cyclictest3846824-21apt-get09:05:123
3727281992522,2cyclictest3914615-21kworker/u8:1+flush-259:010:20:002
3727281992522,2cyclictest3867805-21kworker/u8:3+flush-259:009:45:232
3727281992522,2cyclictest3773904-21kworker/u8:2+events_unbound08:32:312
3727281992520,3cyclictest3862024-21kworker/u8:2+flush-259:009:40:012
3727279992522,2cyclictest3960672-21kworker/u8:1+flush-259:011:05:161
3727279992522,2cyclictest3800274-21kworker/u8:3+flush-259:008:39:591
3727279992521,3cyclictest3929376-21kworker/u8:0+flush-259:011:00:201
3727274992519,4cyclictest3867805-21kworker/u8:3+flush-259:009:55:170
372727499251,21cyclictest3872780-21munin-run09:30:010
3727285992418,4cyclictest4049055-21latency_hist12:20:013
3727285992418,4cyclictest3950422-21latency_hist10:45:013
3727285992418,4cyclictest3950422-21latency_hist10:45:003
3727279992421,2cyclictest3919785-21kworker/u8:2+events_unbound10:16:321
3727279992421,2cyclictest3773904-21kworker/u8:2+flush-259:008:30:211
3727279992420,3cyclictest3909008-21kworker/u8:4+events_unbound10:48:311
3727281992319,3cyclictest3773904-21kworker/u8:2+events_unbound08:15:042
3727279992320,2cyclictest3929376-21kworker/u8:0+flush-259:011:55:211
3727279992320,2cyclictest3758165-21kworker/u8:1+events_unbound07:54:351
3727279992319,3cyclictest3909008-21kworker/u8:4+events_unbound10:59:301
3727274992317,4cyclictest4002404-21base6411:35:010
3727281992219,2cyclictest4043920-21kworker/u8:2+events_unbound12:35:222
3727281992219,2cyclictest3991711-21kworker/u8:3+events_unbound11:30:192
3727281992219,2cyclictest3929376-21kworker/u8:0+events_unbound10:35:172
3727281992219,2cyclictest3867805-21kworker/u8:3+flush-259:009:30:162
3727281992218,3cyclictest3789582-21kworker/u8:1+events_unbound08:10:142
3727281992218,3cyclictest3789582-21kworker/u8:1+events_unbound08:10:132
3727281992217,4cyclictest2698871-21systemd-journal07:25:002
3727279992219,2cyclictest3929376-21kworker/u8:0+events_unbound11:50:221
3727279992219,2cyclictest3000063-21rs:main1
3727279992218,3cyclictest3991711-21kworker/u8:3+flush-259:011:30:251
3727279992218,3cyclictest3673610-21kworker/u8:4+flush-259:007:25:171
3727279992218,3cyclictest2698871-21systemd-journal12:34:591
3727279992216,4cyclictest4038812-21cat12:10:001
3727285992119,1cyclictest0-21swapper/312:39:593
3727281992119,2cyclictest4038008-21kworker/u8:1+events_unbound12:20:252
3727281992118,2cyclictest3986344-21kworker/u8:2+events_unbound11:23:322
3727279992119,1cyclictest3986344-21kworker/u8:2+flush-259:011:20:261
3727279992118,2cyclictest4038008-21kworker/u8:1+flush-259:012:15:201
3727279992118,2cyclictest3986344-21kworker/u8:2+events_unbound11:25:171
3727279992118,2cyclictest3773904-21kworker/u8:2+flush-259:008:05:161
3727279992118,2cyclictest3747863-21kworker/u8:0+flush-259:008:20:201
3727279992115,5cyclictest3898495-21latency_hist09:55:001
3727274992117,3cyclictest3909008-21kworker/u8:4+flush-259:010:45:000
3727274992117,3cyclictest3909008-21kworker/u8:4+flush-259:010:45:000
3727285992014,4cyclictest3955560-21latency_hist10:50:003
3727285992011,1cyclictest0-21swapper/312:29:593
3727281992017,2cyclictest4002436-21kworker/u8:4+flush-259:011:40:002
3727281992017,2cyclictest3929376-21kworker/u8:0+events_unbound11:15:302
3727281992017,2cyclictest3821050-21kworker/u8:4+events_unbound09:15:232
3727281992017,2cyclictest3810781-21kworker/u8:1+events_unbound08:35:242
3727281992017,2cyclictest3773904-21kworker/u8:2+events_unbound07:58:322
3727281992016,3cyclictest3929376-21kworker/u8:0+events_unbound11:40:222
3727281992015,4cyclictest2698871-21systemd-journal08:52:292
3727279992017,2cyclictest3856791-21kworker/u8:1+events_unbound09:30:261
3727279992017,2cyclictest3856791-21kworker/u8:1+events_unbound09:26:301
3727279992017,2cyclictest3732607-21kworker/u8:3+flush-259:007:15:231
3727279992017,2cyclictest0-21swapper/109:25:011
3727279992016,3cyclictest3862024-21kworker/u8:2+events_unbound09:35:141
3727279992016,3cyclictest3773904-21kworker/u8:2+events_unbound08:15:121
3727279992015,4cyclictest2698871-21systemd-journal10:41:301
3727279992015,4cyclictest2698871-21systemd-journal10:41:301
3727274992015,3cyclictest2698871-21systemd-journal11:19:310
3727274992011,6cyclictest4022037-21ntpq11:50:240
3727285991915,3cyclictest3867805-21kworker/u8:3+flush-259:009:30:013
3727285991915,2cyclictest3945260-21cron10:40:003
3727285991913,4cyclictest3779502-21/usr/sbin/munin08:00:163
3727285991913,4cyclictest2698871-21systemd-journal08:20:013
3727281991916,2cyclictest4043920-21kworker/u8:2+events_unbound12:31:312
3727281991916,2cyclictest3867557-21idleruntime-cro09:25:012
3727281991915,3cyclictest3929376-21kworker/u8:0+events_unbound11:00:172
3727281991914,3cyclictest3000063-21rs:main2
3727281991914,3cyclictest2698871-21systemd-journal07:51:302
3727279991916,2cyclictest3960672-21kworker/u8:1+events_unbound11:15:221
3727279991916,2cyclictest3909008-21kworker/u8:4+flush-259:010:10:261
3727279991915,3cyclictest3867805-21kworker/u8:3+events_unbound10:07:301
3727279991915,3cyclictest3836530-21kworker/u8:1+events_unbound08:55:171
3727279991914,3cyclictest2698871-21systemd-journal11:47:311
3727274991912,5cyclictest3887247-21ntpq09:40:200
3727274991912,5cyclictest3851730-21latency_hist09:10:010
3727274991912,4cyclictest3732377-21latency_hist07:15:010
3727285991815,2cyclictest4008072-21irqcore11:40:183
3727285991814,3cyclictest3970779-21kworker/u8:2+flush-259:011:05:003
3727285991811,5cyclictest3882116-21ntpq09:35:213
3727281991816,1cyclictest0-21swapper/207:15:012
3727281991815,2cyclictest3929376-21kworker/u8:0+events_unbound10:25:282
3727281991815,2cyclictest3909008-21kworker/u8:4+flush-259:010:45:162
3727281991815,2cyclictest3747863-21kworker/u8:0+flush-259:007:45:242
3727281991814,3cyclictest3909008-21kworker/u8:4+events_unbound10:33:302
3727281991814,3cyclictest3778836-21apt-get08:00:002
3727281991814,3cyclictest3732607-21kworker/u8:3+events_unbound07:39:322
3727281991813,4cyclictest3918669-21ntpq10:10:232
3727279991815,2cyclictest3929376-21kworker/u8:0+flush-259:010:30:151
3727279991815,2cyclictest3800274-21kworker/u8:3+flush-259:009:00:181
3727279991815,2cyclictest3800274-21kworker/u8:3+events_unbound08:25:211
3727274991814,3cyclictest3747863-21kworker/u8:0+flush-259:007:45:250
3727274991812,4cyclictest3867805-21kworker/u8:3+flush-259:010:05:180
3727285991714,2cyclictest4065022-21idleruntime-cro12:35:003
3727285991714,2cyclictest3841387-21idleruntime-cro09:00:003
3727285991713,3cyclictest3862024-21kworker/u8:2+flush-259:009:50:153
3727285991712,3cyclictest3909008-21kworker/u8:4+flush-259:010:55:003
3727285991711,4cyclictest3862013-21latency_hist09:20:013
3727281991714,2cyclictest4038332-21sh12:09:592
3727281991714,2cyclictest3856791-21kworker/u8:1+events_unbound09:44:312
3727279991714,2cyclictest4038301-21latency_hist12:09:591
3727279991714,2cyclictest3800274-21kworker/u8:3+events_unbound08:45:291
3727279991714,2cyclictest3673610-21kworker/u8:4+events_unbound07:30:201
3727279991713,3cyclictest3909008-21kworker/u8:4+flush-259:010:20:201
3727279991712,3cyclictest2698871-21systemd-journal12:37:301
372727499179,6cyclictest2698871-21systemd-journal08:49:590
3727274991713,3cyclictest60650irq/133-enp0s29f1:rx-008:05:210
3727274991713,2cyclictest3805418-21turbostat08:25:000
3727274991712,4cyclictest3795408-21apt-get08:15:120
3727274991712,1cyclictest60650irq/133-enp0s29f1:rx-011:20:170
3727274991711,4cyclictest3835342-21ntpq08:50:220
3727285991613,2cyclictest3909219-21sh10:05:003
3727285991613,1cyclictest0-21swapper/308:29:593
3727281991614,2cyclictest3800274-21kworker/u8:3+events_unbound09:00:222
3727281991613,2cyclictest3929376-21kworker/u8:0+flush-259:011:45:112
3727279991613,2cyclictest3909008-21kworker/u8:4+events_unbound10:35:211
3727279991613,2cyclictest3800274-21kworker/u8:3+flush-259:008:40:261
3727279991613,2cyclictest3747863-21kworker/u8:0+events_unbound08:15:001
3727279991613,2cyclictest3747863-21kworker/u8:0+events_unbound08:14:591
3727279991610,5cyclictest3762460-21ntpq07:40:231
3727274991613,2cyclictest61450irq/134-enp0s29f1:tx-010:10:270
3727274991613,2cyclictest61450irq/134-enp0s29f1:tx-008:10:140
3727274991613,2cyclictest61450irq/134-enp0s29f1:tx-008:10:130
3727274991613,2cyclictest60650irq/133-enp0s29f1:rx-012:35:260
3727274991613,2cyclictest60650irq/133-enp0s29f1:rx-012:23:000
3727274991613,2cyclictest60650irq/133-enp0s29f1:rx-012:14:290
3727274991613,2cyclictest60650irq/133-enp0s29f1:rx-011:46:510
3727274991613,2cyclictest60650irq/133-enp0s29f1:rx-009:00:330
3727274991613,2cyclictest60650irq/133-enp0s29f1:rx-008:26:120
3727274991613,2cyclictest60650irq/133-enp0s29f1:rx-007:50:010
3727274991613,2cyclictest3000063-21rs:main0
3727274991612,2cyclictest60650irq/133-enp0s29f1:rx-009:50:110
3727274991611,3cyclictest3815941-21/usr/sbin/munin08:35:140
372728599159,5cyclictest3913488-21ntpq10:05:223
372728599158,5cyclictest4033123-21latency_hist12:05:013
3727285991511,3cyclictest3929376-21kworker/u8:0+events_unbound12:06:323
3727285991511,3cyclictest3826222-21/usr/sbin/munin08:45:163
372728199159,2cyclictest3732607-21kworker/u8:3+events_unbound07:15:162
3727281991513,2cyclictest3773904-21kworker/u8:2+flush-259:008:05:262
3727281991512,2cyclictest3929376-21kworker/u8:0+flush-259:011:55:002
3727279991513,2cyclictest3867805-21kworker/u8:3+events_unbound09:55:241
3727279991512,2cyclictest3960672-21kworker/u8:1+flush-259:011:10:161
3727279991512,2cyclictest3747863-21kworker/u8:0+flush-259:008:05:001
3727279991511,3cyclictest3909008-21kworker/u8:4+flush-259:010:50:211
3727274991513,2cyclictest60650irq/133-enp0s29f1:rx-010:00:430
3727274991513,1cyclictest60650irq/133-enp0s29f1:rx-010:29:080
3727274991512,2cyclictest60650irq/133-enp0s29f1:rx-010:23:130
3727274991512,2cyclictest60650irq/133-enp0s29f1:rx-010:16:240
3727274991512,2cyclictest60650irq/133-enp0s29f1:rx-009:48:150
3727274991512,2cyclictest60650irq/133-enp0s29f1:rx-009:10:140
372727499151,12cyclictest3971588-21/usr/sbin/munin11:05:170
3727274991511,2cyclictest3763383-21turbostat07:44:590
3727274991510,3cyclictest3929376-21kworker/u8:0+flush-259:010:35:000
372728599148,4cyclictest3867566-21latency_hist09:25:013
372728599148,4cyclictest3789799-21/usr/sbin/munin08:10:213
372728599148,4cyclictest3789799-21/usr/sbin/munin08:10:203
372728599148,4cyclictest3768512-21kworker/u8:4+flush-259:007:55:173
3727285991412,1cyclictest0-21swapper/311:15:003
3727285991410,3cyclictest4012811-21/usr/sbin/munin11:45:133
372728199149,4cyclictest3748078-21/usr/sbin/munin07:30:192
3727281991411,2cyclictest3929376-21kworker/u8:0+events_unbound12:10:302
3727281991410,3cyclictest3673610-21kworker/u8:4+events_unbound07:25:152
3727279991411,2cyclictest3800274-21kworker/u8:3+events_unbound09:12:321
3727279991411,2cyclictest3570980-21kworker/u8:0+flush-259:007:10:221
372727499148,4cyclictest2702718-21systemd-udevd11:04:560
3727274991412,2cyclictest60650irq/133-enp0s29f1:rx-007:35:320
3727274991412,1cyclictest60650irq/133-enp0s29f1:rx-010:55:180
3727274991412,1cyclictest60650irq/133-enp0s29f1:rx-007:15:230
3727274991411,2cyclictest61450irq/134-enp0s29f1:tx-011:30:190
3727274991411,2cyclictest60650irq/133-enp0s29f1:rx-012:27:590
372728599139,3cyclictest3732607-21kworker/u8:3+flush-259:007:39:003
372728599138,4cyclictest3867805-21kworker/u8:3+flush-259:009:45:203
372728599138,4cyclictest2698871-21systemd-journal08:25:003
372728599138,3cyclictest4023081-21kworker/u8:3+flush-259:012:10:013
372728599137,4cyclictest3996883-21apt-get11:30:123
372728199139,3cyclictest3747863-21kworker/u8:0+events_unbound08:26:322
3727281991311,2cyclictest3800274-21kworker/u8:3+events_unbound09:10:312
372727999139,3cyclictest3929376-21kworker/u8:0+flush-259:010:29:561
3727279991311,1cyclictest0-21swapper/112:05:001
3727279991310,2cyclictest3851275-21unixbench_singl09:05:251
3727279991310,2cyclictest3821050-21kworker/u8:4+flush-259:009:15:191
3727279991310,2cyclictest0-21swapper/112:30:001
372727499137,5cyclictest60650irq/133-enp0s29f1:rx-010:45:130
3727274991311,1cyclictest0-21swapper/007:30:000
372727499131,10cyclictest2698871-21systemd-journal07:34:590
3727274991310,2cyclictest3862759-21cpuspeed_turbos09:20:140
372728599129,2cyclictest3852397-21sed09:10:173
372728599129,2cyclictest3002-21cron07:25:003
372728199127,4cyclictest4023113-21apt-get11:55:122
3727281991210,2cyclictest3960672-21kworker/u8:1+flush-259:011:10:262
372727999129,2cyclictest4038008-21kworker/u8:1+events_unbound12:20:191
372727999127,4cyclictest4007387-21latency_hist11:40:011
3727279991210,2cyclictest3800274-21kworker/u8:3+flush-259:008:50:221
372727499128,3cyclictest3862024-21kworker/u8:2+flush-259:009:35:170
372728599119,1cyclictest0-21swapper/308:50:263
372728599118,2cyclictest4002727-21fschecks_time11:35:153
372728599118,2cyclictest0-21swapper/310:30:003
372728599117,3cyclictest3800274-21kworker/u8:3+flush-259:008:39:353
372728599110,3cyclictest3753001-21latency_hist07:35:013
372728199118,1cyclictest0-21swapper/207:10:002
372727999117,3cyclictest3909008-21kworker/u8:4+events_unbound10:04:581
372728599108,1cyclictest0-21swapper/310:10:193
372728599105,4cyclictest4023376-21uniq11:55:143
37272859995,3cyclictest3960672-21kworker/u8:1+flush-259:011:05:173
37272859995,3cyclictest3877032-21ntpq09:30:233
37272859994,3cyclictest3821278-21cut08:40:133
37272859984,3cyclictest4018111-21cut11:50:153
37272859984,3cyclictest3988675-21cut11:20:203
37272859984,3cyclictest3982104-21cut11:15:193
37272859984,3cyclictest3923343-21cut10:15:193
37272859980,2cyclictest3940261-21cat10:35:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional