You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 04:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Sun Nov 23, 2025 00:44:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3408761994942,5cyclictest3683137-21latency_hist23:35:003
3408761994539,4cyclictest3476165-21latency_hist20:15:013
3408757994035,4cyclictest3492254-21kworker/u8:1+flush-259:021:20:002
3408751994036,3cyclictest3512412-21gpgv20:50:010
3408757993628,5cyclictest3419070-21latency_hist19:20:002
3408757993529,4cyclictest3678008-21latency_hist23:30:002
3408755993428,4cyclictest2698871-21systemd-journal22:35:011
3408755993328,3cyclictest3424247-21latency_hist19:25:001
3408761993226,4cyclictest3512292-21latency_hist20:50:013
3408755993126,3cyclictest3413937-21latency_hist19:15:001
3408757993023,5cyclictest3517017-21ntpq20:50:222
3408757993022,7cyclictest3002-21cron00:05:002
3408757993022,7cyclictest3002-21cron00:05:002
3408757992924,3cyclictest3590144-21/usr/sbin/munin22:05:142
3408757992923,4cyclictest3647069-21latency_hist23:00:012
3408751992926,2cyclictest3678026-21kworker/u8:1+flush-259:023:35:120
3408757992823,3cyclictest3739677-21apt-get00:30:002
3408757992822,4cyclictest3708806-21latency_hist00:00:012
3408751992822,4cyclictest3600228-21latency_hist22:15:010
3408761992719,5cyclictest3481326-21latency_hist20:20:003
3408755992721,4cyclictest3641527-21latency_hist22:55:011
3408757992622,3cyclictest3709085-21kworker/u8:3+flush-259:000:10:172
3408751992623,2cyclictest3709085-21kworker/u8:3+flush-259:000:05:130
3408751992622,3cyclictest3621133-21kworker/u8:3+flush-259:022:40:250
3408755992520,4cyclictest3739707-21gpgv00:30:001
3408755992519,4cyclictest2698871-21systemd-journal00:34:341
3408751992520,4cyclictest3492254-21kworker/u8:1+events_unbound20:55:000
3408757992421,2cyclictest3465878-21kworker/u8:0+flush-259:020:30:252
3408757992418,4cyclictest3439681-21latency_hist19:40:002
3408755992421,2cyclictest3445077-21apt-get19:45:111
3408751992420,3cyclictest3647293-21kworker/u8:4+flush-259:023:25:220
3408757992319,3cyclictest3693118-21kworker/u8:0+flush-259:023:45:182
3408755992319,3cyclictest3709085-21kworker/u8:3+flush-259:000:00:201
3408755992319,3cyclictest3709085-21kworker/u8:3+flush-259:000:00:201
3408755992318,4cyclictest3349678-21kworker/u8:0+flush-259:019:35:011
3408755992318,4cyclictest3349678-21kworker/u8:0+flush-259:019:35:001
3408755992318,3cyclictest2698871-21systemd-journal20:22:171
3408755992317,3cyclictest3492254-21kworker/u8:1+events_unbound21:00:201
3408751992319,3cyclictest3719137-21kworker/u8:1+flush-259:000:35:250
3408751992319,3cyclictest3719137-21kworker/u8:1+events_unbound00:30:150
3408751992319,3cyclictest3387789-21kworker/u8:3+events_unbound19:50:020
3408761992219,2cyclictest0-21swapper/321:35:003
3408757992219,2cyclictest3349678-21kworker/u8:0+flush-259:019:45:162
3408757992216,4cyclictest0-21swapper/221:35:002
3408755992219,2cyclictest3455791-21kworker/u8:4+flush-259:020:00:201
340875599221,19cyclictest1-21systemd22:45:011
3408751992219,2cyclictest3349678-21kworker/u8:0+flush-259:019:40:270
3408751992218,3cyclictest3678026-21kworker/u8:1+flush-259:023:50:240
3408751992218,3cyclictest3349678-21kworker/u8:0+events_unbound19:45:190
3408751992218,3cyclictest3349678-21kworker/u8:0+events_unbound19:10:170
3408751992216,4cyclictest3528171-21latency_hist21:05:010
3408751992216,4cyclictest3492254-21kworker/u8:1+events_unbound21:20:180
3408751992216,4cyclictest3434543-21latency_hist19:35:010
3408751992216,4cyclictest3434543-21latency_hist19:35:000
3408761992117,3cyclictest3595068-21kworker/u8:1+flush-259:022:15:003
3408761992115,4cyclictest3492254-21kworker/u8:1+flush-259:021:05:183
3408757992118,2cyclictest3693118-21kworker/u8:0+flush-259:023:50:182
3408757992117,3cyclictest3465878-21kworker/u8:0+flush-259:020:20:262
3408757992115,4cyclictest3693393-21latency_hist23:45:002
3408755992116,3cyclictest3455828-21apt-get19:55:151
3408751992118,2cyclictest3672871-21latency_hist23:25:000
3408751992118,2cyclictest3600459-21kworker/u8:0+events_unbound22:15:230
3408751992117,3cyclictest3693118-21kworker/u8:0+events_unbound00:14:420
3408751992117,3cyclictest3621133-21kworker/u8:3+flush-259:022:50:270
3408761992013,5cyclictest3496834-21latency_hist20:35:003
3408757992017,2cyclictest3360429-21kworker/u8:1+events_unbound19:30:002
3408755992016,3cyclictest3719137-21kworker/u8:1+flush-259:000:35:161
3408755992015,3cyclictest3662547-21sh23:15:001
3408751992016,3cyclictest3657663-21apt-get23:10:120
3408751992016,3cyclictest3620946-21kworker/u8:1+events_unbound22:47:560
3408751992016,3cyclictest3466091-21kworker/u8:2+events_unbound21:30:180
3408751992014,4cyclictest2698871-21systemd-journal20:15:000
3408757991916,2cyclictest3466091-21kworker/u8:2+flush-259:020:27:222
3408757991914,4cyclictest2698871-21systemd-journal23:02:422
3408757991913,4cyclictest3492254-21kworker/u8:1+flush-259:021:00:182
3408757991912,5cyclictest2698871-21systemd-journal00:16:422
3408755991916,2cyclictest3000063-21rs:main1
3408755991915,2cyclictest3647087-21kworker/u8:2+flush-259:023:40:171
3408755991914,4cyclictest3444835-21kworker/u8:1+events_unbound20:10:171
3408751991915,3cyclictest3647087-21kworker/u8:2+events_unbound23:40:230
3408751991915,3cyclictest3548753-21kworker/u8:0+events_unbound21:44:430
3408751991915,3cyclictest3548753-21kworker/u8:0+events_unbound21:44:420
3408751991915,3cyclictest3548753-21kworker/u8:0+events_unbound21:37:430
3408751991915,3cyclictest3466091-21kworker/u8:2+events_unbound20:44:410
3408751991912,5cyclictest2698871-21systemd-journal22:10:000
340876199189,7cyclictest3517473-21unixbench_singl20:50:263
3408761991815,2cyclictest3628651-21kthreadcore22:40:183
3408761991814,2cyclictest3387789-21kworker/u8:3+flush-259:019:15:163
3408761991813,4cyclictest3466127-21apt-get20:05:133
3408757991814,3cyclictest3548753-21kworker/u8:0+flush-259:021:59:592
3408757991814,3cyclictest3360429-21kworker/u8:1+flush-259:019:20:172
3408755991815,2cyclictest3651311-21ntp_kernel_pll_23:00:211
3408755991814,3cyclictest3349678-21kworker/u8:0+flush-259:019:25:151
3408755991813,3cyclictest3466091-21kworker/u8:2+flush-259:021:50:171
3408751991815,2cyclictest3636409-21kworker/u8:0+flush-259:023:05:110
3408751991815,2cyclictest3636409-21kworker/u8:0+flush-259:023:00:150
3408751991815,2cyclictest3466091-21kworker/u8:2+flush-259:020:25:420
3408751991814,3cyclictest3719137-21kworker/u8:1+flush-259:000:20:260
3408751991814,3cyclictest3620946-21kworker/u8:1+events_unbound22:35:230
3408751991814,3cyclictest3466091-21kworker/u8:2+events_unbound21:55:280
3408751991814,3cyclictest3360429-21kworker/u8:1+events_unbound19:21:440
3408751991814,3cyclictest3349678-21kworker/u8:0+events_unbound19:25:200
3408751991814,3cyclictest0-21swapper/020:59:590
340875199180,16cyclictest1-21systemd20:31:430
340876199174,10cyclictest3688457-21sed23:40:003
340876199172,11cyclictest3652475-21apt-get23:05:113
340876199171,4cyclictest3744360-21ntpq00:30:223
3408761991713,3cyclictest3661660-21ntp_kernel_pll_23:10:223
3408761991711,4cyclictest3729204-40mandb00:17:563
340876199171,14cyclictest3667698-21idleruntime-cro23:20:003
340876199171,14cyclictest1-21systemd21:14:593
340876199170,15cyclictest3688954-21cut23:40:173
3408757991714,2cyclictest3600459-21kworker/u8:0+flush-259:022:15:172
3408757991713,3cyclictest3349678-21kworker/u8:0+flush-259:019:10:212
3408757991712,3cyclictest3465878-21kworker/u8:0+flush-259:020:35:192
3408755991714,2cyclictest3548753-21kworker/u8:0+flush-259:021:45:001
3408755991714,2cyclictest3548753-21kworker/u8:0+flush-259:021:45:001
3408751991714,2cyclictest60650irq/133-enp0s29f1:rx-021:11:170
3408751991713,3cyclictest3481567-21/usr/sbin/munin20:20:170
3408751991713,3cyclictest3466091-21kworker/u8:2+flush-259:022:00:210
3408751991710,5cyclictest2698871-21systemd-journal21:05:410
340876199165,8cyclictest3729918-21tune2fs00:20:173
340876199162,11cyclictest3605836-21sort22:20:153
340876199161,3cyclictest3605398-21latency_hist22:20:003
340876199161,3cyclictest3579590-21sh21:55:003
340876199161,2cyclictest3713067-21ntpq00:00:243
340876199161,2cyclictest3713067-21ntpq00:00:233
340876199161,2cyclictest3708792-21cron00:00:003
340876199161,2cyclictest3671966-21ntpq23:20:213
340876199161,2cyclictest3635496-21ntpq22:45:213
340876199161,2cyclictest3563361-21ntpq21:35:253
340876199161,2cyclictest3506298-21ntpq20:40:233
340876199161,2cyclictest3485583-21ntpq20:20:233
340876199161,2cyclictest3413095-21ntpq19:10:243
340876199161,1cyclictest3443908-21ntpq19:40:233
3408761991611,3cyclictest3615742-21turbostat22:34:553
340876199161,13cyclictest3445488-21tr19:45:173
340876199161,12cyclictest3486842-21tr20:25:173
3408761991610,4cyclictest3620946-21kworker/u8:1+flush-259:022:35:183
340876199160,5cyclictest3583670-21/usr/sbin/munin21:55:193
340876199160,4cyclictest3522393-21/usr/sbin/munin20:55:273
3408757991613,2cyclictest3468429-21kthreadcore20:05:192
3408757991613,2cyclictest0-21swapper/200:25:002
3408757991612,3cyclictest3492254-21kworker/u8:1+flush-259:020:50:002
3408757991612,3cyclictest3444835-21kworker/u8:1+flush-259:019:55:192
3408757991611,4cyclictest3672003-21ntpq23:20:222
340875599169,5cyclictest3615747-21latency_hist22:30:011
3408755991613,2cyclictest3647087-21kworker/u8:2+flush-259:023:05:181
3408755991612,3cyclictest3492254-21kworker/u8:1+flush-259:021:45:181
3408755991612,3cyclictest3492254-21kworker/u8:1+flush-259:021:45:171
3408755991612,3cyclictest3000061-21in:imuxsock22:15:001
3408755991612,2cyclictest0-21swapper/122:50:011
3408755991611,3cyclictest3584982-21/usr/sbin/munin22:00:151
340875199168,5cyclictest3682314-21ntpq23:30:220
340875199167,6cyclictest2698871-21systemd-journal00:20:000
3408751991612,3cyclictest3647293-21kworker/u8:4+events_unbound23:18:010
3408751991612,3cyclictest3600459-21kworker/u8:0+flush-259:022:25:210
3408751991612,3cyclictest3465878-21kworker/u8:0+events_unbound20:35:160
3408751991612,3cyclictest3387789-21kworker/u8:3+events_unbound19:37:100
3408751991612,3cyclictest3387789-21kworker/u8:3+events_unbound19:15:180
3408751991611,4cyclictest3492254-21kworker/u8:1+events_unbound21:17:400
340876199153,10cyclictest3526823-21cut21:00:193
340876199153,10cyclictest3423336-21ntpq19:20:223
340876199152,10cyclictest3434803-21apt-get19:35:133
340876199151,6cyclictest3640701-21ntpq22:50:243
340876199151,4cyclictest3714637-21irqcore00:05:153
340876199151,4cyclictest3501543-21tail20:35:273
340876199151,4cyclictest3450771-21tr19:50:133
340876199151,3cyclictest3749527-21ntpq00:35:213
340876199151,2cyclictest3571882-21grep21:45:203
340876199151,2cyclictest3571882-21grep21:45:193
340876199151,2cyclictest3564708-21fschecks_count21:40:163
340876199151,2cyclictest3564708-21fschecks_count21:40:153
340876199151,2cyclictest3547887-21ntpq21:20:233
340876199151,2cyclictest3538907-21tune2fs21:15:153
340876199151,2cyclictest3464954-21ntpq20:00:233
340876199151,2cyclictest3433716-21ntpq19:30:253
340876199151,2cyclictest3433716-21ntpq19:30:243
340876199151,2cyclictest0-21swapper/300:25:153
340876199151,1cyclictest3589016-21ntpq22:00:213
340876199150,5cyclictest3553469-21/usr/sbin/munin21:25:263
340876199150,5cyclictest3459829-21ntpq19:55:253
340876199150,4cyclictest3719793-21sed00:10:173
340876199150,3cyclictest492-21dbus-daemon23:50:003
340876199150,3cyclictest3648259-21cut23:00:163
340876199150,3cyclictest3590344-21fschecks_count22:05:143
340876199150,3cyclictest3424930-21sed19:25:183
340876199150,2cyclictest3698797-21chrt23:50:123
340876199150,12cyclictest3673233-21cut23:25:153
340875799158,5cyclictest3740187-21cat00:30:012
3408757991511,3cyclictest3600459-21kworker/u8:0+flush-259:022:25:202
3408757991511,3cyclictest3455791-21kworker/u8:4+flush-259:020:18:462
3408757991510,4cyclictest3647087-21kworker/u8:2+flush-259:023:35:052
340875599159,4cyclictest3626084-21latency_hist22:40:001
340875599159,4cyclictest3486685-21/usr/sbin/munin20:25:231
3408755991512,2cyclictest3558161-21ntp_kernel_pll_21:30:201
3408755991512,2cyclictest3466091-21kworker/u8:2+flush-259:021:10:171
3408755991512,1cyclictest0-21swapper/122:09:591
3408755991511,3cyclictest3673130-21apt-get23:25:141
3408755991511,3cyclictest3600459-21kworker/u8:0+flush-259:022:20:191
340875599150,4cyclictest35199811kthreadcore20:55:191
340875199157,6cyclictest3573563-21ntpq21:45:230
340875199157,6cyclictest3573563-21ntpq21:45:230
3408751991511,3cyclictest3703715-21kworker/u8:2+events_unbound00:00:150
3408751991511,3cyclictest3703715-21kworker/u8:2+events_unbound00:00:150
3408751991511,3cyclictest3678026-21kworker/u8:1+events_unbound23:55:250
3408751991511,3cyclictest3466091-21kworker/u8:2+events_unbound21:25:280
340876199140,7cyclictest3614916-21ntpq22:25:223
340876199140,2cyclictest3642342-21tr22:55:143
340875799147,5cyclictest2698871-21systemd-journal23:10:012
3408757991411,2cyclictest3600459-21kworker/u8:0+events_unbound22:20:222
3408757991411,2cyclictest3430088-21irqcore19:30:192
3408757991411,2cyclictest3430088-21irqcore19:30:182
3408757991410,3cyclictest3615979-21/usr/sbin/munin22:30:252
340875599149,4cyclictest3718194-21ntpq00:05:191
3408755991411,2cyclictest3706226-21sort23:55:201
3408755991411,2cyclictest3543570-21sh21:20:001
3408755991410,3cyclictest3387789-21kworker/u8:3+events_unbound19:15:201
3408751991412,1cyclictest60650irq/133-enp0s29f1:rx-019:55:180
3408751991410,3cyclictest3620946-21kworker/u8:1+events_unbound22:55:020
340876199130,4cyclictest3408786-21latency_hist19:10:003
340875799132,4cyclictest3544249-21ps21:20:182
3408757991310,2cyclictest3621133-21kworker/u8:3+flush-259:022:40:262
340875599139,3cyclictest3709085-21kworker/u8:3+flush-259:000:10:251
340875599139,3cyclictest3387789-21kworker/u8:3+flush-259:019:45:011
340875599137,4cyclictest3647087-21kworker/u8:2+events_unbound23:35:171
340875599131,6cyclictest3564233-21munin-run21:39:591
3408751991311,2cyclictest60650irq/133-enp0s29f1:rx-021:50:520
3408751991311,1cyclictest60650irq/133-enp0s29f1:rx-022:20:490
3408751991310,2cyclictest60650irq/133-enp0s29f1:rx-020:17:150
3408751991310,2cyclictest3693834-21uniq23:45:150
3408751991310,2cyclictest3466385-21fschecks_time20:05:170
340875799126,4cyclictest2698871-21systemd-journal20:45:002
340875599129,3cyclictest3719137-21kworker/u8:1+flush-259:000:20:181
340875599129,2cyclictest3439209-21unixbench_multi19:35:311
340875599129,2cyclictest0-21swapper/121:30:011
340875599126,4cyclictest2698871-21systemd-journal20:35:011
340875599125,5cyclictest3662984-21diskstats23:15:141
340875599121,2cyclictest3533277-21awk21:10:001
340875599120,2cyclictest3672873-21basename23:25:001
340875599118,2cyclictest3466091-21kworker/u8:2+flush-259:020:40:171
340875599115,4cyclictest3698768-21/usr/sbin/munin23:50:141
340875799108,1cyclictest0-21swapper/223:15:152
340875799108,1cyclictest0-21swapper/221:10:002
340875799107,2cyclictest3750437-21turbostat.cron00:40:002
340875799107,2cyclictest3636824-21fschecks_count22:50:152
340875599108,1cyclictest0-21swapper/123:30:221
34087579996,2cyclictest3559320-21sort21:35:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional