You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 05:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Thu Nov 27, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3672940994236,4cyclictest2698871-21systemd-journal20:00:002
3672939994037,2cyclictest3850117-21kworker/u8:2+events_unbound22:14:551
3672937994034,4cyclictest3762549-21latency_hist20:45:010
3672942993936,2cyclictest3757020-21kworker/u8:3+events_unbound20:50:173
3672937993932,5cyclictest2698871-21systemd-journal23:05:000
3672939993834,3cyclictest3912416-21kworker/u8:3+events_unbound00:30:231
3672940993731,4cyclictest3750949-21latency_hist20:25:012
3672940993730,5cyclictest3709284-21latency_hist19:45:002
3672939993633,2cyclictest3871123-21kworker/u8:0+events_unbound22:59:441
3672940993531,3cyclictest3871123-21kworker/u8:0+flush-259:022:40:002
3672942993428,4cyclictest3985994-21latency_hist00:20:003
3672940993428,4cyclictest3907680-21apt-get23:05:102
3672939993330,2cyclictest3861362-21kworker/u8:2+flush-259:022:30:471
3672939993329,3cyclictest3871123-21kworker/u8:0+events_unbound23:04:371
3672939993228,3cyclictest3772483-21kworker/u8:1+flush-259:021:05:341
3672939993128,2cyclictest3772483-21kworker/u8:1+flush-259:021:00:391
3672939993127,3cyclictest2698871-21systemd-journal20:55:001
3672937993124,5cyclictest3803731-21latency_hist21:25:010
3672942993023,5cyclictest3938823-21latency_hist23:35:013
3672940993026,3cyclictest3960043-21kworker/u8:0+flush-259:000:07:592
3672940993024,4cyclictest2698871-21systemd-journal20:35:002
3672939993027,2cyclictest3912416-21kworker/u8:3+events_unbound23:19:221
3672939993027,2cyclictest3876276-21kworker/u8:1+events_unbound23:06:481
3672939993027,2cyclictest3772483-21kworker/u8:1+flush-259:022:15:151
3672937993026,3cyclictest3923309-21kworker/u8:4+events_unbound23:25:230
3672942992922,5cyclictest2698871-21systemd-journal22:40:003
3672939992825,2cyclictest3772483-21kworker/u8:1+events_unbound22:05:081
3672939992824,3cyclictest3960043-21kworker/u8:0+flush-259:000:10:201
3672939992824,3cyclictest3871123-21kworker/u8:0+flush-259:022:50:001
3672942992724,2cyclictest3772483-21kworker/u8:1+flush-259:021:10:243
3672942992720,5cyclictest3824302-21latency_hist21:45:003
3672940992722,4cyclictest3960043-21kworker/u8:0+flush-259:000:04:592
3672939992724,2cyclictest3944635-21kworker/u8:1+events_unbound23:55:571
3672939992723,3cyclictest3861362-21kworker/u8:2+flush-259:022:26:241
3672939992723,3cyclictest3861362-21kworker/u8:2+flush-259:022:26:241
3672939992723,3cyclictest3693859-21kworker/u8:0+flush-259:019:43:391
3672942992622,3cyclictest3772483-21kworker/u8:1+flush-259:022:00:233
3672939992623,2cyclictest3912416-21kworker/u8:3+flush-259:023:12:171
3672939992623,2cyclictest3772483-21kworker/u8:1+events_unbound21:53:371
3672939992623,2cyclictest3767747-21kworker/u8:0+flush-259:021:15:181
3672939992623,2cyclictest3750967-21kworker/u8:0+events_unbound20:31:101
3672939992623,2cyclictest3625953-21kworker/u8:3+flush-259:019:55:261
3672939992622,3cyclictest3954966-21apt-get23:50:151
3672939992620,4cyclictest3746004-21/usr/sbin/munin20:20:131
3672937992622,3cyclictest3772483-21kworker/u8:1+flush-259:022:00:170
3672940992518,5cyclictest2698871-21systemd-journal23:40:002
3672939992522,2cyclictest3881739-21kworker/u8:2+events_unbound22:44:591
3672939992522,2cyclictest3783420-21kworker/u8:2+events_unbound21:35:211
3672939992522,2cyclictest3672392-21kworker/u8:2+events_unbound19:29:271
3672939992522,2cyclictest3625953-21kworker/u8:3+flush-259:019:47:291
3672939992522,2cyclictest3625953-21kworker/u8:3+events_unbound20:00:361
3672939992521,3cyclictest3960043-21kworker/u8:0+events_unbound00:20:341
3672939992521,3cyclictest3772483-21kworker/u8:1+events_unbound20:55:201
3672937992522,2cyclictest3635204-21kworker/u8:1+flush-259:019:10:250
3672942992417,5cyclictest3688689-21latency_hist19:25:003
3672940992419,3cyclictest3798607-21sh21:20:002
3672939992421,2cyclictest3814208-21kworker/u8:4+flush-259:021:55:151
3672939992421,2cyclictest3783420-21kworker/u8:2+events_unbound21:13:151
3672939992421,2cyclictest3772483-21kworker/u8:1+events_unbound21:29:371
3672939992421,1cyclictest0-21swapper/121:45:001
3672939992420,3cyclictest3693859-21kworker/u8:0+events_unbound19:31:381
3672937992419,3cyclictest3876276-21kworker/u8:1+flush-259:023:10:010
3672939992320,2cyclictest3912416-21kworker/u8:3+events_unbound00:36:561
3672939992319,3cyclictest3772483-21kworker/u8:1+events_unbound22:00:101
3672937992319,3cyclictest3871123-21kworker/u8:0+flush-259:022:50:250
3672942992216,4cyclictest3834555-21apt-get21:54:593
3672940992217,3cyclictest3891989-21cron22:50:012
3672939992219,2cyclictest3960043-21kworker/u8:0+events_unbound00:27:061
3672939992219,2cyclictest3960043-21kworker/u8:0+events_unbound00:01:581
3672939992219,2cyclictest3672392-21kworker/u8:2+events_unbound19:19:381
3672939992219,2cyclictest3625953-21kworker/u8:3+flush-259:020:15:001
3672939992219,2cyclictest3625953-21kworker/u8:3+flush-259:020:15:001
3672937992219,2cyclictest3912416-21kworker/u8:3+events_unbound23:50:180
3672937992218,3cyclictest3960043-21kworker/u8:0+events_unbound00:00:200
3672937992218,3cyclictest3625953-21kworker/u8:3+events_unbound19:15:200
3672937992216,4cyclictest3944532-21cp23:40:010
3672940992118,2cyclictest3635204-21kworker/u8:1+flush-259:019:30:002
3672940992117,3cyclictest3960043-21kworker/u8:0+flush-259:000:14:012
3672940992114,5cyclictest2698871-21systemd-journal23:55:012
3672939992118,2cyclictest3923309-21kworker/u8:4+events_unbound23:21:341
3672939992118,2cyclictest3735700-21kworker/u8:1+flush-259:020:40:001
3672939992118,2cyclictest3693859-21kworker/u8:0+events_unbound19:35:201
3672942992016,3cyclictest3944635-21kworker/u8:1+flush-259:023:55:163
3672940992017,2cyclictest3855617-21kworker/u8:3+flush-259:022:30:002
3672940992017,2cyclictest3855617-21kworker/u8:3+flush-259:022:30:002
3672940992017,2cyclictest3772483-21kworker/u8:1+events_unbound22:00:182
3672940992016,3cyclictest3772483-21kworker/u8:1+flush-259:021:05:012
3672940992015,3cyclictest2698871-21systemd-journal20:08:192
3672939992017,2cyclictest3912416-21kworker/u8:3+events_unbound23:30:231
3672939992017,2cyclictest3861362-21kworker/u8:2+events_unbound22:23:411
3672939992016,3cyclictest3871123-21kworker/u8:0+events_unbound22:35:091
3672939992016,3cyclictest3715077-21kworker/u8:2+events_unbound20:05:221
3672937992018,1cyclictest3625953-21kworker/u8:3+events_unbound20:15:170
3672937992016,3cyclictest3715077-21kworker/u8:2+events_unbound20:05:000
3672937992015,3cyclictest3986226-21/usr/sbin/munin00:20:140
3672942991914,3cyclictest3850845-21diskstats22:10:153
3672939991916,2cyclictest3735700-21kworker/u8:1+flush-259:020:40:101
3672939991915,3cyclictest3944014-21gpgv23:40:011
3672939991915,3cyclictest3892027-21kworker/u8:3+flush-259:022:50:151
3672939991915,3cyclictest3783420-21kworker/u8:2+flush-259:021:20:231
3672939991912,5cyclictest2698871-21systemd-journal19:55:001
3672942991815,2cyclictest3683511-21cron19:20:013
3672942991815,2cyclictest3635204-21kworker/u8:1+flush-259:019:25:163
3672942991815,2cyclictest0-21swapper/323:45:013
3672942991814,3cyclictest3824343-21kworker/u8:0+flush-259:021:45:253
3672942991814,3cyclictest3746451-21irqcore20:20:153
3672942991813,4cyclictest3866530-21apt-get22:25:133
3672942991813,4cyclictest3866530-21apt-get22:25:123
367294099180,16cyclictest1-21systemd21:51:192
3672939991816,2cyclictest3635204-21kworker/u8:1+events_unbound19:22:541
3672939991814,3cyclictest3933656-21kworker/u8:2+flush-259:023:40:231
3672939991814,3cyclictest3871123-21kworker/u8:0+events_unbound23:28:411
3672942991715,1cyclictest0-21swapper/319:50:003
3672942991712,3cyclictest3720266-21/usr/sbin/munin19:55:173
3672940991714,2cyclictest3964998-21idleruntime-cro00:00:002
3672940991713,3cyclictest3694071-21kworker/u8:4+flush-259:020:00:182
3672940991712,3cyclictest3912416-21kworker/u8:3+events_unbound00:15:162
3672940991710,5cyclictest3802795-21ntpq21:20:192
3672937991714,2cyclictest3871123-21kworker/u8:0+flush-259:022:45:250
3672937991714,2cyclictest3783420-21kworker/u8:2+flush-259:021:30:160
3672937991714,2cyclictest3625953-21kworker/u8:3+flush-259:019:45:150
367294299162,13cyclictest492-21dbus-daemon22:50:013
3672942991612,3cyclictest3912416-21kworker/u8:3+flush-259:000:35:263
3672940991613,2cyclictest3772483-21kworker/u8:1+events_unbound21:10:012
3672940991612,3cyclictest3714744-21cat19:50:002
3672939991612,3cyclictest3912416-21kworker/u8:3+events_unbound00:05:191
3672939991612,3cyclictest3772483-21kworker/u8:1+events_unbound21:47:381
3672939991611,3cyclictest3635204-21kworker/u8:1+flush-259:019:10:171
3672937991614,1cyclictest60650irq/133-enp0s29f1:rx-022:40:230
3672937991614,1cyclictest60650irq/133-enp0s29f1:rx-022:20:530
3672937991613,2cyclictest60650irq/133-enp0s29f1:rx-022:28:140
3672937991613,2cyclictest60650irq/133-enp0s29f1:rx-022:28:130
3672937991613,2cyclictest60650irq/133-enp0s29f1:rx-021:45:370
3672937991613,2cyclictest60650irq/133-enp0s29f1:rx-019:25:210
3672937991613,2cyclictest3871123-21kworker/u8:0+events_unbound22:35:230
3672937991613,2cyclictest3861362-21kworker/u8:2+flush-259:022:30:300
3672937991613,2cyclictest3834810-21latency_hist21:55:010
3672937991613,2cyclictest3772483-21kworker/u8:1+events_unbound21:40:230
3672937991613,1cyclictest0-21swapper/023:35:000
3672937991612,3cyclictest3912416-21kworker/u8:3+flush-259:000:05:240
367294299159,4cyclictest3735462-21latency_hist20:10:003
3672942991513,1cyclictest0-21swapper/321:00:003
367294099158,5cyclictest2698871-21systemd-journal23:25:002
3672940991512,2cyclictest0-21swapper/219:40:002
3672939991511,3cyclictest3933656-21kworker/u8:2+events_unbound23:49:581
3672939991511,3cyclictest3625953-21kworker/u8:3+events_unbound20:25:441
367293799155,8cyclictest2698871-21systemd-journal21:59:590
3672937991513,1cyclictest60650irq/133-enp0s29f1:rx-023:22:520
3672937991513,1cyclictest60650irq/133-enp0s29f1:rx-021:18:290
3672937991513,1cyclictest60650irq/133-enp0s29f1:rx-020:33:450
3672937991513,1cyclictest60650irq/133-enp0s29f1:rx-019:55:180
3672937991513,1cyclictest60650irq/133-enp0s29f1:rx-019:50:410
3672937991513,1cyclictest60650irq/133-enp0s29f1:rx-000:16:170
3672937991512,2cyclictest60650irq/133-enp0s29f1:rx-021:10:190
3672937991512,2cyclictest4004434-21sort00:35:190
3672937991512,2cyclictest3923278-21idleruntime-cro23:20:000
3672937991511,3cyclictest3912416-21kworker/u8:3+flush-259:000:35:000
3672937991510,4cyclictest3897588-21diskstats22:55:140
367294299148,4cyclictest3730308-21latency_hist20:05:013
367294299147,5cyclictest3703236-21ntpq19:35:213
3672942991410,3cyclictest503-21systemd-logind19:15:013
3672940991410,3cyclictest3944924-21fschecks_time23:40:132
3672939991410,3cyclictest3912416-21kworker/u8:3+events_unbound00:15:211
3672939991410,3cyclictest3783420-21kworker/u8:2+events_unbound21:30:101
3672937991413,1cyclictest61450irq/134-enp0s29f1:tx-020:20:140
3672937991413,1cyclictest60650irq/133-enp0s29f1:rx-020:10:420
3672937991413,1cyclictest60650irq/133-enp0s29f1:rx-020:10:420
3672937991412,2cyclictest60650irq/133-enp0s29f1:rx-023:49:480
3672937991412,1cyclictest60650irq/133-enp0s29f1:rx-020:51:090
3672937991412,1cyclictest60650irq/133-enp0s29f1:rx-019:30:240
3672937991412,1cyclictest60650irq/133-enp0s29f1:rx-019:20:180
3672937991412,1cyclictest60650irq/133-enp0s29f1:rx-000:12:230
3672937991411,2cyclictest60650irq/133-enp0s29f1:rx-023:14:180
3672937991411,2cyclictest60650irq/133-enp0s29f1:rx-000:25:010
3672937991411,2cyclictest3871123-21kworker/u8:0+events_unbound23:35:200
3672937991411,2cyclictest3783420-21kworker/u8:2+flush-259:021:35:190
3672937991411,2cyclictest3772483-21kworker/u8:1+events_unbound20:55:230
3672937991410,3cyclictest3944635-21kworker/u8:1+flush-259:023:57:440
367294299139,3cyclictest3757864-21fschecks_count20:40:103
367294299138,3cyclictest2698871-21systemd-journal21:10:003
367294299138,3cyclictest0-21swapper/322:35:013
3672942991311,1cyclictest3952114-21kthreadcore23:45:163
3672942991311,1cyclictest0-21swapper/323:40:003
3672942991310,2cyclictest3965347-21cpuspeed_turbos00:00:133
3672942991310,2cyclictest3783195-21idleruntime-cro21:05:003
3672942991310,2cyclictest3767261-21unixbench_multi20:45:253
3672942991310,2cyclictest0-21swapper/323:05:013
367294099137,4cyclictest4000956-21ntpq00:30:212
3672940991311,1cyclictest4004747-21kthreadcore00:35:202
3672940991310,2cyclictest3903564-21sort23:00:172
3672940991310,1cyclictest0-21swapper/222:25:012
367294099130,8cyclictest3829426-21idleruntime-cro21:49:592
367293999137,4cyclictest3740865-21/usr/sbin/munin20:15:161
3672937991311,1cyclictest60650irq/133-enp0s29f1:rx-021:25:130
3672937991311,1cyclictest60650irq/133-enp0s29f1:rx-021:06:010
3672937991310,2cyclictest3850880-21cut22:10:140
3672937991310,2cyclictest3772483-21kworker/u8:1+events_unbound22:15:200
367294299129,2cyclictest3892335-21cpuspeed_turbos22:50:133
367294299128,3cyclictest3850419-21awk22:09:593
367294299128,3cyclictest3767747-21kworker/u8:0+flush-259:021:15:163
3672942991210,1cyclictest0-21swapper/323:25:203
3672942991210,1cyclictest0-21swapper/322:55:193
367294099129,1cyclictest0-21swapper/220:15:002
367294099129,1cyclictest0-21swapper/220:15:002
3672940991210,1cyclictest0-21swapper/200:20:232
367293799128,3cyclictest3772483-21kworker/u8:1+events_unbound22:05:140
3672937991210,2cyclictest60650irq/133-enp0s29f1:rx-020:26:470
3672937991210,1cyclictest60650irq/133-enp0s29f1:rx-020:39:380
367294299119,1cyclictest0-21swapper/320:40:013
367294299119,1cyclictest0-21swapper/319:45:003
367294299118,2cyclictest3882094-21cpuspeed_turbos22:40:113
367294299118,2cyclictest3715172-21cpuspeed_turbos19:50:133
367294099119,1cyclictest3901300-21needreboot22:55:202
367294099119,1cyclictest0-21swapper/223:30:172
367294099118,2cyclictest3892362-21cpuspeed_turbos22:50:132
367294099118,2cyclictest3769007-21kthreadcore20:50:162
367294099117,3cyclictest3923309-21kworker/u8:4+flush-259:023:25:202
367294099117,2cyclictest0-21swapper/222:15:152
367294099116,3cyclictest0-21swapper/219:50:142
367293799119,1cyclictest171ktimers/021:00:120
367294299108,1cyclictest0-21swapper/321:25:103
367294299106,2cyclictest491ktimers/323:05:193
367294299104,4cyclictest2698871-21systemd-journal22:22:183
367294099107,2cyclictest0-21swapper/220:30:002
367294099104,5cyclictest1-21systemd21:45:002
367294099104,4cyclictest3694387-21fw_conntrack19:30:142
36729429997,1cyclictest0-21swapper/300:05:183
36729409997,1cyclictest3673836-21ps19:10:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional