You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-20 - 11:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Oct 20, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3569126994743,3cyclictest3605097-21kworker/u8:0+events_unbound20:15:020
3569131994539,4cyclictest3838712-21kworker/u8:3+flush-259:023:45:011
3569131994238,3cyclictest3822735-21gpgv23:15:001
3569139994135,4cyclictest3754968-21latency_hist22:10:023
3569136993834,3cyclictest3750039-21kworker/u8:3+flush-259:022:10:172
3569139993731,5cyclictest75-21kswapd023:40:453
3569139993528,5cyclictest3599958-21latency_hist19:40:013
3569136993532,2cyclictest75-21kswapd023:40:442
3569139993427,5cyclictest3589674-21latency_hist19:30:013
3569131993430,3cyclictest3625591-21kworker/u8:2+flush-259:020:55:001
3569139993325,6cyclictest3703472-21latency_hist21:20:003
3569136993328,3cyclictest3775731-21kworker/u8:4+flush-259:022:35:162
3569131993327,4cyclictest3739552-21latency_hist21:55:001
3569126993229,2cyclictest3864292-21kworker/u8:2+flush-259:000:40:010
3569136993128,2cyclictest3525647-21kworker/u8:1+flush-259:019:10:202
3569126993026,3cyclictest3677370-21kworker/u8:4+flush-259:021:10:180
3569136992926,2cyclictest3525647-21kworker/u8:1+flush-259:021:40:182
3569131992925,3cyclictest2698871-21systemd-journal00:39:191
3569136992825,2cyclictest3838712-21kworker/u8:3+events_unbound23:40:002
3569131992822,5cyclictest3584485-21latency_hist19:25:011
3569131992821,5cyclictest3682326-21latency_hist21:00:001
3569126992825,2cyclictest3864292-21kworker/u8:2+flush-259:000:00:010
3569136992721,4cyclictest3780670-21latency_hist22:35:022
3569131992722,3cyclictest3610469-21kworker/u8:4+flush-259:020:00:001
3569131992721,4cyclictest3905610-21latency_hist00:35:011
3569131992720,5cyclictest3708638-21latency_hist21:25:011
3569131992618,6cyclictest3712911-21ntpq21:25:221
3569131992618,5cyclictest2698871-21systemd-journal21:05:001
3569126992624,1cyclictest3677162-21kworker/u8:3+events_unbound21:10:010
3569126992622,3cyclictest3775731-21kworker/u8:4+flush-259:023:10:180
3569139992519,4cyclictest3579591-21/usr/sbin/munin19:20:163
3569136992521,3cyclictest3666619-21kworker/u8:3+flush-259:020:45:162
3569131992518,5cyclictest3775504-21latency_hist22:30:021
3569126992520,4cyclictest3811016-21ntpq23:00:210
3569139992420,3cyclictest3890143-21kworker/u8:1+flush-259:000:35:003
3569136992419,4cyclictest3713791-21kworker/u8:2+flush-259:022:00:182
3569136992418,4cyclictest3610240-21latency_hist19:50:012
3569131992418,4cyclictest3864513-21/usr/sbin/munin23:55:181
3569126992421,2cyclictest3698261-21kworker/u8:0+flush-259:021:25:200
3569126992420,3cyclictest3625591-21kworker/u8:2+events_unbound20:35:160
3569139992314,6cyclictest2698871-21systemd-journal00:10:003
3569136992319,3cyclictest3858756-21kworker/u8:0+events_unbound23:50:192
3569136992319,3cyclictest3525647-21kworker/u8:1+events_unbound22:20:012
3569136992316,5cyclictest2698871-21systemd-journal20:30:012
3569131992319,3cyclictest3605097-21kworker/u8:0+flush-259:020:35:241
3569131992316,5cyclictest3806957-21cat23:00:011
3569126992320,2cyclictest75-21kswapd023:40:450
3569139992217,3cyclictest2698871-21systemd-journal22:11:423
3569136992219,2cyclictest3525647-21kworker/u8:1+flush-259:023:05:202
3569136992219,2cyclictest3525647-21kworker/u8:1+events_unbound23:25:202
3569136992218,3cyclictest3677162-21kworker/u8:3+flush-259:021:15:272
3569126992217,4cyclictest2698871-21systemd-journal23:10:000
3569139992117,3cyclictest3698261-21kworker/u8:0+flush-259:021:25:173
3569139992115,5cyclictest3801215-21latency_hist22:55:003
3569139992114,5cyclictest3666861-21latency_hist20:45:003
3569136992119,1cyclictest3625715-21latency_hist20:05:022
3569131992117,3cyclictest3864292-21kworker/u8:2+flush-259:000:20:001
3569126992118,2cyclictest3838712-21kworker/u8:3+events_unbound23:35:160
3569126992118,2cyclictest3770407-21kworker/u8:0+events_unbound22:25:200
3569126992117,3cyclictest3625591-21kworker/u8:2+events_unbound20:22:060
3569136992017,2cyclictest3838712-21kworker/u8:3+flush-259:000:00:172
3569136992017,2cyclictest3625591-21kworker/u8:2+events_unbound20:10:002
3569136992016,3cyclictest3864292-21kworker/u8:2+flush-259:000:25:182
3569136992016,3cyclictest3785842-21kworker/u8:0+flush-259:022:45:212
3569131992014,4cyclictest3854201-21/usr/sbin/munin23:45:151
3569131992013,5cyclictest3693318-21/usr/sbin/munin21:10:171
3569126992017,2cyclictest3677370-21kworker/u8:4+events_unbound20:55:220
3569126992016,2cyclictest3000061-21in:imuxsock22:40:010
3569139991915,2cyclictest3905874-21apt-get00:35:133
3569139991911,6cyclictest0-21swapper/323:10:003
3569136991916,2cyclictest3807010-21kworker/u8:2+events_unbound23:20:202
3569131991913,4cyclictest2698871-21systemd-journal22:45:411
356912699192,16cyclictest2698871-21systemd-journal20:40:400
3569126991917,2cyclictest3525647-21kworker/u8:1+events_unbound21:40:250
3569126991916,2cyclictest3838712-21kworker/u8:3+flush-259:000:20:190
3569126991916,2cyclictest3807010-21kworker/u8:2+events_unbound23:20:160
3569126991916,2cyclictest3584511-21kworker/u8:2+events_unbound19:25:170
3569126991915,3cyclictest3677162-21kworker/u8:3+events_unbound21:15:230
3569126991915,3cyclictest3525647-21kworker/u8:1+flush-259:023:15:260
356913999187,9cyclictest2698871-21systemd-journal22:46:423
356913999186,10cyclictest3590473-21sed19:30:173
3569139991815,2cyclictest0-21swapper/321:50:003
3569139991814,3cyclictest2698871-21systemd-journal00:15:013
356913999181,15cyclictest3843649-21cron23:35:003
356913999181,15cyclictest3843649-21cron23:34:593
3569139991811,5cyclictest3811949-21latency_hist23:05:003
3569139991811,5cyclictest3646318-21latency_hist20:25:013
356913999180,16cyclictest3569230-21cat19:10:003
356913999180,16cyclictest3569230-21cat19:10:003
3569136991815,2cyclictest3625591-21kworker/u8:2+flush-259:020:15:232
3569136991814,3cyclictest3677370-21kworker/u8:4+events_unbound21:05:152
3569131991814,3cyclictest3525647-21kworker/u8:1+events_unbound22:25:001
3569131991812,4cyclictest2698871-21systemd-journal22:15:011
3569126991815,2cyclictest3713791-21kworker/u8:2+events_unbound22:00:430
3569126991815,2cyclictest3713791-21kworker/u8:2+events_unbound21:55:230
3569126991815,2cyclictest3698261-21kworker/u8:0+events_unbound21:35:170
3569126991815,2cyclictest3677370-21kworker/u8:4+events_unbound21:00:160
3569126991814,3cyclictest3666619-21kworker/u8:3+flush-259:020:45:150
3569126991814,3cyclictest3625591-21kworker/u8:2+events_unbound20:15:190
3569139991713,3cyclictest3739793-21/usr/sbin/munin21:55:153
3569139991713,2cyclictest0-21swapper/320:20:003
356913999170,2cyclictest3677196-21cat20:55:003
3569136991715,1cyclictest0-21swapper/222:10:002
3569136991713,3cyclictest3579617-21apt-get19:20:132
3569136991711,4cyclictest3817570-21cat23:10:012
3569136991711,4cyclictest3645402-21ntpq20:20:202
3569131991714,2cyclictest3729239-21cron21:45:001
3569131991714,2cyclictest0-21swapper/123:10:001
3569126991715,2cyclictest3525647-21kworker/u8:1+flush-259:019:50:250
3569126991713,3cyclictest3750039-21kworker/u8:3+events_unbound22:17:270
3569126991713,3cyclictest3573957-21kworker/u8:0+events_unbound19:15:240
3569126991713,3cyclictest3525647-21kworker/u8:1+flush-259:022:30:240
3569126991713,3cyclictest3525647-21kworker/u8:1+events_unbound19:45:150
3569126991712,3cyclictest3525647-21kworker/u8:1+events_unbound21:50:180
3569126991711,4cyclictest3525647-21kworker/u8:1+flush-259:020:25:160
356912699170,15cyclictest1-21systemd19:21:380
356913999169,6cyclictest3835894-21tr23:25:173
356913999162,12cyclictest3661725-21sh20:39:593
356913999161,6cyclictest3769542-21ntpq22:20:233
356913999161,13cyclictest3895277-21cron00:25:003
356913999161,13cyclictest3781335-21sed22:35:163
356913999161,13cyclictest3739554-21sh21:55:013
356913999161,13cyclictest3620546-21cron20:00:003
356913999161,12cyclictest3760518-21df22:15:153
3569139991610,4cyclictest2698871-21systemd-journal00:05:013
356913999160,8cyclictest835-21runrttasks20:03:583
356913999160,4cyclictest3848779-21awk23:40:003
356913999160,1cyclictest0-21swapper/320:45:023
356913999160,14cyclictest3790535-21sed22:40:253
356913999160,13cyclictest3719322-21grep21:35:153
356913999160,13cyclictest3677812-21ps20:55:193
356913999160,13cyclictest3651825-21cpuspeed_turbos20:30:133
356913999160,13cyclictest3600338-21tr19:40:153
356913999160,13cyclictest3574605-21tr19:15:133
3569136991613,2cyclictest3838712-21kworker/u8:3+flush-259:000:14:572
3569136991613,2cyclictest3807010-21kworker/u8:2+flush-259:023:00:252
3569136991613,2cyclictest0-21swapper/219:30:002
356913199162,6cyclictest3693081-21munin-run21:10:011
3569131991610,4cyclictest3724119-21latency_hist21:40:001
3569126991614,1cyclictest60650irq/133-enp0s29f1:rx-020:51:520
3569126991613,2cyclictest60650irq/133-enp0s29f1:rx-000:27:550
3569126991613,2cyclictest3900432-21kworker/u8:0+flush-259:000:30:160
3569126991613,2cyclictest3864292-21kworker/u8:2+events_unbound00:05:230
3569126991612,3cyclictest3610469-21kworker/u8:4+events_unbound19:55:150
3569126991612,3cyclictest3525647-21kworker/u8:1+flush-259:023:30:250
3569126991612,3cyclictest3525647-21kworker/u8:1+flush-259:023:30:240
3569126991612,3cyclictest3525647-21kworker/u8:1+events_unbound22:23:420
356913999151,2cyclictest3609341-21ntpq19:45:223
3569139991512,2cyclictest3525647-21kworker/u8:1+flush-259:023:15:213
356913999151,13cyclictest3574211-21idleruntime-cro19:15:003
356913999151,12cyclictest3895962-21ps00:25:183
356913999151,12cyclictest3869564-21latency_hist00:00:003
356913999151,12cyclictest3864255-21cron23:54:593
356913999151,12cyclictest3776170-21grep22:30:173
356913999151,12cyclictest3697760-21sed21:10:283
356913999151,12cyclictest3635124-21ntpq20:10:233
356913999151,12cyclictest3615430-21idleruntime-cro19:55:003
356913999150,7cyclictest835-21runrttasks22:28:553
356913999150,14cyclictest3822922-21latency_hist23:15:013
356913999150,13cyclictest3859120-21cron23:50:003
356913999150,13cyclictest3828559-21grep23:20:113
356913999150,13cyclictest3801571-21grep22:55:143
356913999150,13cyclictest3745069-21grep22:00:143
356913999150,13cyclictest3729240-21cron21:45:003
356913999150,13cyclictest3703866-21tr21:20:133
356913999150,13cyclictest3683001-21sort21:00:173
356913999150,12cyclictest3885339-21cpuspeed_turbos00:15:153
356913699159,5cyclictest3910738-21latency_hist00:40:012
3569136991513,1cyclictest3744689-21cron22:00:002
3569136991511,3cyclictest3775731-21kworker/u8:4+events_unbound22:55:222
3569136991511,3cyclictest3677370-21kworker/u8:4+flush-259:020:55:262
3569136991511,3cyclictest3525647-21kworker/u8:1+flush-259:019:30:202
356913199151,2cyclictest3884949-21idleruntime-cro00:15:011
3569131991512,2cyclictest3662072-21cpuspeed_turbos20:40:151
3569131991512,2cyclictest3525647-21kworker/u8:1+flush-259:020:25:261
3569131991512,2cyclictest0-21swapper/122:00:001
3569131991511,2cyclictest0-21swapper/100:05:181
3569126991513,2cyclictest60650irq/133-enp0s29f1:rx-021:20:200
3569126991513,1cyclictest60650irq/133-enp0s29f1:rx-021:46:300
3569126991513,1cyclictest60650irq/133-enp0s29f1:rx-020:30:230
3569126991512,3cyclictest3525647-21kworker/u8:1+events_unbound21:30:280
3569126991512,2cyclictest61450irq/134-enp0s29f1:tx-023:55:280
3569126991512,2cyclictest60650irq/133-enp0s29f1:rx-022:57:110
3569126991512,2cyclictest60650irq/133-enp0s29f1:rx-022:54:060
3569126991512,2cyclictest60650irq/133-enp0s29f1:rx-022:06:260
3569126991512,2cyclictest60650irq/133-enp0s29f1:rx-019:33:230
3569126991512,2cyclictest3525647-21kworker/u8:1+flush-259:023:25:170
3569126991511,3cyclictest3838712-21kworker/u8:3+events_unbound00:15:260
3569126991511,3cyclictest3750039-21kworker/u8:3+events_unbound22:10:200
3569126991511,3cyclictest3584721-21kworker/u8:3+events_unbound19:39:400
3569126991510,4cyclictest3838712-21kworker/u8:3+flush-259:023:45:200
356913999140,5cyclictest3714038-21apt-get21:30:123
356913999140,3cyclictest3688412-21/usr/sbin/munin21:05:153
356913999140,12cyclictest3646695-21tr20:25:133
3569136991410,3cyclictest3874868-21/usr/sbin/munin00:05:152
3569136991410,3cyclictest3698261-21kworker/u8:0+flush-259:021:35:212
3569131991411,2cyclictest3838068-21unixbench_singl23:25:241
3569131991411,2cyclictest3654013-21kthreadcore20:30:171
3569131991411,2cyclictest3525647-21kworker/u8:1+flush-259:020:00:251
3569131991410,3cyclictest3838712-21kworker/u8:3+flush-259:023:40:001
3569131991410,3cyclictest3796365-21apt-get22:50:121
3569131991410,3cyclictest3780195-21unixbench_multi22:30:241
356912699149,4cyclictest3525647-21kworker/u8:1+flush-259:020:00:190
3569126991411,2cyclictest60650irq/133-enp0s29f1:rx-022:49:040
3569126991411,2cyclictest60650irq/133-enp0s29f1:rx-020:08:170
3569126991411,2cyclictest60650irq/133-enp0s29f1:rx-019:44:000
3569126991411,2cyclictest60650irq/133-enp0s29f1:rx-019:14:550
356913699139,3cyclictest3605097-21kworker/u8:0+events_unbound20:30:022
356913699139,2cyclictest3785842-21kworker/u8:0+events_unbound22:50:162
356913699138,3cyclictest3605078-21latency_hist19:45:012
356913699134,7cyclictest3890100-21munin-run00:20:002
3569136991311,1cyclictest3714460-21irqcore21:30:172
3569136991310,2cyclictest3525647-21kworker/u8:1+flush-259:019:50:192
3569136991310,2cyclictest3525647-21kworker/u8:1+events_unbound21:50:202
356913199139,3cyclictest3525647-21kworker/u8:1+flush-259:020:10:271
3569131991310,2cyclictest3595285-21fschecks_time19:35:151
3569126991310,2cyclictest60650irq/133-enp0s29f1:rx-023:50:020
3569126991310,2cyclictest60650irq/133-enp0s29f1:rx-000:12:150
356913699128,3cyclictest3900432-21kworker/u8:0+events_unbound00:30:142
356913699128,3cyclictest2702718-21systemd-udevd23:58:552
356913699125,5cyclictest3713772-21latency_hist21:30:002
356913199126,4cyclictest3823284-21/usr/sbin/munin23:15:161
356913199126,4cyclictest3805477-21ntpq22:55:231
356913199125,5cyclictest3715058-21sed21:30:181
3569126991210,1cyclictest60650irq/133-enp0s29f1:rx-022:40:230
356913699119,1cyclictest0-21swapper/220:35:222
356913699118,2cyclictest3854485-21fschecks_time23:45:152
356913199119,1cyclictest0-21swapper/119:20:001
356913199118,2cyclictest3626068-21cut20:05:111
356913199118,2cyclictest0-21swapper/123:35:011
356913199118,2cyclictest0-21swapper/123:35:011
356913199117,3cyclictest3605097-21kworker/u8:0+flush-259:020:48:121
356913199117,2cyclictest0-21swapper/123:20:161
356913199115,5cyclictest3698507-21apt-get21:15:151
356913199115,4cyclictest3869691-21/usr/sbin/munin00:00:201
356913199111,2cyclictest3749809-21awk22:05:011
356913699107,2cyclictest3729583-21cpuspeed_turbos21:45:112
356913199107,2cyclictest3859450-21cpuspeed_turbos23:50:161
356913199101,1cyclictest35899252sort19:30:121
356913199100,2cyclictest3589674-21latency_hist19:30:001
35691369994,3cyclictest3666860-21awk20:44:592
35691369993,5cyclictest0-21swapper/222:20:182
35691319996,2cyclictest0-21swapper/121:45:141
35691319995,3cyclictest3637234-21sed20:15:201
35691319995,3cyclictest3625591-21kworker/u8:2+flush-259:020:20:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional