You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 04:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Tue Jul 01, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3740900995955,3cyclictest3730854-21kworker/u8:1+events_unbound19:22:033
3740900994441,2cyclictest3824425-21kworker/u8:3+events_unbound20:30:183
3740900994340,2cyclictest3964209-21kworker/u8:1+events_unbound23:15:223
3740900994037,2cyclictest3969580-21kworker/u8:2+events_unbound23:10:223
3740900994037,2cyclictest3834550-21kworker/u8:0+flush-259:020:55:203
3740895994035,4cyclictest3882027-21kworker/u8:0+flush-259:021:35:262
3740900993935,3cyclictest3897541-21kworker/u8:4+events_unbound22:30:193
3740900993832,2cyclictest3932995-21kworker/u8:0+events_unbound22:15:003
3740900993831,2cyclictest3907857-21kworker/u8:3+events_unbound21:55:183
3740900993734,2cyclictest3730854-21kworker/u8:1+flush-259:019:35:303
3740900993632,3cyclictest3932995-21kworker/u8:0+events_unbound22:20:283
3740900993630,4cyclictest3746189-21/usr/sbin/munin19:15:133
3740892993633,2cyclictest3782764-21kworker/u8:0+flush-259:020:05:161
3740900993532,2cyclictest3964209-21kworker/u8:1+flush-259:023:35:163
3740900993532,2cyclictest3834550-21kworker/u8:0+events_unbound21:15:263
3740900993532,2cyclictest3824425-21kworker/u8:3+flush-259:020:50:223
3740900993532,2cyclictest3793220-21kworker/u8:5+events_unbound20:20:253
3740900993532,2cyclictest3782764-21kworker/u8:0+flush-259:019:50:153
3740900993528,5cyclictest3902465-21latency_hist21:45:013
3740900993431,2cyclictest3882027-21kworker/u8:0+events_unbound21:30:243
3740900993431,2cyclictest3824425-21kworker/u8:3+events_unbound20:45:173
3740900993430,3cyclictest3897541-21kworker/u8:4+events_unbound22:40:183
3740900993427,3cyclictest4021126-21kworker/u8:2+events_unbound23:55:153
3740892993429,3cyclictest4041637-21latency_hist00:00:021
3740900993330,2cyclictest3994990-21kworker/u8:0+flush-259:023:40:243
3740900993330,2cyclictest3907857-21kworker/u8:3+flush-259:021:50:163
3740900993329,3cyclictest3984766-21kworker/u8:3+events_unbound00:00:233
3740892993328,3cyclictest3995004-21latency_hist23:15:001
3740900993229,2cyclictest4047520-21kworker/u8:0+events_unbound00:15:223
3740900993229,2cyclictest3882027-21kworker/u8:0+flush-259:021:45:213
3740900993229,2cyclictest3730854-21kworker/u8:1+flush-259:019:40:223
3740895993228,3cyclictest3948997-21kworker/u8:2+flush-259:022:40:002
3740888993226,4cyclictest4004402-21ntpq23:20:250
3740888993225,5cyclictest3989039-21ntpq23:05:260
3740900993128,2cyclictest3897541-21kworker/u8:4+flush-259:022:35:233
3740900993128,2cyclictest1692155-21kworker/u8:3+flush-259:019:25:243
3740895993124,5cyclictest4073241-21latency_hist00:35:002
3740900993027,2cyclictest4021126-21kworker/u8:2+events_unbound23:50:193
3740900993027,2cyclictest3782764-21kworker/u8:0+flush-259:019:55:253
3740900993025,3cyclictest3860294-21latency_hist21:05:013
3740900993022,6cyclictest3754036-21systemd-journal23:35:003
3740892993024,5cyclictest3751094-21latency_hist19:20:001
3740900992926,2cyclictest4057543-21kworker/u8:2+events_unbound00:30:243
3740900992926,2cyclictest3994990-21kworker/u8:0+flush-259:023:45:173
3740900992926,2cyclictest3793220-21kworker/u8:5+events_unbound20:05:143
3740895992927,1cyclictest3755980-21fwupd19:22:042
3740895992926,2cyclictest3964209-21kworker/u8:1+flush-259:023:15:192
3740895992925,3cyclictest3994990-21kworker/u8:0+flush-259:023:40:002
3740900992825,2cyclictest4041664-21kworker/u8:1+events_unbound00:35:163
3740900992824,3cyclictest3865136-21kworker/u8:3+events_unbound21:10:233
3740900992823,3cyclictest3730854-21kworker/u8:1+events_unbound19:30:023
3740895992825,2cyclictest4041664-21kworker/u8:1+flush-259:000:05:002
3740892992822,5cyclictest3854315-21ntpq20:55:221
3740888992823,4cyclictest3824425-21kworker/u8:3+flush-259:020:30:200
374088899280,26cyclictest3803593-21true20:10:000
3740900992724,2cyclictest3969580-21kworker/u8:2+flush-259:022:50:153
3740900992724,2cyclictest3964209-21kworker/u8:1+events_unbound23:05:293
3740900992724,2cyclictest3897541-21kworker/u8:4+events_unbound22:15:263
3740900992723,3cyclictest3932995-21kworker/u8:0+events_unbound22:49:483
3740900992721,4cyclictest3882223-21diskstats21:25:153
3740895992720,5cyclictest3953908-21latency_hist22:35:002
3740892992722,3cyclictest3756488-21/usr/sbin/munin19:25:161
374088899270,26cyclictest1-21systemd19:31:440
3740900992623,2cyclictest3850219-21kworker/u8:2+flush-259:021:05:283
3740900992623,2cyclictest3824425-21kworker/u8:3+events_unbound20:40:153
3740900992621,3cyclictest3881534-21base6421:25:013
3740900992620,4cyclictest4005255-21latency_hist23:25:013
3740892992622,3cyclictest3994990-21kworker/u8:0+flush-259:023:30:001
3740900992523,1cyclictest3964209-21kworker/u8:1+events_unbound23:25:173
3740900992522,2cyclictest3730854-21kworker/u8:1+flush-259:020:10:223
3740900992521,3cyclictest3730854-21kworker/u8:1+flush-259:019:12:453
3740900992520,2cyclictest4041664-21kworker/u8:1+events_unbound00:10:183
3740895992519,4cyclictest3969580-21kworker/u8:2+flush-259:022:50:192
3740895992518,4cyclictest3000063-21rs:main2
3740895992517,5cyclictest4010407-21apt-get23:30:012
3740892992520,4cyclictest3875601-21kworker/u8:2+flush-259:021:20:001
3740900992421,2cyclictest3808749-21kworker/u8:2+flush-259:020:15:233
3740900992420,3cyclictest3875601-21kworker/u8:2+flush-259:022:00:163
3740900992420,3cyclictest3875601-21kworker/u8:2+events_unbound22:08:493
374089599246,9cyclictest3886992-21munin-run21:30:002
3740892992420,3cyclictest3755990-21kworker/u8:2+events_unbound19:40:261
3740892992419,3cyclictest3754036-21systemd-journal20:04:551
3740888992420,3cyclictest3984766-21kworker/u8:3+flush-259:000:25:170
3740900992320,2cyclictest3730854-21kworker/u8:1+events_unbound19:45:123
3740900992319,3cyclictest3984766-21kworker/u8:3+events_unbound00:20:153
3740895992319,3cyclictest3808749-21kworker/u8:2+events_unbound20:15:232
3740895992318,4cyclictest3754036-21systemd-journal23:30:482
3740895992315,5cyclictest3933776-21diskstats22:15:172
3740892992319,3cyclictest4021126-21kworker/u8:2+flush-259:023:55:001
3740892992319,3cyclictest3865136-21kworker/u8:3+flush-259:021:10:181
3740888992317,4cyclictest3833685-21ntpq20:35:230
3740888992317,4cyclictest3833685-21ntpq20:35:220
3740888992317,4cyclictest3787177-21ntpq19:50:220
3740888992317,4cyclictest3777955-21apt-get19:45:120
3740888992316,5cyclictest4026263-21latency_hist23:45:020
3740900992219,2cyclictest3882027-21kworker/u8:0+events_unbound21:35:193
3740895992217,4cyclictest3907857-21kworker/u8:3+flush-259:022:00:262
3740895992217,3cyclictest3932995-21kworker/u8:0+flush-259:022:25:292
3740895992217,3cyclictest3865136-21kworker/u8:3+flush-259:021:15:002
3740895992217,3cyclictest1692155-21kworker/u8:3+flush-259:019:15:162
3740895992215,5cyclictest3754036-21systemd-journal23:15:002
3740895992215,4cyclictest3824425-21kworker/u8:3+flush-259:020:45:192
3740892992218,3cyclictest4011085-21apt-get23:30:131
3740892992217,3cyclictest3730854-21kworker/u8:1+flush-259:019:35:251
3740892992216,4cyclictest3762094-21apt-get19:30:131
3740892992215,4cyclictest3782876-21latency_hist19:50:001
3740900992118,2cyclictest3964209-21kworker/u8:1+events_unbound22:55:253
3740900992118,2cyclictest3824425-21kworker/u8:3+flush-259:020:35:283
3740900992118,2cyclictest3824425-21kworker/u8:3+flush-259:020:35:283
3740895992118,2cyclictest1692155-21kworker/u8:3+flush-259:019:25:272
3740895992116,3cyclictest3730854-21kworker/u8:1+flush-259:020:00:152
3740888992117,3cyclictest3875601-21kworker/u8:2+flush-259:022:10:260
3740888992117,3cyclictest3819040-21kworker/u8:0+flush-259:020:25:190
3740888992116,4cyclictest3754036-21systemd-journal23:55:000
3740888992114,5cyclictest3963292-21ntpq22:40:230
3740900992017,2cyclictest4041664-21kworker/u8:1+events_unbound00:06:473
3740900992017,2cyclictest3964209-21kworker/u8:1+events_unbound23:01:483
3740900992017,2cyclictest3793220-21kworker/u8:5+events_unbound20:02:363
3740895992013,5cyclictest3880456-21ntpq21:20:242
3740892992017,2cyclictest3989848-21cron23:10:001
3740888992018,1cyclictest1693150-21kworker/0:1+pm19:22:050
3740888992017,2cyclictest3948783-21idleruntime-cro22:30:000
3740888992015,4cyclictest3754036-21systemd-journal21:38:470
3740888992015,4cyclictest3754036-21systemd-journal19:55:000
3740888992014,4cyclictest3938921-21diskstats22:20:180
3740892991915,3cyclictest4068348-21apt-get00:30:131
3740892991913,3cyclictest3928179-21latency_hist22:10:001
3740892991912,5cyclictest3754036-21systemd-journal19:22:051
3740895991815,2cyclictest3830070-21irqcore20:35:172
3740895991815,2cyclictest3830070-21irqcore20:35:162
3740895991815,2cyclictest1692155-21kworker/u8:3+flush-259:019:10:182
3740895991814,2cyclictest3932995-21kworker/u8:0+flush-259:022:20:212
3740895991812,4cyclictest3754036-21systemd-journal22:07:482
3740892991813,4cyclictest4021099-21latency_hist23:40:011
3740888991814,3cyclictest4047520-21kworker/u8:0+flush-259:000:15:000
3740888991813,4cyclictest3881380-21latency_hist21:25:010
3740900991714,2cyclictest3793220-21kworker/u8:5+events_unbound20:27:373
3740895991713,3cyclictest3964209-21kworker/u8:1+flush-259:023:00:182
3740895991713,3cyclictest3860975-21irqcore21:05:202
3740895991713,2cyclictest0-21swapper/200:15:012
3740892991715,1cyclictest0-21swapper/121:25:001
3740892991714,2cyclictest3824425-21kworker/u8:3+flush-259:020:50:171
3740892991713,3cyclictest3907857-21kworker/u8:3+flush-259:021:55:261
3740892991713,3cyclictest3819040-21kworker/u8:0+flush-259:020:25:171
3740892991712,4cyclictest3897541-21kworker/u8:4+flush-259:022:15:291
3740892991711,4cyclictest3754036-21systemd-journal22:10:471
3740892991710,5cyclictest4052657-21latency_hist00:15:021
3740892991710,2cyclictest4068090-21grep00:29:591
3740888991714,2cyclictest60650irq/133-enp0s29f1:rx-020:55:210
3740888991711,4cyclictest3776845-21ntpq19:40:270
3740900991614,2cyclictest3984766-21kworker/u8:3+flush-259:000:28:173
3740895991611,4cyclictest3754036-21systemd-journal21:45:012
3740895991611,3cyclictest4041664-21kworker/u8:1+flush-259:000:35:162
3740892991613,2cyclictest503-21systemd-logind21:35:011
3740892991610,4cyclictest3892391-21/usr/sbin/munin21:35:181
374089299160,15cyclictest1-21systemd00:02:471
3740888991614,2cyclictest60650irq/133-enp0s29f1:rx-020:00:550
3740888991614,1cyclictest3002-21cron21:50:000
3740888991613,2cyclictest60650irq/133-enp0s29f1:rx-023:35:190
3740888991613,2cyclictest60650irq/133-enp0s29f1:rx-023:25:240
3740888991613,2cyclictest60650irq/133-enp0s29f1:rx-022:50:230
3740888991613,2cyclictest60650irq/133-enp0s29f1:rx-019:35:530
374088899161,13cyclictest1-21systemd22:47:470
3740888991610,5cyclictest3754036-21systemd-journal20:55:000
374089599159,4cyclictest4067155-21ntpq00:25:222
374089599159,4cyclictest3964209-21kworker/u8:1+events_unbound23:05:182
374089599159,4cyclictest3754036-21systemd-journal19:49:592
3740895991512,2cyclictest4026950-21irqcore23:45:182
3740895991511,3cyclictest3730854-21kworker/u8:1+flush-259:019:40:222
3740895991510,3cyclictest3994990-21kworker/u8:0+events_unbound23:40:222
374089299159,4cyclictest4073480-21/usr/sbin/munin00:35:151
3740892991512,2cyclictest3875601-21kworker/u8:2+flush-259:022:00:181
3740888991513,2cyclictest60650irq/133-enp0s29f1:rx-020:17:230
3740888991513,1cyclictest60650irq/133-enp0s29f1:rx-023:48:300
3740888991513,1cyclictest60650irq/133-enp0s29f1:rx-022:30:510
3740888991513,1cyclictest60650irq/133-enp0s29f1:rx-021:10:240
3740888991513,1cyclictest60650irq/133-enp0s29f1:rx-000:15:240
3740888991512,2cyclictest61450irq/134-enp0s29f1:tx-023:30:180
3740888991512,2cyclictest60650irq/133-enp0s29f1:rx-021:01:160
3740888991512,2cyclictest60650irq/133-enp0s29f1:rx-000:20:170
3740900991410,3cyclictest3932995-21kworker/u8:0+flush-259:022:25:023
374089599149,3cyclictest4041634-21awk00:00:002
3740895991411,2cyclictest3850219-21kworker/u8:2+flush-259:021:00:162
3740895991410,3cyclictest3932995-21kworker/u8:0+events_unbound22:45:282
374089299149,3cyclictest3850219-21kworker/u8:2+flush-259:021:05:201
374089299149,3cyclictest3754036-21systemd-journal19:11:451
3740892991411,2cyclictest3882027-21kworker/u8:0+flush-259:021:25:181
3740888991413,1cyclictest60650irq/133-enp0s29f1:rx-021:40:260
3740888991412,2cyclictest60650irq/133-enp0s29f1:rx-023:10:210
3740888991412,1cyclictest60650irq/133-enp0s29f1:rx-022:15:180
3740888991412,1cyclictest60650irq/133-enp0s29f1:rx-022:05:220
3740888991412,1cyclictest60650irq/133-enp0s29f1:rx-019:25:250
3740888991412,1cyclictest60650irq/133-enp0s29f1:rx-000:30:220
3740888991412,1cyclictest60650irq/133-enp0s29f1:rx-000:01:230
3740888991411,2cyclictest60650irq/133-enp0s29f1:rx-022:58:010
3740888991411,2cyclictest60650irq/133-enp0s29f1:rx-022:00:430
3740888991411,2cyclictest60650irq/133-enp0s29f1:rx-021:55:580
3740888991411,2cyclictest60650irq/133-enp0s29f1:rx-021:25:180
374089599139,3cyclictest4057318-21unixbench_multi00:15:242
374089599139,3cyclictest3788735-21irqcore19:55:182
374089599138,3cyclictest3932995-21kworker/u8:0+flush-259:022:15:002
3740895991311,1cyclictest3918097-21latency_hist22:00:012
3740895991310,2cyclictest3000063-21rs:main2
3740895991310,1cyclictest0-21swapper/221:55:002
374089299139,3cyclictest3897541-21kworker/u8:4+flush-259:022:35:291
374089299139,3cyclictest3782764-21kworker/u8:0+flush-259:019:50:301
374089299139,2cyclictest0-21swapper/122:55:191
3740892991311,1cyclictest0-21swapper/121:50:001
3740892991310,2cyclictest4026786-21fschecks_time23:45:161
374088899138,4cyclictest3814153-21apt-get20:20:160
3740888991312,1cyclictest60650irq/133-enp0s29f1:rx-023:58:000
3740888991311,2cyclictest60650irq/133-enp0s29f1:rx-023:15:280
3740888991311,1cyclictest60650irq/133-enp0s29f1:rx-021:53:270
3740888991311,1cyclictest60650irq/133-enp0s29f1:rx-021:05:180
3740888991311,1cyclictest60650irq/133-enp0s29f1:rx-020:10:270
3740888991311,1cyclictest60650irq/133-enp0s29f1:rx-000:06:360
3740888991310,2cyclictest60650irq/133-enp0s29f1:rx-023:02:480
3740888991310,2cyclictest60650irq/133-enp0s29f1:rx-019:10:140
3740888991310,2cyclictest3875306-21unixbench_multi21:15:260
374089599129,2cyclictest3974980-21fschecks_time22:55:172
374089599128,3cyclictest4031734-21cpuspeed_turbos23:50:152
374089599128,2cyclictest0-21swapper/220:55:132
374089599127,4cyclictest3819040-21kworker/u8:0+flush-259:020:25:152
374089299129,2cyclictest3953491-21unixbench_singl22:30:281
374089299129,2cyclictest3943986-21cpuspeed_turbos22:25:171
374089299128,2cyclictest0-21swapper/100:10:001
374088899129,2cyclictest60650irq/133-enp0s29f1:rx-020:40:260
3740888991210,2cyclictest60650irq/133-enp0s29f1:rx-022:35:540
3740888991210,2cyclictest60650irq/133-enp0s29f1:rx-020:45:250
374089599118,2cyclictest3963208-21needreboot22:40:222
374089599118,1cyclictest0-21swapper/223:20:182
374089599117,2cyclictest3837815-21kthreadcore20:40:192
374089599116,3cyclictest4047482-21cron00:10:002
374089599110,9cyclictest3819014-21latency_hist20:25:012
374089299119,1cyclictest0-21swapper/120:15:241
374089299118,2cyclictest4000608-21fschecks_count23:20:181
374089299118,2cyclictest0-21swapper/119:59:591
374089299116,4cyclictest3754036-21systemd-journal20:49:591
374089299107,2cyclictest3824425-21kworker/u8:3+flush-259:020:40:171
374088899100,9cyclictest4078380-21turbostat00:40:000
37408929994,3cyclictest3860102-21sshd21:02:481
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional