You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 13:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Sat Nov 22, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
222875994135,4cyclictest456433-21kworker/u8:2+flush-259:023:20:000
222883994036,3cyclictest301100-21kworker/u8:4+flush-259:020:30:002
222883993835,3cyclictest238359-21kworker/u8:0+flush-259:019:45:002
222875993631,3cyclictest263864-21kworker/u8:3+flush-259:020:49:590
222886993530,4cyclictest399418-21latency_hist22:00:013
222878993529,4cyclictest508053-21/usr/sbin/munin23:45:151
222878993529,4cyclictest368540-21latency_hist21:30:011
222878993529,4cyclictest368540-21latency_hist21:30:011
222883993431,2cyclictest378854-21kworker/u8:3+events_unbound21:45:152
222878993428,4cyclictest420701-21/usr/sbin/munin22:20:131
222875993327,5cyclictest2698871-21systemd-journal23:30:000
222886993228,3cyclictest280006-21apt-get20:05:033
222883993229,2cyclictest222825-21kworker/u8:3+events_unbound19:15:252
222875993127,3cyclictest544441-21kworker/u8:4+flush-259:000:40:000
222883993027,2cyclictest332227-21kworker/u8:2+flush-259:021:25:252
222883993027,2cyclictest332227-21kworker/u8:2+flush-259:021:25:242
222883993026,3cyclictest311686-21/usr/sbin/munin20:35:162
222883993026,3cyclictest3000063-21rs:main2
222878993024,4cyclictest435883-21latency_hist22:35:011
222875993026,3cyclictest347692-21kworker/u8:0+flush-259:022:35:260
222883992822,5cyclictest222790-21latency_hist19:10:012
222875992821,5cyclictest320821-21ntpq20:40:190
222886992722,3cyclictest456433-21kworker/u8:2+flush-259:023:35:253
222883992722,3cyclictest477199-21/usr/sbin/munin23:15:142
222875992720,5cyclictest310547-21ntpq20:30:200
222886992623,2cyclictest189541-21kworker/u8:2+flush-259:019:20:153
222878992619,4cyclictest389114-21latency_hist21:50:001
222886992522,2cyclictest263864-21kworker/u8:3+flush-259:020:30:253
222886992521,3cyclictest435896-21kworker/u8:3+flush-259:022:50:013
222886992521,3cyclictest347692-21kworker/u8:0+events_unbound22:35:253
222886992520,4cyclictest269278-21latency_hist19:55:013
222883992522,2cyclictest263864-21kworker/u8:3+flush-259:020:15:002
222883992521,3cyclictest332017-21kworker/u8:1+events_unbound21:30:142
222875992520,4cyclictest232624-21kworker/u8:4+flush-259:019:35:010
222886992421,2cyclictest347692-21kworker/u8:0+flush-259:022:30:203
222886992420,3cyclictest523257-21kworker/u8:1+flush-259:000:05:253
222886992420,3cyclictest189541-21kworker/u8:2+flush-259:019:20:003
222883992422,2cyclictest212258-21kworker/u8:1+flush-259:019:15:012
222883992421,2cyclictest332017-21kworker/u8:1+flush-259:021:40:212
222883992418,4cyclictest2698871-21systemd-journal20:00:002
222878992422,1cyclictest399432-21kworker/u8:3+flush-259:022:10:001
222886992319,3cyclictest290774-21kworker/u8:0+events_unbound20:17:013
222886992318,4cyclictest564828-21latency_hist00:40:013
222883992318,3cyclictest543288-21ntpq00:15:232
222878992317,4cyclictest2698871-21systemd-journal21:32:241
222883992218,3cyclictest456433-21kworker/u8:2+flush-259:023:45:162
222883992218,3cyclictest399432-21kworker/u8:3+events_unbound22:11:262
222883992215,5cyclictest2698871-21systemd-journal22:40:012
222878992217,3cyclictest189541-21kworker/u8:2+events_unbound20:20:011
222886992118,2cyclictest290774-21kworker/u8:0+events_unbound20:40:223
222886992118,2cyclictest189541-21kworker/u8:2+events_unbound20:00:163
222886992117,3cyclictest332227-21kworker/u8:2+events_unbound21:35:203
222886992117,3cyclictest2698871-21systemd-journal20:51:153
22288399217,3cyclictest301100-21kworker/u8:4+events_unbound20:50:202
222883992118,2cyclictest456433-21kworker/u8:2+events_unbound23:20:222
222883992118,2cyclictest232624-21kworker/u8:4+flush-259:019:25:182
222875992117,3cyclictest189541-21kworker/u8:2+flush-259:020:15:210
222875992115,4cyclictest456433-21kworker/u8:2+flush-259:023:30:170
222875992115,4cyclictest341485-21ntpq21:00:220
222875992114,5cyclictest456415-21latency_hist22:55:000
222875992114,5cyclictest238338-21latency_hist19:25:010
22287599210,19cyclictest3000063-21rs:main0
222886992017,2cyclictest435896-21kworker/u8:3+events_unbound23:10:193
222886992016,3cyclictest347692-21kworker/u8:0+events_unbound21:50:233
222886992013,5cyclictest445247-21ntpq22:40:213
222883992017,2cyclictest399432-21kworker/u8:3+events_unbound22:04:562
222883992017,2cyclictest290774-21kworker/u8:0+events_unbound20:16:232
222883992017,2cyclictest238359-21kworker/u8:0+events_unbound19:35:142
222883992016,3cyclictest472036-21kworker/u8:1+events_unbound23:14:562
222883992016,3cyclictest2698871-21systemd-journal21:35:002
222878992016,3cyclictest493075-21irqcore23:30:161
222878992016,3cyclictest435896-21kworker/u8:3+flush-259:023:09:541
222878992016,3cyclictest248848-21/usr/sbin/munin19:35:221
22287899201,5cyclictest518097-21cron23:55:001
222878992014,4cyclictest2698871-21systemd-journal20:30:011
222875992014,4cyclictest533950-21kworker/u8:2+flush-259:000:15:010
222875992012,5cyclictest549376-21latency_hist00:25:010
222886991916,2cyclictest435896-21kworker/u8:3+events_unbound23:40:243
222886991916,2cyclictest263864-21kworker/u8:3+events_unbound20:25:223
222883991916,2cyclictest263864-21kworker/u8:3+flush-259:019:50:262
222883991916,2cyclictest0-21swapper/219:50:002
222883991913,4cyclictest280642-21cp20:05:022
22287899197,4cyclictest497848-21apt-get23:35:121
22287899195,2cyclictest539074-21awk00:15:001
222878991914,4cyclictest456433-21kworker/u8:2+flush-259:023:25:001
222875991913,4cyclictest533912-21latency_hist00:10:010
222886991815,3cyclictest332227-21kworker/u8:2+events_unbound20:55:203
222886991815,2cyclictest399645-21kworker/u8:4+events_unbound22:14:253
222886991815,2cyclictest232624-21kworker/u8:4+events_unbound19:25:233
222886991814,3cyclictest435896-21kworker/u8:3+events_unbound23:02:263
222886991814,3cyclictest353255-21kworker/u8:4+events_unbound21:20:153
222886991814,3cyclictest263864-21kworker/u8:3+events_unbound20:20:193
222883991815,2cyclictest290774-21kworker/u8:0+events_unbound20:21:122
222883991814,3cyclictest523257-21kworker/u8:1+flush-259:000:10:192
222883991814,3cyclictest347692-21kworker/u8:0+events_unbound22:24:562
222883991814,3cyclictest332227-21kworker/u8:2+events_unbound20:56:182
222883991812,5cyclictest2698871-21systemd-journal21:10:002
222883991812,4cyclictest528766-21cp00:05:012
22287899187,1cyclictest0-21swapper/120:30:171
22287899181,5cyclictest404574-21awk22:04:591
222878991815,2cyclictest394271-21cron21:55:001
222878991811,5cyclictest419580-21ntpq22:15:211
222875991815,2cyclictest435896-21kworker/u8:3+flush-259:022:55:250
222875991814,2cyclictest553659-21ntp_offset00:25:200
222875991814,2cyclictest0-21swapper/023:55:000
222886991715,2cyclictest456433-21kworker/u8:2+events_unbound23:50:213
222886991714,2cyclictest467334-21irqcore23:05:163
222886991714,2cyclictest435896-21kworker/u8:3+events_unbound23:21:413
222886991714,2cyclictest290739-21idleruntime-cro20:15:003
222886991714,2cyclictest263864-21kworker/u8:3+events_unbound20:45:153
222886991713,3cyclictest253798-21kworker/u8:1+events_unbound19:45:113
222886991713,3cyclictest189541-21kworker/u8:2+flush-259:019:10:233
222883991714,2cyclictest430437-21kworker/u8:2+events_unbound22:31:252
222883991714,2cyclictest430437-21kworker/u8:2+events_unbound22:29:242
222883991714,2cyclictest347692-21kworker/u8:0+flush-259:021:15:002
222883991714,2cyclictest347692-21kworker/u8:0+flush-259:021:14:592
222883991714,2cyclictest263864-21kworker/u8:3+flush-259:020:45:212
222883991714,2cyclictest232624-21kworker/u8:4+flush-259:019:20:212
222883991713,3cyclictest559691-21kworker/u8:1+flush-259:000:35:442
222883991713,3cyclictest523257-21kworker/u8:1+flush-259:000:05:232
222883991711,4cyclictest465783-21ntpq23:00:232
222883991711,4cyclictest363355-21latency_hist21:25:012
22287899174,3cyclictest3000061-21in:imuxsock23:18:241
22287899174,2cyclictest456393-21cron22:55:011
222878991715,1cyclictest274874-21cut20:00:001
222878991712,3cyclictest2698871-21systemd-journal22:25:241
222878991711,4cyclictest472070-21apt-get23:10:171
22287899171,14cyclictest2698871-21systemd-journal20:00:211
22287899170,5cyclictest554479-21idleruntime-cro00:30:011
22287899170,5cyclictest528406-21dpkg00:05:001
22287899170,1cyclictest0-21swapper/120:35:181
222875991714,2cyclictest61450irq/134-enp0s29f1:tx-000:18:410
222875991714,2cyclictest60650irq/133-enp0s29f1:rx-021:48:520
222875991712,3cyclictest3000063-21rs:main0
222886991613,2cyclictest238359-21kworker/u8:0+events_unbound19:33:223
222886991613,2cyclictest0-21swapper/320:00:003
222886991612,3cyclictest523257-21kworker/u8:1+flush-259:000:00:173
222886991612,3cyclictest399645-21kworker/u8:4+flush-259:022:00:263
222886991611,4cyclictest358188-21latency_hist21:20:003
222886991610,4cyclictest363592-21/usr/sbin/munin21:25:163
222886991610,4cyclictest363592-21/usr/sbin/munin21:25:163
222883991613,2cyclictest544441-21kworker/u8:4+events_unbound00:34:562
222883991613,2cyclictest456433-21kworker/u8:2+flush-259:023:25:162
222883991613,2cyclictest435896-21kworker/u8:3+flush-259:023:35:212
222883991613,2cyclictest332017-21kworker/u8:1+events_unbound21:00:232
222883991613,2cyclictest290774-21kworker/u8:0+events_unbound20:40:202
222883991612,3cyclictest543940-21kworker/u8:3+events_unbound00:20:132
222883991612,3cyclictest446424-21apt-get22:45:142
222883991612,3cyclictest435896-21kworker/u8:3+events_unbound23:58:002
222883991612,3cyclictest332227-21kworker/u8:2+events_unbound21:15:032
22287899169,5cyclictest446163-21latency_hist22:45:001
22287899164,3cyclictest534132-21cut00:10:011
22287899164,2cyclictest327342-21fschecks_time20:50:131
22287899164,2cyclictest0-21swapper/100:15:161
22287899161,5cyclictest414471-21ntpq22:10:211
22287899161,4cyclictest523211-21cron00:00:001
222878991610,5cyclictest3002-21cron23:05:001
22287899160,4cyclictest441032-21idleruntime-cro22:40:011
222875991614,1cyclictest60650irq/133-enp0s29f1:rx-023:00:170
222875991614,1cyclictest60650irq/133-enp0s29f1:rx-019:54:390
222875991613,2cyclictest60650irq/133-enp0s29f1:rx-023:39:200
222875991613,2cyclictest60650irq/133-enp0s29f1:rx-022:44:100
222875991613,2cyclictest60650irq/133-enp0s29f1:rx-021:23:490
222875991613,2cyclictest60650irq/133-enp0s29f1:rx-020:27:150
222875991610,4cyclictest2698871-21systemd-journal20:07:190
222886991512,2cyclictest533950-21kworker/u8:2+events_unbound00:15:143
222886991512,2cyclictest435896-21kworker/u8:3+events_unbound23:29:553
222886991512,2cyclictest420479-21munin-plugin-st22:19:593
222886991512,2cyclictest347692-21kworker/u8:0+flush-259:022:25:193
222886991512,2cyclictest332017-21kworker/u8:1+flush-259:021:30:163
222886991511,3cyclictest435896-21kworker/u8:3+events_unbound23:15:023
222886991511,3cyclictest332017-21kworker/u8:1+flush-259:021:05:153
222886991511,3cyclictest263864-21kworker/u8:3+events_unbound20:35:143
222883991512,3cyclictest347692-21kworker/u8:0+events_unbound22:15:192
222883991512,2cyclictest441709-21irqcore22:40:172
222883991512,2cyclictest435896-21kworker/u8:3+events_unbound22:57:252
222883991512,2cyclictest435896-21kworker/u8:3+events_unbound22:54:082
222883991512,2cyclictest189541-21kworker/u8:2+flush-259:020:00:212
222883991510,4cyclictest189541-21kworker/u8:2+flush-259:020:30:442
22287899159,4cyclictest544476-21apt-get00:20:121
22287899151,4cyclictest374219-21fschecks_time21:35:171
222878991512,2cyclictest0-21swapper/119:35:011
222878991511,3cyclictest399400-21cron22:00:001
222878991510,3cyclictest2698871-21systemd-journal21:06:251
22287899150,5cyclictest2698871-21systemd-journal21:25:001
22287599159,4cyclictest2698871-21systemd-journal23:10:000
22287599158,5cyclictest347468-21latency_hist21:10:010
222875991513,1cyclictest60650irq/133-enp0s29f1:rx-023:46:020
222875991513,1cyclictest60650irq/133-enp0s29f1:rx-022:25:230
222875991513,1cyclictest60650irq/133-enp0s29f1:rx-022:00:140
222875991513,1cyclictest60650irq/133-enp0s29f1:rx-021:25:230
222875991513,1cyclictest60650irq/133-enp0s29f1:rx-021:25:230
222875991513,1cyclictest60650irq/133-enp0s29f1:rx-020:55:120
222875991513,1cyclictest60650irq/133-enp0s29f1:rx-020:54:400
222875991513,1cyclictest60650irq/133-enp0s29f1:rx-020:35:240
222875991513,1cyclictest60650irq/133-enp0s29f1:rx-019:10:140
222875991512,2cyclictest61450irq/134-enp0s29f1:tx-019:25:210
222875991512,2cyclictest60650irq/133-enp0s29f1:rx-022:18:160
222875991512,2cyclictest60650irq/133-enp0s29f1:rx-021:52:380
222875991512,2cyclictest60650irq/133-enp0s29f1:rx-021:11:410
222875991512,2cyclictest60650irq/133-enp0s29f1:rx-021:11:400
222886991411,2cyclictest554357-21kworker/u8:0+events_unbound00:30:253
222886991411,2cyclictest253355-21unixbench_singl19:35:243
222886991410,3cyclictest544441-21kworker/u8:4+events_unbound00:29:353
222886991410,3cyclictest523257-21kworker/u8:1+events_unbound00:10:183
222886991410,3cyclictest347692-21kworker/u8:0+events_unbound22:21:343
22288399141,3cyclictest554482-21awk00:30:002
222883991411,2cyclictest456433-21kworker/u8:2+events_unbound23:30:272
222883991411,2cyclictest238359-21kworker/u8:0+events_unbound19:34:562
222883991410,3cyclictest347692-21kworker/u8:0+events_unbound21:50:212
22287899149,3cyclictest332017-21kworker/u8:1+events_unbound21:40:171
22287899148,4cyclictest289907-21sshd20:10:191
22287899141,5cyclictest232223-21ntpq19:15:211
22287899140,7cyclictest342327-21cron21:04:591
22287899140,7cyclictest0-21swapper/121:10:171
22287899140,7cyclictest0-21swapper/121:10:161
22287599149,4cyclictest2698871-21systemd-journal22:07:240
22287599148,4cyclictest263864-21kworker/u8:3+flush-259:020:20:200
222875991412,2cyclictest60650irq/133-enp0s29f1:rx-021:58:170
222875991412,1cyclictest60650irq/133-enp0s29f1:rx-022:10:130
222875991412,1cyclictest60650irq/133-enp0s29f1:rx-021:16:410
222875991411,2cyclictest60650irq/133-enp0s29f1:rx-023:40:510
222875991411,2cyclictest60650irq/133-enp0s29f1:rx-023:14:140
222875991411,2cyclictest60650irq/133-enp0s29f1:rx-022:49:390
222875991411,2cyclictest60650irq/133-enp0s29f1:rx-019:37:450
222886991310,2cyclictest238359-21kworker/u8:0+events_unbound19:44:223
22288399139,3cyclictest435896-21kworker/u8:3+events_unbound23:05:212
22288399138,4cyclictest435896-21kworker/u8:3+events_unbound23:51:232
222883991310,2cyclictest347692-21kworker/u8:0+events_unbound22:05:152
22287899138,3cyclictest263864-21kworker/u8:3+flush-259:019:50:171
22287899137,5cyclictest2698871-21systemd-journal00:35:341
22287899137,4cyclictest461545-21latency_hist23:00:001
22287899133,3cyclictest285300-21seq20:06:181
22287599138,1cyclictest60650irq/133-enp0s29f1:rx-021:30:180
222875991312,1cyclictest60650irq/133-enp0s29f1:rx-020:13:170
222875991311,1cyclictest60650irq/133-enp0s29f1:rx-023:20:190
222875991311,1cyclictest60650irq/133-enp0s29f1:rx-019:55:230
222875991311,1cyclictest60650irq/133-enp0s29f1:rx-000:01:250
222875991310,2cyclictest60650irq/133-enp0s29f1:rx-022:30:240
22288699129,2cyclictest339732-21kthreadcore21:00:193
22288699128,3cyclictest456433-21kworker/u8:2+flush-259:023:30:313
22288699128,3cyclictest347692-21kworker/u8:0+events_unbound21:43:243
22288399129,2cyclictest395562-21kthreadcore21:55:202
22287899129,2cyclictest0-21swapper/120:20:271
22287899128,3cyclictest456433-21kworker/u8:2+flush-259:023:41:471
222878991210,1cyclictest0-21swapper/119:30:001
22287599129,2cyclictest60650irq/133-enp0s29f1:rx-023:59:350
22287599129,2cyclictest60650irq/133-enp0s29f1:rx-019:17:030
22287599127,4cyclictest254042-21apt-get19:40:160
222875991210,1cyclictest60650irq/133-enp0s29f1:rx-021:40:130
222875991210,1cyclictest60650irq/133-enp0s29f1:rx-000:30:160
22288699118,2cyclictest457693-21tr22:55:173
22288699118,2cyclictest332017-21kworker/u8:1+flush-259:021:45:163
22288699117,3cyclictest516-21gdbus00:00:003
22287899118,2cyclictest258002-21ntp_kernel_err19:40:241
22287899117,3cyclictest554357-21kworker/u8:0+flush-259:000:31:401
22287899117,3cyclictest347692-21kworker/u8:0+flush-259:022:48:551
22287899116,4cyclictest2698871-21systemd-journal21:00:001
22287899116,4cyclictest232624-21kworker/u8:4+flush-259:019:20:261
22287899115,4cyclictest456433-21kworker/u8:2+flush-259:023:25:101
22287899110,4cyclictest3222932sh20:45:161
22287899108,1cyclictest0-21swapper/121:15:271
22287899106,3cyclictest259173-21apt-get19:45:111
22287899106,3cyclictest227358-21sed19:10:221
22287599108,1cyclictest60650irq/133-enp0s29f1:rx-020:00:140
2228789996,2cyclictest316909-21cpuspeed_turbos20:40:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional