You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 02:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Wed Dec 31, 2025 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1937792994941,5cyclictest1937714-21latency_hist07:10:002
1937788993731,4cyclictest2211521-21ntpq11:30:241
1937785993729,6cyclictest2268055-21ntpq12:25:240
1937797993624,10cyclictest2201034-21kworker/u8:2+flush-259:012:15:003
1937788993631,3cyclictest2077975-21kworker/u8:4+flush-259:010:00:001
1937785993629,5cyclictest2191829-21latency_hist11:15:010
1937797993525,6cyclictest252-21systemd-journal11:20:003
1937788993528,5cyclictest252-21systemd-journal09:15:001
1937788993430,3cyclictest2077975-21kworker/u8:4+events_unbound10:40:201
1937785993427,5cyclictest2062116-21latency_hist09:10:010
1937788993329,2cyclictest3094483-21kworker/u8:5+flush-259:007:45:171
1937788993326,5cyclictest2055626-21ntpq09:00:221
1937785993328,3cyclictest2165809-21apt-get10:50:120
1937785993326,5cyclictest2273179-21ntpq12:30:220
1937792993225,4cyclictest2169865-21ntpq10:50:222
1937788993229,2cyclictest2253528-21kworker/u8:3+events_unbound12:30:151
1937788993127,3cyclictest1879402-21kworker/u8:2+events_unbound07:35:001
1937788993026,3cyclictest2077975-21kworker/u8:4+flush-259:010:30:251
1937788993023,5cyclictest2061254-21ntpq09:05:211
1937788993023,5cyclictest2008987-21ntpq08:15:221
1937792992925,3cyclictest3094483-21kworker/u8:5+flush-259:007:30:242
1937792992925,3cyclictest2077975-21kworker/u8:4+flush-259:010:10:002
1937792992925,3cyclictest2077975-21kworker/u8:4+flush-259:010:10:002
1937788992925,3cyclictest2155453-21kworker/u8:0+flush-259:012:20:151
1937788992821,5cyclictest252-21systemd-journal10:23:331
1937785992821,5cyclictest2139765-21latency_hist10:25:010
1937792992722,3cyclictest2014987-21kworker/u8:1+flush-259:009:25:012
1937792992721,4cyclictest1948081-21latency_hist07:20:002
1937788992724,2cyclictest2201034-21kworker/u8:2+events_unbound11:40:151
1937788992722,3cyclictest2165156-21kworker/u8:3+flush-259:011:20:161
1937788992721,4cyclictest2087423-21ntpq09:30:231
1937785992721,4cyclictest2088266-21latency_hist09:35:010
1937788992623,2cyclictest1879402-21kworker/u8:2+events_unbound08:00:001
1937788992622,3cyclictest1943199-21apt-get07:15:111
1937797992520,3cyclictest2233147-21/usr/sbin/munin11:55:243
1937797992519,4cyclictest2155224-21latency_hist10:40:003
1937788992519,3cyclictest1879402-21kworker/u8:2+flush-259:008:00:191
1937797992420,3cyclictest1999547-21kworker/u8:3+flush-259:009:35:333
1937797992419,4cyclictest1978950-21kworker/u8:0+flush-259:007:55:153
1937797992417,5cyclictest1998656-21ntpq08:05:223
1937792992420,3cyclictest1978950-21kworker/u8:0+events_unbound08:05:232
1937792992419,3cyclictest2029506-21ntpq08:35:212
1937788992421,2cyclictest2201034-21kworker/u8:2+flush-259:012:15:241
1937788992417,5cyclictest2010272-21diskstats08:20:131
1937785992421,1cyclictest1-21systemd11:57:320
1937797992319,3cyclictest507-21in:imuxsock11:40:003
1937797992318,3cyclictest2103724-21kworker/u8:2+flush-259:010:00:173
1937785992317,4cyclictest1968632-21latency_hist07:40:000
1937797992216,4cyclictest1969050-21diskstats07:40:123
1937797992215,5cyclictest1962652-21ntpq07:30:253
1937792992215,5cyclictest2196068-21ntpq11:15:212
1937788992219,2cyclictest1879402-21kworker/u8:2+events_unbound08:25:141
1937788992218,3cyclictest2077975-21kworker/u8:4+flush-259:009:50:191
1937788992218,3cyclictest2025308-21kworker/u8:0+flush-259:008:40:241
1937788992218,3cyclictest1979191-21apt-get07:50:121
1937788992218,3cyclictest1979191-21apt-get07:50:121
1937788992218,2cyclictest1999547-21kworker/u8:3+events_unbound09:25:191
1937785992217,4cyclictest2170974-21apt-get10:55:140
1937797992119,1cyclictest1879402-21kworker/u8:2+flush-259:008:00:173
1937797992116,3cyclictest2253745-21/usr/sbin/munin12:15:143
1937797992115,4cyclictest2150065-21latency_hist10:35:003
1937792992118,2cyclictest2165156-21kworker/u8:3+flush-259:011:10:032
1937792992118,2cyclictest2124365-21kworker/u8:0+flush-259:010:20:162
1937792992116,4cyclictest2025308-21kworker/u8:0+events_unbound08:50:202
1937792992116,3cyclictest2201034-21kworker/u8:2+flush-259:012:15:172
1937788992118,3cyclictest2093419-21kworker/u8:1+events_unbound10:10:251
1937788992118,2cyclictest2143851-21kworker/u8:1+events_unbound10:40:011
1937788992117,3cyclictest2077975-21kworker/u8:4+events_unbound09:40:201
1937788992117,3cyclictest2067698-21kworker/u8:0+flush-259:009:24:381
1937788992117,3cyclictest1978950-21kworker/u8:0+flush-259:008:05:201
1937788992117,3cyclictest1879402-21kworker/u8:2+flush-259:007:10:181
1937788992117,3cyclictest1879402-21kworker/u8:2+events_unbound08:10:131
1937788992115,4cyclictest2050957-21latency_hist09:00:011
1937797992016,3cyclictest2077975-21kworker/u8:4+flush-259:009:40:213
1937797992013,4cyclictest2186243-21latency_hist11:10:013
1937792992017,2cyclictest2143851-21kworker/u8:1+flush-259:010:35:202
1937792992016,3cyclictest2243204-21kworker/u8:1+events_unbound12:10:232
1937792992015,4cyclictest2253528-21kworker/u8:3+events_unbound12:30:192
1937792992014,4cyclictest2274301-21/usr/sbin/munin12:35:182
1937792992014,4cyclictest2155453-21kworker/u8:0+flush-259:012:05:172
1937792992014,4cyclictest2024438-21ntpq08:30:222
1937788992018,1cyclictest2077975-21kworker/u8:4+events_unbound10:25:231
1937788992017,2cyclictest1999547-21kworker/u8:3+events_unbound09:35:231
1937788992017,2cyclictest1879402-21kworker/u8:2+flush-259:007:20:181
1937788992016,3cyclictest2253528-21kworker/u8:3+flush-259:012:29:381
1937788992013,5cyclictest252-21systemd-journal12:39:591
1937788992013,5cyclictest2226951-21ntpq11:45:231
1937785992016,2cyclictest2050960-21apt-get09:00:020
193778599200,19cyclictest1-21systemd10:07:340
193778599200,19cyclictest1-21systemd10:07:330
1937797991913,4cyclictest509-21rs:main3
1937797991912,5cyclictest2071941-21ntpq09:15:223
1937792991916,2cyclictest2253528-21kworker/u8:3+flush-259:012:25:292
1937788991916,2cyclictest2243204-21kworker/u8:1+flush-259:012:10:201
1937788991916,2cyclictest2165156-21kworker/u8:3+flush-259:011:15:211
1937788991916,2cyclictest2165156-21kworker/u8:3+flush-259:011:10:261
1937788991916,2cyclictest2025308-21kworker/u8:0+flush-259:008:35:221
1937788991915,3cyclictest2155453-21kworker/u8:0+flush-259:012:00:161
1937788991915,3cyclictest2124365-21kworker/u8:0+flush-259:010:15:151
193778599193,6cyclictest1988067-21cut07:55:170
193778599193,11cyclictest2029536-21ntpq08:35:210
193778599192,5cyclictest2232172-21ps11:50:240
1937785991916,2cyclictest2014949-21cron08:25:000
1937785991915,3cyclictest2155453-21kworker/u8:0+flush-259:011:35:180
1937797991815,2cyclictest2245748-21kthreadcore12:05:183
1937797991814,3cyclictest252-21systemd-journal10:50:003
1937797991814,3cyclictest2014987-21kworker/u8:1+flush-259:009:05:003
1937797991814,2cyclictest2227780-21idleruntime-cro11:49:593
1937797991813,4cyclictest1948339-21apt-get07:20:113
1937792991815,2cyclictest2201034-21kworker/u8:2+events_unbound11:50:222
1937792991814,3cyclictest2155453-21kworker/u8:0+events_unbound11:20:242
1937792991814,2cyclictest1994434-21unin-run08:05:002
1937788991815,2cyclictest2165156-21kworker/u8:3+flush-259:011:00:251
1937788991815,2cyclictest2134646-21kworker/u8:2+events_unbound10:45:191
1937788991815,2cyclictest2025308-21kworker/u8:0+flush-259:008:50:191
1937788991814,3cyclictest2155453-21kworker/u8:0+flush-259:011:35:261
1937788991814,3cyclictest2155453-21kworker/u8:0+events_unbound11:55:161
193778599184,12cyclictest2036068-21fschecks_time08:45:160
1937785991813,4cyclictest2180212-21ntpq11:00:230
1937785991813,3cyclictest2114286-21apt-get10:00:120
1937785991812,4cyclictest2242270-21ntpq12:00:200
1937785991811,5cyclictest252-21systemd-journal08:05:000
1937797991714,2cyclictest2203365-21kthreadcore11:25:193
1937797991714,2cyclictest1999547-21kworker/u8:3+flush-259:008:40:163
1937797991713,3cyclictest507-21in:imuxsock12:40:003
1937797991713,2cyclictest0-21swapper/309:10:013
1937797991712,3cyclictest2155453-21kworker/u8:0+flush-259:011:35:003
1937792991714,2cyclictest1999547-21kworker/u8:3+flush-259:008:40:232
1937792991713,3cyclictest3094483-21kworker/u8:5+flush-259:007:25:232
1937792991713,3cyclictest2155453-21kworker/u8:0+events_unbound12:00:202
1937792991713,3cyclictest2134608-21cron10:20:002
1937792991713,3cyclictest2093419-21kworker/u8:1+flush-259:010:10:202
1937792991711,4cyclictest2067698-21kworker/u8:0+flush-259:009:30:192
1937788991713,3cyclictest2201034-21kworker/u8:2+events_unbound11:25:011
1937788991713,3cyclictest2103724-21kworker/u8:2+events_unbound10:04:121
1937788991711,4cyclictest2165156-21kworker/u8:3+flush-259:010:55:011
193778599172,10cyclictest2257769-21ntpq12:15:210
1937785991713,3cyclictest2155453-21kworker/u8:0+events_unbound11:05:180
1937785991713,2cyclictest2041335-21irqcore08:50:150
1937785991712,1cyclictest191rcu_preempt09:23:020
1937785991711,1cyclictest191rcu_preempt07:44:130
193778599170,6cyclictest1979150-21date07:50:010
193778599170,6cyclictest1979150-21date07:50:010
1937797991613,2cyclictest2971-21cron09:50:003
1937797991613,2cyclictest2132037-21kthreadcore10:15:193
1937797991613,2cyclictest0-21swapper/309:29:593
1937797991612,3cyclictest2155453-21kworker/u8:0+flush-259:010:43:043
1937797991612,2cyclictest0-21swapper/310:10:183
1937797991611,3cyclictest1879402-21kworker/u8:2+flush-259:007:15:003
1937797991610,4cyclictest2045796-21latency_hist08:55:013
1937792991613,2cyclictest2143851-21kworker/u8:1+flush-259:010:30:152
1937792991612,3cyclictest1879402-21kworker/u8:2+events_unbound07:50:222
1937792991612,3cyclictest1879402-21kworker/u8:2+events_unbound07:50:222
1937792991612,2cyclictest1999547-21kworker/u8:3+flush-259:009:15:172
1937788991612,3cyclictest2077975-21kworker/u8:4+events_unbound10:10:001
1937788991612,3cyclictest2077975-21kworker/u8:4+events_unbound10:10:001
193778599164,2cyclictest1974026-21apt-get07:45:100
193778599163,4cyclictest1952137-21memory07:20:190
193778599162,5cyclictest2022671-21kthreadcore08:30:190
193778599161,4cyclictest2020230-21date08:30:010
1937785991611,1cyclictest191rcu_preempt12:05:120
1937785991611,1cyclictest191rcu_preempt09:25:020
193779799159,4cyclictest1957512-21ntpq07:25:233
193779799158,5cyclictest2274057-21latency_hist12:35:003
1937792991512,2cyclictest2207904-21irqcore11:30:182
1937792991511,3cyclictest1879402-21kworker/u8:2+flush-259:007:35:152
1937792991511,2cyclictest2062565-21latency_hist09:10:012
1937788991512,2cyclictest3094483-21kworker/u8:5+flush-259:007:35:261
1937788991512,2cyclictest1879402-21kworker/u8:2+events_unbound07:40:151
1937788991511,3cyclictest2155453-21kworker/u8:0+flush-259:011:07:441
193778599158,5cyclictest513-21gdbus08:45:010
193778599152,4cyclictest2207150-21sshd11:29:290
193778599151,4cyclictest2003763-21ld08:10:190
1937785991514,1cyclictest60350irq/132-enp0s29f1:rx-009:53:530
1937785991513,2cyclictest60350irq/132-enp0s29f1:rx-007:25:130
1937785991513,1cyclictest60350irq/132-enp0s29f1:rx-012:10:250
1937785991513,1cyclictest60350irq/132-enp0s29f1:rx-008:55:220
1937785991510,3cyclictest2145164-21apt-get10:30:120
193779799149,3cyclictest2014987-21kworker/u8:1+flush-259:008:55:283
193779799148,4cyclictest252-21systemd-journal11:24:333
193779799148,4cyclictest2187088-21diskstats11:10:163
1937797991410,3cyclictest2077975-21kworker/u8:4+flush-259:010:22:033
1937797991410,2cyclictest1967657-21needreboot07:35:203
193779299148,4cyclictest2114019-21latency_hist10:00:002
1937792991412,1cyclictest2077975-21kworker/u8:4+flush-259:010:25:182
1937792991411,2cyclictest2253528-21kworker/u8:3+events_unbound12:25:012
1937792991411,2cyclictest1879402-21kworker/u8:2+events_unbound07:10:232
193779299141,10cyclictest2056537-21awk09:05:002
1937792991410,3cyclictest2201034-21kworker/u8:2+events_unbound11:40:232
1937792991410,3cyclictest1879402-21kworker/u8:2+flush-259:008:25:442
1937792991410,3cyclictest1879402-21kworker/u8:2+flush-259:008:20:212
1937788991411,2cyclictest2077975-21kworker/u8:4+flush-259:009:45:211
1937788991411,2cyclictest1999547-21kworker/u8:3+events_unbound08:48:571
1937788991410,3cyclictest2155453-21kworker/u8:0+flush-259:011:50:341
1937788991410,3cyclictest2134646-21kworker/u8:2+flush-259:010:55:431
1937788991410,3cyclictest1999547-21kworker/u8:3+flush-259:009:15:201
193778599142,4cyclictest1940018-21grep07:10:180
193778599141,4cyclictest2071930-21ntpq09:15:230
193778599141,4cyclictest1945521-21grep07:15:170
1937785991413,1cyclictest60350irq/132-enp0s29f1:rx-010:15:480
1937785991413,1cyclictest60350irq/132-enp0s29f1:rx-010:11:080
1937785991413,1cyclictest60350irq/132-enp0s29f1:rx-009:40:260
1937785991412,1cyclictest60350irq/132-enp0s29f1:rx-010:40:130
1937785991412,1cyclictest60350irq/132-enp0s29f1:rx-009:55:280
1937785991412,1cyclictest60350irq/132-enp0s29f1:rx-009:45:200
1937785991412,1cyclictest60350irq/132-enp0s29f1:rx-007:30:140
1937785991411,2cyclictest60350irq/132-enp0s29f1:rx-010:45:170
1937785991410,1cyclictest60350irq/132-enp0s29f1:rx-011:30:180
193779799138,4cyclictest2134646-21kworker/u8:2+flush-259:010:50:253
193779799137,4cyclictest2232083-21ntpq11:50:243
1937797991310,2cyclictest2020127-21latency_hist08:30:003
193779299138,4cyclictest1879402-21kworker/u8:2+flush-259:008:10:172
193779299138,3cyclictest0-21swapper/209:00:012
1937792991310,2cyclictest2103724-21kworker/u8:2+events_unbound10:00:182
1937792991310,2cyclictest1937775-21cyclictest08:00:002
193778899139,3cyclictest2022672-21kthreadcore08:30:181
1937788991310,2cyclictest2245091-21kthreadcore12:05:181
193778599136,1cyclictest191rcu_preempt09:35:160
193778599132,2cyclictest2008953-21gcc08:15:210
193778599131,4cyclictest16-21ksoftirqd/012:20:230
193778599131,3cyclictest16-21ksoftirqd/011:20:230
193778599131,2cyclictest2217917-21diskstats11:40:140
1937785991312,1cyclictest60350irq/132-enp0s29f1:rx-012:35:060
1937785991312,1cyclictest60350irq/132-enp0s29f1:rx-011:17:220
1937785991312,1cyclictest60350irq/132-enp0s29f1:rx-010:25:160
1937785991311,1cyclictest60350irq/132-enp0s29f1:rx-009:10:110
193779799129,2cyclictest2008837-21needreboot08:15:203
193779799129,2cyclictest0-21swapper/310:30:003
193779799129,1cyclictest0-21swapper/308:35:253
193779799128,3cyclictest2155453-21kworker/u8:0+flush-259:012:00:053
193779799127,4cyclictest2155453-21kworker/u8:0+flush-259:012:20:113
193779799126,4cyclictest252-21systemd-journal08:47:333
193779799126,4cyclictest252-21systemd-journal07:53:323
193779799126,4cyclictest252-21systemd-journal07:53:323
193779799126,4cyclictest2264226-21df_inode12:25:153
193779299129,2cyclictest2077975-21kworker/u8:4+events_unbound09:45:222
193779299129,2cyclictest2051002-21kworker/u8:2+flush-259:009:10:252
193779299129,1cyclictest0-21swapper/211:45:002
193778899128,3cyclictest1957513-21grep07:25:231
193778599125,5cyclictest2150615-21fw_packets10:35:160
193779799119,1cyclictest2073491-21sed09:20:173
193779799117,3cyclictest1879402-21kworker/u8:2+flush-259:008:30:083
193779799110,9cyclictest1948082-21awk07:20:003
193779299117,3cyclictest2155453-21kworker/u8:0+flush-259:011:25:152
193779299117,3cyclictest2155453-21kworker/u8:0+events_unbound11:05:252
193779299116,3cyclictest1973541-21sshd07:42:322
193779299115,4cyclictest2212642-21apt-get11:35:142
193779799106,3cyclictest2083492-21cut09:30:143
19377929997,1cyclictest0-21swapper/211:55:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional