You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-22 - 02:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Mon Jun 22, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28432239164,24sleep20-21swapper/219:06:592
30272201167,22sleep10-21swapper/119:09:241
28142201165,24sleep30-21swapper/319:06:363
27672200164,24sleep00-21swapper/019:06:000
1072021730,5sleep1316799cyclictest19:25:121
454421700,1sleep20-21swapper/219:10:252
308702610,1sleep30-21swapper/323:50:183
3173995754,2cyclictest0-21swapper/219:53:322
308422530,1sleep130846-21fschecks_count20:10:141
208152530,1sleep20-21swapper/221:00:162
316799473,31cyclictest7542-40dnf22:55:591
316799404,7cyclictest23542-40dnf19:54:251
316799391,11cyclictest32516-21unixbench-2d21:25:251
316799373,22cyclictest23-21ksoftirqd/123:12:591
3167993712,21cyclictest10604-21grep20:35:241
316799371,11cyclictest32011-21diskstats22:40:151
316799368,2cyclictest579-21systemd-journal19:45:011
3167993613,20cyclictest17537-21/usr/sbin/munin00:35:251
316799361,11cyclictest25081-21users23:35:271
316799358,4cyclictest0-21swapper/100:01:071
316799351,8cyclictest22294-21/usr/sbin/munin23:30:201
316799351,30cyclictest1-21systemd20:00:021
316799349,22cyclictest10715-21fschecks_count23:05:151
316799349,22cyclictest10715-21fschecks_count23:05:151
316799341,30cyclictest8230-21perf19:20:001
3167993412,19cyclictest24459-21/usr/sbin/munin21:10:231
316799341,11cyclictest9319-21processes23:00:241
316799335,3cyclictest32216-21kworker/u8:119:20:271
316799331,7cyclictest5748-21cut22:55:011
316799331,29cyclictest17098-21irqstats22:05:171
316799331,11cyclictest3714-21dnf19:10:151
3167993311,18cyclictest15379-21sh19:35:141
3167993310,21cyclictest25143-21kworker/u8:123:40:271
316799328,3cyclictest87550irq/128-enp1s0-22:22:281
3167993213,17cyclictest31828-21sshd23:52:161
316799321,27cyclictest5870-21gltestperf21:40:161
3167993212,17cyclictest7483-21/usr/sbin/munin21:45:111
316799321,11cyclictest11556-21grep00:20:151
3167993211,18cyclictest20054-21/usr/sbin/munin21:00:211
3167993210,3cyclictest28246-21forks23:45:151
3167993210,3cyclictest27956-21hwlatdetect22:30:181
3167993210,3cyclictest19826-21munin-run21:00:011
316799320,30cyclictest0-21swapper/120:45:011
316799318,3cyclictest2081-21seq00:00:011
3167993112,16cyclictest14704-21/usr/sbin/munin23:15:131
3167993110,4cyclictest13250-21tr00:25:131
3167993110,3cyclictest16902-21/usr/sbin/munin23:20:151
3167993110,2cyclictest1-21systemd21:50:201
316799310,7cyclictest15826-21fschecks_time00:30:161
316799309,3cyclictest7267-21cat00:10:171
316799309,3cyclictest0-21swapper/122:35:201
316799305,3cyclictest4868-21cut00:05:171
316799302,6cyclictest27914-21smart_sda21:15:241
316799301,26cyclictest29770-21memory21:20:211
3167993011,3cyclictest4147-21ntpq21:35:221
316799301,11cyclictest0-21swapper/120:45:241
316799301,11cyclictest0-21swapper/120:45:241
316799301,10cyclictest0-21swapper/122:13:451
316799300,11cyclictest0-21swapper/120:00:071
316799299,3cyclictest827-21runrttasks22:02:231
316799298,3cyclictest30372-21cat20:10:031
316799297,4cyclictest1353-21pluginstate20:15:211
316799296,4cyclictest20020-21/usr/sbin/munin19:45:161
316799296,3cyclictest0-21swapper/120:50:261
316799295,21cyclictest2111-21gltestperf22:45:161
316799291,2cyclictest2851-21dump-pmu-power21:35:001
3167992910,3cyclictest19515-21grep23:25:141
316399291,13cyclictest20470-21awk22:15:010
317399281,12cyclictest21965-21hddtemp_smartct23:30:162
316799289,3cyclictest827-21runrttasks19:32:181
316799289,3cyclictest827-21runrttasks19:32:181
316799289,3cyclictest12584-21basename21:55:151
316799289,3cyclictest12584-21basename21:55:151
316799283,2cyclictest9968-21/usr/sbin/munin00:15:211
316799283,2cyclictest9968-21/usr/sbin/munin00:15:211
316799281,24cyclictest25318-21expr22:25:131
3167992811,2cyclictest827-21runrttasks22:17:171
3167992810,3cyclictest23816-21head21:05:281
312082280,3sleep07-21ksoftirqd/023:50:220
316799271,3cyclictest5022-21diskstats20:25:161
3167992711,3cyclictest2712-21expr20:20:131
316399271,13cyclictest12286-21munin-run23:10:010
316399271,13cyclictest12286-21munin-run23:10:010
3173992623,2cyclictest0-21swapper/222:32:182
316399262,14cyclictest87350irq/126-enp1s0-23:40:230
3178992519,3cyclictest23437-21proc_pri21:05:253
316799256,3cyclictest7371-21cut20:30:161
3163992512,4cyclictest45462cut19:10:240
316399251,12cyclictest14478-21munin-run23:15:010
248012250,4sleep024798-21df_inode21:10:130
317399241,8cyclictest0-21swapper/222:06:362
317399241,11cyclictest0-21swapper/219:24:042
316399242,8cyclictest14648-21kworker/0:219:35:340
100822241,19sleep20-21swapper/221:50:172
276242230,4sleep339-21rcuc/322:30:143
3178992212,5cyclictest13385-21awk21:55:253
3178992212,5cyclictest13385-21awk21:55:253
317899220,5cyclictest0-21swapper/300:29:173
317399221,3cyclictest0-21swapper/223:39:002
316399221,4cyclictest0-21swapper/022:00:260
210272220,1sleep011-21rcuc/022:15:150
317899216,9cyclictest0-21swapper/323:29:483
317899214,7cyclictest16369-21kworker/u8:223:40:163
317899211,6cyclictest0-21swapper/323:02:143
317899211,13cyclictest0-21swapper/322:05:443
317899211,11cyclictest4674-21/usr/sbin/munin20:25:213
317899211,11cyclictest24466-21acpi21:10:113
317899211,11cyclictest0-21swapper/300:30:253
317399214,8cyclictest87550irq/128-enp1s0-22:35:222
316399211,6cyclictest4622-21memory22:50:190
316399211,11cyclictest31047-21/usr/sbin/munin21:25:220
166612211,3sleep00-21swapper/023:20:010
317899208,4cyclictest0-21swapper/321:20:073
317899204,5cyclictest0-21swapper/321:17:413
317899202,7cyclictest0-21swapper/321:36:513
317899202,15cyclictest1489-21vmstat23:55:263
317899201,9cyclictest0-21swapper/322:53:543
3178992014,3cyclictest21403-21idleruntime-cro19:50:013
317899201,11cyclictest0-21swapper/323:31:393
317899201,11cyclictest0-21swapper/322:57:433
317899201,11cyclictest0-21swapper/319:55:263
317899201,11cyclictest0-21swapper/319:40:253
317899201,10cyclictest4955-21users19:10:273
317899201,10cyclictest30695-21gzip22:35:233
317899201,10cyclictest26839-21irqstats20:00:173
317899201,10cyclictest24047-21perl23:35:173
317899201,10cyclictest22954-21cat22:20:123
317899201,10cyclictest14734-21grep22:00:163
317399207,4cyclictest2858-21awk21:35:002
317399204,5cyclictest3541-21timerwakeupswit00:00:292
317399203,4cyclictest28228-21df_inode23:45:142
317399202,15cyclictest9725-21gltestperf20:35:152
3173992014,3cyclictest9643-21sed00:15:172
3173992014,3cyclictest9643-21sed00:15:172
317399201,13cyclictest0-21swapper/222:40:162
317399201,11cyclictest0-21swapper/223:20:442
317399201,11cyclictest0-21swapper/200:29:002
316399204,8cyclictest87550irq/128-enp1s0-23:45:180
316399204,8cyclictest87550irq/128-enp1s0-00:20:130
316399203,9cyclictest87550irq/128-enp1s0-21:05:220
316399201,5cyclictest0-21swapper/022:25:000
316399201,4cyclictest15342-21/usr/sbin/munin00:30:120
3163992013,4cyclictest9534-21grep20:35:130
3163992012,5cyclictest19778-21hwlatdetect23:25:150
316399201,11cyclictest6550-21/usr/sbin/munin00:10:200
316399201,11cyclictest6256-21/usr/sbin/munin19:15:180
316399201,11cyclictest26655-21/usr/sbin/munin21:15:270
316399201,11cyclictest12846-21/usr/sbin/munin19:30:120
316399201,11cyclictest12846-21/usr/sbin/munin19:30:120
316399201,10cyclictest0-21swapper/020:45:170
316399201,10cyclictest0-21swapper/020:45:170
148412200,3sleep3401ktimersoftd/319:35:013
148412200,3sleep3401ktimersoftd/319:35:013
317899195,4cyclictest0-21swapper/321:30:053
317899194,8cyclictest32216-21kworker/u8:121:25:173
317899193,3cyclictest0-21swapper/323:05:553
317899193,3cyclictest0-21swapper/323:05:553
317899192,9cyclictest30699-21expr20:10:123
317899192,7cyclictest827-21runrttasks19:54:243
317899191,6cyclictest25143-21kworker/u8:100:20:063
317899191,6cyclictest0-21swapper/322:49:113
317899191,3cyclictest0-21swapper/321:45:233
3178991911,5cyclictest0-21swapper/300:00:303
317899191,10cyclictest5310-21cpu21:40:123
317899191,10cyclictest26301-21gzip22:25:223
317899191,10cyclictest14995-21expr23:15:123
317899191,10cyclictest0-21swapper/323:10:173
317899191,10cyclictest0-21swapper/300:39:333
317899190,10cyclictest21367-21hwlatdetect22:15:173
317399197,3cyclictest87350irq/126-enp1s0-19:45:012
317399191,7cyclictest0-21swapper/223:00:192
317399191,6cyclictest19379-21sed22:10:182
317399191,4cyclictest0-21swapper/223:14:292
317399191,10cyclictest0-21swapper/221:38:042
316399199,3cyclictest9445-21cut19:20:200
316399197,11cyclictest0-21swapper/021:35:000
316399194,3cyclictest87550irq/128-enp1s0-20:55:170
316399193,4cyclictest6356-21ntpq21:40:210
316399192,9cyclictest2142-21cpu00:00:110
316399191,6cyclictest0-21swapper/021:20:170
316399191,15cyclictest27586-21grep20:00:250
3163991911,5cyclictest1-21systemd20:20:200
316399191,10cyclictest3953-21cat21:35:190
316399191,10cyclictest31664-21/usr/sbin/munin22:40:120
316399191,10cyclictest30430-21/usr/sbin/munin20:10:200
316399191,10cyclictest10649-21/usr/sbin/munin19:25:140
201132191,3sleep30-21swapper/322:10:453
102482190,3sleep339-21rcuc/320:35:213
83762180,2sleep20-21swapper/200:13:202
317899181,9cyclictest8814-21cpuspeed_turbos00:15:113
317899181,9cyclictest8814-21cpuspeed_turbos00:15:113
317899181,9cyclictest5086-21cat00:05:193
317899181,7cyclictest20094-21cpu21:00:113
317899181,6cyclictest0-21swapper/322:40:253
317899181,6cyclictest0-21swapper/320:32:053
3178991811,4cyclictest15164-21grep19:35:133
3178991811,4cyclictest0-21swapper/319:25:183
317899181,11cyclictest0-21swapper/323:22:283
317899181,10cyclictest0-21swapper/323:45:543
317899181,10cyclictest0-21swapper/321:54:353
317899181,10cyclictest0-21swapper/300:11:243
317399185,3cyclictest27801-21awk21:15:232
317399184,4cyclictest0-21swapper/222:29:432
317399184,3cyclictest15550-21unixbench-2d22:00:252
317399182,3cyclictest8564-1kworker/u9:121:48:342
317399181,8cyclictest1903-21awk00:00:012
317399181,7cyclictest6642-21awk20:30:002
317399181,7cyclictest17183-21cat19:40:012
317399181,7cyclictest1415-21ps20:15:222
317399181,7cyclictest0-21swapper/221:20:332
317399181,6cyclictest7251-21dump-pmu-power21:45:012
317399181,6cyclictest0-21swapper/200:37:392
317399181,6cyclictest0-21swapper/200:22:012
317399181,4cyclictest16870-21grep00:30:262
317399181,11cyclictest0-21swapper/222:55:242
316399185,2cyclictest87550irq/128-enp1s0-00:25:190
316399184,3cyclictest0-21swapper/023:20:140
316399182,13cyclictest7-21ksoftirqd/021:45:200
316399181,7cyclictest16649-21needreboot20:50:190
316399181,6cyclictest0-21swapper/023:56:380
316399181,5cyclictest0-21swapper/021:51:040
316399181,3cyclictest17048-21ethtool22:05:160
3163991812,3cyclictest6374-21fschecks_time22:55:150
3163991812,3cyclictest3474-21idleruntime-cro22:50:000
3163991812,3cyclictest23493-21/usr/sbin/munin23:35:180
291912180,2sleep00-21swapper/022:34:300
224492181,2sleep20-21swapper/222:17:342
55122170,4sleep232-21ksoftirqd/200:05:252
317899171,7cyclictest29061-21kernelversion20:05:193
3178991711,3cyclictest480-21cat20:15:123
3178991711,3cyclictest17854-21/usr/sbin/munin20:55:253
3178991711,3cyclictest13457-21/usr/sbin/munin20:45:183
3178991711,3cyclictest13457-21/usr/sbin/munin20:45:183
317899171,10cyclictest0-21swapper/319:18:253
317399174,4cyclictest579-21systemd-journal19:20:002
317399172,7cyclictest24459-21/usr/sbin/munin21:10:132
317399172,3cyclictest129792tr19:30:112
317399172,3cyclictest129792tr19:30:112
317399171,6cyclictest28007-21sh20:05:012
317399171,6cyclictest0-21swapper/223:40:242
317399171,5cyclictest567-21perf21:30:002
317399171,5cyclictest0-21swapper/222:23:412
3173991712,3cyclictest5258-21grep22:50:272
3173991712,3cyclictest25336-21awk19:55:262
3173991712,3cyclictest2128-21grep22:45:152
3173991712,3cyclictest11258-21/usr/sbin/munin20:40:222
317399171,14cyclictest0-21swapper/219:27:042
3173991711,3cyclictest2812-21diskstats20:20:132
316399178,2cyclictest0-21swapper/021:00:180
316399171,7cyclictest23834-21/usr/sbin/munin19:55:170
316399171,7cyclictest22550-21memory19:50:190
316399171,6cyclictest0-21swapper/022:26:210
316399171,6cyclictest0-21swapper/000:07:290
316399171,4cyclictest11881-21/usr/sbin/munin21:55:190
316399171,4cyclictest11881-21/usr/sbin/munin21:55:190
316399171,3cyclictest0-21swapper/023:00:030
3163991711,3cyclictest21439-21cpuspeed_turbos23:30:130
316399171,13cyclictest17537-21/usr/sbin/munin00:35:170
296762170,2sleep00-21swapper/022:35:130
164852171,2sleep20-21swapper/223:17:462
317899163,12cyclictest41-21ksoftirqd/320:20:213
317899161,12cyclictest8451-21/usr/sbin/munin19:20:133
3178991610,3cyclictest16367-21basename20:50:163
317399163,4cyclictest28237-21/usr/sbin/munin20:05:192
317399162,5cyclictest12222-21df_inode21:55:122
317399162,5cyclictest12222-21df_inode21:55:122
317399161,7cyclictest10980-21gltestperf23:05:172
317399161,7cyclictest10980-21gltestperf23:05:172
317399161,6cyclictest0-21swapper/223:26:592
317399161,5cyclictest6870-21/usr/sbin/munin20:30:252
317399161,2cyclictest0-21swapper/221:09:072
317399161,2cyclictest0-21swapper/219:45:072
317399161,14cyclictest0-21swapper/220:50:472
317399161,14cyclictest0-21swapper/220:14:332
3173991611,3cyclictest30893-21irqstats23:50:172
3173991611,3cyclictest17854-21/usr/sbin/munin20:55:172
316399166,3cyclictest87550irq/128-enp1s0-19:40:150
316399165,3cyclictest773-21NetworkManager20:17:430
316399165,3cyclictest6870-21/usr/sbin/munin20:30:200
316399161,6cyclictest8750-21/usr/sbin/munin00:15:200
316399161,6cyclictest8750-21/usr/sbin/munin00:15:200
316399161,14cyclictest0-21swapper/020:40:300
3163991611,3cyclictest20725-21sort19:45:250
316399161,12cyclictest28237-21/usr/sbin/munin20:05:260
316399161,10cyclictest0-21swapper/020:25:230
317899152,6cyclictest0-21swapper/320:40:093
317399141,4cyclictest0-21swapper/220:45:142
317399141,4cyclictest0-21swapper/220:45:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional