You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 15:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat May 16, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
134792234161,60sleep00-21swapper/007:09:110
133552221193,19sleep10-21swapper/107:07:351
133162215163,39sleep20-21swapper/207:07:042
115892203168,23sleep30-21swapper/307:05:123
1999521370,3sleep31364499cyclictest07:20:213
2463821360,4sleep21363899cyclictest11:11:122
1140521350,3sleep01362899cyclictest10:40:480
136389910899,8cyclictest0-21swapper/211:17:572
209622660,4sleep11363399cyclictest07:25:011
13638996662,3cyclictest0-21swapper/207:55:562
13638995854,3cyclictest0-21swapper/207:36:122
23322530,2sleep02337-21if_enp1s011:35:180
153722510,2sleep20-21swapper/208:25:132
89942460,2sleep00-21swapper/010:35:260
161612450,1sleep30-21swapper/309:40:153
1362899367,13cyclictest31878-21munin-run07:50:010
1362899345,14cyclictest23382-21awk12:25:010
1362899339,11cyclictest22131-21perf09:55:010
1362899336,12cyclictest25583-21idleruntime-cro12:30:010
1362899291,22cyclictest7811-21kworker/0:311:53:220
45542270,5sleep04552-21fschecks_count08:00:150
1362899271,9cyclictest8239-21kworker/0:308:25:260
13638992623,2cyclictest0-21swapper/212:19:562
13633992615,3cyclictest15317-21grep12:05:161
92472250,1sleep30-21swapper/308:10:183
160232250,4sleep121-21rcuc/109:40:131
1362899252,10cyclictest22749-21kworker/0:107:43:540
68652240,3sleep26869-21unixbench-2d10:30:262
263012240,2sleep10-21swapper/110:02:171
248112240,2sleep10-21swapper/112:25:231
1364499245,4cyclictest0-21swapper/310:14:153
1364499242,6cyclictest21976-21chrt09:53:193
1363399241,3cyclictest0-21swapper/112:30:231
1362899245,16cyclictest909-21sssd_nss08:10:010
1362899242,11cyclictest15977-21kworker/0:110:54:380
1362899241,9cyclictest11381-21kworker/0:310:45:260
1362899241,7cyclictest14747-21kworker/0:108:35:460
1363899238,6cyclictest24415-21kworker/u8:110:05:142
13638992316,5cyclictest12757-21wc12:00:112
1363399232,8cyclictest12266-21memory09:30:191
1363399232,10cyclictest0-21swapper/109:10:181
1363399230,16cyclictest2333-21diskmemload10:51:561
1362899232,8cyclictest32767-21kworker/0:010:17:120
1362899232,8cyclictest2683-21kworker/0:210:25:440
1362899232,10cyclictest25512-21kworker/0:312:36:300
1362899232,10cyclictest25512-21kworker/0:312:36:300
1364499222,8cyclictest23703-21munin-run08:45:013
1363399228,4cyclictest23280-21grep08:40:251
1363399221,9cyclictest0-21swapper/109:47:461
13633992217,3cyclictest0-21swapper/110:35:261
1363399221,16cyclictest0-21swapper/108:31:231
1363399221,15cyclictest0-21swapper/110:32:081
1363399221,15cyclictest0-21swapper/108:57:331
1363399220,5cyclictest0-21swapper/109:57:261
1362899229,2cyclictest21440-21kworker/0:308:53:440
1362899222,9cyclictest29249-21kworker/0:211:26:010
1362899222,9cyclictest29249-21kworker/0:211:26:000
1362899222,8cyclictest32767-21kworker/0:010:21:040
1362899222,8cyclictest22475-21kworker/0:311:11:360
1362899221,9cyclictest27026-21kworker/0:111:23:430
1362899221,8cyclictest4532-21kworker/0:107:10:180
1362899221,8cyclictest25462-21kworker/0:012:32:100
1362899221,8cyclictest20984-21kworker/0:112:19:500
1364499219,3cyclictest16374-21/usr/sbin/munin10:55:233
1364499218,3cyclictest87650irq/129-enp1s0-11:55:233
1364499215,11cyclictest808-21snmpd07:12:253
1364499214,9cyclictest29567-21/usr/sbin/munin11:25:203
1364499214,9cyclictest29567-21/usr/sbin/munin11:25:203
1364499212,11cyclictest6961-21/usr/sbin/munin09:20:203
1364499212,10cyclictest17336-21/usr/sbin/munin08:30:173
1364499211,7cyclictest0-21swapper/310:43:323
1364499211,13cyclictest0-21swapper/309:15:193
1364499211,12cyclictest0-21swapper/311:10:123
1364499211,11cyclictest31760-21/usr/sbin/munin11:30:233
1364499211,11cyclictest1620-21smart_sda09:05:253
1364499211,10cyclictest16726-21/usr/sbin/munin07:15:193
1363899219,3cyclictest0-21swapper/207:50:182
1363899211,3cyclictest9368-21kworker/u8:210:41:112
1363399216,4cyclictest0-21swapper/111:12:551
1363399211,7cyclictest0-21swapper/111:41:441
1363399211,6cyclictest26754-21/usr/sbin/munin10:05:141
13633992114,1cyclictest0-21swapper/107:15:261
1363399211,3cyclictest26471-21diskstats08:50:141
1363399211,3cyclictest21121-21users08:35:271
13633992113,3cyclictest0-21swapper/110:55:181
1363399211,17cyclictest14359-21chrt12:02:201
1363399211,15cyclictest0-21swapper/112:22:391
1363399211,15cyclictest0-21swapper/110:46:531
1363399211,14cyclictest0-21swapper/107:50:141
1363399211,14cyclictest0-21swapper/107:39:291
1363399211,12cyclictest14420-21unixbench-2d08:20:241
1363399211,11cyclictest10067-21load09:25:171
1363399210,4cyclictest0-21swapper/112:16:391
1363399210,15cyclictest0-21swapper/109:15:171
1363399210,15cyclictest0-21swapper/107:10:521
1362899219,3cyclictest20779-21/usr/sbin/munin11:05:120
1362899218,3cyclictest8239-21kworker/0:308:24:260
1362899218,2cyclictest30044-21kworker/0:209:20:040
1362899217,3cyclictest25882-21kworker/0:009:00:180
1362899217,3cyclictest25882-21kworker/0:009:00:180
1362899215,7cyclictest20263-21kworker/0:109:59:040
1362899212,8cyclictest24802-21kworker/0:011:18:520
1362899212,7cyclictest10011-21kworker/0:012:01:380
1362899212,11cyclictest17000-21/usr/sbin/munin12:10:220
1362899212,11cyclictest10390-21/usr/sbin/munin11:55:160
1362899212,10cyclictest25882-21kworker/0:009:06:060
1362899211,9cyclictest4532-21kworker/0:107:17:260
1362899211,9cyclictest4321-21kworker/0:009:15:160
1362899211,9cyclictest2683-21kworker/0:210:32:240
1362899210,9cyclictest29249-21kworker/0:211:30:040
1364499208,3cyclictest3181-21/usr/sbin/munin10:25:193
1364499204,2cyclictest41-21ksoftirqd/310:35:193
1364499202,8cyclictest87650irq/129-enp1s0-12:32:573
1364499201,6cyclictest0-21swapper/310:05:573
13644992014,3cyclictest21513-21tr12:20:133
1364499201,13cyclictest0-21swapper/311:08:213
1364499201,11cyclictest0-21swapper/307:50:233
1364499201,10cyclictest31144-21/usr/sbin/munin10:15:173
1364499201,10cyclictest23926-21/usr/sbin/munin08:45:163
1364499201,10cyclictest0-21swapper/308:15:203
1364499200,9cyclictest2587-21cpu09:10:133
1363899205,4cyclictest25615-21ntp_states10:00:232
1363899205,3cyclictest2333-21diskmemload12:10:122
1363899202,4cyclictest12267-21runrttasks11:58:582
1363899201,5cyclictest14787-21grep09:35:252
1363399209,3cyclictest579-21systemd-journal08:20:011
1363399208,3cyclictest23082-21sh07:30:001
1363399201,9cyclictest0-21swapper/111:52:111
1363399201,9cyclictest0-21swapper/111:00:311
13633992016,3cyclictest0-21swapper/108:08:031
13633992013,2cyclictest0-21swapper/110:16:531
13633992013,2cyclictest0-21swapper/109:20:221
1363399201,18cyclictest0-21swapper/111:33:421
1363399201,17cyclictest0-21swapper/111:27:241
1363399201,17cyclictest0-21swapper/111:27:241
1363399201,17cyclictest0-21swapper/108:46:231
1363399201,14cyclictest0-21swapper/110:40:331
1363399201,14cyclictest0-21swapper/108:13:531
1363399201,14cyclictest0-21swapper/107:58:041
1363399201,14cyclictest0-21swapper/107:47:571
1363399201,14cyclictest0-21swapper/107:43:531
1363399201,14cyclictest0-21swapper/107:32:591
1363399201,13cyclictest0-21swapper/111:05:321
1363399201,11cyclictest17091-21cat08:29:561
1363399201,10cyclictest790-21in:imjournal11:20:011
1363399200,15cyclictest0-21swapper/111:58:081
1363399200,14cyclictest0-21swapper/112:12:431
1363399200,14cyclictest0-21swapper/110:11:391
1363399200,14cyclictest0-21swapper/109:02:531
1363399200,14cyclictest0-21swapper/109:02:531
1362899208,2cyclictest30044-21kworker/0:209:26:040
1362899208,2cyclictest22749-21kworker/0:107:33:060
1362899207,2cyclictest22749-21kworker/0:108:14:320
1362899207,2cyclictest21440-21kworker/0:308:46:300
1362899207,2cyclictest15780-21kworker/0:207:26:420
1362899205,3cyclictest17700-21kworker/0:009:49:300
1362899202,9cyclictest22749-21kworker/0:108:15:300
1362899201,8cyclictest30044-21kworker/0:209:10:400
1362899201,8cyclictest30016-21kworker/0:110:11:360
1362899201,11cyclictest28322-21/usr/sbin/munin08:55:200
1362899201,11cyclictest0-21swapper/012:06:080
1362899200,8cyclictest26467-21kworker/0:210:05:220
1362899200,13cyclictest15314-21kworker/0:209:40:300
1364499198,3cyclictest0-21swapper/308:37:053
1364499193,7cyclictest0-21swapper/309:36:233
1364499191,9cyclictest0-21swapper/312:11:313
1364499191,9cyclictest0-21swapper/311:24:553
1364499191,7cyclictest0-21swapper/310:03:213
1364499191,6cyclictest0-21swapper/312:35:193
1364499191,6cyclictest0-21swapper/312:35:193
1364499191,5cyclictest24231-21taskset09:59:053
13644991911,5cyclictest25840-21grep11:15:163
1364499191,10cyclictest0-21swapper/310:33:533
1363899197,2cyclictest87650irq/129-enp1s0-12:31:222
1363899194,8cyclictest4133-21ls10:25:192
1363899194,8cyclictest22344-21/usr/sbin/munin09:55:132
1363899194,8cyclictest21584-21cat08:40:012
1363899192,4cyclictest0-21swapper/210:13:592
13638991914,3cyclictest9685-21grep09:25:142
1363899191,3cyclictest0-21swapper/211:30:152
1363899191,15cyclictest29574-21users12:35:262
1363899191,15cyclictest29574-21users12:35:262
13638991910,3cyclictest4173-1kworker/u9:109:15:542
13638991910,2cyclictest87650irq/129-enp1s0-11:26:132
13638991910,2cyclictest87650irq/129-enp1s0-11:26:132
1363399198,3cyclictest28005-21grep12:35:121
1363399198,3cyclictest28005-21grep12:35:121
1363399196,5cyclictest0-21swapper/111:46:191
1363399195,11cyclictest0-21swapper/110:25:161
1363399191,8cyclictest0-21swapper/109:36:341
1363399191,16cyclictest0-21swapper/109:07:331
1363399191,14cyclictest0-21swapper/110:20:391
13633991911,3cyclictest0-21swapper/108:04:271
1362899196,2cyclictest3300-21kworker/0:111:41:160
1362899192,8cyclictest15977-21kworker/0:111:00:540
1362899191,7cyclictest22749-21kworker/0:107:50:540
1362899191,4cyclictest0-21swapper/009:37:560
13628991913,3cyclictest2360-21fschecks_count07:55:160
1362899191,10cyclictest0-21swapper/011:48:180
1362899191,10cyclictest0-21swapper/010:56:050
200942180,2sleep339-21rcuc/312:15:183
1364499185,3cyclictest87650irq/129-enp1s0-09:30:153
1364499185,3cyclictest87350irq/126-enp1s0-09:05:013
1364499185,3cyclictest87350irq/126-enp1s0-09:05:013
1364499184,5cyclictest755-21rngd10:20:083
1364499181,6cyclictest0-21swapper/312:25:553
1364499181,6cyclictest0-21swapper/312:05:283
1364499181,6cyclictest0-21swapper/311:54:573
1364499181,6cyclictest0-21swapper/311:42:543
1364499181,6cyclictest0-21swapper/310:51:163
1364499181,6cyclictest0-21swapper/309:47:393
1364499181,6cyclictest0-21swapper/307:40:243
1364499181,4cyclictest0-21swapper/311:46:413
1364499181,10cyclictest0-21swapper/312:01:093
1364499181,10cyclictest0-21swapper/307:25:183
1363899188,5cyclictest87650irq/129-enp1s0-09:43:312
1363899188,2cyclictest87650irq/129-enp1s0-10:24:542
1363899186,7cyclictest6491-21kworker/u8:209:22:212
1363899186,6cyclictest87650irq/129-enp1s0-10:45:082
1363899186,4cyclictest19379-21cut08:35:022
1363899185,4cyclictest1592-21/usr/sbin/munin11:35:212
1363899185,3cyclictest21929-21fschecks_time12:20:162
1363899185,3cyclictest18056-21cpuspeed_turbos09:45:132
1363899184,4cyclictest8553-21/usr/sbin/munin08:10:132
1363899184,4cyclictest15280-21pluginstate10:50:222
1363899183,8cyclictest15215-21timerandwakeup07:10:252
1363899182,9cyclictest665-21munin-run10:20:012
1363899181,9cyclictest9785-21users11:50:272
1363899181,9cyclictest8664-21open_files10:35:232
1363899181,5cyclictest0-21swapper/212:29:532
1363899181,4cyclictest0-21swapper/207:25:242
1363899181,3cyclictest0-21swapper/209:30:192
13638991812,3cyclictest11414-21hddtemp_smartct08:15:172
1363899181,15cyclictest0-21swapper/207:40:322
13633991814,3cyclictest0-21swapper/111:37:241
1363399180,14cyclictest0-21swapper/109:50:411
1362899186,3cyclictest15780-21kworker/0:207:20:260
1362899186,3cyclictest0-21swapper/010:02:500
1362899186,2cyclictest16990-21kworker/0:008:31:140
1362899181,10cyclictest21728-21/usr/sbin/munin08:40:210
1364499175,3cyclictest87350irq/126-enp1s0-08:55:153
1364499173,7cyclictest0-21swapper/308:09:223
1364499173,5cyclictest0-21swapper/310:49:543
1364499171,9cyclictest0-21swapper/309:25:173
1364499171,6cyclictest0-21swapper/311:35:563
1364499171,5cyclictest4239-21cut08:00:133
1364499171,3cyclictest0-21swapper/308:25:053
1364499171,14cyclictest0-21swapper/307:31:513
1364499171,13cyclictest18703-21cpuspeed_turbos11:00:123
1363899177,3cyclictest87350irq/126-enp1s0-11:45:192
1363899176,3cyclictest13347-21fschecks_count08:20:142
1363899175,3cyclictest23499-21cstates07:30:142
1363899175,3cyclictest11686-21kworker/u8:011:40:212
1363899174,4cyclictest30659-21expr09:00:132
1363899174,4cyclictest30659-21expr09:00:132
1363899174,4cyclictest19124-21perl11:00:152
1363899173,7cyclictest29538-21chrt08:55:232
1363899171,4cyclictest0-21swapper/209:50:262
1363899171,3cyclictest0-21swapper/209:13:302
1363899171,2cyclictest0-21swapper/208:40:562
1363899171,15cyclictest0-21swapper/208:52:432
1363899171,15cyclictest0-21swapper/208:03:542
13638991711,4cyclictest16485-21tr10:55:132
13638991711,3cyclictest18923-21/usr/sbin/munin07:20:222
13638991710,4cyclictest27810-21tune2fs11:20:152
1363899170,4cyclictest0-21swapper/209:05:252
1363899170,2cyclictest87350irq/126-enp1s0-12:07:032
1363399171,15cyclictest0-21swapper/111:21:321
1362899171,4cyclictest25515-21/usr/sbin/munin07:35:250
1362899171,4cyclictest11353-21/usr/sbin/munin09:30:180
1364499161,6cyclictest31166-21smart_sda07:45:253
13644991611,3cyclictest27377-21sendmail_mailtr08:50:233
1364499161,12cyclictest25506-21sshd07:35:073
13644991610,3cyclictest1941-21/usr/sbin/munin07:55:183
1364499160,14cyclictest0-21swapper/308:23:103
1363899165,3cyclictest87550irq/128-enp1s0-08:05:172
1363899161,8cyclictest20779-21/usr/sbin/munin11:05:232
1363899161,8cyclictest0-21swapper/207:45:152
1363899161,3cyclictest0-21swapper/207:15:212
1363899161,2cyclictest0-21swapper/208:45:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional