You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-15 - 17:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Jun 15, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1615122710,6sleep30-21swapper/307:05:273
164962233161,21sleep10-21swapper/107:09:481
163762233161,59sleep00-21swapper/007:08:150
162382201166,23sleep20-21swapper/207:06:272
309262590,4sleep339-21rcuc/311:45:163
49722530,4sleep21660899cyclictest10:45:232
246962490,2sleep00-21swapper/008:40:120
246862480,1sleep10-21swapper/110:16:131
16596994111,14cyclictest16551-21munin-run09:50:010
16614993824,9cyclictest0-21swapper/309:30:523
1659699388,14cyclictest15683-21idleruntime-cro08:20:000
1661499351,25cyclictest0-21swapper/312:33:533
1659699359,13cyclictest13731-21munin-run09:40:010
1659699359,13cyclictest13731-21munin-run09:40:010
16614993422,3cyclictest0-21swapper/310:58:253
1660499341,14cyclictest0-21swapper/110:27:151
1661499331,30cyclictest0-21swapper/311:13:493
1661499330,17cyclictest0-21swapper/312:25:493
1660499331,23cyclictest0-21swapper/108:15:511
38112320,4sleep11660499cyclictest11:55:251
16614993224,3cyclictest0-21swapper/312:23:373
1661499321,23cyclictest0-21swapper/311:31:333
1661499321,13cyclictest0-21swapper/311:15:173
1661499320,25cyclictest0-21swapper/312:13:523
16614993118,8cyclictest0-21swapper/310:38:173
1661499311,19cyclictest0-21swapper/310:18:413
1661499310,18cyclictest0-21swapper/308:35:173
1660499318,11cyclictest0-21swapper/108:01:031
1660499310,23cyclictest0-21swapper/109:02:231
1659699319,11cyclictest32211-21awk11:50:010
16614993016,7cyclictest0-21swapper/309:28:043
16614993015,6cyclictest0-21swapper/311:56:123
16614993013,3cyclictest0-21swapper/312:08:483
1661499300,23cyclictest0-21swapper/309:56:533
1660499301,12cyclictest0-21swapper/108:25:021
1660499301,12cyclictest0-21swapper/108:25:021
1660499301,11cyclictest0-21swapper/107:37:031
1660499300,24cyclictest0-21swapper/108:07:271
1659699304,13cyclictest24467-21munin-run08:40:010
1659699302,13cyclictest13475-21munin-run08:15:000
26792290,3sleep11660499cyclictest10:40:211
16614992914,9cyclictest0-21swapper/308:29:133
16614992914,9cyclictest0-21swapper/308:29:133
1661499291,17cyclictest0-21swapper/307:42:083
1661499291,12cyclictest20196-21df12:35:133
1661499291,12cyclictest0-21swapper/310:42:133
1661499290,12cyclictest0-21swapper/311:01:573
1660499291,9cyclictest0-21swapper/112:13:031
16604992913,13cyclictest1037-21systemd-journal10:35:311
1660499291,16cyclictest0-21swapper/107:26:471
1660499291,11cyclictest0-21swapper/107:57:511
1661499281,13cyclictest0-21swapper/309:05:373
1660499281,10cyclictest0-21swapper/109:09:391
1660499281,10cyclictest0-21swapper/107:21:351
1660499280,19cyclictest0-21swapper/109:15:231
1660499280,12cyclictest0-21swapper/111:23:221
1659699281,13cyclictest21676-21awk07:20:010
1659699281,13cyclictest21676-21awk07:20:010
1659699281,11cyclictest18857-21kworker/0:011:23:330
16614992713,3cyclictest0-21swapper/312:02:443
16614992713,3cyclictest0-21swapper/309:14:133
16614992712,3cyclictest0-21swapper/307:58:333
1661499271,20cyclictest0-21swapper/311:28:333
1661499271,18cyclictest0-21swapper/309:15:323
1661499271,11cyclictest0-21swapper/308:04:133
1661499270,13cyclictest0-21swapper/310:20:453
1660499271,16cyclictest0-21swapper/108:44:181
1660499271,11cyclictest0-21swapper/112:37:271
1660499271,11cyclictest0-21swapper/108:54:191
1660499271,10cyclictest0-21swapper/112:24:031
1660499271,10cyclictest0-21swapper/111:00:101
1660499271,10cyclictest0-21swapper/107:50:031
1660499270,11cyclictest0-21swapper/111:37:151
1660499270,10cyclictest0-21swapper/111:52:071
1660499270,10cyclictest0-21swapper/108:31:021
1660499270,10cyclictest0-21swapper/107:46:471
1661499267,13cyclictest0-21swapper/309:24:323
1661499261,19cyclictest0-21swapper/308:42:533
1661499261,17cyclictest0-21swapper/307:16:333
1661499261,17cyclictest0-21swapper/307:16:333
1661499261,13cyclictest0-21swapper/308:07:333
1661499261,10cyclictest5300-21diskmemload11:07:533
1661499261,10cyclictest0-21swapper/307:12:453
1661499260,19cyclictest0-21swapper/310:50:053
1661499260,19cyclictest0-21swapper/310:50:053
1661499260,19cyclictest0-21swapper/310:00:213
1661499260,12cyclictest0-21swapper/310:30:053
1660499261,9cyclictest0-21swapper/112:16:111
1660499261,9cyclictest0-21swapper/111:07:541
1660499261,9cyclictest0-21swapper/110:04:511
1660499261,8cyclictest0-21swapper/108:45:351
1660499261,7cyclictest0-21swapper/107:12:271
1660499261,11cyclictest0-21swapper/110:33:471
1660499261,11cyclictest0-21swapper/109:54:471
1660499261,10cyclictest0-21swapper/107:16:391
1660499261,10cyclictest0-21swapper/107:16:381
1661499258,8cyclictest0-21swapper/308:50:293
1661499258,3cyclictest0-21swapper/308:22:093
1661499257,3cyclictest28367-21df_inode11:40:133
16614992515,3cyclictest0-21swapper/309:36:243
16614992515,3cyclictest0-21swapper/309:36:243
16614992511,8cyclictest0-21swapper/310:09:413
1661499251,15cyclictest0-21swapper/308:15:053
1661499251,14cyclictest0-21swapper/311:20:133
1661499251,12cyclictest0-21swapper/310:11:253
1661499251,11cyclictest0-21swapper/308:46:253
16614992510,7cyclictest0-21swapper/308:58:573
16614992510,3cyclictest0-21swapper/307:53:293
1660499255,3cyclictest0-21swapper/109:58:511
1660499251,9cyclictest0-21swapper/110:54:551
1660499251,9cyclictest0-21swapper/110:54:541
1660499251,9cyclictest0-21swapper/108:13:231
1660499251,8cyclictest0-21swapper/109:25:071
1660499251,8cyclictest0-21swapper/108:56:591
1660499251,6cyclictest0-21swapper/110:49:151
1660499251,11cyclictest0-21swapper/111:43:111
1660499251,11cyclictest0-21swapper/107:32:271
1660499251,10cyclictest0-21swapper/111:48:391
1660499251,10cyclictest0-21swapper/111:13:381
16604992510,3cyclictest0-21swapper/109:47:391
1660499250,9cyclictest0-21swapper/109:42:031
1660499250,9cyclictest0-21swapper/109:14:111
16596992510,14cyclictest0-21swapper/012:35:010
18012240,3sleep011-21rcuc/009:00:190
1661499248,8cyclictest0-21swapper/311:53:123
1661499243,13cyclictest0-21swapper/307:22:563
16614992412,3cyclictest0-21swapper/309:52:213
1661499241,16cyclictest0-21swapper/311:37:053
16614992410,3cyclictest0-21swapper/309:49:213
1661499240,17cyclictest0-21swapper/310:49:233
1661499240,11cyclictest0-21swapper/308:34:293
1660899241,5cyclictest0-21swapper/210:10:022
1660499241,9cyclictest0-21swapper/108:35:261
1660499241,8cyclictest0-21swapper/112:26:151
1660499241,10cyclictest0-21swapper/109:30:151
16604992410,3cyclictest0-21swapper/112:01:071
1660499240,9cyclictest0-21swapper/112:30:431
1659699248,13cyclictest3628-21awk10:45:000
1659699241,11cyclictest16823-21munin-run11:15:020
1661499239,8cyclictest0-21swapper/309:43:243
1661499238,3cyclictest0-21swapper/308:12:373
1661499237,9cyclictest0-21swapper/310:25:453
1661499235,3cyclictest0-21swapper/307:47:293
1660899231,10cyclictest0-21swapper/209:48:452
1660499235,2cyclictest0-21swapper/110:57:301
1660499231,11cyclictest0-21swapper/108:20:341
16604992310,3cyclictest0-21swapper/110:08:511
1660499230,9cyclictest0-21swapper/112:08:311
1659699238,3cyclictest27765-21diskstats10:25:130
1659699231,13cyclictest32512-21kworker/dying08:05:170
1659699231,10cyclictest9245-21grep09:15:250
1659699230,11cyclictest27204-21perf10:25:000
154962230,1sleep20-21swapper/212:25:122
54112220,4sleep011-21rcuc/009:10:020
1661499226,3cyclictest0-21swapper/309:04:293
1661499226,3cyclictest0-21swapper/307:39:163
1661499225,3cyclictest0-21swapper/307:26:283
1661499221,9cyclictest0-21swapper/312:17:363
1660899221,5cyclictest0-21swapper/209:30:162
1660499227,3cyclictest0-21swapper/111:32:511
1660499224,3cyclictest0-21swapper/109:35:031
1660499224,3cyclictest0-21swapper/109:35:031
1660499220,11cyclictest0-21swapper/110:20:191
16596992215,4cyclictest21860-21grep08:30:280
1659699220,10cyclictest0-21swapper/011:26:080
1660899211,9cyclictest0-21swapper/211:23:232
1660899211,4cyclictest4612-21cpuspeed_turbos12:00:122
16608992110,3cyclictest32-21ksoftirqd/211:35:002
1660499216,3cyclictest0-21swapper/109:23:551
1660499211,3cyclictest32128-21timerwakeupswit07:40:241
1660499211,15cyclictest0-21swapper/111:25:251
1659699216,6cyclictest0-21swapper/009:13:210
1659699215,6cyclictest0-21swapper/010:14:340
1659699213,9cyclictest2267050kworker/u8:110:36:100
1659699212,11cyclictest18103-21/usr/sbin/munin08:25:210
1659699212,11cyclictest18103-21/usr/sbin/munin08:25:210
1659699211,14cyclictest0-21swapper/009:40:480
1659699211,12cyclictest30891-21sshd08:53:060
1659699211,11cyclictest6673-21/usr/sbin/munin12:05:270
1659699210,12cyclictest12206-21pmu-power12:15:230
1659699210,10cyclictest0-21swapper/011:55:130
1661499206,3cyclictest0-21swapper/307:31:163
1660899201,6cyclictest0-21swapper/209:17:002
1660899201,5cyclictest0-21swapper/209:22:202
1660899201,10cyclictest12013-21unixbench_singl11:00:272
1660899200,4cyclictest18680-21kworker/u8:211:46:362
1660499205,3cyclictest0-21swapper/111:15:341
16604992013,2cyclictest0-21swapper/110:12:211
1659699204,4cyclictest0-21swapper/011:40:120
1659699201,5cyclictest0-21swapper/012:25:280
1659699201,11cyclictest29952-21df10:30:130
1659699201,10cyclictest6435-21fschecks_count10:50:150
1659699201,10cyclictest6435-21fschecks_count10:50:150
1659699201,10cyclictest21222-21netstat10:05:190
1659699201,10cyclictest17789-21strings07:10:180
1659699201,10cyclictest11952-21tail11:00:270
1659699201,10cyclictest0-21swapper/009:27:220
1659699200,9cyclictest0-21swapper/012:22:400
1659699200,9cyclictest0-21swapper/011:05:460
1659699200,9cyclictest0-21swapper/010:03:000
323432190,2sleep20-21swapper/207:41:202
1660899191,7cyclictest0-21swapper/210:23:402
1660899191,6cyclictest14728-21sed12:20:272
1660899191,6cyclictest0-21swapper/212:08:552
1659699195,12cyclictest11-21rcuc/007:20:230
1659699194,6cyclictest7-21ksoftirqd/008:45:110
1659699191,9cyclictest0-21swapper/007:33:450
1659699191,10cyclictest808-21snmpd11:53:340
1659699191,10cyclictest12310-21ps09:30:180
1659699191,10cyclictest0-21swapper/012:39:180
1659699191,10cyclictest0-21swapper/007:50:220
1659699190,8cyclictest0-21swapper/011:39:260
1659699190,8cyclictest0-21swapper/010:47:200
1659699190,2cyclictest87650irq/129-enp1s0-12:11:170
1660899186,4cyclictest17558-21cstates09:55:122
1660899181,8cyclictest4224-21grep07:50:262
1660899181,8cyclictest0-21swapper/210:44:242
1660899181,7cyclictest0-21swapper/207:15:312
1660899181,7cyclictest0-21swapper/207:15:302
1660899181,6cyclictest0-21swapper/210:54:272
1660899181,6cyclictest0-21swapper/210:54:272
1660899181,6cyclictest0-21swapper/209:36:402
1660899181,6cyclictest0-21swapper/209:36:402
1660899181,5cyclictest0-21swapper/211:41:172
1660899181,4cyclictest0-21swapper/210:58:412
1660899181,15cyclictest29307-21kernelversion07:35:202
1659699188,2cyclictest25347-21smart_sda07:25:260
1659699186,4cyclictest4907-21/usr/sbin/munin07:55:230
1659699183,5cyclictest0-21swapper/009:53:070
1659699182,9cyclictest0-21swapper/010:15:130
1659699181,9cyclictest0-21swapper/012:01:140
1659699181,3cyclictest0-21swapper/008:55:240
1659699181,2cyclictest31662-21cut07:40:180
1659699180,8cyclictest0-21swapper/010:57:170
73142170,3sleep20-21swapper/209:12:492
38952171,2sleep20-21swapper/211:55:522
303072171,2sleep00-21swapper/007:38:250
187852171,2sleep00-21swapper/011:17:130
1660899176,3cyclictest87350irq/126-enp1s0-07:46:232
1660899176,3cyclictest87350irq/126-enp1s0-07:35:012
1660899175,4cyclictest24507-21cat08:40:012
1660899173,4cyclictest31958-21kworker/u8:011:50:292
1660899172,6cyclictest18245-1kworker/u9:210:15:202
1660899171,8cyclictest0-21swapper/212:14:462
1660899171,7cyclictest610-21sensors_temp10:35:242
1660899171,7cyclictest29612-21/usr/sbin/munin10:30:252
1660899171,7cyclictest11788-21if_enp1s012:15:172
1660899171,7cyclictest0-21swapper/210:29:172
1660899171,6cyclictest0-21swapper/211:26:192
1660899171,6cyclictest0-21swapper/211:07:102
1660899171,6cyclictest0-21swapper/209:54:582
1660899171,6cyclictest0-21swapper/209:28:052
1660899171,3cyclictest87350irq/126-enp1s0-12:38:202
16608991712,3cyclictest26883-21/usr/sbin/munin08:45:222
16608991712,3cyclictest18918-21munin-node09:00:132
1660899171,15cyclictest0-21swapper/208:07:442
1660899171,14cyclictest0-21swapper/211:10:052
1660899171,14cyclictest0-21swapper/210:11:572
16608991711,3cyclictest22027-21cpuspeed_turbos07:20:122
16608991711,3cyclictest18098-21ntpq11:15:212
1660899171,13cyclictest14119-21grep08:15:142
16608991710,4cyclictest0-21swapper/208:00:232
1659699177,3cyclictest17573-21cstates09:55:130
1659699176,3cyclictest0-21swapper/009:21:380
1659699173,12cyclictest11-21rcuc/007:45:230
1659699171,6cyclictest0-21swapper/011:31:270
16596991712,3cyclictest16068-21/usr/sbin/munin08:20:130
142472170,2sleep20-21swapper/209:44:002
1660899165,3cyclictest87350irq/126-enp1s0-08:55:162
1660899164,3cyclictest0-21swapper/210:04:102
1660899161,6cyclictest4907-21/usr/sbin/munin07:55:192
1660899161,3cyclictest0-21swapper/207:10:212
16608991612,3cyclictest0-21swapper/209:07:362
16608991612,2cyclictest18103-21/usr/sbin/munin08:25:192
16608991612,2cyclictest18103-21/usr/sbin/munin08:25:192
16608991611,3cyclictest29383-21ls08:50:122
16608991611,3cyclictest21580-21wc08:30:262
16608991611,3cyclictest18461-21interrupts12:30:182
1659699165,4cyclictest7931-21kernelversion08:00:180
1660899154,3cyclictest87350irq/126-enp1s0-11:35:132
1660899151,5cyclictest24692-21/usr/sbin/munin08:40:142
1660899151,5cyclictest16237-21df08:20:132
1660899151,4cyclictest25075-21strings07:25:202
16608991510,3cyclictest11795-21cat08:10:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional