You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 21:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Mon Dec 08, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1075495212053,62sleep30-21swapper/307:05:283
107571328553,9sleep10-21swapper/107:08:141
107576228352,10sleep20-21swapper/207:08:512
107560827859,13sleep00-21swapper/007:06:540
1075973993127,2cyclictest0-21swapper/310:50:163
1075973993024,3cyclictest0-21swapper/312:00:153
107597399261,9cyclictest1120909-21taskset07:50:043
148617622513,9sleep00-21swapper/012:15:270
14263012250,3sleep00-21swapper/011:36:210
12679912251,8sleep11268089-21ntpq09:50:201
107597399242,16cyclictest1268151-21taskset09:50:203
1075973992421,1cyclictest0-21swapper/311:35:013
1075973992320,1cyclictest0-21swapper/311:59:593
1075973992314,5cyclictest13672382sleep310:57:083
107597399231,19cyclictest1425378-21taskset11:35:233
107597399229,9cyclictest13749222sleep311:02:453
107597399222,15cyclictest1451034-21taskset11:55:003
107597399222,13cyclictest1225708-21taskset09:22:193
1075973992219,2cyclictest0-21swapper/309:40:013
1075973992219,1cyclictest0-21swapper/309:25:183
1075973992218,2cyclictest0-21swapper/310:40:123
1075973992218,2cyclictest0-21swapper/309:49:553
107597399221,3cyclictest0-21swapper/307:33:423
107597399221,17cyclictest1284013-21taskset10:00:583
107597399212,13cyclictest1351584-21taskset10:45:333
1075973992117,2cyclictest0-21swapper/309:40:163
107597399210,13cyclictest1390041-21taskset11:13:473
14418252201,13sleep11205265-21diskmemload11:46:411
107597399207,9cyclictest1167999-21chrt08:30:283
107597399202,13cyclictest1481711-21taskset12:15:123
107597399202,13cyclictest1205272-21turbostat09:10:003
1075973992016,2cyclictest0-21swapper/309:30:153
1075973992016,2cyclictest0-21swapper/309:10:183
1075973992016,2cyclictest0-21swapper/309:00:243
107597399201,15cyclictest1489659-21chrt12:20:143
1075973992011,5cyclictest14432152sleep311:49:193
107597399201,14cyclictest1504220-21chrt12:28:353
107597399200,18cyclictest0-21swapper/310:09:233
12756062191,12sleep1802-21snmpd09:55:211
107597399192,11cyclictest1395069-21taskset11:15:193
107597399192,11cyclictest1308811-21taskset10:20:063
1075973991916,1cyclictest0-21swapper/312:10:173
1075973991916,1cyclictest0-21swapper/310:28:033
1075973991916,1cyclictest0-21swapper/309:55:363
1075973991915,2cyclictest0-21swapper/311:20:263
1075973991915,2cyclictest0-21swapper/307:45:143
107597399191,14cyclictest0-21swapper/311:09:463
107597399191,11cyclictest1514660-21taskset12:35:163
14718732181,4sleep11205265-21diskmemload12:07:261
14050312181,4sleep01205265-21diskmemload11:23:550
12777142181,2sleep20-21swapper/209:58:142
12181042181,3sleep20-21swapper/209:16:332
107597399182,12cyclictest1338305-21taskset10:38:593
107597399182,11cyclictest1076172-21taskset07:10:123
1075973991815,1cyclictest0-21swapper/307:56:543
1075973991814,2cyclictest0-21swapper/311:40:183
1075973991814,2cyclictest0-21swapper/311:28:253
1075973991814,1cyclictest0-21swapper/308:40:013
107597399181,14cyclictest0-21swapper/312:31:293
107597399181,14cyclictest0-21swapper/310:30:473
107597399181,13cyclictest1218112-21taskset09:16:333
107597399181,13cyclictest0-21swapper/307:35:193
107597399181,12cyclictest1302295-21chrt10:15:183
107597399181,12cyclictest1178997-21sleep08:45:003
15031072171,2sleep20-21swapper/212:26:502
13809562171,10sleep00-21swapper/011:05:380
13294492171,10sleep10-21swapper/110:31:491
13219862171,3sleep00-21swapper/010:27:030
13152022171,4sleep11205265-21diskmemload10:23:271
13072382171,2sleep00-21swapper/010:16:590
12634772171,3sleep01205265-21diskmemload09:49:510
12554882171,3sleep20-21swapper/209:44:102
12540922175,7sleep01205265-21diskmemload09:41:170
12325082171,4sleep01205265-21diskmemload09:26:170
1075973991714,2cyclictest0-21swapper/308:59:593
1075973991713,2cyclictest0-21swapper/308:10:123
107597399171,13cyclictest0-21swapper/312:05:113
107597399171,13cyclictest0-21swapper/310:11:313
107597399171,10cyclictest1114066-21taskset07:40:183
15118732161,4sleep21205265-21diskmemload12:32:532
14958312161,3sleep01205265-21diskmemload12:22:440
14896372161,3sleep11489646-21ssh12:20:131
14887842161,2sleep10-21swapper/112:19:161
14785572161,3sleep10-21swapper/112:10:391
14582792161,3sleep20-21swapper/212:00:012
14566592161,3sleep21205265-21diskmemload11:56:202
14183192161,4sleep21418328-21ssh11:30:472
13528652161,3sleep10-21swapper/110:48:001
12774272161,2sleep00-21swapper/009:57:430
12401862161,6sleep21240196-21ssh09:31:562
11141132161,4sleep01114161-21ntpq07:40:200
107597399161,9cyclictest1093039-21taskset07:20:223
1075973991613,1cyclictest0-21swapper/308:49:593
1075973991613,1cyclictest0-21swapper/307:15:023
107597399161,12cyclictest0-21swapper/308:00:173
107597399161,12cyclictest0-21swapper/308:00:173
107597399161,11cyclictest1163218-21turbostat08:30:013
107597399160,10cyclictest1136790-21taskset08:05:073
15202442151,2sleep00-21swapper/012:38:280
15040142150,9sleep01205265-21diskmemload12:28:170
14190472151,9sleep01205265-21diskmemload11:32:090
13445782151,3sleep21205265-21diskmemload10:41:362
13442262150,1sleep00-21swapper/010:40:580
13316642151,9sleep01331671-21/usr/sbin/munin10:35:120
12185352150,9sleep01205265-21diskmemload09:17:270
11847722151,9sleep21184946-21expr08:50:162
11359622151,9sleep21136162-21users08:00:262
11359622151,9sleep21136162-21users08:00:252
107597399151,12cyclictest0-21swapper/308:50:143
1075973991511,2cyclictest0-21swapper/308:15:143
1075973991510,3cyclictest0-21swapper/308:20:133
15036802141,6sleep10-21swapper/112:27:461
14797232141,7sleep20-21swapper/212:12:322
14721102141,8sleep01472120-21ssh12:07:570
14648292141,8sleep11205265-21diskmemload12:03:051
14580902140,1sleep10-21swapper/111:59:361
14499022140,8sleep01205265-21diskmemload11:53:020
14482212141,7sleep21448293-21ls11:50:232
14281392141,7sleep21428163-21munin-run11:40:002
14055502141,6sleep11205265-21diskmemload11:24:521
13852612141,8sleep11387043-21grep11:10:171
13382922141,7sleep20-21swapper/210:38:592
13056692141,8sleep21305815-21ntpq10:15:222
12645312141,7sleep21265733-21tr09:50:162
12185482141,1sleep10-21swapper/109:17:281
12125512141,8sleep21212556-21ssh09:14:532
11726782141,9sleep01172723-21modprobe08:35:190
11692972141,8sleep11170826-21tr08:35:161
107597399141,8cyclictest1098905-21taskset07:27:523
15184862131,8sleep11518500-21ssh12:35:441
15105022131,8sleep01510513-21ssh12:30:420
14946572131,8sleep21494669-21ssh12:20:462
14418972131,6sleep01205265-21diskmemload11:46:520
14330592131,7sleep21433313-21users11:40:292
14121562131,6sleep00-21swapper/011:28:200
13673632131,5sleep20-21swapper/210:57:232
13375762131,7sleep11234363-21kworker/u16:0+xprtiod10:37:301
12612972131,8sleep21205265-21diskmemload09:45:482
12469952131,6sleep11205265-21diskmemload09:36:291
12261682131,6sleep21205265-21diskmemload09:23:102
12059302131,5sleep11206150-21smartctl09:10:141
15123782120,7sleep11205265-21diskmemload12:33:401
14731432120,1sleep20-21swapper/212:09:512
14181672121,5sleep11205265-21diskmemload11:30:301
14128552121,5sleep10-21swapper/111:29:411
13968522121,5sleep21205265-21diskmemload11:17:472
13902292120,8sleep01390243-21ssh11:14:170
13696842120,8sleep21371122-21tr11:00:202
13554622121,7sleep21356334-21kthreadcore10:50:182
12695442120,7sleep01269549-21ssh09:51:510
12624522120,7sleep11262456-21sh09:47:591
12551862120,7sleep11205265-21diskmemload09:43:381
12476332121,7sleep21247638-21ssh09:38:062
12285322121,7sleep10-21swapper/109:25:181
12046712121,5sleep21204888-21users09:05:272
11464362121,5sleep11146493-21ntpq08:10:181
10993592121,3sleep21099469-21cpuspeed_turbos07:30:122
10766662121,7sleep21076912-21grep07:10:162
15190462111,4sleep21205265-21diskmemload12:36:362
14816752111,4sleep21205265-21diskmemload12:15:122
14800262111,4sleep01205265-21diskmemload12:12:580
14640882111,4sleep01205265-21diskmemload12:01:440
14497812111,4sleep11205265-21diskmemload11:52:471
14340202111,3sleep00-21swapper/011:41:510
14334772111,4sleep11205265-21diskmemload11:40:481
13970672111,3sleep10-21swapper/111:18:141
13958472111,4sleep01205265-21diskmemload11:15:370
13905762111,4sleep21205265-21diskmemload11:14:592
13811712111,2sleep20-21swapper/211:06:092
13678602111,3sleep10-21swapper/110:58:221
13675022111,4sleep01205265-21diskmemload10:57:400
13536772110,1sleep20-21swapper/210:49:382
13530362113,3sleep01205265-21diskmemload10:48:220
13454662111,3sleep10-21swapper/110:43:151
13307312111,4sleep21205265-21diskmemload10:34:022
13242142111,5sleep01324358-21tail10:30:150
13219972111,3sleep11205265-21diskmemload10:27:041
13214052111,4sleep21205265-21diskmemload10:26:002
12996162111,4sleep11205265-21diskmemload10:12:041
12918342111,4sleep11205265-21diskmemload10:06:351
12833492111,3sleep10-21swapper/110:00:261
12479952111,4sleep01205265-21diskmemload09:39:080
12400152111,4sleep11205265-21diskmemload09:31:351
12394162111,4sleep01205265-21diskmemload09:30:300
12122942111,4sleep01205265-21diskmemload09:14:220
11902812112,2sleep20-21swapper/208:55:172
11797982111,5sleep11180567-21cut08:45:181
11097712111,4sleep01109842-21munin-run07:40:000
11043292110,1sleep00-21swapper/007:34:470
10959952111,5sleep11097827-21cut07:25:171
14434912101,2sleep20-21swapper/211:49:502
14263512101,3sleep10-21swapper/111:36:271
14128252101,3sleep20-21swapper/211:29:382
13829082100,3sleep10-21swapper/111:09:521
13748202100,6sleep01374831-21ssh11:02:350
13741252100,3sleep10-21swapper/111:01:031
13594902101,2sleep00-21swapper/010:51:240
13592952101,3sleep10-21swapper/110:51:011
13146602101,3sleep20-21swapper/210:22:242
13141172101,4sleep01205265-21diskmemload10:21:210
12995852101,3sleep21205265-21diskmemload10:12:012
12918232101,2sleep00-21swapper/010:06:330
12864202101,3sleep20-21swapper/210:05:082
12831042101,4sleep21283169-21ntp_states10:00:232
12829892101,4sleep01283042-21ntpq10:00:210
12326702101,4sleep21205265-21diskmemload09:26:352
12263942101,3sleep01205265-21diskmemload09:23:490
12244302101,4sleep11224482-21ntpq09:20:201
11955022101,4sleep21195573-21irqstats09:00:182
11677512100,6sleep21167996-21unixbench_multi08:30:272
11593362101,4sleep21161063-21tr08:25:152
11531482100,1sleep20-21swapper/208:20:132
11267722101,5sleep01128267-21kthreadcore07:55:190
11203812101,4sleep21120434-21apt-config07:49:592
10879002101,4sleep21088117-21users07:15:242
1451322291,2sleep00-21swapper/011:55:100
1398430290,3sleep21398449-21apt-get11:20:122
1301999291,2sleep10-21swapper/110:15:171
1299607291,2sleep00-21swapper/010:12:030
1204923290,1sleep00-21swapper/009:05:450
1174266291,4sleep21174359-21fschecks_time08:40:142
1173987291,3sleep01174006-21apt-get08:40:110
1162047290,1sleep00-21swapper/008:25:170
1152971290,4sleep00-21swapper/008:20:110
1136797290,4sleep00-21swapper/008:05:130
1135556291,3sleep01135603-21ntpq08:00:200
1135556291,3sleep01135603-21ntpq08:00:190
1120379290,1sleep00-21swapper/007:49:580
1115370292,2sleep1171rcu_preempt07:45:101
1110127291,3sleep11110343-21diskstats07:40:131
1109440291,4sleep11109385-31find07:35:551
1089386291,4sleep01091039-21cut07:20:160
1083478291,4sleep01083565-21cut07:15:110
1185054281,3sleep01185088-21kthreadcore08:50:170
1183096281,3sleep01183209-21ntpq08:45:210
1158766281,3sleep11159335-21grep08:25:141
1149362282,2sleep20-21swapper/208:15:172
1148220281,3sleep11149359-21kthreadcore08:15:161
1142921281,3sleep01143778-21cut08:10:130
1136819281,3sleep10-21swapper/108:05:141
1120903280,2sleep00-21swapper/007:50:020
1094616281,3sleep01095994-21kthreadcore07:25:170
1093432280,1sleep10-21swapper/107:20:371
1083566281,3sleep11083744-21diskstats07:15:111
1199910270,1sleep10-21swapper/109:04:011
1199655270,1sleep00-21swapper/009:00:470
1194432270,1sleep00-21swapper/008:56:230
1174009271,3sleep11174264-21df08:40:121
1163470270,2sleep00-21swapper/008:30:140
1136794270,1sleep20-21swapper/208:05:112
1130943270,1sleep20-21swapper/207:56:322
1121307271,3sleep21120922-21/usr/sbin/munin07:50:162
1098899270,1sleep20-21swapper/207:27:472
1089067270,3sleep21089281-21smartctl07:20:142
1080861270,1sleep10-21swapper/107:10:351
1186244261,2sleep10-21swapper/108:50:191
1180568260,2sleep20-21swapper/208:45:192
1173441260,4sleep20-21swapper/208:36:412
1152967260,1sleep10-21swapper/108:20:081
1147651260,2sleep00-21swapper/008:15:110
1142644261,2sleep20-21swapper/208:10:112
1135842261,2sleep10-21swapper/108:00:251
1135842261,2sleep10-21swapper/108:00:241
1125921261,2sleep11125950-21perf07:55:001
1105847260,2sleep20-21swapper/207:35:162
1080966260,4sleep00-21swapper/007:11:550
1204328250,1sleep10-21swapper/109:05:221
1194625250,1sleep10-21swapper/108:58:531
1164669250,2sleep10-21swapper/108:30:181
1130286250,1sleep10-21swapper/107:55:231
1099337250,1sleep10-21swapper/107:30:111
1110344240,1sleep20-21swapper/207:40:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional