You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-16 - 19:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Mon Jun 16, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
920645211758,55sleep30-21swapper/307:05:573
92074728454,9sleep10-21swapper/107:07:151
92064428354,9sleep20-21swapper/207:05:562
91620227959,13sleep00-21swapper/007:05:130
13653252271,2sleep20-21swapper/212:33:322
921096992419,2cyclictest0-21swapper/312:26:063
92109699241,8cyclictest1258227-21taskset11:26:293
10980092241,8sleep11052886-21diskmemload09:40:171
921096992319,2cyclictest0-21swapper/311:55:263
92109699222,14cyclictest1127204-21chrt10:00:153
921096992219,1cyclictest0-21swapper/308:30:153
921096992218,2cyclictest0-21swapper/309:57:373
921096992217,2cyclictest0-21swapper/311:45:213
921096992217,2cyclictest0-21swapper/311:45:213
921096992217,2cyclictest0-21swapper/309:21:553
921096992216,3cyclictest0-21swapper/309:15:233
921096992212,7cyclictest958800-21taskset07:40:263
921096992210,8cyclictest1215370-21taskset10:57:563
13471362221,2sleep00-21swapper/012:22:310
92109699212,13cyclictest1359640-21taskset12:30:203
921096992116,2cyclictest0-21swapper/310:30:013
921096992115,3cyclictest0-21swapper/310:45:183
92109699211,16cyclictest1082059-21taskset09:29:333
13447702211,5sleep11344830-21ntpq12:20:191
92109699202,13cyclictest1116238-21taskset09:50:213
921096992017,1cyclictest0-21swapper/312:05:153
92109699201,15cyclictest0-21swapper/311:30:153
92109699200,17cyclictest0-21swapper/311:35:093
92109699198,8cyclictest1223379-21turbostat11:09:553
92109699194,13cyclictest0-21swapper/310:41:213
921096991915,2cyclictest0-21swapper/312:36:573
921096991915,2cyclictest0-21swapper/310:30:153
921096991915,2cyclictest0-21swapper/307:55:153
92109699191,15cyclictest0-21swapper/312:20:123
92109699191,14cyclictest0-21swapper/312:00:213
92109699191,14cyclictest0-21swapper/311:50:093
92109699191,14cyclictest0-21swapper/308:10:153
13648432190,2sleep00-21swapper/012:32:530
12400662190,3sleep2441rcuc/211:15:172
9547772181,8sleep0956424-21grep07:40:170
92109699188,7cyclictest1337241-21taskset12:15:593
92109699182,11cyclictest1184313-21taskset10:35:433
92109699182,11cyclictest1103601-21taskset09:43:113
92109699182,11cyclictest1057648-21taskset09:10:233
921096991815,1cyclictest0-21swapper/309:45:133
921096991815,1cyclictest0-21swapper/308:50:203
921096991815,1cyclictest0-21swapper/307:50:153
921096991814,2cyclictest0-21swapper/312:10:213
921096991814,2cyclictest0-21swapper/309:30:313
921096991814,2cyclictest0-21swapper/307:25:263
921096991812,3cyclictest0-21swapper/310:21:313
92109699181,14cyclictest82250irq/126-enp1s0-TxRx-309:39:383
92109699181,14cyclictest0-21swapper/311:20:183
92109699181,14cyclictest0-21swapper/311:00:023
92109699181,13cyclictest0-21swapper/311:40:253
92109699181,13cyclictest0-21swapper/310:10:513
92109699181,13cyclictest0-21swapper/310:05:273
92109699181,12cyclictest943877-21chrt07:30:123
92109699180,14cyclictest0-21swapper/311:11:113
92109699180,14cyclictest0-21swapper/310:50:503
92109699180,14cyclictest0-21swapper/310:50:493
92109699180,13cyclictest9375692sleep307:20:233
10640322180,2sleep20-21swapper/209:15:202
921096991714,1cyclictest0-21swapper/309:00:203
921096991712,3cyclictest0-21swapper/310:15:133
921096991712,2cyclictest0-21swapper/307:50:003
92109699171,13cyclictest0-21swapper/307:10:103
92109699171,12cyclictest1242920-21chrt11:15:583
92109699171,10cyclictest1026314-21taskset08:50:003
92109699171,10cyclictest1005655-21taskset08:30:003
12895582171,7sleep01289568-21ssh11:46:400
12895582171,7sleep01289568-21ssh11:46:400
12500342170,8sleep11052886-21diskmemload11:20:311
11637792170,3sleep00-21swapper/010:24:450
11326162170,2sleep00-21swapper/010:01:580
10314052171,10sleep11031434-21perf08:54:591
921096991613,1cyclictest0-21swapper/309:05:213
92109699161,12cyclictest0-21swapper/308:07:413
92109699161,11cyclictest1021182-21turbostat08:45:003
92109699160,13cyclictest0-21swapper/308:15:263
92109699160,13cyclictest0-21swapper/308:00:193
92109699160,13cyclictest0-21swapper/307:38:283
92109699160,12cyclictest0-21swapper/308:20:213
13742842160,10sleep00-21swapper/012:38:470
13552212161,2sleep00-21swapper/012:26:450
13128542161,2sleep00-21swapper/012:00:260
10575902162,7sleep21057641-21ntpq09:10:232
92109699151,9cyclictest926611-21taskset07:15:023
92109699151,2cyclictest0-21swapper/308:35:213
921096991512,1cyclictest0-21swapper/308:55:273
13381662150,2sleep00-21swapper/012:17:150
12642972151,9sleep11264427-21cut11:30:221
12596922150,3sleep00-21swapper/011:29:240
12438602151,10sleep01243870-21ssh11:17:540
12293912151,10sleep01229405-21ssh11:06:470
12161422150,2sleep00-21swapper/010:59:260
12076402150,2sleep00-21swapper/010:52:280
12076402150,2sleep00-21swapper/010:52:280
10890052151,6sleep10-21swapper/109:33:471
13164842141,8sleep21316514-21ssh12:05:142
12921382141,7sleep00-21swapper/011:50:120
12006372141,9sleep21052886-21diskmemload10:48:342
11642632140,2sleep00-21swapper/010:25:150
11406032140,3sleep01052886-21diskmemload10:08:110
11089242142,6sleep21108970-21ntpq09:45:202
10813792141,8sleep21052886-21diskmemload09:28:042
13323952132,7sleep21332605-21idleruntime12:15:202
12880952131,5sleep21288151-21ntpq11:45:262
12880952131,5sleep21288151-21ntpq11:45:252
12779462131,8sleep01279721-21ssh11:40:230
12671502131,2sleep20-21swapper/211:34:032
11855602131,5sleep00-21swapper/010:38:210
11017332131,8sleep21101868-21cat09:40:262
10958312131,8sleep21052886-21diskmemload09:37:262
10826132131,6sleep00-21swapper/009:30:060
9683982120,8sleep2968474-21gcc07:50:202
9494482121,3sleep0950446-21kthreadcore07:35:160
9487252120,8sleep1948955-21cat07:35:001
9340482123,5sleep1935087-21cat07:20:181
9227002121,7sleep2923206-21kthreadcore07:10:162
13243622121,7sleep01052886-21diskmemload12:10:160
13224082121,7sleep11322413-21ssh12:07:261
13055772120,1sleep00-21swapper/011:56:180
12836012121,6sleep10-21swapper/111:45:011
12731002120,3sleep01273114-21ssh11:36:020
12522342121,7sleep21252546-21gpg-connect-age11:25:012
12178582121,7sleep11219500-21grep11:00:191
12073472121,6sleep21052886-21diskmemload10:51:542
12073472121,6sleep21052886-21diskmemload10:51:532
11926742121,3sleep21192680-21ssh10:42:232
11703502120,1sleep20-21swapper/210:28:172
11481402121,3sleep00-21swapper/010:13:150
11407532121,6sleep21052886-21diskmemload10:08:292
11271742122,4sleep21127196-21apt-get10:00:142
10821542120,8sleep11082160-21ssh09:29:441
10749142121,7sleep01074924-21sh09:24:360
10604242122,5sleep01060444-21apt-get09:15:150
10457242121,7sleep20-21swapper/209:05:232
9944642110,7sleep2994622-21sed08:15:262
9800882111,5sleep1980238-21df_inode08:05:151
9730532111,6sleep2973572-21grep07:55:182
9538702111,5sleep1953813-31find07:38:371
13660782111,3sleep10-21swapper/112:34:311
13471502111,3sleep20-21swapper/212:22:342
13303802111,4sleep11052886-21diskmemload12:12:381
12748582111,3sleep20-21swapper/211:38:592
12729882111,4sleep11052886-21diskmemload11:35:501
12490212110,6sleep01052886-21diskmemload11:20:220
12371682111,3sleep10-21swapper/111:14:101
12292022113,3sleep2451ktimers/211:06:202
12079882111,4sleep11052886-21diskmemload10:53:061
12079882111,4sleep11052886-21diskmemload10:53:051
11929762111,2sleep00-21swapper/010:42:540
11929032110,7sleep11192913-21ssh10:42:461
11769412111,6sleep10-21swapper/110:31:281
11543622111,4sleep11052886-21diskmemload10:15:461
11420952111,4sleep11052886-21diskmemload10:10:151
11418262111,6sleep21141835-21ssh10:10:042
11254352111,3sleep10-21swapper/109:57:261
11179092111,3sleep20-21swapper/209:52:242
11114132111,3sleep10-21swapper/109:49:041
10977682111,5sleep01098008-21cut09:40:160
10588302112,4sleep11052886-21diskmemload09:11:591
10269982115,3sleep21027930-21cut08:50:192
10267142114,3sleep01026842-21fschecks_time08:50:170
10157392111,5sleep01015817-21munin-run08:40:000
9538632101,4sleep2953813-31find07:38:342
13722252101,3sleep21052886-21diskmemload12:36:092
13669102100,7sleep11367031-21cpuspeed_turbos12:35:161
13552472101,4sleep21052886-21diskmemload12:26:472
13550292101,4sleep11052886-21diskmemload12:26:301
13384852101,3sleep10-21swapper/112:17:401
13216982100,6sleep01321709-21ssh12:06:160
13159512101,3sleep20-21swapper/212:04:452
13150092101,3sleep11052886-21diskmemload12:03:121
13076942100,4sleep11052886-21diskmemload11:59:281
12920772101,4sleep21052886-21diskmemload11:50:062
12840072101,4sleep11284031-21apt-get11:45:171
12840072101,4sleep11284031-21apt-get11:45:161
12758722100,1sleep20-21swapper/211:40:132
12582962101,3sleep20-21swapper/211:26:412
12432052100,3sleep10-21swapper/111:16:341
12375032101,2sleep20-21swapper/211:14:512
12353982100,3sleep01235526-21diskmemload11:10:280
12296482101,3sleep10-21swapper/111:07:201
12168302101,4sleep01216849-21apt-get11:00:140
12092062101,2sleep20-21swapper/210:55:022
12090132101,4sleep11208682-21apt-get10:55:011
11998062101,4sleep11052886-21diskmemload10:47:001
11993792101,3sleep00-21swapper/010:46:150
11843302101,3sleep10-21swapper/110:35:461
11767722101,5sleep21052886-21diskmemload10:31:092
11707422101,4sleep11052886-21diskmemload10:28:561
11320182101,3sleep11052886-21diskmemload10:00:411
11191352101,3sleep10-21swapper/109:54:401
11186122101,3sleep00-21swapper/009:53:430
11114072101,3sleep00-21swapper/009:49:030
10884722101,4sleep21052886-21diskmemload09:32:322
10802542101,4sleep01052886-21diskmemload09:25:340
10728262100,4sleep11052886-21diskmemload09:20:411
10604472101,4sleep11060610-21diskstats09:15:161
10206112101,4sleep01020796-21users08:40:280
10043342101,5sleep21004440-21ntpq08:25:202
10007442101,5sleep11000845-21tune2fs08:25:151
990802295,1sleep10-21swapper/108:15:171
980066291,3sleep00-21swapper/008:05:140
969740291,3sleep00-21swapper/007:55:120
964320291,4sleep2964369-21munin-run07:50:012
943855291,2sleep2943874-21apt-get07:30:122
938923291,4sleep1939132-21hddtemp_smartct07:25:151
928384291,2sleep20-21swapper/207:15:142
1307873291,2sleep20-21swapper/211:59:452
1292063291,3sleep11052886-21diskmemload11:50:051
1266326291,4sleep01266333-21ssh11:32:370
1259468290,2sleep10-21swapper/111:28:561
1221998291,3sleep21052886-21diskmemload11:01:122
1179452291,4sleep21179611-21df_inode10:35:192
1172909291,4sleep01174624-21grep10:30:190
1162191291,4sleep11052886-21diskmemload10:21:341
1149430291,2sleep00-21swapper/010:15:080
1140574291,2sleep10-21swapper/110:08:081
1125198291,3sleep00-21swapper/009:56:400
1123814291,4sleep21123874-21ntpq09:55:222
1095594291,3sleep10-21swapper/109:37:011
1072834290,3sleep21052886-21diskmemload09:20:422
1058319291,2sleep00-21swapper/009:10:590
1025166291,3sleep21025272-21ntpq08:45:232
1004756291,5sleep01004771-21grep08:25:260
995789281,3sleep1995905-21/usr/sbin/munin08:20:171
989807280,1sleep10-21swapper/108:11:591
971234281,3sleep1973052-21kthreadcore07:55:171
969250280,1sleep10-21swapper/107:52:071
940613280,1sleep00-21swapper/007:25:170
933061280,1sleep10-21swapper/107:17:011
925417280,1sleep10-21swapper/107:10:241
1161896281,3sleep21161901-21ssh10:20:592
1154418280,2sleep20-21swapper/210:15:512
1090232281,4sleep01090329-21fschecks_time09:35:160
1043856281,4sleep01045560-21grep09:05:210
1042116281,3sleep1341rcuc/109:05:181
1032210281,3sleep01033881-21cut08:55:190
1016285281,2sleep21015748-21apt-get08:40:012
1010896281,3sleep11011046-21df_inode08:35:161
1010606281,4sleep11010743-21latency_hist08:34:591
1010287282,3sleep0944-21snmpd08:30:580
1000136280,4sleep00-21swapper/008:22:440
995443270,2sleep20-21swapper/208:20:142
994135271,2sleep0994176-21ntpq08:15:200
985347271,3sleep2441rcuc/208:10:162
985178270,2sleep00-21swapper/008:10:140
979631270,1sleep20-21swapper/208:02:502
979111271,3sleep1979179-21cat08:00:251
959595270,4sleep10-21swapper/107:45:151
948397270,1sleep00-21swapper/007:30:510
937438270,2sleep00-21swapper/007:20:210
935088271,3sleep20-21swapper/207:20:182
933289270,1sleep00-21swapper/007:19:520
925802271,3sleep0944-21snmpd07:10:430
1324259271,2sleep21324273-21ssh12:10:112
1041554270,1sleep00-21swapper/009:03:330
1038755270,2sleep20-21swapper/209:00:212
1036499270,1sleep20-21swapper/208:59:502
1005709270,2sleep20-21swapper/208:30:112
984229261,2sleep2984451-21unixbench_multi08:05:282
965932260,3sleep0967722-21kthreadcore07:50:190
960140261,2sleep00-21swapper/007:45:180
1040956260,2sleep10-21swapper/109:00:281
1035828261,2sleep10-21swapper/108:55:281
1022643261,2sleep00-21swapper/008:45:210
1014848260,1sleep20-21swapper/208:35:262
959081251,2sleep2921090-21cyclictest07:43:582
954263250,1sleep10-21swapper/107:40:051
942549250,1sleep20-21swapper/207:25:202
1015830250,2sleep10-21swapper/108:40:001
974927240,1sleep00-21swapper/008:00:130
1021416240,1sleep10-21swapper/108:45:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional