You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-02 - 06:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Wed Jul 02, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10681042150102,15sleep30-21swapper/319:05:063
1072640214498,15sleep00-21swapper/019:07:520
1072532213698,13sleep10-21swapper/119:06:341
10681022131107,16sleep20-21swapper/219:05:052
10835592990,2sleep1321rcuc/119:15:171
1072928993833,3cyclictest1326898-21cpuspeed_turbos22:30:143
1072928993524,9cyclictest1269725-21gltestperf21:55:153
1072928993322,9cyclictest1530349-21needreboot00:35:213
1072928993322,9cyclictest1195911-21cpuspeed_turbos21:10:143
1072928993218,11cyclictest0-21swapper/321:55:003
107292899318,2cyclictest0-21swapper/321:30:153
1072928993115,3cyclictest0-21swapper/300:10:003
1072928993024,3cyclictest1363853-21grep22:50:283
1072928993021,6cyclictest1378426-21cut23:00:193
1072928993021,6cyclictest1378426-21cut23:00:193
1072928993017,9cyclictest1290259-21users22:05:273
1072928993016,2cyclictest1153949-21needreboot20:25:203
1072928992918,9cyclictest709-21polkitd22:20:003
1072928992916,2cyclictest0-21swapper/323:20:143
1072928992915,2cyclictest0-21swapper/321:10:003
1072928992913,2cyclictest0-21swapper/300:00:143
1072928992911,6cyclictest1453788-21ls23:45:283
107292899290,6cyclictest1442403-21kthreadcore23:40:193
1072928992813,2cyclictest0-21swapper/322:15:013
1072928992811,9cyclictest1312140-21cut22:20:193
1072928992811,6cyclictest1129243-21ps20:00:243
1072928992810,6cyclictest1395100-21cut23:10:173
107292899279,7cyclictest0-21swapper/321:43:573
107292899277,7cyclictest1281597-21grep22:00:233
1072928992718,6cyclictest1094551-21grep19:25:273
1072928992717,7cyclictest1425215-21uname23:30:193
1072928992716,2cyclictest1358810-21sh22:49:473
1072928992715,2cyclictest1211529-21cron21:20:003
1072928992714,5cyclictest511ktimers/300:10:163
107292899271,16cyclictest1525639-21sh00:34:113
107292899271,16cyclictest1324394-21sh22:27:103
1072928992711,6cyclictest1211831-21ssh21:20:033
107292899270,17cyclictest1350669-21sh22:44:363
14614242260,2sleep20-21swapper/223:50:192
1072928992618,6cyclictest0-21swapper/323:27:393
1072928992615,4cyclictest511ktimers/323:58:473
107292899261,16cyclictest1518194-21sh00:29:113
107292899261,12cyclictest1391534-21cron23:10:003
107292899261,11cyclictest1257498-21cat21:45:253
107292899258,8cyclictest1073121-21apt-get19:10:133
107292899250,6cyclictest1226326-21ssh21:27:253
107292899250,16cyclictest1439143-21sh23:37:523
107292899250,16cyclictest1405864-21sh23:17:103
107292899248,7cyclictest1242961-21ssh21:37:303
1072928992412,3cyclictest1510929-21sh00:23:553
107292899235,6cyclictest1080235-21diskstats19:15:143
1072928992315,6cyclictest1104505-21grep19:35:273
107292899231,10cyclictest1504447-21sleep00:20:003
1072928992310,6cyclictest1462114-21ssh23:50:303
1072928992217,3cyclictest733-21gdbus20:45:003
1072928992210,8cyclictest262-21systemd-journal22:38:323
1072928992210,5cyclictest1185799-21/usr/sbin/munin21:00:143
14056452211,2sleep20-21swapper/223:16:542
1072928992116,3cyclictest1089931-21sh19:25:003
1072928992116,3cyclictest0-21swapper/322:55:293
1072928992112,3cyclictest0-21swapper/319:58:513
14651212201,5sleep11465147-21latency_hist23:55:011
1072928992011,6cyclictest1114854-21taskset19:48:343
1072928991911,6cyclictest1135672-21hddtemp_smartct20:10:163
107292899191,16cyclictest1109966-21runrttasks19:44:543
14375272181,4sleep21195377-21diskmemload23:35:372
14075302181,4sleep11195377-21diskmemload23:19:321
12659682181,2sleep10-21swapper/121:50:421
1072928991811,5cyclictest1144702-21taskset20:16:293
14973212171,4sleep11497478-21df00:15:121
14488892171,3sleep11195377-21diskmemload23:44:461
12991572171,2sleep10-21swapper/122:11:301
107292899176,7cyclictest717-21systemd-logind20:47:013
107292899171,6cyclictest1165046-21/usr/sbin/munin20:40:003
107292899171,3cyclictest0-21swapper/319:53:293
107292899171,3cyclictest0-21swapper/319:53:293
1072928991713,2cyclictest1180352-21sh20:55:013
1072928991712,3cyclictest0-21swapper/320:10:003
14865282160,2sleep10-21swapper/100:05:421
13818872161,2sleep20-21swapper/223:02:502
13818872161,2sleep20-21swapper/223:02:502
13339972161,1sleep21195377-21diskmemload22:34:162
13313562160,9sleep10-21swapper/122:30:301
125336121611,2sleep20-21swapper/221:45:132
107292899167,6cyclictest1183670-21tr20:55:173
1072928991612,2cyclictest1095247-21apt-get19:30:133
107292899161,13cyclictest1149804-21runrttasks20:22:313
15228492151,3sleep21522899-21ntpq00:30:212
13743392151,8sleep11195377-21diskmemload22:58:481
13604902151,9sleep11362138-21cut22:50:181
12280332151,8sleep00-21swapper/021:29:560
1072928991511,2cyclictest0-21swapper/320:30:153
15334122140,1sleep20-21swapper/200:39:592
15268922141,6sleep11526972-21irqstats00:35:171
14997792141,8sleep21501013-21memory00:15:172
14899692141,9sleep11490050-21/usr/sbin/munin00:10:191
14726622140,2sleep20-21swapper/223:59:222
13092762141,3sleep01309286-21ssh22:19:220
12757742141,8sleep01275788-21sh21:58:060
12733192141,9sleep21273377-21ssh21:55:232
12444502141,3sleep21244463-21ssh21:39:402
11266652141,9sleep11128438-21cut20:00:191
11099752141,9sleep11110054-21sed19:45:001
15254162131,8sleep11525422-21sshd00:33:481
15107992131,6sleep21195377-21diskmemload00:23:392
14694142131,6sleep11469461-21ntpq23:55:241
14316392131,8sleep01431645-21ssh23:33:500
13557452131,8sleep11195377-21diskmemload22:45:291
13412312131,8sleep01341244-21rm22:37:500
13313992131,8sleep01195377-21diskmemload22:30:330
13167302130,1sleep10-21swapper/122:23:201
12998052130,3sleep20-21swapper/222:12:272
11156882131,8sleep00-21swapper/019:50:160
11156882131,8sleep00-21swapper/019:50:160
14868362121,5sleep21195377-21diskmemload00:06:112
14564922121,7sleep11456502-21ssh23:49:031
14232452121,7sleep11423257-21ssh23:28:301
13923352121,7sleep21392413-21ssh23:10:152
13907222121,7sleep21390735-21ssh23:08:512
13509312121,7sleep01351100-21latency_hist22:45:000
13436452121,7sleep21343719-21/usr/sbin/munin22:40:212
12989562121,7sleep01195377-21diskmemload22:11:130
12934462121,7sleep01293606-21date22:10:000
12735612121,7sleep11273573-21mailstats21:55:261
12588432121,7sleep01258845-21ssh21:47:030
11250042121,7sleep21125037-21perf20:00:002
15312552114,3sleep0171ktimers/000:35:440
15253402111,3sleep00-21swapper/000:33:390
14792202111,4sleep01195377-21diskmemload00:01:530
14645342111,3sleep00-21swapper/023:54:110
14499812111,6sleep21451340-21cat23:45:192
14303022111,3sleep10-21swapper/123:31:551
14222572111,4sleep21195377-21diskmemload23:27:082
14146902111,3sleep20-21swapper/223:22:582
13972532111,3sleep10-21swapper/123:11:351
13911362110,5sleep10-21swapper/123:09:271
13598162111,6sleep01360489-21cut22:50:170
13573782111,3sleep00-21swapper/022:47:410
13400942111,5sleep21195377-21diskmemload22:36:132
13257002110,1sleep10-21swapper/122:29:071
13190622111,6sleep01320022-21kthreadcore22:25:200
13018332111,3sleep20-21swapper/222:15:042
12932162111,4sleep11195377-21diskmemload22:09:411
12775802110,7sleep01277695-21/usr/sbin/munin22:00:140
12523432111,3sleep20-21swapper/221:44:212
12499812111,3sleep10-21swapper/121:40:551
12425352111,3sleep10-21swapper/121:36:541
12358732111,7sleep11235886-21ssh21:34:061
12350502110,5sleep00-21swapper/021:32:560
12196672111,4sleep11195377-21diskmemload21:25:001
12091652110,7sleep21209176-21ssh21:16:392
12005212111,3sleep20-21swapper/221:10:462
12004012111,3sleep00-21swapper/021:10:360
11793812111,5sleep01179426-21ntpq20:50:200
11792242112,4sleep21179322-21ntpq20:50:182
11568052111,6sleep21158612-21kthreadcore20:30:182
11549952110,7sleep11155111-21latency_hist20:30:011
15169102101,3sleep10-21swapper/100:26:441
15163232100,3sleep20-21swapper/200:25:332
15103412104,3sleep1331ktimers/100:22:421
15029612100,1sleep00-21swapper/000:17:170
14968152101,3sleep21195377-21diskmemload00:14:492
14959942101,2sleep00-21swapper/000:13:310
14871712101,4sleep01195377-21diskmemload00:06:460
14805382100,3sleep20-21swapper/200:03:442
14789612101,6sleep11478974-21ssh00:01:291
14727412101,2sleep00-21swapper/023:59:280
14540802101,4sleep01195377-21diskmemload23:45:330
14469102101,3sleep01195377-21diskmemload23:41:570
14408702101,4sleep21440621-21apt-get23:40:022
14381912101,3sleep01195377-21diskmemload23:36:310
14367292101,4sleep11195377-21diskmemload23:35:241
14315192101,4sleep21195377-21diskmemload23:33:402
14217662101,3sleep00-21swapper/023:26:250
13821502101,3sleep01195377-21diskmemload23:03:120
13821502101,3sleep01195377-21diskmemload23:03:120
13736062101,3sleep00-21swapper/022:57:430
13726462101,3sleep21195377-21diskmemload22:56:162
13560692100,3sleep20-21swapper/222:45:472
13492792100,3sleep10-21swapper/122:42:401
13172062100,4sleep21195377-21diskmemload22:24:012
13161862101,3sleep01195377-21diskmemload22:22:310
13067472101,4sleep11195377-21diskmemload22:15:471
12912882101,5sleep21291299-21ssh22:06:542
12825202101,3sleep10-21swapper/122:01:041
12650802101,4sleep01265133-21ntpq21:50:200
12171132100,3sleep00-21swapper/021:21:180
12166622100,1sleep20-21swapper/221:20:402
12099622101,3sleep01195377-21diskmemload21:17:450
12026862101,3sleep10-21swapper/121:13:571
11843952101,4sleep11184448-21ntp_states20:55:221
11340302101,4sleep21134076-21ntpq20:05:212
10947402100,1sleep10-21swapper/119:27:091
10939642101,4sleep21094014-21ntpq19:25:212
1516540291,2sleep00-21swapper/000:25:580
1509288292,4sleep01473274-21kworker/0:0+events00:20:280
1412138291,4sleep11412196-21ntpq23:20:211
1412004290,3sleep00-21swapper/023:20:200
1407860290,3sleep01195377-21diskmemload23:19:590
1395822291,4sleep01395890-21ntp_states23:10:210
1383747290,3sleep01195377-21diskmemload23:05:130
1364263290,3sleep20-21swapper/222:50:592
1339750291,2sleep10-21swapper/122:35:441
1323281291,2sleep20-21swapper/222:25:382
1282706290,2sleep20-21swapper/222:01:222
1261153291,2sleep20-21swapper/221:50:022
1258060290,3sleep10-21swapper/121:45:561
1251771290,2sleep00-21swapper/021:43:320
1243632290,1sleep00-21swapper/021:38:300
1226204290,2sleep10-21swapper/121:27:161
1224892291,4sleep21224969-21ssh21:25:302
1208749290,2sleep10-21swapper/121:16:021
1160267291,3sleep21160286-21apt-get20:35:152
1155259291,3sleep11155278-21apt-get20:30:141
1135772291,4sleep21137311-21cut20:10:182
1134428291,4sleep01134645-21users20:05:270
1109084291,4sleep21109169-21ps19:40:242
1381442280,1sleep10-21swapper/123:02:131
1381442280,1sleep10-21swapper/123:02:131
1228675281,4sleep21228686-21ssh21:30:062
1191057281,3sleep21191356-21tr21:05:172
1187009281,3sleep01188785-21cut21:00:180
1154868280,1sleep20-21swapper/220:28:252
1125748281,4sleep01126663-21cut20:00:180
1115253281,3sleep11115405-21df_inode19:50:131
1115253281,3sleep11115405-21df_inode19:50:131
1114648280,2sleep20-21swapper/219:45:572
1104098281,4sleep21104221-21sendmail_mailqu19:35:232
1077746283,3sleep1331ktimers/119:13:141
1195369271,3sleep11195594-21latency_hist21:10:001
1189294271,2sleep21189367-21ntpq21:00:202
1185177270,6sleep00-21swapper/020:58:010
1169619270,1sleep20-21swapper/220:40:292
1149276271,3sleep21149283-21sendmail_mailtr20:20:242
1138967270,2sleep10-21swapper/120:10:211
1129356271,3sleep21129570-21grep20:00:282
1120249270,2sleep00-21swapper/019:55:130
1104856270,1sleep00-21swapper/019:38:260
1085122270,2sleep20-21swapper/219:20:122
1073102270,2sleep00-21swapper/019:10:120
1185044260,1sleep20-21swapper/220:56:232
1174970260,1sleep10-21swapper/120:49:331
1174104260,2sleep00-21swapper/020:45:230
1170254260,1sleep20-21swapper/220:45:132
1165437260,1sleep10-21swapper/120:40:151
1164864260,1sleep10-21swapper/120:38:381
1164863260,1sleep00-21swapper/020:38:370
1149969260,1sleep10-21swapper/120:24:401
1141781260,3sleep01143585-21cut20:15:170
1121396260,3sleep11123127-21kthreadcore19:55:181
1105268261,2sleep00-21swapper/019:40:140
1104046261,2sleep10-21swapper/119:35:221
1099961260,1sleep10-21swapper/119:34:581
1099637260,1sleep20-21swapper/219:30:542
1084636260,1sleep20-21swapper/219:17:282
1077916260,3sleep01078440-21cat19:15:010
1077767260,1sleep20-21swapper/219:13:312
1180193250,1sleep10-21swapper/120:53:091
1158958250,1sleep00-21swapper/020:30:200
1149716250,1sleep00-21swapper/020:21:260
1143586250,2sleep10-21swapper/120:15:181
1133971250,2sleep10-21swapper/120:05:201
1118449250,2sleep21118934-21strings19:50:192
1118449250,2sleep21118934-21strings19:50:182
1114969250,1sleep00-21swapper/019:50:000
1114619250,3sleep1331ktimers/119:45:361
1095228250,1sleep00-21swapper/019:30:130
1088625250,1sleep00-21swapper/019:20:180
1085510250,1sleep10-21swapper/119:20:151
1190805240,1sleep00-21swapper/021:05:150
1185795240,1sleep10-21swapper/121:00:151
1165281240,1sleep00-21swapper/020:40:140
1150286240,1sleep00-21swapper/020:25:150
1140246240,1sleep20-21swapper/220:15:112
1135462240,1sleep00-21swapper/020:10:160
1093437240,1sleep00-21swapper/019:25:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional