You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-19 - 19:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel

Intel(R) Atom(TM) CPU D510 @ 1.66GHz, Linux 3.12.61-rt81 (Profile)

Latency plot of shadow in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot6s.osadl.org (updated Sun Sep 19, 2021 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
227499820,7rtkit-daemon0-21swapper/207:00:320
227499700,5rtkit-daemon16331-21kworker/u8:006:56:243
300872657,51sleep00-21swapper/009:45:340
300872657,51sleep00-21swapper/009:45:340
1267499654,54cyclictest0-21swapper/109:34:031
1267699644,53cyclictest0-21swapper/308:40:353
138272630,5sleep013828-21munin-node06:55:310
1267499633,54cyclictest0-21swapper/107:20:311
1267499633,54cyclictest0-21swapper/107:20:311
12674996234,12cyclictest0-21swapper/108:40:181
12674996233,12cyclictest0-21swapper/112:10:171
12674996232,12cyclictest0-21swapper/109:35:341
1267699613,51cyclictest0-21swapper/310:00:433
12674996137,8cyclictest31374-21unixbench_multi10:55:371
12674996136,11cyclictest0-21swapper/108:23:381
1267499613,52cyclictest0-21swapper/112:05:381
1267599604,40cyclictest0-21swapper/210:55:172
12674996035,12cyclictest0-21swapper/110:17:261
268882595,10sleep00-21swapper/009:40:220
247322593,11sleep30-21swapper/307:20:303
247322593,11sleep30-21swapper/307:20:303
191322592,10sleep00-21swapper/009:25:070
182392593,8sleep018264-21systemd-cgroups08:15:090
12674995931,12cyclictest0-21swapper/109:45:351
12674995931,12cyclictest0-21swapper/109:45:351
1267499593,51cyclictest0-21swapper/107:01:471
1267499593,50cyclictest0-21swapper/109:00:271
12674995836,8cyclictest16130-21idleruntime-cro11:35:011
12674995836,8cyclictest16130-21idleruntime-cro11:35:011
12674995831,11cyclictest0-21swapper/110:40:251
12674995831,11cyclictest0-21swapper/108:45:551
12674995830,12cyclictest0-21swapper/109:55:511
1267299584,46cyclictest0-21swapper/008:25:270
73912573,46sleep10-21swapper/111:15:201
217952573,9sleep121910-21unixbench_multi10:35:371
217952573,9sleep121910-21unixbench_multi10:35:371
12676995743,8cyclictest31734-21uniq09:50:233
12674995733,11cyclictest0-21swapper/108:35:301
12674995733,11cyclictest0-21swapper/107:52:331
12674995732,11cyclictest0-21swapper/109:10:251
12674995731,11cyclictest0-21swapper/110:09:461
12674995730,9cyclictest20855-21latency_hist11:45:011
12674995730,11cyclictest0-21swapper/112:15:171
1267499573,48cyclictest0-21swapper/110:10:211
12674995729,13cyclictest0-21swapper/109:50:351
12674995728,12cyclictest0-21swapper/111:50:451
87882563,46sleep20-21swapper/211:17:142
194982565,10sleep00-21swapper/008:15:450
1267699564,43cyclictest0-21swapper/307:55:513
12676995632,8cyclictest1251-21snmpd07:01:483
1267699563,46cyclictest0-21swapper/309:10:173
12674995634,12cyclictest0-21swapper/110:45:231
12674995631,11cyclictest0-21swapper/107:55:331
1267499563,47cyclictest0-21swapper/107:45:411
12674995628,12cyclictest0-21swapper/111:13:251
12674995626,11cyclictest0-21swapper/110:30:041
42992555,10sleep00-21swapper/011:10:030
283842554,43sleep20-21swapper/208:35:272
12676995530,9cyclictest1251-21snmpd12:20:053
12675995538,11cyclictest6162-21head12:20:202
1267499555,40cyclictest0-21swapper/107:33:431
1267499554,35cyclictest18297-21proc_pri09:20:381
12674995529,11cyclictest0-21swapper/110:52:311
12674995528,11cyclictest0-21swapper/111:26:401
12674995528,11cyclictest0-21swapper/107:30:021
12674995527,12cyclictest0-21swapper/112:20:211
12674995525,16cyclictest0-21swapper/107:10:421
1267299554,34cyclictest0-21swapper/008:45:410
12676995431,9cyclictest18717-21munin-node11:40:183
1267699543,44cyclictest13124-21munin-node10:20:343
1267699543,43cyclictest28341-21iostat_ios10:50:263
1267499545,44cyclictest0-21swapper/109:40:281
12674995433,7cyclictest18-21rcuc/112:00:141
12674995428,12cyclictest0-21swapper/108:15:561
12674995427,12cyclictest0-21swapper/111:00:341
12674995427,12cyclictest0-21swapper/108:30:221
12674995427,10cyclictest0-21swapper/111:35:171
12674995425,14cyclictest0-21swapper/110:32:371
12672995439,10cyclictest0-21swapper/007:35:410
12672995438,11cyclictest0-21swapper/008:52:560
12672995438,11cyclictest0-21swapper/008:52:560
12672995421,27cyclictest0-21swapper/010:02:420
315032534,42sleep20-21swapper/207:34:122
146672532,10sleep00-21swapper/010:20:410
141822532,10sleep20-21swapper/208:05:272
141822532,10sleep20-21swapper/208:05:272
12676995330,9cyclictest1797-21munin-node07:40:433
12676995330,9cyclictest1251-21snmpd07:10:563
1267699533,43cyclictest15485-21munin-node10:25:283
12676995329,9cyclictest22501-21gltestperf09:30:233
12676995325,13cyclictest0-21swapper/308:40:073
12676995323,13cyclictest0-21swapper/306:55:123
1267699532,43cyclictest20220-21munin-node10:35:313
1267699532,43cyclictest20220-21munin-node10:35:313
12676995315,13cyclictest0-21swapper/308:30:083
1267499536,27cyclictest1336-21runrttasks10:20:321
1267499533,45cyclictest0-21swapper/110:00:431
1267499533,44cyclictest0-21swapper/107:35:281
12674995328,11cyclictest0-21swapper/108:05:451
12674995328,11cyclictest0-21swapper/108:05:451
12674995327,12cyclictest0-21swapper/108:50:221
12674995327,12cyclictest0-21swapper/108:50:221
12672995332,9cyclictest4288-21munin-node07:46:060
311992523,7sleep331317-21perl12:05:203
255442525,40sleep20-21swapper/209:35:412
197932522,9sleep30-21swapper/310:32:453
12676995239,8cyclictest27303-21tr07:25:293
12676995230,8cyclictest1251-21snmpd11:39:583
1267699523,42cyclictest25093-21iostat09:35:303
1267699523,42cyclictest24162-21yum07:15:593
12676995229,9cyclictest13754-21munin-node08:05:403
12676995229,9cyclictest13754-21munin-node08:05:403
12676995228,10cyclictest29457-21munin-node07:30:303
12675995237,9cyclictest14637-21grep06:55:452
12675995213,13cyclictest0-21swapper/207:55:532
1267499525,30cyclictest0-21swapper/111:49:081
1267499523,44cyclictest0-21swapper/111:22:351
1267499523,43cyclictest0-21swapper/111:05:291
12674995228,10cyclictest0-21swapper/109:25:281
12674995227,11cyclictest0-21swapper/107:15:461
12674995226,13cyclictest0-21swapper/106:57:191
12674995224,14cyclictest0-21swapper/107:09:271
12674995224,13cyclictest0-21swapper/109:07:431
12674995224,13cyclictest0-21swapper/108:10:551
12674995223,14cyclictest0-21swapper/111:59:351
12674995222,13cyclictest0-21swapper/108:27:081
12672995238,8cyclictest41ktimersoftd/008:40:360
234642512,9sleep20-21swapper/211:50:072
22822514,40sleep20-21swapper/209:55:402
12676995134,10cyclictest16361-21grep09:15:533
12676995130,8cyclictest1251-21snmpd10:43:213
12676995129,8cyclictest1251-21snmpd08:56:523
12676995129,8cyclictest1251-21snmpd08:56:523
1267699512,45cyclictest0-21swapper/309:50:053
1267699512,45cyclictest0-21swapper/309:50:053
12675995136,8cyclictest18748-21wc10:30:242
12675995135,9cyclictest2328-21unixbench_multi12:10:352
1267499516,28cyclictest0-21swapper/108:56:491
1267499516,28cyclictest0-21swapper/108:56:491
12674995127,10cyclictest0-21swapper/107:40:401
12674995125,12cyclictest0-21swapper/109:15:281
12672995133,13cyclictest0-21swapper/011:45:280
1267299513,41cyclictest0-21swapper/012:05:380
1267299513,41cyclictest0-21swapper/007:10:110
12672995127,9cyclictest27723-21munin-run08:35:080
12672995124,12cyclictest0-21swapper/010:30:280
121552512,9sleep30-21swapper/308:00:333
108932512,9sleep30-21swapper/310:15:283
292352504,38sleep20-21swapper/210:52:072
285352502,9sleep30-21swapper/309:44:283
235492502,10sleep20-21swapper/209:31:132
15692505,9sleep00-21swapper/011:02:090
12676995030,8cyclictest1251-21snmpd10:45:423
1267699503,40cyclictest1613-21df_inode09:55:293
12676995028,9cyclictest28202-21munin-node12:00:223
12676995028,9cyclictest1251-21snmpd08:19:263
12676995021,12cyclictest0-21swapper/308:10:293
12675995036,8cyclictest3854-21tr10:00:212
1267599503,22cyclictest0-21swapper/209:02:302
1267599503,22cyclictest0-21swapper/207:24:142
1267599503,22cyclictest0-21swapper/207:24:142
12674995025,12cyclictest0-21swapper/108:04:411
12672995026,9cyclictest21841-21sort10:35:370
12672995026,9cyclictest21841-21sort10:35:370
12672995023,12cyclictest0-21swapper/007:10:370
116902502,41sleep30-21swapper/309:05:553
87492492,41sleep00-21swapper/007:54:490
69872492,40sleep00-21swapper/008:56:170
69872492,40sleep00-21swapper/008:56:170
5742492,40sleep00-21swapper/009:51:140
50252492,40sleep30-21swapper/312:18:343
33522492,10sleep00-21swapper/007:40:470
197392492,9sleep20-21swapper/209:25:212
150752492,9sleep20-21swapper/210:22:532
1267699493,40cyclictest4279-21latency_hist11:10:033
12676994928,8cyclictest20962-21munin-node09:25:383
12676994926,9cyclictest25839-21munin-node11:55:333
12676994923,8cyclictest1560-21munin-run07:40:103
12676994920,13cyclictest0-21swapper/307:50:093
12676994913,24cyclictest0-21swapper/308:31:183
1267599495,22cyclictest0-21swapper/211:45:342
1267599495,14cyclictest0-21swapper/211:25:352
12675994933,9cyclictest31754-21grep12:05:272
12675994929,11cyclictest8779-21crond07:55:092
12672994935,8cyclictest1251-21snmpd07:59:200
12672994934,9cyclictest41ktimersoftd/009:55:460
1267299493,39cyclictest0-21swapper/011:55:380
12672994923,11cyclictest0-21swapper/007:20:450
12672994923,11cyclictest0-21swapper/007:20:450
12672994922,12cyclictest0-21swapper/010:45:030
93242482,40sleep30-21swapper/309:00:553
133242482,40sleep20-21swapper/208:02:452
12676994833,9cyclictest4470-21unixbench_multi08:50:443
12676994833,9cyclictest4470-21unixbench_multi08:50:443
1267699483,38cyclictest23521-21cpu11:50:123
1267699483,15cyclictest0-21swapper/307:50:513
12676994824,11cyclictest7138-21expr11:15:153
12676994821,11cyclictest0-21swapper/310:10:293
12675994835,8cyclictest30464-21cpuspeed_turbos08:40:182
12675994833,9cyclictest22536-21cpuspeed_turbos07:15:332
12675994831,11cyclictest438-21cpuspeed_turbos08:45:372
1267599482,21cyclictest0-21swapper/211:40:142
12675994810,9cyclictest0-21swapper/212:04:592
1267299483,39cyclictest0-21swapper/012:15:280
12672994827,14cyclictest0-21swapper/010:25:210
12672994822,12cyclictest0-21swapper/012:10:190
12672994818,13cyclictest0-21swapper/011:55:010
105362482,39sleep00-21swapper/010:15:050
78252472,9sleep00-21swapper/010:05:540
273712472,9sleep20-21swapper/208:30:472
269632472,40sleep00-21swapper/010:48:180
261912472,9sleep00-21swapper/009:39:420
238382472,40sleep00-21swapper/009:34:540
228872474,9sleep00-21swapper/008:24:000
13472472,9sleep20-21swapper/207:37:342
1267699474,27cyclictest0-21swapper/309:20:383
1267699473,38cyclictest0-21swapper/311:22:023
12676994729,11cyclictest31355-21unixbench-2d10:55:373
12676994725,9cyclictest13645-21seq11:28:483
12676994718,12cyclictest0-21swapper/308:20:283
1267599474,26cyclictest0-21swapper/210:45:402
1267599474,13cyclictest0-21swapper/210:35:352
1267599474,13cyclictest0-21swapper/210:35:352
12675994732,9cyclictest17798-21unixbench_multi08:10:462
12672994732,10cyclictest0-21swapper/010:55:190
1267299473,16cyclictest0-21swapper/010:53:500
12672994722,7cyclictest13517-21latency_hist08:05:100
12672994720,11cyclictest0-21swapper/009:00:290
12672994710,32cyclictest0-21swapper/012:20:220
62962463,37sleep36026-21munin-node10:05:243
306402462,9sleep20-21swapper/209:46:052
306402462,9sleep20-21swapper/209:46:052
242402462,39sleep00-21swapper/007:17:000
210822462,9sleep30-21swapper/311:45:083
15672462,38sleep30-21swapper/311:02:083
142272462,39sleep00-21swapper/009:13:050
12676994630,9cyclictest17904-21df07:05:293
12676994630,10cyclictest14125-21cpuspeed_turbos11:30:143
12676994630,10cyclictest14125-21cpuspeed_turbos11:30:143
12676994630,10cyclictest1101-21sed08:45:433
1267599468,21cyclictest6021-21taskset11:10:332
1267599465,14cyclictest0-21swapper/209:05:262
1267599465,13cyclictest0-21swapper/209:51:252
1267599465,13cyclictest0-21swapper/207:45:392
1267599464,14cyclictest0-21swapper/212:15:042
1267599464,14cyclictest0-21swapper/211:20:272
1267599464,14cyclictest0-21swapper/209:15:502
1267599464,13cyclictest0-21swapper/211:06:032
1267599464,13cyclictest0-21swapper/211:05:012
1267599464,13cyclictest0-21swapper/210:15:422
1267599464,13cyclictest0-21swapper/209:10:232
1267599464,13cyclictest0-21swapper/208:29:282
12675994632,9cyclictest27899-21grep07:25:352
1267599463,38cyclictest0-21swapper/207:05:482
1267599463,34cyclictest0-21swapper/210:25:342
1267599463,25cyclictest0-21swapper/210:05:292
12675994629,11cyclictest6058-21sed08:55:262
12675994629,11cyclictest6058-21sed08:55:262
12675994627,12cyclictest16059-21grep07:00:322
1267599462,39cyclictest0-21swapper/211:56:432
12675994617,13cyclictest0-21swapper/209:40:212
12675994614,13cyclictest0-21swapper/210:43:262
1267299463,36cyclictest0-21swapper/011:10:160
1267299463,36cyclictest0-21swapper/007:25:180
12672994627,13cyclictest0-21swapper/011:37:460
12672994626,9cyclictest27956-21munin-node08:35:390
12672994626,9cyclictest11610-21munin-node11:25:210
12672994621,11cyclictest0-21swapper/012:00:180
63422452,37sleep30-21swapper/311:11:103
211892452,37sleep00-21swapper/009:26:150
159992452,9sleep20-21swapper/211:33:392
159992452,9sleep20-21swapper/211:33:392
12676994522,9cyclictest1251-21snmpd12:10:033
1267599454,13cyclictest0-21swapper/207:10:572
12675994530,10cyclictest21679-21head08:20:302
1267599453,25cyclictest0-21swapper/209:20:352
12675994527,11cyclictest18692-21cpuspeed_turbos08:15:312
12672994531,10cyclictest0-21swapper/006:55:120
1267299453,38cyclictest0-21swapper/011:24:370
1267299453,36cyclictest0-21swapper/009:05:410
1267299453,35cyclictest0-21swapper/009:15:300
12672994524,11cyclictest0-21swapper/011:40:310
12672994522,9cyclictest14614-21fschecks_time11:30:190
12672994522,9cyclictest14614-21fschecks_time11:30:190
12672994520,11cyclictest0-21swapper/008:05:310
12672994520,11cyclictest0-21swapper/008:05:310
12672994516,9cyclictest31589-21latency_hist07:35:110
12675994426,12cyclictest16941-21tune2fs11:35:192
12675994416,13cyclictest0-21swapper/207:41:052
12675994416,12cyclictest0-21swapper/210:14:042
12675994410,14cyclictest0-21swapper/208:50:322
12675994410,14cyclictest0-21swapper/208:50:322
1267299443,35cyclictest0-21swapper/011:15:160
1267299423,32cyclictest1251-21snmpd10:18:130
1267599383,29cyclictest113850irq/46-eth006:55:122
1267499383,30cyclictest0-21swapper/106:55:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional