-
Notifications
You must be signed in to change notification settings - Fork 35
/
Copy pathcloud-native-enterprise-data-lake.yml
422 lines (421 loc) · 11.2 KB
/
cloud-native-enterprise-data-lake.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
ROSTemplateFormatVersion: '2015-09-01'
Description:
zh-cn: 创建一个包含EMR集群的大数据分析环境，搭配OSS存储与VPC安全配置，支持云上数据处理与存储。
en: Establish a big data analysis environment featuring an EMR (Elastic Map Reduce)
cluster, integrated with OSS (Object Storage Service) for storage and fortified
with VPC (Virtual Private Cloud) security configurations, facilitating cloud-based
data processing and storage capabilities.
Parameters:
CommonName:
Type: String
Default: data-analysis
ZoneId:
Type: String
Label:
en: Availability Zone
zh-cn: 可用区
AssociationProperty: ZoneId
EcsInstanceType:
Type: String
Label:
en: ECS instance type.
zh-cn: ECS实例规格
AssociationProperty: 'ALIYUN::ECS::Instance::InstanceType'
AssociationPropertyMetadata:
ZoneId: '${ZoneId}'
SystemDiskCategory:
Type: String
Label:
en: System Disk Type
zh-cn: 系统盘类型
Description:
en: >-
<font color='blue'><b>Optional values:</b></font><br>[cloud_efficiency:
<font color='green'>Efficient Cloud Disk</font>]<br>[cloud_ssd: <font
color='green'>SSD Cloud Disk</font>]<br>[cloud_essd: <font
color='green'>ESSD Cloud Disk</font>]<br>[cloud: <font
color='green'>Cloud Disk</font>]<br>[ephemeral_ssd: <font
color='green'>Local SSD Cloud Disk</font>]
zh-cn: >-
<font color='blue'><b>可选值：</b></font><br>[cloud_efficiency: <font
color='green'>高效云盘</font>]<br>[cloud_ssd: <font
color='green'>SSD云盘</font>]<br>[cloud_essd: <font
color='green'>ESSD云盘</font>]<br>[cloud: <font
color='green'>普通云盘</font>]<br>[ephemeral_ssd: <font
color='green'>本地SSD盘</font>]
AssociationProperty: 'ALIYUN::ECS::Disk::SystemDiskCategory'
AssociationPropertyMetadata:
LocaleKey: DiskCategory
InstanceType: '${EcsInstanceType}'
ZoneId: '${ZoneId}'
Default: cloud_essd
MasterRootPassword:
Type: String
Label:
en: ECS Master Node Instance Password
zh-cn: ECS Master节点实例密码
Description:
en: >-
Server login password, Length 8-30, must contain three(Capital letters,
lowercase letters, numbers, ()`~!@#$%^&*_-+=|{}[]:;'<>,.?/ Special
symbol in).
zh-cn: >-
服务器登录密码,长度8-30，必须包含三项（大写字母、小写字母、数字、 ()`~!@#$%^&*_-+=|{}[]:;'<>,.?/
中的特殊符号）。
ConstraintDescription:
en: >-
Length 8-30, must contain three(Capital letters, lowercase letters,
numbers, ()`~!@#$%^&*_-+=|{}[]:;'<>,.?/ Special symbol in).
zh-cn: '长度8-30，必须包含三项（大写字母、小写字母、数字、 ()`~!@#$%^&*_-+=|{}[]:;''<>,.?/ 中的特殊符号）。'
AllowedPattern: '[0-9A-Za-z\_\-\&:;''<>,=%`~!@#\(\)\$\^\*\+\|\{\}\[\]\.\?\/]+$'
MinLength: 8
MaxLength: 30
NoEcho: true
BucketName:
ConstraintDescription:
zh-cn: 3 到 63 个字符，不以连字符 (-) 开头和结尾，可以包含小写字母、数字和连字符 (-)
en: >-
3 to 63 characters, not beginning and ending with a hyphen (-), can
contain lowercase letters, Numbers and hyphens (-)
Description:
zh-cn: >-
3到63个字符，不以连字符(-)开头和结尾，可以包含小写字母、数字和连字符(-)；<br>备注：<font
color='blue'><b>需要保证整个网络的唯一性，已经存在的无法创建。</b></font>
en: >-
3 to 63 characters, not beginning and ending with a hyphen (-), can
contain lowercase letters, Numbers and hyphens (-);<br>Note: <font
color='blue'><b>need whole network uniqueness, already existing can not
be created.</b></font>
Label:
zh-cn: 存储空间名称
en: Bucket Name
AssociationProperty: AutoCompleteInput
AssociationPropertyMetadata:
Length: 3
Prefix: example-bucket-data-lake
CharacterClasses:
- Class: lowercase
min: 1
AllowedPattern: '^[a-z0-9]+[a-z0-9\-]*[a-z0-9]+$'
MaxLength: 63
MinLength: 3
Type: String
WorkspaceName:
Label:
zh-cn: 工作空间名称。
en: The name of the dataset.
Description:
zh-cn: >-
工作空间名称，长度为3~23 个字符，可以包含字母、下划线或数字。必须以大小字母开头。<br>备注：<font
color='blue'><b>需要保证当前地域内唯一，已经存在的无法创建。</b></font>
en: >-
The workspace name contains 3 to 23 characters and can contain letters,
underscores (_), or digits. Must start with a small or large letter.
<br> Note:<font color='blue'><b> Must be unique in the current locale.
An existing one cannot be created. </b></font>
AssociationProperty: AutoCompleteInput
AssociationPropertyMetadata:
Length: 2
Prefix: hymenoptera_workspace
CharacterClasses:
- Class: lowercase
min: 1
MaxLength: 23
MinLength: 3
Type: String
Resources:
Vpc:
Type: 'ALIYUN::ECS::VPC'
Properties:
CidrBlock: 192.168.0.0/16
VpcName:
'Fn::Sub': 'vpc-created-by-ros-${CommonName}'
VSwitch:
Type: 'ALIYUN::ECS::VSwitch'
Properties:
ZoneId:
Ref: ZoneId
VpcId:
Ref: Vpc
CidrBlock: 192.168.0.0/24
VSwitchName:
'Fn::Sub': 'vsw-created-by-ros-${CommonName}'
SecurityGroup:
Type: 'ALIYUN::ECS::SecurityGroup'
Properties:
VpcId:
Ref: Vpc
SecurityGroupIngress_80:
Type: 'ALIYUN::ECS::SecurityGroupIngress'
Properties:
SecurityGroupId:
Ref: SecurityGroup
SourceCidrIp: 0.0.0.0/0
IpProtocol: tcp
NicType: intranet
PortRange: 80/80
SecurityGroupIngress_3306:
Type: 'ALIYUN::ECS::SecurityGroupIngress'
Properties:
SecurityGroupId:
Ref: SecurityGroup
SourceCidrIp: 0.0.0.0/0
IpProtocol: tcp
NicType: intranet
PortRange: 3306/3306
SecurityGroupIngress_443:
Type: 'ALIYUN::ECS::SecurityGroupIngress'
Properties:
SecurityGroupId:
Ref: SecurityGroup
SourceCidrIp: 0.0.0.0/0
IpProtocol: tcp
NicType: intranet
PortRange: 443/443
Cluster:
Type: 'ALIYUN::EMR::Cluster2'
Properties:
ApplicationConfigs:
- ConfigFileName: hivemetastore-site.xml
ApplicationName: HIVE
ConfigItemKey: hive.metastore.type
ConfigScope: CLUSTER
ConfigItemValue: DLF
- ConfigFileName: hivemetastore-site.xml
ApplicationName: SPARK2
ConfigItemKey: hive-site.xml
ConfigScope: CLUSTER
ConfigItemValue: DLF
Applications:
- ApplicationName: HADOOP-COMMON
- ApplicationName: HIVE
- ApplicationName: TEZ
- ApplicationName: YARN
- ApplicationName: HDFS
- ApplicationName: SPARK2
PaymentType: PayAsYouGo
ClusterType: DATALAKE
NodeGroups:
- NodeCount: 1
NodeGroupName: emr-master
DataDisks:
- Category: cloud_essd
Size: 80
Count: 1
VSwitchIds:
- Ref: VSwitch
SystemDisk:
Category:
Ref: SystemDiskCategory
Size: 80
NodeGroupType: MASTER
InstanceTypes:
- Ref: EcsInstanceType
- NodeCount: 2
NodeGroupName: emr-core
DataDisks:
- Category: cloud_essd
Size: 80
Count: 4
VSwitchIds:
- Ref: VSwitch
SystemDisk:
Category:
Ref: SystemDiskCategory
Size: 80
NodeGroupType: CORE
InstanceTypes:
- Ref: EcsInstanceType
ReleaseVersion: EMR-5.13.0
NodeAttributes:
VpcId:
Ref: Vpc
ZoneId:
Ref: ZoneId
SecurityGroupId:
Ref: SecurityGroup
MasterRootPassword:
Ref: MasterRootPassword
RamRole: AliyunECSInstanceForEMRRole
ClusterName:
'Fn::Sub': 'Emr-DataLake-created-by-ros-${CommonName}'
Bucket:
Type: 'ALIYUN::OSS::Bucket'
Properties:
BucketName:
Ref: BucketName
DeletionForce: true
Workspace:
Type: 'ALIYUN::PAI::Workspace'
Properties:
EnvTypes:
- prod
- dev
Description: Workspace
WorkspaceName:
Ref: WorkspaceName
Metadata:
'ALIYUN::ROS::Interface':
ParameterGroups:
- Parameters:
- ZoneId
- EcsInstanceType
- SystemDiskCategory
- MasterRootPassword
Label:
default: EMR
- Parameters:
- BucketName
Label:
default: OSS
- Parameters:
- WorkspaceName
Label:
default: PAI
TemplateTags:
- 'acs:technical-solution:data-analysis:云原生企业级数据湖-tech_solu_55'
Hidden:
- CommonName
ALIYUN::ROS::Composer:
5219acf4:
Rect:
- 691
- 498
- 27
- 212
- 1
- 0
ResT: Composer::ROSParameter::AlibabaCloud
fa1b0685:
Parent: 5219acf4
Rect:
- 600
- 443
- 47
- 251
- 2
- 0
ResT: Composer::ROSParameter::Region
84b4885e:
Res:
- Vpc
Parent: fa1b0685
Rect:
- 449
- 378
- 64
- 298
- 3
- 0
da095e36:
Res:
- Bucket
Parent: fa1b0685
Rect:
- 40
- 40
- 560
- 438
- 3
- 0
935fc597:
Res:
- Workspace
Parent: fa1b0685
Rect:
- 40
- 40
- 560
- 601
- 3
- 0
dc4ed5fe:
Res:
- VSwitch
Parent: 84b4885e
Rect:
- 319
- 190
- 81
- 342
- 4
- 0
c2376e7e:
Res:
- SecurityGroupIngress_80
Parent: 84b4885e
Rect:
- 40
- 40
- 122
- 601
- 4
- 0
d1c3883d:
Res:
- SecurityGroupIngress_3306
Parent: 84b4885e
Rect:
- 40
- 40
- 263
- 601
- 4
- 0
a4133e95:
Res:
- SecurityGroupIngress_443
Parent: 84b4885e
Rect:
- 40
- 40
- 405
- 601
- 4
- 0
812cd60a:
Res:
- Cluster
Parent: dc4ed5fe
Rect:
- 40
- 40
- 122
- 438
- 5
- 0
f6a58d5c:
Parent: 84b4885e
Edge:
- c2376e7e
- 9e96a37b
Line: 0:0:0:gray:0
496514a7:
Parent: 84b4885e
Edge:
- d1c3883d
- 9e96a37b
Line: 0:0:0:gray:0
2f02f506:
Parent: 84b4885e
Edge:
- a4133e95
- 9e96a37b
Line: 0:0:0:gray:0
55aceb74:
Parent: 84b4885e
Edge:
- 812cd60a
- 9e96a37b
Line: 0:0:0:gray:0
9e96a37b:
Res:
- SecurityGroup
Parent: 84b4885e
Rect:
- 94
- 70
- 236
- 423
- 11
- 0