spark-hadoop-distributed-env-3-ecs.yml

ROSTemplateFormatVersion: '2015-09-01'
Description:
  zh-cn: 在ROS模版中，配置了3台ECS实例（1主2从）以CentOS-7为基础，自动搭建无密码SSH访问，并安装Java环境、Hadoop 2.7.7与Spark分布式计算环境，适用于大数据处理场景，依赖资源自动创建于专有网络VPC内，考虑到了防火墙调整与依赖包下载速度对部署时长的影响。
  en: In the ROS template, three ECS instances (1 master and 2 slaves) are configured
    based on CentOS-7, automatically establishing passwordless SSH access, with Java
    environment, Hadoop 2.7.7, and Spark distributed computing environment installed,
    tailored for big data processing scenarios. The required resources are created
    within a dedicated VPC, taking into account firewall adjustments and the impact
    of dependency package download speeds on deployment duration.
Parameters:
  VpcName:
    Type: String
    Label:
      en: VPC Name
      zh-cn: VPC 名称
    Description:
      en: Name of VPC, [2, 128] English or Chinese characters, must start with a letter
        or Chinese in size, can contain numbers, '_' or '.', '-'.
      zh-cn: VPC 名称，[2，128]英文或中文字符，必须以字母或中文开头，可以包含数字、下划线或“.”、“-”。
    ConstraintDescription:
      en: '[2, 128] English or Chinese characters'
      zh-cn: '[2, 128] 英文或中文字符'
    Default: DefaultVPCTest
    MinLength: 2
    MaxLength: 128
  VpcCidrBlock:
    Type: String
    Label:
      en: VPC Cidrblock
      zh-cn: VPC网段
    Description:
      en: 'The IP address range of the VPC in the CIDR block form. You can use the
        following IP address ranges and their subnets: 10.0.0.0/8 172.16.0.0/12  192.168.0.0/16'
      zh-cn: 专有网络的网段，可选值：10.0.0.0/8，172.16.0.0/12，192.168.0.0/16。
    Default: 192.168.0.0/16
    AllowedValues:
    - 192.168.0.0/16
    - 172.16.0.0/12
    - 10.0.0.0/8
  VSwitchCidrBlock:
    Type: String
    Label:
      en: VSwitch CidrBlock
      zh-cn: VSwitch网段
    Description:
      en: CIDR Block of created VSwitch,It must belong to itself VPC CIDR Block.
      zh-cn: 创建的虚拟交换机的子网，它必须属于自己的VPC网段内
    Default: 192.168.1.0/24
  ZoneId:
    Type: String
    Label:
      en: Available Zone ID
      zh-cn: 可用区ID
    Description:
      en: ECS Available Zone ID,<br><b>note： <font color='blue'>Before selecting,
        please confirm that the Availability Zone supports the specification of creating
        ECS resources</font></b>
      zh-cn: ECS可用区ID,<br><b>注： <font color='blue'>选择可用区前请确认该可用区是否支持创建ECS资源的规格</font></b>
    AssociationProperty: ALIYUN::ECS::Instance:ZoneId
  InstanceName:
    Type: String
    Label:
      en: Instance Name
      zh-cn: 实例名称
    Description:
      en: The name of ECS instance,[2, 128] English or Chinese characters, must start
        with a letter or Chinese in size, can contain numbers, '_' or '.', '-'.
      zh-cn: ECS实例名称，[2，128]英文或中文字符，必须以字母或中文开头，可以包含数字、下划线或“.”、“-”。
    Default: DefaultECSInstance
  InstanceType:
    Type: String
    Label:
      en: Instance Type
      zh-cn: 实例规格
    Description:
      en: <font color='blue'><b>1.Before selecting the model please confirm that the
        current available zone under the model is in stock, some models need to be
        reported in advance</b></font><br><font color='blue'><b>2.List of optional
        models</font><br></b></font>[ecs.c5.large <font color='green'>2vCPU 4GiB Intranet
        bandwidth1Gbps In-grid sending and receiving packages30MillionPPS</font>]<br></b>[ecs.c5.xlarge
        <font color='green'>4vCPU 8GiB Intranet bandwidth1.5Gbps In-grid sending and
        receiving packages50MillionPPS</font>]<br></b>[ecs.c5.2xlarge <font color='green'>8vCPU
        16GiB Intranet bandwidth2.5Gbps In-grid sending and receiving packages80MillionPPS</font>]
      zh-cn: <font color='blue'><b>1.选择机型前请先确认当前可用区下该机型是否有货，部分机型需要提前报备</b></font><br><font
        color='blue'><b>2.可选机型列表</font><br></b></font>[ecs.c5.large <font color='green'>2vCPU
        4GiB 内网带宽1Gbps 内网收发包30万PPS</font>]<br></b>[ecs.c5.xlarge <font color='green'>4vCPU
        8GiB 内网带宽1.5Gbps 内网收发包50万PPS</font>]<br></b>[ecs.c5.2xlarge <font color='green'>8vCPU
        16GiB 内网带宽2.5Gbps 内网收发包80万PPS</font>]
    AssociationProperty: ALIYUN::ECS::Instance::InstanceType
    AssociationPropertyMetadata:
      ZoneId: ZoneId
  SystemDiskCategory:
    Type: String
    Label:
      en: System Disk Category
      zh-cn: 系统盘类型
    AssociationProperty: ALIYUN::ECS::Disk::SystemDiskCategory
    AssociationPropertyMetadata:
      InstanceType: InstanceType
      ZoneId: ZoneId
  ImageId:
    Type: String
    Label:
      en: Image ID
      zh-cn: 镜像ID
    Description:
      en: 'Instance Image ID. see detail: <b><a href=''https://www.alibabacloud.com/help/doc-detail/112977.html''
        target=''_blank''><font color=''blue''>Find the mirror</font></a></b>'
      zh-cn: 实例镜像，详见：<b><a href='https://help.aliyun.com/document_detail/112977.html'
        target='_blank'><font color='blue'>查找镜像</font></a></b>
    AssociationProperty: ALIYUN::ECS::Image::ImageId
    AssociationPropertyMetadata:
      InstanceType: ${InstanceType}
      SupportedImageOwnerAlias:
        - system
        - self
        - others
    Default: centos_7_04_64_20G_alibase_201701015.vhd
  InstancePassword:
    Type: String
    Label:
      en: Instance Password
      zh-cn: 实例密码
    Description:
      en: The 8-30 long login password of instance, consists of the uppercase, lowercase
        letter and number. <br> special characters include()`~!@#$%^&*_-+=|{}[]:;'<>,.?/
      zh-cn: 长度8-30，必须包含大写字母、小写字母、数字、特殊符号三个；<br>特殊字符包括：()`~!@#$%^&*_-+=|{}[]:;'<>,.?/
    ConstraintDescription:
      en: 'Length 8-30, must contain upper case letters, lower case letters, Numbers,
        special symbols three; special characters include: ()`~!@#$%^&*_-+=|{}[]:;''<>,.?/'
      zh-cn: 长度8-30，必须包含大写字母、小写字母、数字、特殊符号三种；特殊字符包括：()`~!@#$%^&*_-+=|{}[]:;' <>,.?/
    AllowedPattern: '[0-9A-Za-z\_\-&:;''<>,=%`~!@#\(\)\$\^\*\+\|\{\}\[\]\.\?\/]+$'
    MinLength: '8'
    MaxLength: '30'
    NoEcho: true
  MasterName:
    Type: String
    Label:
      en: Master Name
      zh-cn: 主机名
    Description:
      en: Master hostname, [2, 128] English or Chinese characters, must start with
        a letter or Chinese in size, can contain numbers, '_' or '.', '-'.
      zh-cn: 主机名，[2，128]英文或中文字符，必须以字母或中文开头，可以包含数字、下划线或“.”、“-”。
    Default: master.hadoop
  HadoopHome:
    Type: String
    Label:
      en: Hadoop Home
      zh-cn: Hadoop主目录
    Description:
      en: HADOOP_HOME directory should be new.
      zh-cn: HADOOP_HOME 主目录必须是新的
    Default: /usr/local/hadoop/hadoop-2.7.7
  Slave1Name:
    Type: String
    Label:
      en: 'Slave 1 Name '
      zh-cn: 从机1名称
    Description:
      en: Slave 1 hostname, [2, 128] English or Chinese characters, must start with
        a letter or Chinese in size, can contain numbers, '_' or '.', '-'.
      zh-cn: 从机1名称，[2，128]英文或中文字符，必须以字母或中文开头，可以包含数字、下划线或“.”、“-”。
    Default: slave1.hadoop
  Slave2Name:
    Type: String
    Label:
      en: 'Slave 2 Name '
      zh-cn: 从机2名称
    Description:
      en: Slave 2 hostname, [2, 128] English or Chinese characters, must start with
        a letter or Chinese in size, can contain numbers, '_' or '.', '-'.
      zh-cn: 从机2名称，[2，128]英文或中文字符，必须以字母或中文开头，可以包含数字、下划线或“.”、“-”。
    Default: slave2.hadoop
  HadoopUrl:
    Type: String
    Label:
      en: Hadoop Url
      zh-cn: Hadoop网址
    Description:
      en: The official download path of .tar.gz
      zh-cn: .tar.gz的官方下载路径
    Default: https://ros-template-resources.oss-cn-beijing.aliyuncs.com/Hadoop/hadoop-2.7.7.tar.gz
  ScalaHome:
    Type: String
    Label:
      en: Scala Home
      zh-cn: Scala主目录
    Description:
      en: SCALA_HOME directory should be new.
      zh-cn: SCALA_HOME 主目录必须是新的
    Default: /usr/local/scala/scala-2.12.1
  ScalaUrl:
    Type: String
    Label:
      en: Scala Url
      zh-cn: Scala网址
    Description:
      en: The official download path of .tar.gz
      zh-cn: .tar.gz的官方下载路径
    Default: https://downloads.lightbend.com/scala/2.12.1/scala-2.12.1.tgz
  SparkHome:
    Type: String
    Label:
      en: Spark Home
      zh-cn: Spark主目录
    Description:
      en: SPARK_HOME directory should be new.
      zh-cn: SPARK_HOME 主目录必须是新的
    Default: /usr/local/hadoop/spark-2.1.0-bin-hadoop2.7
  SparkUrl:
    Type: String
    Label:
      en: Spark Url
      zh-cn: Spark网址
    Description:
      en: The official download path of -bin-hadoopx.x.tgz
      zh-cn: -bin-hadoopx.x.tgz的官方下载路径
    Default: http://d3kbcqa49mib13.cloudfront.net/spark-2.1.0-bin-hadoop2.7.tgz
Resources:
  EipSNat:
    Type: ALIYUN::VPC::EIP
    Properties:
      Bandwidth: 5
      InternetChargeType: PayByTraffic
  Vpc:
    Type: ALIYUN::ECS::VPC
    Properties:
      CidrBlock:
        Ref: VpcCidrBlock
      VpcName:
        Ref: VpcName
  VSwitch:
    Type: ALIYUN::ECS::VSwitch
    Properties:
      ZoneId:
        Ref: ZoneId
      VpcId:
        Ref: Vpc
      CidrBlock:
        Ref: VSwitchCidrBlock
  NatGateway:
    Type: ALIYUN::VPC::NatGateway
    Properties:
      VpcId:
        Fn::GetAtt:
        - Vpc
        - VpcId
      VSwitchId:
        Ref: VSwitch
      Description: My NAT Gateway
      NatGatewayName: nat_gateway_1
    DependsOn: VSwitch
  EIpSnatAssociation:
    Type: ALIYUN::VPC::EIPAssociation
    Properties:
      InstanceId:
        Ref: NatGateway
      AllocationId:
        Ref: EipSNat
    DependsOn: NatGateway
  EipForward:
    Type: ALIYUN::VPC::EIP
    Properties:
      Bandwidth: 5
      InternetChargeType: PayByTraffic
  EIpForwardAssociation:
    Type: ALIYUN::VPC::EIPAssociation
    Properties:
      InstanceId:
        Ref: NatGateway
      AllocationId:
        Ref: EipForward
    DependsOn: EIpSnatAssociation
  SNatEntry:
    Type: ALIYUN::ECS::SNatEntry
    Properties:
      SNatIp:
        Fn::GetAtt:
        - EipSNat
        - EipAddress
      SNatTableId:
        Fn::GetAtt:
        - NatGateway
        - SNatTableId
      SourceVSwitchId:
        Fn::GetAtt:
        - VSwitch
        - VSwitchId
    DependsOn:
    - EIpSnatAssociation
  SecurityGroup:
    Type: ALIYUN::ECS::SecurityGroup
    Properties:
      VpcId:
        Ref: Vpc
      SecurityGroupEgress:
      - DestCidrIp: 0.0.0.0/0
        IpProtocol: all
        NicType: internet
        PortRange: -1/-1
        Priority: 1
      SecurityGroupIngress:
      - IpProtocol: all
        NicType: internet
        PortRange: -1/-1
        Priority: 1
        SourceCidrIp: 0.0.0.0/0
  Slave1:
    Type: ALIYUN::ECS::Instance
    Properties:
      VpcId:
        Fn::GetAtt:
        - Vpc
        - VpcId
      VSwitchId:
        Ref: VSwitch
      SecurityGroupId:
        Ref: SecurityGroup
      ImageId:
        Ref: ImageId
      InstanceName:
        Ref: InstanceName
      InstanceType:
        Ref: InstanceType
      IoOptimized: optimized
      Password:
        Ref: InstancePassword
      SystemDiskCategory:
        Ref: SystemDiskCategory
      UserData:
        Fn::Replace:
        - ros-notify:
            Fn::GetAtt:
            - Slave1ConditionHandle
            - CurlCli
        - Fn::Join:
          - ''
          - - "#!/bin/bash \n"
            - Slave1Name=
            - Ref: Slave1Name
            - '

              '
            - "hostnamectl --static set-hostname \"$Slave1Name\" \n"
            - "export HOME=/root \n"
            - "export HOSTNAME=`hostname` \n"
            - "cd /root \n"
            - "rm -rf .ssh \n"
            - "ssh-keygen -t rsa -P '' -f '/root/.ssh/id_rsa' \n"
            - "cd /root/.ssh \n"
            - "rm -f * \n"
            - "echo '-----BEGIN RSA PRIVATE KEY-----' > id_rsa \n"
            - "echo 'MIIEowIBAAKCAQEAzfQ/QHwWB1njU9+Wu3RYi9g+g5rydSpAE0klefTJuZjtcaic'\
              \ >> id_rsa \n"
            - "echo 'SCeBN5avih8UToZ148+Ef2YzOtoosqluZpoYLCPSpAqr8pmviBJIU3vfu9mnDG9L'\
              \ >> id_rsa \n"
            - "echo 'oevT6K8w3wCBRCmqu+vc0Tpju/EeCuYK3+8w7e2I6F3+zIYzhhX3qmkocje7ACnV'\
              \ >> id_rsa \n"
            - "echo 'yh2DB/2m3sogTMc0RT+5y3kJAnC6TOIlGsYjOOkPbEF3Ifn1o4ZjOFOmQlcRJer4'\
              \ >> id_rsa \n"
            - "echo '1E6rTdXAxS2uDNMFBCf9Xyx7O9J+ELTCAXc4h7AE6WLdQb5Apzv4t1KswCAtRenP'\
              \ >> id_rsa \n"
            - "echo '1xGcUYY8I/JUT2VvBWtQJennrk9jrPZUFDcmcwIDAQABAoIBAAwzDZQaRYvF7UtI'\
              \ >> id_rsa \n"
            - "echo 'kTslVyFhe8J76SS7jfQWfxvMPi66OkZjQG6duG+8g0VhNei42j7WSfjp6trvlT2P'\
              \ >> id_rsa \n"
            - "echo '/7QgKJJkxNNmtmy2Ycljm9kmG0ibSePYq9g5ieHcjr6G3yFUfoKHJBtYpBO74pWu'\
              \ >> id_rsa \n"
            - "echo 'rrI5DuLpERUCjFc9E8w7fOIhPH4XZ6wk/EmPxHTgxZk+aMvqptyPSbUyiUOvCiZf'\
              \ >> id_rsa \n"
            - "echo 'MD0ircs9vgtslMVDlz9m6CoiNz6B3Yf6eLRoGGMiGnsQzZHIfnHCMX8i65Jc+TvQ'\
              \ >> id_rsa \n"
            - "echo 'fLopIHzwBwI55xpcOIBRgYiEAQJhLsSNSFugoxMcwe9RalTGGS21HOQu4b3ZylKM'\
              \ >> id_rsa \n"
            - "echo '8ofEVKECgYEA/Y4MzN04wAsM1yNuHN+9sdiVLG28LWH3dgpcXqa9gyNsWs9Gf8uf'\
              \ >> id_rsa \n"
            - "echo 'qbuvQGeKDRXCW93wO1jO+pCYVrkyY3l+KhBKIqmkJT5gFsa8dBUvEBLALiHNg3+o'\
              \ >> id_rsa \n"
            - "echo 'jR2Vqsemk8kMZA8zfJ3FKcMb1pt4S2GqepqsdC3DgzIIsxufCh7jSzsCgYEAz/Cv'\
              \ >> id_rsa \n"
            - "echo 'Z7gAdSFC8q6QxFxSyhfZwGA6QW6ZU7rZv9ZdGySvZg+vHbpNVmQ0BAQzJui3Qs9u'\
              \ >> id_rsa \n"
            - "echo 'XQMpYafXWzKsPzG6ZWvYXTF7fuxlovvG8Qd2A2QnGtGMB9YtQMHVqbsUDwxMjiTn'\
              \ >> id_rsa \n"
            - "echo 'VBZILkDf+WCwQ98P5UMoumI0goIcNZ+AXhcqrikCgYBkSwLvKfYfqH9Mvfv5Odsr'\
              \ >> id_rsa \n"
            - "echo '9NKUv1c20FB1BYYh/mxp6eIbTW/CbwXZup6IqCvoHxpBAlna77b3T6iibSDsTgtE'\
              \ >> id_rsa \n"
            - "echo 'kirw6Q8/mBukBrpWZGa4QeJ4nPBQuncuUmx4H/7Y6CaZkZW5DiMF8OIbEmYT0y7+'\
              \ >> id_rsa \n"
            - "echo 'zh222r9CLtFYH23aL/uSLwKBgQCS1xyG2eE41aw5RBznDWtJW15iA5If8sJD5ocu'\
              \ >> id_rsa \n"
            - "echo 'eWp2aImUQS8ghxdmEozI6U5WA7CmdWUyObFXTPc/Z6FLXwqJ5IZ+CRt0neuIFNSA'\
              \ >> id_rsa \n"
            - "echo 'EQy9iFQ1FBUW06BRQpBns7yOg9jr6BOTxchjIV0I9caDp1nKRIrWU9NQ9iCFnYVA'\
              \ >> id_rsa \n"
            - "echo '7IsvQQKBgFxgF7UOhwaiMb/ATSuhm2v9kVvRPEO9umdo7YJ9I4L09lYbAtpcnusQ'\
              \ >> id_rsa \n"
            - "echo 'fIROYL25VeEMgYcQyInc3Fm/sgJdbXQnUy+3QbbCcBCmcLj27LPQyxuu7p9hbPPT'\
              \ >> id_rsa \n"
            - "echo 'Mxx37OmYvOkSVTQz0T9HfDGvOJgt4t4cXD4T/7ewk62p6jdpSQrt' >> id_rsa\
              \ \n"
            - "echo '-----END RSA PRIVATE KEY-----' >> id_rsa \n"
            - "echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN9D9AfBYHWeNT35a7dFiL2D6DmvJ1KkATSSV59Mm5mO1xqJxIJ4E3lq+KHxROhnXjz4R/ZjM62iiyqW5mmhgsI9KkCqvyma+IEkhTe9+72acMb0uh69PorzDfAIFEKaq769zROmO78R4K5grf7zDt7YjoXf7MhjOGFfeqaShyN7sAKdXKHYMH/abeyiBMxzRFP7nLeQkCcLpM4iUaxiM46Q9sQXch+fWjhmM4U6ZCVxEl6vjUTqtN1cDFLa4M0wUEJ/1fLHs70n4QtMIBdziHsATpYt1BvkCnO/i3UqzAIC1F6c/XEZxRhjwj8lRPZW8Fa1Al6eeuT2Os9lQUNyZz\
              \ root@iZ2zee53wf4ndvajz30cdvZ' > id_rsa.pub \n"
            - "cp id_rsa.pub authorized_keys \n"
            - "chmod 600 authorized_keys \n"
            - "chmod 600 id_rsa \n"
            - "sed -i 's/#   StrictHostKeyChecking ask/StrictHostKeyChecking no/'\
              \ /etc/ssh/ssh_config \n"
            - "sed -i 's/GSSAPIAuthentication yes/GSSAPIAuthentication no/' /etc/ssh/ssh_config\
              \ \n"
            - "service sshd restart \n"
            - 'ros-notify -d ''{"data" : "Install Spark."}''

              '
  Slave1ConditionHandle:
    Type: ALIYUN::ROS::WaitConditionHandle
    Properties: {}
  Slave1WaitCondition:
    Type: ALIYUN::ROS::WaitCondition
    Properties:
      Count: 1
      Handle:
        Ref: Slave1ConditionHandle
      Timeout: 1800
    DependsOn: Slave1
  Slave2:
    Type: ALIYUN::ECS::Instance
    Properties:
      VpcId:
        Fn::GetAtt:
        - Vpc
        - VpcId
      VSwitchId:
        Ref: VSwitch
      SecurityGroupId:
        Ref: SecurityGroup
      ImageId:
        Ref: ImageId
      InstanceName:
        Ref: InstanceName
      InstanceType:
        Ref: InstanceType
      IoOptimized: optimized
      Password:
        Ref: InstancePassword
      SystemDiskCategory:
        Ref: SystemDiskCategory
      UserData:
        Fn::Replace:
        - ros-notify:
            Fn::GetAtt:
            - Slave2ConditionHandle
            - CurlCli
        - Fn::Join:
          - ''
          - - "#!/bin/bash \n"
            - Slave2Name=
            - Ref: Slave2Name
            - '

              '
            - "hostnamectl --static set-hostname \"$Slave2Name\" \n"
            - "export HOME=/root \n"
            - "export HOSTNAME=`hostname` \n"
            - "cd /root \n"
            - "rm -rf .ssh \n"
            - "ssh-keygen -t rsa -P '' -f '/root/.ssh/id_rsa' \n"
            - "cd /root/.ssh \n"
            - "rm -f * \n"
            - "echo '-----BEGIN RSA PRIVATE KEY-----' > id_rsa \n"
            - "echo 'MIIEowIBAAKCAQEAzfQ/QHwWB1njU9+Wu3RYi9g+g5rydSpAE0klefTJuZjtcaic'\
              \ >> id_rsa \n"
            - "echo 'SCeBN5avih8UToZ148+Ef2YzOtoosqluZpoYLCPSpAqr8pmviBJIU3vfu9mnDG9L'\
              \ >> id_rsa \n"
            - "echo 'oevT6K8w3wCBRCmqu+vc0Tpju/EeCuYK3+8w7e2I6F3+zIYzhhX3qmkocje7ACnV'\
              \ >> id_rsa \n"
            - "echo 'yh2DB/2m3sogTMc0RT+5y3kJAnC6TOIlGsYjOOkPbEF3Ifn1o4ZjOFOmQlcRJer4'\
              \ >> id_rsa \n"
            - "echo '1E6rTdXAxS2uDNMFBCf9Xyx7O9J+ELTCAXc4h7AE6WLdQb5Apzv4t1KswCAtRenP'\
              \ >> id_rsa \n"
            - "echo '1xGcUYY8I/JUT2VvBWtQJennrk9jrPZUFDcmcwIDAQABAoIBAAwzDZQaRYvF7UtI'\
              \ >> id_rsa \n"
            - "echo 'kTslVyFhe8J76SS7jfQWfxvMPi66OkZjQG6duG+8g0VhNei42j7WSfjp6trvlT2P'\
              \ >> id_rsa \n"
            - "echo '/7QgKJJkxNNmtmy2Ycljm9kmG0ibSePYq9g5ieHcjr6G3yFUfoKHJBtYpBO74pWu'\
              \ >> id_rsa \n"
            - "echo 'rrI5DuLpERUCjFc9E8w7fOIhPH4XZ6wk/EmPxHTgxZk+aMvqptyPSbUyiUOvCiZf'\
              \ >> id_rsa \n"
            - "echo 'MD0ircs9vgtslMVDlz9m6CoiNz6B3Yf6eLRoGGMiGnsQzZHIfnHCMX8i65Jc+TvQ'\
              \ >> id_rsa \n"
            - "echo 'fLopIHzwBwI55xpcOIBRgYiEAQJhLsSNSFugoxMcwe9RalTGGS21HOQu4b3ZylKM'\
              \ >> id_rsa \n"
            - "echo '8ofEVKECgYEA/Y4MzN04wAsM1yNuHN+9sdiVLG28LWH3dgpcXqa9gyNsWs9Gf8uf'\
              \ >> id_rsa \n"
            - "echo 'qbuvQGeKDRXCW93wO1jO+pCYVrkyY3l+KhBKIqmkJT5gFsa8dBUvEBLALiHNg3+o'\
              \ >> id_rsa \n"
            - "echo 'jR2Vqsemk8kMZA8zfJ3FKcMb1pt4S2GqepqsdC3DgzIIsxufCh7jSzsCgYEAz/Cv'\
              \ >> id_rsa \n"
            - "echo 'Z7gAdSFC8q6QxFxSyhfZwGA6QW6ZU7rZv9ZdGySvZg+vHbpNVmQ0BAQzJui3Qs9u'\
              \ >> id_rsa \n"
            - "echo 'XQMpYafXWzKsPzG6ZWvYXTF7fuxlovvG8Qd2A2QnGtGMB9YtQMHVqbsUDwxMjiTn'\
              \ >> id_rsa \n"
            - "echo 'VBZILkDf+WCwQ98P5UMoumI0goIcNZ+AXhcqrikCgYBkSwLvKfYfqH9Mvfv5Odsr'\
              \ >> id_rsa \n"
            - "echo '9NKUv1c20FB1BYYh/mxp6eIbTW/CbwXZup6IqCvoHxpBAlna77b3T6iibSDsTgtE'\
              \ >> id_rsa \n"
            - "echo 'kirw6Q8/mBukBrpWZGa4QeJ4nPBQuncuUmx4H/7Y6CaZkZW5DiMF8OIbEmYT0y7+'\
              \ >> id_rsa \n"
            - "echo 'zh222r9CLtFYH23aL/uSLwKBgQCS1xyG2eE41aw5RBznDWtJW15iA5If8sJD5ocu'\
              \ >> id_rsa \n"
            - "echo 'eWp2aImUQS8ghxdmEozI6U5WA7CmdWUyObFXTPc/Z6FLXwqJ5IZ+CRt0neuIFNSA'\
              \ >> id_rsa \n"
            - "echo 'EQy9iFQ1FBUW06BRQpBns7yOg9jr6BOTxchjIV0I9caDp1nKRIrWU9NQ9iCFnYVA'\
              \ >> id_rsa \n"
            - "echo '7IsvQQKBgFxgF7UOhwaiMb/ATSuhm2v9kVvRPEO9umdo7YJ9I4L09lYbAtpcnusQ'\
              \ >> id_rsa \n"
            - "echo 'fIROYL25VeEMgYcQyInc3Fm/sgJdbXQnUy+3QbbCcBCmcLj27LPQyxuu7p9hbPPT'\
              \ >> id_rsa \n"
            - "echo 'Mxx37OmYvOkSVTQz0T9HfDGvOJgt4t4cXD4T/7ewk62p6jdpSQrt' >> id_rsa\
              \ \n"
            - "echo '-----END RSA PRIVATE KEY-----' >> id_rsa \n"
            - "echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN9D9AfBYHWeNT35a7dFiL2D6DmvJ1KkATSSV59Mm5mO1xqJxIJ4E3lq+KHxROhnXjz4R/ZjM62iiyqW5mmhgsI9KkCqvyma+IEkhTe9+72acMb0uh69PorzDfAIFEKaq769zROmO78R4K5grf7zDt7YjoXf7MhjOGFfeqaShyN7sAKdXKHYMH/abeyiBMxzRFP7nLeQkCcLpM4iUaxiM46Q9sQXch+fWjhmM4U6ZCVxEl6vjUTqtN1cDFLa4M0wUEJ/1fLHs70n4QtMIBdziHsATpYt1BvkCnO/i3UqzAIC1F6c/XEZxRhjwj8lRPZW8Fa1Al6eeuT2Os9lQUNyZz\
              \ root@iZ2zee53wf4ndvajz30cdvZ' > id_rsa.pub \n"
            - "cp id_rsa.pub authorized_keys \n"
            - "chmod 600 authorized_keys \n"
            - "chmod 600 id_rsa \n"
            - "sed -i 's/#   StrictHostKeyChecking ask/StrictHostKeyChecking no/'\
              \ /etc/ssh/ssh_config \n"
            - "sed -i 's/GSSAPIAuthentication yes/GSSAPIAuthentication no/' /etc/ssh/ssh_config\
              \ \n"
            - "service sshd restart \n"
            - 'ros-notify -d ''{"data" : "Install Spark."}''

              '
  Slave2ConditionHandle:
    Type: ALIYUN::ROS::WaitConditionHandle
    Properties: {}
  Slave2WaitCondition:
    Type: ALIYUN::ROS::WaitCondition
    Properties:
      Count: 1
      Handle:
        Ref: Slave2ConditionHandle
      Timeout: 1800
    DependsOn: Slave2
  Master:
    Type: ALIYUN::ECS::Instance
    Properties:
      VpcId:
        Fn::GetAtt:
        - Vpc
        - VpcId
      VSwitchId:
        Ref: VSwitch
      SecurityGroupId:
        Ref: SecurityGroup
      ImageId:
        Ref: ImageId
      AllocatePublicIP: false
      InstanceName:
        Ref: InstanceName
      InstanceType:
        Ref: InstanceType
      IoOptimized: optimized
      Password:
        Ref: InstancePassword
      SystemDiskCategory:
        Ref: SystemDiskCategory
      UserData:
        Fn::Replace:
        - ros-notify:
            Fn::GetAtt:
            - MasterConditionHandle
            - CurlCli
        - Fn::Join:
          - ''
          - - "#!/bin/bash \n"
            - "# set -e \n"
            - ipaddr_slave1=
            - Fn::GetAtt:
              - Slave1
              - PrivateIp
            - '

              '
            - ipaddr_slave2=
            - Fn::GetAtt:
              - Slave2
              - PrivateIp
            - '

              '
            - HADOOP_HOME=
            - Ref: HadoopHome
            - '

              '
            - SCALA_HOME=
            - Ref: ScalaHome
            - '

              '
            - SPARK_HOME=
            - Ref: SparkHome
            - '

              '
            - HadoopUrl=
            - Ref: HadoopUrl
            - '

              '
            - ScalaUrl=
            - Ref: ScalaUrl
            - '

              '
            - SparkUrl=
            - Ref: SparkUrl
            - '

              '
            - MasterName=
            - Ref: MasterName
            - '

              '
            - Slave1Name=
            - Ref: Slave1Name
            - '

              '
            - Slave2Name=
            - Ref: Slave2Name
            - '

              '
            - " \n"
            - "hostnamectl --static set-hostname \"$MasterName\" \n"
            - "export HOME=/root \n"
            - "export HOSTNAME=`hostname` \n"
            - "ip_addr=`ifconfig eth0 | awk '/inet /{print $2}'` \n"
            - "cd /root \n"
            - "rm -rf .ssh \n"
            - "ssh-keygen -t rsa -P '' -f '/root/.ssh/id_rsa' \n"
            - "cd /root/.ssh \n"
            - "# rm -f * \n"
            - "mv id_rsa bak.id_rsa \n"
            - "mv id_rsa.pub bak.id_rsa.pub \n"
            - "echo '-----BEGIN RSA PRIVATE KEY-----' > id_rsa \n"
            - "echo 'MIIEowIBAAKCAQEAzfQ/QHwWB1njU9+Wu3RYi9g+g5rydSpAE0klefTJuZjtcaic'\
              \ >> id_rsa \n"
            - "echo 'SCeBN5avih8UToZ148+Ef2YzOtoosqluZpoYLCPSpAqr8pmviBJIU3vfu9mnDG9L'\
              \ >> id_rsa \n"
            - "echo 'oevT6K8w3wCBRCmqu+vc0Tpju/EeCuYK3+8w7e2I6F3+zIYzhhX3qmkocje7ACnV'\
              \ >> id_rsa \n"
            - "echo 'yh2DB/2m3sogTMc0RT+5y3kJAnC6TOIlGsYjOOkPbEF3Ifn1o4ZjOFOmQlcRJer4'\
              \ >> id_rsa \n"
            - "echo '1E6rTdXAxS2uDNMFBCf9Xyx7O9J+ELTCAXc4h7AE6WLdQb5Apzv4t1KswCAtRenP'\
              \ >> id_rsa \n"
            - "echo '1xGcUYY8I/JUT2VvBWtQJennrk9jrPZUFDcmcwIDAQABAoIBAAwzDZQaRYvF7UtI'\
              \ >> id_rsa \n"
            - "echo 'kTslVyFhe8J76SS7jfQWfxvMPi66OkZjQG6duG+8g0VhNei42j7WSfjp6trvlT2P'\
              \ >> id_rsa \n"
            - "echo '/7QgKJJkxNNmtmy2Ycljm9kmG0ibSePYq9g5ieHcjr6G3yFUfoKHJBtYpBO74pWu'\
              \ >> id_rsa \n"
            - "echo 'rrI5DuLpERUCjFc9E8w7fOIhPH4XZ6wk/EmPxHTgxZk+aMvqptyPSbUyiUOvCiZf'\
              \ >> id_rsa \n"
            - "echo 'MD0ircs9vgtslMVDlz9m6CoiNz6B3Yf6eLRoGGMiGnsQzZHIfnHCMX8i65Jc+TvQ'\
              \ >> id_rsa \n"
            - "echo 'fLopIHzwBwI55xpcOIBRgYiEAQJhLsSNSFugoxMcwe9RalTGGS21HOQu4b3ZylKM'\
              \ >> id_rsa \n"
            - "echo '8ofEVKECgYEA/Y4MzN04wAsM1yNuHN+9sdiVLG28LWH3dgpcXqa9gyNsWs9Gf8uf'\
              \ >> id_rsa \n"
            - "echo 'qbuvQGeKDRXCW93wO1jO+pCYVrkyY3l+KhBKIqmkJT5gFsa8dBUvEBLALiHNg3+o'\
              \ >> id_rsa \n"
            - "echo 'jR2Vqsemk8kMZA8zfJ3FKcMb1pt4S2GqepqsdC3DgzIIsxufCh7jSzsCgYEAz/Cv'\
              \ >> id_rsa \n"
            - "echo 'Z7gAdSFC8q6QxFxSyhfZwGA6QW6ZU7rZv9ZdGySvZg+vHbpNVmQ0BAQzJui3Qs9u'\
              \ >> id_rsa \n"
            - "echo 'XQMpYafXWzKsPzG6ZWvYXTF7fuxlovvG8Qd2A2QnGtGMB9YtQMHVqbsUDwxMjiTn'\
              \ >> id_rsa \n"
            - "echo 'VBZILkDf+WCwQ98P5UMoumI0goIcNZ+AXhcqrikCgYBkSwLvKfYfqH9Mvfv5Odsr'\
              \ >> id_rsa \n"
            - "echo '9NKUv1c20FB1BYYh/mxp6eIbTW/CbwXZup6IqCvoHxpBAlna77b3T6iibSDsTgtE'\
              \ >> id_rsa \n"
            - "echo 'kirw6Q8/mBukBrpWZGa4QeJ4nPBQuncuUmx4H/7Y6CaZkZW5DiMF8OIbEmYT0y7+'\
              \ >> id_rsa \n"
            - "echo 'zh222r9CLtFYH23aL/uSLwKBgQCS1xyG2eE41aw5RBznDWtJW15iA5If8sJD5ocu'\
              \ >> id_rsa \n"
            - "echo 'eWp2aImUQS8ghxdmEozI6U5WA7CmdWUyObFXTPc/Z6FLXwqJ5IZ+CRt0neuIFNSA'\
              \ >> id_rsa \n"
            - "echo 'EQy9iFQ1FBUW06BRQpBns7yOg9jr6BOTxchjIV0I9caDp1nKRIrWU9NQ9iCFnYVA'\
              \ >> id_rsa \n"
            - "echo '7IsvQQKBgFxgF7UOhwaiMb/ATSuhm2v9kVvRPEO9umdo7YJ9I4L09lYbAtpcnusQ'\
              \ >> id_rsa \n"
            - "echo 'fIROYL25VeEMgYcQyInc3Fm/sgJdbXQnUy+3QbbCcBCmcLj27LPQyxuu7p9hbPPT'\
              \ >> id_rsa \n"
            - "echo 'Mxx37OmYvOkSVTQz0T9HfDGvOJgt4t4cXD4T/7ewk62p6jdpSQrt' >> id_rsa\
              \ \n"
            - "echo '-----END RSA PRIVATE KEY-----' >> id_rsa \n"
            - "echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN9D9AfBYHWeNT35a7dFiL2D6DmvJ1KkATSSV59Mm5mO1xqJxIJ4E3lq+KHxROhnXjz4R/ZjM62iiyqW5mmhgsI9KkCqvyma+IEkhTe9+72acMb0uh69PorzDfAIFEKaq769zROmO78R4K5grf7zDt7YjoXf7MhjOGFfeqaShyN7sAKdXKHYMH/abeyiBMxzRFP7nLeQkCcLpM4iUaxiM46Q9sQXch+fWjhmM4U6ZCVxEl6vjUTqtN1cDFLa4M0wUEJ/1fLHs70n4QtMIBdziHsATpYt1BvkCnO/i3UqzAIC1F6c/XEZxRhjwj8lRPZW8Fa1Al6eeuT2Os9lQUNyZz\
              \ root@iZ2zee53wf4ndvajz30cdvZ' > id_rsa.pub \n"
            - "cp id_rsa.pub authorized_keys \n"
            - "chmod 600 authorized_keys \n chmod 600 id_rsa \n"
            - "sed -i 's/#   StrictHostKeyChecking ask/StrictHostKeyChecking no/'\
              \ /etc/ssh/ssh_config \n"
            - "sed -i 's/GSSAPIAuthentication yes/GSSAPIAuthentication no/' /etc/ssh/ssh_config\
              \ \n"
            - "service sshd restart \n"
            - "scp -r /root/.ssh/bak.*  root@$ipaddr_slave1:/root/.ssh/  \n"
            - "scp -r /root/.ssh/bak.*  root@$ipaddr_slave2:/root/.ssh/  \n"
            - "ssh root@$ipaddr_slave1 \"cd /root/.ssh; rm -f id_rsa; mv bak.id_rsa\
              \ id_rsa; rm -f id_rsa.pub; mv bak.id_rsa.pub id_rsa.pub; rm -f authorized_keys;\
              \ cp id_rsa.pub authorized_keys; chmod 600 authorized_keys; chmod 600\
              \ id_rsa; exit\" \n"
            - "ssh root@$ipaddr_slave2 \"cd /root/.ssh; rm id_rsa; mv bak.id_rsa id_rsa;\
              \ rm id_rsa.pub; mv bak.id_rsa.pub id_rsa.pub; rm authorized_keys; cp\
              \ id_rsa.pub authorized_keys; chmod 600 authorized_keys; chmod 600 id_rsa;\
              \ exit\" \n"
            - "cd /root/.ssh \n rm id_rsa \n mv bak.id_rsa id_rsa \n rm id_rsa.pub\
              \ \n mv bak.id_rsa.pub id_rsa.pub \n rm authorized_keys \n cp id_rsa.pub\
              \ authorized_keys \n chmod 600 authorized_keys \n chmod 600 id_rsa \n\
              \ cd \n"
            - " \n"
            - "echo $ip_addr $MasterName >> /etc/hosts \n"
            - "echo $ipaddr_slave1 $Slave1Name >> /etc/hosts \n"
            - "echo $ipaddr_slave2 $Slave2Name >> /etc/hosts \n"
            - "ssh root@$ipaddr_slave1 \"echo $ip_addr $MasterName >> /etc/hosts;\
              \ exit\" \n"
            - "ssh root@$ipaddr_slave1 \"echo $ipaddr_slave1 $Slave1Name >> /etc/hosts;\
              \ exit\" \n"
            - "ssh root@$ipaddr_slave1 \"echo $ipaddr_slave2 $Slave2Name >> /etc/hosts;\
              \ exit\" \n"
            - "ssh root@$ipaddr_slave2 \"echo $ip_addr $MasterName >> /etc/hosts;\
              \ exit\" \n"
            - "ssh root@$ipaddr_slave2 \"echo $ipaddr_slave1 $Slave1Name >> /etc/hosts;\
              \ exit\" \n"
            - "ssh root@$ipaddr_slave2 \"echo $ipaddr_slave2 $Slave2Name >> /etc/hosts;\
              \ exit\" \n"
            - " \n"
            - "cd /root \n"
            - "yum -y install aria2 \n"
            - "yum -y install java \n"
            - "ssh root@$ipaddr_slave1 \"yum -y install java; exit\" \n"
            - "ssh root@$ipaddr_slave2 \"yum -y install java; exit\" \n"
            - "cd /root \n"
            - "aria2c $HadoopUrl \n"
            - "mkdir -p $HADOOP_HOME \ntar zxvf hadoop-*.tar.gz -C $HADOOP_HOME \n\
              cd $HADOOP_HOME \nmv hadoop-*.*/* ./ \nrmdir hadoop-*.* \n"
            - "echo  >> /etc/profile \n"
            - "echo export HADOOP_HOME=$HADOOP_HOME >> /etc/profile \n"
            - "echo export HADOOP_MAPRED_HOME=$HADOOP_HOME >> /etc/profile \n"
            - "echo export HADOOP_COMMON_HOME=$HADOOP_HOME >> /etc/profile \n"
            - "echo export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native >>\
              \ /etc/profile \n"
            - "echo export YARN_HOME=$HADOOP_HOME >> /etc/profile \n"
            - "echo export HADOOP_HDFS_HOME=$HADOOP_HOME >> /etc/profile \n"
            - "echo export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop >> /etc/profile\
              \ \n"
            - "echo export HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native\
              \ >> /etc/profile \n"
            - "echo export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH >> /etc/profile\
              \ \n"
            - "echo export CLASSPATH=.:$CLASSPATH:$HADOOP__HOME/lib >> /etc/profile\
              \ \n"
            - "source /etc/profile \n"
            - " \n"
            - "mkdir -p $HADOOP_HOME/tmp \n"
            - "mkdir -p $HADOOP_HOME/dfs/name \n"
            - "mkdir -p $HADOOP_HOME/dfs/data \n"
            - "echo >> /etc/profile \n"
            - "JAVA_HOME=`find /usr/lib/jvm -name 'java-1.8.0-openjdk-1.8.0*'` \n"
            - "echo export JAVA_HOME=$JAVA_HOME/jre   >> /etc/profile \n"
            - "echo export JRE_HOME=$JAVA_HOME/jre  >> /etc/profile \n"
            - " \n"
            - "cd $HADOOP_HOME/etc/hadoop \n"
            - "cp hadoop-env.sh bak.hadoop-env.sh \n"
            - "sed -i '/export JAVA_HOME=/{s/^/# /}' hadoop-env.sh \n"
            - "sed -i \"/# export JAVA_HOME=/a export JAVA_HOME=$JAVA_HOME\" hadoop-env.sh\
              \ \n"
            - " \n"
            - "cd $HADOOP_HOME/etc/hadoop \n"
            - "cp core-site.xml bak.core-site.xml \n"
            - "echo '<configuration>' > core-site.xml \n"
            - "echo '  <property>' >> core-site.xml \n"
            - "echo '    <name>hadoop.tmp.dir</name>' >> core-site.xml \n"
            - "echo \"    <value>$HADOOP_HOME/tmp</value>\" >> core-site.xml \n"
            - "echo '  </property>' >> core-site.xml \n"
            - "echo '' >> core-site.xml \n"
            - "echo '  <property>' >> core-site.xml \n"
            - "echo '    <name>fs.default.name</name>' >> core-site.xml \n"
            - "echo \"    <value>hdfs://$ip_addr:9000</value>\" >> core-site.xml \n"
            - "echo '  </property>' >> core-site.xml \n"
            - "echo '</configuration>' >> core-site.xml \n"
            - " \n"
            - "cd $HADOOP_HOME/etc/hadoop \n"
            - "cp hdfs-site.xml bak.hdfs-site.xml \n"
            - "echo '<configuration>' > hdfs-site.xml \n"
            - "echo '  <property>' >> hdfs-site.xml \n"
            - "echo '    <name>dfs.namenode.name.dir</name>' >> hdfs-site.xml \n"
            - "echo \"    <value>file://$HADOOP_HOME/dfs/name</value>\" >> hdfs-site.xml\
              \ \n"
            - "echo '  </property>' >> hdfs-site.xml \n"
            - "echo '' >> hdfs-site.xml \n"
            - "echo '  <property>' >> hdfs-site.xml \n"
            - "echo '    <name>dfs.datanode.data.dir</name>' >> hdfs-site.xml \n"
            - "echo \"    <value>file://$HADOOP_HOME/dfs/data</value>\" >> hdfs-site.xml\
              \ \n"
            - "echo '  </property>' >> hdfs-site.xml \n"
            - "echo '' >> hdfs-site.xml \n"
            - "echo '  <property>' >> hdfs-site.xml \n"
            - "echo '    <name>dfs.replication</name>' >> hdfs-site.xml \n"
            - "echo '    <value>1</value>' >> hdfs-site.xml \n"
            - "echo '  </property>' >> hdfs-site.xml \n"
            - "echo '' >> hdfs-site.xml \n"
            - "echo '  <property>' >> hdfs-site.xml \n"
            - "echo '    <name>dfs.nameservices</name>' >> hdfs-site.xml \n"
            - "echo '    <value>hadoop-cluster1</value>' >> hdfs-site.xml \n"
            - "echo '  </property>' >> hdfs-site.xml \n"
            - "echo '' >> hdfs-site.xml \n"
            - "echo '  <property>' >> hdfs-site.xml \n"
            - "echo '    <name>dfs.namenode.secondary.http-address</name>' >> hdfs-site.xml\
              \ \n"
            - "echo \"    <value>$ip_addr:50090</value>\" >> hdfs-site.xml \n"
            - "echo '  </property>' >> hdfs-site.xml \n"
            - "echo '' >> hdfs-site.xml \n"
            - "echo '  <property>' >> hdfs-site.xml \n"
            - "echo '    <name>dfs.webhdfs.enabled</name>' >> hdfs-site.xml \n"
            - "echo '    <value>true</value>' >> hdfs-site.xml \n"
            - "echo '  </property>' >> hdfs-site.xml \n"
            - "echo '</configuration>' >> hdfs-site.xml \n"
            - " \n"
            - "cd $HADOOP_HOME/etc/hadoop \n"
            - "cp mapred-site.xml.template  mapred-site.xml \n"
            - "echo '<configuration>' > mapred-site.xml \n"
            - "echo '  <property>' >> mapred-site.xml \n"
            - "echo '    <name>mapreduce.framework.name</name>' >> mapred-site.xml\
              \ \n"
            - "echo '    <value>yarn</value>' >> mapred-site.xml \n"
            - "echo '    <final>true</final>' >> mapred-site.xml \n"
            - "echo '  </property>' >> mapred-site.xml \n"
            - "echo '' >> mapred-site.xml \n"
            - "echo '  <property>' >> mapred-site.xml \n"
            - "echo '    <name>mapreduce.jobtracker.http.address</name>' >> mapred-site.xml\
              \ \n"
            - "echo \"    <value>$ip_addr:50030</value>\" >> mapred-site.xml \n"
            - "echo '  </property>' >> mapred-site.xml \n"
            - "echo '' >> mapred-site.xml \n"
            - "echo '  <property>' >> mapred-site.xml \n"
            - "echo '    <name>mapreduce.jobhistory.address</name>' >> mapred-site.xml\
              \ \n"
            - "echo \"    <value>$ip_addr:10020</value>\" >> mapred-site.xml \n"
            - "echo '  </property>' >> mapred-site.xml \n"
            - "echo '' >> mapred-site.xml \n"
            - "echo '  <property>' >> mapred-site.xml \n"
            - "echo '    <name>mapreduce.jobhistory.webapp.address</name>' >> mapred-site.xml\
              \ \n"
            - "echo \"    <value>$ip_addr:19888</value>\" >> mapred-site.xml \n"
            - "echo '  </property>' >> mapred-site.xml \n"
            - "echo '' >> mapred-site.xml \n"
            - "echo '  <property>' >> mapred-site.xml \n"
            - "echo '    <name>mapred.job.tracker</name>' >> mapred-site.xml \n"
            - "echo \"    <value>http://$ip_addr:9001</value>\" >> mapred-site.xml\
              \ \n"
            - "echo '  </property>' >> mapred-site.xml \n"
            - "echo '</configuration>' >> mapred-site.xml \n"
            - " \n"
            - "cd $HADOOP_HOME/etc/hadoop \n"
            - "cp yarn-site.xml bak.yarn-site.xml \n"
            - "echo '<configuration>' > yarn-site.xml \n"
            - "echo '  <property>' >> yarn-site.xml \n"
            - "echo '    <name>yarn.resourcemanager.hostname</name>' >> yarn-site.xml\
              \ \n"
            - "echo \"    <value>$ip_addr</value>\" >> yarn-site.xml \n"
            - "echo '  </property>' >> yarn-site.xml \n"
            - "echo '  <property>' >> yarn-site.xml \n"
            - "echo '    <name>yarn.nodemanager.aux-services</name>' >> yarn-site.xml\
              \ \n"
            - "echo '    <value>mapreduce_shuffle</value>' >> yarn-site.xml \n"
            - "echo '  </property>' >> yarn-site.xml \n"
            - "echo '  <property>' >> yarn-site.xml \n"
            - "echo '    <name>yarn.resourcemanager.address</name>' >> yarn-site.xml\
              \ \n"
            - "echo \"    <value>$ip_addr:8032</value>\" >> yarn-site.xml \n"
            - "echo '  </property>' >> yarn-site.xml \n"
            - "echo '  <property>' >> yarn-site.xml \n"
            - "echo '    <name>yarn.resourcemanager.scheduler.address</name>' >> yarn-site.xml\
              \ \n"
            - "echo \"    <value>$ip_addr:8030</value>\" >> yarn-site.xml \n"
            - "echo '  </property>' >> yarn-site.xml \n"
            - "echo '  <property>' >> yarn-site.xml \n"
            - "echo '    <name>yarn.resourcemanager.resource-tracker.address</name>'\
              \ >> yarn-site.xml \n"
            - "echo \"    <value>$ip_addr:8031</value>\" >> yarn-site.xml \n"
            - "echo '    </property>' >> yarn-site.xml \n"
            - "echo '  <property>' >> yarn-site.xml \n"
            - "echo '    <name>yarn.resourcemanager.admin.address</name>' >> yarn-site.xml\
              \ \n"
            - "echo \"    <value>$ip_addr:8033</value>\" >> yarn-site.xml \n"
            - "echo '  </property>' >> yarn-site.xml \n"
            - "echo '  <property>' >> yarn-site.xml \n"
            - "echo '    <name>yarn.resourcemanager.webapp.address</name>' >> yarn-site.xml\
              \ \n"
            - "echo \"    <value>$ip_addr:8088</value>\" >> yarn-site.xml \n"
            - "echo '  </property>' >> yarn-site.xml \n"
            - "echo '</configuration>' >> yarn-site.xml \n"
            - " \n"
            - "sed -i 304s/'$JAVA'/'java'/ /usr/local/hadoop/hadoop-2.7.7/bin/hdfs\
              \ \n"
            - "ssh root@$ipaddr_slave1 \"mkdir -p $HADOOP_HOME; echo export HADOOP_HOME=$HADOOP_HOME\
              \ >> /etc/profile; echo export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin\
              \ >> /etc/profile; source /etc/profile;exit\" \n"
            - "ssh root@$ipaddr_slave2 \"mkdir -p $HADOOP_HOME; echo export HADOOP_HOME=$HADOOP_HOME\
              \ >> /etc/profile; echo export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin\
              \ >> /etc/profile; source /etc/profile;exit\" \n"
            - "scp -r $HADOOP_HOME/*  root@$ipaddr_slave1:$HADOOP_HOME \n"
            - "scp -r $HADOOP_HOME/*  root@$ipaddr_slave2:$HADOOP_HOME \n"
            - " \n"
            - "cd $HADOOP_HOME/etc/hadoop \n"
            - "echo $ipaddr_slave1 > slaves \n"
            - "echo $ipaddr_slave2 >> slaves \n"
            - " \n"
            - "cd /root \n"
            - "aria2c $ScalaUrl \n"
            - "mkdir -p $SCALA_HOME \ntar zxvf scala-*.tgz -C $SCALA_HOME \ncd $SCALA_HOME\
              \ \nmv scala-*.*/* ./ \nrmdir scala-*.* \n"
            - "echo >> /etc/profile \n"
            - "echo export SCALA_HOME=$SCALA_HOME >> /etc/profile \n"
            - "echo export PATH=$PATH:$SCALA_HOME/bin >> /etc/profile \n"
            - "source /etc/profile \n"
            - " \n"
            - "cd /root \n"
            - "aria2c $SparkUrl \n"
            - "mkdir -p $SPARK_HOME \ntar zxvf spark-*hadoop*.tgz -C $SPARK_HOME \n\
              cd $SPARK_HOME \nmv spark-*hadoop*/* ./ \nrmdir spark-*hadoop* \n"
            - "echo >> /etc/profile \n"
            - "echo export SPARK_HOME=$SPARK_HOME >> /etc/profile \n"
            - "echo export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin >> /etc/profile\
              \ \n"
            - "source /etc/profile \n"
            - " \n"
            - "cp $SPARK_HOME/conf/slaves.template $SPARK_HOME/conf/slaves \n"
            - "echo $ipaddr_slave1 > $SPARK_HOME/conf/slaves \n"
            - "echo $ipaddr_slave2 >> $SPARK_HOME/conf/slaves \n"
            - "cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh\
              \ \n"
            - "echo export SCALA_HOME=$SCALA_HOME > $SPARK_HOME/conf/spark-env.sh\
              \ \n"
            - "echo export JAVA_HOME=$JAVA_HOME   >> $SPARK_HOME/conf/spark-env.sh\
              \ \n"
            - " \n"
            - "ssh root@$ipaddr_slave1 \"mkdir -p $SCALA_HOME; mkdir -p $SPARK_HOME;\
              \ exit\" \n"
            - "ssh root@$ipaddr_slave2 \"mkdir -p $SCALA_HOME; mkdir -p $SPARK_HOME;\
              \ exit\" \n"
            - "scp -r $SCALA_HOME/*  root@$ipaddr_slave1:$SCALA_HOME \n"
            - "scp -r $SCALA_HOME/*  root@$ipaddr_slave2:$SCALA_HOME \n"
            - "scp -r $SPARK_HOME/*  root@$ipaddr_slave1:$SPARK_HOME \n"
            - "scp -r $SPARK_HOME/*  root@$ipaddr_slave2:$SPARK_HOME \n"
            - " \n"
            - "ssh root@$ipaddr_slave1 \"echo >> /etc/profile;echo export SCALA_HOME=$SCALA_HOME\
              \ >> /etc/profile;echo export PATH=$PATH:$SCALA_HOME/bin >> /etc/profile;echo\
              \ export SPARK_HOME=$SPARK_HOME >> /etc/profile;echo export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin\
              \ >> /etc/profile;exit\" \n"
            - "ssh root@$ipaddr_slave2 \"echo >> /etc/profile;echo export SCALA_HOME=$SCALA_HOME\
              \ >> /etc/profile;echo export PATH=$PATH:$SCALA_HOME/bin >> /etc/profile;echo\
              \ export SPARK_HOME=$SPARK_HOME >> /etc/profile;echo export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin\
              \ >> /etc/profile;exit\" \n"
            - " \n"
            - "hadoop namenode -format \n"
            - "systemctl stop  firewalld \n"
            - "start-dfs.sh \n"
            - "start-yarn.sh \n"
            - "$SPARK_HOME/sbin/start-all.sh \n"
            - 'ros-notify -d ''{"data" : "Install Spark."}''

              '
    DependsOn:
    - SNatEntry
    - Slave1WaitCondition
    - Slave2WaitCondition
  ForwardEntry:
    Type: ALIYUN::ECS::ForwardEntry
    Properties:
      ExternalIp:
        Fn::GetAtt:
        - EipForward
        - EipAddress
      ExternalPort: Any
      ForwardTableId:
        Fn::GetAtt:
        - NatGateway
        - ForwardTableId
      InternalIp:
        Fn::GetAtt:
        - Master
        - PrivateIp
      InternalPort: Any
      IpProtocol: Any
    DependsOn:
    - EIpForwardAssociation
    - Master
  MasterConditionHandle:
    Type: ALIYUN::ROS::WaitConditionHandle
    Properties: {}
  MasterWaitCondition:
    Type: ALIYUN::ROS::WaitCondition
    Properties:
      Count: 1
      Handle:
        Ref: MasterConditionHandle
      Timeout: 3600
    DependsOn: Master
Outputs:
  HadoopWebsiteURL:
    Description:
      en: URL for newly created Hadoop environment.
      zh-cn: 新创建的Hadoop环境的网址。
    Value:
      Fn::Join:
      - ''
      - - http://
        - Fn::GetAtt:
          - EipForward
          - EipAddress
        - :50070
  SparkWebsiteURL:
    Description:
      en: URL for newly created Spark environment.
      zh-cn: 新创建的Spark环境的网址。
    Value:
      Fn::Join:
      - ''
      - - http://
        - Fn::GetAtt:
          - EipForward
          - EipAddress
        - :8080
Metadata:
  ALIYUN::ROS::Interface:
    ParameterGroups:
    - Parameters:
      - VpcName
      - VpcCidrBlock
      - VSwitchCidrBlock
      Label:
        default:
          en: VPC
          zh-cn: 专有网络
    - Parameters:
      - ZoneId
      - InstanceName
      - InstanceType
      - SystemDiskCategory
      - ImageId
      - InstancePassword
      Label:
        default:
          en: ECS
          zh-cn: 云服务器
    - Parameters:
      - MasterName
      - HadoopHome
      - Slave1Name
      - Slave2Name
      - HadoopUrl
      Label:
        default: Hadoop
    - Parameters:
      - ScalaHome
      - ScalaUrl
      Label:
        default: Scala
    - Parameters:
      - SparkHome
      - SparkUrl
      Label:
        default: Spark
    TemplateTags:
    - acs:example:大数据:基于3个ECS实例和NAT网关并部署Spark和Hadoop分布式环境
  ALIYUN::ROS::Composer:
    7ab82dc8:
      Rect:
        - 640
        - 560
        - 40
        - 100
        - 1
        - 0
      ResT: Composer::ROSParameter::AlibabaCloud
    144d2642:
      Parent: 7ab82dc8
      Rect:
        - 600
        - 490
        - 60
        - 150
        - 2
        - 0
      ResT: Composer::ROSParameter::Region
    9dec1243:
      Res:
        - EipSNat
        - EIpSnatAssociation
      Parent: 144d2642
      Rect:
        - 40
        - 40
        - 180
        - 560
        - 3
        - 0
    d2e4f42a:
      Res:
        - Vpc
      Parent: 144d2642
      Rect:
        - 440
        - 300
        - 80
        - 200
        - 3
        - 0
    7b2973b4:
      Res:
        - EipForward
        - EIpForwardAssociation
      Parent: 144d2642
      Rect:
        - 40
        - 40
        - 388
        - 560
        - 3
        - 0
    44dd2b7c:
      Res:
        - Slave1ConditionHandle
      Parent: 144d2642
      Rect:
        - 40
        - 40
        - 600
        - 510
        - 3
        - 0
    97ca9cf0:
      Res:
        - Slave2ConditionHandle
      Parent: 144d2642
      Rect:
        - 40
        - 40
        - 600
        - 580
        - 3
        - 0
    9fa3501f:
      Res:
        - Slave1WaitCondition
      Parent: 144d2642
      Rect:
        - 40
        - 40
        - 460
        - 510
        - 3
        - 0
    d44bb514:
      Res:
        - Slave2WaitCondition
      Parent: 144d2642
      Rect:
        - 40
        - 40
        - 460
        - 580
        - 3
        - 0
    c1679111:
      Res:
        - MasterWaitCondition
      Parent: 144d2642
      Rect:
        - 40
        - 40
        - 560
        - 238
        - 3
        - 0
    76aecc24:
      Res:
        - MasterConditionHandle
      Parent: 144d2642
      Rect:
        - 40
        - 40
        - 560
        - 400
        - 3
        - 0
    276e680f:
      Res:
        - VSwitch
      Parent: d2e4f42a
      Rect:
        - 400
        - 230
        - 100
        - 250
        - 4
        - 0
    fea0ca95:
      Res:
        - NatGateway
        - ForwardEntry
      Parent: d2e4f42a
      Rect:
        - 40
        - 40
        - 388
        - 420
        - 5
        - 0
    0639f47e:
      Res:
        - SNatEntry
      Parent: 276e680f
      Rect:
        - 40
        - 40
        - 180
        - 420
        - 5
        - 0
    02b7a934:
      Res:
        - SecurityGroup
      Parent: d2e4f42a
      Rect:
        - 328
        - 106
        - 120
        - 294
        - 10
        - 0
    89d76d4a:
      Parent: 144d2642
      Edge:
        - 9dec1243
        - fea0ca95
      Line: 0:0:0:gray:0
    854d1d8b:
      Parent: 144d2642
      Edge:
        - fea0ca95
        - 7b2973b4
      Line: 0:0:0:gray:0
    6f1d3aac:
      Parent: d2e4f42a
      Edge:
        - fea0ca95
        - 53b35f4a
      Line: 0:0:0:gray:0
    8385b7a0:
      Parent: 144d2642
      Edge:
        - 7b2973b4
        - fea0ca95
      Line: 0:0:0:gray:0
    ffad2188:
      Parent: 144d2642
      Edge:
        - 0639f47e
        - 9dec1243
      Line: 0:0:0:gray:0
    299bdb28:
      Parent: d2e4f42a
      Edge:
        - 0639f47e
        - fea0ca95
      Line: 0:0:0:gray:0
    7b080350:
      Parent: 144d2642
      Edge:
        - 9fa3501f
        - 44dd2b7c
      Line: 0:0:0:gray:0
    0479d04b:
      Parent: 144d2642
      Edge:
        - d44bb514
        - 97ca9cf0
      Line: 0:0:0:gray:0
    613685c7:
      Parent: 276e680f
      Edge:
        - 53b35f4a
        - 1c0cb985
      Line: 0:0:0:gray:0
    fb2e44cb:
      Parent: 276e680f
      Edge:
        - 53b35f4a
        - 6e8d4bf1
      Line: 0:0:0:gray:0
    d7ba91d9:
      Parent: 144d2642
      Edge:
        - c1679111
        - 76aecc24
      Line: 0:0:0:gray:0
    53b35f4a:
      Res:
        - Master
      Parent: 276e680f
      Rect:
        - 40
        - 40
        - 388
        - 314
        - 11
        - 0
      Layer:
        - 02b7a934
    6e8d4bf1:
      Res:
        - Slave2
      Parent: 276e680f
      Rect:
        - 40
        - 40
        - 250
        - 314
        - 11
        - 0
      Layer:
        - 02b7a934
    1c0cb985:
      Res:
        - Slave1
      Parent: 276e680f
      Rect:
        - 40
        - 40
        - 160
        - 314
        - 11
        - 0
      Layer:
        - 02b7a934