AWS Cloudformation – Meta data

In a serie of blog articles I’ll take a closer look at AWS CloudFormation. Read more about what AWS CloudFormation is, how to design templates and bootstrapping an EC2 with userdata in my previous blog articles:

• AWS CloudFormation – Templates, stacks and change sets
• AWS CloudFormation – Designing Templates
• AWS CloudFormation – Bootstrapping an Ec2 instance with Userdata

In this blog article I’ll take a closer look at using metadata in the resource section to prevent a mess.

CloudFormation Metadata

Additional to bootstrap scripts you can include meta data on an EC2 instance. In comparison to the UserData, where basic shell scripting is used, metad ata will follow a declarative approach to setting up the EC2 instances. Use the AWS::CloudFormation::Init type to include meta data on an Amazon EC2 instance for the cfn-init helper script. When your template calls the cfn-init script, the script looks for resource meta data rooted in the AWS::CloudFormation::Init metadata key. The cfn-init command can be found in UserData property. Basically the previous UserData example will be extended with this extra line:

/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} \
--resource MyInstance --region ${AWS::Region}

More about this later.

The metadata configuration is seperated into sections. The following example shows how you can attach metadata for cfn-init to an Amazon EC2 instance resource within the template.

Resources: 
  MyInstance: 
    Type: AWS::EC2::Instance
    Metadata: 
      AWS::CloudFormation::Init: 
        config: 
          packages: 
            :
          groups: 
            :
          users: 
            :
          sources: 
            :
          files: 
            :
          commands: 
            :
          services: 
            :
    Properties: 

The metadata is organized into config keys, which you can group into configsets. You can specify a configset when you call cfn-init in your template. If you don’t specify a configset, cfn-init looks for a single config key named config. The cfn-init helper script processes these configuration sections in the following order: packages, groups, users, sources, files, commands, and then services. If you require a different order, separate your sections into different config keys, and then use a configset that specifies the order in which the config keys should be processed.

AWS::CloudFormation::Init: 
  configSets: 
    ascending: 
      - "config1"
      - "config2"
    descending: 
      - "config2"
      - "config1"
  config1: 
    commands: 
      test: 
        command: "echo \"$CFNTEST\" > test.txt"
        env: 
          CFNTEST: "I come from config1."
        cwd: "~"
  config2: 
    commands: 
      test: 
        command: "echo \"$CFNTEST\" > test.txt"
        env: 
          CFNTEST: "I come from config2"
        cwd: "~"

With cfn-init -c <configSet> you can call a specific set and force an ordering. Also, using configSets allows you to combine multiple configSets within your template.

AWS::CloudFormation::Init: 
  1: 
    commands: 
      test: 
        command: "echo \"$MAGIC\" > test.txt"
        env: 
          MAGIC: "I come from the environment!"
        cwd: "~"
  2: 
    commands: 
      test: 
        command: "echo \"$MAGIC\" >> test.txt"
        env: 
          MAGIC: "I am test 2!"
        cwd: "~"
  configSets: 
    test1: 
      - "1"
    test2: 
      - ConfigSet: "test1"
      - "2"
    default: 
      - ConfigSet: "test2"

Let’s dive into more detail and explore each section keys. Every key contains a set of (sub) keys. Some of them are required.

Commands key

The only required key in this section is command. This can either be a string or an array. Commands are processed in alphabatical order by name. Optionally you can specify environment variables, working directory and test-runs before executed by cfn-init (dry run). Additionally you can ignore errors by setting ignoreErrors to true.

Files key

You can use the files key to create files on the EC2 instance. The content can be either inline in the template or the content can be pulled from a URL.

files: 
  /tmp/setup.mysql: 
    content: !Sub | 
      CREATE DATABASE ${DBName};
      CREATE USER '${DBUsername}'@'localhost' IDENTIFIED BY '${DBPassword}';
      GRANT ALL ON ${DBName}.* TO '${DBUsername}'@'localhost';
      FLUSH PRIVILEGES;
    mode: "000644"
    owner: "root"
    group: "root"

Creating a symlink can easily done by specify the symlink target in the content key. The mode key uses the first three digits for symlinks and the last three digits for setting permissions. To create a symlink, specify 120000. To specify permissions for a file, use the last three digits, such as 000644

files:
  /tmp/file1:
    content: "/tmp/file2"
    mode: "120644"

Sources key

Besides declaring the content inside the template, you can use the source key to specify a specific URL, like github etc. It can also be a S3 bucket. Additionally you can configure your access keys in the authentication key to succesfully pull the content from S3. You need to configure the type key correctly to tell CloudFormation about the kind of authentication. use s3 to authenticate to S3 buckets. Use basic to authenticate to gitlab or another site. The type of authentication defines the properties to use. So s3 will require secretkey,accessKeyId and bucket. Whereas basic requires an url, username and password.

Packages key

If you’re familiar with configuration managent this is quite similar. It enables you to install packages and specify the version. Version is not required, if you leave this blank, CloudFormation assumes you want the latest version.

yum: 
  httpd: []

in this example CloudFormation leverages the yum repository to install httpd with the latest version.

Services key

This key managed the services, but has some extra keys which you not expect at first. It the following code snippet you’ll see two examples of services that are managed by CloudFormation init.

services: 
  sysvinit: 
    nginx: 
      enabled: "true"
      ensureRunning: "true"
      files: 
        - "/etc/nginx/nginx.conf"
    php-fastcgi: 
      enabled: "true"
      ensureRunning: "true"
      packages: 
        yum: 
          - "php"
          - "spawn-fcgi"
    

Besides declaring the run- and startup-state, you also can manage the files, sources and packages that are required to run the services. Any changes to this will trigger a restart of the service.

User and Group keys

These keys will manage your user and groups during creation.

groups: 
  groupOne: 
    gid: "45"
users: 
  myUser: 
    groups: 
      - "groupOne"
    uid: "50"
    homeDir: "/tmp"

As already mentioned the metadata is accessed by cfn-init via UserData. With Fn::Sub you can substitute the stackname and region pseudo parameters for the actual stackname and and region at runtime. Always update aws-cfn-bootstrap to have to latest version installed.

Ec2Instance:
  Type: AWS::EC2::Instance
  Metadata: 
    AWS::CloudFormation::Init: 
      config:
      .. 
  Properties:
    UserData:
      Fn::Base64:
        !Sub |
          #!/bin/bash -xe
          yum update -y aws-cfn-bootstrap
          /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} \ 
          --resource MyInstance --region ${AWS::Region}

In this example cfn-init will look for the metadata of the resource MyInstance within the CloudFormation template and executes the configuration (search for config by default).

As described cfn-init is only applicable to the bootstrap process of your EC2 instance. Updating the configuration of your instances can be done with the cfn-hup deamon. cfn-hup runs user defined actions when a change is detected in the resource metadata. Next time cfn-hup will be covered in more detail. You can read it here.