AWS Cloudformation – Metadata

In a serie of blog articles I’ll take a closer look at AWS CloudFormation. Read more about what AWS CloudFormation is, how to design templates and bootstrapping an EC2 with userdata in my previous blog articles:

• AWS CloudFormation – Templates, stacks and change sets
• AWS CloudFormation – Designing Templates
• AWS CloudFormation – Bootstrapping an Ec2 instance with Userdata

In this blog article I’ll take a closer look at using metadata in the resource section to prevent a mess.

CloudFormation Metadata

Additional to bootstrap scripts you can include meta data on an EC2 instance. In comparison to the UserData, where basic shell scripting is used, metad ata will follow a declarative approach to setting up the EC2 instances. Use the AWS::CloudFormation::Init type to include meta data on an Amazon EC2 instance for the cfn-init helper script. When your template calls the cfn-init script, the script looks for resource meta data rooted in the AWS::CloudFormation::Init metadata key. The cfn-init command can be found in UserData property. Basically the previous UserData example will be extended with this extra line:

/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} \
--resource MyInstance --region ${AWS::Region}

More about this later.

The Metadata configuration is separated into sections. The following example shows how to attach Metadata for cfn-init to an Amazon EC2 instance resource within the template.

Resources: 
  MyInstance: 
    Type: AWS::EC2::Instance
    Metadata: 
      AWS::CloudFormation::Init: 
        config: 
          packages: 
            :
          groups: 
            :
          users: 
            :
          sources: 
            :
          files: 
            :
          commands: 
            :
          services: 
            :
    Properties: 

Metadata is organized into config keys, which you can group into config sets. You can specify a config set when you call cfn-init in your template. If you don’t specify a config set, cfn-init looks for a single config key named config. The cfn-init helper script processes these configuration sections in the following order: packages, groups, users, sources, files, commands, and then services. If you require a different order, separate your sections into different config keys, and then use a config set that specifies the order in which the config keys should be processed.

AWS::CloudFormation::Init: 
  configSets: 
    ascending: 
      - "config1"
      - "config2"
    descending: 
      - "config2"
      - "config1"
  config1: 
    commands: 
      test: 
        command: "echo \"$CFNTEST\" > test.txt"
        env: 
          CFNTEST: "I come from config1."
        cwd: "~"
  config2: 
    commands: 
      test: 
        command: "echo \"$CFNTEST\" > test.txt"
        env: 
          CFNTEST: "I come from config2"
        cwd: "~"

With cfn-init -c <configSet> you can call a specific set and force an ordering. Also, using configSets allows you to combine multiple configSets within your template.

AWS::CloudFormation::Init: 
  1: 
    commands: 
      test: 
        command: "echo \"$MAGIC\" > test.txt"
        env: 
          MAGIC: "I come from the environment!"
        cwd: "~"
  2: 
    commands: 
      test: 
        command: "echo \"$MAGIC\" >> test.txt"
        env: 
          MAGIC: "I am test 2!"
        cwd: "~"
  configSets: 
    test1: 
      - "1"
    test2: 
      - ConfigSet: "test1"
      - "2"
    default: 
      - ConfigSet: "test2"

Let’s dive into more detail and explore each section keys. Every key contains a set of (sub) keys. Some of them are required.

Commands key

The only required key in this section is command. This can either be a string or an array. Commands are processed in alphabetical order by name. Optionally you can specify environment variables, working directory and test-runs before being executed by cfn-init (dry run). Additionally you can ignore errors by setting ignoreErrors to true.

Files key

You can use the files key to create files on the EC2 instance. The content can be specified inline in the template or can be pulled from a URL.

files: 
  /tmp/setup.mysql: 
    content: !Sub | 
      CREATE DATABASE ${DBName};
      CREATE USER '${DBUsername}'@'localhost' IDENTIFIED BY '${DBPassword}';
      GRANT ALL ON ${DBName}.* TO '${DBUsername}'@'localhost';
      FLUSH PRIVILEGES;
    mode: "000644"
    owner: "root"
    group: "root"

Creating a symlink can easily done by specify the symlink target in the content key. The mode key uses the first three digits for symlinks and the last three digits for setting permissions. To create a symlink, specify 120000. To specify permissions for a file, use the last three digits, such as 000644

files:
  /tmp/file1:
    content: "/tmp/file2"
    mode: "120644"

Sources key

Instead of declaring the content inside the template, you can use the source key to specify a specific URL, like GitHub etc. This may also be a S3 bucket. Additionally, you can configure your access keys in the authentication key to successfully pull the content from S3. You need to configure the type key correctly, and tell CloudFormation about the kind of authentication. Use s3 to authenticate to S3 buckets and basic to authenticate to GitLab or another site. The type of authentication determines which properties to use. s3 requires secretKey, accessorized and bucket. basic on the oterh hand requires an url, user name and password.

Packages key

This is very similar to configuration management. It allows you to install packages and specify the version. Version is not required, leaving this blank, CloudFormation assumes the latest version.

yum: 
  httpd: []

in this example CloudFormation leverages the yum repository to install httpd with the latest version.

Services key

This key manages the services, but has some extra keys, which you may not expect at first sight. The following code snippet shows two examples of services managed by CloudFormation init.

services: 
  sysvinit: 
    nginx: 
      enabled: "true"
      ensureRunning: "true"
      files: 
        - "/etc/nginx/nginx.conf"
    php-fastcgi: 
      enabled: "true"
      ensureRunning: "true"
      packages: 
        yum: 
          - "php"
          - "spawn-fcgi"
    

Besides declaring the run- and startup-state, you can also manage the files, sources and packages that are required to run the services. Any changes to this will trigger a restart of the service.

User and Group keys

These keys will manage your user and groups during creation.

groups: 
  groupOne: 
    gid: "45"
users: 
  myUser: 
    groups: 
      - "groupOne"
    uid: "50"
    homeDir: "/tmp"

As already mentioned the Metadata is accessed by cfn-init via UserData. With Fn::Sub you can substitute the stackname and region pseudo parameters by the actual stackname and region at run-time. Always update aws-cfn-bootstrap to have to latest version installed.

Ec2Instance:
  Type: AWS::EC2::Instance
  Metadata: 
    AWS::CloudFormation::Init: 
      config:
      .. 
  Properties:
    UserData:
      Fn::Base64:
        !Sub |
          #!/bin/bash -xe
          yum update -y aws-cfn-bootstrap
          /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} \ 
          --resource MyInstance --region ${AWS::Region}

In this example cfn-init will look for the meta data of the resource MyInstance within the CloudFormation template and executes the configuration (search for config by default).

As described above, cfn-init only applies to the bootstrap process of your EC2 instance. Updating the configuration of your instances can be done with the cfn-hup daemon. cfn-hup runs user defined actions when a change is detected in the resource Metadata. Next time cfn-hup will be covered in more detail. You can read it here.