AWS and DevOps – Elastic Beanstalk: Deploying an environment

In a serie of blog articles I’ll take a closer look at the available CI/CD development tools, serverless architecture and orchestration services (like Elastic Beanstalk and Cloudformation) in AWS. In my previous blog articles “AWS and DevOps – Elastic Beanstalk” and “”AWS and DevOps – Platform configuration Elastic Beanstalk“you can read more about what Elastic Beanstalk is and the Application Lifecycle and the platform configuration in Elastic Beanstalk. In this blog article I’ll take a closer look at the actual deployment of an application to an EC2 instance.

Deploying an environment

When you create a new application in Elastic Beanstalk you specify the right environment tier. There are two tiers available that eventually will determine which AWS resources will be provisioned. Access management to all the required resources is managed by Elastic Beanstalk by creating the appropiate IAM roles.
An application that serves HTTP requests runs in a web server environment tier. An application that processes long-running workloads on demand or performs tasks on schedule will use an worker environment tier. Workers are specialized applications that have a background processing task that listens for messages on an Amazon SQS queue. Worker applications post those messages to your application by using HTTP. The SQS queue is managed by Elastic Beanstalk because it is one of the components of the worker enviroment tier.

Let’s explore the deployment of an environment more in-depth by taken the web server environment tier as example.

Deployment EC2

The environment is the heart of the application. In the diagram, the environment is shown within the top-level solid line. When you create an environment, Elastic Beanstalk provisions the resources required to run your application. A high available environment for example, include one elastic load balancer, an Auto Scaling group, and one or more Amazon Elastic Compute Cloud (Amazon EC2) instances. The environment has a CNAME (URL) that points to the load balancer, such as This URL is aliased in Amazon Route 53 to an Elastic Load Balancing URL by using a CNAME record. An Alias in Route 53 is a CNAME that points to specific AWS resources. Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service. It provides secure and reliable routing to your infrastructure.

The software stack running on the instances depends on the AMI that is deployed. An AMI (Amazon Machine Image) is a preconfigured template that is used by AWS to launch EC2 instances. The type of AMI defines the infrastructure topology and software stack to be used for that environment. Each instance running the application uses a specific AMI that is specifically created for the language (PHP, Python etc). During bootstrapping every EC2 instance is deployed with set of scripts, hooks and config files specific for your application type. Details can be found in /tmp/ of this running EC2 instance.

Application deployments and configuration updates to the EC2 instances is managed by the Host Manager. This process runs on each instance and listens to events like code deployment or environment variable updates and process these events when triggered. It is responsible for:

  • deploying the application
  • patching instance components
  • monitoring and agregation of events and metrics
  • generating and rotating logs

The daemon process of the Host Manager, that listens to these events, is cfn-hup (/opt/aws/bin/cfn-hup). Cfn-hup is a helper process that’s originally used by CloudFormation. In CloudFormation it detects changes in resource metadata and runs user-specified actions. Apparently they took the basics and integrated cfg-hup with the components for deployments etc within Elastic Beanstalk. Why reinvent the wheel…

The cfn-hup process will log the events to /var/log/cfn-hup.log.  There you’ll find quite a lot of messages about SQS. Cfn-hup listens to SQS (Simple Queue Service) of AWS to process any of these events. Elastic Beanstalk sends the events to SQS, cfn-hup picks them up and executes the required actions on the instance.

The Host Manager eventually triggers some ruby code that does all the magic. You can debug this yourself by logging into the specific EC2 instance. Make sure you deploy your environment with your ssh key pair, otherwise you’ll need to change this afterwards which takes some time because the instance needs to be replaced with a new instance that contains your public key. Let’s go through a few of these steps.

First it starts by reading the content of /etc/elasticbeanstalk/ This file is managed by Elastic beanstalk and contains the ARN (Aws Resource Name) of the S3 bucket that belongs to the environment and the stack_name of the CloudFormation stack that’s deployed. There are some other items in this configuration file like environment_id and region.

Next it will read the container configuration that is stored on the instance. It’s JSON code that describes all kind of settings of your environment. You’ll find a command section with all kind of commands needed to start the container, deploy code, pre/post-build test etc.  There are some php ini settings (in case of PHP environment) that are enforced by EB. One of the interesting parts of this file is the container section. Here you’ll find locations of log files, source bundles and where to find the staging/deploy directory for your app.
To print this section in a readable format, run get-config container --output YAML:

[root@ip-172-31-45-175 log]# /opt/elasticbeanstalk/bin/get-config container --output YAML
app_staging_dir: "/var/app/ondeck"
instance_port: '80'
http_port: '80'
php_version: '7.2'
app_support_dir: "/var/app/support"
app_pids_dir: "/var/app/support/pids"
- "/var/log/httpd/error_log"
- "/var/log/httpd/access_log"
config_staging_dir: "/tmp"
source_bundle: "/opt/elasticbeanstalk/deploy/appsource/source_bundle"
app_deploy_dir: "/var/app/current"
- "/var/log/eb-activity.log"
log_group_name_prefix: "/aws/elasticbeanstalk"
app_assets_dir: "/var/app/support/assets"
app_logs_dir: "/var/app/support/logs"
app_user: webapp
app_base_dir: "/var/app"

The next steps are related to validation, updating configuration files and retrieving metadata and finally loading the manifest from the S3 bucket.

Finally it will start deploying the application:

[2019-05-22T12:31:31.198Z] INFO  [3596]  : Updated manifest cache: deployment ID 4 and serial 4.
[2019-05-22T12:31:31.198Z] DEBUG [3596]  : Loaded definition of Command CMD-AppDeploy.
[2019-05-22T12:31:31.198Z] INFO  [3596]  : Executing Application update
[2019-05-22T12:31:31.199Z] INFO  [3596]  : Executing command: CMD-AppDeploy...
[2019-05-22T12:31:31.199Z] INFO  [3596]  : Executing command CMD-AppDeploy activities...
[2019-05-22T12:31:31.199Z] DEBUG [3596]  : Setting environment variables..
[2019-05-22T12:31:31.199Z] INFO  [3596]  : Running AddonsBefore for command CMD-AppDeploy...
[2019-05-22T12:31:31.317Z] DEBUG [3596]  : Running stages of Command CMD-AppDeploy from stage 0 to stage 1...
[2019-05-22T12:31:31.317Z] INFO  [3596]  : Running stage 0 of command CMD-AppDeploy...
[2019-05-22T12:31:31.317Z] INFO  [3596]  : Running leader election...
[2019-05-22T12:31:31.707Z] INFO  [3596]  : Instance is Leader.
[2019-05-22T12:31:31.707Z] DEBUG [3596]  : Loaded 5 actions for stage 0.
[2019-05-22T12:31:31.707Z] INFO  [3596]  : Running 1 of 5 actions: DownloadSourceBundle...
[2019-05-22T12:31:32.013Z] INFO  [3596]  : Running 2 of 5 actions: EbExtensionPreBuild...
[2019-05-22T12:31:32.434Z] INFO  [3596]  : Running 3 of 5 actions: AppDeployPreHook...
[2019-05-22T12:31:33.915Z] INFO  [3596]  : Running 4 of 5 actions: EbExtensionPostBuild...
[2019-05-22T12:31:34.277Z] INFO  [3596]  : Running 5 of 5 actions: InfraCleanEbextension...
[2019-05-22T12:31:34.278Z] INFO  [3596]  : Running stage 1 of command CMD-AppDeploy...
[2019-05-22T12:31:34.278Z] DEBUG [3596]  : Loaded 2 actions for stage 1.
[2019-05-22T12:31:34.278Z] INFO  [3596]  : Running 1 of 2 actions: AppDeployEnactHook...
[2019-05-22T12:31:34.492Z] INFO  [3596]  : Running 2 of 2 actions: AppDeployPostHook...
[2019-05-22T12:31:34.623Z] INFO  [3596]  : Running AddonsAfter for command CMD-AppDeploy...
[2019-05-22T12:31:34.737Z] INFO  [3596]  : Command CMD-AppDeploy succeeded!
[2019-05-22T12:31:34.737Z] INFO  [3596]  : Command processor returning results:

The AppDeploy command eventually runs some ruby executables that are doing the actual deployment of the code. During the deployment the beanstalk-core gem is loaded.  Finally the /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command-processor.rb contains all the logic for how Elastic Beanstalk handles deployments.

The deployment will be handled by several hook scripts, which are some basic shell scripts located in dedicated hook directories within Elastic beanstalk.

ls /opt/elasticbeanstalk/hooks/

appdeploy/        configdeploy/     .gitignore        postinit/         preinit/          restartappserver/

Inside each directory you’ll find some pre and post scripts that will handle an important part of the deployment of your application. You can debug this by observing /var/log/eb-activity.log. This logfile prints every action, and because tracing is enabled by default the verbosity is very helpful to get a beter understanding of what exactly happens in the background.

Deploying an application

The deployment of a new version of your application is done by uploading a application source bundle containing a new code. This source bundle must meet the following requirements:

  • Zip or WAR file (multiple WAR files can be included inside one ZIP file)
  • Not exceed 512 MB
  • Not include parent or top-level directory in your ZIP file.

The source bundle contains the necessary code to run your application. In addition, you can add .ebextentions files to configure your environment and customize the AWS resources it contains. These configuration files are YAML or JSON formatted documents with a .config file extention. The files are placed in .ebextention directory. On Github you can find a lot of sample configuration files.

After you have written your new code and packaged it into a zip archive you can go to the AWS console and upload the source bundle.

At first glance Elastic Beanstalk looks like a blackbox. The EC2 instances are being deployed with some default set of scripts and processes that will handle all the necessary tasks on behalf of Elastic Beanstalk. The code is stored in a S3 bucket and the interaction between Elastic Beanstalk and the Host Manager relies on SQS service. By debugging and tracing on the EC2 instance you get an idea what happens under the hood of Elastic beanstalk which gives you eventually a better understanding of it.

The provisioning of the environment in Elastic Beanstalk relies fully on Cloudformation. In the next post I will explore Cloudformation.

Vincent LamersVincent Lamers, Linux-consultant @ AT Computing

Actieve filters: Wis alle filters