There are two mechanisms to initialize instances in AWS. Cloud init and CloudFormation Init. Both are widely used and we discuss each of them in this posting.
Cloud-Init is a service originally built for Ubuntu, as a bootstrapping utility to customize a Linux VM as it boots for the first time. It has evolved to be an industry standard multi-distribution method for cross-platform (public or private) cloud instance initialization, or even bare-metal installation. In cloud-init you can install packages and write files, or configure users and security. Because cloud-init is called during the initial boot process, there are no additional steps or required agents to apply your configuration.
Cloud-Init uses UserData, which is part of instance metadata. With AWS, you can pass two types of user data to Amazon EC2: shell scripts and cloud-init directives.
The cloud-config files are text files encoded in base64, with more details covered in the documentation here. cloud-init also works across distributions. For example, you don’t use apt-get install or yum install to install a package. Instead you can define a list of packages to install. cloud-init automatically uses the native package management tool for the distro you select.
The cloudformation init mechanism does not only initialize instance, it also provides a mechanism for the resource being created to communicate with other resources. It allows an instance to emit signal to a different resource (via cfn-signal). It can also monitor changes to external resource and invoke local action (using cfn-hup with hooks). CloudFormation Init requires several components to work together:
- The cloudformation resource should have metadata. The metadata must have a key AWS::CloudFormation::Init in which configsets are declared.
- The UserData must use helper script (cfn-init) to invoke configuration jobs
- The UserData can use helper script (cfn-signal) to signal with a CreationPolicy or WaitCondition (of the same or different resource), so you can synchronize other resources in the stack when the prerequisite resource or application is ready.
- The cfn-hup service on the instance can be configured, to check for updates to metadata and execute custom hooks when changes are detected.
While there are overlaps between the functionalities of Cloud Init and CloudFormation Init, the major difference is the latter support extended features (signal, update, etc); whereas the former is vendor neutral. The table below summarized some the differences:
|Cloud Init||CloudFormation Init|
|Works on||Linux OS distribution||CloudFormation resource, in combination with cfn helper scripts|
|Usecase||Initialization only||Both initialization and resource update|
|Trigger||cloud-init systemd service||Initial: from UserData|
Update: by cfn hook
|Adoption||Multiple cloud vendors and bare-metal system||AWS cloud instances|
Instance Metadata -> User Data, encoded in base 64
|CloudFormation Resource -> Metadata section -> AWS::CloudFormation::Init -> configSets and configs|
|Log file and stdout||/var/log/cloud-init.log|
AWS Cloud Development Toolkit (CDK)
Traditionally, AWS CloudFormation uses template in YAML or JSON for resource declaration. As the size of system grows, the amount of resource involved grows quickly and the size of such declaration file may grow beyond manageable. Nested stacks and export of output are mechanisms designed to combat the template sprawling, but to a very limited extent. Two reasons it is hard to control template size are:
- In declarative statements, each line carries very small piece of information. Without flow controls such as if-else, loops, object oriented structure, the level of code reusability is very low;
- Some auxiliary resources (such as AWS::EC2::VPCGatewayAttachment) must be declared explicitly, even though they are insignificant to the stack functionality
An Example in Python
I have created an example for AWS CDK in Python. The purpose is to create some EC2 instance to complete a lab for Kubernetes (without using managed EKS service). The example provisions the followings:
- VPC, a public and private subnets, Internet and NAT gateways;
- Relevant security groups and permissions
- Bastion host, public instances in public subnet
- Private instances in private subnet, with public route through NAT gateway
Here is the code repo for this example. With CloudFormation only, the single template could go well beyond 1000 lines. With CDK, the code are organized into several different python files, each representing a stack. The stacks can be stood up with command:
cdk deploy vpc-stack cdk deploy security-stack cdk deploy bastion-stack cdk deploy private-stack
Although the documentation in Python is available, there are generally not a lot of examples built out on the Internet. The pypi site provides some Python specific examples for each module (e.g. core and aws-ec2). Given these libraries are available for only 2 years (since 2018), many advocates TypeScript as the language. However, I have implemented some CloudFormation init, used helper script, and UserData in this example, without running into any language specific issues.It should be noted that the EC2 instance by default will call cfn-init. So there is no need to explicitly run cfn-signal or cfn-init from user data in python code (example). This can be verified in file /var/lib/cloud/instances/<instance-id>/user-data.txt which automatically includes the following lines:
# fingerprint: e1b32ead13878deb ( set +e /opt/aws/bin/cfn-init -v --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456 -c config_set_1,config_set_2 /opt/aws/bin/cfn-signal -e $? --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456 cat /var/log/cfn-init.log >&2 )