ec2 on Digi Hunch

Managing EC2 instances across accounts with Ansible

Mon, 27 May 2024 14:00:22 -0400

I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is consistency in the VM management, over time, and across platforms (AWS, on-prem, Azure, etc). Even for AWS shops, in the last few years many enterprises have adopted AWS landing zone with the multiple AWS account prescriptive pattern. However AWS Systems Manager still lacks integration with AWS Organization (except for a few non-core capabilities). This creates the demand of managing EC2 instances across AWS accounts. In this post, we propose a secure method to manage a fleet of EC2 instances from multiple AWS accounts, using Systems Manager . It also enables connectivity from an Ansible control node.

Prerequisites

This proposal ties a few CLI tools together, including AWS CLI, SSH, Ansible, etc. It also requires the cloud engineer to understand how they work. I’ll start with the choice of the tools.

Above, I discussed the benefit of Ansible. Since Ansible operates on SSH, we’ll still have to use SSH tools. Even though SSM agent provides a way to connect to EC2 instance without requiring an RSA key pair, we still need SSH since it is a well-established industry standard (RFC4253) and the foundation of Ansible. These two technologies are not mutually exclusive. In fact, the SSM agent provides a secure enhancement to the operation with SSH. Traditionally, on each EC2 instance we’d have to run SSHD services which opens TCP port 22 (or alternative TCP port as configured). For authentication we favour key pair over password but the open port is still an attack surface vulnerable to brute force and DDoS attacks. For EC2 instances on private networks there is no reachability to the instance’s SSH port unless the bastion is also in a connected network.

As I cover in a post, the SSM Session Manager comes in handy. The SSM agent operates from the instance and communicate outbound to AWS backend. Since the SSM agent runs under a privileged user on the OS, you can perform OS-level commands through SSM. Further, AWS developed a Session Manager plugin with AWS CLI, allowing AWS CLI as a proxy command when making an SSH connection. Therefore SSM enables SSH connection without requiring port 22 to be open. In addition we’ll need to use RSA key pair as required for SSH, which is also an improvement to the security posture.

That explains the dependent tools. On the Ansible control node, apart from Ansible itself, we need the latest version of AWS CLI with the Session Manager plugin, we need to configure AWS CLI properly to connect to EC2 instances across multiple AWS accounts.

Configure AWS CLI

This section discusses how to configure AWS CLI. I have a couple of handy aliases for productivity but they are not essential. For example, I often need to check the IAM identity making the call, and I often need to list out all profiles configured. So I added the following two entries in the ~/.aws/cli/alias file:

[toplevel]
whoami = sts get-caller-identity --no-cli-pager --output yaml
profile = configure list-profiles

With that I have an alias to check IAM identity and available profiles. Then we can start configuring the profiles for CLI (in the file ~/.aws/config). Since we’ll be working with multiple AWS accounts, we have to manage multiple CLI profiles, which implies that:

We better use the --profile switch to explicitly specify profile being used, instead of relying on the AWS_PROFILE environment variable;
As a security best practice, we should not configure profiles with long-term IAM credential in the config file;
We must ensure the CLI doesn’t prompt for log-in every time we switch profile

To satisfy #3 there are many ways but we’ll discuss two: using cross-account IAM role, and using AWS SSO.

Bonus point if you enable auto-complete for AWS CLI.

Configure AWS CLI Profiles

With cross-account IAM role, the idea, is that the client start with one IAM identity, and use that IAM identity to assume roles on several other accounts. The configuration looks like this:

[profile jump_account]
credential_process = /opt/bin/awscreds-custom --username helen

[profile target_account_1]
role_arn = arn:aws:iam::123456789011:role/OrganizationAccountAccessRole
source_profile = jump_account

[profile target_account_2]
role_arn = arn:aws:iam::123456789012:role/OrganizationAccountAccessRole
source_profile = jump_account

In this example, you start with an validated identity in the jump account, then assume a privileged IAM role named OrganizationAccountAccessRole on the target accounts. Typically such IAM roles are pre-configured (e.g. in an multi-account landing zone) with appropriate trust policy to allow principals from the jump account. Once you’re validated as the IAM identity in the jump account, then you can use profiles for target accounts without being prompted for credentials again.

If your have configured IAM Identity Center for the multi-account environment, consider an alternative approach using sso login. The configuration usually looks like this:

[sso-session sso]
sso_start_url = https://myorg.awsapps.com/start/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile target_account_1]
sso_session = sso
sso_account_id = 123456789011
sso_role_name = AWSAdministratorAccess

[profile target_account_2]
sso_session = sso
sso_account_id = 123456789012
sso_role_name = AWSAdministratorAccess

This is often used by human users with SSO credential. In this example, to authenticate the sso session, start with command “aws sso login” . Then you can use all profiles by specifying --profile switch without having to log in again.

Configure SSH to EC2 via SSM

The EC2 instance must connect to Systems Manager endpoint, before one can SSH to the instance using the plugin. Once connected, you should find the instance in Fleet Manager. For this to happen, there are a few conditions. First, the instance must be able to reach the endpoint, either via public Internet, or via VPC interface endpoints if in a private subnet. Second, the instance profile must contain an IAM role with appropriate permissions. We can use AWS managed policy AmazonSSMManagedInstanceCore in the role. In addition, if we record the SSM session to an S3 bucket with encryption the instance profile must have permission to use the encryption key.

SSM agent uses the IAM role. The agent runs as a service on Linux or Windows machines. Many AWS managed AMIs come with SSM agent pre-installed. If that is not the case, you’d install the agent in your own AMI, or in user data which requires downloading the installer. With these configuration you’d be able to connect to the instance via SSM. We can use AWS CLI SSM command, or AWS web console to start an SSH session. To use SSH CLI utility, we install the session manager plugin along with AWS CLI on the SSH client machine. We also specify a public key for EC2 instance and run ssh command with the private key. The SSH configuration needs a configuration such as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

With this entry, you may directly SSH by instance ID (usually starting with i-* and mi-*), and the specified Proxy Command with SSM session document AWS-StartSSHSession will be invoked.

Configure Ansible Inventory

We can SSH to an instance (without port 22 open, on top of SSM) using the method above. Similarly, we can also configure Ansible to connect to the instance, without port 22. The inventory configuration looks like this:

mytest:
 hosts:
 instance1:
 ansible_host: i-00aabbffcc7755221
 ansible_user: ubuntu
 ansible_ssh_common_args: -o ProxyCommand="aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1"
 instance2:
 ansible_host: i-eedd88ff66aa22442
 ansible_user: ubuntu
 ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ProxyCommand="sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --profile target_account_2 \""

Note I used two patterns for ansible_ssh_common_args that are similar. Both works. Each entry references its own profile. This is necessary because Ansible does not have the knowledge which instance belongs to which profile’s account.

Another way to get Ansible connect to instances is to use the connection plugin community.aws.aws_ssm, by specifying ansible_connection: aws_ssm (e.g. as host variable) and other required variables (e.g. profile). This method does not require SSH channel but it requires an S3 bucket, and hence IAM permission on the controller node.

Ansible supports dynamic inventory, in three ways: inventory source file (with existing plugin), custom inventory plugin, and inventory script (in Python). Take source file as an example for EC2, add the followings as the content of aws_ec2.yaml:

plugin: amazon.aws.aws_ec2

# Attach the default AWS profile
aws_profile: target_account_1

compose:
 ansible_host: instance_id
 ansible_user: "'ubuntu'"
 ansible_ssh_common_args: "'-o ProxyCommand=\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1 \"'"

Then we can display the rendered inventory list, and Ansible-ping the instances.

ansible-inventory -i aws_ec2.yaml --list -y | less
ansible all -i aws_ec2.yaml -m ping

For more flexibility, for the composed variables, we could use jinja2 expression to generate the value. In both ways, we produce an inventory source per profile using the built-in inventory plugin aws_ec2. For greater flexibility, such as consolidating instances from all accounts into a single inventory, consider writing your own inventory script, or even own inventory plugin.

Conclusion

In this post we propose a way to manage instances across AWS accounts. Two main challenges are establishing the communication channel (SSH on top of SSM) and generating inventory data in Ansible. Some AWS services can generate inventory data, such as resource data sync in Systems Manager, or using AWS Config Aggregator. It is unfortunate that neither way produces the inventory data in a format that is directly compatible with Ansible inventory. Therefore, you might have to create a custom Ansible dynamic inventory script (in Python) that reads from the inventory data from AWS Config Aggregator (which supports AWS Organization). The side benefit of this script is that it is usually faster than the built-in aws_ec2 inventory plugin.

AWS CDK example in Typescript – provision an AWX server

Sat, 19 Dec 2020 17:14:00 -0400

This post provides an example of using AWS CDK in Typescript.

Ansible Tower and AWX

We have used open-source Ansible extensively in the past. While the automation is convenient, the lack of UI makes it not as suitable as a team collaboration tool. One way to allow team collaboration with open-source Ansible, is to use Jenkins to glue the components together, as discussed in the Automated Deployment Pipeline series. In this setup, the open-source Ansible remains command-line driven, with Jenkins building up the command, rather than a human user. There are many upsides in this configuration, but it is not built specifically for Ansible. Ansible is agent-less, and can be run from any host. This sounds appealing and can work well in smaller server fleet. However, since it requires some configuration on the controlling host for Ansible to function properly, it become unnecessary to configure Ansible environment on every single host (e.g. production). A typically environment only has Ansible environment configured on the bastion host. This brings the need for a dedicated controller server to drive all Ansible tasks.

Ansible Tower is Red Hat’s commercial enhancement to the open source Ansible, providing web-based console, REST API and other services such as Role-based Access Control (RBAC). Managing Ansible via REST API is still somewhat involving but this also enables other open-source contributions to simplify the use of API. For example Tower CLI allows you to use Ansible Tower with simplified command. Ansible Tower has an open-source upstream project called AWX, maintained by Red Hat. AWX is essentially a preview release of Ansible Tower without commercial support. AWX can serve as an engine for all Ansible related task. AWX server is essentially an Ansible control server. AWX, or Ansible Tower, also brings several concepts on top of Ansible:

Job template: defines how an Ansible playbook should be executed, including details such as machine credential, project, inventory, and playbook file.
Job: the actual execution of job template
Project: connects Ansible Tower to source control such as BitBucket. It is tied to a Git repository and a branch within that repository

To deploy AWX on EC2 instances, there is a reference deployment by AWS. However, it is provided as CloudFormation template and appears to be outdated (from 2018). In our project (late 2020, named ansible tower lab, or dubbed as “atlab”), we provide the infrastructure in AWS CDK (written in typescript), to provision the AWX environment.

The goal is that once the configuration is completed, you can run ansible ping against a target EC2 instance. The steps are as automated as possible. However, a number of key steps are purposefully left manual for learning purpose, such as the installation of AWX on EC2 instance.

Infrastructure as Code

In previous posting, I created infrastructure as code in AWS CDK with Python, so I decided to change to typescript in this project, with the assumption it is just a matter of syntax mapping. However, I underestimated the transition to a new language I never learned before. A fuzzy understanding of little details such as when to use let a=4 vs this.a=4, may produce elusive errors that takes hours to troubleshoot. I would therefore strongly recommend reading the basic syntax guide for typescript, before getting started.

In typescript, this example project provides an implementation of configuring autoscaling groups, including cloud init, user data, etc on AWS. Other than the language, everything else is very similar to the project in this post, which was developed in Python. Also, note that the project directory structure varies slightly based on the language being used.

If you are absolutely new to AWS cdk, start with this app. It is beyond the scope of this post, to cover extensively the installation and environment configuration of AWS CDK.

In the provision process for Bastion host, the cloudformation init script pulls a specific version from AWX repository, then makes slight modification. User will need to install it manually. Note that AWX can be installed on three types of platforms:

OpenShift
Kubernetes
Docker Compose

All are documented in their README file. For simplicity in this project, the installation is on standalone docker compose. This is the default mode so there is no need to modify the inventory file.

The code repo

The repository is version controlled here. To run the project, you need to have aws cli environment, then install the required packages including node js, and npm packages such as aws cdk. Once configured, validate with command:

cdk ls

This should display the stacks available. Use cdk deploy to deploy each stack.

When BastionStack is deployed, dependent packages should be installed with user data and cloud init. You will just need to SSH on to the server to manually install AWX, as explained in the instruction, to manually install AWX:

ansible-playbook -i inventory install.yml

Then you can browse to the server (at port 80 by default). Before the log-in page for the first time, the AWX will upgrade itself, with the following screen presented:

Now log on with default credential (in README.md), you will have the UI for AWX:

AWX Web Console

From here, you can edit inventory by adding the host. Or use the helper script (~/awxcompose-helper.sh) from bastion host to create a new inventory (named Private Instance Inventory), and populate it with the hosts in the stack. The helper script does so by querying aws resource, and isssue rest API calls to AWX. After executing the script, you can see a new inventory, and the Private instance inventory should contain all hosts in the stack:

Automatically populated inventory

You can then run Ansible ping against the host to validate connectivity. Note that during inventory creation, the ansible_user is already set to ec2-user (by the helper script):

Ping result

Some technical details

The initialization process on Bastion host creates an RSA key pair, stores the public key to AWS, for the upcoming private instances to uses. It keeps the private key locally in order to make outgoing SSH connection to the private instances. To ensure connectivity between AWX and private instances, there are a couple of (bash) helper scripts involved. Both reflects some technical details that I had to work through.

awxcompose-helper.sh: the initialization process pulls AWX installation file from git repo. The installation process will build a docker-compose file in ~/.awx/awxcompose, based on a template (~/awx-*/installer/roles/local_docker/templates/docker-compose.yml.j2). When user tells AWX to connect to private instance, the connection was made out of a docker container (instead of from the OS of bastion host), we need this script to map SSH key file from host to container, by modifying the template file. Without this helper, outgoing SSH connection will fail with error (Permission denied (publickey,gssapi-keyex,gssapi-with-mic)). This script is invoked in the cloud init process without requiring manual execution.
awxinvt-helper.sh: once the private stack is up and the installation has completed, we need to add the hosts to AWX inventory. This script gets the instance ID and IP addresses of the private instances, and uses Rest API calls to create inventory and populate it with hosts. Ansible has multiple ways of authentication. This script uses the non-stateful basic authentication with each curl command requiring credential. Ansible Rest API guide is provided here and be wary of the convention where URI must end with a slash.

This project is just a start of AWX on AWS CDK project using Typescript. In real life scenarios, there are some work to do to make this even more automated. For example, use cfn-hup service to monitor changes of private stack, and therefore update inventories accordingly.

Virtualization 1 of 4 – Hypervisor

Mon, 27 Jul 2020 22:52:00 -0400

In broad terms, virtualization of computing resource is about isolation of resources, at different levels. There are five levels of virtualization:

Application level, such as JVM, .NET CLR
Library (user-level API) level
Operating system level, such as LXC, Docker, OpenVZ
Hardware abstraction layer (HAL) level, such as VMware, Xen, etc
Instruction set architecture (ISA) level

In my context I deal mostly with OS level and HAL (hardware abstraction layer) level of virtualization. In loose terms, the word containerization refers to OS level virtualization, while the word virtualization is exclusively reserved for HAL level virtualization, also referred to as hypervisor-based virtualization. This post will just focus on this family of technology and loosely refers to it as virtualization.

Virtualization technology evolved from on-premise data centre environment and now is the backbone of cloud computing. The challenges of IT operation in the era of virtualization involves managing VM sprawling, investigating performance issues, planning capacity and addressing storage I/O block. The idea of virtualization is sharing (thus isolating) resources for better utilization, leading to better return on investment. This posting is to cover only the very basics of virtualization.

Hypervisor

Hypervisor is the software layer which provides the capability to run multiple virtual machines on the same physical host. It is broken down into two categories:

Type I hypervisor (aka bare metal hypervisor): directly run on physical hardware. They control the hardware as well as manage the virtual machines. For example, Linux KVM, VMware ESXi, Xen and Microsoft Hyper-V
Type II hypervisor: runs as an application or service on top of the host operating system, which is installed on the bare metal. Guest operating system calls need to traverse via the host operating system stack to reach hardware resource. For example, Oracle Virtual Box, VMware Fusion and Linux Containers (LXC)

Hypervisor Types

Virtualization Techniques

The most primitive form of technology that can be arguably categorized under virtualization is hardware emulation, where a piece of (more accessible) hardware imitates another (less accessible). The architecture limits itself in functional testing only, and is not built for performance or production at all.

The original virtualization technology deals with CPU and memory virtualization. In this well-written whitepaper fromVMware, there are three CPU virtualization techniques introduced for x86 architecture.

The x86 architecture offers four levels of privilege known as Ring 0,1,2 and 3 to operating system and applications to manage access to the computer hardware. User-level applications typically run in Ring 3, the OS must execute its privileged instructions in Ring 0 since it needs to have direct access to memory and hardware. The two main challenges with virtualizing x86 architecture are:

A virtualization layer between hardware operating system who expects Ring 0 privilege;
Some instructions with different semantics when not executed in Ring 0 cannot be virtualized effectively. They need to be translated at runtime.

These challenges makes true virtualization of x86 architecture impossible and thus VMware developed three alternative technologies.

Full virtualization (using binary translation): virtual machine presents a complete simulation of the actual hardware environment so that an unmodified guest OS can run in isolation. The Guest OS is not aware that the underlying environment it is running on is virtualized, and issues hardware calls to communicate with (what it thinks as) hardware. The virtual processors have to understand guest CPU instruction, and reproduce the equivalent CPU instructions of the host machine. VMware’s technology to address this is called Binary Translation. This overhead makes true full virtualization difficult to achieve. In real life, a virtual environment that provides “enough representation of the underlying hardware” can be considered to provide full virtualization as long as it allows guest OS to run without modification. Full virtualization comes with a performance penalty.

Paravirtualization (aka OS assisted virtualization): refers to communication between the guest OS and the hypervisor to improve performance and efficiency. In this technology, guest OS is modified with an interface to host hardware to be able to communicate and operate seamlessly. Since the guest OS is modified, the VM does not need to be a complete simulation of the hardware. The modified guest OS knows it is running on a virtualized environment, and (vm driver) makes API calls (known as ‘hyper calls’) to the hypervisor. This allows para-virtualization technology to achieve performance closer to non-virtualized environment. However, since paravirtualization cannot support unmodified operating systems, its compatibility and portability is poor.

Hardware-Assisted Virtualization: hardware vendors such as Intel and AMD both have developed extensions (new features) to simplify virtualization techniques, for example, the introduction of privileged instructions with new CPU execution mode feature to allow hypervisor to run in a new root mode below ring 0. This removed the need for full virtualization and paravirtualization. With VMware originally as a promoter of full virtualization and Xen for paravirtualization, most virtualization technologies today utilizes hardware-assisted virtualization feature, for example, Linux KVM, VMware workstation, VMware fusion, Xen, VirtualBox, etc. Intel’s virtualization extension is VT-x. AMD’s counterpart is AMD-V technology.

To virtualize memory, another level of memory virtualization is required (similar to the virtual memory support in Linux). Hypervisor is responsible for mapping guest physical memory to the actual machine memory, and it uses shadow page tables to accelerate the mappings, usually at a performance cost.

Popular hypervisors

On the market there are a few popular hypervisor technologies. They are all type 1 hypervisors:

Xen is an open-source hypervisor project originally developed in Cambridge University, licensed under GPLv2. . Based on that, Citrix developed its commercial product XenServer, a bare-metal virtualization platform with enterprise-grade features for x86 and AMD environments. Oracle VM is another commercial implementation of Xen. The Xen project also supports many cloud platforms such as Openstack, Cloudstac, etc. Xen project supports paravirtualization (Xen-PV) as well as hardware-assisted virtualization (Xen-HVM) for virtualization of X86, IA64, ARM and other CPU architectures. The earlier versions does not support memory overcommit (aka “dynamic memory optimization”, “memory ballooning“, or as Citrix calls it “dynamic memory control, DMC”). This delivers better performance but also has higher budgetary requirement on hardware since there isn’t room for over-subscription.

Hyper-V is a Microsoft product. It executes in high CPU privilege (Microsoft calls it ring -1 which is equivalent to root mode as Intel calls it). On the guest VM, OS kernel and drivers run in ring 0, application rin in ring 3. This eliminates the need for binary translation. Hyper-V does not support memory overcommit either. Hyper-V is well integrated with Windows platform. It supports Linux as well although with some performance penalty.

Linux KVM (Kernel-based Virtual Machine) is a full open-source virtualization solution for GNU/Linux. What makes KVM a special hypervisor is that it uses a loadable kernel module kvm.ko that turns itself into a hypervisor and provides VMs with direct access to the hardware. So it is a type 1 hypervisor despite of the presence of Linux OS. KVM also contains a processor specific module, kvm-intel.ko or kvm-amd.ko. KVM leverages qemu to access devices. Because KVM runs as a process inside of Linux OS, KVM can use many existing feature in Linux kernel. Redhat has an enterprise solution based on KVM.

Xen vs KVM

VMware ESXi is VMware’s premium hypervisor product (not open-source) and is available for ~~free download~~, although the advanced features are not free. (Update no free download link anymore.) VMware vSphere is virtualization platform built on top of ESXi, including a whole family of virtualization products.

Market segments and players

Virtualization involves many market segments such as virtual desktop infrastructure (VDI, for desktop virtualization), server virtualization is the predominant domain in the virtualization of data centre environment. This effort led to Hyper-Converged Infrastructure (HCI) where almost all the traditional hardware resources are software-defined through the virtualization layer. The management of infrastructure is abstracted away from the physical hardware management. The three most fundamental areas in HCI are:

Server (compute) virtualization: the previous section covers the virtualization of memory and x86 CPU, which are the main focus on computing resource virtualization. Additionally, graphics computing resources can be virtualized today. Example products include: VMware vShpere (compute virtualization based on ESXi hypervisor).

Storage Virtualization: the technology to abstract physical data storage resource to make them appear as if they were a centralized resource. Storage virtualization takes place at three levels depending on the use case: block-level, file-level and object level. Example products include: VMWare vSAN (vSphere-native storage), HPE 3PAR (Tier-1 storage), EMC VxRail, PureStorage Flash Array (Tier 1), etc. Storage Virtualization enables Software-Defined Storage (SDS), the provisioning and management of data storage independent of the underlying hardware.

Network Virtualization: the technology to abstract network resources that were traditionally delivered in hardware to software. Network virtualization decouples network services from the underlying hardware management and allows virtual provisioning of an entire network. VLAN is a classic example of network virtualization. There are also various overlay technologies such as VXLAN, which provides an industry framework for overlaying virtualized layer 2 network over layer 3 network (used in Docker network) using an encapsulation mechanism and a control plane. Example products include: VMware NSX Data Center (L2-L7 network and security virtualization platform), Cisco ACI, Palo Alto Panorama. Network Virtualization enables Software-Defined Network (SDN), an approach to network management that enables dynamic, programmatically efficient network configuration in order to improve network performance and monitoring, making it more like cloud computing than traditional network management.

Delivery model

Virtualization allows managed service providers (MSPs) to deliver IT service in the following three models:

Iaas (Infrastructure as a Service): MSP delivers VM to customers.
PaaS (Platform as a Service): MSP delivers environments to customers (e.g. Database as a Service, managed RabbitMQ service, etc).
SaaS (Software as a Service): MSP delivers entire application for the customer.

IT service delivery models enabled by virtualization technology

Since virtualization is the backbone of cloud computing. This model is also referred to as cloud computing delivery model.

Virtualization and Containerization

These two concepts are similar and could be confusing to beginners. Both provide a mechanism to isolate computing resource for different applications, for the purpose of higher utilization of resource. The difference lies in how and where the isolation is made. Virtualization requires a guest operating system per VM (OS level isolation), whereas the container technology isolates application processes along with its runtime into a container (dependency level isolation), using some new Linux kernel features such as namespaces and cgroups. All containers make their system calls to the container engine on the host operating system. So they share a kernel on the same host. In this sense, container engine running on OS could be considered as type 2 hypervisor.

From VMs to containers

VMware is a major player in enterprise data centre virtualization, which is facing fierce competition from public and private cloud vendors. VMware also has its own private cloud services. Docker is the most popular container technology that conforms to the specifications of Open Container Initiative (OCI), a governance structure for industry standards around container formats and runtimes.

Virtualization and Cloud

Among public cloud vendors, AWS EC2 used Xen PV and Xen HVM in its earlier implementations. It has transitioned to AWS bare metal. The history is well summarized here. Microsoft Azure runs Azure Hypervisor as the native hypervisor in Azure Cloud Services platform. It is a customized version of Microsoft Hyper-V specifically for Azure platform. With GCP, Google Compute Engine (GCE) instance runs VMs on KVM as hypervisor. It can also enable nested virtualization.

The scope of cloud computing is evolving overtime. It originally only refers to a business model of offering IT services (in one of the three delivery models outlined above) based on virtualization technology. Therefore I cannot make comparison between a technology and a business model. Today, with public cloud vendor extending their offerings (with various managed services and platforms) and people’s misuse of the terms, the buzz-word “cloud” seems to suggest anything that is offered in public cloud service. The essence still remain the same where managed services and managed platforms are built on top of virtualized compute unit under the hood, which are driven by virtualization technologies.

AWS Certified DevOps Engineer Exam Tips

Sat, 17 Aug 2019 23:07:59 -0400

The last 30 days have been exhausting for me. I studied hard on the new AWS Certified DevOps Engineer exam and thank goodness I passed (750 out of 1000 is required). This was the hardest professional certification I ever worked on. The exam was re-launched recently in March 2019 so there is still a shortage of information around the community. I was hoping to share my experience to help demystify this new exam.

I had a somewhat solid background to begin with, having taken the AWS Certified Solution Architect Professional exam (before the 2019 update), and worked quite a bit on CloudFormation, automation and Git. However, I still did not anticipate the exam to be this difficult until I was halfway through and already had the exam and materials paid for.

Without clear guideline on study material other than the white papers, I first skimmed through the ACloudGuru course, which helped me form a high level sense of exam coverage. Nonetheless it does not cover any topic in-depth and therefore by no means makes an essential part of my study. I checked out LinuxAcademy course and they are much more in-depth for the major topics indeed. I like the course material in Lucid chart. However, the LinuxAcademy course along does not cover everything you need to know.

What I found extremely helpful is the free training videos from the official training website. I strongly recommend the 7-hour course Exam Readiness: AWS Certified DevOps Engineer – Professional. The instructor did a great job outlining the services and knowledge areas in the assessment . The course also comes with quality sample questions with answers and explanations on what the thinkings are behind the correct answers or why some choices are obviously wrong. I went through these questions twice and feel much better at not only understanding the question, but also understanding the intent of the question.

Apart from the Exam Readiness course, other free introductory courses from the official training website are helpful as well especially for those services that you only need to know the basics. Most of those courses are 5 ~ 10 minutes long, with brief but sufficient introduction and a demo session. The other extremely helpful resource is the official practice questions. The practice exam is harder than the actual exam but they closely resemble the actual question style in the exam. Unfortunately, no answer is provided but they made me spend time finding answers across the documentations and blogs. It is worth-noting that the AWS blogs provides plenty of use cases that are covered in the questions.

When it comes to the real exam, it covers a lot more topics than its predecessor. Many questions are long and confusing. And I wish I could run a diff command to highlight the differences between choices. During exam preparation you really need to train yourself on reading efficiently. I found myself sometimes eyeball through all four choices at the same time, which get my mind scattered.

During my study, I divide all services into three categories based on the level of familiarity, and here is my list:

Category 1. Know these services very well, in and out:

AWS ElasticBeanstalk, OpsWorks Stacks, OpsWorks Chef Automate, CloudFormation, CloudWatch, CodeBuild, CodeCommit, CodeDeploy, CodePipeline, CodeStar, Lambda, API Gateway, Config, Trusted Advisor, CloudTrail, Systems Manager, Autoscaling Group in EC2, DynamoDB

Category 2. Know these services well, but not necessarily down to every single detail:

Amazon Kinesis Firehose, Kinesis Analytics, Kinesis Streams, Step Functions, Elastic Load Balancer, Secrets Manager, Serverless Application Model (SAM), Route53, RDS, Certificate Manager, ElasticSearch, ECS, ECR

Category 3. Know about these services at a high level, but do not skip any:

AWS Organization, X-Ray, GuardDuty, Macie, Inspector, Service Catalog, KMS, Batch, Athena, Single-Sign-On, Data LifeCycle Manager, CloudSearch, Health Dashboard, Glue, QuickSight, LightSail

Although I did not mention much about the white papers, I want to highlight their importance again. I would not attempt the exam without reading and understanding the required white papers.

If you aspire to take the AWS certified DevOps Engineer exam I hope this helps you a little bit. Good luck.