aws organization on Digi Hunch

Managing EC2 instances across accounts with Ansible

Mon, 27 May 2024 14:00:22 -0400

I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is consistency in the VM management, over time, and across platforms (AWS, on-prem, Azure, etc). Even for AWS shops, in the last few years many enterprises have adopted AWS landing zone with the multiple AWS account prescriptive pattern. However AWS Systems Manager still lacks integration with AWS Organization (except for a few non-core capabilities). This creates the demand of managing EC2 instances across AWS accounts. In this post, we propose a secure method to manage a fleet of EC2 instances from multiple AWS accounts, using Systems Manager . It also enables connectivity from an Ansible control node.

Prerequisites

This proposal ties a few CLI tools together, including AWS CLI, SSH, Ansible, etc. It also requires the cloud engineer to understand how they work. I’ll start with the choice of the tools.

Above, I discussed the benefit of Ansible. Since Ansible operates on SSH, we’ll still have to use SSH tools. Even though SSM agent provides a way to connect to EC2 instance without requiring an RSA key pair, we still need SSH since it is a well-established industry standard (RFC4253) and the foundation of Ansible. These two technologies are not mutually exclusive. In fact, the SSM agent provides a secure enhancement to the operation with SSH. Traditionally, on each EC2 instance we’d have to run SSHD services which opens TCP port 22 (or alternative TCP port as configured). For authentication we favour key pair over password but the open port is still an attack surface vulnerable to brute force and DDoS attacks. For EC2 instances on private networks there is no reachability to the instance’s SSH port unless the bastion is also in a connected network.

As I cover in a post, the SSM Session Manager comes in handy. The SSM agent operates from the instance and communicate outbound to AWS backend. Since the SSM agent runs under a privileged user on the OS, you can perform OS-level commands through SSM. Further, AWS developed a Session Manager plugin with AWS CLI, allowing AWS CLI as a proxy command when making an SSH connection. Therefore SSM enables SSH connection without requiring port 22 to be open. In addition we’ll need to use RSA key pair as required for SSH, which is also an improvement to the security posture.

That explains the dependent tools. On the Ansible control node, apart from Ansible itself, we need the latest version of AWS CLI with the Session Manager plugin, we need to configure AWS CLI properly to connect to EC2 instances across multiple AWS accounts.

Configure AWS CLI

This section discusses how to configure AWS CLI. I have a couple of handy aliases for productivity but they are not essential. For example, I often need to check the IAM identity making the call, and I often need to list out all profiles configured. So I added the following two entries in the ~/.aws/cli/alias file:

[toplevel]
whoami = sts get-caller-identity --no-cli-pager --output yaml
profile = configure list-profiles

With that I have an alias to check IAM identity and available profiles. Then we can start configuring the profiles for CLI (in the file ~/.aws/config). Since we’ll be working with multiple AWS accounts, we have to manage multiple CLI profiles, which implies that:

We better use the --profile switch to explicitly specify profile being used, instead of relying on the AWS_PROFILE environment variable;
As a security best practice, we should not configure profiles with long-term IAM credential in the config file;
We must ensure the CLI doesn’t prompt for log-in every time we switch profile

To satisfy #3 there are many ways but we’ll discuss two: using cross-account IAM role, and using AWS SSO.

Bonus point if you enable auto-complete for AWS CLI.

Configure AWS CLI Profiles

With cross-account IAM role, the idea, is that the client start with one IAM identity, and use that IAM identity to assume roles on several other accounts. The configuration looks like this:

[profile jump_account]
credential_process = /opt/bin/awscreds-custom --username helen

[profile target_account_1]
role_arn = arn:aws:iam::123456789011:role/OrganizationAccountAccessRole
source_profile = jump_account

[profile target_account_2]
role_arn = arn:aws:iam::123456789012:role/OrganizationAccountAccessRole
source_profile = jump_account

In this example, you start with an validated identity in the jump account, then assume a privileged IAM role named OrganizationAccountAccessRole on the target accounts. Typically such IAM roles are pre-configured (e.g. in an multi-account landing zone) with appropriate trust policy to allow principals from the jump account. Once you’re validated as the IAM identity in the jump account, then you can use profiles for target accounts without being prompted for credentials again.

If your have configured IAM Identity Center for the multi-account environment, consider an alternative approach using sso login. The configuration usually looks like this:

[sso-session sso]
sso_start_url = https://myorg.awsapps.com/start/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile target_account_1]
sso_session = sso
sso_account_id = 123456789011
sso_role_name = AWSAdministratorAccess

[profile target_account_2]
sso_session = sso
sso_account_id = 123456789012
sso_role_name = AWSAdministratorAccess

This is often used by human users with SSO credential. In this example, to authenticate the sso session, start with command “aws sso login” . Then you can use all profiles by specifying --profile switch without having to log in again.

Configure SSH to EC2 via SSM

The EC2 instance must connect to Systems Manager endpoint, before one can SSH to the instance using the plugin. Once connected, you should find the instance in Fleet Manager. For this to happen, there are a few conditions. First, the instance must be able to reach the endpoint, either via public Internet, or via VPC interface endpoints if in a private subnet. Second, the instance profile must contain an IAM role with appropriate permissions. We can use AWS managed policy AmazonSSMManagedInstanceCore in the role. In addition, if we record the SSM session to an S3 bucket with encryption the instance profile must have permission to use the encryption key.

SSM agent uses the IAM role. The agent runs as a service on Linux or Windows machines. Many AWS managed AMIs come with SSM agent pre-installed. If that is not the case, you’d install the agent in your own AMI, or in user data which requires downloading the installer. With these configuration you’d be able to connect to the instance via SSM. We can use AWS CLI SSM command, or AWS web console to start an SSH session. To use SSH CLI utility, we install the session manager plugin along with AWS CLI on the SSH client machine. We also specify a public key for EC2 instance and run ssh command with the private key. The SSH configuration needs a configuration such as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

With this entry, you may directly SSH by instance ID (usually starting with i-* and mi-*), and the specified Proxy Command with SSM session document AWS-StartSSHSession will be invoked.

Configure Ansible Inventory

We can SSH to an instance (without port 22 open, on top of SSM) using the method above. Similarly, we can also configure Ansible to connect to the instance, without port 22. The inventory configuration looks like this:

mytest:
 hosts:
 instance1:
 ansible_host: i-00aabbffcc7755221
 ansible_user: ubuntu
 ansible_ssh_common_args: -o ProxyCommand="aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1"
 instance2:
 ansible_host: i-eedd88ff66aa22442
 ansible_user: ubuntu
 ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ProxyCommand="sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --profile target_account_2 \""

Note I used two patterns for ansible_ssh_common_args that are similar. Both works. Each entry references its own profile. This is necessary because Ansible does not have the knowledge which instance belongs to which profile’s account.

Another way to get Ansible connect to instances is to use the connection plugin community.aws.aws_ssm, by specifying ansible_connection: aws_ssm (e.g. as host variable) and other required variables (e.g. profile). This method does not require SSH channel but it requires an S3 bucket, and hence IAM permission on the controller node.

Ansible supports dynamic inventory, in three ways: inventory source file (with existing plugin), custom inventory plugin, and inventory script (in Python). Take source file as an example for EC2, add the followings as the content of aws_ec2.yaml:

plugin: amazon.aws.aws_ec2

# Attach the default AWS profile
aws_profile: target_account_1

compose:
 ansible_host: instance_id
 ansible_user: "'ubuntu'"
 ansible_ssh_common_args: "'-o ProxyCommand=\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1 \"'"

Then we can display the rendered inventory list, and Ansible-ping the instances.

ansible-inventory -i aws_ec2.yaml --list -y | less
ansible all -i aws_ec2.yaml -m ping

For more flexibility, for the composed variables, we could use jinja2 expression to generate the value. In both ways, we produce an inventory source per profile using the built-in inventory plugin aws_ec2. For greater flexibility, such as consolidating instances from all accounts into a single inventory, consider writing your own inventory script, or even own inventory plugin.

Conclusion

In this post we propose a way to manage instances across AWS accounts. Two main challenges are establishing the communication channel (SSH on top of SSM) and generating inventory data in Ansible. Some AWS services can generate inventory data, such as resource data sync in Systems Manager, or using AWS Config Aggregator. It is unfortunate that neither way produces the inventory data in a format that is directly compatible with Ansible inventory. Therefore, you might have to create a custom Ansible dynamic inventory script (in Python) that reads from the inventory data from AWS Config Aggregator (which supports AWS Organization). The side benefit of this script is that it is usually faster than the built-in aws_ec2 inventory plugin.

Orchestrate Landing Zone with AWS Control Tower

Sat, 19 Aug 2023 17:25:00 -0400

Following an introduction to AWS Landing Zone, I’ll dive deeper into Control Tower as an orchestration tool in this post.

More on Landing Zone

In data center operation, there are numerous tasks that other teams have to complete before the the deployment of an application. For example, the 42U server cabinet must be in place with dual powers. The cabinet comes with a network switch in the middle and each ethernet port must be provisioned. Once the server is connected to the central network, the NOC team assigns IP address, configure dynamic VLAN on the switches, and configures firewalls etc according to the connectivity requirement. The new server also needs to report to centralized monitoring solutions such as SolarWinds. In cloud operations, the scope and target of a landing zone is similar to those data center operations work, with networking being the core. The idea is that the landing zone ensures security, compliance and governance, so that applications (analogous to paratroopers) can focus on their primary responsibility.

Although Landing Zone is a general concept for any cloud service provider (CSP), each CSP has some prescriptive guidances on setting up landing zone in their particular environment. For example, an AWS prescriptive landing zone typically covers the following apsects:

Multi-account structure
Identity and Access Management
Governance (controls and guardrails)
Networking
Additional Security Services

The multi-account best practice is an important aspect in AWS as the OU structure dictates how effective Service Control Policies (SCPs) can govern the entire footprint. A landing zone orchestration solution should also apply guardrails and controls based on the organization’s security and compliance requirement. In addition, it is common expectation that landing zone orchestration solutions create required networking resources such that applications are ready to deploy securely.

This post is about Control Tower.

Intro to Control Tower

Control Tower is a landing zone orchestration solution available as an AWS service. I have three impressions over Control Tower. First, it is good with managing multi-account structure. It integrates closely with AWS Organizations and requires client to have a dedicated log archive account and a dedicated security tooling account, which also serves as audit account.

The second impression is Control Tower makes governance more straightforward. AWS seems to use the term control and guardrail interchangeably. Below is an illustration:

The detective controls only captures and reports violations. The preventive controls stops the violating API request. The proactive controls remediates the detected violations. Another way to look at the controls are the priority of the controls:

Mandatory controls: there are about 23 mandatory controls that are enforced on each account. As soon as a new workload account becomes part of Control Tower, these controls will come enforced.
Optional controls including strongly recommended controls and Elective controls.

There are many existing controls and AWS is still releasing new one (example). AWS Config service uses conformance pack to organize relevant controls. For example, there are sample conformance packs such as “Operational Best Practices for “NIST 800 53 rev4” and “Security Best Practices for EKS”. In addition, you can even build your own conformance pack. Once you deploy a conformance pack, AWS config deploys additional rules against the current environment.

The third impression is that Control Tower falls short with infrastructure automation. It provides a number of customization mechanism to leave infrastructure (mostly networking) automation with users. I will explain later.

Mandatory Accounts

The function of Control Tower does not depend on a specific account structure that aligns with the multi-account best practice. However it does require three mandatory accounts to set up automatically when creating control tower. The account where Control Tower is configured is the management account. The log archive account is the owner of S3 buckets that hosts buckets for loggings. The audit account is a restricted account that’s designed to give your security and compliance teams read and write access to all accounts in your landing zone. We also use this account as delegated administrator account for several security services at organization level. I’ll elaborate in the next section.

The management account is directly under root OU and there is no SCPs applied to the management account. Also Control Tower does not turn on AWS Config recorder and configure delivery channel on the management account itself.

The log archive account and audit account are under the Security OU (or otherwise named during the setup). At the end of Control Tower setup a set of mandatory controls will be applied at the OU level and thus effective to each account.

Apart from the mandatory accounts, oftentimes there is an Infrastructure OU. Typically we place shared services account under this OU. The account is for common services such as central networking, managed Active Directory, DevOps pipeline etc. Another optional OU is workload OU, where we can create new workload accounts or enrol existing workload accounts.

Each enrolled workload account have its CloudTrail configured with an organization-level trail, which sends log to the central bucket that log archive account owns. Each enrolled account also has AWS Config recorder configured with a delivery channel pointing to the configuration log bucket also owned by log archive account. Note that Control Tower does not turn on recorder on the management account itself.

Account Enrolment

Most likely a customer is already in use of AWS Organization. They can delegate one account as management account, and use Control Tower console to create a landing zone for the intended regions. During the creation, they will have to create two additional account, one as log archive account, and the other as security tooling or audit account. This is in alignment with multi-account best practices. After the creation of landing zone, the two new account are automatically enrolled in Control Tower.

Going forward, the client should create new accounts using Control Tower’s Account Factory to save a separate enrolment step. For existing accounts and OUs we’ll need to enroll them into Control Tower. At the beginning, the Control Tower Landing Zone (CTLZ) only has the mandatory controls, which are the bare minimum governance. When we enroll existing OUs and accounts, these controls (mostly detective and preventive) will extend to the newly enrolled OUs or accounts. Therefore, it is important to not introduce numerous controls prior to having all accounts and OUs enrolled. It is important to have a proper OU hierarchy upfront because both detective and preventive controls (Config Rules and SCPs) are typically applied at OU level and effective to all the children of the the OU.

To enroll an existing account, the account must meet some prerequisites. An IAM role with cross-account trust must be manually created. Since each account can have only one AWS Config configuration recorder and delivery channel, if an existing account already has one of them, they must be manually deleted using CLI.

Account Factory defines what happens when we create a new account and it is where we can bake in the customizations. When it comes to infrastructure automation, Control Tower leaves pretty much everything up to the account factory customization.

Delegated Administrator

Many AWS services are available as organization wide service. For example, CloudTrail, GuardDuty and even AWS Organization. Most of these organization level services allows you to specify one account as delegated administrator for the entire organization. For example, for System Manager and Service Catalog, the best practice is to designate an operation account or shared services account as the delegated administrator.

There are several security-related, organization-wide services, such as GuardDuty, Detective, Macie, Security Hub, Inspector, Audit Manager, and Firewall Manager. The best practices for these services is to designate the audit account as the delegated administrator. We think of the audit account as an aggregation point (or points for organizations that split the functionality across multiple accounts) for these AWS services. You can find this recommendation from the documentation of each service. However, currently Control Tower does not enforce this best practice. Even though Control Tower dashboard gives you visibility to security services such as GuardDuty and SecurityHub, you are still on your own to configure these services outside of Control Tower.

Whether you had never used these services, or you had previously designated another account as administrator for these services, it is good time to align with the best practice for delegated administrator right after Control Tower setup.

If you had previously designated other accounts, you need to take additional steps to revoke the old delegation and designate new delegated administrator account. These general steps can be performed from console or CLI and apply to all these services:

From the old administrator account, remove all the members
From the management account, revoke delegation
From the management account, delegate new administrator account (audit account)
From the new administrator account, invite all the members, and make sure to tell the service to automatically include new accounts going forward

Note that when you invite an account, that account must not have created, invited or associated with other accounts (as a member). In other words, we should never let a non-delegated administrator account create, invite or associate other account as member.

Control Tower Customization

Control Tower by itself does not play a big role in networking automation. You can create a VPC when provisioning a new account in Control Tower. That’s about what you can do. It does not address how the new VPC connects to existing networking space. It does not distinguish the existing VPC topology. To address this, Control Tower has a few customization options:

	AFC Account Factory Customization	AFT Account Factory Terraform	CfCT Customizations for Control Tower
Supported IaC language	Anything that Service Catalog Support (e.g. CloudFormation and Terraform HCL)	Terraform HCL	CloudFormation
Readiness	AFC is a native mechanism. You can specify the Service Catalog product during creation of an Account in Control Tower console.	Users need to first bootstrap the solution, which creates Terraform pipelines and other components.	Users need to first bootstrap the solution which creates a pipeline and other components.
How it works	Control Tower launches the specified Service Catalog product (aka blueprint) during account creation.	Quite complex. Explained in the section below	Control Tower posts a lifecycle events to Amazon EventBridge. A lambda function will process the event using pre-baked CloudFormation templates and Step Functions.
Prebuilt customizations	No. However, for certain products, there are some partner-maintained blueprints	Yes	No

AFC is fairly straightforward to use based on this post, as long as you know Service Catalog. It is a native capability of Control Tower. The other two ways for CT landing zone customization are not native capability and each require a separate bootstrapping process to deploy the pipeline-based solution.

Customizations for Control Tower

In CfCT, we first bootstrap the solution from management account, using this CloudFormation template on GitHub. We usually need to version control the custom configuration so we should select “AWS CodeCommit” as the value for the CodePipelineSource parameter. The bootstrapping (execution of this CloudFormation template) takes about 5 minutes.

The solution is based on Control Tower’s lifecycle event to trigger the pipeline with a source code repository (or S3 bucket). The source code repository (AWS CodeCommit) stores the custom configuration. A custom configuration consists of a manifest file in YAML format, made up with a number of resource sections. Each resource section references an artifact, either a CloudFormation template or a policy file to apply. The artifact can either be stored in the same repository, or in a remote S3 bucket.

Account Factory Terraform

Introduced in late 2021, the AFT solution is based on the terraform-aws-control_tower_account_factory repository. It is too complex as a solution in my opinion. At the beginning you need to have a designated OU and account for AFT and install the solution itself with Terraform. Note that, the solution does not address the state storage of the installation of the solution itself (chicken-or-egg). The installation also installed a few IAM roles required on the AFT management account. Having a landing zone management account and a AFT management account is not a neat setup.

Once installed, the solution consists of four repositories. Their purposes are as follows (copied from workshop instruction):

Account requests – handles placing or updating account requests. See example here.
AFT account provisioning customizations – manages non-Terraform customizations that are applied to all accounts. This stage runs before the global customizations stage. Examples available here
Global customizations – Global customizations – manages Terraform-based customizations that are applied to all accounts created by and managed with AFT. Examples available
Account customizations – Account customizations – manages Terraform-based customizations that are applied only to specific accounts created by and managed with AFT. Examples available

With this solution in place, when we create a new account via account request repo, quite a number of steps will happen after that, as the workshop instruction illustrated:

When I first looked into how this work I’m very concerned about the maintainability of this “solution”. The amount of services and pipelines involved in this solution makes it difficult to troubleshoot end to end. To improve that AWS even added an enhancement for request tracing. Nonetheless, I would not go with AFT just with how complex it looks like. For more details, check out the Control Tower workshop.

Summary

Control Tower helps you set up a Landing Zone without network infrastructure automation. It provides a couple of customization mechanisms, allowing you to bake in your own infrastructure automation. To build a landing zone, you can go with Control Tower in the following situations:

you have general regulatory requirement and need to enable governance capabilities
your networking stack isn’t centrally managed or isn’t large enough to warrant infrastructure automation
If you do need networking automation, then you will need the customization capability of Control Tower. In this case, your team should have good handle on infrastructure as code (CloudFormation or Terraform)

Out of the customization options, AFC seems the least complex to me. The other two, especially AFT, is too complex. We use pipelines to trail and error with other stacks, and the pipeline solution itself must be simple.

On the other hand, if you environment has strong regulatory requirement, or you are seeking a prescriptive network architecture, or your team does not have the capacity with CloudFormation or Terraform Templates, consider Landing Zone Accelerator on AWS.

Landing Zone in AWS – An Introduction

Thu, 08 Dec 2022 22:37:00 -0400

Cloud adoption has gone through phases. Hashicorp’s CTO Armon Dadgar has a great stream on Hashcorp’s narrative of the three Phases of Cloud Adoption:

Phase 1, with main focus on agility, app teams in wild west, account sprawl, inconsistent configuration, security & compliance challenge
Phase 2. consistent platform layer providing an opinionated way of configurations, security and compliance control, introduction of platform team, whose customers are application team, scaling challenges with platform team capacity not keeping up with application team’s demand
Phase 3. self-service platform at scale for many application teams. CICD, infra-as-code

New greenfield clients today should aim at phase 2 or phase 3 depending on their target operation size. Either way, they need a landing zone for consistency. This post focuses on the landing zone options on AWS and the key constructs.

Overview

AWS Prescriptive Guidance defines landing zone as a well-architected (secure, scalable, compliant, etc), multi-account AWS environment that is a secure baseline from which you can deploy workloads and applications. A landing zone may consists of:

Multi-account architecture
Identity and Access management solution
Governance, compliance, logging and auditing solutions
Security and networking design

Landing zone reflects an enterprise opinion on how to configure networking and IAM. Establishing a landing zone can be a very involving process. In early days of cloud operation people used build landing zone by “clickops” on AWS console, which cannot keep up with the growth of landing zone and associated security services. Compared to SMB clients, some sectors such healthcare and national security have much more regulations and compliance frameworks.

In fact, landing zone is such a buzzword that I have learned to be very sensitive to the context. If a solution has words “landing zone” in its name, given the complexity and loose use of the words, there is a good chance that the solution only delivers some of the aspects above. Cloud consultants are still to address the gaps. Therefore I decided to write this post about what I learned about landing zone in AWS.

Multi-account and Organization

Since 2017, AWS has been officially advocating the use of multiple accounts as a best practice and security strategy. They encourage clients to view an account as a resource container, just like Resource Group in Azure (here is an article on the account structure between AWS and Azure) Client builds an account for security boundary and financial container. Many blog post came along (such as this one) on the implementation details. From tooling perspective, AWS launched AWS Organization in 2017 to facilitate multi-account management. This presentation from re:Inforce 2019 is a good material to understand multi-account environment with AWS Organizations. In 2021, AWS published a new white paper on best practices with multiple accounts. My previous coworkers authored two blog posts here and here to reflect multi-account setup as of late 2020.

AWS Organization is an account management service that provides a vehicle to centrally manage AWS accounts by groups, which brings many benefits, such as centralized logging, compliance management, consolidated billing, etc. As an administrator, you can create accounts in your organization and invite existing accounts to join the organization.

Typical account structure involves:

A root organization named Root
A organization hierarchy with one or more OUs under root, with each OU having one or more child OUs.
Each OU can have multiple accounts, with each account having one email address.

You can also configure service control policies (SCPs), a type of organization-level policy that you can use to manage permissions in your organization. SCPs offer central control over the maximum available permissions for all accounts in your organization. SCPs are a means of implementing guardrails in your AWS organization.

To vend multiple account automatically, AWS introduced the Account Factory (which became part of Control Tower later in 2019) and let users create new account from AWS console and specify which OU it belongs to. It also allows users to implement customization after account creation with Service Catalog products. Some legacy orchestration solution (e.g. Augmented Account Factory) were based on this mechanism. Another orchestration solution prior to the launch of Control Tower was the AWS Landing Zone solution (ALZ, introduced in 2018), which uses AWS CodePipeline to provision accounts and deploy resources.

From Landing Zone Solution (ALZ) to Control Tower

The launch of Control Tower was a game changer in 2019 to bring users to multi-account best practices. Control Tower, as an AWS product (instead of a solution by some service teams at AWS) matured over several years. It gradually deprecated the previous generation orchestration solutions. Newer orchestration solutions all have to support Control Tower. According to this page, we can customize Control Tower based Landing Zone in these ways:

AWS Control Tower console:
- Instead of creating OU in AWS organization, do it under Control Tower in the console, under “create required OUs”.
Outside of AWS Control Tower console
- using Account Factory for Terraform (AFT): Terraform-based account provisioning pipeline, for heavy Terraform shops.
- using Customizations for AWS Control Tower (CfCT) solution. The LZ created remains in sync with Control Tower.
  - You first launch a standardized CF stack to set up the mechanisms for customization.
  - Then you create a custom package to define the customization. This includes a manifest file.

On the other hand, the original ALZ solution is currently in long-term support and will not receive any additional features. It is deprecated. AWS advises its customers migrate to AWS Control Tower based landing zone. The ALZ page is redirected to a page about customizing Control Tower landing zone.

The Control Tower based landing zone configures OUs, accounts, SSO and guardrails. It however does not entail networking design except for the guardrails.

Security Reference Architecture (SRA)

SRA is just a reference architecture and accompanying recommendations on AWS security services, and how they work together in a multi-account environment to host a single-page application. It comes with a repository to demonstrate how to configure a secure multi-account environment with Control Tower, CfCT, as well as security services. Alternatively, you can deploy it with CloudFormation StackSets. For fully automated deployment of this architecture, check out Landing Zone Accelerator down below.

One important feature that reflects the notion of centralized security in a multi-VPC topology, is the use of a dedicated VPC for centralized interface endpoints. This pattern is seen in all generations of security reference architectures.

AWS Security Environment Accelerator (ASEA)

Regulated customers often find they need to add additional controls and capabilities to be defined and setup outside of Control Tower. ASEA, as an orchestration solution, aims to remove the complexity of having to develop and maintain a separate codebase to manage the additional customizations, by providing a tool to help deploy and operate secure multi-account, multi-region AWS environments on an ongoing basis.

Overtime, as Control Tower introduces new capabilities to support the customizations required in heavily regulated environments, the capabilities will be removed from ASEA and enabled directly within the Control Tower managed service, further reducing operational burden. Read this for its relationship with ALZ and Control Tower.

ASEA was first released in late 2020. It covers more on networking design and has a fairly comprehensive installation process. ASEA primarily cater to government of Canada‘s PBMM Security Configuration Profile with an opinionated configuration. The first few revisions were referred to as PMBB architecture. While it provides a great reference architecture for highly regulated landing zone, it has not gotten much traction elsewhere. The recommendation going forward, is to use Landing Zone Accelerator, which incorporates the features and lessons learned from ASEA.

Landing Zone Accelerator (LZA)

First released in May 2022, the Landing Zone Accelerator on AWS solution deploys a cloud foundation that is architected to align with AWS best practices and multiple global compliance frameworks. LZA operates on top of Control Tower managed landing zone. This page in the documentation brings a good explanation of how it works. Basically it employs CodeBuild as an orchestration engine, and leverages CDK to drive resource deployment. There are two repositories: the GitHub repository for Landing Zone Accelerator itself, and a CodeCommit configuration repository provisioned during preparation.

As document states, LZA is a fully automated implementation of the architecture guidelines documented in the SRA. LZA also incorporates features and lessons learned from ASEA and Compliant Framework for Federal and DoD Workloads in GovCloud (US), neither of which are recommended for new deployment. In addition, LZA aims to enable iterations and extensions of a secure environment over time. The vision is to eventually replace AFT, CfCT, and ASEA. As AWS releases newer versions of LZA, client should be able to upgrade it in a pipeline run.

Baseline architecture for LZA

The Landing Zone Accelerator project also provides a samples configurations in each regulated frameworks. For example, the healthcare best practice come up in Oct 2022 for healthcare industry. The healthcare best practice sample incorporates healthcare specific configurations, such as the detective guardrails defined in the Operational Best Practices for HIPAA Security conformance pack. To deploy the best practices, modify the configuration in config repo, and run the pipeline again.

Landing Zone Orchestration Options

To summarize, we have the following landing zone options in AWS:

Orchestration Layer	Lifecycle	Summary
Solutions based on account factory with Service Catalog (e.g. ALZ, Augmented Account Factory)	All solutions pre-dates the launch of control tower have been or will soon be deprecated.	Pre-Control Tower solutions should migrate to current alternatives.
AWS Control Tower	GA in June 2019	To customize the landing zone, use AWS console, or alternatively, one of the following three options: – CfCT: Customization for Control Tower – AFT: Account Factory Terraform – AFC: Account Factory Customizations For example, CfCT can be used to deploy Security Reference Architecture (SRA)
AWS Secure Environment Accelerator (ASEA)	Released 2020	Canadian Centre for Cyber Security (CCCS) Medium Cloud Control Profile, formerly known as PBMM. This approach will be replaced by LZA
AWS Landing Zone Accelerator (LZA)	Released in 2022	A low-code deployment option. Samples provided in support of reference architectures that align with industry best practices or compliance frameworks. Examples for industry best practices: – general best practice – healthcare – finance and tax – education Example for compliance frameworks: – US state local government – FedRAMP for US Federal and Department of Defence (DoD) – CCCS Medium for Canadian government – AWS Trusted Secure Enclave (TSE) Sensitive Edition (SE), which also aligns with other medium level security profiles such as NIST 800-53, ITSG-33, FedRAMP moderate, CCCS-Medium, IRAP, etc

If your organization has a compliance framework that LZA supports, it makes sense to start with LZA. Otherwise, it is sufficient to use Control Tower for multi-account setup.

Networking constructs

Landing zone involves multiple VPCs so it’s important to understand VPC peering and Transit Gateway. VPC peering can only be setup between two VPCs and transitive peering relationship is not supported. For 6 VPCs to talk to all each other, we’d need 15 peering setups, which is not sustainable. We have two options:

We can connect many VPCs to a single Transit Gateway and the VPCs will be able to talk to each other. We can also connect Transit Gateway to site-to-site VPN or Direct Connect. This is a good explanation.
We can still use peering, but dedicate one VPC as “Transit VPC” in a hub and spoke model.

When you attach a VPC to a transit gateway, you must specify one subnet from each AZ to be used by the transit gateway to route traffic. Specifying one subnet from an AZ enables traffic to reach resources in every subnet in that AZ.

Here is a comparison table. Read the white-paper “Building a Scalable and Secure Multi-VPC AWS Network Infrastructure” for more about network design. For example, Some clients need to inspect traffic. When deploying multiple VPCs, we also need to ensure DNS resolution works across VPCs, and between on-prem networks, by configuring Amazon DNS server.

Our VPCs also need to connect to managed AWS services. For services like S3 or DynamoDB (of the same or different account), workload in VPC can access them via public DNS. The traffic goes through Internet Gateway of VPC and then public Internet. It is neither secure nor economical. We often want all network traffic to stay on the global AWS backbone. There are three types of Endpoint under VPC to help us.

	Gateway VPC Endpoint or Gateway Endpoint	Interface VPC Endpoint or Interface Endpoint	Gateway Load Balancer Endpoint
Purpose	VPC access native AWS services	VPC access native AWS services	VPC access your own service (aka Endpoint Service)
Enabled by Private Link	No	Yes	Yes
Traffic remain on AWS network	Yes	Yes	Yes
Mechanism	Use the public IP address of the service along with configuration in routing table to access target resource. The routing table acts as gateway. You can use the public DNS name of the service.	Use private IP address from the VPC to access the target service. Require endpoint-specific DNS name for the target service. Incur extra charge. S3 as example. More secure because there is ENI in the VPC controlled by security group.	Your Endpoint Services is hosted in front of a fleet of network virtual appliances. You can select endpoint type as you create it in console under VPC
Access from clients on premise or from other region	Does NOT allow access from clients on premise or in VPC from other region	Allows access from clients on premise or clients in VPC from other region	Allowed

Private Link integrates with a subset of AWS services. To check the list of interface endpoint, use:

aws ec2 describe-vpc-endpoint-services --query "ServiceDetails[?ServiceType[0].ServiceType=='Interface'].ServiceName"

Given the inter-VPC connectivity, most deployment centralizes interface endpoint into a dedicated VPC.

Centralized Interface Endpoints

There are several benefits to use a single VPC as dedicated provider of interface endpoint in a multi-VPC topology. First, the interface endpoints incurs a standing charge and it makes financial sense to consolidate them in one VPC. Second, this setup centralizes the configuration and security aspects as well.

In most cases, interface endpoint services follow the format of com.amazonaws.<region>.<endpoint>, with the dns name looking like: <endpoint>.<region>.amazonaws.com. For example:

Endpoint Service Name: com.amazonaws.us-east-1.ssm
Private Hosted Zone: ssm.us-east-1.amazonaws.com
Zone Record (A alias): ssm.us-east-1.amazonaws.com

However, there are a few exceptions to that which can make it tricky to implement interface endpoint with infrastructure as code.

Exception 1. Private DNS name suffix is api.aws instead of amazonaws.com:

Endpoint Service Name: com.amazonaws.us-east-1.eks-auth
Private Hosted Zone: eks-auth.us-east-1.api.aws
Zone Record (A alias): eks-auth.us-east-1.api.aws

Exception 2. Endpoint Service Name doesn’t start with com.amazonaws

Endpoint Service Name: aws.sagemaker.us-east-1.notebook
Private Hosted Zone: notebook.us-east-1.sagemaker.aws
Zone Record (A alias): notebook.us-east-1.sagemaker.aws

Exception 3. Two A-records are required under the same PHZ

Endpoint Service Name: com.amazonaws.us-east-1.ecr.dkr
Private Hosted Zone: dkr.ecr.us-east-1.amazonaws.com
Zone Record (A alias): dkr.ecr.us-east-1.amazonaws.com and *.dkr.ecr.us-east-1.amazonaws.com

Exception 4. Two PHZs are required:

Endpoint Service Name: com.amazonaws.us-east-1.ec2
Private Hosted Zone: ec2.us-east-1.amazonaws.com and ec2.us-east-1.api.aws
Zone Record (A alias): ec2.us-east-1.amazonaws.com and ec2.us-east-1.api.aws

The infrastructure code that handles interface endpoints should be able to take care of the exceptions.

Security Constructs

AWS Network firewall is a configuration under VPC. You associate a Network Firewall with one or more subnets in the VPC. You also associate the Network Firewall with Firewall Policies. Each policy consists of rule groups. Each rule blocks or filters traffic. The log can be published to CloudWatch or S3 via Kinesis. Usecases of AWS Network firewall include:

inspect VPC-to-VPC traffic;
filter outbound traffic;
prevent inbound internet traffic;
secure AWS Direct Connect and VPN traffic

When you associate a firewall to your VPC, you must provide a subnet for each Availability Zone where you want to place a firewall endpoint to filter traffic

There are many points of configuration for firewall rules in AWS. AWS Firewall Manager is a place for central management. It connects with other services such as:

AWS WAF (L7)
AWS Network Firewall (L4)
AWS Shield (DDos)
Amazon Route 53 Resolver DNS Firewall
Security Groups
Third party firewall support

IAM Identity Center (previously AWS SSO) is for logging into AWS portal, giving each identity from an external identity store (such as AD) an identity on an AWS account. It addresses several challenges as a result of having multiple AWS accounts.

First, a user from AD needs to access multiple accounts. Second, in each account with access, s/he needs to have an IAM role. These IAM roles are based on attributes of his/her external identity, and can be different per AWS account. With Permission Set IAM Identity Center:

Allow you to login on different AWS account using the same credential
Allow you to federate with external identity store( e.g. using SAML)
Manage permission set for each account.

This video has a good walk-through of how to configure Azure AD as identity store, and tie it to specific AWS accounts and specify permission set.

Note that IAM Identity Center has a different purpose than Cognito. Cognito is to connect your app with an IdP. Your app can be an EC2-hosted application, serverless application on API gateway, or container-based application on Kubernetes, as long as the support open identity standards (e.g. OAuth 2.0, SAML 2.0 and OIDC). Cognito has two pools:

User pool for authentication. With a user pool, your app
Identity pool for authorization

So Cognito is for AuthN & AuthZ to your own app’s endpoint (business traffic) and it supports a number of standards. IAM Identity Center is for AuthN & AuthZ to your AWS account (management traffic). If you use it for your own application, your application user will have direct access your AWS resources. It mainly supports SAML (and OIDC but less used) and is commonly connected with Active Directory (Microsoft or AWS).

One can use AWS Directory Service to host a compatible and managed directory service. AWS Directory Service supports four modes:

AWS managed Microsoft AD: actual Microsoft Active Directory in AWS Cloud
Simple AD, powered by Linux-Samba Active Directory-compatible server
AD Connector: a proxy for redirecting directory request to your existing Microsoft AD without caching any information in the cloud
Cognito user pools

If you combine two AD domains, you will need domain trust.

Other security services

In addition to the core services above, LZA involves the following four services as well.

AWS Macie is a fully managed data security and privacy service based on ML and pattern matching. It continually evaluates your S3 environments to discovery PII and act on them. It also reports alerts on unencrypted buckets, publicly accessible buckets, etc.

AWS GuardDuty continuously analyze S3, container & instance workloads, user and accounts for potential threads, across account, based on ML for intelligent threat detection. In addition, GuardDuty also acts on findings.

AWS Config is an essential service that traces resource inventory, their changes and monitors for compliance:

When you turn on AWS Config, it first discovers the supported AWS resources in your account and generates a configuration item (poin-in-time view of attributes) for each resource. AWS Config also generates configuration items when the configuration of a resource changes, and it maintains historical records of the configuration items of your resources from the time you start the configuration recorder.
On an ongoing basis, AWS Config keeps track of all changes to your resources, whether or not it is initiated by the API
If you are using AWS Config rules, AWS Config continuously evaluates your AWS resource configurations for desired settings. You can deploy several related rules in a pre-built compliance pack.

As you may have noticed, there are many AWS services related to firewall, security and compliance. AWS Security Hub aims to be a a consolidated view of your security status in AWS. You can automate security checks, manage security findings, and identify the highest priority security issues across accounts in client environment. It can also:

consolidate security findings from GuardDuty, Inspector (vulnerability scanner and management), Macie, Config, Systems Manager, Firewall Manager, IAM Access Analyzer and other Integrated APN solutions
Check findings against best practices
Client can take action: i.e. investigate findings or take response and remediation actions

Summary

Landing zone deployment can take numerous iterations to finalize the requirement. It is important to have a vision of the client’s cloud operating model, which is usually discussed prior to implementation. At the implementation phase, I discuss the topic on two more separate posts on control tower and landing zone accelerator.