<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>multi-account on Digi Hunch</title><link>https://www.digihunch.com/tag/multi-account/</link><description>Recent content in multi-account on Digi Hunch</description><generator>Hugo -- gohugo.io</generator><language>en-US</language><lastBuildDate>Fri, 04 Apr 2025 22:58:29 -0400</lastBuildDate><atom:link href="https://www.digihunch.com/tag/multi-account/index.xml" rel="self" type="application/rss+xml"/><item><title>Managing EC2 instances across accounts with Ansible</title><link>https://www.digihunch.com/2024/05/managing-ec2-instances-across-aws-accounts-ssm/</link><pubDate>Mon, 27 May 2024 14:00:22 -0400</pubDate><guid>https://www.digihunch.com/2024/05/managing-ec2-instances-across-aws-accounts-ssm/</guid><description>&lt;img src="https://www.digihunch.com/wp-content/uploads/2025/04/feature-ec2-ssm.webp" alt="Featured image of post Managing EC2 instances across accounts with Ansible" /&gt;&lt;p class="wp-block-paragraph"&gt;I regard &lt;a href="https://aws.amazon.com/systems-manager/"&gt;AWS Systems Manager&lt;/a&gt; as &lt;a href="https://www.digihunch.com/2023/10/the-systems-manager-hodgepodge/"&gt;omnipotent&lt;/a&gt;. Nonetheless, there are a few reasons that makes &lt;a href="https://www.ansible.com/"&gt;Ansible&lt;/a&gt; still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible &lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html"&gt;roles&lt;/a&gt; and &lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_intro.html"&gt;playbooks&lt;/a&gt; want to reuse, and expand their assets in Ansible. The benefit is consistency in the VM management, over time, and across platforms (AWS, on-prem, Azure, etc). Even for AWS shops, in the last few years many enterprises have adopted &lt;a href="https://www.digihunch.com/2022/12/landing-zone-in-aws/"&gt;AWS landing zone&lt;/a&gt; with the &lt;a href="https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/benefits-of-using-multiple-aws-accounts.html"&gt;multiple AWS account&lt;/a&gt; prescriptive pattern. However AWS Systems Manager still lacks integration with &lt;a href="https://docs.aws.amazon.com/organizations/latest/userguide/services-that-can-integrate-ssm.html"&gt;AWS Organization&lt;/a&gt; (except for a few non-core capabilities). This creates the demand of managing EC2 instances across AWS accounts. In this post, we propose a secure method to manage a fleet of EC2 instances from multiple AWS accounts, using Systems Manager . It also enables connectivity from an Ansible control node.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Prerequisites&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;This proposal ties a few CLI tools together, including AWS CLI, SSH, Ansible, etc. It also requires the cloud engineer to understand how they work. I&amp;#8217;ll start with the choice of the tools.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Above, I discussed the benefit of Ansible. Since Ansible operates on SSH, we&amp;#8217;ll still have to use SSH tools. Even though SSM agent provides a way to connect to EC2 instance without requiring an RSA key pair, we still need SSH since it is a well-established &lt;a href="https://datatracker.ietf.org/doc/html/rfc4253"&gt;industry standard&lt;/a&gt; (RFC4253) and the foundation of Ansible. These two technologies are not mutually exclusive. In fact, the SSM agent provides a secure enhancement to the operation with SSH. Traditionally, on each EC2 instance we&amp;#8217;d have to run SSHD services which opens TCP port 22 (or alternative TCP port as configured). For authentication we favour key pair over password but the open port is still an attack surface vulnerable to brute force and DDoS attacks. For EC2 instances on private networks there is no reachability to the instance&amp;#8217;s SSH port unless the bastion is also in a connected network. &lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;As I cover in a &lt;a href="https://www.digihunch.com/2023/10/the-systems-manager-hodgepodge/"&gt;post&lt;/a&gt;, the SSM Session Manager comes in handy. The SSM agent operates from the instance and communicate outbound to AWS backend. Since the SSM agent runs under a privileged user on the OS, you can perform OS-level commands through SSM. Further, AWS developed a &lt;a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html"&gt;Session Manager plugin&lt;/a&gt; with AWS CLI, allowing AWS CLI as a proxy command when making an SSH connection. Therefore SSM enables SSH connection without requiring port 22 to be open. In addition we&amp;#8217;ll need to use RSA key pair as required for SSH, which is also an improvement to the security posture. &lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;That explains the dependent tools. On the Ansible &lt;a href="https://docs.ansible.com/ansible/latest/network/getting_started/basic_concepts.html#control-node"&gt;control node&lt;/a&gt;, apart from Ansible itself, we need the latest version of AWS CLI with the &lt;a href="https://docs.aws.amazon.com/organizations/latest/userguide/services-that-can-integrate-ssm.html"&gt;Session Manager plugin&lt;/a&gt;, we need to configure AWS CLI properly to connect to EC2 instances across multiple AWS accounts. &lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Configure AWS CLI&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;This section discusses how to configure AWS CLI. I have a couple of handy aliases for productivity but they are not essential. For example, I often need to check the IAM identity making the call, and I often need to list out all profiles configured. So I added the following two entries in the &lt;code&gt;~/.aws/cli/alias&lt;/code&gt; file:&lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;[toplevel]&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;whoami = sts get-caller-identity --no-cli-pager --output yaml&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;profile = configure list-profiles&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p class="wp-block-paragraph"&gt;With that I have an alias to check IAM identity and available profiles. Then we can start configuring the profiles for CLI (in the file &lt;code&gt;~/.aws/config&lt;/code&gt;). Since we&amp;#8217;ll be working with multiple AWS accounts, we have to manage multiple CLI profiles, which implies that:&lt;/p&gt;&#10;&lt;ol class="wp-block-list"&gt;&#10;&lt;li&gt;We better use the &lt;code&gt;--profile&lt;/code&gt; switch to explicitly specify profile being used, instead of relying on the &lt;code&gt;AWS_PROFILE&lt;/code&gt; environment variable;&lt;/li&gt;&#10;&lt;li&gt;As a security best practice, we should not configure profiles with &lt;a href="https://docs.aws.amazon.com/sdkref/latest/guide/access-iam-users.html"&gt;long-term IAM credential&lt;/a&gt; in the config file;&lt;/li&gt;&#10;&lt;li&gt;We must ensure the CLI doesn&amp;#8217;t prompt for log-in every time we switch profile&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p class="wp-block-paragraph"&gt;To satisfy #3 there are many ways but we&amp;#8217;ll discuss two: using cross-account IAM role, and using AWS SSO.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Bonus point if you enable &lt;a href="https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-completion.html"&gt;auto-complete&lt;/a&gt; for AWS CLI.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Configure AWS CLI Profiles&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;With cross-account IAM role, the idea, is that the client start with one IAM identity, and use that IAM identity to assume roles on several other accounts. The configuration looks like this:&lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;[profile jump_account]&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;credential_process = /opt/bin/awscreds-custom --username helen&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;[profile target_account_1]&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;role_arn = arn:aws:iam::123456789011:role/OrganizationAccountAccessRole&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;source_profile = jump_account&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;[profile target_account_2]&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;role_arn = arn:aws:iam::123456789012:role/OrganizationAccountAccessRole&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;source_profile = jump_account&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p class="wp-block-paragraph"&gt;In this example, you start with an validated identity in the jump account, then assume a privileged IAM role named OrganizationAccountAccessRole on the target accounts. Typically such IAM roles are pre-configured (e.g. in an multi-account landing zone) with appropriate trust policy to allow principals from the jump account. Once you&amp;#8217;re validated as the IAM identity in the jump account, then you can use profiles for target accounts without being prompted for credentials again.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;If your have configured IAM Identity Center for the multi-account environment, consider an alternative approach using sso login. The configuration usually looks like this:&lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;[sso-session sso]&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;sso_start_url = https://myorg.awsapps.com/start/&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;sso_region = us-east-1&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;sso_registration_scopes = sso:account:access&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;[profile target_account_1]&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;sso_session = sso&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;sso_account_id = 123456789011&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;sso_role_name = AWSAdministratorAccess&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;[profile target_account_2]&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;sso_session = sso&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;sso_account_id = 123456789012&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;sso_role_name = AWSAdministratorAccess&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p class="wp-block-paragraph"&gt;This is often used by human users with SSO credential. In this example, to authenticate the sso session, start with command &amp;#8220;aws sso login&amp;#8221; . Then you can use all profiles by specifying &lt;code&gt;--profile&lt;/code&gt; switch without having to log in again.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Configure SSH to EC2 via SSM&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;The EC2 instance must connect to Systems Manager &lt;a href="https://docs.aws.amazon.com/general/latest/gr/ssm.html"&gt;endpoint&lt;/a&gt;, before one can SSH to the instance using the plugin. Once connected, you should find the instance in &lt;a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/fleet.html"&gt;Fleet Manager&lt;/a&gt;. For this to happen, there are a few conditions. First, the instance must be able to reach the endpoint, either via public Internet, or via VPC interface endpoints if in a private subnet. Second, the instance profile must contain an IAM role with appropriate permissions. We can use AWS managed policy AmazonSSMManagedInstanceCore in the role. In addition, if we record the SSM session to an S3 bucket with encryption the instance profile must have permission to use the encryption key.&lt;/p&gt;&#10;&lt;div class="wp-block-image"&gt;&#10;&lt;figure class="aligncenter size-full"&gt;&lt;img loading="lazy" decoding="async" width="511" height="141" src="https://www.digihunch.com/wp-content/uploads/2024/05/ssh-ssm.webp" alt="" class="wp-image-12985" srcset="https://www.digihunch.com/wp-content/uploads/2024/05/ssh-ssm.webp 511w, https://www.digihunch.com/wp-content/uploads/2024/05/ssh-ssm-300x83.webp 300w" sizes="auto, (max-width: 511px) 100vw, 511px" /&gt;&lt;/figure&gt;&#10;&lt;/div&gt;&#10;&lt;p class="wp-block-paragraph"&gt;SSM agent uses the IAM role. The agent runs as a service on Linux or Windows machines. Many AWS managed AMIs come with SSM agent pre-installed. If that is not the case, you&amp;#8217;d install the agent in your own AMI, or in user data which requires downloading the installer. With these configuration you&amp;#8217;d be able to connect to the instance via SSM. We can use AWS &lt;a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html#sessions-start-cli"&gt;CLI SSM command&lt;/a&gt;, or AWS web console to start an SSH session. To use SSH CLI utility, we install the &lt;a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/plugin-version-history.html"&gt;session manager plugin&lt;/a&gt; along with AWS CLI on the SSH client machine. We also specify a public key for EC2 instance and run ssh command with the private key. The SSH configuration needs a configuration such as:&lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;host i-* mi-*&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ProxyCommand sh -c &amp;#34;aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters &amp;#39;portNumber=%p&amp;#39;&amp;#34;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; User ec2-user&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; IdentityFile ~/.ssh/id_rsa&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p class="wp-block-paragraph"&gt;With this entry, you may directly SSH by instance ID (usually starting with i-* and mi-*), and the specified Proxy Command with SSM session document AWS-StartSSHSession will be invoked. &lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Configure Ansible Inventory&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;We can SSH to an instance (without port 22 open, on top of SSM) using the method above. Similarly, we can also configure Ansible to connect to the instance, without port 22. The inventory configuration looks like this:&lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;mytest:&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; hosts:&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; instance1:&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ansible_host: i-00aabbffcc7755221&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ansible_user: ubuntu&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ansible_ssh_common_args: -o ProxyCommand=&amp;#34;aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1&amp;#34;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; instance2:&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ansible_host: i-eedd88ff66aa22442&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ansible_user: ubuntu&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ProxyCommand=&amp;#34;sh -c \&amp;#34;aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters &amp;#39;portNumber=%p&amp;#39; --profile target_account_2 \&amp;#34;&amp;#34;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p class="wp-block-paragraph"&gt;Note I used two patterns for ansible_ssh_common_args that are similar. Both works. Each entry references its own profile. This is necessary because Ansible does not have the knowledge which instance belongs to which profile&amp;#8217;s account.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Another way to get Ansible connect to instances is to use the connection plugin &lt;a href="https://docs.ansible.com/ansible/latest/collections/community/aws/aws_ssm_connection.html"&gt;community.aws.aws_ssm&lt;/a&gt;, by specifying ansible_connection: aws_ssm (e.g. as host variable) and other required variables (e.g. profile). This method does not require SSH channel but it requires an S3 bucket, and hence IAM permission on the controller node.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Ansible supports dynamic inventory, in three ways: inventory source file (with existing plugin), custom inventory plugin, and inventory script (in Python). Take source file as an example for EC2, add the followings as the content of aws_ec2.yaml:&lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;plugin: amazon.aws.aws_ec2&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;# Attach the default AWS profile&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;aws_profile: target_account_1&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;compose:&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ansible_host: instance_id&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ansible_user: &amp;#34;&amp;#39;ubuntu&amp;#39;&amp;#34;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; ansible_ssh_common_args: &amp;#34;&amp;#39;-o ProxyCommand=\&amp;#34;aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1 \&amp;#34;&amp;#39;&amp;#34;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p class="wp-block-paragraph"&gt;Then we can display the rendered inventory list, and Ansible-ping the instances. &lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;ansible-inventory -i aws_ec2.yaml --list -y | less&#10;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;ansible all -i aws_ec2.yaml -m ping&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p class="wp-block-paragraph"&gt;For more flexibility, for the composed variables, we could use jinja2 expression to generate the value. In both ways, we produce an inventory source per profile using the &lt;a href="https://vexpose.blog/2023/11/20/ansible-dynamic-inventory-tutorial/"&gt;built-in inventory plugin&lt;/a&gt; aws_ec2. For greater flexibility, such as consolidating instances from all accounts into a single inventory, consider writing your own&lt;a href="https://www.redhat.com/sysadmin/ansible-dynamic-inventories"&gt; inventory script&lt;/a&gt;, or even &lt;a href="https://www.redhat.com/sysadmin/ansible-plugin-inventory-files"&gt;own inventory plugin&lt;/a&gt;.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Conclusion&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;In this post we propose a way to manage instances across AWS accounts. Two main challenges are establishing the communication channel (SSH on top of SSM) and generating inventory data in Ansible. Some AWS services can generate inventory data, such as &lt;a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-inventory-datasync.html"&gt;resource data sync&lt;/a&gt; in Systems Manager, or using AWS Config Aggregator. It is unfortunate that neither way produces the inventory data in a format that is directly compatible with Ansible inventory. Therefore, you might have to create a custom Ansible dynamic inventory script (in Python) that reads from the inventory data from AWS Config Aggregator (which supports AWS Organization). The side benefit of this script is that it is usually faster than the built-in &lt;a href="https://docs.ansible.com/ansible/latest/collections/amazon/aws/aws_ec2_inventory.html"&gt;aws_ec2&lt;/a&gt; inventory plugin.&lt;/p&gt;&#10;&lt;nav class="wp-post-navigation" aria-label="Post navigation"&gt;&#10;&lt;a rel="prev" href="https://www.digihunch.com/2024/03/public-key-infrastructure-3-of-3-use-cases/"&gt;&lt;span class="wp-post-navigation-label"&gt;Previous Post&lt;/span&gt;&lt;strong class="wp-post-navigation-title"&gt;Public Key Infrastructure 3 of 3 – PKI Implementation&lt;/strong&gt;&lt;/a&gt;&#10;&lt;a rel="next" href="https://www.digihunch.com/2024/07/iam-roles-for-any-workload/"&gt;&lt;span class="wp-post-navigation-label"&gt;Next Post&lt;/span&gt;&lt;strong class="wp-post-navigation-title"&gt;IAM Roles for any workload&lt;/strong&gt;&lt;/a&gt;&#10;&lt;/nav&gt;&#10;</description></item><item><title>Landing Zone in AWS – An Introduction</title><link>https://www.digihunch.com/2022/12/landing-zone-in-aws/</link><pubDate>Thu, 08 Dec 2022 22:37:00 -0400</pubDate><guid>https://www.digihunch.com/2022/12/landing-zone-in-aws/</guid><description>&lt;img src="https://www.digihunch.com/wp-content/uploads/2025/04/landing-zone-intro.webp" alt="Featured image of post Landing Zone in AWS – An Introduction" /&gt;&lt;p class="wp-block-paragraph"&gt;Cloud adoption has gone through phases. Hashicorp&amp;#8217;s CTO Armon Dadgar has a great stream on Hashcorp&amp;#8217;s narrative of the &lt;a href="https://www.youtube.com/watch?v=IBTpNptxYF4"&gt;three Phases of Cloud Adoption&lt;/a&gt;:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;Phase 1, with main focus on agility, app teams in wild west, account sprawl, inconsistent configuration, security &amp;amp; compliance challenge&lt;/li&gt;&#10;&lt;li&gt;Phase 2. consistent platform layer providing an opinionated way of configurations, security and compliance control, introduction of platform team, whose customers are application team, scaling challenges with platform team capacity not keeping up with application team&amp;#8217;s demand&lt;/li&gt;&#10;&lt;li&gt;Phase 3. self-service platform at scale for many application teams. CICD, infra-as-code&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;New greenfield clients today should aim at phase 2 or phase 3 depending on their target operation size. Either way, they need a landing zone for consistency. This post focuses on the landing zone options on AWS and the key constructs.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Overview&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;AWS &lt;a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-aws-environment/understanding-landing-zones.html"&gt;Prescriptive Guidance&lt;/a&gt; defines landing zone as a well-architected (secure, scalable, compliant, etc), multi-account AWS environment that is a secure baseline from which you can deploy workloads and applications. A landing zone may consists of:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;Multi-account architecture&lt;/li&gt;&#10;&lt;li&gt;Identity and Access management solution&lt;/li&gt;&#10;&lt;li&gt;Governance, compliance, logging and auditing solutions&lt;/li&gt;&#10;&lt;li&gt;Security and networking design&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Landing zone reflects an enterprise opinion on how to configure networking and IAM. Establishing a landing zone can be a very involving process. In early days of cloud operation people used build landing zone by “clickops” on AWS console, which cannot keep up with the growth of landing zone and associated security services. Compared to SMB clients, some sectors such healthcare and national security have much more regulations and compliance frameworks.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;In fact, landing zone is such a buzzword that I have learned to be very sensitive to the context. If a solution has words “landing zone” in its name, given the complexity and loose use of the words, there is a good chance that the solution only delivers some of the aspects above. Cloud consultants are still to address the gaps. Therefore I decided to write this post about what I learned about landing zone in AWS.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading" id="h-multi-account-and-organization"&gt;Multi-account and Organization&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Since 2017, AWS has been officially advocating the use of &lt;a href="https://d1.awsstatic.com/aws-answers/AWS_Multi_Account_Security_Strategy.pdf"&gt;multiple accounts&lt;/a&gt; as a best practice and security strategy. They encourage clients to view an account as a resource container, just like &lt;a href="https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal"&gt;Resource Group&lt;/a&gt; in &lt;a href="https://medium.com/@richard_64931/account-structure-comparison-between-aws-and-azure-450cb4544fff"&gt;Azure&lt;/a&gt; (&lt;a href="https://www.linkedin.com/pulse/account-structure-comparison-between-aws-azure-richard-lenan-zhao/"&gt;here&lt;/a&gt; is an article on the account structure between AWS and Azure) Client builds an account for security boundary and financial container. Many blog post came along (such as &lt;a href="https://aws.amazon.com/blogs/mt/supercharge-multi-account-management-with-aws-cloudformation/"&gt;this&lt;/a&gt; one) on the implementation details. From tooling perspective, AWS launched &lt;a href="https://aws.amazon.com/about-aws/whats-new/2017/02/aws-organizations-now-generally-available/"&gt;AWS Organization&lt;/a&gt; in 2017 to facilitate multi-account management. &lt;a href="https://www.youtube.com/watch?v=fxo67UeeN1A"&gt;This&lt;/a&gt; presentation from re:Inforce 2019 is a good material to understand multi-account environment with AWS Organizations. In 2021, AWS published a new &lt;a href="https://aws.amazon.com/blogs/architecture/new-whitepaper-provides-best-practices-for-optimizing-aws-accounts/"&gt;white paper&lt;/a&gt; on best practices with multiple accounts. My previous coworkers authored two blog posts &lt;a href="https://medium.com/slalom-build/crafting-secure-aws-environments-using-an-aws-multi-account-environment-1163910b8919"&gt;here&lt;/a&gt; and &lt;a href="https://medium.com/slalom-build/crafting-aws-accounts-applications-accounts-7d9140ab0da6"&gt;here&lt;/a&gt; to reflect multi-account setup as of late 2020.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;AWS Organization is an account management service that provides a vehicle to centrally manage AWS accounts by groups, which brings many benefits, such as centralized logging, compliance management, consolidated billing, etc. As an administrator, you can create accounts in your organization and invite existing accounts to join the organization.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Typical account structure involves:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;A root organization named Root&lt;/li&gt;&#10;&lt;li&gt;A organization hierarchy with one or more OUs under root, with each OU having one or more child OUs.&lt;/li&gt;&#10;&lt;li&gt;Each OU can have multiple accounts, with each account having one email address.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;You can also configure &lt;strong&gt;service control policies (SCPs)&lt;/strong&gt;, a type of organization-level policy that you can use to manage permissions in your organization. SCPs offer central control over the maximum available permissions for all accounts in your organization. SCPs are a means of implementing guardrails in your AWS organization.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;To vend multiple account automatically, AWS introduced the Account Factory (which became part of Control Tower later in 2019) and let users create new account from AWS console and specify which OU it belongs to. It also allows users to implement customization after account creation with Service Catalog products. Some legacy orchestration solution (e.g. Augmented Account Factory) were based on this mechanism. Another orchestration solution prior to the launch of Control Tower was the AWS &lt;a href="https://aws.amazon.com/about-aws/whats-new/2018/06/introducing-aws-landing-zone/"&gt;Landing Zone solution&lt;/a&gt; (ALZ, introduced in 2018), which uses AWS CodePipeline to provision accounts and deploy resources.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading" id="h-from-landing-zone-solution-alz-to-control-tower"&gt;From Landing Zone Solution (ALZ) to Control Tower&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;The launch of &lt;a href="https://aws.amazon.com/about-aws/whats-new/2019/06/aws-control-tower-is-now-generally-available/"&gt;Control Tower&lt;/a&gt; was a game changer in 2019 to bring users to &lt;a href="https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/organizing-your-aws-environment.html"&gt;multi-account&lt;/a&gt; best practices. Control Tower, as an AWS product (instead of a solution by some service teams at AWS) matured over several years. It gradually deprecated the previous generation orchestration solutions. Newer orchestration solutions all have to support Control Tower. According to &lt;a href="https://docs.aws.amazon.com/controltower/latest/userguide/customize-landing-zone.html"&gt;this&lt;/a&gt; page, we can customize &lt;a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/designing-control-tower-landing-zone/welcome.html"&gt;Control Tower based Landing Zone&lt;/a&gt; in these ways:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;AWS Control Tower console:&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;Instead of creating OU in AWS organization, do it under Control Tower in the console, under “create required OUs”.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;/li&gt;&#10;&lt;li&gt;Outside of AWS Control Tower console&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;using &lt;a href="https://developer.hashicorp.com/terraform/tutorials/aws/aws-control-tower-aft"&gt;Account Factory for Terraform (AFT)&lt;/a&gt;: Terraform-based account provisioning pipeline, for heavy Terraform shops.&lt;/li&gt;&#10;&lt;li&gt;using &lt;a href="https://docs.aws.amazon.com/controltower/latest/userguide/cfct-overview.html"&gt;Customizations for AWS Control Tower&lt;/a&gt; (CfCT) solution. The LZ created remains in sync with Control Tower.&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;You first launch a standardized CF stack to set up the mechanisms for customization.&lt;/li&gt;&#10;&lt;li&gt;Then you create a custom package to define the customization. This includes a manifest file.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;On the other hand, the original ALZ solution is currently in long-term support and will not receive any additional features. It is deprecated. AWS advises its customers &lt;a href="https://aws.amazon.com/blogs/mt/migrate-aws-landing-zone-solution-to-aws-control-tower/"&gt;migrate to AWS Control Tower&lt;/a&gt; based landing zone. The &lt;a href="https://aws.amazon.com/solutions/implementations/customizations-for-aws-control-tower/"&gt;ALZ page&lt;/a&gt; is redirected to a page about customizing Control Tower landing zone.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;The &lt;a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/designing-control-tower-landing-zone/welcome.html"&gt;Control Tower based landing zone&lt;/a&gt; configures OUs, accounts, SSO and guardrails. It however does not entail networking design except for the guardrails.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading" id="h-security-reference-architecture-sra"&gt;Security Reference Architecture (SRA)&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/security-reference-architecture/welcome.html"&gt;SRA&lt;/a&gt; is just a reference architecture and accompanying recommendations on AWS security services, and how they work together in a multi-account environment to host a single-page application. It comes with a &lt;a href="https://github.com/aws-samples/aws-security-reference-architecture-examples"&gt;repository&lt;/a&gt; to demonstrate how to configure a secure multi-account environment with Control Tower, CfCT, as well as security services. Alternatively, you can deploy it with CloudFormation StackSets. For fully automated deployment of this architecture, check out Landing Zone Accelerator down below.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;One important feature that reflects the notion of centralized security in a multi-VPC topology, is the use of a dedicated VPC for centralized interface endpoints. This pattern is seen in all generations of security reference architectures.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading" id="h-aws-security-environment-accelerator-asea"&gt;AWS Security Environment Accelerator (ASEA)&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Regulated customers often find they need to add additional controls and capabilities to be defined and setup outside of Control Tower. &lt;a href="https://aws-samples.github.io/aws-secure-environment-accelerator/"&gt;ASEA&lt;/a&gt;, as an orchestration solution, aims to remove the complexity of having to develop and maintain a separate codebase to manage the additional customizations, by providing a tool to help deploy and operate secure multi-account, multi-region AWS environments on an ongoing basis.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Overtime, as Control Tower introduces new capabilities to support the customizations required in heavily regulated environments, the capabilities will be removed from ASEA and enabled directly within the Control Tower managed service, further reducing operational burden. Read this for its &lt;a href="https://github.com/aws-samples/aws-secure-environment-accelerator#relationship-with-aws-landing-zone-solution-alz"&gt;relationship&lt;/a&gt; with ALZ and Control Tower.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;ASEA was first released in late 2020. It covers more on networking design and has a fairly comprehensive &lt;a href="https://github.com/aws-samples/aws-secure-environment-accelerator#accelerator-installation-process-summary"&gt;installation process&lt;/a&gt;. ASEA primarily cater to &lt;a href="https://docs.aws.amazon.com/whitepapers/latest/public-sector-cloud-transformation/secure-environment-accelerator-canada.html"&gt;government of Canada&lt;/a&gt;&amp;#8216;s &lt;a href="https://www.canada.ca/en/government/system/digital-government/digital-government-innovations/cloud-services/government-canada-security-control-profile-cloud-based-it-services.html#toc4"&gt;PBMM Security Configuration Profile&lt;/a&gt; with an opinionated configuration. The first few revisions were referred to as PMBB architecture. While it provides a great &lt;a href="https://aws-samples.github.io/aws-secure-environment-accelerator/v1.5.6-a/architectures/sensitive/network/"&gt;reference architecture&lt;/a&gt; for highly regulated landing zone, it has not gotten much traction elsewhere. The recommendation going forward, is to use Landing Zone Accelerator, which incorporates the features and lessons learned from ASEA.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading" id="h-landing-zone-accelerator-lza"&gt;Landing Zone Accelerator (LZA)&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;First released in May 2022, the &lt;a href="https://aws.amazon.com/solutions/implementations/landing-zone-accelerator-on-aws/"&gt;Landing Zone Accelerator&lt;/a&gt; on AWS solution deploys a cloud foundation that is architected to align with AWS best practices and multiple global compliance frameworks. LZA operates on top of Control Tower managed landing zone. This &lt;a href="https://docs.aws.amazon.com/solutions/latest/landing-zone-accelerator-on-aws/architecture-overview.html"&gt;page&lt;/a&gt; in the documentation brings a good explanation of how it works. Basically it employs CodeBuild as an orchestration engine, and leverages &lt;a href="https://www.digihunch.com/2020/12/ansible-tower-lab-environment-on-aws/"&gt;CDK&lt;/a&gt; to drive resource deployment. There are two repositories: the GitHub repository for Landing Zone Accelerator itself, and a CodeCommit configuration repository provisioned during preparation.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;As &lt;a href="https://docs.aws.amazon.com/solutions/latest/landing-zone-accelerator-on-aws/reference.html"&gt;document&lt;/a&gt; states, LZA is a fully automated implementation of the architecture guidelines documented in the &lt;a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/security-reference-architecture/welcome.html"&gt;SRA&lt;/a&gt;. LZA also incorporates features and lessons learned from ASEA and &lt;a href="https://docs.aws.amazon.com/solutions/latest/compliant-framework-for-federal-and-dod-workloads-in-aws-govcloud-us/welcome.html"&gt;Compliant Framework for Federal and DoD Workloads in GovCloud (US)&lt;/a&gt;, neither of which are recommended for new deployment. In addition, LZA aims to enable iterations and extensions of a secure environment over time. The vision is to eventually replace AFT, CfCT, and ASEA. As AWS releases newer versions of LZA, client should be able to upgrade it in a pipeline run.&lt;/p&gt;&#10;&lt;div class="wp-block-image"&gt;&#10;&lt;figure class="aligncenter size-large"&gt;&lt;img loading="lazy" decoding="async" width="1024" height="549" src="https://www.digihunch.com/wp-content/uploads/2022/12/base-arch-lza-1024x549.webp" alt="" class="wp-image-12877" srcset="https://www.digihunch.com/wp-content/uploads/2022/12/base-arch-lza-1024x549.webp 1024w, https://www.digihunch.com/wp-content/uploads/2022/12/base-arch-lza-300x161.webp 300w, https://www.digihunch.com/wp-content/uploads/2022/12/base-arch-lza-768x412.webp 768w, https://www.digihunch.com/wp-content/uploads/2022/12/base-arch-lza.webp 1288w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /&gt;&lt;figcaption class="wp-element-caption"&gt;Baseline architecture for LZA&lt;/figcaption&gt;&lt;/figure&gt;&#10;&lt;/div&gt;&#10;&lt;p class="wp-block-paragraph"&gt;The Landing Zone Accelerator project also provides a &lt;a href="https://github.com/awslabs/landing-zone-accelerator-on-aws/tree/main/reference/sample-configurations"&gt;samples&lt;/a&gt; configurations in each regulated frameworks. For example, the healthcare best practice come up in Oct 2022 for &lt;a href="https://www.healthcareitnews.com/news/aws-launches-new-healthcare-specific-accelerator"&gt;healthcare industry&lt;/a&gt;. The &lt;a href="https://aws.amazon.com/blogs/industries/introducing-landing-zone-accelerator-for-healthcare/"&gt;healthcare best practice&lt;/a&gt; sample incorporates healthcare specific configurations, such as the detective guardrails defined in the&amp;nbsp;&lt;a href="https://docs.aws.amazon.com/config/latest/developerguide/operational-best-practices-for-hipaa_security.html"&gt;Operational Best Practices for HIPAA Security&lt;/a&gt; &amp;nbsp;conformance pack. To deploy the best practices, modify the configuration in config repo, and run the pipeline again.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Landing Zone Orchestration Options&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;To summarize, we have the following landing zone options in AWS:&lt;/p&gt;&#10;&lt;figure class="wp-block-table is-style-stripes"&gt;&lt;table class="has-black-color has-pale-cyan-blue-background-color has-text-color has-background"&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Orchestration Layer&lt;/th&gt;&lt;th&gt;Lifecycle&lt;/th&gt;&lt;th&gt;Summary&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Solutions based on account factory with Service Catalog (e.g. ALZ, Augmented Account Factory)&lt;/td&gt;&lt;td&gt;All solutions pre-dates the launch of control tower have been or will soon be deprecated.&lt;/td&gt;&lt;td&gt;Pre-Control Tower solutions should migrate to current alternatives.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;AWS Control Tower&lt;/td&gt;&lt;td&gt;GA in June 2019&lt;/td&gt;&lt;td&gt;To customize the landing zone, use AWS console, or alternatively, one of the following three options:&lt;br&gt;&amp;#8211; CfCT: &lt;a href="https://docs.aws.amazon.com/controltower/latest/userguide/cfct-overview.html"&gt;Customization for Control Tower&lt;/a&gt;&lt;br&gt;&amp;#8211; AFT: &lt;a href="https://docs.aws.amazon.com/controltower/latest/userguide/taf-account-provisioning.html"&gt;Account Factory Terraform&lt;/a&gt;&lt;br&gt;&amp;#8211; AFC: &lt;a href="https://docs.aws.amazon.com/controltower/latest/userguide/af-customization-page.html"&gt;Account Factory Customizations&lt;/a&gt;&lt;br&gt;For example, CfCT can be used to deploy Security Reference Architecture (SRA)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;AWS Secure Environment Accelerator (ASEA)&lt;/td&gt;&lt;td&gt;Released 2020&lt;/td&gt;&lt;td&gt;Canadian Centre for Cyber Security (CCCS) Medium Cloud Control Profile, formerly known as PBMM. This approach will be replaced by LZA&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;AWS Landing Zone Accelerator (LZA)&lt;/td&gt;&lt;td&gt;Released in 2022&lt;/td&gt;&lt;td&gt;A low-code deployment option. Samples provided in support of reference architectures that align with industry best practices or compliance frameworks. Examples for industry best practices:&lt;br&gt;&amp;#8211; general best practice&lt;br&gt;&amp;#8211; healthcare&lt;br&gt;&amp;#8211; finance and tax&lt;br&gt;&amp;#8211; education&lt;br&gt;Example for compliance frameworks:&lt;br&gt;&amp;#8211; US state local government&lt;br&gt;&amp;#8211; FedRAMP for US Federal and Department of Defence (DoD)&lt;br&gt;&amp;#8211; CCCS Medium for Canadian government&lt;br&gt;&amp;#8211; AWS Trusted Secure Enclave (TSE) Sensitive Edition (SE), which also aligns with other medium level security profiles such as NIST 800-53, ITSG-33, FedRAMP moderate, CCCS-Medium, IRAP, etc&lt;br&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/figure&gt;&#10;&lt;p class="wp-block-paragraph"&gt;If your organization has a compliance framework that LZA supports, it makes sense to start with LZA. Otherwise, it is sufficient to use Control Tower for multi-account setup.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Networking constructs&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Landing zone involves multiple VPCs so it&amp;#8217;s important to understand VPC peering and Transit Gateway. VPC peering can only be setup between two VPCs and transitive peering relationship is not supported. For 6 VPCs to talk to all each other, we&amp;#8217;d need 15 peering setups, which is not sustainable. We have two options:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;We can connect many VPCs to a single Transit Gateway and the VPCs will be able to talk to each other. We can also connect Transit Gateway to site-to-site VPN or Direct Connect. &lt;a href="https://ngoyal16.medium.com/vpc-peering-or-transit-gateway-b0f1176874f"&gt;This&lt;/a&gt; is a good explanation.&lt;/li&gt;&#10;&lt;li&gt;We can still use peering, but dedicate one VPC as “Transit VPC” in a hub and spoke model.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;When you attach a VPC to a transit gateway, you must specify one subnet from each AZ to be used by the transit gateway to route traffic. Specifying one subnet from an AZ enables traffic to reach resources in every subnet in that AZ.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;a href="https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/transit-vpc-solution.html"&gt;Here&lt;/a&gt; is a comparison table. Read the white-paper &amp;#8220;Building a Scalable and Secure Multi-VPC AWS Network Infrastructure&amp;#8221; for more about network design. For example, Some clients need to &lt;a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/inline-traffic-inspection-third-party-appliances/welcome.html"&gt;inspect traffic&lt;/a&gt;. When deploying multiple VPCs, we also need to ensure &lt;a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver.html"&gt;DNS resolution&lt;/a&gt; works across VPCs, and between on-prem networks, by configuring &lt;a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#AmazonDNS"&gt;Amazon DNS server&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Our VPCs also need to connect to managed AWS services. For services like S3 or DynamoDB (of the same or different account), workload in VPC can access them via public DNS. The traffic goes through Internet Gateway of VPC and then public Internet. It is neither secure nor economical. We often want all network traffic to stay on the global AWS backbone. There are three types of Endpoint under VPC to help us.&lt;/p&gt;&#10;&lt;figure class="wp-block-table is-style-stripes"&gt;&lt;table class="has-black-color has-pale-cyan-blue-background-color has-text-color has-background"&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;&lt;/th&gt;&lt;th&gt;Gateway VPC Endpoint or &lt;a href="https://docs.aws.amazon.com/vpc/latest/privatelink/gateway-endpoints.html" target="_blank" rel="noreferrer noopener"&gt;Gateway Endpoint&lt;/a&gt;&lt;/th&gt;&lt;th&gt;Interface VPC Endpoint or &lt;a href="https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html" target="_blank" rel="noreferrer noopener"&gt;Interface Endpoint&lt;/a&gt;&lt;/th&gt;&lt;th&gt;&lt;a href="https://docs.aws.amazon.com/vpc/latest/privatelink/create-gateway-load-balancer-endpoint-service.html" target="_blank" rel="noreferrer noopener"&gt;Gateway Load Balancer Endpoint&lt;/a&gt;&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;&lt;strong&gt;Purpose&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;VPC access native AWS services&lt;/td&gt;&lt;td&gt;VPC access native AWS services&lt;/td&gt;&lt;td&gt;VPC access your own service (aka Endpoint Service)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;strong&gt;Enabled by &lt;a href="https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html" target="_blank" rel="noreferrer noopener"&gt;Private Link&lt;/a&gt;&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;No&lt;/td&gt;&lt;td&gt;Yes&lt;/td&gt;&lt;td&gt;Yes&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;strong&gt;Traffic remain on AWS network&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;Yes&lt;/td&gt;&lt;td&gt;Yes&lt;/td&gt;&lt;td&gt;Yes&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;strong&gt;Mechanism&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;Use the public IP address of the service along with configuration in routing table to access target resource. The routing table acts as gateway. You can use the public DNS name of the service.&lt;/td&gt;&lt;td&gt;Use private IP address from the VPC to access the target service. Require endpoint-specific DNS name for the target service. Incur extra charge. &lt;a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html" target="_blank" rel="noreferrer noopener"&gt;S3&lt;/a&gt; as example. More secure because there is ENI in the VPC controlled by security group.&lt;/td&gt;&lt;td&gt;Your Endpoint Services is hosted in front of a fleet of network &lt;a href="https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-gateway-load-balancer.html" target="_blank" rel="noreferrer noopener"&gt;virtual appliances&lt;/a&gt;. You can select endpoint type as you create it in console under VPC&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;strong&gt;Access from clients on premise or from other region&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;Does NOT allow access from clients on premise or in VPC from other region&lt;/td&gt;&lt;td&gt;Allows access from clients on premise or clients in VPC from other region&lt;/td&gt;&lt;td&gt;Allowed&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/figure&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Private Link integrates with a subset of &lt;a href="https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-privatelink-support.html" target="_blank" rel="noreferrer noopener"&gt;AWS services&lt;/a&gt;. To check the list of interface endpoint, use:&lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;aws ec2 describe-vpc-endpoint-services --query &lt;span style="color:#e6db74"&gt;&amp;#34;ServiceDetails[?ServiceType[0].ServiceType==&amp;#39;Interface&amp;#39;].ServiceName&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p class="wp-block-paragraph"&gt;Given the inter-VPC connectivity, most deployment centralizes interface endpoint into a dedicated VPC.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Centralized Interface Endpoints&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;There are several benefits to use a single VPC as dedicated provider of interface endpoint in a multi-VPC topology. First, the interface endpoints incurs a standing charge and it makes financial sense to consolidate them in one VPC. Second, this setup centralizes the configuration and security aspects as well. &lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;In most cases, interface endpoint services follow the format of &lt;code&gt;com.amazonaws.&amp;lt;region&amp;gt;.&amp;lt;endpoint&amp;gt;&lt;/code&gt;, with the dns name looking like: &lt;code&gt;&amp;lt;endpoint&amp;gt;.&amp;lt;region&amp;gt;.amazonaws.com&lt;/code&gt;. For example:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;Endpoint Service Name: &lt;code&gt;com.amazonaws.us-east-1.ssm&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Private Hosted Zone: &lt;code&gt;ssm.us-east-1.amazonaws.com&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Zone Record (A alias): &lt;code&gt;ssm.us-east-1.amazonaws.com&lt;/code&gt;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;However, there are a few exceptions to that which can make it tricky to implement interface endpoint with infrastructure as code.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;strong&gt;Exception 1&lt;/strong&gt;. Private DNS name suffix is &lt;code&gt;api.aws&lt;/code&gt; instead of &lt;code&gt;amazonaws.com&lt;/code&gt;:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;Endpoint Service Name: &lt;code&gt;com.amazonaws.us-east-1.eks-auth&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Private Hosted Zone: &lt;code&gt;eks-auth.us-east-1.api.aws&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Zone Record (A alias): &lt;code&gt;eks-auth.us-east-1.api.aws&lt;/code&gt;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;strong&gt;Exception 2&lt;/strong&gt;. Endpoint Service Name doesn&amp;#8217;t start with com.amazonaws&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;Endpoint Service Name: &lt;code&gt;aws.sagemaker.us-east-1.notebook&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Private Hosted Zone: &lt;code&gt;notebook.us-east-1.sagemaker.aws&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Zone Record (A alias): &lt;code&gt;notebook.us-east-1.sagemaker.aws&lt;/code&gt;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;strong&gt;Exception 3&lt;/strong&gt;. Two A-records are required under the same PHZ&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;Endpoint Service Name: &lt;code&gt;com.amazonaws.us-east-1.ecr.dkr&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Private Hosted Zone: &lt;code&gt;dkr.ecr.us-east-1.amazonaws.com&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Zone Record (A alias): &lt;code&gt;dkr.ecr.us-east-1.amazonaws.com&lt;/code&gt; and &lt;code&gt;*.dkr.ecr.us-east-1.amazonaws.com&lt;/code&gt;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;strong&gt;Exception 4&lt;/strong&gt;. Two PHZs are required:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;Endpoint Service Name: &lt;code&gt;com.amazonaws.us-east-1.ec2&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Private Hosted Zone: &lt;code&gt;ec2.us-east-1.amazonaws.com&lt;/code&gt; and &lt;code&gt;ec2.us-east-1.api.aws&lt;/code&gt;&lt;/li&gt;&#10;&lt;li&gt;Zone Record (A alias): &lt;code&gt;ec2.us-east-1.amazonaws.com&lt;/code&gt; and &lt;code&gt;ec2.us-east-1.api.aws&lt;/code&gt;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;The infrastructure code that handles interface endpoints should be able to take care of the exceptions.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Security Constructs&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;strong&gt;AWS Network firewall &lt;/strong&gt;is a configuration under VPC. You associate a Network Firewall with one or more subnets in the VPC. You also associate the Network Firewall with Firewall Policies. Each policy consists of rule groups. Each rule blocks or filters traffic. The log can be published to CloudWatch or S3 via Kinesis. Usecases of AWS Network firewall include:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;inspect VPC-to-VPC traffic;&lt;/li&gt;&#10;&lt;li&gt;filter outbound traffic;&lt;/li&gt;&#10;&lt;li&gt;prevent inbound internet traffic;&lt;/li&gt;&#10;&lt;li&gt;secure AWS Direct Connect and VPN traffic&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;When you &lt;a href="https://docs.aws.amazon.com/network-firewall/latest/developerguide/vpc-config.html"&gt;associate a firewall to your VPC&lt;/a&gt;, you must provide a subnet for each Availability Zone where you want to place a firewall endpoint to filter traffic&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;There are many points of configuration for firewall rules in AWS. &lt;strong&gt;AWS Firewall Manager&lt;/strong&gt; is a place for central management. It connects with other services such as:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;AWS WAF (L7)&lt;/li&gt;&#10;&lt;li&gt;AWS Network Firewall (L4)&lt;/li&gt;&#10;&lt;li&gt;AWS Shield (DDos)&lt;/li&gt;&#10;&lt;li&gt;Amazon Route 53 Resolver DNS Firewall&lt;/li&gt;&#10;&lt;li&gt;Security Groups&lt;/li&gt;&#10;&lt;li&gt;Third party firewall support&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;strong&gt;IAM Identity Center&lt;/strong&gt; (previously AWS SSO) is for logging into AWS portal, giving each identity from an external identity store (such as AD) an identity on an AWS account. It addresses several challenges as a result of having multiple AWS accounts. &lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;First, a user from AD needs to access multiple accounts. Second, in each account with access, s/he needs to have an IAM role. These IAM roles are based on attributes of his/her external identity, and can be different per AWS account. With &lt;a href="https://docs.aws.amazon.com/singlesignon/latest/userguide/permissionsetsconcept.html"&gt;Permission Set&lt;/a&gt; IAM Identity Center:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;Allow you to login on different AWS account using the same credential&lt;/li&gt;&#10;&lt;li&gt;Allow you to federate with external identity store( e.g. using SAML)&lt;/li&gt;&#10;&lt;li&gt;Manage permission set for each account.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;This &lt;a href="https://www.youtube.com/watch?v=x7TCs9HxRFg"&gt;video&lt;/a&gt; has a good walk-through of how to configure Azure AD as identity store, and tie it to specific AWS accounts and specify permission set. &lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Note that IAM Identity Center has a different purpose than Cognito. &lt;strong&gt;Cognito&lt;/strong&gt; is to connect your app with an IdP. Your app can be an EC2-hosted application, serverless application on API gateway, or container-based application on Kubernetes, as long as the support open identity standards (e.g. OAuth 2.0, SAML 2.0 and OIDC). Cognito has two pools:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;User pool for authentication. With a user pool, your app&lt;/li&gt;&#10;&lt;li&gt;Identity pool for authorization&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;span class="underline" style="text-decoration: underline"&gt;So Cognito is for AuthN &amp;amp; AuthZ to your own app’s endpoint (business traffic) and it supports a number of standards. IAM Identity Center is for AuthN &amp;amp; AuthZ to your AWS account (management traffic).&lt;/span&gt; If you use it for your own application, your application user will have direct access your AWS resources. It mainly supports SAML (and OIDC but less used) and is commonly connected with Active Directory (Microsoft or AWS).&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;One can use AWS Directory Service to host a compatible and managed directory service. &lt;strong&gt;AWS Directory Service&lt;/strong&gt; supports four modes:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;AWS managed Microsoft AD: actual Microsoft Active Directory in AWS Cloud&lt;/li&gt;&#10;&lt;li&gt;Simple AD, powered by Linux-Samba Active Directory-compatible server&lt;/li&gt;&#10;&lt;li&gt;AD Connector: a proxy for redirecting directory request to your existing Microsoft AD without caching any information in the cloud&lt;/li&gt;&#10;&lt;li&gt;Cognito user pools&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;If you combine two AD domains, you will need &lt;a href="https://learn.microsoft.com/en-us/azure/active-directory-domain-services/concepts-forest-trust#trust-relationship-flows"&gt;domain trust&lt;/a&gt;.&lt;/p&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Other security services&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;In addition to the core services above, LZA involves the following four services as well.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;strong&gt;AWS Macie&lt;/strong&gt; is a fully managed data security and privacy service based on ML and pattern matching. It continually evaluates your S3 environments to discovery PII and act on them. It also reports alerts on unencrypted buckets, publicly accessible buckets, etc.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;strong&gt;AWS GuardDuty&lt;/strong&gt; continuously analyze S3, container &amp;amp; instance workloads, user and accounts for potential threads, across account, based on ML for intelligent threat detection. In addition, GuardDuty also acts on findings.&lt;/p&gt;&#10;&lt;p class="wp-block-paragraph"&gt;&lt;strong&gt;AWS Config&lt;/strong&gt; is an essential service that traces resource inventory, their changes and monitors for compliance:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;When you turn on AWS Config, it first discovers the supported AWS resources in your account and generates a &lt;a href="https://docs.aws.amazon.com/config/latest/developerguide/config-concepts.html#config-items"&gt;configuration item&lt;/a&gt; (poin-in-time view of attributes) for each resource. AWS Config also generates configuration items when the configuration of a resource changes, and it maintains historical records of the configuration items of your resources from the time you start the configuration recorder.&lt;/li&gt;&#10;&lt;li&gt;On an ongoing basis, AWS Config keeps track of all changes to your resources, whether or not it is initiated by the API&lt;/li&gt;&#10;&lt;li&gt;If you are using AWS Config rules, AWS Config continuously evaluates your AWS resource configurations for desired settings. You can deploy several related rules in a pre-built &lt;a href="https://docs.aws.amazon.com/config/latest/developerguide/conformance-packs.html"&gt;compliance pack&lt;/a&gt;.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p class="wp-block-paragraph"&gt;As you may have noticed, there are many AWS services related to firewall, security and compliance. &lt;strong&gt;AWS Security Hub&lt;/strong&gt; aims to be a a consolidated view of your security status in AWS. You can automate security checks, manage security findings, and identify the highest priority security issues across accounts in client environment. It can also:&lt;/p&gt;&#10;&lt;ul class="wp-block-list"&gt;&#10;&lt;li&gt;consolidate security findings from GuardDuty, Inspector (vulnerability scanner and management), Macie, Config, Systems Manager, Firewall Manager, IAM Access Analyzer and other Integrated APN solutions&lt;/li&gt;&#10;&lt;li&gt;Check findings against best practices&lt;/li&gt;&#10;&lt;li&gt;Client can take action: i.e. investigate findings or take response and remediation actions&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 class="wp-block-heading"&gt;Summary&lt;/h2&gt;&#10;&lt;p class="wp-block-paragraph"&gt;Landing zone deployment can take numerous iterations to finalize the requirement. It is important to have a vision of the client&amp;#8217;s &lt;a href="https://docs.aws.amazon.com/whitepapers/latest/building-cloud-operating-model/building-cloud-operating-model.html"&gt;cloud operating model&lt;/a&gt;, which is usually discussed prior to implementation. At the implementation phase, I discuss the topic on two more separate posts on &lt;a href="https://www.digihunch.com/2023/08/control-tower-aws-landing-zone/"&gt;control tower&lt;/a&gt; and &lt;a href="https://www.digihunch.com/2023/09/orchestrate-landing-zone-with-landing-zone-accelerator-on-aws/"&gt;landing zone accelerator&lt;/a&gt;.&lt;/p&gt;&#10;&lt;nav class="wp-post-navigation" aria-label="Post navigation"&gt;&#10;&lt;a rel="prev" href="https://www.digihunch.com/2022/11/key-mapping-on-external-pc-keyboard-on-macbook/"&gt;&lt;span class="wp-post-navigation-label"&gt;Previous Post&lt;/span&gt;&lt;strong class="wp-post-navigation-title"&gt;Key mapping for external PC keyboard on Mac&lt;/strong&gt;&lt;/a&gt;&#10;&lt;a rel="next" href="https://www.digihunch.com/2022/12/eks-impression/"&gt;&lt;span class="wp-post-navigation-label"&gt;Next Post&lt;/span&gt;&lt;strong class="wp-post-navigation-title"&gt;EKS impression&lt;/strong&gt;&lt;/a&gt;&#10;&lt;/nav&gt;&#10;</description></item></channel></rss>