instance profile on Digi Hunch

IAM Roles for any workload

Sun, 14 Jul 2024 23:53:49 -0400

Background

A few month back a client of mine wanted to use GitLab pipeline to deploy infrastructure on AWS with Terraform. The key question is how to authenticate the Terraform process running in the pipeline to AWS with temporary credential. Having worked it out on GitHub, my proposal at time was to add OIDC provider to represent the GitLab runner.

After a few months, they told me that they are self-hosting their GitLab instance. The idea above was based on exposing an identity provider document on the public Internet, which the client is unable to do.

Now, I have an idea: IAM Roles Anywhere.

IAM Role Introduction

Many 101 tutorials asks beginners to create standalone IAM users (or group) with IAM policies directly attached. For programatic access they also include creating a pair of access key and secret access key and pass them along to an external application. The keys are long term credentials, and worse, never expires. The leakage of these long-term credentials had been such a headache that AWS strongly discourage the use of long term credentials. You can feel the discouragement when trying to create an access key through the web console, or by the banners on top of the documentation page about how to do so.

The recommendation is use temporary security credentials. In the context of AWS that means IAM roles. The users must assume an IAM role by issuing an API call, and the Security Token Service (STS) grants temporary credential in response.

This diagram has several variations. For example, the request can be AssumeRole, AssumeRoleWithSAML and AssumeRoleWithWebIdentity, depending on whether and how the user info is federated with external identity store. The returned response, a triplet of three values, makes the temporary credential that we should use in any secure environment. They must be renewed before expiry. This model works not only for human identity (e.g. SAML integration, OIDC integration, cross-account access) but also for workload identity (e.g. EC2 instance profile, Lambda execution role, ECS task role, etc).

Another good example is IAM Role for Service Account (IRSA), where a web identity represents a Kuberentes Service Account to gain role credential using the AssumeRoleWithWebIdentity API. In this post however, I’d like to explore more about the IAM role for EC2 instance profile.

The IMDS service

For EC2 instance, we all know that we can associate an IAM role as the instance profile and grant the process using AWS SDK running on the instance with permissions associated with the IAM role. At a lower level, this relies on the IMDS (Instance Metadata Service) running on the instance.

If an EC2 instance’s profile points to an IAM role, a process running on the instance using AWS SDK will also need to get the triplet from the STS. It is summarized in this diagram:

The IMDS is a service available on one of the link-local IP address (169.254.169.254) on the EC2 instance. Requests made to this IP address are not routed elsewhere. The Instance Metadata Service (IMDS) is a means for the cloud service provider’s virtualization layer to share information with the processes on the operating system of a virtual machine. It responds with information related to the instance itself, such as the subnets, IAM role, instance ID, AMI ID, security group. The instance metadata also includes user data script for cloud init process to consume, and most relevantly, the role credential for the instance. This also requires that the IMDS service to have connectivity to the STS endpoint, either via interface endpoint or over the Internet. The AssumeRole calls are logged in CloudTrail.

All major cloud vendors (AWS, Azure and GCP) uses the IMDS mechanism, and this mechanism obviously draws the attention of bad actors. I find some good articles on this here and here.

IAM Role for EC2 Workload

For EC2 instances at AWS, the initial IMDS v1 was introduced in 2012 and allows a GET method to fetch instance metadata. The IMDS v1 is subject to attacks such as SSRF (Server-side request forgery). In 2019 AWS introduced IMDS v2 which tackles those vulnerabilities. As of date, the recommendation is to use IMDSv2. Here is an example of how to fetch instance metadata, including the credential:

# Grab a token
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
# Get top-level instance metadata information
curl http://169.254.169.254/latest/meta-data/profile -H "X-aws-ec2-metadata-token: $TOKEN"
# Get the name of the role
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials
# Get the credential for the role session
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/InstanceProfileRoleName

These commands emulate how the SDK library fetches the credentials to assume the instance profile role. You can also find similar commands on the documentation. However, there isn’t much details about how the instance metadata service interacts with the STS service, except a general statement:

These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes before the expiration of the old credentials.

When creating a new EC2 instance, make sure that the instance metadata option has http_endpoint enabled, to enable the IMDS service. Also set http_tokens to required, which would run IMDSv2 exclusively. With that setup, the application does not have to mana to use an SDK version that supports IMDSv2. Another metadata option is http_put_response_hop_limit, with default of 1. This limits the number of hops in the metadata request. If the process runs from a Docker container with bridge networking mode, set it to 2 or the process cannot even secure a token.

The diagram above illustrates this scenario with two hops.

IAM Role Anywhere Concept

As this point, we know that the process of workload assuming an IAM role, is essentially using SDK to gain role credentials from instance metadata. In addition to using SDK and instance metadata, AWS also supports using X.509 certificate to gain role credentials. As a result, workload no longer needs AWS SDK, and it doesn’t rely on instance metadata from an EC2 instance. This mechanism is known as IAM Role Anywhere, and it greatly expands the use cases for IAM Role.

To make this work, we first have to provide a certificate authority (CA) to AWS as a trust anchor. It can be any X.509 CA including AWS Private CA. The IAM Roles Anywhere will allow any end-entity endorsed by this trust anchor, to assume an IAM role as specified. We also need to create a profile, in which we can add IAM policies directly, or link to an IAM roles with a trust policy for service principal rolesanywhere.amazonaws.com.

To gain role credential, the requestor must provide both the private key, and its end-entity certificate. The certificate proofs the endorsement of the CA as Role’s trust anchor specifies. The private key proofs the requestor’s identity. The requestor uses the aws_signing_helper utility to request role credentials.

The utility is compatible with the credential_process feature in AWS config, which passes the returned role credentials to the AWS config profile for AWS CLI or SDK running on external virtual machine.

IAM Role Anywhere Lab

Let’s tweak the three commands from this old post of mine to create the test materials: a self-signed CA and a certificate signed by the CA:

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Health Certificate Authority' -addext basicConstraints=critical,CA:TRUE,pathlen:1 -addext keyUsage=keyCertSign

cat > ext.cnf <<EOF
[v3_leaf]
keyUsage = digitalSignature
basicConstraints=CA:false
EOF

openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=*.digihunch.com'

openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt -extfile ext.cnf -extensions v3_leaf

I tweak them to add the X.509 extensions to meet the requirement for signature validation. We need the following files from the output.

ca.crt -> the certificate of the CA. We provide this file as the trust anchor
server.crt -> the certificate of the server, we need it as the end-entity certificate
server.key -> we need to present this file to proof identity of the requestor

First, we go to AWS console and create a new trust anchor. Copy the content of ca.crt as the certificate. Then we can create a profile with an IAM role, with the trust policy looking like this example. Then we can request the role credential with one command:

aws_signing_helper credential-process --certificate server.crt --private-key server.key --trust-anchor-arn $TRUST_ANCHOR_ARN --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN

Moreover, if the workload supports AWS SDK or can use CLI but not an EC2 instance, we can bake this in the AWS profile on the external machine:

[profile myprofile]
output = json
credential_process = aws_signing_helper credential-process --certificate /path/server.crt --private-key /path/server.key --trust-anchor-arn $TRUST_ANCHOR_ARN --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN

We can even configure this in any pipeline as code to allow deployment from a non-AWS pipeline.

Summary

In summary, apart from native AWS services, an IAM role can trust the following types of principals:

Native IAM identity such as an IAM user or an IAM group
Authenticated identity from SAML identity provider that IAM is configured to trust
Authenticated identity from OIDC identity provider that IAM is configured to trust
Validated identity endorsed by a Certificate Authority that IAM designate as a trust anchor

The first type is rarely used because few organizations uses AWS IAM as their identity store. Most organizations have their identity store with federation capability via SAML. On the other hand, a lot of modern applications adopts identity stores with OIDC compliance. Now with IAM Role Anywhere, any entity with X.509 identity can also assume an IAM role. It works with any CI/CD pipeline, whether it is self-hosted. Also, it is now more important to keep the keys safe.

Managing EC2 instances across accounts with Ansible

Mon, 27 May 2024 14:00:22 -0400

I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is consistency in the VM management, over time, and across platforms (AWS, on-prem, Azure, etc). Even for AWS shops, in the last few years many enterprises have adopted AWS landing zone with the multiple AWS account prescriptive pattern. However AWS Systems Manager still lacks integration with AWS Organization (except for a few non-core capabilities). This creates the demand of managing EC2 instances across AWS accounts. In this post, we propose a secure method to manage a fleet of EC2 instances from multiple AWS accounts, using Systems Manager . It also enables connectivity from an Ansible control node.

Prerequisites

This proposal ties a few CLI tools together, including AWS CLI, SSH, Ansible, etc. It also requires the cloud engineer to understand how they work. I’ll start with the choice of the tools.

Above, I discussed the benefit of Ansible. Since Ansible operates on SSH, we’ll still have to use SSH tools. Even though SSM agent provides a way to connect to EC2 instance without requiring an RSA key pair, we still need SSH since it is a well-established industry standard (RFC4253) and the foundation of Ansible. These two technologies are not mutually exclusive. In fact, the SSM agent provides a secure enhancement to the operation with SSH. Traditionally, on each EC2 instance we’d have to run SSHD services which opens TCP port 22 (or alternative TCP port as configured). For authentication we favour key pair over password but the open port is still an attack surface vulnerable to brute force and DDoS attacks. For EC2 instances on private networks there is no reachability to the instance’s SSH port unless the bastion is also in a connected network.

As I cover in a post, the SSM Session Manager comes in handy. The SSM agent operates from the instance and communicate outbound to AWS backend. Since the SSM agent runs under a privileged user on the OS, you can perform OS-level commands through SSM. Further, AWS developed a Session Manager plugin with AWS CLI, allowing AWS CLI as a proxy command when making an SSH connection. Therefore SSM enables SSH connection without requiring port 22 to be open. In addition we’ll need to use RSA key pair as required for SSH, which is also an improvement to the security posture.

That explains the dependent tools. On the Ansible control node, apart from Ansible itself, we need the latest version of AWS CLI with the Session Manager plugin, we need to configure AWS CLI properly to connect to EC2 instances across multiple AWS accounts.

Configure AWS CLI

This section discusses how to configure AWS CLI. I have a couple of handy aliases for productivity but they are not essential. For example, I often need to check the IAM identity making the call, and I often need to list out all profiles configured. So I added the following two entries in the ~/.aws/cli/alias file:

[toplevel]
whoami = sts get-caller-identity --no-cli-pager --output yaml
profile = configure list-profiles

With that I have an alias to check IAM identity and available profiles. Then we can start configuring the profiles for CLI (in the file ~/.aws/config). Since we’ll be working with multiple AWS accounts, we have to manage multiple CLI profiles, which implies that:

We better use the --profile switch to explicitly specify profile being used, instead of relying on the AWS_PROFILE environment variable;
As a security best practice, we should not configure profiles with long-term IAM credential in the config file;
We must ensure the CLI doesn’t prompt for log-in every time we switch profile

To satisfy #3 there are many ways but we’ll discuss two: using cross-account IAM role, and using AWS SSO.

Bonus point if you enable auto-complete for AWS CLI.

Configure AWS CLI Profiles

With cross-account IAM role, the idea, is that the client start with one IAM identity, and use that IAM identity to assume roles on several other accounts. The configuration looks like this:

[profile jump_account]
credential_process = /opt/bin/awscreds-custom --username helen

[profile target_account_1]
role_arn = arn:aws:iam::123456789011:role/OrganizationAccountAccessRole
source_profile = jump_account

[profile target_account_2]
role_arn = arn:aws:iam::123456789012:role/OrganizationAccountAccessRole
source_profile = jump_account

In this example, you start with an validated identity in the jump account, then assume a privileged IAM role named OrganizationAccountAccessRole on the target accounts. Typically such IAM roles are pre-configured (e.g. in an multi-account landing zone) with appropriate trust policy to allow principals from the jump account. Once you’re validated as the IAM identity in the jump account, then you can use profiles for target accounts without being prompted for credentials again.

If your have configured IAM Identity Center for the multi-account environment, consider an alternative approach using sso login. The configuration usually looks like this:

[sso-session sso]
sso_start_url = https://myorg.awsapps.com/start/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile target_account_1]
sso_session = sso
sso_account_id = 123456789011
sso_role_name = AWSAdministratorAccess

[profile target_account_2]
sso_session = sso
sso_account_id = 123456789012
sso_role_name = AWSAdministratorAccess

This is often used by human users with SSO credential. In this example, to authenticate the sso session, start with command “aws sso login” . Then you can use all profiles by specifying --profile switch without having to log in again.

Configure SSH to EC2 via SSM

The EC2 instance must connect to Systems Manager endpoint, before one can SSH to the instance using the plugin. Once connected, you should find the instance in Fleet Manager. For this to happen, there are a few conditions. First, the instance must be able to reach the endpoint, either via public Internet, or via VPC interface endpoints if in a private subnet. Second, the instance profile must contain an IAM role with appropriate permissions. We can use AWS managed policy AmazonSSMManagedInstanceCore in the role. In addition, if we record the SSM session to an S3 bucket with encryption the instance profile must have permission to use the encryption key.

SSM agent uses the IAM role. The agent runs as a service on Linux or Windows machines. Many AWS managed AMIs come with SSM agent pre-installed. If that is not the case, you’d install the agent in your own AMI, or in user data which requires downloading the installer. With these configuration you’d be able to connect to the instance via SSM. We can use AWS CLI SSM command, or AWS web console to start an SSH session. To use SSH CLI utility, we install the session manager plugin along with AWS CLI on the SSH client machine. We also specify a public key for EC2 instance and run ssh command with the private key. The SSH configuration needs a configuration such as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

With this entry, you may directly SSH by instance ID (usually starting with i-* and mi-*), and the specified Proxy Command with SSM session document AWS-StartSSHSession will be invoked.

Configure Ansible Inventory

We can SSH to an instance (without port 22 open, on top of SSM) using the method above. Similarly, we can also configure Ansible to connect to the instance, without port 22. The inventory configuration looks like this:

mytest:
 hosts:
 instance1:
 ansible_host: i-00aabbffcc7755221
 ansible_user: ubuntu
 ansible_ssh_common_args: -o ProxyCommand="aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1"
 instance2:
 ansible_host: i-eedd88ff66aa22442
 ansible_user: ubuntu
 ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ProxyCommand="sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --profile target_account_2 \""

Note I used two patterns for ansible_ssh_common_args that are similar. Both works. Each entry references its own profile. This is necessary because Ansible does not have the knowledge which instance belongs to which profile’s account.

Another way to get Ansible connect to instances is to use the connection plugin community.aws.aws_ssm, by specifying ansible_connection: aws_ssm (e.g. as host variable) and other required variables (e.g. profile). This method does not require SSH channel but it requires an S3 bucket, and hence IAM permission on the controller node.

Ansible supports dynamic inventory, in three ways: inventory source file (with existing plugin), custom inventory plugin, and inventory script (in Python). Take source file as an example for EC2, add the followings as the content of aws_ec2.yaml:

plugin: amazon.aws.aws_ec2

# Attach the default AWS profile
aws_profile: target_account_1

compose:
 ansible_host: instance_id
 ansible_user: "'ubuntu'"
 ansible_ssh_common_args: "'-o ProxyCommand=\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1 \"'"

Then we can display the rendered inventory list, and Ansible-ping the instances.

ansible-inventory -i aws_ec2.yaml --list -y | less
ansible all -i aws_ec2.yaml -m ping

For more flexibility, for the composed variables, we could use jinja2 expression to generate the value. In both ways, we produce an inventory source per profile using the built-in inventory plugin aws_ec2. For greater flexibility, such as consolidating instances from all accounts into a single inventory, consider writing your own inventory script, or even own inventory plugin.

Conclusion

In this post we propose a way to manage instances across AWS accounts. Two main challenges are establishing the communication channel (SSH on top of SSM) and generating inventory data in Ansible. Some AWS services can generate inventory data, such as resource data sync in Systems Manager, or using AWS Config Aggregator. It is unfortunate that neither way produces the inventory data in a format that is directly compatible with Ansible inventory. Therefore, you might have to create a custom Ansible dynamic inventory script (in Python) that reads from the inventory data from AWS Config Aggregator (which supports AWS Organization). The side benefit of this script is that it is usually faster than the built-in aws_ec2 inventory plugin.