iam on Digi Hunch

Managing EC2 instances across accounts with Ansible

Mon, 27 May 2024 14:00:22 -0400

I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is consistency in the VM management, over time, and across platforms (AWS, on-prem, Azure, etc). Even for AWS shops, in the last few years many enterprises have adopted AWS landing zone with the multiple AWS account prescriptive pattern. However AWS Systems Manager still lacks integration with AWS Organization (except for a few non-core capabilities). This creates the demand of managing EC2 instances across AWS accounts. In this post, we propose a secure method to manage a fleet of EC2 instances from multiple AWS accounts, using Systems Manager . It also enables connectivity from an Ansible control node.

Prerequisites

This proposal ties a few CLI tools together, including AWS CLI, SSH, Ansible, etc. It also requires the cloud engineer to understand how they work. I’ll start with the choice of the tools.

Above, I discussed the benefit of Ansible. Since Ansible operates on SSH, we’ll still have to use SSH tools. Even though SSM agent provides a way to connect to EC2 instance without requiring an RSA key pair, we still need SSH since it is a well-established industry standard (RFC4253) and the foundation of Ansible. These two technologies are not mutually exclusive. In fact, the SSM agent provides a secure enhancement to the operation with SSH. Traditionally, on each EC2 instance we’d have to run SSHD services which opens TCP port 22 (or alternative TCP port as configured). For authentication we favour key pair over password but the open port is still an attack surface vulnerable to brute force and DDoS attacks. For EC2 instances on private networks there is no reachability to the instance’s SSH port unless the bastion is also in a connected network.

As I cover in a post, the SSM Session Manager comes in handy. The SSM agent operates from the instance and communicate outbound to AWS backend. Since the SSM agent runs under a privileged user on the OS, you can perform OS-level commands through SSM. Further, AWS developed a Session Manager plugin with AWS CLI, allowing AWS CLI as a proxy command when making an SSH connection. Therefore SSM enables SSH connection without requiring port 22 to be open. In addition we’ll need to use RSA key pair as required for SSH, which is also an improvement to the security posture.

That explains the dependent tools. On the Ansible control node, apart from Ansible itself, we need the latest version of AWS CLI with the Session Manager plugin, we need to configure AWS CLI properly to connect to EC2 instances across multiple AWS accounts.

Configure AWS CLI

This section discusses how to configure AWS CLI. I have a couple of handy aliases for productivity but they are not essential. For example, I often need to check the IAM identity making the call, and I often need to list out all profiles configured. So I added the following two entries in the ~/.aws/cli/alias file:

[toplevel]
whoami = sts get-caller-identity --no-cli-pager --output yaml
profile = configure list-profiles

With that I have an alias to check IAM identity and available profiles. Then we can start configuring the profiles for CLI (in the file ~/.aws/config). Since we’ll be working with multiple AWS accounts, we have to manage multiple CLI profiles, which implies that:

We better use the --profile switch to explicitly specify profile being used, instead of relying on the AWS_PROFILE environment variable;
As a security best practice, we should not configure profiles with long-term IAM credential in the config file;
We must ensure the CLI doesn’t prompt for log-in every time we switch profile

To satisfy #3 there are many ways but we’ll discuss two: using cross-account IAM role, and using AWS SSO.

Bonus point if you enable auto-complete for AWS CLI.

Configure AWS CLI Profiles

With cross-account IAM role, the idea, is that the client start with one IAM identity, and use that IAM identity to assume roles on several other accounts. The configuration looks like this:

[profile jump_account]
credential_process = /opt/bin/awscreds-custom --username helen

[profile target_account_1]
role_arn = arn:aws:iam::123456789011:role/OrganizationAccountAccessRole
source_profile = jump_account

[profile target_account_2]
role_arn = arn:aws:iam::123456789012:role/OrganizationAccountAccessRole
source_profile = jump_account

In this example, you start with an validated identity in the jump account, then assume a privileged IAM role named OrganizationAccountAccessRole on the target accounts. Typically such IAM roles are pre-configured (e.g. in an multi-account landing zone) with appropriate trust policy to allow principals from the jump account. Once you’re validated as the IAM identity in the jump account, then you can use profiles for target accounts without being prompted for credentials again.

If your have configured IAM Identity Center for the multi-account environment, consider an alternative approach using sso login. The configuration usually looks like this:

[sso-session sso]
sso_start_url = https://myorg.awsapps.com/start/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile target_account_1]
sso_session = sso
sso_account_id = 123456789011
sso_role_name = AWSAdministratorAccess

[profile target_account_2]
sso_session = sso
sso_account_id = 123456789012
sso_role_name = AWSAdministratorAccess

This is often used by human users with SSO credential. In this example, to authenticate the sso session, start with command “aws sso login” . Then you can use all profiles by specifying --profile switch without having to log in again.

Configure SSH to EC2 via SSM

The EC2 instance must connect to Systems Manager endpoint, before one can SSH to the instance using the plugin. Once connected, you should find the instance in Fleet Manager. For this to happen, there are a few conditions. First, the instance must be able to reach the endpoint, either via public Internet, or via VPC interface endpoints if in a private subnet. Second, the instance profile must contain an IAM role with appropriate permissions. We can use AWS managed policy AmazonSSMManagedInstanceCore in the role. In addition, if we record the SSM session to an S3 bucket with encryption the instance profile must have permission to use the encryption key.

SSM agent uses the IAM role. The agent runs as a service on Linux or Windows machines. Many AWS managed AMIs come with SSM agent pre-installed. If that is not the case, you’d install the agent in your own AMI, or in user data which requires downloading the installer. With these configuration you’d be able to connect to the instance via SSM. We can use AWS CLI SSM command, or AWS web console to start an SSH session. To use SSH CLI utility, we install the session manager plugin along with AWS CLI on the SSH client machine. We also specify a public key for EC2 instance and run ssh command with the private key. The SSH configuration needs a configuration such as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

With this entry, you may directly SSH by instance ID (usually starting with i-* and mi-*), and the specified Proxy Command with SSM session document AWS-StartSSHSession will be invoked.

Configure Ansible Inventory

We can SSH to an instance (without port 22 open, on top of SSM) using the method above. Similarly, we can also configure Ansible to connect to the instance, without port 22. The inventory configuration looks like this:

mytest:
 hosts:
 instance1:
 ansible_host: i-00aabbffcc7755221
 ansible_user: ubuntu
 ansible_ssh_common_args: -o ProxyCommand="aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1"
 instance2:
 ansible_host: i-eedd88ff66aa22442
 ansible_user: ubuntu
 ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ProxyCommand="sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --profile target_account_2 \""

Note I used two patterns for ansible_ssh_common_args that are similar. Both works. Each entry references its own profile. This is necessary because Ansible does not have the knowledge which instance belongs to which profile’s account.

Another way to get Ansible connect to instances is to use the connection plugin community.aws.aws_ssm, by specifying ansible_connection: aws_ssm (e.g. as host variable) and other required variables (e.g. profile). This method does not require SSH channel but it requires an S3 bucket, and hence IAM permission on the controller node.

Ansible supports dynamic inventory, in three ways: inventory source file (with existing plugin), custom inventory plugin, and inventory script (in Python). Take source file as an example for EC2, add the followings as the content of aws_ec2.yaml:

plugin: amazon.aws.aws_ec2

# Attach the default AWS profile
aws_profile: target_account_1

compose:
 ansible_host: instance_id
 ansible_user: "'ubuntu'"
 ansible_ssh_common_args: "'-o ProxyCommand=\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1 \"'"

Then we can display the rendered inventory list, and Ansible-ping the instances.

ansible-inventory -i aws_ec2.yaml --list -y | less
ansible all -i aws_ec2.yaml -m ping

For more flexibility, for the composed variables, we could use jinja2 expression to generate the value. In both ways, we produce an inventory source per profile using the built-in inventory plugin aws_ec2. For greater flexibility, such as consolidating instances from all accounts into a single inventory, consider writing your own inventory script, or even own inventory plugin.

Conclusion

In this post we propose a way to manage instances across AWS accounts. Two main challenges are establishing the communication channel (SSH on top of SSM) and generating inventory data in Ansible. Some AWS services can generate inventory data, such as resource data sync in Systems Manager, or using AWS Config Aggregator. It is unfortunate that neither way produces the inventory data in a format that is directly compatible with Ansible inventory. Therefore, you might have to create a custom Ansible dynamic inventory script (in Python) that reads from the inventory data from AWS Config Aggregator (which supports AWS Organization). The side benefit of this script is that it is usually faster than the built-in aws_ec2 inventory plugin.

Workload Identity on Kubernetes 2 of 2 – EKS

Mon, 08 Jan 2024 00:05:00 -0400

I discussed in my previous post on workload identity and dived into how it works in AKS (Azure Kubernetes Service). In this post I will continue the topic with AWS as the example. From the perspective of CSP, we consider any running process on the cloud resource as workload. Therefore, I’ll start with control plan and node identities. From the perspective of a Kubernetes platform, the term workload mostly refers to applications running in Pods. So later in this article I’ll distinguish two mechanisms for Pod Identity: IRSA and EKS Pod Identity.

EKS Control Plane and Node Identity

AWS directly associate an IAM role with EKS control plane and an IAM role with each node group. We don’t need an extra step of assigning a “managed identity” (as in Azure) to a cluster or to a node group ( and then bind a role to the identity). You can find this pattern from Terraform code. Each aws_eks_node_group resource has a node_role_arn attribute to link to its IAM role, and a cluster_name attribute to link to the cluster. Each aws_eks_cluster resource has a role_arn attribute for cluster’s permission.

The cluster’s IAM role is usually bound to managed policies like AmazonEKSVPCResourceController and AmazonEKSClusterPolicy. The IAM role that is assigned to the node group is the exact IAM role of the instance profile of each node. The kubelet process on the nodes are the main users of this role and the permission should not be broader than what it needs to do. This role usually have a few managed policies such as AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonSSMManagedInstanceCore and AmazonEC2ContainerRegistryReadOnly.

The node role applies to self-managed node and managed node. When using Fargate to provide computing capacity, each Fargate profile will use its own IAM role, to connect to the cluster and pull container images. This IAM role is known as Pod Execution Role. For a private cluster, the place to run the command would be a bastion host with connectivity to the cluster’s API endpoint. Refer to this post about the connectivity to private cluster.

IAM Role for Service Account (IRSA)

When AWS launched EKS in 2018, Kiam was a popular open-source project to grant Pods access to AWS resources. In 2019, AWS introduced the official mechanism, IRSA (IAM Role for Service Account). IRSA ties a Kubernetes identity (in the form of Service Account) to an IAM role in AWS. IAM allows creation of web identity based on OIDC. EKS can act as an OIDC issuer.

This requires a few points of configurations, via the cluster API and via cloud the endpoint. The eksctl utility makes it simple with two commands:

$ eksctl utils associate-iam-oidc-provider \
 --cluster $CLUSTER_NAME \
 --approve

$ eksctl create iamserviceaccount \
 --cluster=$CLUSTER_NAME \
 --namespace=kube-system \
 --name=aws-load-balancer-controller \
 --role-name AmazonEKSLoadBalancerControllerRole \
 --attach-policy-arn=arn:aws:iam::112233445566:policy/AWSLoadBalancerControllerIAMPolicy \
 --approve

The first command creates an OIDC web identity integrated with the EKS cluster, and the second creates a Service Account in Kubernetes and links it to the identity, and assign an IAM policy. These two commands must run under certain conditions. The AWS CLI identity for first command requires the the permission to add OIDC provider. The second needs the permission to create an IAM role. In addition, it requires kube API access to the cluster. So the command needs to run from an environment that can access both the cluster’s API and AWS API.

The IAM identity provider is somewhat similar to a managed identity with OIDC federated credential in Azure. However, unlike managed identity, here in AWS we cannot create the OIDC identity until after the cluster creation. In other words, the OIDC web identity’s lifecycle does not decouple with the cluster lifecycle. We have to create a new web identity every time we create a new EKS cluster. In large organizations, the permission to create a new web identity is highly restricted.

EKS Pod Identity

There are a few other limitations with IRSA. As this blog post suggests:

Further, cluster administrators have to update the IAM role trust policy each time the role is used in a new cluster during scenarios like blue-green upgrades or failover testing. Additionally, as customers grow their EKS cluster footprint, due to the per cluster OIDC provider requirement in IRSA, customers run into the per account OIDC provider limit. Similarly, as they scale the number of clusters or Kubernetes namespaces in which an IAM role is used, they run into IAM trust policy size limit, which makes them duplicate the IAM roles to overcome the trust policy size limit.

AWS brings the new mechanism “EKS Pod Identity” at reInvent 2023. In this mechanism, user can hook up an IAM role directly to a Kubernetes service account, without having to resort to a web identity and OIDC integration. Users just need to create a Pod Identity Association, using the CreatePodIdentityAssociation API, with the following parameters:

Cluster name
Namespace
ARN of the IAM role
serviceAccount

Both AWS CLI and ekscli already support the CreatePodIdentityAssociation API. Before creating a Pod Identity Association, we need to install the add-on “Amazon EKS Pod Identity Agent”, and ensure that the node roles have the permission. That is because the agent needs to use AssumeRoleForPodIdentity API. We also need an IAM role, with the trust policy principal being “pods.eks.amazonaws.com” and our own choice of resource tags as condition. Note that another implicit prerequisite is that the programming running in the Pod use a newer version of AWS SDK to access cloud resource.

This blog post has good details, including a diagram and a walk-through.

Comparison

Both EKS Pod Identity and IRSA are here to stay. I’m afraid this is going to create confusions. I put the following table for their comparision:

	IRSA	EKS Pod Identity
Pros	– in use since 2019 – support EKS, EKS-A, ROSA – support all EKS versions	– support role session tags – no dependency on OIDC identity provider – create an IAM role once for all clusters. the role can be created before cluster – cross account access through resource policies and chained AssumeRole operation
Cons	– Cannot create OIDC identity provider, until the cluster is ready – One OIDC provider per cluster, with the risk of hitting quota – Trust policy sprawl as more clusters are created	– the program has to use newer version of SDK. – ony support EKS – Pod Identity Agent (DaemonSet) can’t run on Fargate

Compairing IRSA with EKS Pod Identity

The blog post also contains a long table for their comparison. In the near future, I will have to check the SDK version of a workload in order to assess whether EKS Pod Identity will function. This is a restriction because it depends upon software builder disclosing the SDK version used. The EKS cluster also needs to host daemonSet on a node agent. On the other hand, go with IRSA if portability between EKS and EKS-A and ROSA is of concern, because the IAM service principal pods.eks.amazonaws.com is dedicated to EKS.

The blog post also gives the migration step as follows:

Ensure EKS cluster is above 1.24, and install the add-on for EKS pod identity agent.
Ensure the SDK running in pod meets the version requirement.
Update the IAM role’s trust policy with the new principal “pods.eks.amazonaws.com”

So the EKS Pod Identity mechanism still requires an IAM role. It does not required an OIDC identity. The service account connects to IAM role via an agent on the node.

Summary

A good design concerns not only functionality, but also streamlined configuration experience. EKS Pod Identity is a great improvement over IRSA heading the right direction. It just came out two months ago so still too early to adopt, especially without knowing the workload details. For now I tend to use pod identity as a backup mechanism when IRSA isn’t available for some reason. However, I recommend starting to introduce the Pod Identity mechanism for all new EKS clusters and new workloads.

OAuth 2.0 and OIDC 2 of 2

Thu, 13 Jul 2023 21:24:01 -0400

I wrote a brief on this topic a while back. Now I need to configure OIDC in a few occasions I decide to dive deeper into the flows this time. As I stated in the last post, Nate Barbettini’s presentation from 2017 was awesome and I viewed it again. Slides are available here. Another great reference is this post from DeepNetwork Developer’s blog.

Back Channel and Front Channel

To understand why there are several different flows, it is important to understand the difference between back channel and front end channel.

In web development, traditional architecture involves a frontend (e.g. Browser, or any client-side app) and backend server. The web frontend is written in HTML, CSS, JavaScript, etc. There are also web frameworks such as Django, Angular, to save developers time. Backend (server-side) is responsible for storing and organizing data to ensure frontend can function. There might be multiple backend servers, such as session cache, data store, API server, etc.

In a nutshell, server-to-server communication is back channel, and browser-to-server communication is front channel. From security perspective, we regard front-channel as less secure, because we have less control of the location of the front-end and browser is easy to tamper with.

Back Channel and Front Channel (source https://www.okta.com/blog/2019/04/oauth-when-things-go-wrong/)

For traditional web applications with client-server architecture, when they communicate with third-party servers, they can initiate the communication from their backends, creating a back channel for better security posture. Single-page applications (SPAs) are applications without their own backends. When building an SPA, the front-end developer deals with frontend frameworks such as React, Angular or Next.js. When SPAs have to communicate with third-party APIs, they have to create a front channel. Also the API must support Cross-Origin Resource Sharing (CORS) for the browser to allow the cross-domain communication.

In the web development world, there is also Native App (aka Mobile App). Native App to Server communication is also considered back channel as we consider the client-side (Mobile App) secure. Nate’s talk makes the following recommendation for the flow (grant type) and I rephrase it as below:

Architecture	Recommendation
Traditional Web Application (client-server architecture)	There are both front and back channels. Use authorization code flow
Single Page Application (SPA, e.g. JavaScript) with API backend	Front channel only. Use implicit flow
Native mobile App	Use authorization code flow with PKCE
Microservices and APIs (Machine-to-Machine)	Back-channel only. Use Client credentials flow

The message, is that we should use Authorization Code Flow and use back channel, so long as the required component (backend) exists in the architecture. Note that the terminology for these types of applications may differ. For example, when you try to create an OIDC client for Amazon Cognito user pool, here’s how it categorizes client capability (app type):

Public client: A native, browser, or mobil-device app. Cognito API requests are made from user systems that are not trusted with a client secret.
Confidential client: A server-side application that can securely store a client secret. Cognito API requests are made from a central server.

Get used to different terms describing the same concept.

OAuth 2.0 and OIDC flows

The original problem that OAuth 2.0 (RFC6749) addresses is delegated authorization. In all OAuth flows, the authorization server issues an Access Token for the client to carry. The Access Token identifies the scope of resources that its carrier is authorized to access. However, third-party services do not always want to delegate authorization to the identity provider. They often just need identity information and want to perform authorization on their own. The OAuth 2.0 Access Token itself is all about permission and does not care about the identify of its carrier. It is not designed for authentication. OAuth 2.0 does not provide a standard way for Authorization Server to keep identity information of the principal. Many developers needs to address identity issue and they started to (mis)use the AccessToken to store identity information in custom fields, until OIDC came about.

OAuth: Access Token only

OIDC: Access + Identity Token

OIDC is a thin layer (5%) on top of OAuth 2.0 and one important addition is the ID token. The resource server can, in addition to being asked to allow access, now can understand the identity of the principal requesting to access resources from the ID token. The OIDC layer also uses standard set of scopes and proposes a userinfo endpoint for client to get more details about user information. The authorization code flows in OIDC and OAuth2.0 are roughly the same except for the additions.

In terms of the flows supported, the OAuth flows are defined in RFC6749, including the following grants:

Authorization Code grant
Implicit grant
Resource Owner Password Credentials grant
Client Credentials grant

The classic grant type is Authorization Code. After verifying with user, the Authorization Server fires a call-back to the client to pass the authorization code. The client, then takes the authorization code, along with its client ID and client secret, to fire a request to Authorization server in exchange for Access Token. The implicit grant skips the Authorization Code step and the client gets the Access Token in a one-stop shop via callback over front channel, which is less secure. The other two grants are less often used.

On the OIDC side, the specification document discusses three flows:

Authorization Code Flow (specification 3.1)
Implicit Flow (specification 3.2)
Hybrid Flow (specification 3.3)

The original OAuth2.0 flows should only be used in delegated authorization scenario. In most contexts, if we talk about ID token, and if our use case involves authentication, then we’re talking about OIDC not just OAuth. Since Authorization Code Flow is the classic one, out of all these flows, in the next section we take a closer look at the Authorization Code Flow in OIDC.

Authorization Code Flow in OIDC

We consider the Authorization Code Flow the baseline flow and others as variations of it owing to architectural limitations. When we mention OIDC we implicitly refers to the Authorization Code Flow unless the context suggests otherwise. Now let’s zoom in on it:

Here is the narrative from my own words:

The user launches client application, which detects that user has not logged in, and redirect to log in page.
The client app sends an HTTP request for authorization code to the /authorize endpoint of Authorization server. This request consists of the following fields:
- Response type: code, indicating it is requesting authorization code
- Scope: openid, standard for oidc
- RedirectURI: my.com/oidc-callback, Authorization Server will use this to call back with code.
The authorization server redirects the user to a prompt for log-in
The user completes authentication and consent
The Authorization server validates user information within its identity provider
The Authorization server fires an HTTP request call-back at the Redirect URI (on the backend), with Authorization Code.
The client app issues an HTTP request for ID Token and Access Token to the /token endpoint of Authorization server. This request consists of:
- Authorization Code (received from previous step)
- Client ID
- Client Secret
The Authorization server validates the information and process the request, and prepare the response with the following fields:
- ID Token: identifies the resource owner.
- Access Token: identifies what the client app can access
- Expiration
- (Optional) Refresh Token
The client app receives the tokens above in the HTTP response from the /token endpoint
With the Tokens, the client app issues API requests to the resource server
The resource server independently validates the token
The resource server send API response back the the client app.

There are also a few points of configurations. First, the Resource Server needs to trust the Authorization Server. The Authorization Server uses its private key to sign the JWT tokens and the Resource Server needs the public key to validates it. Second, the authorization server needs to know about the client app. We usually configure the Authorization Server upfront, to generate the client ID and secret. The Client app will keep them as part of its configuration.

It is important to note that, when I use the term client app (OIDC calls it client), the word “client” is relative to the Authorization server. The client app itself consists of both frontend (browser) and backend (aka relying party). In this flow, the authorization code is not exposed to browser.

Other OIDC Flows

Now we can discuss some flows for other architectures.

The Authorization Code flow has a close variation with the use of PKCE (Proof Key for Code Exchange). For native apps, postman and Okta recommend Authorization Code flow with PKCE. When client app first requests for authorization code, it also includes a challenge. After the callback, when it sends the authorization code back to authorization server in exchange for tokens, the request now adds a verifier. This way, even if the authorization code may not be securely saved, the authorization server can ensure it is the same client app that requests authorization code and that requests tokens.

An SPA or JavaScript app does not have a way to a. store Authorization code and b. listen on a call-back URI. As a result, it makes sense for the SPA to just fetch the Tokens directly. This make the implicit flow. The spec doc refers to it as simplified authorization code flow. The grant type is “implicit” because there is no intermediate credentials issued. In this flow, the Authorization server does not authenticate its client. The tokens may be exposed to resource owner or other applications with access to resource owner’s user-agent. This flow improve the responsiveness but we should be wary of the security implications.

As stated above, for SPAs (or in-browser JavaScript), we have to choice but the Implicit Flow because the client app is front-end only and we do not consider it able to securely store credentials. Client apps that can securely store client credentials may benefit from Hybrid Flow. In the hybrid flow, when the authorization server fires callback, the callback includes a single-use authorization code, along with ID token, access token, or both, depending on the provided response_type. Then the client app sends it back to authorization server, along with client credentials, in exchange for a second ID token and access token. The specification has a good table that compares the three flows:

Apart from the three flows, there are also some “unofficial” OIDC flows, that are not discussed in the specification. For example, Auth0 adopted some original OAuth2.0 grants in conformance to OIDC, including Client Credentials Flow with OIDC and Resource Owner Password Flow with OIDC. The client credentials flow is for machine-to-machine identity and is not concerned with user identity.

Identify the Flow

A challenge that I faced is to make sense of the Authorization Code Flow in real life. I realized that the components (Client App, Authorization Server and Resource Server) in Authorization Code Flow are conceptual. In real life we do not always find a counterpart that perfectly match their features. When we try to introduce OIDC for authentication, we often need to build our own solution, with additional tools.

Take the client app for example. We would need one or several components in real life to perform the followings in order to qualify as a Client App in the sense of Authorization Code Flow, it needs to:

know the authorization endpoint and construct the HTTP request for Authorization code;
stand up an HTTP service (relying party) to listen to call back, and parse the Authorization Code;
securely store Authorization code, and have access to client ID and client secret;
construct a request for tokens using client ID, secret and authorization Code received;
parse the tokens from the response from Token endpoint
to pass the tokens along

When developers builds an application with OIDC integration capability, they’d have to implement all these using the library of their programming language. In addition to application’s own server, the OIDC module will need its own backend capable of doing all the activities above. The alternative option is to introduce a OIDC capable client proxy service.

As for the Resource Server, it needs to have a trust on the Authorization Server, so that it can cryptographically validate the tokens that the Authorization Server has issued using the well-known public key.

On the Authorization Server side, as we discussed. It needs to provision client ID and client secret that itself can later recognize when client app connects to it. It also needs to have both authorization endpoint and token endpoint. Often times, the authorization server contains identity store and we’d like to call it the identity provider, but that is not always the case. A company may have a home grown identity store that does not support OIDC. In that case, to qualify as an OIDC Authorization server, they need a server proxy.

Open ID Connect Specifications

Despite the different implementation by different vendors, we often need to resort to the official standard documentation. This Open ID connect page lists all the specification if you expand “OpenID Connect specification” under Final Specifications. The most commonly used ones are:

OpenID Connect Core specification [OpenID.Core.Errata2], which covers the foundation and three login flows (Authorization Code, Implicit and Hybrid). This was developed early and the current version [OpenID.Core.Errata2] is from Dec 2023 but the two previous versions [OpenID.Core.Errata1] and [OpenID.Core.Final] had been around since 2014;
Open ID Connect Session Management [OpenID.Session], another core document that stipulates how to manage sessions, finalized in Sept 2022;
Open ID Connect Discovery 1.0 [OpenID.Discovery], which stipulates the hosting OIDC discovery document, finalized in Dec 2023;
Open ID RP-Initiated Logout [OpenID.RPInitiated], one of the logout flow specification, drafted in 2020 and finalized in Sep 2022;
Open ID Front-Channel Logout [OpenID.FrontChannel], one of the logout flow specification, drafted from March 2016 and finalized in Oct 2022;
Open ID Back-Channel Logout [OpenID.BackChannel], one of the logout flow specification, drafted in 2016 and finalized in Sep 2022;

Do read the specification when you’re configuring integration. It is worth noting that apart from the Core specification which has been finalized for a decade, most of the other specifications did not finalize until late 2022. Therefore, it is important for integrators to validate the compliance state of the components in the implementation.

OIDC Proxy

As a result, with regard to OIDC, there are two categories of proxies: OIDC client proxy and OIDC server proxy.

For example, in my previous post, I explained how to configure external authorization via OIDC in Istio. Looking at the diagram, it uses OAuth2 proxy to integrate with GCP as the authorization server. In this use case, GCP is natively OIDC capable, the the Hello Word App isn’t. Therefore, the OAuth2-proxy that we introduced is an OIDC client proxy.

For a corporate with Active Directory, the identity store only supports LDAP protocol. In order to quality the identity store as an OIDC Authorization Server, we would need a server-side proxy such as the LDAP connector in Dex, with the Active Directory as authentication source. The diagram of dex is a good summary of its role:

In some scenarios, we refer to this role as identity broker. Dex is an identity broker. Another important project to know is KeyCloak, which is sponsored by Red Hat and now a CNCF project. Although you can configure KeyCloak as an identity broker, it is much more than a broker. KeyCloadk is a full-fledged identity and access management solution on its own. It can act as the entire Authorization server. The diagram in this blog post summarizes its features well. It is for teams that wants to build their home grown identity store. Think of KeyCloak as a self-managed open-source alternative to IAM solutions such as Okta or Auth0.

The Amazon Cognito user pool plays a similar role. A user pool serves as an identity store to an app. The integration (no matter which flow and how Cognito calls them) is supposed to be OIDC compliant. However, as of date, the integration with Cognito user pool isn’t. For example, the logout endpoint requires client_id parameter where as the RP initiated logout specification has it optional. On the other hand, it can federate its own identity pool with a third party via standard protocol including OIDC.

In any use case where we need to bring OIDC integration, we need to start with the flow recommendation for each architecture, then we examine the existing component against the flow diagram. From there, we can identify the missing pieces and determine where and how we should configure the proxy.

Summary

The OIDC topic confuses me big time every time I need to configure identity store. With this post, I was hoping to elaborate on the Authorization Code Flow for OIDC. See OpenID certification for a list of providers. Further I discussed the two categories of proxies in the OIDC picture. Hopefully, when the OIDC topic comes back again, I will be able to quickly match which is which, and identify the missing piece to build a solution.

EKS impression

Fri, 23 Dec 2022 18:18:19 -0400

I’ve worked on a few AKS projects previously. Since I joined AWS I wanted to put aside some time to check out EKS (Elastic Kubernetes Service). Here in this post, I put down my first impression on EKS, and also share my Terraform template in cloudkube project to create an EKS cluster.

Similar to AKS, EKS exposes API endpoint and the control plane components are hidden from AWS users. When creating EKS cluster it does not create the underlying VPC and subnets. Therefore, you have create an existing VPC and at least two subnets ahead of time, and specify them during EKS creation. Bear in mind that there is a list of requirement for the VPC and subnets.

In the cluster, the CNI that EKS officially supports is Amazon VPC CNI plugin. It is available as an add-on. Similar to Azure CNI, each Pod gets its own IP address. In addition, EKS supports other compatible CNI plugins such as Calico, Cilium, Weave Net and Antrea.

Computing Nodes in EKS

There are three modes to address computing capacity: self-managed nodes, EKS managed node groups and AWS Fargate. The documentation has a comparison table.

With self-managed nodes, users create EC2 instances separately and then register them to the control plane. The instances must use the same IAM role and AMI. You can use Auto Scaling groups of Bottlerocket (AWS-sponsored purpose-built Linux distro for container host) nodes. The self-managed node option is mostly for AWS outpost customers who bring in their own computing capacity from data centre.

If you provision computing capacity from AWS, it makes sense to assign EKS managed node groups when creating EKS cluster. We can turn on Cluster Autoscaler, a Kubernetes construct to manage the auto scaling of node groups. Sometimes we want to have more than one node groups. For example, to build a multi-architecture cluster, we need one node group with amd64 nodes and the other with arm64 nodes (e.g. instances with Graviton processor). In general, arm-based CPU delivers better performance with less power consumption and the industry is slowly moving towards more arm-based CPU architecture.

Fargate is what I call managed computing service for EKS. With Fargate you do not need to tweak Cluster Autoscaler to self-manage computing capacity. The Fargate documentation has a long list of considerations. For example, Pods must match a Fargate profile (here‘s an example) at the time that they’re scheduled to run on Fargate. So we need to build Fargate profile and Pod labelling properly. Also, Fargate does not support DaemonSet. Another big consideration is that Fargate does not support non-VPC CNI. In my opinion these are pretty significant limitations. Many workloads (system-level or application-level) would need Daemonset (e.g. kube-proxy, some CNI or CSI drivers, Dynatrace monitoring).

The pro of Fargate is the serverless computing model. The construct of a Fargate profile isn’t complicated. You just specify subnets, namespace and labels. However, the downside is the long list of considerations. Some teams may consider these restrictions too much. The other overhead is the need to manage Fargate profile to ensure all Pods are scheduled somewhere.

To me, using Fargate alone impairs portability of workload. The good thing is that Fargate and Managed Node Group are not mutually exclusive on a cluster. In most cases, we can go partially serverless, and reap the benefits of both of them.

Node AutoScaling

For workloads that don’t have a matching Fargate profile, we have to figure out node autoscaling ourselves. I touched on Cluster Autoscaler in “Autoscaling on Kubernetes Platform“. CA works on AWS as well and is triggered upon a Pod coming to unschedulable status in Scheduler. There is some limitations though. For example, CA interacts with Autoscaling Group (instead of EC2 instances directly). When it determines it’s time to scale up, it bumps up the desired capacity by one at a time in the Autoscaling group. The configurations in Autoscaling group may also be at play and CA do not have direct control. For example, the “scaling cooldown“. The pool of nodes is homogenous as per the pre-configured launch template and CA has no control. If a Pod requires a different type of node (e.g. ARM64 CPU, spot instance, etc), then we’d first have to create a node group with the desired node type. Moreover, in the worst cases, one-at-a-time scale-up does not meet the increase of demand driven by Pod increases, causing nuances such as racing conditions.

Because the Cluster Autoscaler doesn’t really deal with the nodes themselves, this kind of integration is clunky and slow. Nearly half of Kubernetes customers on AWS report that configuring cluster auto scaling using the Kubernetes Cluster Autoscaler is challenging and restrictive, according to this blog post. As a result, AWS launched an open-source cluster autoscaler project, Karpenter. Karpenter first only supported EKS but now the support includes other CSPs. For EKS, Karpenter directly interact with different types of EC2 instances.

Karpenter makes node scaling work in a more cloud-native manner. In the presence of unschedulable Pods, Karpenter bypasses the Kubernetes scheduler and works directly with the Cloud provider, to launch the minimal compute resources needed to fit those Pods and immediately binds the Pods to the newly provisioned Nodes without waiting for scheduler. As Pods are removed or rescheduled to other nodes, Karpenter looks for opportunities to terminate under-utilized nodes. Karpender defines a CR called Provisioner to specify node provisioning configuration, such as instance size, zone, CPU architecture, etc. It is a manifest that describes a node group so the node scaler is aware of all the available node types. You can have multiple Provisioners for different needs, just like node groups. The Provisioner CR can also set TTL for empty Nodes, such that once a Node has no pods other than DaemonSet, Karpenter will terminate the Node on TTL expiry.

Karpenter’s idea is similar to the idea of AutoPilot cluster in GKE. The new EKS workshop has an section on how to set up CA and Karpenter in practice.

Identity Management for EKS

For IAM, we need to be concerned with three aspects. The management traffic to the cloud service, the management traffic for Kubernetes cluster and business traffic.

Traffic type	AWS	Azure
I. Cloud Service Endpoint (Management Traffic for Cloud Service)	AWS IAM identity	Azure RBAC
II. Kubernetes API (Management Traffic for K8s Cluster)	IAM mapping or OIDC	Azure RBAC (implementation of OIDC)
III. Business traffic	Up to Kubernetes Ingress	Up to Kubernetes Ingress

For business traffic (type III), it is all up to the Ingress. I’ve written another article on managing ingress traffic on Kubernetes platforms. We interact with cloud service endpoint (type II) with either AWS CLI or Terraform, to create any object, including resources needed for a cluster. This is generally how we work with cloud service, not specific to Kubernetes. Usually the IAM identity assumes another IAM role, which empowers it with a lot of permissions.

For access to Kubernetes API (type III), EKS supports OIDC and IAM mapping. AWS documentation refers to this as “Cluster Authentication“. There is one special scenario where your identity for type II access inherits your identity for type I access. As the document puts:

When you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s role-based access control (RBAC) configuration in the Amazon EKS control plane. This IAM entity doesn’t appear in any visible configuration, so make sure to keep track of which IAM entity originally created the cluster.

This special scenario (I call it the “implicit master user“) allows us to perform critical activities on the cluster, such as creating IAM mapping, or OIDC configuration.

The above addressed how AWS resource access Kubernetes resource. On the other hand, to address how a Kubernetes resource access AWS resources, we need IRSA (IAM Roles for Service Account). We have a service account in Kubernetes and map it to an IAM role.

AppMesh

AppMesh is AWS’ Envoy based service-mesh offering supporting Kubernetes cluster, ECS service and even EC2 instance. AppMesh’s control plane is a managed AWS service, with a controller running on the Kubernetes cluster. To install AppMesh on the cluster:

On the EKS cluster, install AppMesh Controller using Helm
Associate the cluster with IAM OIDC provider
Create an IAM role for the appmesh-controller service account

After these steps, you can create a mesh using CloudFormation, Terraform, etc. The data plane (Envoy proxy) can run on Kubernetes (as sidecar). Traffic between control plane and data plane can go through private link (Interface VPC endpoint) for added security. Like Istio, AppMesh enables mTLS. For observability, you can export Envoy metrics with Prometheus. Coupled with XRay, AppMesh also supports distributed tracing.

AppMesh uses a different set of CRDs than Istio. Key CRDs are:

Mesh: represents an entire service mesh. At mesh level you can configure Egress filter (to allow or deny external traffic) and set IP version (v4 vs v6)
VirtualGateway: a CRD that represents an Ingress in to the Mesh. A virtual gateway allows resources that are outside of your mesh to communicate to resources that are inside of your mesh. A virtual gateway references Envoy proxy deployment by podSelector. It references GatewayRoutes by namespaceSelector, and optionally gatewayRouteSelector. You also specify listeners in the manifest to reference Envoy proxy Service (LoadBalancer Type).
GatewayRoute: A gateway route is attached to a virtual gateway and routes traffic to an existing virtual service. If a route matches a request, it can distribute traffic to a target virtual service. In the manifest, you specify a list of httpRoute, each with matching condition and action. In the action section you can specify virtualService as target.
VirtualService: an abstraction of a real service provided by a virtual node directly or indirectly by means of a virtual router. Dependent services call your virtual service by its virtualServiceName, and those requests are routed to the VirtualNode or VirtualRouter that is specified as the provider for the VirtualService.
VirtualRouter: Virtual routers handle traffic for virtual services. In a virtual router manifest, you can define Route to direct incoming requests to virtual nodes as target.
VirtualNode: A virtual node acts as a logical pointer to a particular task group (i.e. ECS service, Kubernetes deployment). It represent a Service in the AppMesh. In the manifest, you reference Pods by podSelector, specify listeners for any inbound traffic that your virtual node expects, and specify serviceDiscovery for your task group.

You can configure those Custom Resources using YAML manifests (and check the API reference a lot). Alternatively, you can configure them from AWS CLI or AWS console. The console will help you visualize what can be configured. For further details on how these CRs play together, there is a workshop for AppMesh.

EKS cluster using Terraform

Officially, there is an EKS blueprint project for provisioning EKS cluster in Terraform.

I also keep my own Terraform code in the AWS directory of cloudkube project. It works out to be a little more complex than my Terraform template to create Azure Kubernetes Cluster (Azure directory). Because I had to create Cognito resources with initial credential to allow users to connect to cluster without using the implicit master account.

Below is the diagram of the processes.

Create EKS cluster with Terraform module

The template configures kubectl access on a Bastion host, which assumed the same role that our IAM user uses to create the Kubernetes cluster. Therefore, the IAM role is the master identity.

Note that the IAM user (power-user) has very powerful permissions. Usually it is ideal to assign lots of permission to IAM Roles (temporary credential) instead of IAM user (long-term credential). So the role chaining would look like:

The IAM user that Terraform uses has no permission other than assuming a “PowerUser” role
The PowerUser role trusts the IAM user. It also has the permission to assume the “EKS-Manager” role
The EKS-Manager role trusts PowerUser’s role session.

However, the role chaining scenario above is not currently supported in Terraform.

I use a Bastion host because the cluster endpoint is on private subnet. The bastion host is on a public subnet. However, if we do not like public subnet and public IP, we can place the bastion host on a private subnet, and use SSM system manager agent with SSH tunnel plugin to have SSH access to private bastion host.

Summary

I first came across this article about EKS and its awfulness and then decided to check out EKS. I’m not sure all points are still valid but it’s generally real-life experiences. There are also many peripheral services, such as AMP (AWS Managed Prometheus), AMG (AWS Managed Grafana), ADOT (AWS Distro for Open Telemetry), AppMesh (Another Envoy-based Service Mesh, easier to manage than Istio but less Powerful), with a lot to explore.