aks on Digi Hunch

Workload Identity on Kubernetes 1 of 2 – AKS

Sat, 23 Dec 2023 16:43:00 -0400

As applications are moved to the cloud, the application workload hosted on virtual machines need to interact with cloud resources. For this, we need an IAM solution with two mechanisms:

a (non-human) identity in the cloud service platform (CSP), to represent the application;
a way to grant permission to this identity, so it can manage resources

CSPs such as Azure and AWS have their own implementations of the two mechanism. In Azure, we have Entra workload identity (including service principal and managed identity) for the first mechanism, and Azure roles for the second. On AWS, they are the identity pool capability of Amazon Cognito and IAM role. Next, what about the workload on managed Kubernetes service? Essentially, we will need to more mechanisms:

a native Kubernetes identity to represent the workload (Pod);
a way to map the Kubernetes identity to the identity in CSP

Kubernetes Service Account is designed for the first item. The second mechanism is for the CSP to address. In this post, let’s examine this in Azure. Specifically, how does Azure manage workload identity with Azure Kubernetes Service (AKS).

Node Identity and Cluster Identity in AKS

Let’s define what exactly is a workload identity. In Azure we think of it as one type of non-human identity. In our context, workload identity in the broader sense contains:

the identity that represents the control plane (or the whole cluster)
the identity that represents the node (or kubelet process)
the identity that represents the application in a Pod (workload identity in the narrow sense);

So it is important not to confuse these identities. In this section, I’ll focus on 1 and 2 since they are part of workload identity in the broad sense. In the rest of the sections, I’ll discuss 3, and use the narrow sense of workload identity.

When we create an AKS cluster, we create both a cluster control plane and a node pool. Both the control plane and the nodes need to provision cloud resources using Cloud API from Azure. For example, if we use Terraform’s AzureRM provider to create an azurerm_kubernetes_cluster resource, then we specify the cluster’s identity using service_principal or identity block. We specify the nodes’ identity using the kubelet_identity block, because kubelet is the process that runs on each node.

Even though a cluster builder might be tempted to assign the same identity to both Control plane and kubelet, the security best practice is to keep them separated. It is also the responsibility of the cluster builder to distinguish activities by the control plane and by kubelet process on each node, and attache an Azure Role with minimum privilege to each of the identities.

These two types of identities (control plane and kubelet) are relatively straightforward. In order to use them, we don’t have to play with Kubernetes objects. In the next section, we’ll continue to discuss the identities that represents each Pod in Azure. We now refer to them as workload identities, but the first available technology was pod managed identity.

Pod Managed Identity in AKS

When I first worked on Azure Kubernetes, Pod managed identity was in preview and was the recommendation. However, Microsoft renamed it (to Microsoft Entra pod-managed identities) and then deprecated it after a couple years of preview. As of Oct 2022, the recommended mechanism becomes Microsoft Entra Workload ID. For simplicity, we refer to the deprecated mechanism as “Pod Identity”. We discuss pod identity only for the purpose of understanding why it is no longer recommended and what is missing in it. For new workload deployment, we should always use workload identity.

For Pod Identity to work, a feature flag EnablePodIdentityPreview must turn on. Pod Managed Identity operates on a Kubernetes controller called MIC (Managed Identity Controller) and a DaemonSet called NMI (Node Managed identity). You start with an Azure managed identity with appropriate roles. Once you installed Pod Identity, there will be two CRDs AzureIdentity and AzureIdentityBinding. To grant Azure permissions to a Pod, you create a CR for each CRD. The AzureIdentity CR connects to your Azure managed identity. You also create an AzureIdentityBindign CR. When declaring a Pod, you link to AzureIdentityBinding by using the label aadpodidbinding.

There are two problems with pod managed identity. First, there is a vulnerability when it works with kubenet as network plugin. This vulnerability requires an additional mitigation step. Second, it does not make use of Kubernete’s Service Account. Let’s discuss in the next section why it’s favourable to use Kubernetes’ ServiceAccount.

Kubernetes Service Account

In Kubernetes RBAC model, Service Account can bind to Roles to gain access to other Kubernetes resources. The most common use case is allowing the running application in a Pod to access other Kubernetes resources. When it comes to letting an application in Pod access cloud resources in the CSP, it makes sense to use Service Account, for a consistent pattern.

A service account must carry a token to function. Each namespace has a default service account with the token mounted automatically. Each Pod created in a namespace uses the default service account of the namespace, unless otherwise specified. However, many security organizations do not considered this default behaviour as the best practice. For example, CIS Kubernetes benchmark 1.8 has these two recommendations:

Ensure that the default service accounts are not actively used (5.1.5)
Ensure that Service Account Tokens are only mounted where necessary (5.1.6)

In other words, we should create non-default service account with automountServiceAccountToken set to false. Then when declaring a Pod, we explicitly specify the service account and where to grab the token for the service account. One way to pass ServiceAccount token is through volume projection.

To allow a Pod to access Azure resources, we use the combination of Kubernetes Service Account and Microsoft Entra workload identity.

Workload Identity for AKS

Microsoft introduced Entra Workload Identities in late 2022 to address IAM issues around machine identities. It comes with some modern features such as conditional access (e.g. location-based access, anomaly sign-in detection, etc). A workload identity can be:

application: an abstract entity as the global representation of your application for use across all tenants;
service principal: the local representation of a global application object in a specific tenants;
managed identity: a special type of service principal that eliminates the need for developers to manage credentials

In our use case for AKS workload, we also make use of Azure’s Managed Identity. This part is the same as the pod identity mechanism. However, here we create a federated identity credential for managed identity. The OIDC federated identity credential is issued by the AKS cluster. Within the AKS, the service account references the identity by client_id. Here is the documentation for the whole process.

One of the improvements in Entra workload identity for AKS, is the use of service account, which obviates the use of CRDs. Another improvement is the use of federated identity, whose lifecycle is tied to the cluster. This pattern is not only neater, but also standard. We map a service account to a managed identity with federated credential.

Summary

On managed Kubernetes services, we need an integration mechanism to grant Kubernetes workload access to cloud resources. We discussed what’s needed in this integration mechanism and looked at Azure Kubernetes as an example. In the next post, we’ll discuss how this issue is addressed in Elastic Kubernetes Service on the AWS side.

Authentication to kube-apiserver via OIDC

Fri, 28 Jul 2023 09:20:00 -0400

Background

There are many benefits of using OIDC to authenticate to kube-api server, especially with multiple clusters that need consistent log-in experience. With the last post on how OIDC Authorization Code Flow works, now I will discuss options with authentication for kubectl to connect to kube API server.

To start, let’s look at the anatomy of kubeconfig file. The full schema is in the documentation. Looking at my kubeconfig file, there are three sections:

clusters: each entry specifies a cluster’s name, server address and certificate authority data (in base64 encoding or a file location).
users: each entry specifies a username. Some users are identified with client key and certificate. Some specify a command to provide client authentication. Refer to the authentication strategies.
contexts: each entry links a user to a cluster

Therefore, the key to use OIDC integration, is to use command to provide client authentication.

Vanilla Kubernetes

The documentation on authenticating has a diagram on how to use OpenID Connect tokens. The diagram does not give details on how access_token and id_token were obtained. So it could be any OIDC flow (Authorization Code Flow, Implicit Flow, etc) as we have discussed.

Although the instruction does not mandate which OIDC flow to use, we should use Authorization Code Flow in this architecture. The API server needs to trust the OIDC issuer, and the document covers how to configure API server.

In the diagram, step 2 and step 3 are required by kubectl itself does not perform these activities. All kubectl does is carry the JWT token in the Authorization Bearer. Vanilla Kubernetes does not provide a solution for OIDC integration. It only provides some instructions and we still need some helper scripts to glue all these instruction steps together.

There are many open-source project for this purpose. For example, Jetstack has kube-oidc-proxy and Int128 developed kubelogin. Other projects such as k8s-auth-client, k8s-oidc-helper, and gangway are no longer being updated. The kubelogin project remains influential. It has a clear diagram too:

From the diagram we can see kubelogin proposes authorization code flow.

Also, one design concern with Kubernetes control plane is the placement of endpoint. From this diagram we can see that even if the cluster endpoint is on private subnet, OIDC integration should still work. The control plane (specifically kube-apiserver) initiates outbound connection to OIDC Provider. There is no inbound connection to it from the OIDC provider.

In the kubelogin setup, the redirect URI is set to localhost:8080 because it stands up a server on the same host where browser is running. The browser can always resolve localhost. For a full configuration steps, Okta has this blog post on how to use kubelogin as helper, and Okta as Authorization Server to authenticate kubectl via OIDC.

Step-by-step with kubelogin

We’ll go through an example with int128/kubelogin because it works with any Kubernetes flavour, including managed Kubernetes services. It is also fairly simple. The instruction covers a few types of Authorization Servers (Google Identity Platform, KeyCloak, Dex with GitHub, Okta and Ping Identity). I’ll take KinD cluster as an example and use Azure AD as Authorization Server.

First, we’ll register an App in Azure Portal. Go to “App Registrations” and “New registration”. Give it a name “kubeoidc” and set Redirect URI to “Web” with URL “localhost:8000”. Click on Register.

The next page shows the details for this app. The Application (client) ID is important for next steps. Click on “Add a certificate or secret”, then “New client secret”, put in expiry and description. The secret value is generated and displayed on the next page, which is important for our next step.

We also need to find out the issuer URL. From the App page above, click on “Endpoints” and find out the URL from field “OpenID Connect metadata document”. My metadata document URL looks like:

https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0/.well-known/openid-configuration

The OIDC Issuer URL is the part before .well-known. In this case, it is:

https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0

Now we have collected what we need for the next few steps: ClientID, ClientSecret and OIDC Issuer URL. We can then create the KinD cluster, and reference ClientID and IssuerURL in the cluster configuration:

cat << EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
 - role: control-plane
 kubeadmConfigPatches:
 - |
 kind: ClusterConfiguration
 apiServer:
 extraArgs:
 oidc-issuer-url: https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0
 oidc-client-id: 99999e88-e777-6666-c5c5-c444444d3d22
 - role: worker
 - role: worker
 - role: worker
EOF

kind create cluster --config=kind-config.yaml

The command will create cluster and configure kubeconfig file with a user named kind-kind as admin, which isn’t what we need. Now we use kubelogin helper command:

kubectl oidc-login setup \
 --oidc-issuer-url=ISSUER_URL \
 --oidc-client-id=YOUR_CLIENT_ID \
 --oidc-client-secret=YOUR_CLIENT_SECRET

This command will act as the client in the OIDC flow and prompts you to log in to Azure in a Browser. Once logged on, it gives you the next few commands to run. Since we’ve already created a cluster with the Issuer URL and Client ID, we can skip creating cluster, and run the steps to:

bind the Azure user to a ClusterRole
set up the kubeconfig locally with a user oidc, which needs to execute the oidc-login command

Lastly, we can test the oidc user with kubectl --user=oidc get nodes. We can also set the context to use oidc user by default. Voila.

From this example, we learned how to configure OIDC integration for any Kubernetes distros. The steps that we need to take are:

On the cluster side, we customize kube-api-server flag with OIDC provider info. Specifically, we provided Issuer URL and client ID in this example. In the OIDC Flow diagram, this step establish a trust from the Resource Server (K8s cluster) to the OIDC provider.
On the kubectl side, kubectl itself cannot fulfill all the duties of a client app in the OIDC Flow. It needs a helper script and we’ve made friend with int128/kubelogin.
On the Identity Store side, we expect it to be an OIDC-compliant Authorization Server. Otherwise, we consider using Dex as a broker in between.

When it comes to managed service, many allows us to customize the OIDC related flags for kube-api-server. Let’s look at how some managed services get this to work.

Azure Kubernetes Service

I find the kubectl authentication with AKS highly opinionated in its documentation. The recommendation is using Azure Active Directory as identity store but I don’t find it work with other OIDC providers.

To work with Azure Active Directory, you would configure the cluster and specify Azure role-based access control with the Group UUID in Azure AD. On the client side, you will need to install Azure’s kubelogin utility. Do not confuse it with int128’s kubelogin, which is for any cluster. This kubelogin is just for Azure. Once installed, you can use az-cli command to update your kubeconfig file, which call this utility from kubeconfig. To examine details about client configuration and AAD integration, check out the Terraform template in the azure directory of my cloudkube project.

The document only covers Azure AD integration and I tried to find if there’s a way to integrate with third-party OIDC providers. Unfortunately I have no luck. ChatGPT points me to a page about enabling OIDC provider but it is in the context of workload identity and it does not allow you to customize the issuer. So it’s completely irrelevant. Because you cannot customize OIDC issuer, etc, it simply won’t work with any third-party OIDC provider. Sure enough, most of Azure’s client use Azure AD anyways.

Azure makes it streamlined to configure OIDC integration of AKS with Azure AD, its own identity store. To my disappointment, it is currently not possible to integrate with third-party OIDC provider for authentication at cluster endpoint.

ROSA

RedHat OpenShift on AWS (ROSA) is a. However it reflects how OpenShift configures third-party identity provider.

Let’s first create a cluster off custom VPC and private endpoint. This is covered in my previous post about ROSA. After the cluster creation, we’re at the point where we can run oc command against cluster endpoint from Bastion host because it is a private cluster. However, being a private cluster is irrelevant to how we configure OIDC integration.

We’ll use Azure AD again as OIDC provider. So we need to register an app the same way as I did above in the kubelogin example. We need to have Client ID, Secret and OIDC issuer URL. For RedirectURI, go to your OpenShift console, and under Cluster, click on the cluster name → access control → identity providers → select OpenID. Note the page clearly states that this is Authorization Code Flow, and the OAuth Callback URL is provided. Use it to regiter App in Azure, and fill in the page with Client ID, Secret and OIDC issuer URL. Also fill in other fields accordingly and click on Add. Now you should have it configured!

Identity Providers for OpenShift cluster

The next step is trying to login. From Bastion host, run oc login and it will give me an URL to use. The URL contains the cluster endpoint, which resolvable from the Bastion Host itself. However I need a Browser session here, so I have to run Bastion host as SOCKS5 proxy and tell Chrome on my MacBook to use it:

~ open /Applications/Google\ Chrome.app --args --proxy-server="socks5://localhost:1080"

The browser session redirects to Azure AD for log in. Once completed the webpage will display a token that I can use with oc login command. Run this command with token from Bastion, I’m logged in:

oc login --token=sha256~3ZT5JGWELOcBzfohftEm9D2UwoOVFvATASuZk3_uxps --server=https://api.dhc.62q3.p1.openshiftapps.com:6443
oc whoami
oc get no

At this point, if I run oc whoami, I get the user name. However, this user cannot do anything. This is because it is not associated with a role yet. You grant more permission to this user: go back to OpenShift console, Clusters → ClusterName → Access Control → Cluster Roles and Access → Add user. Here you can map the user name to a role (let’s say ClusterAdmin). Then this user will have its priviledge:

The whole Flow works with private cluster, because the redirect URI is resolvable from the Bastion host. If you chose to expose cluster endpoint publicly (not recommended), you can perform the above steps directly from your MacBook or Laptop.

So the ROSA experience has been smooth. Unlike kubectl, the oc utility can act as the Client App in Authorization Code Flow. The other part of the configuration such as client secret and issuer URL are made in OpenShift console. Good job!

Elastic Kuberentes Service

EKS allows you to specify OIDC issuers from console or CLI to set up third-party OIDC configuration. There is a blog post from Okta on this, which works for private clusters. In the instruction, the author first manually created kubeconfig file with int128 kubelogin, and then bind ClusterRole with the user.

The blog post is very detailed. Instead of repeating it, I would like to discuss two SSO models available in AWS. I summarize them as below:

Both are SSO models for EKS. Model 1 (IAM Identity Center) is home grown as AWS using IAM Identity Center (formerly AWS SSO). Users start with an IAM principal (AWS construct) and use the aws-auth config map to tie them to Kubernetes groups. This AWS blog post and this support article are based on the IAM Identity Center model. On the other hand, Model 2 (Direct OIDC) is the vanilla Kubernetes approach. It takes group claim from OIDC identity token. The Okta blog post is based on this model.

The IAM identity center model works with SAML compliant identity providers, oftentimes Active Directory, although there seems to be a plan to support OIDC-compliant identity provider as well in the future. Even if it was supported today, I’d still prefer model 2 for working with any OIDC compliant identity provider because it’s simple. Why not leverage K8s’ native capability? For identity providers that do not support OIDC natively, or does not issue group claim (e.g. Google workspace), as we discussed, we can also consider alternatives such as Dex as identity broker. However, this model comes handy when a pipeline user with IAM role needs to authenticate into EKS.

One should always go for the direct OIDC model, if the upstream identity provider supports OIDC. The provider itself can even be AWS Cognito User pool. Cognito User Pool itself supports federated identity but again, I would directly connect EKS cluster to the OIDC compatible identity provider, instead of going through Cognito User Pool. As a result, the only use case where Cognito user pool is absolutely necessary, is when we need the Cognito user pool itself as the identity provider, as we have in CloudKube‘s eks directory.

Summary

Configuring OIDC provider for cluster endpoint can be confusing and we need to understand how OIDC flows work. I dived into OIDC in the previous post and in this post, I explained how to get it to work with vanilla Kubernetes. I summarized the three requirements and looked at the OIDC provider option with some managed services. Then I went through OIDC options for AKS, ROSA and EKS.

Connect kubectl to private Kubernetes cluster in EKS and AKS

Sat, 10 Jun 2023 19:31:00 -0400

Managed Kubernetes services give user a cluster endpoint and a number of worker nodes, with the choice. For each access, users have the choice of making them publicly available, or keeping them on private networking. In my opinion, any deployment beyond personal hobbies, should use Kubernetes private cluster, with both cluster endpoint and worker nodes on private subnet. There is no reason to expose computing nodes or Kubernetes management traffic publicly.

For worker nodes, it is fairly easy to put VMs on private network, but many companies still have the cluster endpoint exposed publicly. There are usually two reasons. First, their CI/CD agent is hosted somewhere else on the Internet (instead of on private network with private connectivity to Kubernetes cluster) and need to access Kubernetes cluster endpoint. Second, when the cluster needs to connect with third-party identity provider as OIDC provider, a two-way communication is needed.

There is a classic pattern of using a public bastion host (jump box), with a bastion host on the public subnet, routable to the private endpoint of managed Kubernetes service. Clients then connect to the bastion host via port 22 on a public IP address. The authentication is based on SSH key pair, or worse, password. The port forwarding (aka SSH tunnelling) capability enables all the magics.

Exposing a jump box in the public subnet with RSA key authentication is still not favourable. In this post, I’ll examine some secure patterns to connect to private endpoint with improved security posture.

AWS options

There are two problems. First, how to establish connectivity to the Bastion host in a private subnet. Second, how to use the Bastion host to proxy traffic to the cluster endpoint also in private subnet. To the first problem, there are two potential solutions: SSM Session Manager, and EC2 Instance Connect (EIC) with EIC endpoint (EICE).

SSM Session Manager was introduce in 2018. It runs an agent on the EC2, which initiates a connection to the SSM endpoint on the AWS side. This connection enables not only Session Manager, but also other Systems Managers (SSM) services such as Fleet Manager, Patch Manager and State Manager. The problem that session manager originally addresses is server management.

AWS launched EC2 Instance Connect (EIC) in 2019, and EIC Endpoint (EICE) in 2023. EIC addresses the problem with managing SSH key pairs at scale. It dynamically generates an SSH key pair for server access, based on IAM permission. However, it still requires an instance to have its SSH port publicly accessible. With EICE, it is no longer a requirement. In the diagram, EICE is placed in a private subnet, allowing EICE service to reach private instances at their SSH port.

Here is a comparison of the two:

	EC2 Instance Connect (EIC) with EIC Endpoint	SSM Session Manager
Location of Bastion host	Private Subnet.	Private Subnet
Need Ingress Port	Yes. Port 22 must open to the endpoint.	No. SSM agent initiate outbound connection from the instance
Traffic Path	AWS CLI → AWS EIC ES → EICE→EC2 Inst	AWS CLI → AWS SSM ES → SSM ← EC2 Inst
Authentication	AWS IAM and ephemeral SSH key when using AWS CLI directly AWS IAM and long-term SSH key when using SSH proxy command	AWS IAM and long-term SSH key when using AWS CLI directly or SSH proxy command
Work with OpenSSH	Yes	Yes
Cost	There is no additional cost for using EIC.	No additional cost, unless private SSM Endpoint.

Let’s take a look at each option.

EC2 Instance Connect

To use EIC, pick an AMI that has it pre-installed and ensure instance profile has correct policy, as the document states here. AWC CLI will make use of local OpenSSL client. So make sure there connection at port 22 is open. To make it work with EC2 instance on a private subnet, create an EC2 Instance Connect Endpoint on the VPC, and ensure that the security group of EC2 allows port 22 from the Endpoint. Run this command:

$ aws ec2-instance-connect ssh --instance-id i-00ea30a6e02db33fe

The command above simply generates a key pair internally, add the public key to the server side, and connect with SSH from the client side. The command takes you to an SSH session. Checking ps -ef | grep ssh on the client machine, you can see the full parameter of SSH, including the location of the ephemeral private key.

However, if you use AWS CLI open-tunnel as proxy command to ssh, then you’d still have to use the key pair used to create the EC2 instance. As suggested at the bottom of this blog post, the command is:

$ ssh ec2-user@[INSTANCE] -i [SSH-KEY] -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id %h'

This is a bummer, because with native SSH tool you do not get the primary benefit of EIC – ephemeral key pair.

SSM Session Manager

Now let’s look at SSM session manager. Similarly, it needs an agent installed and IAM role configured. You can connect to from web console but more importantly, from AWS CLI:

$ aws ssm start-session --target i-0531b19bec8ad022d

This command takes you to an SSH session with user ssm-user, without starting an OpenSSH client process locally. User do not have to manage key pair. There is also a document about using this command as proxy command, which uses an SSM document. I have one of the SSH config entry as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

This allows me to directly SSH to client using OpenSSL client (e.g. ssh i-0531b19bec8ad022d) by Instance ID. With this, I also need to specify my own OS user and matching private key.

I know I will use the OpenSSH client a lot from pipelines because it is very powerful. In both options, I have to live with managing key pairs myself. With SSM session manager’s proxy command, the instance does not need port 22 to open, which is a great advantage, in terms of security and operation. SSM Session Manager is a winner.

SOCKS5 proxy for kubectl

Either SSM Session Manager or EIC with EICE enables an SSH tunnel with key encryption between client (a local computer or a pipeline agent). On top of the SSH tunnel, we can build a SOCKS5 proxy. Kubernetes document has a good page on how to do this. I managed to get this to work with a few gotchas.

To put this in practice, I first created a VPC stack with a bastion host using terraform template from my vpc-base project. The terraform output will give the next set of commands to run to create a private cluster, using a manifest rendered from the file private-cluster.yaml.tmpl:

# cd aws_vpc
# terraform init
# terraform plan
# terraform apply
# ... run the given command ...
# envsubst < private-cluster.yaml.tmpl | tee | eksctl create cluster -f -

# Run this from a remote host without access to cluster endpoint.
# Run terraform apply and terraform output contains the variables needed for the next steps

# the command below may take 15 minutes to create a private cluster

eksctl create cluster -f private-cluster.yaml 

aws eks update-kubeconfig --name private-cluster

At this point, the kubeconfig file has been updated, but kubectl (from Internet or on-prem) is unable to connect to cluster endpoint (on private network). In order to

BASTION_SECURITY_GROUP_ID=$(terraform output -raw bastion_sg_id)
CLUSTER_SECURITY_GROUP_ID=$(aws eks describe-cluster --name private-cluster --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text)

# In Cluster Endpoint's security group, open up port 443 to Bastion host
aws ec2 authorize-security-group-ingress --group-id $CLUSTER_SECURITY_GROUP_ID --source-group $BASTION_SECURITY_GROUP_ID --protocol tcp --port 443

# Test with connecting to Bastion host with ssh i-0750643179667a5b6, assuming .ssh/config file is configured as above. From the bastion host, you can test:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api 
# if you get an HTTP response, even an error code 403, the bastion host has TCP connectivity to cluster endpoint

# then we can start an SSH session as a SOCKS5 proxy on the remote host
ssh -D 1080 -q -N i-0750643179667a5b6

# add > /dev/null 2>&1 & to push it to background, or use ctrl+z after running the command
# to validate that the SOCKS5 proxy is working, you can run the same curl command with a proxy parameter:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api --proxy socks5://localhost:1080

# you can instruct kubectl to use the SOCKS5 proxy with the following environment variable
export HTTPS_PROXY=socks5://localhost:1080

kubectl get node

# alternatively, add "proxy-url: socks5://localhost:1080" below server attribute in ~/.kube/config file.

There are some pitfalls to watch for. On the remote host both ssh command and kubectl command implicitly uses AWS CLI. Therefore, make sure the profile and IAM role are correctly configured. For example, if SSM agent requires one IAM role, and kubectl is created with another IAM role, then make sure AWS CLI assumes the correct IAM role using environment variables, and use “aws sts get-caller-identity” to validate the IAM identity being used.

What about AKS in Azure

I touched on this in my Azure notes in 2021 and did a research again. Unfortunately, options are still fairly limited.

The first option is to use a managed service called “Azure Bastion”, which requires public IP and a dedicated subnet with the exact name of AzureBastionSubnet, as well as some additional requirement. I’m not impressed with these requirement because it is meant to be a managed service. The other option, is essentially to DIY a JumpBox. The idea is the same: put the jumpbox in a public subnet, which is routable to private subnets. When you need to connect to private VMs, get to the jumpbox first.

Apart from having to put the bastion VM on a public subnet, the pattern that we discussed above involving SOCKS5 proxy still works. Exposing a bastion host isn’t ideal but it still reduces attack surface significantly, comparing to exposing the cluster endpoints of all Kubernetes API servers.

Summary

Many immature Kubernetes configurations exposes private endpoint publicly. Having cluster endpoint in private subnet greatly improves security posture. In my opinion, there are very few situations where cluster endpoint must exposed publicly. Having private endpoint should be mandatory for all Kubernetes cluster. In the next post, I also cover how to create a ROSA cluster with private endpoint.

Kubernetes Storage on Azure 2 of 3 – Portworx

Fri, 12 Aug 2022 15:23:00 -0400

In the previous post, we have discussed built-in storage classes on Azure Kubernetes. Further to that, we will examine some third-party software defined storage (SDS) options that are compatible with Azure Kubernetes Service in this post. Then we take Portworx on Azure as an example. Although, these options are specific to Azure, most of the players also have solutions for other managed Kubernetes platforms. Also, the methodology to study storage options remain the same regardless of cloud service provider.

In fact, I touched on software defined storage (SDS) in the context of general non-containerized workload in a separate post. In short, storage providers decouple the SDS appliance from the full storage solution in order to lower the cost and increase flexibility. To the storage consumer (e.g. a process running on Linux OS), SDS can present a block disk or file system. There are also SDS solution that can host your own object storage and we will discuss that later. In fact, SDS has gained significant popularity in recent years. For example, the report “Validating Software-Defined Storage Operating Models for the Enterprise” by archiectingit divided the evolution into four phases and cited that Gartner predicts the SDS revolution to reach 50% of the storage market by 2024, from 15% in 2020.

This post discusses SDS in the context of container storage. Then we will install Portworx on Azure Kubernetes.

SDS for Kubernetes

Many SDS appliances also developed the capability to present storage volumes to containerized workload. I put together a list of SDS products that works on Azure with their supported access modes:

Solution	Licence	Development and Support	Access Mode
Portworx	Closed source. Free Essential tier. Enterprise features on License.	Commercially supported by PureStorage	RWO, RWX
Ceph by Rook	Open source. Rook is a graduated CNCF project.	Developed and commercially supported by Red Hat, Canonical and SoftIron	RWO, ROX, RWX
OpenEBS	Open source. Sandbox CNCF project.	Developed and commercially supported by MayaData et al.	RWO
Longhorn	Open source. Incubating CNCF project.	Originally developed by Rancher, and commercially supported by SUSE	RWO
StorageOS	Closed source. License required.	Commercial support by Ondat.	RWO
LINBIT	Open-source with enterprise plans	Enterprise support

RWO is the most commonly supported mode. The report “Performance Benchmarking Cloud Native Storage Solutions for Kubernetes” makes a comparison of performance among some of the options in early 2021. Another potentially opinionated comparison list is by LINBIT.

Portworx is a leading player with commercial SDS solution and I will test its free Essential tier in the rest of this post. Ceph is one of the most mature leading open-source offering and I will test it in the next post.

For Portworx, we can use the terraform template cloudkube for Azure. The template assigns the kubelet’s managed identity as contributor of the node resource group. The template also creates a bastion host with direct SSH access to the nodes.

Portworx Operator on Azure

We use Portworx Operator to configure storage cluster. Portworx has an instruction for AKS but it is not tailored to specific identity model. For simplicity, use my cloudkube Terraform template to create the AKS cluster, and skip the “Prepare Your AKS Platform” page. Instead, follow the “Deploy Portworx using Azure managed identity on new AKS cluster” page starting at step 7.

At that step, we need to create a secret with the client ID of the managed identity for node agent. The terraform template outputs the BYO identity’s client ID. After cluster creation, we simply SSH to the bastion host and create the secret using the output.

To install Porworx using operator, we can follow a wizard in PX-central. If this is the first time, we need to create an account and log in to the portal. If this is not the first time and you have previously created a cluster, you need to detach that cluster by going to Profile from bottom left corner on the portal page.

Follow the guide in “Install Portworx on AKS using the Operator“. In the wizard, click on “Portworx Essentials” for free tier, or “Portworx Enterprise” for the 30-day trial. Then select operator with latest version. In the rest of the wizard steps, select options applicable to Azure environment. The last step will present two kubectl commands to install operator and install the CR. Run the command to install operator and verify result:

kubectl apply -f 'https://install.portworx.com/2.9?comp=pxoperator'
kubectl -n kube-system get deployment portworx-operator

Portworx Custom Resource

To install the CR, we need to customize the given manifest in order to use our managed identity. We can download the YAML manifest (portworx_essentials.yml) and modify it in text editor. As the page “Deploy Portworx using Azure managed identity on new AKS cluster” suggest at step 9: in the env section, remove the AZURE_CLIENT_SECRET and AZURE_TENANT_ID sections but keep the AZURE_CLIENT_ID section. My CRD declaration looks like this:

# SOURCE: https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=myuserid&b=true&kd=type%3DPremium_LRS%2Csize%3D150&s=%22type%3DPremium_LRS%2Csize%3D150%22&c=my-very-long-px-cluster-id&aks=true&stork=true&csi=true&mon=true&tel=false&st=k8s&promop=true
kind: StorageCluster
apiVersion: core.libopenstorage.org/v1
metadata:
 name: my-very-long-px-cluster-id
 namespace: kube-system
 annotations:
 portworx.io/install-source: "https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=myuserid&b=true&kd=type%3DPremium_LRS%2Csize%3D150&s=%22type%3DPremium_LRS%2Csize%3D150%22&c=my-very-long-px-cluster-id&aks=true&stork=true&csi=true&mon=true&tel=false&st=k8s&promop=true"
 portworx.io/is-aks: "true"
 portworx.io/misc-args: "--oem esse"
spec:
 image: portworx/oci-monitor:2.10.2
 imagePullPolicy: Always
 kvdb:
 internal: true
 cloudStorage:
 deviceSpecs:
 - type=Premium_LRS,size=150
 kvdbDeviceSpec: type=Premium_LRS,size=150
 secretsProvider: k8s
 stork:
 enabled: true
 args:
 webhook-controller: "true"
 autopilot:
 enabled: true
 monitoring:
 prometheus:
 enabled: true
 exportMetrics: true
 featureGates:
 CSI: "true"
 env:
 - name: AZURE_CLIENT_ID
 valueFrom:
 secretKeyRef:
 name: px-azure
 key: AZURE_CLIENT_ID
---
apiVersion: v1
kind: Secret
metadata:
 name: px-essential
 namespace: kube-system
data:
 px-essen-user-id: aaaabbbbccccddddmyverylongpxessenuserid
 px-osb-endpoint: ssssssyyyyyyyzzzzzzmyverylongpxosbendpoint

In this custom resource manifest, I specify to get the cluster to create disk from Azure, even though it has the capability of using an existing disk. Then apply the CRD manifest:

 kubectl apply -f portworx_essentials.yml

This will take up to 10 minutes to create several related resources. There are several check points to ensure the cluster is created successfully. Here are some useful commands:

kubectl -n kube-system get po # all Pods related to portworx should be ready and running, especially the portworx-api ones
kubectl -n kube-system get storagecluster # the status should report online
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') 
# get the name of one of the portworx Pod for storage cluster
kubectl -n kube-system exec -c portworx -it $PX_POD --tty --stdin -- /opt/pwx/bin/pxctl status 
# should report "Status: PX is operational" with valid license loaded
kubectl get sc # portworx related storage classes are available

The pre-built storage classes are not CSI based. However, CSI should be automatically enabled in recent operator versions. We should create our own CSI storage classes and PVCs using our own storage classes. Here are two examples:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: px-csi-database
provisioner: pxd.portworx.com
parameters:
 repl: "2"
 priority_io: "high"
 io_profile: "db"
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: px-csi-artifact
provisioner: pxd.portworx.com
parameters:
 repl: "2"
 priority_io: "medium"
 io_profile: "sequential"

That is a bare minimum Portworx installation. Check out their documentation for the full features. With this minimum install we can go to the section “Validate Persistent Storage” from the previous post to validate the persistent volume.

For troubleshooting purpose, pxctl is the utility and it is available on Portworx Pods.

Performance

We care not only the functionality, but also the performance. So I ran a quick performance test using kube-str, using all default configuration. The result is as follows:

	read_iops	write_iops	read_bw	write_bw
px-csi-database	IOPS=969.614136 BW(KiB/s)=3894	IOPS=729.698059 BW(KiB/s)=2935	IOPS=1172.772827 BW(KiB/s)=150639	IOPS=691.626526 BW(KiB/s)=89053
px-csi-artifact	IOPS=780.681946 BW(KiB/s)=3139	IOPS=682.522766 BW(KiB/s)=2746	IOPS=773.548584 BW(KiB/s)=99549	IOPS=659.015320 BW(KiB/s)=84890

Note that those numbers reflect performance under default configuration, and they should not be considered as the best performance that Portworx can deliver on Azure Kubernetes. Before moving to production, it is important to establish your own test parameters that best represents the container workload, and then iterate through different parameters for the storage class based on the requirement and performance output.

Kubernetes Storage on Azure 1 of 3 – built-in storage and NFS

Sun, 31 Jul 2022 15:22:00 -0400

In the previous post, we understand that to host stateful workload, we need to manage persistent storage to the Kubernetes platform. In this post, I will explore the different storage options. These options are specific to Azure Kubernetes service. However, the principals apply to any Kubernetes platform regardless of cloud vendor.

In another old post, I discussed in-tree and CSI storage classes, and from a developer’s perspective, how to mount volumes statically and dynamically once the storage class is available. Here in this post we are concerned with how to make storage classes available, from a platform specialist perspective.

Storage in Azure Kubernetes

As soon as the Azure Kubernetes cluster is launched, a number of built-in storage classes are available. Unlike third-party storage classes, they do not require kubelet identity to be contributor for node resource group. As discussed we shall use CSI based storage classes.

managed-csi
managed-csi-premium
azurefile-csi
azurefile-csi-premium

The main difference between them is the backing technology. However, form Kubernetes workload’s perspective, the Pods as storage consumers are concerned with the access mode instead of backing technology. Here are supported access modes:

ReadWriteOnce: read-write by a single node
ReadOnlyMany: read only by many nodes
ReadWriteMany: read-write by many nodes
ReadWriteOncePod: new in Kubernetes 1.22 to restrict volume access to a single Pod

The storage classes managed-csi and managed-csi-premium support ReadWriteOnce. The storage classes azurefile-csi and azurefile-csi-premium support ReadWriteMany.

Apart from these built-in options, Azure also suggests a few more options based on other types of Azure resources. For example, AKS can integrate with HPC cache and it for HPC. We can also self-manage a virtual machine configured as NFS server, and use the NFS subdir external provisioner to configure storage class. Despite of the overhead with managing a VM, you have more configurability. My previous client reports that they gain better performance than the built-in options.

Another alternative is Azure NetApp Files. However, being a full enterprise grade solution (similar to FSx ONTAP), Azure NetApp Files costs an arm and a leg. In this comparison, it cost 60 to 100 times as the cost by built-in options.

Another option is Azure Ultra Disk, which needs to be enabled at cluster level. You can provision performance target (DiskIOPSReadWrite and DiskMBpsReadWrite) in the storage class. Ultra Disk is a good middle ground between the pricey NetApp files and the less performant built-in options.

Terraform Template for AKS

To explore the storage options, I use my own terraform template to create an AKS cluster. The template is in the azure directory of the cloudkube repo. The template consists a few configurations with Azure Kubernetes to facilitate storage configuration. First, it configures an SSH key pair to use between the bastion host and the node. Users can SSH to Kubernetes nodes from bastion host as soon as terraform apply is completed. Second, the third party storage options installed after the cluster creation need their Pod to instruct Azure to create Azure disks. This requires that a Kubernetes node agent have the permission to provision resources in the node resource group. This is important to understand because there are a couple of managed identities at play (refer to this post) when building an AKS cluster.

In Azure Kubernetes, it is the managed identity of kubelet, that needs to have contributor permission over the resource group for the nodes (not the one for the AKS cluster itself). A managed identity is expressed by a client ID, an object ID (aka principal ID), and the identity ID. We can find them out with an AZ CLI command as below:

We can also tell that the kubelet managed identity represents node agent, by connecting to a node and looking at the argument (kubernetes.azure.com/kubelet-identity-client-id) of kubelet process:

In the template, I also chose to designate the same BYO identity for both the cluster and for kubelet (node agent), in order to minimize my requirement on permission. If I had left it with a system assigned identity for node agent, I would have to assign that identity as a contributor for the node resource group, either as a user, or via Terraform’s identity. Either way, it is beyond what a Contributor is allowed to do.

Benchmarking with kubestr

I used fio utility for storage benchmarking from virtual machines. However, fio utility is not for container. For fio testing on Kubernetes, I’d have to use a Docker image, and test with target volume attached. Fortunately, the Kasten team shared their initiative in the open source project Kubestr. The kubestr release is available as an executable on common platforms. It connects to the cluster the same way as kubectl and here is a demo. To begin with, download the utility to bastion host, and run it without any argument, which prints the storage classes and volume snapshot classes:

curl -L -o kubestr.tar.gz https://github.com/kastenhq/kubestr/releases/download/v0.4.31/kubestr_0.4.31_Linux_amd64.tar.gz
tar -xvf kubestr.tar.gz && rm kubestr.tar.gz && chmod +x kubestr

./kubestr # if kubectl is configured, this command will print out the details of storage classes and volume snapshot classes

In addition to outputting details, it is also very simple to perform storage benchmarking with kubestr. All we need to do is giving it the storage class name and it will run four tests by default with common global options (ioengine=libaio verify=0 direct=1 gtod_reduce=1). The four tests are:

JobName	block_size	filesize	iodepth	rw
read_iops	4k	2G	64	randread
write_iops	4k	2G	64	randwrite
read_bw	128k	2G	64	randread
write_bw	128k	2G	64	randwrite

During each test, it measures and reports IOPS and bandwidth (throughput). If your I/O profile falls out of the four jobs, you can even customize your test by supplying a fio config file. For example, you need a longer test duration, or you need a larger total size for the test. Before the test, kubestr automatically mount their test volumes using the storage class being tested.

./kubestr fio -s my-storage-class # benchmarking a storage class

For read_iops and write_iops, we mainly look at the IOPS. For read_bw and write_bw, we mainly look at the bandwidth. The iops and bw based on samples are reported as first line of result, followed by min, max and average.

Metrics

With kubestr I ran a performance test amongst the native storage classes with Azure Kubernetes Service, with results as below:

	read_iops	write_iops	read_bw	write_bw
managed-csi	IOPS=314.729797 BW(KiB/s)=1275	IOPS=297.071136 BW(KiB/s)=1204	IOPS=315.311188 BW(KiB/s)=40887	IOPS=261.048645 BW(KiB/s)=33941
managed-csi-premium	IOPS=493.395844 BW(KiB/s)=1990	IOPS=426.731812 BW(KiB/s)=1723	IOPS=455.950348 BW(KiB/s)=58894	IOPS=422.888855 BW(KiB/s)=54662
azurefile-csi	IOPS=259.333282 BW(KiB/s)=1053	IOPS=283.985779 BW(KiB/s)=1152	IOPS=240.447403 BW(KiB/s)=31298	IOPS=230.689804 BW(KiB/s)=30048
azurefile-csi-premium	IOPS=394.044739 BW(KiB/s)=1586	IOPS=371.181793 BW(KiB/s)=1494	IOPS=380.360535 BW(KiB/s)=49018	IOPS=491.313446 BW(KiB/s)=63310

These numbers are based on default test parameters (e.g. 100Gi PVC size). The numbers indicate that block storage generally performs better in default setting. So we should use managed disk instead of azure file unless multiple pods needs to access the same volume.

Validate Persistent Storage

We can use mysql as a quick and dirty test of storage classes. We can deploy the following manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: mysql-pvc
spec:
 storageClassName: managed-csi-premium # the storage class being tested
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 5Gi
---
apiVersion: v1
kind: Secret
metadata:
 name: mysql
data:
 password: eHl6  # base64 code of xyz
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mysql
 labels:
 app: mysql
spec:
 replicas: 1
 selector:
 matchLabels:
 app: mysql
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - image: mysql:5.6
 name: mysql
 env:
 - name: MYSQL_ROOT_PASSWORD
 valueFrom:
 secretKeyRef:
 name: mysql
 key: password
 ports:
 - containerPort: 3306
 name: mysql
 volumeMounts:
 - name: mysql-persistent-storage
 mountPath: /var/lib/mysql
 volumes:
 - name: mysql-persistent-storage
 persistentVolumeClaim:
 claimName: mysql-pvc
---
apiVersion: v1
kind: Service
metadata:
 name: mysql-service
 labels:
 service: mysql
spec:
 selector:
 app: mysql
 ports:
 - name: tcp-mysql
 protocol: TCP
 port: 3306
 targetPort: 3306

Once the Pod has been created, then we use a throw-away Pod to connect to mysql service and build some data:

$ kubectl run mysql-cli --rm -i --tty --image imega/mysql-client -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz
Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 9 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]>

From the mysql shell, we create a test database with dummy data populated:

CREATE DATABASE `testdb`; 
USE testdb; 
CREATE TABLE IF NOT EXISTS tasks ( 
 task_id INT AUTO_INCREMENT PRIMARY KEY, 
 title VARCHAR(255) NOT NULL, 
 description TEXT, 
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP 
) ENGINE=INNODB; 
INSERT INTO tasks (title, description) VALUES ('Job A','Morning Standup');
INSERT INTO tasks (title, description) VALUES ('Job B','Latte with two shots of espresso');
INSERT INTO tasks (title, description) VALUES ('Job C','Coding coding and coding');
INSERT INTO tasks (title, description) VALUES ('Job D','git commit');

Each SQL command should return with number of rows affected and then we can exit the MySQL shell and Pod shell. Once we exit out of the Pod shell, the Pod is deleted. We can re-connect to validate the data are still present:

$ kubectl run mysql-tester --rm -i --tty --image imega/mysql-client -- mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz --database testdb --execute='SELECT * FROM tasks;'
If you don't see a command prompt, try pressing enter.
Got error: Access denied for user 'root'@'147.206.3.15' (using password: NO)
+---------+-------+----------------------------------+---------------------+
| task_id | title | description | created_at |
+---------+-------+----------------------------------+---------------------+
| 1 | Job A | Morning Standup | 2022-06-22 20:29:12 |
| 2 | Job B | Latte with two shots of espresso | 2022-06-22 20:29:12 |
| 3 | Job C | Coding coding and coding | 2022-06-22 20:29:12 |
| 4 | Job D | git commit | 2022-06-22 20:29:13 |
+---------+-------+----------------------------------+---------------------+
Session ended, resume using 'kubectl attach mysql-tester -c mysql-tester -i -t' command when the pod is running
pod "mysql-tester" deleted

The output validates the persistent storage of the data. When testing a different storage class, simply start over with a different storage class specified in the PVC part of the manifest.

Summary

This post focuses on storage options for persistent volumes on Azure Kubernetes service. I use my own Terraform template with some custom configuration. I also covered kubestr as benchmarking tool and ran it against the built-in storage classes.

In some situations, the built-in options do not suit your needs. For example, you might want your application to use persistent volume in a consistent way across multiple cloud vendors. We will have to resort to third-party software defined storage layer. In the next post, I will explore a couple of SDS-based options, namely Portworx and Ceph. The custom configurations in the Terraform template will be helpful when we configure Portworx.

From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms

Wed, 29 Dec 2021 22:50:16 -0400

Update: also read my other article here on the different generations of ingress technologies.

In my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.

The word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article.

Background

In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node’s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node’s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.

Kubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.

Requirement

Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:

The ability to proxy TCP traffic over an arbitrary TCP port
The ability to terminate TLS/SSL encryption for TCP traffic

The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I’m not confident.

I also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:

Product	Requirement #1	Requirement #2
F5 driven Nginx Ingress Controller	Supported	There have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.
HA Proxy Ingress controller	Supported	There’s no mention of this in documentation.
Kong’s Kubernetes Controller (with TCPIngress CRD)	Supported	Documentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.
Traefik Lab’s IngressRoute (with IngressRouteTCP CRD)	Supported	Documentation seems to suggest that it is supported by IngressRoute.

My general impression is that Req #2 isn’t very popular so the providers either don’t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP’s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.

Introduction to Gateway

In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:

This sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to “Route” resource type.

This conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The diagram above stolen from Gateway API website illustrate the management model for each resource.

Kubernetes Gateway Implementations

The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.

Because of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification.

For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API’s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong’s Kubernetes Ingress Controller follows the same path.

The Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.

Until the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.

Istio Gateway implementation

As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.

First, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I’ve compared three major service mesh technologies as below:

Linkderd	Istio	Consul
– lightweight – uses linkerd2-proxy – the original service mesh – no Gateway implementation (deal breaker in my use case)	– feature rich – uses Envoy proxy – complex but getting better	– initially a service discovery and distributed key-value store – lacks observability features but getting better – uses Envoy proxy

It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in.

Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.

Istio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).

In addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:

Here is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called “Ingress”, nor a resource type named “Ingress Gateway”, although they may be loosely used to refer to “Gateway resources configured to manage ingress traffic”.

Istio Gateway Installation

There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm.

With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:

install CRDs using the base chart
install istiod using the istiod chart. Here is an example of values provided to Helm installer.
install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition.

This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.

AKS Lessons Learned 2 of 2

Sat, 18 Dec 2021 01:18:00 -0400

Even though Azure Kubernetes Service (AKS) is a managed service, building a cluster is not trivial. For help resources, I would start with the webinar “Configure Your AKS cluster with Confidence” from April 2021, which focuses on a set of working best practices (convention over configuration) but obviously not every recommendation suits every use case. For a deeper technical tour, the John Savill’s Technical Training channel has good videos (from 2020) on AKS overview, high availability and networking. Lastly, there is also an AKS checklist to remind you of the implementation details to consider.

All the references aside, I need to write down some gotchas from my implementation experience in the last two month.

Identity and Access Management

AKS is a special type of Azure resource in the sense that it manages other Azure services on user’s behalf. Therefore the access management needs to consider several aspects:

Access Type	Mechanisms involved	Example
User access Kubernetes API	Azure AD, Azure RBAC and Kubernetes RBAC. – Azure AD is for authentication – Azure RBAC for Kubernetes – Kubernetes RBAC	A user connects to Kube-API server using kubectl
AKS access other Azure resource	There are several identities that represents different components of AKS. For example, the AKS cluster, the node agent pool, and each add-on. The AKS cluster can be represented as a service principal, or managed identity (system assigned or user assigned). The node agent pool can be represented as a managed identity	AKS cluster connects to a VNet in a different resource group. (requiring cluster’s identity to have network contributor role on the target network resource group) AKS node agent pulls images from ACR (requiring the node agent pool’s identity to have ArcPull role on the target ACR)
Pod access other Azure resource	AAD-Pod Managed Identity	Business workload connects to managed database service such as PostgreSQL on Azure.
Pod access Kubernetes API	Access Kubernetes API using Service Account. This issue is solved completely by Kubernetes native mechanisms. Roles and ClusterRoles define permissions. RoleBindings and ClusterRoleBindings associates Service Accounts with permissions	Workload access ConfigMap, Secret etc.

In the first access type, for RBAC with user to access Kubernetes API, there is an overlap between Azure RBAC and Kubnernetes RBAC. Azure RBAC has four built-in roles and three of them (reader, writer, admin) are namespaced. When you use Azure CLI to manage to assign one of those roles, the rolebinding and cluster rolebinding record stored in etcd will be updated accordingly.

RBAC mechansim	Use case
Azure RBAC for Kubernetes	Manage RBAC programmatically using Azure CLI, or infrastructure as code
Kubernetes RBAC	Manage RBAC declaratively with more granularity for all types of Kubernetes resources including CRD

For ease of operation it is advised to use Kubernetes RBAC whenever possible. Azure RBAC is still used for RBAC at the level of Azure resource but not at the level of Kubernetes resource.

In the second access type, AKS cluster may use managed identity or service principal. Azure’s recommendation is managed identity over service principal. Managed Identity is a wrapper around Service Principal with less overhead. Managed Identity can be system assigned (created at the time of cluster creation), or user assigned (can be created ahead of time by Azure administrator and imported to the cluster’s context).

The second access type can be further broken down because there are several components in AKS that uses their own identities. I list the related managed identities as below:

Name	Purpose	BYO identity with Terraform
cluster identity	This identity represents the cluster	Specify in identity block in kubernetes_cluster resource
agent pool identity	This identity represents kubelet running in the agent pool	Specify in kubelet_identity block.
addon: azurepolicy	This identity represents azure policy addon to access the policy	N/A
addon: omsagent	This identity represents OMS agent to access monitoring etc	Specify in oms_agent_identity block
addon: secret	This identity represents to the secret addon, to access AKV	Specify in secret_identity block
addon: ingress gateway	This identity represents the ingress application gateway	ingress_application_gateway_identity block

By default, the system creates a new managed identity for each of the required identity above. For simplicity with identity management, we may create a managed identity and use it for all the occasions where an identity is needed and user assigned (BYO) identity is supported.

In the “az aks show” command return (a JSON document), the identity section (root level) reports the cluster identity, the identityProfile section (root level) reports the agent pool (kubelet) identity. Other identities such as omsagent, are reported in their own child document.

Node Networking

In Azure, a subnet can span across multiple availability zones. Therefore an AKS cluster can put its nodes on a single subnet with nodes evenly distributed across three AZs for high availability. The AZ of each node is indicated in the node label, and can be displayed with kubectl command.

Within a single AZ, a good practice to minimize latency between nodes is to place the nodes in a proximity placement group (PPG). However, only a single PPG can be associated with a node group. You can’t have three PPGs, one in each AZ, for a single subnet.

Pod Networking

The default Pod networking model is kubenet, which involves overlay network. Pod-to-Pod traffic across nodes requires Network Address Translation (NAT). To overcome this performance tax, Azure introduces Azure CNI which gives each Pod an routable IP address from the VNet’s CIDR. This requires advanced IP planning to prevent IP exhaustion. A risk introduced in Azure CNI is that all Pods are exposed on the V-net, which needs to be protected by Network Security Group and/or outbound firewall.

DNS

On the DNS side, when AKS cluster integrate with an external node network, it may create weird issues that are hard to troubleshoot. Another example is with DNS. If the V-Net uses an external DNS server (which is common for enterprises with hybrid network to use an on-premise DNS server), then the cluster creation failed with time-out with misleading error messages (for example, this comment). This is because the DNS name of the newly created cluster is not resolvable within the V-NET, which points to the on-prem DNS server. The fix to that is:

Use a BYO DNS zone (in Azure) for AKS cluster creation;
The AKS cluster will publish the A-record to the zone. To allow this to happen, the AKS cluster’s managed identity needs to have DNS contributor permission for the zone;
Configure the on-prem DNS for conditional forwarding to the DNS zone

This fix will allow AKS to resolve its name and therefore confirm its own creation. Here is a good blog about the DNS zone options.

Another potential issue introduced with the use of on-prem DNS server, is the resolution of single-label hostname of the nodes. This is not just an issue in the context of AKS. It is a generic issue with VMs running on a V-Net pointing to on-prem DNS, as explained in detail here.

In this situation, we should use the fully qualified hostname instead of single-label hostname. The fully qualified hostname with DNS suffix can help the on-prem server to configure conditional forwarding. For example, when the DNS suffix is *.internal.cloudapp.net, then forward it to Azure’s virtual internal DNS server 168.63.129.16 which can resolve the hostname.

If only the Pods need to resolve those FQDNs, then we can configure Core-DNS with conditional forwarding, which will take effect only at the cluster level without the need for changing the on-prem DNS. The Core-DNS configuration looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
 name: coredns-custom # this is the name of the configmap you can overwrite with your changes
 namespace: kube-system
data:
 cloudapp.override: | # you may select any name here, but it must end with the .override file extension
 log
 rewrite continue {
 name regex ^(.*[0-9]{7}-vmss[0-9]{6})$ {1}.internal.cloudapp.net
 answer name ^(.*)\.internal\.cloudapp\.net$ {1}
 }
 forward internal.cloudapp.net 168.63.129.16
 cloudapp.server: |
 internal.cloudapp.net:53 {
 errors
 log
 cache 10
 forward . 168.63.129.16
 }

Alternatively, use Pod DNS policy so that the Pod can use an external DNS server.

Initial Service Account

When a cluster is created, an Azure AD user or group can be assigned as cluster administrator. For a CI/CD pipeline to interact with the newly created cluster, a service account in Kubernetes is needed. Suppose we use Terraform to create the AKS cluster, we can create such service account automatically with the Kubernetes provider. This requires that the Terraform execution environment to have network access to the cluster. If the AKS cluster is located in a private network, then the agent where Terraform CLI runs should also be on the network. Alternatively, use Terraform Enterprise hosted in an environment with access to the cluster’s network.

Integration with Azure KeyVault

Azure Key Vault can store several types of secrets, key value pair, X509 keys and certificate. When AKV is integrated with an AKS cluster, the Kubernetes workload can access the secrets as mounted volumes, using CRD named SecretProviderClass. Further, they can be presented as Kubernetes Secret, using a Pod to sync between mounted content and Secret. AKV has three types of entries: key, certificate and secret (key-value). The certificate entry requires both key and certificate are stored, with optional certificate chain. In my opinion this is an over design. Unless we need Azure to manage the certificate (e.g. rotation) I would simply use secret to store my own X509 key and certificate.

AKS Lessons Learned 1 of 2

Sat, 04 Dec 2021 02:11:06 -0400

In general, troubleshooting Kubernetes is tricky. That is because one has to get in and out of pods. I took two days to troubleshoot some networking issues with private AKS cluster. For the amount of tricks I had to employ, I need to take some notes.

The issue

After writing the Terraform code, I used the following dummy service to test the private AKS cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: aks-helloworld-one
spec:
 replicas: 1
 selector:
 matchLabels:
 app: aks-helloworld-one
 template:
 metadata:
 labels:
 app: aks-helloworld-one
 spec:
 containers:
 - name: aks-helloworld-one
 image: neilpeterson/aks-helloworld:v1
 ports:
 - containerPort: 80
 env:
 - name: TITLE
 value: "Welcome to Azure Kubernetes Service (AKS)"
---
apiVersion: v1
kind: Service
metadata:
 name: aks-helloworld-one
spec:
 type: LoadBalancer
 ports:
 - port: 80
 selector:
 app: aks-helloworld-one

The expected behaviour, is that the service object will tell cloud API to provision a load balancer, with public IP listing at port 80. I should be able to curl to the IP address and connect to the site in the Pod. However, I was not able to. On the bastion host, I was able to curl to nodePort of the node address. But anything on public IP does not work, no matter where I ran curl from. This feels like a basic issue, but is quite annoying because the native troubleshooting tool for Azure Load Balancer is horrible. In and out of a bunch of components named “insights”, “diagnostic log”, or “Metrics”, I can’t simply find a way to trace whether it received an HTTP request. Most of the information I was able to see was irrelevant or useless.

The approach

The hard way to troubleshooting infrastructure as code, is configuration comparison approach: revert to a baseline configuration, and see if the expected function works. Then from the baseline, change one configuration at a time and see where it starts to break. This approach is very time consuming, and AKS cluster as a relatively large resource, with numerous attributes, takes this effort to extreme.

The baseline configuration I started with is:

az aks create -g AutomationTest -n orthCluster --generate-ssh-keys --node-count 1 --tags Owner=MyOwner

With this baseline, I simply use kubectl to apply the YAML file above. Then I can tell that the port is working. With a good start point, I started to apply one change at a time and repeat the test. I ran into a snug when I’m using the following configuration:

az aks create -g AutomationTest -n orthCluster --generate-ssh-keys --node-count 1 --tags Owner=MyOwner --enable-private-cluster --network-plugin azure --network-policy calico

With the cluster created from the command above, the variable introduced is –enable-private-cluster. This puts the cluster on a private network. I cannot connect to the cluster via a public endpoint anymore, and thus have to figure out some tricks to run the kubectl commands. I had to play with the Command Run feature of AKS cluster because I don’t have a bastion host when using AZ CLI command. The Command Run feature would not allow me to use any file from bastion host. So i had to create my test objects, the Deploy and the Service objects all by imperative commands. The equivalent commands I worked out is:

kubectl create deployment aks-helloworld-one --image=neilpeterson/aks-helloworld:v1 --replicas=1 --port=80
kubectl expose deploy aks-helloworld-one --port 80 --target-port 80 --type='LoadBalancer'

Then I realized a limitation with Command Run feature: it only supports basic command switches and doesn’t like switches such as –replicas. So I used the following commands:

az aks command invoke -g AutomationTest -n orthCluster -c "kubectl get no"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl create deployment aks-helloworld-one --image=neilpeterson/aks-helloworld:v1"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl get deploy"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl expose deploy aks-helloworld-one --port 80 --target-port 80 --type=LoadBalancer"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl get svc"

This trick allows me to continue with the testing eliminate Azure CNI and Calico policy as the cause. Testing after each cluster creation is painful because the cluster creation can take 10 minutes.I had to temporarily minimize the size of the cluster to speed up provisioning.

I finally came to the point that I can reproduce the issue using TF template. I realized that when I set the vnet_subnet_id attribute of azurerm_kubernetes_cluster’s default_node_pool, the problem came back. That’s the smoking gun that the node subnet is the issue.

The Network Security Group on Node Subnet

The node subnet has an associated network security group. I discovered that once I add an allow rule for port 80 to the security group, the curl test will work. I also noticed the security group rule change will take 60 sec to come to effect and load balancer will also take 60 sec to warm up.

This confuses me because port 80 is only listened by the load balancer and not by any of the nodes. It’s most likely when public load balancer is used the load balancer is placed on the node subnet. According to this note:

Inbound, external traffic flows from the load balancer to the virtual network for your AKS cluster. The virtual network has a Network Security Group (NSG) which allows all inbound traffic from the load balancer. This NSG uses a service tag of type LoadBalancer to allow traffic from the load balancer.

The packet coming from external source can travel up to the VNet, but it was blocked at the NSG of node subnet.

Lessons Learned

We always need to have some dummy service ready to test what we need. We can use nginx dummy service like:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-nginx
spec:
 selector:
 matchLabels:
 run: my-nginx
 replicas: 2
 template:
 metadata:
 labels:
 run: my-nginx
 spec:
 containers:
 - name: my-nginx
 image: nginx
 ports:
 - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: my-nginx
 labels:
 run: my-nginx
spec:
 type: LoadBalancer
 ports:
 - port: 80
 protocol: TCP
 selector:
 run: my-nginx

As discussed above, it’s also important to have a Bastion host able to access the control plane when the AKS cluster is private. Azure touts about CloudShell (and its ability to run in specified V-Net) but it’s pretty useless in troubleshooting. CloudShell sessions run inside of Kubernetes cluster and lacks common network troubleshooting tool such as nc. Azure has a managed service for Bastion but it requires a subnet with the exact name of AzureBastionSubnet.

We will explore more issues in the next post.