encryption on Digi Hunch

Managing EC2 instances across accounts with Ansible

Mon, 27 May 2024 14:00:22 -0400

I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is consistency in the VM management, over time, and across platforms (AWS, on-prem, Azure, etc). Even for AWS shops, in the last few years many enterprises have adopted AWS landing zone with the multiple AWS account prescriptive pattern. However AWS Systems Manager still lacks integration with AWS Organization (except for a few non-core capabilities). This creates the demand of managing EC2 instances across AWS accounts. In this post, we propose a secure method to manage a fleet of EC2 instances from multiple AWS accounts, using Systems Manager . It also enables connectivity from an Ansible control node.

Prerequisites

This proposal ties a few CLI tools together, including AWS CLI, SSH, Ansible, etc. It also requires the cloud engineer to understand how they work. I’ll start with the choice of the tools.

Above, I discussed the benefit of Ansible. Since Ansible operates on SSH, we’ll still have to use SSH tools. Even though SSM agent provides a way to connect to EC2 instance without requiring an RSA key pair, we still need SSH since it is a well-established industry standard (RFC4253) and the foundation of Ansible. These two technologies are not mutually exclusive. In fact, the SSM agent provides a secure enhancement to the operation with SSH. Traditionally, on each EC2 instance we’d have to run SSHD services which opens TCP port 22 (or alternative TCP port as configured). For authentication we favour key pair over password but the open port is still an attack surface vulnerable to brute force and DDoS attacks. For EC2 instances on private networks there is no reachability to the instance’s SSH port unless the bastion is also in a connected network.

As I cover in a post, the SSM Session Manager comes in handy. The SSM agent operates from the instance and communicate outbound to AWS backend. Since the SSM agent runs under a privileged user on the OS, you can perform OS-level commands through SSM. Further, AWS developed a Session Manager plugin with AWS CLI, allowing AWS CLI as a proxy command when making an SSH connection. Therefore SSM enables SSH connection without requiring port 22 to be open. In addition we’ll need to use RSA key pair as required for SSH, which is also an improvement to the security posture.

That explains the dependent tools. On the Ansible control node, apart from Ansible itself, we need the latest version of AWS CLI with the Session Manager plugin, we need to configure AWS CLI properly to connect to EC2 instances across multiple AWS accounts.

Configure AWS CLI

This section discusses how to configure AWS CLI. I have a couple of handy aliases for productivity but they are not essential. For example, I often need to check the IAM identity making the call, and I often need to list out all profiles configured. So I added the following two entries in the ~/.aws/cli/alias file:

[toplevel]
whoami = sts get-caller-identity --no-cli-pager --output yaml
profile = configure list-profiles

With that I have an alias to check IAM identity and available profiles. Then we can start configuring the profiles for CLI (in the file ~/.aws/config). Since we’ll be working with multiple AWS accounts, we have to manage multiple CLI profiles, which implies that:

We better use the --profile switch to explicitly specify profile being used, instead of relying on the AWS_PROFILE environment variable;
As a security best practice, we should not configure profiles with long-term IAM credential in the config file;
We must ensure the CLI doesn’t prompt for log-in every time we switch profile

To satisfy #3 there are many ways but we’ll discuss two: using cross-account IAM role, and using AWS SSO.

Bonus point if you enable auto-complete for AWS CLI.

Configure AWS CLI Profiles

With cross-account IAM role, the idea, is that the client start with one IAM identity, and use that IAM identity to assume roles on several other accounts. The configuration looks like this:

[profile jump_account]
credential_process = /opt/bin/awscreds-custom --username helen

[profile target_account_1]
role_arn = arn:aws:iam::123456789011:role/OrganizationAccountAccessRole
source_profile = jump_account

[profile target_account_2]
role_arn = arn:aws:iam::123456789012:role/OrganizationAccountAccessRole
source_profile = jump_account

In this example, you start with an validated identity in the jump account, then assume a privileged IAM role named OrganizationAccountAccessRole on the target accounts. Typically such IAM roles are pre-configured (e.g. in an multi-account landing zone) with appropriate trust policy to allow principals from the jump account. Once you’re validated as the IAM identity in the jump account, then you can use profiles for target accounts without being prompted for credentials again.

If your have configured IAM Identity Center for the multi-account environment, consider an alternative approach using sso login. The configuration usually looks like this:

[sso-session sso]
sso_start_url = https://myorg.awsapps.com/start/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile target_account_1]
sso_session = sso
sso_account_id = 123456789011
sso_role_name = AWSAdministratorAccess

[profile target_account_2]
sso_session = sso
sso_account_id = 123456789012
sso_role_name = AWSAdministratorAccess

This is often used by human users with SSO credential. In this example, to authenticate the sso session, start with command “aws sso login” . Then you can use all profiles by specifying --profile switch without having to log in again.

Configure SSH to EC2 via SSM

The EC2 instance must connect to Systems Manager endpoint, before one can SSH to the instance using the plugin. Once connected, you should find the instance in Fleet Manager. For this to happen, there are a few conditions. First, the instance must be able to reach the endpoint, either via public Internet, or via VPC interface endpoints if in a private subnet. Second, the instance profile must contain an IAM role with appropriate permissions. We can use AWS managed policy AmazonSSMManagedInstanceCore in the role. In addition, if we record the SSM session to an S3 bucket with encryption the instance profile must have permission to use the encryption key.

SSM agent uses the IAM role. The agent runs as a service on Linux or Windows machines. Many AWS managed AMIs come with SSM agent pre-installed. If that is not the case, you’d install the agent in your own AMI, or in user data which requires downloading the installer. With these configuration you’d be able to connect to the instance via SSM. We can use AWS CLI SSM command, or AWS web console to start an SSH session. To use SSH CLI utility, we install the session manager plugin along with AWS CLI on the SSH client machine. We also specify a public key for EC2 instance and run ssh command with the private key. The SSH configuration needs a configuration such as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

With this entry, you may directly SSH by instance ID (usually starting with i-* and mi-*), and the specified Proxy Command with SSM session document AWS-StartSSHSession will be invoked.

Configure Ansible Inventory

We can SSH to an instance (without port 22 open, on top of SSM) using the method above. Similarly, we can also configure Ansible to connect to the instance, without port 22. The inventory configuration looks like this:

mytest:
 hosts:
 instance1:
 ansible_host: i-00aabbffcc7755221
 ansible_user: ubuntu
 ansible_ssh_common_args: -o ProxyCommand="aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1"
 instance2:
 ansible_host: i-eedd88ff66aa22442
 ansible_user: ubuntu
 ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ProxyCommand="sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --profile target_account_2 \""

Note I used two patterns for ansible_ssh_common_args that are similar. Both works. Each entry references its own profile. This is necessary because Ansible does not have the knowledge which instance belongs to which profile’s account.

Another way to get Ansible connect to instances is to use the connection plugin community.aws.aws_ssm, by specifying ansible_connection: aws_ssm (e.g. as host variable) and other required variables (e.g. profile). This method does not require SSH channel but it requires an S3 bucket, and hence IAM permission on the controller node.

Ansible supports dynamic inventory, in three ways: inventory source file (with existing plugin), custom inventory plugin, and inventory script (in Python). Take source file as an example for EC2, add the followings as the content of aws_ec2.yaml:

plugin: amazon.aws.aws_ec2

# Attach the default AWS profile
aws_profile: target_account_1

compose:
 ansible_host: instance_id
 ansible_user: "'ubuntu'"
 ansible_ssh_common_args: "'-o ProxyCommand=\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1 \"'"

Then we can display the rendered inventory list, and Ansible-ping the instances.

ansible-inventory -i aws_ec2.yaml --list -y | less
ansible all -i aws_ec2.yaml -m ping

For more flexibility, for the composed variables, we could use jinja2 expression to generate the value. In both ways, we produce an inventory source per profile using the built-in inventory plugin aws_ec2. For greater flexibility, such as consolidating instances from all accounts into a single inventory, consider writing your own inventory script, or even own inventory plugin.

Conclusion

In this post we propose a way to manage instances across AWS accounts. Two main challenges are establishing the communication channel (SSH on top of SSM) and generating inventory data in Ansible. Some AWS services can generate inventory data, such as resource data sync in Systems Manager, or using AWS Config Aggregator. It is unfortunate that neither way produces the inventory data in a format that is directly compatible with Ansible inventory. Therefore, you might have to create a custom Ansible dynamic inventory script (in Python) that reads from the inventory data from AWS Config Aggregator (which supports AWS Organization). The side benefit of this script is that it is usually faster than the built-in aws_ec2 inventory plugin.

Public Key Infrastructure 3 of 3 – PKI Implementation

Sat, 30 Mar 2024 00:13:00 -0400

After the last two post, now we can focus on PKI implementation. The use case is software testing, where we need to create and recycle a lot of short-lived certificates. Typically, we don’t have to create public certificates because testing workload is internal. Also, hosting a public CA is much more involving. In this post we go over some options to host private CAs.

Root CA

I’m assuming we are creating a self-signed root CA for the organization. Large organizations usually keep their root CA offline and store the key material in highly protected configuration such as HSM. That is not something we can easily emulate. Nor are we concerned with the details of protecting root CA. In order to enable configuration of intermediate CAs, all we need for root CA, is to create a self-signed certificate along with the key. We can do this with OpenSSL following the first section of this post. Now we’ll use open-source Step CA. In my opinion, this tool is handier. For example, we can create a root CA this with a single command:

step ca init

This will start a prompt with a few questions. Then it will create a root CA and an intermediate CA. The keys and certificates are stored in ~/.step/ directory.

Note that by default, the root CA certificate has a path length of 1. To make the path length more than 1, we’d have to customize the creation. We can specified the desired path length in a certificate template and create a self-signed certificate with the template:

cat << EOF > root.tpl
{
	"subject": {{ toJson .Subject }},
	"issuer": {{ toJson .Subject }},
	"keyUsage": ["certSign", "crlSign"],
	"basicConstraints": {
		"isCA": true,
		"maxPathLen": 2
	}
}
EOF

step certificate create --kty=RSA --size 4096 --template root.tpl "DigiHunch Root CA" root_ca.crt root_ca.key

The command outputs the certificate and key of our test root CA. With that, next, we’ll create subordinate CAs with a few different tools.

AWS Private CA as intermediate CA

I’ll start with AWS Private CA. It was a spin-off service from AWS Certificate Manager and is fairly simple. However, we need to understand its capacity limit. AWS Private CA documentation has a page on RFC Compliance, which lists what in RFC 5280 are supported and what are not. It performs very basic CA functions. It does not perform domain validation, hence no ACME support. We can create root CA as well but here we’ll create an intermediate CA:

In AWS Console, create a Private CA. Indicate you want to create a subordinate CA. You may specify CRL distribution and OCSP endpoint. However, you’re responsible for the providing CRL and/or hosting OCSP service;
In the Private CA, we need to create CA Certificate. We need to provide CSR to our root CA to sign this CA’s certificate. The root CA can be either another AWS Private CA or external CA. In this case we choose external CA and export the CSR to a file (e.g. dh.csr).
We can use Step CA to sign the request. Note that we have to give reasonable value for expiry date. Because ACM create certificate with 13 month validity period by default, the expiry date must be at least 13 months from the current date. We also need to set path length. In this case, I put 0 so the Private CA can only issue end-entity certificate. The command looks like this:

step certificate sign pca.csr \
 /Users/digihunch/.step/certs/root_ca.crt \
 /Users/digihunch/.step/secrets/root_ca_key \
 --profile intermediate-ca \
 --not-after=2027-01-24T07:20:50.52Z \
 --path-len=0

In the AWS console, we paste the generate certificate content in the as certificate body, and the root CA’s certificate as certificate chain.

If the Private CA serves as parent CA of one more layer of CA, set the path length to 1. This requires the path length to be at least 2.

Once the CA certificate is installed, we have completed creating a private CA. We can reference this CA from ACM (AWS certificate manager) when creating a private certificate. Since there is no validation support, ACM will allow you to claim any domain. Currently it does not support ACME-based certificate automation. However, you can use AWS CLI or any AWS based automation tool to create your private certificate.

Step CA

As a cheaper alternative, we can host private CA using step CA on virtual machines. One of the benefits is ACME support (http-01 challenge). In the init command in this post, we already create the intermediate CA certificate and configuration. In this part of the lab, we’ll have two servers: the CA server (pki.digihunch.internal) and the web server (web.digihunch.internal). Make sure the DNS resolution works for both servers. The architecture looks like this:

The CA server runs the Step CA process acting as ACME server. We deploy the CA in standalone mode (instead of linked or hosted deployment), meaning it’s not connected to any cloud services. We can host the service on port 443 so make sure the process has port binding permission within the operating system, and the firewall (security group) allows traffic via port 443.

The Web Server runs Step CLI acting as ACME client. On the web server, we run Step CLI in standalone mode (instead of webroot). During the challenge-response phase, the CLI will get the required random number from ACME server, and host it the as the response on port 80. Make sure that Step CLI process has port binding permission in the OS, and the firewall (security group) allows traffic via port 80.

Make sure the DNS resolution works for both servers. On the CA server, we fetch the fingerprint (in preparation for setting up Step CLI on web server). Then we add an ACME provisioner, and host the CA server with a single command:

step certificate fingerprint $(step path)/certs/root_ca.crt
step ca provisioner add myacme --type ACME
step-ca $(step path)/config/ca.json

The CA server is listening on port 443 (by default). Now we can test ACME process with any ACME compatible client. Let’s use step CLI on the web server:

step ca bootstrap --ca-url https://pki.digihunch.internal:443 \
 --fingerprint <fingerprintvalue>
step ca certificate web.digihunch.internal acme.crt acme.key \
 --acme https://pki.digihunch.internal/acme/myacme/directory

The bootstrapping step is to establish trust on the CA server. Then the provisioning process should complete automatically:

Here the step CLI command uses standalone mode by default so it is important to ensure reachability (port 80, DNS name) from the CA server when step CLI hosts the response. You can also go with webroot mode, where Step CLI generate the file to the web root directory so the response becomes available. This is helpful when you already run another web hosting process such as Nginx on the web server. This blog post covers the usage of other ACME-compatible tools with Step CA as private CA server.

Another well-known tool is HashiCorp Vault. There is already a good tutorial here and all the OpenSSL command in the tutorial can be replaced with Step commands.

Cert Manager on Kubernetes

The Cert Manager project on Kubernetes makes PKI work simple. In my discussion about self-signed certificate on Kubernetes, I covered how to use Cert Manager to create self-signed certificate. However, a corporate with multiple clusters may need to chain each cluster-wide issuing CA to an intermediate CA outside of the cluster. Let’s look at this architecture as an example of a full PKI implementation:

In this architecture, we extend corporate on-premise root CA to the cloud. The Ops account hosts the AWS Private CA, and shares it out to workload account(s) using Resource Access Manager. In each EKS cluster, the Private CA serves as the cluster-level issuer, which can issue CA certificates across Kubernetes namespaces. In each namespace, there is a Cert Manager issuer responsible for issuing certificates within the namespace. Cert Manager supports many issuers and we’re using the AWS Private CA issuer.

To demonstrate the gist of this architecture, in our lab we’ll create one Kubernetes cluster in the same AWS account as the Private CA, and create a Cert Manager certificate in one namespace (e.g. ingress). To get started, create a private CA with the instruction in this post and ensure that the private CA has path length of 1. Then create an EKS cluster. You may use my CloudKube project to provision this cluster in Terraform or any other ways. We use the IRSA model to grant a service account access to the Private CA. First, we create an IAM policy and reference the ARN of the private CA:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "awspcaissuer",
 "Action": ["acm-pca:DescribeCertificateAuthority", "acm-pca:GetCertificate", "acm-pca:IssueCertificate"],
 "Effect": "Allow",
 "Resource": "arn:aws:acm-pca:<region>:<account_id>:certificate-authority/<resource_id>"
 }
 ]
}

Name this policy ‘PCA-Access’. It is the minimum required permission. Now let’s create a service account, along with an IAM role that uses this policy:

eksctl utils associate-iam-oidc-provider \
 --region $AWS_REGION \
 --cluster $CLUSTER_NAME \
 --approve

eksctl create iamserviceaccount \
 --cluster=$CLUSTER_NAME \
 --namespace=cert-manager \
 --name=aws-pca-sa \
 --role-name EKSCertManagerPrivateCARole-$CLUSTER_NAME \
 --attach-policy-arn=arn:aws:iam::123456789012:policy/PCA-Access \
 --approve

In order to run the command successfully, you need the IAM permission to create IAM role, as well as API access to the cluster. Now we can install both Cert Manager and the Private CA Issuer.

helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --version v1.13.3 \
 --set installCRDs=true \
 --create-namespace

helm install aws-ca awspca/aws-privateca-issuer \
 --namespace cert-manager \
 --version v1.2.7 \
 --set serviceAccount.create=false \
 --set serviceAccount.name=aws-pca-sa

Note that when installing Private CA issuer with helm, specify the service account that we created earlier (aws-pca-sa). We install both to the cert-manager namespace, where we’ll create a ClusterIssuer. Now we can declare the following manifest:

apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
 name: pca-cluster-issuer-rsa
 namespace: cert-manager
spec:
 arn: arn:aws:acm-pca:ca-central-1:383500642091:certificate-authority/7f2d7b38-2508-4492-81f0-b5b85427c99c
 region: ca-central-1
---
apiVersion: v1
kind: Namespace
metadata:
 name: ingress
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: ingress-ca-cert
 namespace: ingress
spec:
 isCA: true
 commonName: ingress-ca
 secretName: ingress-ca-secret
 privateKey:
 algorithm: RSA
 size: 2048
 issuerRef:
 name: pca-cluster-issuer-rsa
 kind: AWSPCAClusterIssuer
 group: awspca.cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: ingress-ca-issuer
 namespace: ingress
spec:
 ca:
 secretName: ingress-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: ingress-cert
 namespace: ingress
spec:
 commonName: web.digihunch.com
 secretName: web-digihunch-secret
 duration: 2160h
 renewBefore: 72h
 subject:
 organizations:
 - digihunch
 dnsNames:
 - web.digihunch.com
 privateKey:
 algorithm: RSA
 size: 2048
 issuerRef:
 name: ingress-ca-issuer
 kind: Issuer
 group: cert-manager.io

In this manifest, we create a CA certificate, along with a CA in the ingress namespace. With that, we create an end-entity certificate. As a result, we should find the certificates ready:

kubectl -n ingress get certificate
NAME READY SECRET AGE
ingress-ca-cert True ingress-ca-secret 24s
ingress-cert True web-digihunch-secret 24s

The certificate can be reference by the workload in the namespace. Because an Issuer can only issuer Certificate in the same namespace, we need a issuer in the namespace where the certificate will live. The other benefit that Cert Manager brings, is the support of ACME challenges, which is an enhancement to AWS Private CA.

Summary

We discussed several approaches in PKI implementation. PKI is a fundamental requirement in a software testing environment today. Having an internal public key infrastructure enables many other use cases too. For example, the team may use SSH user certificate. You can also host a CA with Step. PKI allows an enterprise to configure SSL inspection on their Next Generation Firewall. The IAM Role Anywhere feature on AWS also operates on an organization’s own PKI.

Public Key Infrastructure 1 of 3 – Basics

Tue, 20 Feb 2024 00:05:00 -0400

In 2021, I wrote an intro to Public Key Infrastructure (PKI). Now that I have to host my own certificate authority, I decide to dive a little deeper into PKI in this series of posts. In software testing scenario, we need to issue (and recycle) a lot of certificates, and manage their lifecycle events such as (renewal, revocation). As a result, the corporate should establish its own private key infrastructure (PKI). This effort includes hosing their own Certificate Authority.

Concepts

IT professions often use the acronyms PKI and CA interchangeably in the context of implementation. However, strictly speaking, PKI is the entire framework (strategy, policy, etc) around managing certificate at corporate level, and CA is the specific entity that issues certificates. When it comes to architecting PKI, a key design decision is the hierarchy of CAs. Microsoft has a good page explaining the common options.

In a single-tier CA hierarchy, the root CA is also issuing CA. The root CA as the anchor of trust of the PKI also issues certificates to the end entities. Obviously, this does not scale. It is only for sandbox testing, and not suitable for any environment that needs to scale. It also carries significant risk because the private key at the anchor of trust has to remain online.

With a two-tier CA hierarchy, the root CA only issues certificates to its subordinate CAs. Therefore the root CA can go offline. As a result, the subordinate CAs issue certificates to end entities. Since the root CA can remain offline (except for issuing certificates for new CAs), the chance compromise is reduced significantly. This option also scales better with multiple issuing CAs. If one of them gets compromised, then only the end entities of the compromised CA are impacted. The root CA still needs to be online from time to time. Apart from granting new subordinate CA, the team may also need root CA online to sign CRLs, which is a management overhead.

The two-tier hierarchy introduces another problem. The root CA also needs to restrict the certificates that its subordinate CAs can issue. For example, one issuing CA can only issue certificates in the *.dev.digihunch.com domain, and another CA can only issue *.ops.digihunch.com. No subordinate CA shall issue “rogue” certificate beyond their authorized scope. In order to enforce issue boundaries on issuing CAs, RFC 5280 defines multiple ways to express constraints, such as basic constraints (including path length), name constraints, policy constraints, and EKU.

In a three-tier CA hierarchy, the top-level is still a root CA that stays offline. One level below, is a layer of CAs that also stay offline and we refer to them as intermediate CAs. Going down one more level down, there are the issuing CAs for end-entity certificates. Oftentimes, we use the intermediate CAs as policy CAs where we introduce restrictions to the subordinates. In this model, the root CA can remain offline nearly all the time because we can issue CRLs at the intermediate level. This is more flexible, but also more management overhead.

Implementation Options

The two-tier hierarchy is good in most scenarios but the three-tier hierarchy is also common for large organizations. The next consideration is implementation strategy. Take two-tier model for example. We can think about these options:

Implement a self-managed PKI, with an internal root CA. The internal subordinate CAs are the issuing CAs. They are chained to the internal root CA.
Implement a self-managed PKI, with its certificate purchased from a commercial CA. The internal subordinate CAs are the issuing CAs. They are chained to the external root CA, as the TPP (trusted third party). The external CA may or may not be a public root CA.
Purchase certificates from a commercial CA that are chained to a public root CA

Option 3 essentially delegates the PKI to a commercial provider. This is usually not a favourable option due to the hefty charge and minimal control. In option 2, when the third party is a public root CA, you can have all your certificates with public trust, although this is a pricey option too. Even if the third-party isn’t a public root CA, there is still a benefit of delegating the management of Root CA to commercial provider.

Option 1 is for use cases where we need a lot of certificates quick, for example, in agile development iterations. The certificates do not need trust beyond the organization, and the risk of root CA being compromised is manageable. In this option all CAs are private CAs. Option 2 on the other hand, is flexible in terms of trust boundary. You can just rely on the commercial provider as a TTP. If the TPP supports public root CA, you can issue certificates for public facing workload. In other words, you can choose either a private CA or a public CA. When we work with a private CA, we have to import the certificate to the trust stores of the organization.

Some PKI topologies splits a CA into two sub-components: the Certificate Authority Service (CAS, or simply CA) and the Registration Authority (RA), as the diagram above shows. With many locations, each location has an RA that communicates with a with a central CAS. The RAs receive requests from local clients and are responsible for authenticating these requests and pass the validated ones along to the CAS. The validation can take place in ACME protocol. The CAS implicitly trusts RAs and will sign the validated requests from RAs before sending them via API calls. The RA then relays the certificates back to the local requestors. This topology is more scalable with the CAS focusing only on signing and the RAs on validation and passing the requests along. Large organizations may have their central CAS hosted on-prem, and remote RAs in the CSP regions.

Components for X.509 Certificate

X.509 is the standard for digital certificate. The X.509 standard has the following most important fields:

Subject: The name of the subject (e.g. a user, service, device), commonly represented as X.500 formate distinguished name (DN). For website, the value can be CN=digihunch.com
Serial Number: A unique identifier for each certificate that a CA issues.
Issuer: DN of the CA. For a self-signed root CA, the issuer is the subject. They are different otherwise, such as in subordinate CA certificates and end entity certificates.

These fields were introduced in version 1. In addition, X.509 version 3 certificates introduces extensions that provide additional functionality and features to the certificate. Each extension comes in two flavours: critical and non-critical. A certificate-using system MUST reject the certificate if it encounters a critical extension it does not recognize or a critical extension that contains information that it cannot process. A non-critical extension MAY be ignored if it is not recognized, but MUST be processed if it is recognized.

Here are some common extensions that a lot of implementations use:

Subject Alternative Name (SAN): only used in end entity certificates, not in CA certificates. The format of SAN is flexible and it does not have to be X.500 DN. For website certificate, we often place alternative DNS names here.
Key Usage: The intended scope of usage for a private key is specified through the Key Usage and Extended Key Usage (EKU) extensions in the associated certificate. Example: “Certificate Sign, CRL Sign”
Basic Constraints: Used to distinguish between end-entity cert and CA cert. You should also specify path length. The value can be: “CA:TRUE, pathlen:1”. If the path length constraint is 0, the CA may have one more level of subordinate CA. But these subordinate CAs must have path length of 0 on their own certs, and cannot extend one more level as their own subordinates. Those subordinate CAs can only issue end-entity certificates. If the path length value is none, then there is no restriction on the levels of subordinate CAs.
Name Constraints: for CA certs only, defined in RFC5280, to limit the scope to certain names on the certificate that the CA issues. Client must verify that a certificate is allowed to be signed by CA.
CDP (CRL distribution points): URL(s) where the application or service can retrieve the certificate revocation list (CRL).
AIA (Authority Information Access): URL(s) where the application or service can retrieve the revocation list for CA’s certificate
SKI (Subject Key Identifier): the SHA-1 hash of the subject’s public key
AKI (Authority Key Identifier): the SHA-1 hash of the issuer’s public key
EKU (Extended Key Usage, also Enhanced Key Usage): an object identifier (OID) for each application or service a certificate can be used for. It needs to align with Key Usage
Certificate Policy: reference to the certification practice statement (CPS) of the issuer. During exchange, any relying party can access the assurance level associated with the certificate, and decide on the level of trust to put in the certificate.
Policy Constraints: for path validation, it can be used to prohibit policy mapping or to require that each certificate in a path contain an acceptable policy identifier.
Policy Mappings: in CA certificates to restrict the certificates that the CA can issue.

This isn’t a complete list. Plus, X.509 V3 also supports custom extensions. Here is a sample certificate. In real life, the way each implementation uses these extensions may vary slightly, so interoperability issues between PKIs exist. When designing your own PKI, it is important to examine the usage of extensions to comply with the standard. For example, Microsoft has the following recommendations to restrict certificates:

For subordinate CA certificates, the Basic Constraints extension should be present and marked as critical
The cA field should be set to TRUE
The pathLenConstraint field should be set to the minimum value required to enable the business scenario (i.e. 0 if that CA will issue certificates only to End Entities)
The EKU extension should be present and contain the minimum set of EKU object identifiers (OIDs) to enable the business scenario. Furthermore, the anyExtendedKeyUsage OID (2.5.29.37.0) should not be specified.

Summary

This post discussed the basic concepts in public key infrastructure. In the next post, I’ll cover the automation of certificate issuance.

Landing Zone in Azure – Introduction

Sat, 25 Mar 2023 01:30:00 -0400

I recently renewed my associate administrator certification, and feel it’s a good opportunity to brush up on Azure landing zone.

The lame part of this is the semantics. I found many similar terms across cloud service provider (CSPs). In the context of Azure, it makes sense to clarify the terms again for Cloud Adoption Framework (CAF) and Cloud Operating Models.

Cloud Adoption Framework (CAF)

Similar to AWS Cloud Adoption Framework (CAF), Azure also has the concept of CAF and it means the same thing. This part may feel lofty, but it’s in fact foundational. To get started on the cloud there are thousand ways to configure the foundation (right or wrong). The adopter needs CAF to navigate through the offerings and define what they can achieve.

The CAF documentation is good although length. The most “beefy” part is Ready section.

Cloud Operating Models

Every cloud company has some narrative about cloud operating model. For example, Here’s Hashicorp’s definition, and here’s AWS‘ white paper on it. In the context of Azure, the CAF document gives some guidance on developing your own operating model in alignment with the CAF. In addition, it also gives a few example cloud operating models:

Decentralized operations
Centralized operations
Enterprise operations
Distributed operations

There is a comparison table that highlights their differences as well as an accountability chart proposing team divisions. Another insightful table is the one that list out implementation starting point and typical path of iterations for each operating model. The table also suggests that Azure Landing Zone includes two implementation options: starting small and CAF enterprise-scale.

Landing Zone at High Level

Followed by Cloud Operating Model is the design and implementation of Azure Landing Zone. There are currently eight design areas:

Billing and Active Directory tenant: including Azure AD tenant
Identity and Access Management: including hybrid identity
Network Topology and Connectivity
Resource Organization: different levels of resource containers
Security
Management
Governance
Platform automation and DevOps

Out of the many design areas, I fell short off IAM and Network so I’ll try to discuss them in more details below in the next section. As for resource organization, apart from Resource Group and Subscription, it is also important to understand management group.

Most cloud engineers work with subscription and resource group. That is where a lot are going on. For enterprises however, Azure has to address the requirement for the capability of top-down enforcement. Management Group provides a governance scope above subscriptions, provided that all subscriptions trust a single Azure AD account. Management groups may form a hierarchy of up to six levels to help you configure policies and access, so that the all the subscriptions under each management group have unified policy and access configuration. At the very top is root management group. Any assignment of user access or policy on the root management group applies to all resources within the directory. Because of this, all customers should evaluate the need to have items defined on this scope.

We can apply policy guardrails (e.g Azure Policy) at management group level so that the policies are effect across subscriptions. Azure Policy can also address operational compliance considerations by monitoring configuration drift.

Identity and Access Management

First, we really need to distinguish AD DS on Windows Server, Azure AD and Azure AD DS. In an old post, I discussed what is a Windows domain, the key role of a domain controller (to manage user identity, as well computer identity), and the fact that Active Directory is a complete redesign of Windows Domain system since Windows 2000. So we can start with AD DS on Windows Server:

AD DS on Windows Server: In the good old days, some common network administrative activities were to configure Active Directory (including the X.500 compatible database, the OUs, domains and forests) on Windows Servers, joining computers to the company’s domain, configure group policy, configure LDAP and Kerberos, upgrading Domain controllers, etc. Over the years, Microsoft moved these activities to the cloud and offer them as a managed service, known as Azure AD DS.
Azure Active Directory Domain Service (Azure AD DS): allows you to use managed domain services (e.g. Windows Domain Join, group policy, LDAP, Kerberos authentication) without having to deploy, manage or patch domain controllers. It is a SaaS offering to manage your domain controllers in the cloud, with a pay-as-you-go model. The counterpart in AWS is “AWS Directory Service” which lets you run Microsoft Active Directory (AD) as an AWS managed service.

In summary, both AD DS on Windows Server (self-hosted) and Azure AD DS (managed service) are identity stores that operates on Windows domains. Even though the latter is a managed service, it supports LDAP or Kerberos as integration protocol for third party applications (usually on-premis) to use. Both LDAP and Kerberos came around prior to the cloud era and they are not optimized for cloud connectivity. For example, insecure bind (on port 389) in LDAP is still prevalent. Kerberos is fairly complex to configure. However, they are not phased out right away because of their established presence as well as the domain’s awareness to authenticate devices.

Many organizations have to keep their domain service and when they move to cloud so they still have to use Active Directory as identity store. For this, Azure has Azure AD connect. On the AWS side, there is also an AD connector tool to allow on-prem users to log into AWS applications and services. With AD connector you can also join EC2 instance to existing AD domain.

Now let’s examine Azure AD.

Azure AD: is an IAM solution. It contains an identity store (with users and groups in a flat directory structure) but more importantly it integrates with external identity stores (including Domain Service, self-hosted or SaaS managed), which gives it hybrid-identity capability. A company can even sync their own on-prem identity store to Azure AD using Azure AD Connect. As an IAM solution, Azure AD also allows a company to tie their identity store to applications using modern protocols such as SAML and OAuth. Azure AD treats applications as objects, and they can represents either Microsoft Applications (Office 365, Dynamics 365, Azure) or third-party ones (Slack, Salesforce) as long as they use the supported protocol for SSO. The closest AWS counterpart of Azure AD is Amazon Cognito (arguably), even though their capabilities are not identical in every aspect.

Compared to Domain Service, Azure AD alone doesn’t have the concept of domain. Therefore you cannot join a server or PC to a domain and configure group policy. Azure AD’s native identity store is a flat directory structure without OUs or forests. Azure AD is NOT a replacement of domain service, either self-hosted or managed.

Now coming back to the Azure landing zone literature, the document lays out the key decision to make about identity:

A critical design decision for enterprise organizations adopting Azure is whether to extend current on-premises identity domains into Azure or to create new identity domains.
Azure Active Directory (Azure AD) and hybrid identity

The document even includes a comprehensive identity decision guide. After this decision, we’ll know what identity store to use. Then we can address the problem of platform access vs workload access. In other words, IAM of management traffic vs business traffic, which opens up topics such as RBAC, service principle and managed identities.

Networking

Back in 2017, Azure published a white paper about V-Net and it focuses on mesh network and hub-and-spoke. Back then Azure customers run multiple lines of business (LOB) on different V-Nets. The V-Net peering feature allows early cloud adopters to organize all their V-Nets in a mesh topology, ensuring all peers have access to all other peers, or a hub-and-spoke topology to aggregate shared resources in hubs so they can be shared by the spokes in the network.

When setting up a landing zone, network topology is a big decision. In the landing zone document today, clients need to consider the followings:

Traditional Azure networking topologies, including:
- large flat V-Net
- multiple V-Nets connected with multiple Azure ExpressRoute circuits/connections
- hub-and-spoke
- full mesh
- hybrid
Microsoft managed networking topology (on top of Virtual WAN)

From the 2017 white paper, most organizations at that time solve their need for network isolation and connectivity by creating a mesh architecture among various V-Nets. All nodes in the network are interconnected so network traffic is fast and can be easily redirected. However, mesh topology has significant disadvantages because it requires too many connections as the footprint expands, making it very costly to operate and quick to reach limit of number of peering links. It is not scalable. The white paper is to advocate the use of hub-and-spoke topology, which I will discuss in the next section.

It is worth-noting that, today (Jan 2023) one can create both topologies with Azure Virtual Network Manager. It is currently a preview service but I can foresee it will eventually get integrated with landing zone.

No matter which topology, another issue to address is connectivity to on-prem network, and to Azure PaaS services. If the traffic is light, we can use VPN gateway to configure IPSec tunnel that goes over public internet encrypted. It is simple to configure with a good aggregate bandwidth. This connection requires a VPN device on premise as well. A faster alternative is Azure ExpressRoute, which runs a private connection with a third-party connectivity provider. ExpressRoute is more complex and expensive to set up, but it supports much higher bandwidth with direct access and better SLA. In reality, many clients configures ExpressRoute with VPN failover for connectivity to on-prem network. For connectivity to PaaS services, options are service endpoint and private link endpoint.

Hub-and-spoke topology

In the traditional topologies, hub-and-spoke network topology is popular as the hub network provides a central point of management. Also it overcomes subscription limits and institutes a separation of concerns. The Azure documentation recommends hub-and-spoke architecture for larger cloud adoption efforts. If the footprint is even massive, we can even extend the model to a cluster of hubs and spokes.

A cluster of multiple hub-and-spoke

We can connect multiple hubs using:

V-Net peering
Azure ExpressRoute
Azure Virtual WAN
Site-to-site VPN

Within a single hub-and-spoke model, the Hub V-Net hosts shared services and acts as central point of connectivity (to many spoke V-Nets). Often in the Hub V-Net are Azure Bastion, Azure Firewall and VPN Gateway or ExpressRoute gateway. The spoke V-Nets (in same or different subscriptions) isolates and manage workloads in prod, non-prod, etc. Since a single V-Net cannot traverse subscription boundaries, you have to use V-Net peering (preferred), ExpressRoute circuit, or VPN Gateways. V-Net peering works across regions, and across Azure AD tenants. It is low-latency but isn’t transitive.

In some cases we also configure perimeter networks (aka DMZs) in the hub-and-spoke architecture, to handle external traffic. Perimeter networks host services such as External Load balancer, Azure Firewall, Azure Application Firewall on Azure Application Gateway or on Azure FrontDoor) , network virtual appliances (NVAs), IDS, IPS, and other security appliances. Incoming packets flow through the security appliances before reaching back-end servers. Internet-bound packets from workloads must also flow through security appliances in the perimeter network before they can leave the network. The document gives an example of a DMZ hub V-Net with two perimeter networks.

Virtual WANs

This page discusses what is WAN and SD-WAN. WAN connects multiple LANs in different geographic areas and is common with companies with multiple offices in different regions. WAN infrastructure may be privately owned or leased as a service from a third-party service provider (hybrid WAN). Companies may use IPSec VPN, SSL VPN or direct connection to build their WANs. Software-defined WAN (SD-WAN) leverages virtualization technologies, network overlays, on-site SD-WAN devices and software platforms to build hybrid WANs.

Azure Virtual WAN (similar to AWS cloud WAN) is a managed service to build a virtual WAN with a single operational interface that brings many networking, security and routing functionalities together. It simplifies end-to-end network connectivity (within Azure, between Azure and on-prem) by creating a hub-and-spoke architecture.

Virtual WAN

Virtual WAN is essentially an integrated connectivity solutions (in hub and spoke), with a global transit network architecture. The configurations, including spoke setup) is automated and troubleshooting is more intuitive. Global transit network configures multiple virtual WAN hubs with hub-to-hub connectivity, which ultimately enables any-to-any connectivity, with different paths discussed here.

The landing zone document recommends Virtual WAN for new large or global network deployments in Azure where you need global transit connectivity across Azure regions and on-premises locations.

Summary

Landing Zone configuration involves many components and there is no way to discuss everything thoroughly. In this post I put down my notes reading Azure landing zone documentation. Overall, working on landing zones requires learning a variety of services by the CSP.

A taste of IoT device tracking

Fri, 03 Mar 2023 10:53:00 -0400

Last fall I worked on a demo project for IoT device tracking on AWS so I had some reading.

Overview

From a strategic level, as servers move away to the cloud, AWS envisions that the footprint left on premise will mainly be IoT devices. The role of the cloud therefore becomes a central point of management for IoT devices. As a result, most of the AWS services for IoT are managed services. The best resources are two white papers: IoT Lens – AWS Well-Architected Framework and Securing Internet of Things (IoT) with AWS. The most important services are:

IoT Analytics: Makes it easy to run sophisticated analytics on volumes on IoT data. It connects to the underlying IoT data store and allow you to build your own analytical queries and Jupiter notebooks.
IoT Events
IoT Core: Core features for IoT.
IoT SiteWise
IoT Device Management
IoT Greengrass
FreeRTOS: IoT devices usually need to be small and power efficient. The software and OS running on the device is important. FreeRTOS is a real-time operating system for micro-controllers supported by AWS. FreeRTOS provides kernel, OS and libraries to securely connect your edge device to the cloud in no time.

In this post I will explore IoT architecture at high level. In real life, you program your device with AWS IoT Device SDK and AWS IoT API in different languages. In this post, I use a script to simulate GPS data, and push it to AWS IoT using SDK. Then I render the location using sample AWS code for Amazon Location.

IoT Architecture

When creating IoT services we consider registration and telemetry capturing flow. The Well architected white paper proposes registration flow as such:

Registration Flow

Device Registry to keep track of devices (aka Things). You can find where your devices are, and filter by a common feature (e.g. ModelX device only). Registration flow usually involves a testing of communication between device and server. The authentication must be 2-way where server needs to validate device identity, and device needs to validate server identity. You can use a unique X.509 certificate per device to adhere to security best practices on AWS. This way, if one device gets hacked, the entire fleet of devices is not affected by one certificate being compromised. An alternative authentication method is Cognito. With Cognito you can sign your users into a mobile application, so you use IAM policies to authenticate them into viewing different dashboards or viewing the data that pertains to them specifically. IoT Core policies can help manage authorization.

The white paper also proposes a few options for capturing telemetry:

Options for capturing telemetry

These options presents a common pub-sub pattern, where the device streams message by topic to the Message Broker in IoT core. The IoT core also involves policy and rules. A rule may involve a subscriber to consume the messages. A more comprehensive architecture from AWS IoT device connectivity workshop looks like this:

The message path from device to IoT core remains the same. When building an IoT solution, we first address the messaging path.

IoT Protocols

In IoT core, Device Gateway is the entry point for IoT devices connecting to AWS.It supports MQTT, WebSockets and HTTP 1.1 protocols, on top of TLS. Registration flow uses HTTP/REST protocol for provisioning, and with MQTT protocol for a message test. For pushing telemetry, we can use both but prefer MQTT (topic based) because of its advantages in IoT messaging. Here is a good article on the differences.

AWS has a white paper on designing MQTT topics, with a few communication patterns and best practices. The SDK documentation also explained the communication protocols, including:

HTTPS: publish only
MQTT: publish and subscribe
MQTT over WebSocket: publish and subscribe. Device Gateway will maintain long lived, bi-directional connections, enabling devices to send and receive messages at any time with low latency.

Pay attention to the authentication mechanism. As to what protocol is used in a communication, they can be dynamically negotiated using the ALPN protocol. ALPN (Application-Layer Protocol Negotiation) is a TLS protocol extension that allows the application layer to negotiate which protocol should be performed over a secure connection in a manner that avoids additional round trips and which is independent of the application-layer protocols.

GPS data simulator

I don’t have a GPS chip. To get sample GPS data, I used geojson.io website, use a pen to paint the points and collect the result in JSON format. The data looks like this:

{
 "type": "FeatureCollection",
 "features": [
 {
 "type": "Feature",
 "properties": {},
 "geometry": {
 "coordinates": [
 [
 -119.4966304331144,
 49.88901098598916
 ],
 [
 -119.4966304331144,
 49.889903059931726
 ],
 [
 -119.49658503509582,
 49.89066350338888
 ],
 [
 -119.49658503509582,
 49.891365440561145
 ],
 [
 -119.4966304331144,
 49.89219897769374
 ],
 [
 -119.4966304331144,
 49.88735841200943
 ],
 [
 -119.49660773410511,
 49.8880603972496
 ],
 [
 -119.4966304331144,
 49.88885011844141
 ]
 ],
 "type": "LineString"
 }
 }
 ]
}

Save this file as map.geojson to later feed it to device simulation script.

Rendering location data

In AWS location samples project, the sample project maplibre-js-react-iot-asset-tracking is a good demo of IT. The readme document contains a walk through, using AWS amplify services. The steps includes creating certificates, configuring lambda function to add location data to tracker. The project directory also includes the device simulation script, as index.js. I slightly modified the content to this:

const awsIot = require("aws-iot-device-sdk");

// Replace with your AWS IoT endpoint
const THING_ENDPOINT = "safdsa-ats.iot.us-east-1.amazonaws.com"; // get from console
const CLIENT_ID = "trackThing01";
const IOT_TOPIC = "iot/trackedAssets";
const DEVICE_ID = "thing123";
const GEOJSON_FILEPATH="geojson/map.geojson";
const fs = require('fs')
const file_raw = fs.readFileSync(GEOJSON_FILEPATH).toString();
const positions_raw = JSON.parse(file_raw).features[0].geometry.coordinates
const POINTS_ON_MAP=[]
for (var i=0;i<positions_raw.length;i++){
 POINTS_ON_MAP.push({lat:positions_raw[i][1],long:positions_raw[i][0]})
}

const device = awsIot.device({
 host: THING_ENDPOINT,
 keyPath: `${__dirname}/certs/private.pem.key`,
 certPath: `${__dirname}/certs/certificate.pem.crt`,
 caPath: `${__dirname}/certs/root-CA.pem`,
 clientId: CLIENT_ID,
 keepalive: 60000,
});

console.log("Connecting to %s with client ID %s", THING_ENDPOINT, CLIENT_ID);

device.on("connect", async function () {
 console.log("Connected to device %s", CLIENT_ID);

 for (const point of POINTS_ON_MAP) {
 const message = {
 payload: {
 deviceId: DEVICE_ID,
 timestamp: new Date().getTime(),
 location: point,
 },
 };
 console.log(
 "Publishing message to topic %s: %s",
 IOT_TOPIC,
 JSON.stringify(message)
 );
 device.publish(IOT_TOPIC, JSON.stringify(message), { qos: 1 });

 // Set timeout to sleep
 await new Promise((resolve) => setTimeout(resolve, 10000));
 }

 device.end();
});

We can obtain certificate ID from AWS console or by CLI command:

aws iot list-certificates --output text --query 'reverse(sort_by(certificates,&creationDate))[:1].[certificateId]' | cat

When running the script, it pushes data to IoT core service. The AWS Amplify project creates Lambda function that is subscribed to the topic and trigger actions. The data are used to render points on a map, which is available on the front end.

Summary

This is an overly simplified use case but it covers the basics. IoT solution will use a lot managed service and familiar technologies (e.g. TLS, certificate). Creating an IoT solution is mostly about address the onboarding services and make use of the MQTT based workflow. AWS managed services makes these easier.

More IoT workshops are available on AWS workshops.

MinIO for S3-compatible Object Storage

Fri, 09 Sep 2022 09:00:00 -0400

I reviewed some storage technologies on Kubernetes but they are all for block and file storage. In this post, I will discuss the current available options for container workload to use object storage. I will also touch on MinIO as an object storage solution.

Object storage

Block and file system are more native to operating system because they present themselves to the OS as a block device or file system attached to the OS. In other words, application processes running on the OS will be able to access the storage by address expressed as a POSIX-compatible path. On the contrary, object storage is a REST API service, operating at the application layer in the TCP/IP stack. Therefore, we can think of object storage as “storage as a web service”.

Object storage can be made very cheap. However, the application protocol may vary depending on the object storage provider. Amazon S3 is a forerunner in object storage market and its protocol has emerged as the de-facto standard for object storage. When building an application and if there is one object storage protocol to support, it should be S3. For non-S3 object storage services, we can front them with an S3 interface, if the provider itself does not have one. For example Ceph storage has its Gateway S3 API.

Container Object Storage Interface

If we use S3 as the universal object storage protocol, does that also address object storage access with container workload on Kubernetes? Absolutely. Nonetheless, for a number of reasons using REST API from containers are not the best option. From platform’s perspective, it is the platform that should define how to access object storage, instead of leaving it with an application-layer protocol.

When a pattern (for storage, or networking, etc) turns out very common, the platform layer should incorporate it as an infrastructure service, manage it with its own standard, and provide it to application so that developer can focus on business features. With that vision, the community brought up the Container Object Storage Interface (COSI) initiative. It is currently in very early stage, but the idea is to commoditize object storage in Kubernetes platform with a unified interface. For more background about this initiative, refer to the post “Beyond block and file – COSI enables object storage in Kubernetes“.

COSI is the ultimate cloud native solution but it is still in pre-alpha phase as of mid 2022. Unfortunately, it is not a recommended solution to any real-life project in 2022, and we are stuck with the unified API approach until COSI matures.. The unified API approach is by no means cloud native, but has come to maturity for adoption. S3 Rest API is our friend, regardless of whether the client process is in a container or not.

Update: on Sept 2, 2022, Kubernetes introduced COSI as alpha feature.

MinIO Introduction

In order to use S3 protocol without using Amazon S3 storage, we can use MinIO to build our own object storage service serve client via a S3-compatible REST API interface. The main developer of the MinIO project is MinIO Inc, a startup from 2014. Having learned the lessons from GlusterFS, the founders and developers make MinIO very simple. MinIO operates in two modes: gateway mode (soon to be legacy) and server mode.

In the Gateway mode, MinIO as a gateway between client and destination storage, and does not persist data to itself. In the past, the destination storage can be Azure Blob and Google Cloud Storage (GCS) and HDFS as backend. However, these supports are deprecated now. The current release (July 2022) only supports S3 and NAS backend. According to MinIO’s blog post from February 2022, the entire MinIO Gateway feature will be removed in August, leaving server mode the only option for MinIO.

In the Server mode, the MinIO service will persist data to itself in a file system (or volume). You can specify that file system (or volume) as you launch the server. As one of the quick-start guides shows, we can host MinIO server using a single executable. For administrative tasks, MinIO has a web console and a client utility called mc.

MinIO Deployment Options

For storage service, there are a number of deployment options:

SNSD (single-node, single-drive): single MinIO server with a single storage volume or folder.
SNMD (signle-node, multi-drive): single MinIO server with four or more storage volumes.
MNMD (multi-node, multi-drive, aka distributed): multiple MinIO servers with at least four drives across all servers. This should be considered for production grade configuration.

The deployment options above describes the node and volume topology. No matter which topology option, there are also a number of ways to host the MinIO service process: on Linux OS, Windows OS, MacOS, Docker Container, and on Kubernetes platform.

In addition, MinIO Inc ships the software under different business models. For example, there are fully managed applications in Azure Marketplace, AWS Marketplace, and GCP Marketplace all hosted on virtual machines with extra charges. Clients not willing to pay can host MinIO storage all on their own, either on virtual machines, or on managed Kubernetes environment provided by each cloud provider.

MinIO Hosting solutions

MinIO lists these hosting solutions under multi-cloud products. These hosting solutions (or “products” in MinIO’s term) vary in terms of where peripheral services and data tiers are hosted. Here is the list of the supported platforms:

To illustrate how these solutions are different, I put some details on a few options together for an incomplete comparison below:

	Kubernetes	EKS	AKS	GKE
Hot Storage	Direct PV (NVMe)	EKS EBS CSI	Azure CSI	GKE Standard SSD
Warm Storage	Direct PV (HDD)	S3 IA	Azure BlobStore	GCS
Cold Storage	Public Cloud storage	Glacier	Azure Cool Blob	GCS for Data Archiving
Encryption	HashiCorp Vault	KMS	Azure Key Vault	Cloud Key Management
Observability	Elastic Stack and Grafana	Managed ElasticSearch Prometheus	Azure Monitor	Stack Driver
Identity Provider	KeyCloak	LDAP, SSO	Azure Active Directory	GCP Cloud Identity
LB and Cert Mgmt	Nginx, Let’s Entrypt	AWS Cert Mgr, ELB	Azure Load Balancer, JetStack, Let’s Encrypt	GCP Cloud LB and Managed Cert

Note that all of these hosting solutions are based on some flavour of Kubernetes. The hot tier is usually based on storage options available to the platform. MinIO service access this hot tier via Kubernetes persistent volume. The warm and cold tiers are backed by different object storage service. Between MinIO and storage client, it always use the same S3 compatible Rest API.

MinIO also has tiering capability. While the hot storage destination has to be either a file system or Kubernetes persistent volume, remote tiers can be S3 , Azure Blob, or GCS. MinIO supports encryption at rest (SSE-KMS, SSE-S3, SSE-C) and in transit (TLS) for security, as well as many other useful features such as object replication, versioning, locking, events, Prometheus metrics, lifecycle management etc.

Connect to MinIO server with S3 client

To validate that the client is compatible, we use MinIO’s client utility (mc) to connect to an AWS S3 bucket. Then we use AWS CLI to connect to a MinIO server, similar to this instruction. To do so, we first install client and server utilities:

brew install minio/stable/minio
brew install minio/stable/mc
minio --version
mc --version

Then, we start MinIO server and store an object using AWS CLI’s S3 tool. In our working directory, we create a new directory called minio_data and launch MinIO server with it:

mkdir minio_data
minio server minio_data --console-address :9090

Once the server is up, the screen should display the details, including the portal URL and the default username and password will be used as Access Key ID and Secret Key:

Note that the MinIO service does NOT have TLS enabled by default, on the console or API service. At this point, we can browse to the console web page using the given credential. Then, we can configure AWS CLI with a new profile just to act as a client to communicate with the MinIO server:

$ aws configure --profile minio-cli
AWS Access Key ID [None]: minioadmin
AWS Secret Access Key [None]: minioadmin
Default region name [None]: us-east-1
Default output format [None]: json
$ aws configure set default.s3.signature_version s3v4 --profile minio-cli

At this point, the AWS CLI is configured to communicate with MinIO server. Then, we can create bucket, list object in the bucket, copy an object to the bucket, etc

$ aws --endpoint-url http://127.0.0.1:9000 s3 ls --profile minio-cli # list all bucket, should return empty
$ aws --endpoint-url http://127.0.0.1:9000 s3 mb s3://hehebucket --profile minio-cli # create new bucket
make_bucket: hehebucket
$ aws --endpoint-url http://127.0.0.1:9000 s3 cp README.md s3://hehebucket --profile minio-cli # copy a file to bucket as a new object
upload: ./README.md to s3://hehebucket/README.md
$ aws --endpoint-url http://127.0.0.1:9000 s3 ls s3://hehebucket --profile minio-cli # list objects in the bucket
2022-07-09 00:30:23 631 README.md

The created bucket and object are also visible in MinIO web console, under “Bucket”:

The steps above validate that AWS CLI can talk to MinIO server. Because of that, MinIO server can emulate an S3 service in any development environment so users do not always have to use S3 from AWS. This makes sense for both cost and security reasons for the organization.

Connect to S3 with MinIO client

In this lab, we create an S3 bucket and use mc utility to store an object to it. In order to consistently create S3 bucket and associated permissions, I use the CloudFormation template in this repo. The output of the CloudFormation stack returns the Access Key ID and Secret Key required for the client to access the bucket. Once we cloned the repo, let’s enter the obj-store-helper directory, and run aws cli command to launch the CloudFormation template, assuming it has been configured:

BUCKET_NAME=c0sas2dsadigihunch
S3_STACK_NAME=$BUCKET_NAME-stack
aws cloudformation create-stack --template-body file://aws-s3-stack.yaml --stack-name $S3_STACK_NAME --parameters ParameterKey=S3BucketName,ParameterValue=$BUCKET_NAME --capabilities CAPABILITY_IAM

# to delete stack after test, run: aws cloudformation delete-stack --stack-name $S3_STACK_NAME

In the AWS console, we should see the configuration information as below:

Supposed the bucket name is vna-tst-c0sas2dsadigihunch as shown above, this allows us to configure the client utility MC as below:

mc alias set awss3 https://s3.amazonaws.com # Fill in access key ID and Secret key at the prompt
mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, should return empty
mc cp README.md awss3/vna-tst-c0sas2dsadigihunch/README.md # upload and object to bucket
mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, the uploaded object should be there
mc rm awss3/vna-tst-c0sas2dsadigihunch/README.md # delete the object
mc alias remove awss3 # remove awss3 alias

Once we emptied the bucket, we can delete the CloudFormation stack. This test only needs client utility mc to verify that MinIO client is able to talk to AWS S3 server.

Erasure Coding

For scalable production use, we should deploy MinIO in distributed mode. When MinIO is configured in distributed deployment (MNMD, or multi-node, multi-drive), it implicitly enables an important feature called erasure coding. This erasure coding feature further unlocks a number of other MinIO features:

Erasure coding is MinIO’s data redundancy and availability feature that allows MinIO deployments to automatically reconstruct objects on-the-fly despite the loss of multiple drives or nodes in the cluster. Erasure coding provides object-level handling with less overhead than adjacent technologies such as RAID. The key concept is Erasure Set, a set of drives in a MinIO deployment that supports Erasure Coding. MinIO evenly distributes object data and parity blocks among the drives in the Erasure Set.

Two important variables are M and N: for a given erasure set of size M, MinIO splits objects into N parity blocks, and M-N data blocks. MinIO uses the EC:N notation to refer to the number of parity blocks (N) in the deployment. To determine optimal erasure set size for the cluster, use MinIO’s Erasure Coding Calculator tool.

To help client to specify per-object parity with Erasure Coding, MinIO uses storage classes. Note that the storage class concept in MinIO is distinct from AWS S3 storage class or Kubernetes storage class. In MinIO, a storage class defines parity settings per object. The STANDARD storage class (default) defines EC:N based on M, which can be overridden. In addition, there is REDUCED_REDUNDANCY storage class, whose parity must be less than or equal to that of STANDARD storage class.

The erasure coded backend also protects the storage against Bit Rot with HighwayHash algorithm.

More Features

Authentication and authorization between MinIO client and MinIO server have a number of options. MinIO client may use the built-in standalone identity management in MinIO server. This is the default mode. In addition, one may delegate IAM to external service. To Active Directory via LDAP, or any Identity provider that supports OIDC (JWT with Authorization Code Flow).

As to Object Lifecycle Management (OLM), MinIO allows you to define a remote tier storage for each local target (bucket). The remote tier can be Amazon S3, Google Cloud Storage or Azure Blob storage. We can use mc utility to administer the remote tier and OLM. Configuration steps (e.g. Azure Blob, AWS S3) usually include:

Configure required permissions on the MinIO bucket, create user account for OLM activities.
Configure the Remote Storage Tier
Create and Apply an ILM Transition Rule. The rule can be expressed in a json document.
Validate the creation of ILM transition rule
Validate the effect of transition rule.

As for encryption, MinIO can support encryption at rest. It can also work with etcd store to store encrypted IAM assets if KMS is configured.

Conclusion

Even though we watch for the progress of COSI initiative, we still use Rest API to access object storage from container, which is no different than from a virtual machine. If we develop an application, then we should make it support S3 protocol, a de-facto standard protocol for object storage. As for the storage backend, if we want to be vendor neutral, the feature-rich MinIO is the best bet. We can use MinIO to build our own Object storage as a service compatible with S3. We can also lifecycle our object to remote object storage tier backed by Azure, GCP or S3. In this post we validated the S3 compatibility, and discussed some advanced MinIO features.

FSx ONTAP – Enterprise storage on AWS

Sat, 14 May 2022 22:28:00 -0400

Even though object storage has gained a lot of popularity, file storage is still prevalent. AWS has Elastic File System but the performance is insufficient for enterprise workload. The FSx product line has enterprise storage options and on Sept 2, 2021, AWS launched FSx ONTAP.

This post is my impression about FSx ONTAP. As previously discussed, FSx ONTAP is a managed NetApp storage service by AWS. Essentially, AWS installs NetApp arrays in their data centres, so that users can provision ONTAP volumes from AWS console, or using AWS CLI.

FSx ONTAP

ONTAP (or Data OONTAP) has been a very successful operating system to manage storage arrays. It was so successful that NetApp uses ONTAP as brandname for their storage arrays. This is similar to Isilon, a name of BSD based operating system to manage storage and later becomes brandname of EMC’s storage products. ONTAP has been in competition with other enterprise storage players such as EMC Isilon, HP 3PAR, etc and AWS landed on ONTAP as their partner for enterprise storage. It appears that NetApp still owns their ONTAP storage technology. AWS operates the data centre and provides capability via the CLI layer and console.

Sometimes, people confuses FSx ONTAP with NetApps offering Cloud volumes ONTAP. The two are fundamentally different. Cloud volumes ONTAP works with Cloud Manager (available as self-hosted or SaaS) as the management UI. It manages volumes provisioned from cloud vendor such as AWS, Azure, etc. Clients often configure these “Cloud volumes” as extension to an existing on-premise ONTAP storage deployment. When their on-prem ONTAP volumes fall short of space, the Cloud Manage is aware of a remotely available, cloud backed volume to move cold data off to.

Client Tool

FSx ONTAP is essentially an ONTAP storage cluster sitting in AWS data centre. Users have the options of using AWS CLI or ONTAP CLI to manage the cluster. Users with storage administrator background are likely to prefer the latter. In my professional service experience, I have taken some iterations to come to best practice to use the right tool to interact with ONTAP resources. In a nut shell, it depends on the level of resource that we are interacting with. I categorize those resources into two classes:

Category	Example	Identification
AWS-level resource	FSx ONTAP file system, Storage Virtual Machine	These resources are from ONTAP but they are identified as AWS resources (with ARN). They are also exposed to AWS SDK and can be managed by AWS CLI.
ONTAP-native resource	Volume, Snapshot policy, Schedule, Snapmirror relationship, Vserver	These resources come from ONTAP and can only be managed using ONTAP CLI. The AWS CLI cannot manage _these resources simply because they are not exposed to AWS SDK. Some type of resource such as Volume, may be managed by AWS CLI with very limited options. So we still prefer ONTAP CLI to manage resources.

The AWS CLI (v 2.2.37) has very limited options when creating volumes. For example, the create-volume documentation states that the OntapVolumeType section of output can display types of RW, DP (data protection), or LS. However, it doesn’t allow users to create a volume other than the default RW type. When we configure SnapMirror destination, we need a volume of DP type. We had to use ONTAP CLI to achieve that.

Our practice works out to be: use AWS CLI to create a file system and storage virtual machine. Then we use ONTAP CLI to create everything else. Even though AWS CLI intend to support volume creation, we prefer ONTAP CLI for full functionality support, and alignment with the ONTAP literature.

Administration Tasks

You should configure most of the administrative tasks with ONTAP CLI. We use the documentation by ONTAP as reference. For example, when a volume runs out of Inode, the AWS CLI reports that there is no space left. We need to increase the inode limit and this is, again, not something that AWS CLI can manage. We’d have to use the volume modify command from ONTAP CLI.

NFS version

The AWS document states that FSx ONTAP supports NFSv3.0, v4.0 and v4.1. However, FSx ONTAP is currently backed by NetApp ONTAP 9.10.0, which partially supports NFSv4.2, with basic protocol and Labled NFS feature. NetApp’s ONTAP Best Practices and Implementation Guide suggests a method for clients to mount as NFSv4.2. In the POC we mount as NFS v4.2 in all of our testings.

Clarity of terminology

The ONTAP storage system has been around for a while and many of its concepts are well known in the storage community. For example, A “Snapshot copy” is a read-only, point-in-time image of a volume. (ref: ONTAP 9 documentation -> ONTAP concepts -> Replication -> Snapshot copies) This concept becomes “Snapshot” in AWS literature. It took us some research to come to realize that “Snapshot” in AWS document, essentially maps to “Snapshot copy” in ONTAP literature.

This creates confusion, because we use ONTAP documentation for operation guidance because we can’t get enough help from AWS documentation. The terminology in AWS documentation should align with ONTAP.

The other example is the difference between “backup” and “snapshot” in AWS documentation. It is my understanding that they both use the same underlying Snapshot technology on the ONTAP side. I’m not exactly sure what their difference is.

ONTAP can create a Snapshot copy nearly instantaneously. However, when taking a snapshot using web console in AWS, it takes up to 10 minutes to update the status. This is confusing because it creates a perception that it takes 10 minutes to complete snapshot.

Cross Region Replication

There is a document page on AWS about using SnapMirror at a very high level. It points to two documents: using NetApp Cloud Manager and ONTAP CLI.

The former is not a viable option as we started natively on FSx ONTAP and do not have NetApp Cloud Manager. As to the latter, we managed to configure cross-region replication with ONTAP CLI following the document and identified some gaps in the documentation. Specifically, it would be helpful if AWS documentation calls out that:

inter-cluster network connectivity is a prerequisite (e.g. via VPC peering, transit gateway)
Port 10000, 11104-11105 must be added to security group for inter cluster communication.
The ONTAP CLI command to validate connectivity between clusters (using the ping command from ONTAP CLI).

With AWS CLI alone it is not possible to configure cross-region replication.

Final words

As someone who lived with enterprise storage for more than a decade, I’m glad to see that cloud vendors brings enterprise storage into their data centre, acknowledging that consumer grade file storage are just insufficient for heavy storage use cases such as medical imaging. FSx ONTAP seems to be in early maturity level. However, since AWS exposes ONTAP CLI access to users, ONTAP professionals are able to leverage its full potential.

Istio Lab – Authentication and Authorization

Sun, 13 Feb 2022 13:21:13 -0400

My previous blog discussed as service mesh what Istio can offer in terms of authentication and authorization capabilities. Istio can authenticate an incoming HTTP request, ensuring the JWT issued has not been tampered somewhere in the middle. The fields in the JWT allows for more flexibilities at the point of authorization. This combination allows Istio to integrate with identity providers that can issue JWT.

We’ve also discussed how JWT works, and pointed out that the two key element of request authentication is the JWT (payload signed with private key) itself as well as the JWK (carrying public key). In this post, we will test it in a lab. To start with this lab, we need a test cluster (e.g. Minikube) with Istio installed.

Preparation

I have not find a native Bash way to produce JWT. We will do that with python packages python_jwt and jwcrypto in Python3. Let’s install the modules and import them in Python environment.

$ python3 -m pip install python_jwt jwcrypto datetime
$ python3
>>> import python_jwt as jwt, jwcrypto.jwk as jwk, datetime

Now we’re in the Python3 shell with needed modules loaded. We can take the following steps to produce the JWT as well as the JWK:

RSAkey = jwk.JWK.generate(kty='RSA', size=2048)
private_key = RSAkey.export_private()
public_key = RSAkey.export_public()
raw_payload = {'iss':'digihunch.com','sub':'DIGIHUNCH','role':'reader','permission':'read'}

## use private key to generate jwt token. HTTP request will bear this token 
jwt_token = jwt.generate_jwt(raw_payload, jwk.JWK.from_json(private_key), 'RS256', datetime.timedelta(minutes=50))
print(jwt_token)

## The JWKS keeps public key and is referenced by Istio RequestAuthentication object
jwks='{"keys":['+public_key+']}'
print(jwks)

# Helpful command to print the key in PEM format:
## RSAkey.export_to_pem(private_key=False)
# Helpful command to verify JWT token:
## header, claims = jwt.verify_jwt(jwt_token, jwk.JWK.from_json(public_key), ['RS256'])

To complete this lab, we need the values of jwt_token and jwks.

Next in the preparation is a local cluster, istio with metallb installed, which is covered in a previous post. We should be able to get the ingress IP address:

$ export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo $INGRESS_HOST

We create our own namespace and use the httpbin application:

$ kubectl create ns web 
$ kubectl label namespace web istio-injection=enabled
$ kubectl apply -n web -f httpbin.yaml
$ kubectl apply -n web -f httpbin-gateway.yaml
$ curl -I -XGET $INGRESS_HOST/headers

The curl command above (without any token) should return HTTP 200 code, indicating that no authorization token is needed to connect to the service. Note that the Pod for httpbin has the label app=httpbin which will be used in the request authentication. Also note that traffic is served over a named port called http in the Service object for http, which will implicitly enable HTTP based conditions for authorization policies we will build later.

Request Authentication

Let’s create a request authentication object with the following manifest:

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
 name: jwt-req-authn
 namespace: web
spec:
 selector:
 matchLabels:
 app: httpbin
 jwtRules:
 - issuer: "digihunch.com"
 jwks: |
 ## jwks output from previous step ##

Replace the last line with the jwks output from the preparation step and store it to jwt-req-authn.yaml. It should look like this:

jwt-req-authn.yaml

Then apply it to the web namespace:

$ kubectl -n web apply -f jwt-req-authn.yaml

Now the request authentication resource is applied to the httpbin workload. We first test it with a random authentication token and it should be denied of 401 error:

$ curl -I -XGET $INGRESS_HOST/headers --header "Authorization: Bearer randomstring"
HTTP/1.1 401 Unauthorized
www-authenticate: Bearer realm="http://192.168.64.16/headers", error="invalid_token"
content-length: 79
content-type: text/plain
date: Sun, 13 Feb 2022 16:13:32 GMT
server: istio-envoy
x-envoy-upstream-service-time: 31

Then we take the jwt_token from the preparation step and present it to the request authentication resource by sending an HTTP request with the appropriate authorization token. It should return an HTTP 200 code this time:

$ curl -I -XGET $INGRESS_HOST/headers --header "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDQ3NzE0NjUsImlhdCI6MTY0NDc2ODQ2NSwiaXNzIjoiZGlnaWh1bmNoLmNvbSIsImp0aSI6Im1faUhla2pNbWRmUmlsWEdCaTFBR3ciLCJuYmYiOjE2NDQ3Njg0NjUsInBlcm1pc3Npb24iOiJyZWFkIiwicm9sZSI6InJlYWRlciIsInN1YiI6IkRJR0lIVU5DSCJ9.sVppwmvDqKvSsVdB05a_mDHymZq7Okvnwu-caTywXQgsUvOA6HfaySp_WXMyTp1HQ4WcTqKE4frZm7QNtrZsPso4bdD_4mEDYTswTCWhblaPy236NJBEH3ilB2BVySBVQKsjyxd94F1KV24SFWiR6lUxk52wKKE3ipBwR79jPizhAu9xxrfJ2Lfi5ypNa_kjBdJi63KCt2Y0eW94Fjq3PZs4ZalHJyaXYSx5Gxyei5f7QdpEOBpvs13mSdi9RqkgVQOjE0V1uBRpMckMyZs-IijknJcSu4fkrjgfNmXsrm__-vlM9UjUl2Jlj0x8bRC8l20IZ6t1ml-GFkwj39JC0g"

HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 13 Feb 2022 16:13:54 GMT
content-type: application/json
content-length: 589
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 17

In the example above, jwtRules can be used with other keys such as jwksUri to reference the jwks by Uri. More fields in JWTRules can be found here.

Now we have tested three curl commands:

Without authentication token at all: http server (istio-envoy) returns 200 code.
With an invalid authentication token: http server (istio-envoy) returns 401 code for error.
with a valid authentication: http server (istio-envoy) returns 200 code.

So we have the capability to validate token, but it is not yet mandatory to present the token. We can change this behaviour by tweaking authorization policy.

Authorization Policy

According to Istio documentation, to restrict access to authenticated requests only, this should be accompanied by an authorization rule. We start with the following policy:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
 name: auth-pol
 namespace: web
spec:
 selector:
 matchLabels:
 app: httpbin
 action: ALLOW
 rules:
 - from:
 - source:
 requestPrincipals: ["*"]

Applying the manifest above to namespace web to it applies to workload httpbin. Then we

$ curl -I -XGET $INGRESS_HOST/headers
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Sun, 13 Feb 2022 16:29:58 GMT
server: istio-envoy
x-envoy-upstream-service-time: 35

The requestPrincipals clause makes it mandatory to present a token. The RequestAuthentication validates the token.

We can beef up the authorization policies by adding claims to the conditions, for example:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
 name: auth-pol
 namespace: web
spec:
 selector:
 matchLabels:
 app: httpbin
 action: ALLOW
 rules:
 - from:
 - source:
 requestPrincipals: ["*"]
 to:
 - operation:
 methods: ["GET"]
 when:
 - key: request.auth.claims[iss]
 values: ["digihunch.com"]
 - key: request.auth.claims[role]
 values: ["reader"]

Both the to and when conditions are for HTTP traffic only, and we must tell Istio to inspect the traffic as HTTP, which is done implicitly with named ports on the Service object. Refer to this common problem from Istio’s documentation. The request principals, if a none wildcard value is specified, will be a SPIFFE format identity, the same one used for peer authentication, as discussed in the previous post.

The when clause above contains two key-value pairs. The first looks for the value of a standard claim (iss), the second for a custom claim (role). In the preparation step, we created the claims with those claims in Python and they will match the condition here.

Should any of the conditions above not match, a 403 (Forbidden) error code will be returned by the workload’s istio-envoy proxy.

Verify mTLS connection

In some cases, we need to audit whether the TLS traffic between workloads actually take place in mTLS. As I was developing my korthweb project, I don’t find a straightforward way of validating TLS. We can validate that mTLS mode on a workload using the following istio CTL command:

$ istioctl x describe pod my-workload-pod -n [namespace]

The output shows Effective PeerAuthentication and Applied PeerAuthentication. This verifies configuration. But how can we ensure that traffic are indeed using mTLS? For most of mTLS traffic, we can use Kiali’s observability feature. In Graph, we need to ensure “Security” is checked in the display drop-down. The pad lock will indicate the traffic is mTLS. To get reliable results, we have to artificially create some live traffic between workloads (e.g. curl from one Pod to another) so Kiali can pick up the update.

Validate mTLS traffic in Kiali

Unfortunately, even artificial traffic does not make Kiali the most reliable way to detect mTLS. There are three other approach to verify TLS traffic.

The first approach is to let Envoy proxy emit TLS related traffic. We can apply the following annotation line to a Pod, to tell its Envoy proxy to emit measurements related to tls_inspector:

sidecar.istio.io/statsInclusionPrefixes: "tls_inspector,listener.0.0.0.0_15006"

The metrics will be exposed to Envoy’s admin port (15000 on istio-proxy) with the path /stats. For example:

$ kubectl -n orthweb exec orthanc-7f4c9b759-lxnrb -c istio-proxy -- curl localhost:15000/stats | grep tls_inspector
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 22194 0 22194 0 0 21.1M 0 --:--:-- --:--:-- --:--:-- 21.1M
tls_inspector.alpn_found: 13
tls_inspector.alpn_not_found: 0
tls_inspector.client_hello_too_large: 0
tls_inspector.connection_closed: 0
tls_inspector.read_error: 0
tls_inspector.sni_found: 13
tls_inspector.sni_not_found: 0
tls_inspector.tls_found: 13
tls_inspector.tls_not_found: 0

We still need some live traffic to bump up the measurement. Comparing between tls_found and tls_not_found is a good way to determine if a Pod is receiving both TLS and plaintext traffic in PERMISSIVE mode.

The second approach to tell mTLS is via the connection_security_policy metric label. It is set to mtutual_tls if the connection is in mTLS. We need the dashboard for Prometheus:

$ istioctl dashboard prometheus

In the prometheus dashboard, click on “Graph” at the top, then search for “istio_tcp_connections_closed_total” for “istio_tcp_connections_opened_total“, the result should include a metrics called connection_security_policy which is labelled as mtutual_tls, as illustrated below:

mTLS metric

For more about collecting and querying metrics from Prometheus, check out Istio’s documentation here and here.

The third approach is to utilize the AUDIT feature of Authorization Policy. When a rule in Authorization Policy has a source with namespace or notNamespace field, it requires the incoming connection to have an SPIFFE identity and use mTLS. We can set the Authorization Policy’s action to AUDIT and use RBAC access logging, the same way we would do to troubleshoot RBAC access issues. From the log we can check RBAC failures due to missing SPIFFE identity.

Public Key Infrastructure (PKI) – Introduction

Thu, 08 Apr 2021 22:07:00 -0400

A public-key infrastructure (PKI) is a set of roles, policies, hardware, software and procedures needed to create, manage, distribute, use, store and revoke digital certificates and manage public-key encryption. The algorithms are based on Publick-key cryptography. The format of the digital certificate is defined in X.509 standard.

Certificate Authority – CA digitally signs and publishes the public key to user. Website requesting certificates start with a key pair. It then converts public key into CSR (certificate signing request), including the identity. Once the identity of requestor is validated, CA will sign the public key of requestor, using its own private key. The output of this is the certificate.

Registration Authority – CA may delegate some roles to registration authority (RA). RA is responsible for accepting requests for certificates and authenticating the entity making the request. However, RAs do not have the signing authority of a CA. Note that Microsoft may have referred to a subordinate CA as an RA, which is incorrect according to X.509 PKI standards.

Validation types

Domain Validation: domain ownerships is usually verified via DNS record.
Organization Validation: the organization name and address are verified and put into the certificate.
Extended Validation: verifies existence and location of the legal entity, as well as domain ownership. EV cannot be issued as a wildcard certificate.

Private Certificate Authorities

You can create private CA and use it to sign certificates. Your user need to manually install and trust your private CA so that all certificates issued from the CA will inherit that trust. For revocation, you will also need to maintain an HTTP server for the certificate revocation list, or an OCSP responder.

Certificate Revocation List (CRL)

SSL certificates include information on how to access a certificate revocation list. Client will download and check this list to make sure the certificate has not been revoked. This mechanism has largely been replaced by OCSP responders.

Online Certificate Status Protocol (OCSP)

The OCSP protocol is a replacement for CRLs, with the benefit of being more real-time and requiring less bandwidth. The general operation is similar: clients are to query to OCSP responder to check if a certificate has been revoked.

Commercial vs non-profit CA

	Commercial (e.g. SSLs.com)	Non-profit (e.g. Let’s Encrypt)
Validation	DV, OV and EV	DV only
Wildcard	Supported	Supported (using DNS-01 challenge via ACME v2)
Cost	Not Free	Free
Expiration	1-3 years	90 days

ACME protocol

Traditionally, there are several command-line utilities such as openssl, cfssl, or keytool (Java) to manage certificate related tasks. The process are mostly manual. The Internet Security Research Group (ISRG) developed the ACME (Automated Certificate management Environment) protocol.

In this protocol, there is a certificate management agent (client) on the given web server. The agent generates a key pair and shares it with the CA at the outset of the validation process. Once validation is finished and the agent is verified as the proven owner of the key pair. It can use its key to digitally sign the CSRs it generates and sends to the CA via HTTPS requests. The CA uses the CSR, along with its associated public key, to issue the certificate and send it back to the agent. The agent downloads and installs it, then notifies the designated contact. The agent can be automated to check in with the CA at given intervals to rotate certificates and keys.

Let’s encrypt adopts ACME protocol by using Boulder on the server side, and the most commonly used client is certbot. Smallstep also introduced ACME support in step CA in 2019.

Lets Encrypt

I have used let’s encrypt several times because it is free and easy to manage with certbot, which can be installed using brew on Mac. Here’s how I quickly generate certificate manually:

DOMAIN=orthwebdemo.digihunch.com
echo $DOMAIN
sudo certbot -d $DOMAIN --manual --preferred-challenges dns certonly
# get ready to change txt record
ls /etc/letsencrypt/live/orthwebdemo.digihunch.com/

The --manual switch starts interactive prompts, which includes configuring TXT record and wait for the update.

Open source implementations

Here is a list of open-source implementations of PKI management:

OpenSSL: classic tool for PKI management. The Mac/BSD implementation and GNU implementation are slightly different.
Keytool: Java’s Key and Certificate Management Tool that supports formats used in Java
Cfssl: introduced by CloudFlare to simplify the PKI management process. On Ubuntu, the apt package name is golang-cfssl
Hashicorp Vault: CA, secret management and encryption.
Boulder: implemented in Go based on ACME protocol. Let’s Encrypt uses Boulder on the server side.
EJBCA: a full-featured, enterprise-grade implementation in Java, managed by Swedish company PrimeKey Solutions AB.

Managed CA as service

AWS Certificate Manager: [Update] as of Sep 2022, the managed CA capability was spun off as a new service called AWS Private Certificate Authority, to distinguish from the certificate management capability.
EJBCA Enterprise, as Azure Market place
Google Cloud Certificate Authority Service API

OAuth 2.0 and OIDC 1 of 2

Sat, 14 Mar 2020 21:10:00 -0400

OAuth 2.0 and OpenID Connect (OIDC 1.0) are different but highly related protocols and they are often confused. When we talk about IAM (identity and access management), we should first distinguish between Authentication (AuthN) and Authorization (AuthZ):

AutheNtication (AuthN, aka Identity Management) is about validating user’s identity by verifying that the user trying to connect is actually who it claims itself to be;
AuthoriZation (AuthZ, aka Access Management) refers to granting or denying access to specific resources based on the requesting user’s identity. It is usually performed after a user is identified through authentication. The most common approach is Role-Based Access Control (RBAC).

In a nutshell, OAuth 2.0 deals with authorization. OIDC is a layer later developed on top of OAuth 2.0, to deal with authentication. This post is greatly influenced by a presentation (1 hour) delivered by Nate Barbettini from Okta, with the slides available here. There is also an abridged and illustrated video (16 min) by Okta available here.

OAuth 2.0

OAuth was originally developed by Twitter and Google in 2006 as an open standard for API authorization. OAuth 2.0 is published in 2012. It allows user to delegate authorization. The original scenario is a user signed up to a new application and allows it to automatically import her Gmail contact. The technical problem to solve is: how can a user (Resource Owner) let an app (Client) to access his contact list stored in Google server (Resource Server)? The proposal is that it redirects user to Google Account page (Authorization Server) for user to log in. Then Google Account issues token to the application (Client) with user’s approval. Note that the user did NOT log in to the application itself with her Google account. From the application’s standpoint, the user had been authenticated already, and was simply importing contact after logging in. The roles involved in OAuth 2.0 are:

Resource: the contact list of the user
Resource owner: the user
Client: the application
Resource Server: contact.google.com
Authorization Server: accounts.google.com

The diagram below illustrates the interactions:

OAuth 2.0 Authorization Code Flow

This page has further details for each step. Note that at step 3 to 5 may seem unnecessary because Auth Server could have send Access Token Grant to Application via User-Agent at step 3, which could have eliminate the need for step 4 and 5. In fact this design is to avoid sending critical information (Access Token Grant) to User-Agent (browser) which is considered in secure. In other words, it avoids front channel (User-agent to auth server) and prefers back channel (Client to Auth server) for security. This is the difference between Authorization code flow and the implicit flow.

With OAuth 2.0, there are a number of flows:

Authorization code (front channel and back channel)
Implicit (front channel only, token returned to user agent directly)
Resource owner password credentials (back channel only)
Client credentials (back channel only)

In Authorization Code Flow, the Application (client) needs a one-time registration with the Auth Server and is given a client ID and client secret, which are sent to Auth Server at step 4 along with Access Token Request, to prove the identity of the client application.

Note that OAuth 2.0 is an inherently insecure protocol since it does not support signature, encryption, channel binding or client verification. The protocol relies entirely on the underlying transport layer security (TLS) to provide confidentiality and integrity.

Also note that throughout the process (Authorization Code Flow as an example), the Client application eventually is granted access to user’s data. However, it does not know anything about the user itself. Neither the authorization code grant, nor the access token grant is obligated to present information about the user itself. Therefore, OAuth 2.0 is designed strictly for permission purpose without the intent to address identity issue. In the flow, the auth server does the authentication (for the purpose of granting access to resources, but none of the authentication. A user logs in to client application as Bob, when he requests to imports contact, he is redirected to account.google.com and there he could put in the credential of Alice and therefore load Alice’s Google contacts into Bob’s App account!

Pseudo-authentication with OAuth 2.0

In many real life OAuth 2.0 implementations, at step 3, the Auth server chooses to include a field about the user’s identity. This makes user’s identity visible to the client, and the client is therefore able to confirm user’s identity in its own code. This also allows client application to use OAuth 2.0 as an authentication method, which is referred to as pseudo-authentication. The access token acts as a kind of “valet key” that the application can include with its request to the auth server, as a proof that it has user’s permission to access the resources (or APIs).

Because the identity provider (auth server) typically (but not always) authenticates the user as part of the process of granting an OAuth access token, it’s tempting to view a successful OAuth access token request as an authentication method itself. However, because OAuth was not designed with this use case in mind, making this assumption can lead to major security flaws.

Nate’s presentation outlined some scenarios where OAuth 2.0 is applied up to 2012, and which ones are misuses:

Simple login – pseudo authentication with OAuth 2.0
Single sign-on across sites – pseudo authentication with OAuth 2.0
Mobile app login – pseudo authentication with OAuth 2.0
Delegated authorization – the only intended use case for OAuth 2.0

To address the authentication issue properly, and in a standard approach, we need OpenID Connect.

OpenID Connect (OIDC)

OpenID Connect is an open standard for authentication, promoted by the non-profit OpenID Foundation. It allows user to be authenticated using a third-party service called identity providers. User may choose to use their preferred OpenID Connect providers to log in to websites that accept the OpenID Connect authentication scheme. For example, a user uses her Facebook to login to an online application.

OpenID Connect is an extension to OAuth 2.0 with a just few additions:

In addition to access token, an ID token is returned by the authorization server;
Userinfo end point is provided in case Id token is not sufficient and more user information is needed;
“openid” is passed as a parameter in the Scope during the initial call to the authorization server;

Therefore OpenID Connect is considered an identity layer on top of OAuth 2.0. Many application supports OpenID Connect such as Apache Nifi. OIDC is comparable with SAML in the sense that both provide SSO feature (federated identity). Here is a comparison table:

	OpenID Connect	SAML
Main Purpose	SSO for consumer/mobile applications	SSO for enterprise applications
Load	Relatively light weight	Heavy weight due to the size of XML messages
Use case	Satisfies both authentication and authorization use cases, often combined with OAuth 2.0	Generally not used for API security
Transport	HTTP GET and HTTP POST	HTTP Redirect (GET) binding, SAML SOAP binding, HTTP POST binding, et

Here are more details about their differences. In general SAML is more common in the enterprise world for SSO and it has been around for a while. When developing new applications for enterprise it is advised to consider OIDC first.

OIDC also has authorization code flow, with the additional fields on top of its counterpart in OAuth 2.0. The authorization server returns both access and ID tokens, wrapped in a data structure named JWT (JSON Web Token). The JWT includes a signature field, allowing the client application to verify it with authorization server’s public key. Nate’s presentation proposes the following flows for each application type:

Web application with server backend: authorization code flow
Native mobile app: authorization code flow with PKCE
Java Script app (SPA) with API backend: implicit flow
Microservices and APIs: client credential flow

In addition, this page from Okta developer has a good summary of how to select flow type (grant) based on each use case.

Summary

OpenID Connect is an authentication protocol for the purpose of validating user’s identity. OAuth 2.0 is an authorization protocol. You should use OAuth 2.0 for granting access to your API, or access to user data in other systems. If you need to log user in, or make your accounts available in other systems, you need OIDC.

The next post about OAuth and OIDC was posted in 2023.

Nginx as a reverse proxy for Nifi web UI and Kibana

Thu, 16 Jan 2020 22:22:51 -0400

Nginx can act as a application neutral proxy. One example is to front Nifi. The nifi default configuration provides an HTTP access point, specified in the following entries in nifi.properties:

nifi.web.http.host=192.168.133.5
nifi.web.http.port=8080

Nifi can provide secure port by commenting out the lines above and provide the followings:

nifi.web.https.host=192.168.133.5
nifi.web.https.port=8083

However, it does require configuring JKS keystore for Java, as well as authentication. Customers with existing AD servers are likely to require authentication via LDAP. While Nifi does support LDAP integration according to its administration guide. The configuration is quite involving. You need to configure the identity provider, as well as authorizes. I have personally spent a couple days on this without much progress. The information in the logging isn’t to the point. Restarting nifi also is a long process, making it painful to troubleshoot.

I then moved to Nginx (open-source) as an alternative and it is quite enlightening. I already knew that the SSL termination in nginx is super easy to configure. This time I learned that the opensource community even has a support for LDAP integration. Here is a diagram of how it works:

This approach is outlined in a blog post on Nginx website. The ldap-auth daemon is implemented in Python can can be wrapped up as a systemd service. Once a client sends a request in https, security layer is terminated in nginx, and an authentication request in http is sent to the ldap-auth daemon, which proxies converts the request into LDAP searches and proxies it over to customer’s Active Directory server, for authentication. Once authenticated, the http request can make to one of the backend container or server which hosts Nifi. Below is an example of how this can be configure on RedHat.

Install python3 and python-ldap

RedHat may have both python2 and python3 pre-installed, python2 being the default. We do not want to change the default because other applications such as yum still depends on python2 as of early 2020.

yum -y install python3
yum -y install gcc python3-devel openldap-devel
pip3 install python-ldap

Once python3 is installed, pip3 will be available and we use that to install python-ldap. This is a module in Python3 that will be used by the script that act as ldap daemon.

Configure ldap-auth daemon as systemd service

In the github project for ldap-auth, download nginx-ldap-auth-daemon.py to local location such as /usr/bin, then we create nginx-ldap-auth.service in /etc/systemd/system/ with the following content.

[Unit]
Description=LDAP authentication helper for Nginx
After=network.target network-online.target
 
[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/var/run
ExecStart=/usr/bin/python3 /usr/bin/nginx-ldap-auth-daemon.py
KillMode=process
KillSignal=SIGINT
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

Then, run the following command to load, start and check nginx-ldap-auth service.

systemctl reload-daemon
systemctl start nginx-ldap-auth
systemctl status nginx-ldap-auth

This service will be up and listening to port 8888 for http traffic.

Configure Nginx

Then configure nginx with the following entries in its default.conf file, typically located in /etc/nginx/conf.d.

upstream nifibackend {
 # default: round robin
 server container1.nifi.digihunch.com:8080;
 server container2.nifi.digihunch.com:8080;
}
proxy_cache_path cache/ keys_zone=auth_cache:10m;
# nifi proxy
server {
 listen 8083ssl;
 include /etc/nginx/ssl/default.conf;
 
 location / {
 auth_request /auth-proxy;
 
 proxy_pass http://nifibackend;
 proxy_set_header Host $host:$server_port;
 proxy_set_header X-ProxyScheme https;
 proxy_set_header X-ProxyHost $1;
 proxy_set_header X-ProxyPort 8083;
 proxy_set_header X-ProxyContextPath /;
 }
 
 location /auth-proxy {
 internal;
 proxy_pass http://127.0.0.1:8888;
 
 proxy_pass_request_body off;
 proxy_set_header Content-Length "";
 proxy_cache auth_cache;
 proxy_cache_valid 200 10m;
 
 proxy_cache_key "$http_authorization$cookie_nginxauth";
 proxy_set_header X-Ldap-URL "ldaps://ldap.digihunch.com:636";
 proxy_set_header X-Ldap-BaseDN "OU=Corporate User Accounts,DC=digihunch,DC=org";
 proxy_set_header X-Ldap-BindDN "CN=Digi Hunch Service Account,OU=Digi,OU=ServiceAccounts,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org";
 proxy_set_header X-Ldap-BindPass "myownpasswordtricks";
 proxy_set_header X-CookieName "nginxauth";
 proxy_set_header Cookie nginxauth=$cookie_nginxauth;
 proxy_set_header X-Ldap_Starttls "true";
 proxy_set_header X-Ldap-Template "(&(sAMAccountName=%(username)s)(objectClass=organizationalPerson)(memberOf=CN=GH_SYSADMIN,OU=GHCO,OU=Groups,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org))";
 }
}

We need the full distinguished name of bind user to get this to work. Once configured properly, and user attempts to connect through a browser, Nginx will pop up a prompt for username and password. The username will be plugged into the X-Ldap-Template for further queries. The same HTTP header also allows you to filter by membership that the user is associated with.

Cryptography Basics 1 of 2

Wed, 10 Jul 2019 20:07:18 -0400

I have been dabbling with OpenSSL commands to achieve what I needed during IT implementation, but I decided to spent some time to overcome the conceptual hurdles around cryptography. In this domain, following other people’s instructions through the project does not produce much learning value when too many concepts cloud around. Let’s take the bull by the horn.

This article is purely conceptual. There are already lots of step-by-step guideline about acquiring a website certificate. The intention is to elucidate the core concepts on IT cryptography, and then connect the dots to form the big picture in cryptography.

First, let’s distinguish three basic concepts: encoding, hashing and encryption. They are in essence all mathematical functions, but one does not need to understand the underlying algorithm in order to understand what they are.

Encoding – Transform data into a format so it is readable by external system. Encoding is about interoperability. It is not about security whatsoever. Example: ASCII, BASE64, UNICODE.

Hashing – Mathematic algorithms to generate digest of content. A digest is usually fixed-size, non-reversible and deterministic. You cannot restore content from its digest (non-reversible, one-way calculation). Two different contents results guarantees different digests (deterministic). Digest as a result of hashing is mostly about data integrity. For example, in file download you can calculate MD5 hash and compare the result against the digest given by the source. Another common use, is to hash all password in database. In that sense, hashing has to do with security. Popular algorithms are MD5 and SHA-256.

Encryption – Mathematic algorithms that only succeeds if correct parameter (key) is provided. The function is deterministic and reversible (two-way). The parameter (key) used for calculation (encryption) and reverse calculation (decryption) can either be the same or different.

Cryptography involves all three concepts, but it mainly addresses issues around encryption. We break down encryption into two categories, symmetric key encryption (aka private key encryption) and asymmetric key encryption (aka public key encryption).

Symmetric Key Encryption: using the same key (shared secret) for encryption and decryption. Both parties need to keep it secret. Popular algorithm is AES.

Asymmetric Key Encryption: involves a pair of public key and private key. Public key is given out to external systems. Private key is kept secret. Popular algorithm is RSA. You can run some experiments on this page, and I’ve made some additional notes (all based on RSA algorithm):

Public Key and Private Key are NOT interchangeable. The size of an RSA private key is usually much larger than its public key;
You can encrypt with either key, and decrypt with the other, so long as you specify the key type at the time of encryption or decryption;
You can generate public key from private key (ssh-keygen -y); but not the other way round;
Using a wrong key to encrypt or decrypt leads to failure, instead of wrong result.

Obviously symmetric key encryption is the original form of encryption and requires both parties to keep the key secret, permanently. This is not realistic in real life between organizations. This challenge leads to the adoption of asymmetric key where the public key can be published, for external party to encrypt outgoing messages, whereas the private key is kept secret within the owner, only to decrypt incoming messages.

Knowing this distinction, we can introduce two concepts that are built on top of asymmetric key encryption.

Digital Signature: if an entity signs a document digitally. The digital signature is the digest of the document (hash of the content) encrypted with the signer’s private key.

Digital Certificate: contains owner’s public key and digital signature of issuer (digest of owner’s public key encrypted by issuer’s private key). Client (e.g. browser with CA’s public key preloaded) should not trust the owner’s public key until it compares its digest against decryption result of digital signature (using CA’s public key).

The key difference here is that a digital certificate involves third party. Digital signature by itself cannot address impersonation, which is addressed by digital certificate issued by third party (certified authority). This requires digital certificate must follow some standard, and the most prevalent one is:

X.509: a standard format for digital certificates. It contains:

a public key
an identity (a hostname, or an organization, or an individual)
a signature (either signed by CA or self-signed)

Since the advent of certificate, there are tons of global organizations that need to manage keys and certificates. When we create a secure connection now, we only need to get a certificate from an intermediate CA, thanks to existing certificate chain. It involves lots of work for large organizations to maintain keys and certificates, which requires:

Public Key Infrastructure (PKI): the IT infrastructure to create, manage, distribute, use, store and revoke digital certificates and public keys.

One of the important technique to manage keys and certificates is the encoding. Keys and certificates are usually wrapped with different encoding formats based on what need to be done on them. Here is a summary of encoding formats

Format	Encoding	what is stored	Possible Suffix
PEM	DER file (binary content) encoded in Base64. Certificate files typically include clear text statement “BEGIN CERTIFICATE” and “END CERTIFICATE”	single certificate, certificate chains or private keys	.pem, .crt, .cer, .key
DER	Binary format in early days; it’s complex and don’t use unless with a specific purpose.	single certificate, certificate chains, or private keys	.der, .cer
PKCS#7	Base64 encoded ASCII file, typically include clear text statement “BEGIN PKCS7” and “END PKCS7”	only certificates or certificate chains; no private keys	.p7b .p7s
PKCS#8	Similar to PEM as base64 encoded format but for storing private key only, can be password protected	Private key	.key
PKCS#12	a binary format, heavily used by Microsoft products	certificate, certificate chains or private keys; public private key pair	.pfx .p12
OpenSSH	used by OpenSSH to store public keys (as specified in RFC4253)		.pub

Note: above are just encoding formats. Just by file extension you cannot tell whether the file is a key or a X.509 certificate. When you are configuring certificates, you may come across the following file extensions as well:

.key this extension can indicate any kind of key, but usually it is a private key (used along with .crt file)

.csr certificate signing request, including a public key and an identity required by CA. CA needs this file to issue a certificate. CSR could be encoded in Base-64 or DER

.cer or .crt a certificate, usually in X.509 v3 (public key + identity + signature), the encoding could be PEM or DER.

.jks -> java key store file type. It can be either a key store (private key along with certificate) or trust store (certificate) for Java application. Refer to the section for Java applications.

In the next article, we will examine some use case involving the concepts introduced in this post.