Public Key Infrastructure 3 of 3 – PKI Implementation

After the last two post, now we can focus on PKI implementation. The use case is software testing, where we need to create and recycle a lot of short-lived certificates. Typically, we don’t have to create public certificates because testing workload is internal. Also, hosting a public CA is much more involving. In this post we go over some options to host private CAs.

Root CA

I’m assuming we are creating a self-signed root CA for the organization. Large organizations usually keep their root CA offline and store the key material in highly protected configuration such as HSM. That is not something we can easily emulate. Nor are we concerned with the details of protecting root CA. In order to enable configuration of intermediate CAs, all we need for root CA, is to create a self-signed certificate along with the key. We can do this with OpenSSL following the first section of this post. Now we’ll use open-source Step CA. In my opinion, this tool is handier. For example, we can create a root CA this with a single command:

step ca init

This will start a prompt with a few questions. Then it will create a root CA and an intermediate CA. The keys and certificates are stored in ~/.step/ directory.

Note that by default, the root CA certificate has a path length of 1. To make the path length more than 1, we’d have to customize the creation. We can specified the desired path length in a certificate template and create a self-signed certificate with the template:

cat << EOF > root.tpl
{
	"subject": {{ toJson .Subject }},
	"issuer": {{ toJson .Subject }},
	"keyUsage": ["certSign", "crlSign"],
	"basicConstraints": {
		"isCA": true,
		"maxPathLen": 2
	}
}
EOF

step certificate create --kty=RSA --size 4096 --template root.tpl "DigiHunch Root CA" root_ca.crt root_ca.key

The command outputs the certificate and key of our test root CA. With that, next, we’ll create subordinate CAs with a few different tools.

AWS Private CA as intermediate CA

I’ll start with AWS Private CA. It was a spin-off service from AWS Certificate Manager and is fairly simple. However, we need to understand its capacity limit. AWS Private CA documentation has a page on RFC Compliance, which lists what in RFC 5280 are supported and what are not. It performs very basic CA functions. It does not perform domain validation, hence no ACME support. We can create root CA as well but here we’ll create an intermediate CA:

  1. In AWS Console, create a Private CA. Indicate you want to create a subordinate CA. You may specify CRL distribution and OCSP endpoint. However, you’re responsible for the providing CRL and/or hosting OCSP service;
  2. In the Private CA, we need to create CA Certificate. We need to provide CSR to our root CA to sign this CA’s certificate. The root CA can be either another AWS Private CA or external CA. In this case we choose external CA and export the CSR to a file (e.g. dh.csr).
  3. We can use Step CA to sign the request. Note that we have to give reasonable value for expiry date. Because ACM create certificate with 13 month validity period by default, the expiry date must be at least 13 months from the current date. We also need to set path length. In this case, I put 0 so the Private CA can only issue end-entity certificate. The command looks like this:
step certificate sign pca.csr \
   /Users/digihunch/.step/certs/root_ca.crt \
   /Users/digihunch/.step/secrets/root_ca_key \
   --profile intermediate-ca \
   --not-after=2027-01-24T07:20:50.52Z \
   --path-len=0
  1. In the AWS console, we paste the generate certificate content in the as certificate body, and the root CA’s certificate as certificate chain.

If the Private CA serves as parent CA of one more layer of CA, set the path length to 1. This requires the path length to be at least 2.

Once the CA certificate is installed, we have completed creating a private CA. We can reference this CA from ACM (AWS certificate manager) when creating a private certificate. Since there is no validation support, ACM will allow you to claim any domain. Currently it does not support ACME-based certificate automation. However, you can use AWS CLI or any AWS based automation tool to create your private certificate.

Step CA

As a cheaper alternative, we can host private CA using step CA on virtual machines. One of the benefits is ACME support (http-01 challenge). In the init command in this post, we already create the intermediate CA certificate and configuration. In this part of the lab, we’ll have two servers: the CA server (pki.digihunch.internal) and the web server (web.digihunch.internal). Make sure the DNS resolution works for both servers. The architecture looks like this:

CA Server
(ACME Server)
CA Server…
Step CA
Step…
pki.digihunch.internal
pki.digihunch.internal
Web Server
(ACME Client)
Web Server…
Step CLI 
Step…
web.digihunch.internal
web.digihunch.internal
ephemeralephemeralTCP 443TCP 80
/challenge/response
/challenge/response
standalone
mode
standalone…
/acme/order/
/acme/order/
Text is not SVG – cannot display

The CA server runs the Step CA process acting as ACME server. We deploy the CA in standalone mode (instead of linked or hosted deployment), meaning it’s not connected to any cloud services. We can host the service on port 443 so make sure the process has port binding permission within the operating system, and the firewall (security group) allows traffic via port 443.

The Web Server runs Step CLI acting as ACME client. On the web server, we run Step CLI in standalone mode (instead of webroot). During the challenge-response phase, the CLI will get the required random number from ACME server, and host it the as the response on port 80. Make sure that Step CLI process has port binding permission in the OS, and the firewall (security group) allows traffic via port 80.

Make sure the DNS resolution works for both servers. On the CA server, we fetch the fingerprint (in preparation for setting up Step CLI on web server). Then we add an ACME provisioner, and host the CA server with a single command:

step certificate fingerprint $(step path)/certs/root_ca.crt
step ca provisioner add myacme --type ACME
step-ca $(step path)/config/ca.json 

The CA server is listening on port 443 (by default). Now we can test ACME process with any ACME compatible client. Let’s use step CLI on the web server:

step ca bootstrap --ca-url https://pki.digihunch.internal:443 \
    --fingerprint <fingerprintvalue>
step ca certificate web.digihunch.internal acme.crt acme.key \
    --acme https://pki.digihunch.internal/acme/myacme/directory

The bootstrapping step is to establish trust on the CA server. Then the provisioning process should complete automatically:

Here the step CLI command uses standalone mode by default so it is important to ensure reachability (port 80, DNS name) from the CA server when step CLI hosts the response. You can also go with webroot mode, where Step CLI generate the file to the web root directory so the response becomes available. This is helpful when you already run another web hosting process such as Nginx on the web server. This blog post covers the usage of other ACME-compatible tools with Step CA as private CA server.

Another well-known tool is HashiCorp Vault. There is already a good tutorial here and all the OpenSSL command in the tutorial can be replaced with Step commands.

Cert Manager on Kubernetes

The Cert Manager project on Kubernetes makes PKI work simple. In my discussion about self-signed certificate on Kubernetes, I covered how to use Cert Manager to create self-signed certificate. However, a corporate with multiple clusters may need to chain each cluster-wide issuing CA to an intermediate CA outside of the cluster. Let’s look at this architecture as an example of a full PKI implementation:

Data Center
Data Center
Root CA
Offline
Root CA…
Root CA
Certificate
Root CA…
self-sign
self-sign
Intermediate CA
on-prem
Inte…
Intermediate CA
on-prem
Inte…
AWS Cloud
AWS Cloud
sign
sign
Ops Account
Ops Account
Intermediate CA
Certificate
Intermedi…
AWS Private CA
Intermediate CA
AWS Pri…
Resource Access Manager
Resou…
Workload Account
Workload Account
Elastic Kubernetes
Service Cluster
Elastic…
ns
namespace
workload1
namesp…
ns
namespace
cert-manager
namesp…
ClusterIssuerWorkload1Issuer
Workload1
Certificate
Workload…
pod
Workload1
Workload1
CA Certificate
CA Certif…
sign
sign
ns
namespace
workload2
namesp…
Workload2Issuer
Workload2
Certificate
Workload…
pod
Workload2
Workload2
CA Certificate
CA Certif…
sign
sign
ns
namespace
ingress
namesp…
IngressIssuer
Ingress
Certificate
Ingress…
pod
Ingress
Pod
Ingress…
CA Certificate
CA Certif…
sign
sign
sign
sign
sign
sign
sign
sign
sa
Service
Account
Service…
Elastic Kubernetes
Service Cluster
Elastic…
ns
namespace
workload1
namesp…
ns
namespace
cert-manager
namesp…
ClusterIssuerWorkload1Issuer
Workload1
Certificate
Workload…
pod
Workload1
Workload1
CA Certificate
CA Certif…
sign
sign
ns
namespace
workload2
namesp…
Workload2Issuer
Workload2
Certificate
Workload…
pod
Workload2
Workload2
CA Certificate
CA Certif…
sign
sign
ns
namespace
ingress
namesp…
IngressIssuer
Ingress
Certificate
Ingress…
pod
Ingress
Pod
Ingress…
CA Certificate
CA Certif…
sign
sign
sign
sign
sign
sign
sign
sign
sa
Service
Account
Service…
Extend Corporate PKI to Cloud for Kubernetes workload
Extend Corporate PKI to Cloud f…
Text is not SVG – cannot display

In this architecture, we extend corporate on-premise root CA to the cloud. The Ops account hosts the AWS Private CA, and shares it out to workload account(s) using Resource Access Manager. In each EKS cluster, the Private CA serves as the cluster-level issuer, which can issue CA certificates across Kubernetes namespaces. In each namespace, there is a Cert Manager issuer responsible for issuing certificates within the namespace. Cert Manager supports many issuers and we’re using the AWS Private CA issuer.

To demonstrate the gist of this architecture, in our lab we’ll create one Kubernetes cluster in the same AWS account as the Private CA, and create a Cert Manager certificate in one namespace (e.g. ingress). To get started, create a private CA with the instruction in this post and ensure that the private CA has path length of 1. Then create an EKS cluster. You may use my CloudKube project to provision this cluster in Terraform or any other ways. We use the IRSA model to grant a service account access to the Private CA. First, we create an IAM policy and reference the ARN of the private CA:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "awspcaissuer",
      "Action": ["acm-pca:DescribeCertificateAuthority", "acm-pca:GetCertificate", "acm-pca:IssueCertificate"],
      "Effect": "Allow",
      "Resource": "arn:aws:acm-pca:<region>:<account_id>:certificate-authority/<resource_id>"
    }
  ]
}

Name this policy ‘PCA-Access’. It is the minimum required permission. Now let’s create a service account, along with an IAM role that uses this policy:

eksctl utils associate-iam-oidc-provider \
    --region $AWS_REGION \
    --cluster $CLUSTER_NAME \
    --approve

eksctl create iamserviceaccount \
  --cluster=$CLUSTER_NAME \
  --namespace=cert-manager \
  --name=aws-pca-sa \
  --role-name EKSCertManagerPrivateCARole-$CLUSTER_NAME \
  --attach-policy-arn=arn:aws:iam::123456789012:policy/PCA-Access \
  --approve

In order to run the command successfully, you need the IAM permission to create IAM role, as well as API access to the cluster. Now we can install both Cert Manager and the Private CA Issuer.

helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
   --namespace cert-manager \
   --version v1.13.3 \
   --set installCRDs=true \
   --create-namespace

helm install aws-ca awspca/aws-privateca-issuer \
   --namespace cert-manager \
   --version v1.2.7 \
   --set serviceAccount.create=false \
   --set serviceAccount.name=aws-pca-sa

Note that when installing Private CA issuer with helm, specify the service account that we created earlier (aws-pca-sa). We install both to the cert-manager namespace, where we’ll create a ClusterIssuer. Now we can declare the following manifest:

apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
  name: pca-cluster-issuer-rsa
  namespace: cert-manager
spec:
  arn: arn:aws:acm-pca:ca-central-1:383500642091:certificate-authority/7f2d7b38-2508-4492-81f0-b5b85427c99c
  region: ca-central-1
---
apiVersion: v1
kind: Namespace
metadata:
  name: ingress
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-ca-cert
  namespace: ingress
spec:
  isCA: true
  commonName: ingress-ca
  secretName: ingress-ca-secret
  privateKey:
    algorithm: RSA
    size: 2048
  issuerRef:
    name: pca-cluster-issuer-rsa
    kind: AWSPCAClusterIssuer
    group: awspca.cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ingress-ca-issuer
  namespace: ingress
spec:
  ca:
    secretName: ingress-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-cert
  namespace: ingress
spec:
  commonName: web.digihunch.com
  secretName: web-digihunch-secret
  duration: 2160h
  renewBefore: 72h
  subject:
    organizations:
      - digihunch
  dnsNames:
    - web.digihunch.com
  privateKey:
    algorithm: RSA
    size: 2048
  issuerRef:
    name: ingress-ca-issuer
    kind: Issuer
    group: cert-manager.io

In this manifest, we create a CA certificate, along with a CA in the ingress namespace. With that, we create an end-entity certificate. As a result, we should find the certificates ready:

kubectl -n ingress get certificate
NAME              READY   SECRET                 AGE
ingress-ca-cert   True    ingress-ca-secret      24s
ingress-cert      True    web-digihunch-secret   24s

The certificate can be reference by the workload in the namespace. Because an Issuer can only issuer Certificate in the same namespace, we need a issuer in the namespace where the certificate will live. The other benefit that Cert Manager brings, is the support of ACME challenges, which is an enhancement to AWS Private CA.

Summary

We discussed several approaches in PKI implementation. PKI is a fundamental requirement in a software testing environment today. Having an internal public key infrastructure enables many other use cases too. For example, the team may use SSH user certificate. You can also host a CA with Step. PKI allows an enterprise to configure SSL inspection on their Next Generation Firewall. The IAM Role Anywhere feature on AWS also operates on an organization’s own PKI.