Decryption on Digi Hunch

Public Key Infrastructure 3 of 3 – PKI Implementation

Sat, 30 Mar 2024 00:13:00 -0400

After the last two post, now we can focus on PKI implementation. The use case is software testing, where we need to create and recycle a lot of short-lived certificates. Typically, we don’t have to create public certificates because testing workload is internal. Also, hosting a public CA is much more involving. In this post we go over some options to host private CAs.

Root CA

I’m assuming we are creating a self-signed root CA for the organization. Large organizations usually keep their root CA offline and store the key material in highly protected configuration such as HSM. That is not something we can easily emulate. Nor are we concerned with the details of protecting root CA. In order to enable configuration of intermediate CAs, all we need for root CA, is to create a self-signed certificate along with the key. We can do this with OpenSSL following the first section of this post. Now we’ll use open-source Step CA. In my opinion, this tool is handier. For example, we can create a root CA this with a single command:

step ca init

This will start a prompt with a few questions. Then it will create a root CA and an intermediate CA. The keys and certificates are stored in ~/.step/ directory.

Note that by default, the root CA certificate has a path length of 1. To make the path length more than 1, we’d have to customize the creation. We can specified the desired path length in a certificate template and create a self-signed certificate with the template:

cat << EOF > root.tpl
{
	"subject": {{ toJson .Subject }},
	"issuer": {{ toJson .Subject }},
	"keyUsage": ["certSign", "crlSign"],
	"basicConstraints": {
		"isCA": true,
		"maxPathLen": 2
	}
}
EOF

step certificate create --kty=RSA --size 4096 --template root.tpl "DigiHunch Root CA" root_ca.crt root_ca.key

The command outputs the certificate and key of our test root CA. With that, next, we’ll create subordinate CAs with a few different tools.

AWS Private CA as intermediate CA

I’ll start with AWS Private CA. It was a spin-off service from AWS Certificate Manager and is fairly simple. However, we need to understand its capacity limit. AWS Private CA documentation has a page on RFC Compliance, which lists what in RFC 5280 are supported and what are not. It performs very basic CA functions. It does not perform domain validation, hence no ACME support. We can create root CA as well but here we’ll create an intermediate CA:

In AWS Console, create a Private CA. Indicate you want to create a subordinate CA. You may specify CRL distribution and OCSP endpoint. However, you’re responsible for the providing CRL and/or hosting OCSP service;
In the Private CA, we need to create CA Certificate. We need to provide CSR to our root CA to sign this CA’s certificate. The root CA can be either another AWS Private CA or external CA. In this case we choose external CA and export the CSR to a file (e.g. dh.csr).
We can use Step CA to sign the request. Note that we have to give reasonable value for expiry date. Because ACM create certificate with 13 month validity period by default, the expiry date must be at least 13 months from the current date. We also need to set path length. In this case, I put 0 so the Private CA can only issue end-entity certificate. The command looks like this:

step certificate sign pca.csr \
 /Users/digihunch/.step/certs/root_ca.crt \
 /Users/digihunch/.step/secrets/root_ca_key \
 --profile intermediate-ca \
 --not-after=2027-01-24T07:20:50.52Z \
 --path-len=0

In the AWS console, we paste the generate certificate content in the as certificate body, and the root CA’s certificate as certificate chain.

If the Private CA serves as parent CA of one more layer of CA, set the path length to 1. This requires the path length to be at least 2.

Once the CA certificate is installed, we have completed creating a private CA. We can reference this CA from ACM (AWS certificate manager) when creating a private certificate. Since there is no validation support, ACM will allow you to claim any domain. Currently it does not support ACME-based certificate automation. However, you can use AWS CLI or any AWS based automation tool to create your private certificate.

Step CA

As a cheaper alternative, we can host private CA using step CA on virtual machines. One of the benefits is ACME support (http-01 challenge). In the init command in this post, we already create the intermediate CA certificate and configuration. In this part of the lab, we’ll have two servers: the CA server (pki.digihunch.internal) and the web server (web.digihunch.internal). Make sure the DNS resolution works for both servers. The architecture looks like this:

The CA server runs the Step CA process acting as ACME server. We deploy the CA in standalone mode (instead of linked or hosted deployment), meaning it’s not connected to any cloud services. We can host the service on port 443 so make sure the process has port binding permission within the operating system, and the firewall (security group) allows traffic via port 443.

The Web Server runs Step CLI acting as ACME client. On the web server, we run Step CLI in standalone mode (instead of webroot). During the challenge-response phase, the CLI will get the required random number from ACME server, and host it the as the response on port 80. Make sure that Step CLI process has port binding permission in the OS, and the firewall (security group) allows traffic via port 80.

Make sure the DNS resolution works for both servers. On the CA server, we fetch the fingerprint (in preparation for setting up Step CLI on web server). Then we add an ACME provisioner, and host the CA server with a single command:

step certificate fingerprint $(step path)/certs/root_ca.crt
step ca provisioner add myacme --type ACME
step-ca $(step path)/config/ca.json

The CA server is listening on port 443 (by default). Now we can test ACME process with any ACME compatible client. Let’s use step CLI on the web server:

step ca bootstrap --ca-url https://pki.digihunch.internal:443 \
 --fingerprint <fingerprintvalue>
step ca certificate web.digihunch.internal acme.crt acme.key \
 --acme https://pki.digihunch.internal/acme/myacme/directory

The bootstrapping step is to establish trust on the CA server. Then the provisioning process should complete automatically:

Here the step CLI command uses standalone mode by default so it is important to ensure reachability (port 80, DNS name) from the CA server when step CLI hosts the response. You can also go with webroot mode, where Step CLI generate the file to the web root directory so the response becomes available. This is helpful when you already run another web hosting process such as Nginx on the web server. This blog post covers the usage of other ACME-compatible tools with Step CA as private CA server.

Another well-known tool is HashiCorp Vault. There is already a good tutorial here and all the OpenSSL command in the tutorial can be replaced with Step commands.

Cert Manager on Kubernetes

The Cert Manager project on Kubernetes makes PKI work simple. In my discussion about self-signed certificate on Kubernetes, I covered how to use Cert Manager to create self-signed certificate. However, a corporate with multiple clusters may need to chain each cluster-wide issuing CA to an intermediate CA outside of the cluster. Let’s look at this architecture as an example of a full PKI implementation:

In this architecture, we extend corporate on-premise root CA to the cloud. The Ops account hosts the AWS Private CA, and shares it out to workload account(s) using Resource Access Manager. In each EKS cluster, the Private CA serves as the cluster-level issuer, which can issue CA certificates across Kubernetes namespaces. In each namespace, there is a Cert Manager issuer responsible for issuing certificates within the namespace. Cert Manager supports many issuers and we’re using the AWS Private CA issuer.

To demonstrate the gist of this architecture, in our lab we’ll create one Kubernetes cluster in the same AWS account as the Private CA, and create a Cert Manager certificate in one namespace (e.g. ingress). To get started, create a private CA with the instruction in this post and ensure that the private CA has path length of 1. Then create an EKS cluster. You may use my CloudKube project to provision this cluster in Terraform or any other ways. We use the IRSA model to grant a service account access to the Private CA. First, we create an IAM policy and reference the ARN of the private CA:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "awspcaissuer",
 "Action": ["acm-pca:DescribeCertificateAuthority", "acm-pca:GetCertificate", "acm-pca:IssueCertificate"],
 "Effect": "Allow",
 "Resource": "arn:aws:acm-pca:<region>:<account_id>:certificate-authority/<resource_id>"
 }
 ]
}

Name this policy ‘PCA-Access’. It is the minimum required permission. Now let’s create a service account, along with an IAM role that uses this policy:

eksctl utils associate-iam-oidc-provider \
 --region $AWS_REGION \
 --cluster $CLUSTER_NAME \
 --approve

eksctl create iamserviceaccount \
 --cluster=$CLUSTER_NAME \
 --namespace=cert-manager \
 --name=aws-pca-sa \
 --role-name EKSCertManagerPrivateCARole-$CLUSTER_NAME \
 --attach-policy-arn=arn:aws:iam::123456789012:policy/PCA-Access \
 --approve

In order to run the command successfully, you need the IAM permission to create IAM role, as well as API access to the cluster. Now we can install both Cert Manager and the Private CA Issuer.

helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --version v1.13.3 \
 --set installCRDs=true \
 --create-namespace

helm install aws-ca awspca/aws-privateca-issuer \
 --namespace cert-manager \
 --version v1.2.7 \
 --set serviceAccount.create=false \
 --set serviceAccount.name=aws-pca-sa

Note that when installing Private CA issuer with helm, specify the service account that we created earlier (aws-pca-sa). We install both to the cert-manager namespace, where we’ll create a ClusterIssuer. Now we can declare the following manifest:

apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
 name: pca-cluster-issuer-rsa
 namespace: cert-manager
spec:
 arn: arn:aws:acm-pca:ca-central-1:383500642091:certificate-authority/7f2d7b38-2508-4492-81f0-b5b85427c99c
 region: ca-central-1
---
apiVersion: v1
kind: Namespace
metadata:
 name: ingress
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: ingress-ca-cert
 namespace: ingress
spec:
 isCA: true
 commonName: ingress-ca
 secretName: ingress-ca-secret
 privateKey:
 algorithm: RSA
 size: 2048
 issuerRef:
 name: pca-cluster-issuer-rsa
 kind: AWSPCAClusterIssuer
 group: awspca.cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: ingress-ca-issuer
 namespace: ingress
spec:
 ca:
 secretName: ingress-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: ingress-cert
 namespace: ingress
spec:
 commonName: web.digihunch.com
 secretName: web-digihunch-secret
 duration: 2160h
 renewBefore: 72h
 subject:
 organizations:
 - digihunch
 dnsNames:
 - web.digihunch.com
 privateKey:
 algorithm: RSA
 size: 2048
 issuerRef:
 name: ingress-ca-issuer
 kind: Issuer
 group: cert-manager.io

In this manifest, we create a CA certificate, along with a CA in the ingress namespace. With that, we create an end-entity certificate. As a result, we should find the certificates ready:

kubectl -n ingress get certificate
NAME READY SECRET AGE
ingress-ca-cert True ingress-ca-secret 24s
ingress-cert True web-digihunch-secret 24s

The certificate can be reference by the workload in the namespace. Because an Issuer can only issuer Certificate in the same namespace, we need a issuer in the namespace where the certificate will live. The other benefit that Cert Manager brings, is the support of ACME challenges, which is an enhancement to AWS Private CA.

Summary

We discussed several approaches in PKI implementation. PKI is a fundamental requirement in a software testing environment today. Having an internal public key infrastructure enables many other use cases too. For example, the team may use SSH user certificate. You can also host a CA with Step. PKI allows an enterprise to configure SSL inspection on their Next Generation Firewall. The IAM Role Anywhere feature on AWS also operates on an organization’s own PKI.

Public Key Infrastructure 1 of 3 – Basics

Tue, 20 Feb 2024 00:05:00 -0400

In 2021, I wrote an intro to Public Key Infrastructure (PKI). Now that I have to host my own certificate authority, I decide to dive a little deeper into PKI in this series of posts. In software testing scenario, we need to issue (and recycle) a lot of certificates, and manage their lifecycle events such as (renewal, revocation). As a result, the corporate should establish its own private key infrastructure (PKI). This effort includes hosing their own Certificate Authority.

Concepts

IT professions often use the acronyms PKI and CA interchangeably in the context of implementation. However, strictly speaking, PKI is the entire framework (strategy, policy, etc) around managing certificate at corporate level, and CA is the specific entity that issues certificates. When it comes to architecting PKI, a key design decision is the hierarchy of CAs. Microsoft has a good page explaining the common options.

In a single-tier CA hierarchy, the root CA is also issuing CA. The root CA as the anchor of trust of the PKI also issues certificates to the end entities. Obviously, this does not scale. It is only for sandbox testing, and not suitable for any environment that needs to scale. It also carries significant risk because the private key at the anchor of trust has to remain online.

With a two-tier CA hierarchy, the root CA only issues certificates to its subordinate CAs. Therefore the root CA can go offline. As a result, the subordinate CAs issue certificates to end entities. Since the root CA can remain offline (except for issuing certificates for new CAs), the chance compromise is reduced significantly. This option also scales better with multiple issuing CAs. If one of them gets compromised, then only the end entities of the compromised CA are impacted. The root CA still needs to be online from time to time. Apart from granting new subordinate CA, the team may also need root CA online to sign CRLs, which is a management overhead.

The two-tier hierarchy introduces another problem. The root CA also needs to restrict the certificates that its subordinate CAs can issue. For example, one issuing CA can only issue certificates in the *.dev.digihunch.com domain, and another CA can only issue *.ops.digihunch.com. No subordinate CA shall issue “rogue” certificate beyond their authorized scope. In order to enforce issue boundaries on issuing CAs, RFC 5280 defines multiple ways to express constraints, such as basic constraints (including path length), name constraints, policy constraints, and EKU.

In a three-tier CA hierarchy, the top-level is still a root CA that stays offline. One level below, is a layer of CAs that also stay offline and we refer to them as intermediate CAs. Going down one more level down, there are the issuing CAs for end-entity certificates. Oftentimes, we use the intermediate CAs as policy CAs where we introduce restrictions to the subordinates. In this model, the root CA can remain offline nearly all the time because we can issue CRLs at the intermediate level. This is more flexible, but also more management overhead.

Implementation Options

The two-tier hierarchy is good in most scenarios but the three-tier hierarchy is also common for large organizations. The next consideration is implementation strategy. Take two-tier model for example. We can think about these options:

Implement a self-managed PKI, with an internal root CA. The internal subordinate CAs are the issuing CAs. They are chained to the internal root CA.
Implement a self-managed PKI, with its certificate purchased from a commercial CA. The internal subordinate CAs are the issuing CAs. They are chained to the external root CA, as the TPP (trusted third party). The external CA may or may not be a public root CA.
Purchase certificates from a commercial CA that are chained to a public root CA

Option 3 essentially delegates the PKI to a commercial provider. This is usually not a favourable option due to the hefty charge and minimal control. In option 2, when the third party is a public root CA, you can have all your certificates with public trust, although this is a pricey option too. Even if the third-party isn’t a public root CA, there is still a benefit of delegating the management of Root CA to commercial provider.

Option 1 is for use cases where we need a lot of certificates quick, for example, in agile development iterations. The certificates do not need trust beyond the organization, and the risk of root CA being compromised is manageable. In this option all CAs are private CAs. Option 2 on the other hand, is flexible in terms of trust boundary. You can just rely on the commercial provider as a TTP. If the TPP supports public root CA, you can issue certificates for public facing workload. In other words, you can choose either a private CA or a public CA. When we work with a private CA, we have to import the certificate to the trust stores of the organization.

Some PKI topologies splits a CA into two sub-components: the Certificate Authority Service (CAS, or simply CA) and the Registration Authority (RA), as the diagram above shows. With many locations, each location has an RA that communicates with a with a central CAS. The RAs receive requests from local clients and are responsible for authenticating these requests and pass the validated ones along to the CAS. The validation can take place in ACME protocol. The CAS implicitly trusts RAs and will sign the validated requests from RAs before sending them via API calls. The RA then relays the certificates back to the local requestors. This topology is more scalable with the CAS focusing only on signing and the RAs on validation and passing the requests along. Large organizations may have their central CAS hosted on-prem, and remote RAs in the CSP regions.

Components for X.509 Certificate

X.509 is the standard for digital certificate. The X.509 standard has the following most important fields:

Subject: The name of the subject (e.g. a user, service, device), commonly represented as X.500 formate distinguished name (DN). For website, the value can be CN=digihunch.com
Serial Number: A unique identifier for each certificate that a CA issues.
Issuer: DN of the CA. For a self-signed root CA, the issuer is the subject. They are different otherwise, such as in subordinate CA certificates and end entity certificates.

These fields were introduced in version 1. In addition, X.509 version 3 certificates introduces extensions that provide additional functionality and features to the certificate. Each extension comes in two flavours: critical and non-critical. A certificate-using system MUST reject the certificate if it encounters a critical extension it does not recognize or a critical extension that contains information that it cannot process. A non-critical extension MAY be ignored if it is not recognized, but MUST be processed if it is recognized.

Here are some common extensions that a lot of implementations use:

Subject Alternative Name (SAN): only used in end entity certificates, not in CA certificates. The format of SAN is flexible and it does not have to be X.500 DN. For website certificate, we often place alternative DNS names here.
Key Usage: The intended scope of usage for a private key is specified through the Key Usage and Extended Key Usage (EKU) extensions in the associated certificate. Example: “Certificate Sign, CRL Sign”
Basic Constraints: Used to distinguish between end-entity cert and CA cert. You should also specify path length. The value can be: “CA:TRUE, pathlen:1”. If the path length constraint is 0, the CA may have one more level of subordinate CA. But these subordinate CAs must have path length of 0 on their own certs, and cannot extend one more level as their own subordinates. Those subordinate CAs can only issue end-entity certificates. If the path length value is none, then there is no restriction on the levels of subordinate CAs.
Name Constraints: for CA certs only, defined in RFC5280, to limit the scope to certain names on the certificate that the CA issues. Client must verify that a certificate is allowed to be signed by CA.
CDP (CRL distribution points): URL(s) where the application or service can retrieve the certificate revocation list (CRL).
AIA (Authority Information Access): URL(s) where the application or service can retrieve the revocation list for CA’s certificate
SKI (Subject Key Identifier): the SHA-1 hash of the subject’s public key
AKI (Authority Key Identifier): the SHA-1 hash of the issuer’s public key
EKU (Extended Key Usage, also Enhanced Key Usage): an object identifier (OID) for each application or service a certificate can be used for. It needs to align with Key Usage
Certificate Policy: reference to the certification practice statement (CPS) of the issuer. During exchange, any relying party can access the assurance level associated with the certificate, and decide on the level of trust to put in the certificate.
Policy Constraints: for path validation, it can be used to prohibit policy mapping or to require that each certificate in a path contain an acceptable policy identifier.
Policy Mappings: in CA certificates to restrict the certificates that the CA can issue.

This isn’t a complete list. Plus, X.509 V3 also supports custom extensions. Here is a sample certificate. In real life, the way each implementation uses these extensions may vary slightly, so interoperability issues between PKIs exist. When designing your own PKI, it is important to examine the usage of extensions to comply with the standard. For example, Microsoft has the following recommendations to restrict certificates:

For subordinate CA certificates, the Basic Constraints extension should be present and marked as critical
The cA field should be set to TRUE
The pathLenConstraint field should be set to the minimum value required to enable the business scenario (i.e. 0 if that CA will issue certificates only to End Entities)
The EKU extension should be present and contain the minimum set of EKU object identifiers (OIDs) to enable the business scenario. Furthermore, the anyExtendedKeyUsage OID (2.5.29.37.0) should not be specified.

Summary

This post discussed the basic concepts in public key infrastructure. In the next post, I’ll cover the automation of certificate issuance.

DICOM testing over TLS

Sat, 18 Feb 2023 01:05:00 -0400

I have two open-source projects to deploy a medical imaging application on different platforms. In both of them, I define DICOM validation scenario, and provide steps to test DICOM traffic with TLS. The steps have been working well, until a recent change in Envoy broke the testing, and led me to revisit the test scenario.

In a nutshell, I changed from dcm4che to dcmtk binary builds. I’ve also expanded the test case from a self-signed server certificate to one involving a self-signed CA. Although this test is about TLS for DICOM traffic, the principles apply to any traffic at TCP level. If you just need instruction for DICOM validation on Orthweb or Korthweb projects, skip the Background section.

Background

To test DICOM traffic with command line tool, I was investigating between dcm4che and dcmtk. Both are open-source projects with builds for multiple platforms. My DICOM test is as simple as a C-Echo command and a C-Store command using the tool, with TLS enabled. Once they work, other DICOM commands usually work as well.

I have been primarily using dcm4che as I was familiar with its previous version from my old job. For example, I can issue a C-ECHO with TLS using storescu command:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123!

To turn that test into a C-Store test, simply add a DCM file as input:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! MY.DCM

The C-Store output tracks each DIMSE command and return codes. Note that in the command, we specify –tls12 as the version, and with –tls-aes switch we enabled AES or 3DES encryption. We also specified a file for trust store and password to the trust store.

This is when I first frowned over dcm4che. The dcm4che utility is a Java-based program we have to take an extra step of turning certificate into Java trust store. Different JVM versions may also cause different behaviours in the test. What later prompted me to switch to dcmtk is that with dcm4che I came across a weird error since Envoy proxy version 1.23, which impacted both Orthweb (Envoy proxy) and Korthweb (Istio Ingress).

Moreover, dcmtk is available as a HomeBrew package, Ubuntu package, and Debian package.

The test case, data and the tool

We can install dcmtk utility simply with brew install dcmtk, and then we need its echoscu and storescu commands with correct TLS options. The DICOM data I used for testing is a CT exam available for download here.

Before mocking with dcmtk‘s TLS options, we first need to understand what would be a good test. Previously I have used a single self-signed certificate on server. It is an over-simplified scenario that is far from a real-life certificate chain, and also does not test client certificate. If I also self-sign the client certificate, the client and server certificates are signed by entirely different parties and have no trust relationship, making it an invalid test case for client certificate.

For a full-blown testing with TLS, we should have two levels of CA as below:

The chart represents a typical hierarchy of three-level certificate authorities. Sometimes we need simplicity in our testing, and it is reasonable to simplify the diagram to the following, with one CA that issues certificate for both client and server:

Our DICOM validation testing will be based on both approaches depending on the project and deployment option. For Orthweb project and the Helm-chart driven option in Korthweb, we have one level of CA. For the GitOps and manual option in Korthweb, we have two levels of CA. When configuring testing, it is important to have the diagram above in mind.

In addition, we should also be aware of the limitation of the testing. As a personal project I will not pay for the certificates. I have to self-sign the certificate of the CA so there is no way to derive trust on this CA from another level. As a result, we must tell the client and server to trust the CA. The steps to create the needed certificates are:

Generate a key pair for Test CA. Generate the certificate for Test CA by self-signing its own public key
Generate a key pair for the (DICOM) server. Generate the certificate for the server by signing its public key with Test CA’s private key
Generate a key pair for the (DICOM) client. Generate the certificate for the client by signing its public key with Test CA’s private key

Because the client now also has its certificate, we can test with and without client certificate, using the following dcmtck switches:

-d (shorthand for –debug): print out detailed DICOM communication log. For succinct output, use -v (shorthand for –verbose) instead.
+tla (shorthand for –anonymous-tls): enable anonymous TLS (without client certificate)
+tls (shorthand for –enable-tls): enable full TLS (with client certificate), followed by client key and certificate files
-rc (shorthand for –require-peer-cert): –require-peer-cert, require peer (server) certificate
+cf (shorhand for –add-cert-file): –add-cert-file, add server certificate so client can trust it.

For C-ECHO, the testing commands with and without client certificate look like:

$ echoscu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112
$ echoscu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112

For C-STORE, the testing commands with and without client certificate look like:

$ storescu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm
$ storescu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm

As to how to create the certificates, in the post Creating X.509 TLS certificate in Kubernetes, I discussed different ways to create certificates for testing, including using openssl. In the next two sections, I will discuss them in further details.

Orthweb Test

The orthweb project runs Docker containers on an EC2 instance. In the cloud-init script of the EC2 instance, we self-sign a test CA with openssl. Then we create key and certificate for server and client respectively:

IssuerComName=issuer.orthweb.digihunch.com
ClientComName=dcmclient.orthweb.digihunch.com
ServerComName=ca-central-1.compute.amazonaws.com

openssl11 req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$IssuerComName/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ServerComName/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:$IssuerComName -keyout /tmp/server.key -out /tmp/server.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ClientComName/emailAddress=orthweb@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

When simulating DICOM activities from the client side, we supply the three files (ca.key, client.crt and client.key) to the echoscu and storescu executables, to issue DIMSE commands on top of TLS from client side, as shown in the previous section.

Korthweb Test

Currently, the Korthweb project deploys to Kubernetes cluster in three approaches, including two types of ingress controllers: Istio CRD (manual and GitOps deployment options) and Traefik CRD (Helm Chart driven deployment options). With both approaches, it is fairly straightforward to validate the HTTPS port. We export the CA certificate and run a curl command such as:

curl -HHost:web.orthweb.com -k -X GET https://web.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert ca.crt

Note that by default, the curl command adds SNI (server name indication) extension by default to its TLS ClientHello Message (even without -HHost switch). It acts like a modern browser.

On the ingress controller side, most ingress controllers use SNI to drive request routing (because Host field in payload is encrypted). For example, Traefik Proxy has HostSNI matching rule. With Istio, the document states that: TLS implies the connection will be routed based on the SNI header to the destination. With DICOM traffic the Ingress also expects the client to make use of SNI extension in the TLS ClientHello message. The ingress supports multiple sites so the SNI even has an impact of which TLS certificate the ingress serves to the client. We can use openssl to examine which certificate an ingress serves. For example:

openssl s_client -showcerts -connect dicom.orthweb.com:11112 -servername dicom.orthweb.com < /dev/null
openssl s_client -showcerts -connect dicom.orthweb.com:11112 < /dev/null

The second command without -servername switch constructs an ClientHello message without SNI. The ingress may not have a clue of what certificate to serve, depending on its own implementation of TLS protocol. We can also force TLS version with a switch such as -tls1_2.

When it comes to open-source DICOM client, neither dcm4che or dcmtk puts SNI in the TLS request. This created some limitation with my testing. Luckily I do not have multiple routing destinations for now so I only need to direct all DICOM traffic to a service. When I use Istio ingress, I was able to set hosts to “*” so the Ingress does not care missing SNI extension in the client request. With Traefik proxy, I had to set sniStrict to false, and also forgo client certificate check.

The workaround is different per ingress implementation. Even worse, depending on what the available workaround can achieve, the testing steps vary as well.

For example, I perform DICOM validation (Istio ingress) with the steps below:

# bhs: generate client key pair
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=dcmclient.bhs.orthweb.com/emailAddress=dcmclient@www.digihunch.com -keyout bhs.client.key -out bhs.client.csr

# bhs: export intermediate CA credentials
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.key}' | base64 -d > bhs.int.ca.key
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > bhs.int.ca.crt

# bhs: get intermediate CA to sign client cert 
openssl x509 -req -sha256 -days 365 -in bhs.client.csr -CA bhs.int.ca.crt -CAkey bhs.int.ca.key -set_serial 01 -out bhs.client.crt

# bhs: validate web request (without client certificate)
curl -HHost:web.bhs.orthweb.com -k -X GET https://web.bhs.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert bhs.int.ca.crt

# bhs: validate DICOM c-echo request (with client certificate)
echoscu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112

# bhs: validate DICOM c-store request (with client certificate)
storescu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 DICOM_CT/0001.dcm

On the other hand, for Traefik ingress, I have to use anonymous TLS without client certificate:

echoscu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112
storescu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 DICOM_CT/123.dcm

The complexities with different test paths are consequences of the missing SNI capability in both DICOM toolkits. Unfortunately, the developers of the two DICOM tools are not aware of these consequences. I tried to contact dcmtk about this and will see what happens.

TLS profile

Another setting to pay close attention to is the security profile for TLS communication. These profiles defines the behaviours of dcmtk when it establishes TLS connection. The dcmtk has the following security profiles:

–profile-bcp195-nd (+py default): Non-downgrading BCP 195 TLS Profile
–profile-bcp195 (+px): BCP 195 TLS Profile
–profile-bcp195-ex (+pz): Extended BCP 195 TLS Profile
–profile-aes (+pa): AES TLS Secure Transport Connection Profile (retired)
–profile-null (+pn): Authenticated unencrypted communication (retired, was used in IHE ATNA)

The two at the bottom have been retired. The current profiles are all based on BCP195. BCP (best current practice) are sub-series of the corresponding RFC document series. The current revision of DICOM standard discusses bcp195-nd, bcp195 and bcp195-ex profiles in DICOM standard chapter PS 3.15 (B.9-B.11). For example, bcp195-nd requires that:

Implementation shall not negotiate TLS 1.0 or 1.1
Client and server shall prefer strict TLS configuration (as opposed to startTLS)
Ciphers that should be supported.
Recommend port 2762

These BCP profiles were incorporated into DICOM standard since 2018 and are all based on BCP195. BCP 195 states in section 3.6

3.6. Server Name Indication
TLS implementations MUST support the Server Name Indication (SNI)
extension defined in Section 3 of [RFC6066] for those higher-level
protocols that would benefit from it, including HTTPS. However, the
actual use of SNI in particular circumstances is a matter of local
policy.
Rationale: SNI supports deployment of multiple TLS-protected virtual
servers on a single address, and therefore enables fine-grained
security for these virtual servers, by allowing each one to have its
own certificate.

So technically, missing SNI is considered incompliant.

Summary

I find myself switching between dcmtk and dcm4che back and forth in the past. This time, I spent some time hoping to settle with the better tool this time. The effort is insightful but not fruitful. It is unfortunate to realize that neither supports SNI so I had to compromise the feature of my deployment. Hopefully one of those tools will catch up.

Public Key Infrastructure (PKI) – Introduction

Thu, 08 Apr 2021 22:07:00 -0400

A public-key infrastructure (PKI) is a set of roles, policies, hardware, software and procedures needed to create, manage, distribute, use, store and revoke digital certificates and manage public-key encryption. The algorithms are based on Publick-key cryptography. The format of the digital certificate is defined in X.509 standard.

Certificate Authority – CA digitally signs and publishes the public key to user. Website requesting certificates start with a key pair. It then converts public key into CSR (certificate signing request), including the identity. Once the identity of requestor is validated, CA will sign the public key of requestor, using its own private key. The output of this is the certificate.

Registration Authority – CA may delegate some roles to registration authority (RA). RA is responsible for accepting requests for certificates and authenticating the entity making the request. However, RAs do not have the signing authority of a CA. Note that Microsoft may have referred to a subordinate CA as an RA, which is incorrect according to X.509 PKI standards.

Validation types

Domain Validation: domain ownerships is usually verified via DNS record.
Organization Validation: the organization name and address are verified and put into the certificate.
Extended Validation: verifies existence and location of the legal entity, as well as domain ownership. EV cannot be issued as a wildcard certificate.

Private Certificate Authorities

You can create private CA and use it to sign certificates. Your user need to manually install and trust your private CA so that all certificates issued from the CA will inherit that trust. For revocation, you will also need to maintain an HTTP server for the certificate revocation list, or an OCSP responder.

Certificate Revocation List (CRL)

SSL certificates include information on how to access a certificate revocation list. Client will download and check this list to make sure the certificate has not been revoked. This mechanism has largely been replaced by OCSP responders.

Online Certificate Status Protocol (OCSP)

The OCSP protocol is a replacement for CRLs, with the benefit of being more real-time and requiring less bandwidth. The general operation is similar: clients are to query to OCSP responder to check if a certificate has been revoked.

Commercial vs non-profit CA

	Commercial (e.g. SSLs.com)	Non-profit (e.g. Let’s Encrypt)
Validation	DV, OV and EV	DV only
Wildcard	Supported	Supported (using DNS-01 challenge via ACME v2)
Cost	Not Free	Free
Expiration	1-3 years	90 days

ACME protocol

Traditionally, there are several command-line utilities such as openssl, cfssl, or keytool (Java) to manage certificate related tasks. The process are mostly manual. The Internet Security Research Group (ISRG) developed the ACME (Automated Certificate management Environment) protocol.

In this protocol, there is a certificate management agent (client) on the given web server. The agent generates a key pair and shares it with the CA at the outset of the validation process. Once validation is finished and the agent is verified as the proven owner of the key pair. It can use its key to digitally sign the CSRs it generates and sends to the CA via HTTPS requests. The CA uses the CSR, along with its associated public key, to issue the certificate and send it back to the agent. The agent downloads and installs it, then notifies the designated contact. The agent can be automated to check in with the CA at given intervals to rotate certificates and keys.

Let’s encrypt adopts ACME protocol by using Boulder on the server side, and the most commonly used client is certbot. Smallstep also introduced ACME support in step CA in 2019.

Lets Encrypt

I have used let’s encrypt several times because it is free and easy to manage with certbot, which can be installed using brew on Mac. Here’s how I quickly generate certificate manually:

DOMAIN=orthwebdemo.digihunch.com
echo $DOMAIN
sudo certbot -d $DOMAIN --manual --preferred-challenges dns certonly
# get ready to change txt record
ls /etc/letsencrypt/live/orthwebdemo.digihunch.com/

The --manual switch starts interactive prompts, which includes configuring TXT record and wait for the update.

Open source implementations

Here is a list of open-source implementations of PKI management:

OpenSSL: classic tool for PKI management. The Mac/BSD implementation and GNU implementation are slightly different.
Keytool: Java’s Key and Certificate Management Tool that supports formats used in Java
Cfssl: introduced by CloudFlare to simplify the PKI management process. On Ubuntu, the apt package name is golang-cfssl
Hashicorp Vault: CA, secret management and encryption.
Boulder: implemented in Go based on ACME protocol. Let’s Encrypt uses Boulder on the server side.
EJBCA: a full-featured, enterprise-grade implementation in Java, managed by Swedish company PrimeKey Solutions AB.

Managed CA as service

AWS Certificate Manager: [Update] as of Sep 2022, the managed CA capability was spun off as a new service called AWS Private Certificate Authority, to distinguish from the certificate management capability.
EJBCA Enterprise, as Azure Market place
Google Cloud Certificate Authority Service API