bridge network on Digi Hunch

Test Open ID Connect Flows Locally

Mon, 05 Aug 2024 13:24:34 -0400

Earlier this year, I had to integrate an application with an identity provider. Both claim to be compliant with Open ID Connect. But when they don’t get along, I must find out where it breaks to determine which party isn’t compliant. Therefore, I had to really get to the transaction-level details. As a result, I was eager to find out a way to test Open ID Connect Flows locally. Considering that I discussed OIDC in several past articles and use cases (authenticating to kube-api server, Kubernetes workload, ROSA), I feel it is important to be able to test Open ID Connect flows locally.

Architecture

What makes local testing difficult is that there are several parties involved and they act in different roles. Take Authorization Code Flow as an example, the main actors are:

User Agent: the browser;
Authorization Server: an identity store, also referred to as identity provider;
Resource server: an HTTP server that returns protected resource;
OIDC client application: the component that communicates with identity store on behalf of the resource server;

Apart from the bowser (obviously running on my laptop), we can group the the other actors in different patterns. Here I illustrate some options below:

In essence, we need an OIDC client app and an authorization server. And I find the most simplistic test architecture to be pattern #2 (without reverse proxy) on the client side, and pattern #1 on the server side. In the next section let’s discuss why I prefer this test architecture.

Choice of Tools

The principle of this lab is that we can focus on the OIDC flows itself and simplify every other aspects as much as we can.

The authorization server needs to access my client app. As a result, if I use a public authorization server, such as Azure or Google, I’d have to host my client app on a public IP with domain name as well. This creates churns. To see how much hassle this can involve, review my article Istio External Authentication lab. Instead, I need a tool to host the authorization server on my laptop. And yes, it’s KeyCloak. It is a well-renewed open-source project for identity and access management. It is also the upstream project of RedHat SSO. Another reason is it operates on PostgreSQL database which is a common relational database technology with release in Docker images.

On the client side, we can have our web service with built-in OIDC capability, which usually requires development work in a language with OIDC library. Alternatively, if the client app lacks such capability, we would address this capability in a different component on the client side. It can either be a standalone, purpose-built proxy, or a generic reverse proxy with OIDC capability. For this, I examined a few options. Nginx and Traefik have OIDC support for a fee in their Enterprise product. Apache has a module mod_auth_oidc for free but it requires building the plug-in on my own for Mac platform. Eventually I landed on the purpose-built option, using the oauth2-proxy open source project. This project can act as both actor #3 (with a minimal HTTP server) and actor #4 (OIDC client). So it also saves me from hosting a separate web server. Also, I have used it in the past in the Istio External Authentication lab.

In reality, each message in the OIDC flows must be TLS encrypted. In local testing though we don’t really care. Similarly we don’t necessary need a reverse proxy if it plays no role in the OIDC flow. Both Keycloak (with PostgreSQL) and oauth2-proxy are released in Docker images. As a result, we are ready to roll with only three tools: the browser on local host, Keycloak and oauth2-proxy in the Docker daemon, which provides great portability.

To get started, let’s have two fictitious domains, web.digihunch.com, and keycloak.digihunch.com. In the Docker compose manifest, I name the services based on their hostname so that they can reference each other from within the container network namespace. To access the service from the host, I force the DNS resolution to localhost in /etc/hosts on my Mac, and make sure to declare the same host port in the port mapping (4180 for dummy web service; and 8080 for Keycloak).

Configuration

I created a Docker compose file as below, to set up my test:

services:
 web.digihunch.com:
 container_name: oauth2-proxy
 image: quay.io/oauth2-proxy/oauth2-proxy:latest
 command:
 - --http-address
 - 0.0.0.0:4180
 environment:
 OAUTH2_PROXY_COOKIE_SECRET: NYZaClZinINKwxNGzEDeFGh64W6tmq1eB6uHQPa4S5o 
 OAUTH2_PROXY_CLIENT_ID: dh-user-client
 OAUTH2_PROXY_CLIENT_SECRET: pmkwBjkVesrj7fw1MY7h5s9e3cmAKXgc 

 OAUTH2_PROXY_PROVIDER: oidc 
 OAUTH2_PROXY_OIDC_ISSUER_URL: http://keycloak.digihunch.com:8080/realms/digihunch-users

 OAUTH2_PROXY_PASS_ACCESS_TOKEN: true 
 OAUTH2_PROXY_EMAIL_DOMAINS: '*' 
 OAUTH2_PROXY_REDIRECT_URL: http://web.digihunch.com:4180/oauth2/callback
 OAUTH2_PROXY_PROVIDER_DISPLAY_NAME: DHCKC

 OAUTH2_PROXY_COOKIE_CSRF_EXPIRE: '5m'
 OAUTH2_PROXY_COOKIE_CSRF_PER_REQUEST: true
 OAUTH2_PROXY_COOKIE_SECURE: false # Needed for HTTP connection

 #OAUTH2_PROXY_UPSTREAMS: file:///var/www/static/#/home/ # serve page at /home path
 OAUTH2_PROXY_UPSTREAMS: static://202 
 volumes:
 - ./config/oauth2-proxy.cfg:/etc/oauth2-proxy.cfg
 # - ./config/www:/var/www/static/
 ports: 
 - 4180:4180
 networks:
 - oidc_network
 restart: unless-stopped
 depends_on:
 - keycloak.digihunch.com
 
 postgres-db:
 image: postgres
 container_name: postgresdb
 restart: always
 shm_size: 128mb
 ports:
 - 5432:5432
 networks:
 - oidc_network
 volumes:
 - ./data/pgdata:/var/lib/postgresql/data
 environment:
 - POSTGRES_USER=master
 - POSTGRES_PASSWORD=masterpass
 - POSTGRES_DB=keycloak

 keycloak.digihunch.com:
 image: quay.io/keycloak/keycloak
 command: start
 environment: # Based on Hostname:v2 https://www.keycloak.org/docs/25.0.0/upgrading/#migrating-to-25-0-0
 KC_HOSTNAME: http://keycloak.digihunch.com:8080
 #KC_HOSTNAME_ADMIN: For simplicity, no separate management URL or port
 KC_HOSTNAME_BACKCHANNEL_DYNAMIC: true
 KC_HTTP_ENABLED: true ## Otherwise HTTPS is the enforced by default.
 KC_HEALTH_ENABLED: true
 KEYCLOAK_ADMIN: admin 
 KEYCLOAK_ADMIN_PASSWORD: kcadminpass
 KC_DB: postgres
 KC_DB_URL: jdbc:postgresql://postgres-db/keycloak
 KC_DB_USERNAME: master
 KC_DB_PASSWORD: masterpass 
 ports:
 - 8080:8080
 networks:
 - oidc_network
 restart: always
 depends_on:
 - postgres-db

networks:
 oidc_network:
 enable_ipv6: false
 driver: bridge

For OAuth2-Proxy, the official image for oauth2-proxy is distroless. So if you have to troubleshoot its container file system, you need to access it via an assistant container:

docker run --rm -it --name debugger --privileged --pid container:oauth2-proxy --network container:oauth2-proxy busybox sh
# to see target container's file system, go to: ls -l /proc/1/root/

Alternatively, we can use the one that Bitnami releases but be vary of some nuances.

In the Keycloak part, I use environment variables and had to watch out for the recent changes on hostname v2. We should first start up the Keycloak service. From http://keycloak.digihunch.com:8080, we can login using the credential specified in the environment variables, then we can create a client app:

Create a new realm (dropdown-> Create realm) with name digihunch-users
Switch to this realm from the dropdown, create a couple users (e.g. dhadmin@www.digihunch.com), and set password.
Create a group (e.g. myadmin) and join the user to the group
Under the same realm, create a client, with OpenID Connect as type, client ID being dh-user-client. turn on client authentication (without Direct access grants)
Save the client for now and grab the client secret. Note that the OIDC discovery document (http://keycloak.digihunch.com:8080/realms/digihunch-users/.well-known/openid-configuration) should come online.
Update the docker compose file:
- The value for OAUTH2_PROXY_CLIENT_SECRET is from the client secret;
- The value for OAUTH2_PROXY_OIDC_ISSUER_URL should be http://keycloak.digihunch.com:8080/realms/digihunch-users;
- The value for OAUTH2_PROXY_CLIENT_ID is dh-user-client;
Restart all services including the web. Log on to keycloak and go back to the client configuration in the realm. Under settings. Put in valid redirect URIs as http://web.digihunch.com:4180/oauth2/callback and save the client.

Now, let’s start an private browser session, and browse to http://web.digihunch.com:4180/. The browser should redirect you to keycloak’s login page. Once log in is successful, it should redirect you to the static response with 202 code as the manifest configured.

Summary

In this post, I laid out the steps to test the login for OIDC authorization code flow locally as a starting point. For a bullet-proof solution, I recommend taking a look at KeyCloak administration guide. For example, we typically disable the master realm for security.

There is a similar test setup on Otka blog but I simplified all the aspects that I regard as distractors. There are many other flows that can be tested. However, some of the testing still requires client pattern #1 if we need to initiates an activity from the client application.

Host legacy application in Docker 2 of 2

Thu, 22 Oct 2020 17:54:00 -0400

My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months…

Use Case

I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process. When it is up it listens to more than 70 TCP ports for different business services. Here is a simplified list:

Application service	TCP port to bind
Business service A	8030
Business service B	8040
Business service C	8050
……	…

TCP port requirement

The application also communicates with database and search engine on the same server. Since I am building a training environment where multiple instances of our application needs to run on a single server host. All these instances of application share the same underlying database and search engine services. With multiple instances, additional constraints are introduced. For example:

Each instance requires more than 120 configuration files. A small number of them defines what ports the process binds to. The rest of configuration files are the same across all instances.
The OS needs to host 6 processes of the same application all running at the same time;
The OS does not allow multiple processes to bind to a single TCP port (duh!);
It is extremely labourious to change the path for application to read configuration files from. This bad configuration also breaks the upgrade process going forward.

From the statements of constraints, I determine that we need a mechanism to bring running application process into an isolated environment. This is exactly the definition of container and a perfect use case for docker. The following table represents an example of how the multiple instances can be orchestrated.

OS	Container ID	Application Service	container port	published port
Host CentOS	Container 1 (Instance #1)	Business Service A	8030	9301
		Business Service B	8040	9401
		Business Service C	8050	9501
	Container 2 (Instance #2)	Business Service A	8030	9302
		Business Service B	8040	9402
		Business Service C	8050	9502
	Container 3 (Instance #3)	Business Service A	8030	9601
		Business Service B	8040	9602
		Business Service C	8050	9603

This way of orchestration allows the different instances of applications to share as much configuration files as possible, so that each process thinks that they bind to TCP ports (8030, 8040, 8050, etc), by taking advantage of Docker’s ability to map ports for publishing.

Below is an example of the docker compose file:

version: '3.6'
services:
 dapp1:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp1
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9301:8030 # BUSINESS SERVICE A
 - 9401:8040 # BUSINESS SERVICE B
 - 9501:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:cf
 environment:
 - INSTANCE_TAG=dapp1 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp1/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp1/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp2:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp2
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9302:8030 # BUSINESS SERVICE A
 - 9402:8040 # BUSINESS SERVICE B
 - 9502:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d0 
 environment:
 - INSTANCE_TAG=dapp2 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp2/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp2/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp3:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp3
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9601:8030 # BUSINESS SERVICE A
 - 9602:8040 # BUSINESS SERVICE B
 - 9603:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d1 
 environment:
 - INSTANCE_TAG=dapp3 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp3/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp3/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
networks:
 vcnet:
 driver: bridge
 driver_opts:
 com.docker.network.enable_ipv6: "false"

In this compose file, the environment variables are stored in .env file in the same directory and if they are not declared, the default is specified (syntax: ${VAR:-default}).

Helper scripts

The docker commands are fairly long so I had to organize them into several helper scripts. For example:

docker-entrypoint.sh: this script is the ENTRYPOINT script for container. It is responsible for:
- Initialization work that cannot be done in Dockerfile, such as setting environment variable
- Launch the application, including pointing log file to stdout
- Adding host entry for host.docker.internal to /etc/hosts, as a workaround to this issue with Docker on Linux
build_image.sh: this script makes the image build process smoother
- check if image to build already exist, and ask permission to delete the existing image if so;
- build the image with Dockerfile, and create directory structure for Dockerfile to use during COPY instruction
start_dapp_all.sh: this script starts all containers using docker-compose up and also add required iptables rules. We need to edit PREROUTING rules in IP tables to allow traffic between host NIC interface and the docker bridge interface, created each time service is up, as pointed out in previous post.
stop_dapp_all.sh: this script removes the relevant iptables rules and stop all containers using docker-compose. Note that when deleting routing rules by number, start from the highest rule number and work your way down, since each deletion will cause the rules to be re-numbered.

Permission

The container uses a non-root user to run application (e.g. with su dhunch -c “command” from entry point script to run application as dhunch user), because the legacy application uses the same (non-root) user to perform its actions, and it is generally not advised to use root user. To ensure consistency, we need to create the dhunch user in container (in Dockerfile) so it’s uid and gid aligns with those of the host. The file and directory on the host to be access by the process in container also needs to allow dhunch user to read and write. Otherwise, entry point script will fail.

In the docker-compose file, we mount a file or a directory on the host to the container, and specify 😮 if it is read only mount, under volumes. We can alternatively use bind mount (check here for comparison). In either case, we need to keep in mind of the permission – owner alignment. For example, we have the following mount statement under volumes:

– /var/lib/dapp/dcontainer/archive:/var/lib/dapp/dhost/archive

We also need the entire directory hierarchy accessible to dhunch user. To configure this correctly, we need to create the entire directory hierarchy and set proper owner to it. Here is the comparison between the bad configuration and good configuration:

	Dockerfile instruction for container	Permission issue during mount by docker-compose
Bad config	RUN mkdir -p /var/lib/dapp && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was not created until mount time and it is created implicitly with root as owner (since there is no user section in docker-compose, so root as default is used). The application running as dhunch user in container will have permission issue going into dcontainer directory after mount.
Good config	RUN mkdir -p /var/lib/dapp/dcontainer/archive && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was already created with proper permission prior to mount and the main application process running as dhunch user will not have permission issue.

For application process running as dhunch, it also needs to write logs to stdout, so the result can be viewed from outside the container using docker logs command. The docker-entrypoint.sh script makes this happen by:

su dhunch -c "ln -sf /dev/stdout $DHUNCH_LOG_DIR/dhunch.log"

However, this command itself will run into permission issues. To fix, we need to add user dhunch to tty group (e.g. in Dockerfile as it’s needed on every container):

usermod -a -G tty dhunch

For application process to write to a shared volume on host (e.g. NFS), we can either allow access through volume mapping, or for performant access, mount the NFS share directly to container with proper driver.

Java application

For Java applications, only use the needed package (openjdk, openjdk-devel, openjdk-headless) as the Docker image size must be kept as small as possible. The headless package is for non-UI components, the devel package is for development stuff.

It is also worth-noting that the upper limit of heap size (Xmx) should be set based on the reserved memory of container (specified under docker-compose under resource limit and reservation). If heap is larger than container’s available memory, OOM will be triggered and the container will be killed. This article has some good explanation on this.

Host legacy application in Docker 1 of 2

Fri, 04 Sep 2020 16:24:00 -0400

This is my notes from containerizing a legacy application with Docker compose. We have to run multiple instances of our application because we’re unable to secure additional VMs for this single-VM education environment. The application is target of containerization, because it requires mass reconfiguration (around TCP port) to run multiple instances of the application. We want to use the same application configuration file for multiple containers, and map the TCP port to different groups of ports on the host, leveraging port mapping in Docker. On the other hand, the auxiliary services are not being containerized, such as Cassandra database and ElasticSearch because they can be shared for multiple application instances. In other words, we use Docker to isolate processes of the same application.

Prepare environment

The CentOS server needs to have docker-ce (through YUM) as well as docker-compose (direct download). They can be installed this way:

$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
$ sudo yum install docker-ce docker-ce-cli containerd.io

$ curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo systemctl start docker

Our Docker registry is not publicly available. So we need to port the Docker image we need to remote server and load it into the local registry. We first examine the registry locally:

$ curl -XGET https://admin:password@docker.digihunch.com/v2/dhunch/tags/list | python -m json.tool

Once we identify the image, we export it to a tar file:

$ docker save docker.digihunch.com/dhunch > dhunch_image.tar

SCP the file to remote server and load it locally:

$ docker load -i /home/dhunch/dhunch_image.tar
$ docker image ls

We need to distinguish these commands:

docker save: saves an (non-running) image with all layers to file
docker export: saves a running or paused container to file
docker import: import the contents from a tarball to create a filesystem image, most used with docker export
docker load: load an image from a tar archive or STDIN, most used with docker save

Build docker-compose file

I need to cater to the customer environment with a newly create docker-compose file. The customer environment includes specific storage and networking configurations. Docker compose’s official documentation is here. We repeat the following commands for our troubleshooting:

$ docker-compose up -d
$ docker-compose exec -it dhunch1 bash
$ docker container ls

Once we start the container, the status might go unhealthy after it starts. The documentation explains two reasons you’re seeing an unhealthy container:

a single run of the command takes longer than the specified timeout
health check fails; the health check command will retry a number of times before it declares the container as unhealthy.

In our case, It is most likely because it does not pass a built-in health check mechanism. We need to understand where the health check was defined. There are four ways to enable health check:

Dockerfile instruction when building the image
Docker run command
Docker-compose or docker stack yaml file
Docker service

With #1, unfortunately, you can’t reverse engineer an image and view the Dockerfile that were used to built it and review the health check statement. What you can do is check docker events, or inspect the container, and go to the log files as specified under logPath section in the inspection result and look for HealthCheck section. We determined it is the case, then we can disable, or override the built-in healthcheck command from image, with a statement in docker compose.

For network interface, docker compose also allows us to specify MAC address for each container with mac_address keyword (for license key). MAC address generator are available on the internet.

EntryPoint vs CMD

The difference between EntryPoint and CMD is very important when launching container. Some literature also mentions RUN, which is only used when building a new layer of images so it is not relevant here (in the context of launching a container from image). EntryPoint and CMD has similar functionalities both allowing you to specify a command to run. The difference is whether they can be overwritten by command line arguments that user provide to docker-compose or docker run in an ad-hoc manner. As their names suggests, EntryPOINT means what is specified under it must be executed as it launches into the container, regardless of any adhoc commands. On the other hand, CMD is just an entry to save users from typing in a command every time they run docker compose or docker run. Should user prefer a different command, it can be provided as an explicit argument and it will be respected overwriting the pre-defined CMDentry in Dockerfile or command entry in docker-compose.yml.

Both CMD and EntryPoint supports shell and exec forms. More details here.

Choice of Networking

With single-host deployment, the containerized application needs to communicate with other existing, non-containerized service on host, such as database or elastic search. If docker uses host network, the container shares interface with the host and it does not have its own IP address. Host network removes isolation between container and host. This allows container to run the application that was licensed to the host based on MAC address. There is also no port mapping from container to host network. Container simply uses port on host, and is subject to the availability of TCP/UDP port on host.

We will have to use bridge network here. We can force MAC address the app container, and pre-generate license. For container to communicate with a service on host, through bridge network, there are two problems to address:

Container knows the IP of the host (layer-3 connectivity, ping);
Making host service available to container (layer-4 connectivity, telnet);

Docker creates its own interface for bridge network. If it’s an unnamed network, i.e. not explicitly declared under networks section in docker compose, then interface docker0 is used. If it’s a named network, then an interface name starting with br- is used.

The first problem is easier to address, we simply needs to IP address of the host on the interface. We can validate by pinging from container to host. Docker can also use host.docker.internal to reference the host. Unfortunately, this stopped working for linux since 18.09.3.

It is reportedly to be fixed in 20.04 and until it is available, we may add it to manual dns. The following command outputs the entry to add to /etc/hosts in container.

# ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}'

To do this automatically in docker compose, we need some tricks:

Store the Host IP in host environment variable ( use an export command)
Use compose to pass host environment variable to container environment variable
Have the container write its environment variable to /etc/hosts

The compose file will contain a line like this:

services:
 myenv1:
 image: alpine
 command: >
 sh -c "apk update &&
 echo $$HostDNSLine >> /etc/hosts &&
 bash"
 #network_mode: bridge
 environment:
 - HostDNSLine=${HOSTDNSREC}

Note ampersand might be mistakenly displayed as & in the above. Then we run it with the following:

# export HOSTDNSREC=$(echo 1.2.3.4 host.docker.internal) && docker-compose up

The second problem is harder to address because the service on host may not bind to docker’s interface. Some services such as ssh bind to all interfaces on host and you can telnet to port 22 with any IP address the host is associated with. This is however not the case for most other services, such as Cassandra or Elastic Search. They typically only bind to main interface, such as ens192, or eth0, and not to the docker interface. In order to make the service available to container, we either need to bind these services to the docker interface, or use iptables rules as an alternative.

Suppose it is a named network and Docker’s interface name is br-90ae024d5324, and the service on host listens to port 9042, we will need the following two commands from host:

# sysctl -w net.ipv4.conf.br-90ae024d5324.route_localnet=1

# iptables -t nat -A PREROUTING -p tcp -i br-90ae024d5324 --dport 9042 -j DNAT --to-destination 127.0.0.1:9042

Note that docker compose can configure to run sysctl in container but not from host. If there are multiple ports, we can turn this into a shell script:

#!/bin/bash
tcp_port_list="9200 9042 8302 8303 8304 8305 8306"

if_name=$(basename -a /sys/class/net/* | grep ^br- | head -1)
echo enable route localnet on interface $if_name
sysctl -w net.ipv4.conf.$if_name.route_localnet=1
for tcp_port in $tcp_port_list; do
 echo open host tcp port $tcp_port to interface $if_name
 iptables -t nat -A PREROUTING -p tcp -i $if_name --dport $tcp_port -j DNAT --to-destination 127.0.0.1:$tcp_port
done

echo $(ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}')

On the other hand, binding service to multiple interfaces usually require some re-configuration on the service itself. For example, if it is Elastic Search, we need to update [network.host] entry in elasticsearch.yml to include multiple IP addresses. For Cassandra, we need to update rpc_address to 0.0.0.0 or set rpc_interface in cassandra.yml.

Integration with storage

The application in the container need to store files to storage available to host, whether it is an NFS share or a block disk. We can use volume mapping with Docker compose, to map a path in container to a path presented to host as persistent volume. At this step, we might run into permission issues.

By default, containers initializes as root (uid=1) within the container, and the entrypoint script launches application as root. When application writes to persistent volume, files are written as root user. In the legacy non-container setup, we expect the application to write file as dhunch user. Moreover, NFS volume will not allow writing files as root (if the server has root squash configured). To address this, there are two approaches:

launch container as a regular user
launch container as root user, then have the entrypoint script launch application as regular user (dhunch)

For approach 1, we need to tell Docker to launch container as a regular user by specify the uid and gid for container to run application. We can specify the following envrionment variable in the compose yaml:

user: ${CURRENT_UID}

Then we assign the environment variable before running docker-compose:

# export CURRENT_UID=$(id -u dhunch):$(id -g dhunch) &amp;&amp; docker-compose up

This allows container to initialize as the regular user. However, if the entry point script needs to perform activities that requires root permission within the container, it will fail. For example, a regular user in container will not be able to update /etc/hosts;

With approach 2, we do not specify user in docker compose so container initializes as root. Then the entry point script launches application as regular user. For example, use su command before launch Java:

su dhunch -c "
exec java \
 -Xms512M -Xmx8192M \
 -Djava.io.tmpdir=$APP_HOME/var/tmp \
 -server \
 -XX:CompileCommandFile=$APP_HOME/etc/hotspot_compiler \
 -jar $APP_HOME/lib/jar/jruby-complete-*.jar \
 --1.9 \
 $APP_HOME/lib/rubybin/runapp.rb
"

Before doing this, we need to first create user dhunch within container, and the uid and gid must match those of the host. So that when container picks up dhunch user, it converts it to the correct uid.

groupadd -g 1011 dhunch
useradd -m -c 'regular user' -u 1011 -g 1011 dhunch

To further understand how uid and gid work, here are two posts with more information.

This user ownership setup will also work for NFS. To configure NFS, we need some extra client-side configurations in the container, as well as a special volume driver for NFS. Refer to this post.

Virtualization 4 of 4 – Networking

Fri, 21 Aug 2020 21:53:32 -0400

Virtual LAN (VLAN)

Although VLAN emerged before virtualization and is technically not part of virtualization topic. I’d just like to start from here as a refresher. Suppose we have computers from finance department and computers from sales department all connected to a single layer-2 switch. There are at least three problems: 1) too many devices on the same broadcast domain causes traffic congestion; 2) security can be compromised and 3) each department might have several physical locations. We introduce multi-layer switch to address these with two main features: 1) the VLAN feature can map ports to logical networks, so that all hosts are physically connected to a single switch, but logically to their own network (VLAN) 2) the SVI (switch virtual interface) feature allows inter-VLAN routing at layer 3. Such multi-layer switch is sometimes referred to as layer-3 switch.

VLAN is local to a switch and a tag is required in ethernet frame in order to pass VLAN info across switches. This link between switches is called a trunk. IEEE 802.1q (aka dot1q) is the networking standard for VLAN, which standardizes the tagging traffic between switches to tell which traffic belongs to which VLAN. The dot1q trunk (aka dot1q link) provides VLAN IDs fro frames traversing between switches. A trunk can be configured between two switches, or between a switch and a router. Trunking is the process of traversing different VLAN traffic over the trunk. The ports on each switch need to be configured to enable trunking. While Cisco calls such ports trunk port, others call them tagged port. Their function is to add the VLAN tag to ethernet frame. In contrast, regular ports that send and receive frames without VLAN tag are called access port or untagged port. Trunk port carries traffic for multiple VLANS whereas access port carries traffic for a single VLAN. A network device connected to access port has no idea about its VLAN belonging. VLAN creation and management are the responsibility of the switch. Common trunking protocols include VTP (VLAN trunking protocol) and DTP (dynamic trunking protocol)

This video and this video have good explanations on VLAN.

Virtual Extensible LAN (VXLAN)

VXLAN is an overlay protocol. Remember that in the standard TCP/IP stack, you normally encapsulate layer-3 IP datagram into a layer-2 ethernet frame. With the VXLAN encapsulation technique however, layer-2 frames can be encapsulated within layer-4 UDP packet.

VXLAN allows you to stretch layer 2 connection over an intervening layer 3 network. VXLAN tunnel endpoints (VTEPs) are the endpoint device that terminate VXLAN tunnels and it can be either virtual or physical switch ports. It encapsulate VXLAN traffic and de-encapsulate the traffic when it leaves the VXLAN tunnel.

The VXLAN encapsulation includes the followings:

Outer Ethernet Header (source and dest MAC for underlay VTEPs)
Outer IP header (source and dest IP on underlay network)
Outer UDP header (including source and dest ports, 4789 default)
VXLAN Header (including VNI)
Inner Ethernet Frame (with source and dest MAC for overlay interfaces)

The VNI (VXLAN network identifier, aka VNID) included in the VXLAN header is 24-bit long. It is conceptually similiar to VLAN ID in VLAN but only with 12-bit length.

VXLAN Enapsulation

The VXLAN protocol is documented in RFC7348. Its specification was originally created by VMware, Cisco and Arista. As it became more common in network virtualization (with data centre virtualization, and application containerization) several other players joined the list of contributors and they manufacture switches that support VXLAN. This is a section on VXLAN from the document of Huawei Cloud Engine 5800 switch.

Open vSwitch is an example of a software-based virtual network switch that supports VXLAN overlay networks.

In summary, the main benefits of VXLAN over VLAN are:

VXLAN scales up to 16 million logical networks, thanks to the 24-bit length of VNI
VXLAN supports layer 2 adjacency across IP networks. A VM belonging to existing layer 2 domain can be created in different data centre (where more computing resources are available), without being constrained by layer 2 boundaries, or being forced to create geographically stretched layer 2 domains (stretched VLAN).

Virtual Machine Networking

For VM to connect to each other, within or across hosts, we need not only vNIC on VM, but also vSwitch to connect vNICs. A vSwitch (aka bridge) is a logically defined layer-2 device that passes frames between vNICs. On the same host, vNICs are directly connected to vSwitch, which is then connected to the physical NIC. Each vSwtich connects a broadcast domain. When we setup vNIC there are three modes:

Bridged networking: VM connect to outside network using host’s physical NIC, which acts as a bridge between vNIC and outside network. The VM is a full participant in the network as if it were a physical computer on the network. i.e. it obtain IP addressing information from a DHCP server on the outside (physical) network. The VM’s IP address is also visible and directly accessible by other computers on the network. bridge networking is common for servers as VMs.

NAT networking: The VM relies on the host to act as NAT device to make outgoing network connection. The IP address of VM is assigned by virtual DHCP server on host. The guest VMs form a private network and computers on the outside network are external. The host translates private IP address into the host’s IP address on the way out, and listens for returning traffic. Outside network sees traffic from VM guest as if it were from the host. This network mode is common when the VMs are mainly used as a client workstation.

Host-only networking: creates a network that is completely contained within the host computer. The vSwtich is the hub of the private network and the physical NIC on the host is not involved. The VM will not have access to the outside network. This mode is useful when the VMs needs to be isolated from outside network, and only need to communicate with peers on the same host.

The difference between NAT networking and host-only networking is the exposure of VM guest to external network. All of these networking modes are available on VMWare fusion, for example.

Advanced virtualization platform such as vSphere usually support multi-hosting. Multiple host can also be configured to form a distributed vSwitch, such as vSphere Distributed Switch, in addition to standard switches.

Docker Networking

I had a brief on Docker network covering three modes. Out of the three modes, single-host bridge network is the equivalent of host-only networking. MacVLAN driver is similar to bridged networking, in the sense that container may connect to external network, using host NIC as a bridge. However, the external network is still bound by physical location. This is when overlay network comes in handy.

CNM and CNI

In the Docker networking, container needs to map its own port to host, of which the port resource is implemented by IP tables, which limits the scale and performance of the solution. Also, those networking modes do not address the problem of multi-host networking. As multi-host networking became a real need for containers, the industry started looking into different solutions. Container project favour a model where networking is decoupled from the container runtime. This also greatly improves application mobility. In this model, networking is handled by a ‘plugin’ or ‘driver’ that manages the network interface, and how the containers are connected to the network. The plugin also assigns the IP address to the container’s network interfaces. In order for this model to succeed, there needs to be a well-defined interface or API between the container runtime and the network plugins.

Docker, the company behind the Docker container runtime, came up with the Container Network Model (CNM). Around the same time, CoreOS, the company responsible for creating the rkt container runtime, came up with the Container Network Interface (CNI). Kubernetes originally seeks to use CNM for its plugins, but they eventually decided to go with CNI. The primary reason was that CNM was still seen as something designed with Docker container runtime in mind and was hard to decouple from it. After this decision, several other open source project also turned to CNI for their container runtimes.

This article expands further into the difference between CNM and CNI.

Docker network in different modes

Wed, 01 Jul 2020 20:19:00 -0400

Reading notes of “Docker DeepDive”.

Docker networking is backed by libnetwork, which is an implementation of Container Network Model (CNM), an open-source pluggable architecture designed to provide networking to containers. Libnetwork also provides native service discovery and basic container load balancing solution. Docker networking also involves some drivers that extend the CNM model with specific network topology implementation.

Sandbox – an isolated network stack, including Ethernet interfaces, ports, routing tables, and DNS config, usually implemented through Linux namespace.
Endpoints – behave like regular network adapters, and can only be connected to a single network at a time. It connects sandbox to network. Endpoint is implemented in veth pair in Linux.
Networks – software implementation of an 802.1 bridge (aka switch). They group together, and isolate, a collection of endpoints that need to communicate.

Docker company separates network project out from its container project, as a plugin called libnetwork, which is developed in Golang and compliant to CNM. Libnetwork is the official implementation of CNM.

Libnetwork supports the following network modes:

network mode	mechanism	use case
null	no network is provided to containers	quarantined environment for security
bridge	containers communicate with each other through bridge	containers needs to communicate with each other or with host service
host	process in container has access to host network stack and use host port	container needs to use host network stack (e.g. licence by mac address)
container	place containers in a single net namespace so they can communicate with each other as localhost	proxy, kubernetes

Linux veth comes in pairs to connect virtual network devices. For example, connect two net namespaces to allow intercommunication. Linux bridge is a virtual device, to connect two net namespaces.

Dockers ships with several built-in drivers, known as native drivers or local drivers, such as bridge, overlay and macvlan on Linux. There are also 3rd-party network drivers for docker (aka remote drivers).

Host network

In this mode libnetwork will not create network and net namespace for container. Container process shares the network configuration of the host, and therefore uses the ports on host. Other than network sharing, other aspects (e.g. process, file system, hostname, etc) are separated from host.

Bridge networks

This type of network only exist on a single Docker host and can only connect containers that are on the same host. The word bridge refers to 802.1d bridge (layer 2 switch), which is used to connect multiple network interfaces.

Every Docker host gets a default single-host network, called bridge on Linux. This is the network that all new containers will attach to by default.
Docker networks built with the bridge driver on Linux hosts are based on the linux bridge technology that has existed in the Linux kernel for a while. They’re high performance and extremely stable. Linux brctl tool can inspect the linux bridge.

Bridge networks allows container on the same host to communicate with each other. Port mapping allows network connectivity between container and host. Traffic hitting host port will be redirected to container port.

Multi-host overlays

Cross-host networking usually uses an overlay network, which builds a mesh between host and employs a large block of IP addresses within that mesh. A mesh network is a local network topology in which the infrastructure nodes connect directly, dynamically and non-hierarchically to as many other nodes as possible and cooperate with one another to efficiently route data from/to clients.

You can attach a service to overlay network, which spans across multiple Docker hosts so that containers on different hosts can communicate at layer 2. They are much better alternatives than bridge network for container-to-container communication. Overlay networking is very common due to its scalability.

The trick is basically the layer 2 frame of the overlay network is encapsulated into layer 3 datagram transmitted across underlay network, at layer 3. This is achieved through VXLAN tunnels, which allows you to create a virtual Layer 2 network on top of an existing Layer 3 infrastructure. VXLAN is an encapsulation technology that existing routers and network infrastructure just see as regular IP/UDP packets without issue.

To create the virtual Layer 2 overlay network, a VXLAN tunnel is created through the underlying Layer 3 IP infrastructure (aka underlay network). Each end of the VXLAN tunnel is terminated by a VXLAN Tunnel Endpoint (VTEP). It’s this VTEP that performs the encapsulation/de-encapsulation.

VXLAN networking

To accomplish overlay network across multiple hosts, a new network sandbox was created on each host. A sandbox is like a container, but instead of running an application, it runs an isolated network stack – one that’s sandboxed from the network stack of the host itself.

A virtual switch (aka virtual bridge) called Br0 is created inside the sandbox. A VTEP is also created with one end plumbed into the Br0 virtual switch, and the other end plumbed into the host network stack (VTEP). The end in the host network gets an IP address on the underlay network the host is connected to and is bound to a UDP socket on port 4789. The two VTEPs on each host create the overlay via a VXLAN tunnel.

Each container then gets its own virtual Ethernet (veth) adapter that is also plumbed into the local Br0 virtual switch.

Let’s go over an example in the following diagram, where container C1 with an overlay IP needs to communicate to another container C2, with a different overlay IP, sitting on a different node (Docker host). Each node has its own underlay IP.

IP communication details:

C1 creates the IP datagram with destination IP (C2) and sends it over its veth interface, which is connected to the Br0 virtual switch on the host node.
The virtual switch doesn’t know where to send the datagram, as it doesn’t have an entry in its ARP table that corresponds to the destination IP address. As a result, it floods the packet to all ports. The VTEP interface connected to Br0 knows how to forward the frame, so responds with its own MAC address.
This is a proxy APR reply and results in the Br0 switch learning how to forward the packet. So it updates its ARP mapping the destination IP address to the MAC address of the local VTEP.
The VTEP knows about C2 because all newly started containers have their network details propagated to the other nodes in the Swarm using the network’s built-in gossip protocol. When the packet arrives at node2
The VTEP encapsulates the frame so it can be sent over the underlay transport infrastructure, by adding a VXLAN header to the Ethernet frame. The VXLAN header contains the VXLAN network ID (VNID) which is used to map frames from VLANs to VXLANs and vice versa.
Each VLAN gets mapped to VNID, so that the packet can be de-encapsulated on the receiving end and forwarded to the correct VLAN. This is how network isolation is maintained. The encapsulation also wraps the frame in a UDP packet with the IP address of the remote VTEP on node2 in the destination IP field, and the UDP port 4789 socket information. The underlying network does not know that it is transporting data frames for the overlay network.
When the packet arrives at node2, the kernel sees that it’s addressed to UDP port 4789. The kernel also knows that it has a VTEP interface bound to this socket. As a result, it sends the packet to the VTEP, which reads the VNID, de-encapsulates the packet, and sends it on to its own local Br0 switch on the VLAN that corresponds the VNID. From there it is delivered to container C2

Docker also supports Layer 3 routing within the same overlay network. For example, you can create an overlay network with two subnets, and Docker will take care of routing between them. Two subnets will require two virtual switches, Br0 and Br1, being created inside the sandbox, and routing happens by default.

Plugging into existing vLANs

The built-in MACVLAN driver was created for onnect containerized apps to external physical network. A good example is partially containerized app, in which the containerized parts will need a way to communicate with the non-containerized parts still running on existing physical networks.

To connect the container interface through the host interface to an external network, the host NIC needs to be in promiscuous mode. For public cloud, this is most likely prohibited. For data centers, this depends on the network policy.

Docker MACVLAN driver is built on top of Linux kernel driver with the same name. As such, it supports VLAN trunking. This means we can create multiple MACVLAN networks and connect containers on the same Docker host to them.

For connectivity issues between containers, it’s worth checking both the daemon logs (on host) and container logs.

Service discovery

allows all containers and Swarm services to locate each other by name, as long as they are on the same network. This leverages Docker’s embedded DNS server as well as a DNS resolver in each container.
Each Swarm Service and standalone container started with the –name flag will register its name and IP address with the Docker DNS service.
This name resolution, however, only works within the same network.
It is also possible to configure Swarm services and standalone containers with customized DNS options in case embedded Docker DNS server cannot resolve a query (/etc/resolv.conf)

Ingress load balancing

Services published via ingress mode (by default, as opposed to host mode) can be accessed from any node in the Swarm, even nodes not running a service replica. Ingress mode uses a layer 4 routing mesh called the Service Mesh or the Swarm Mode Service Mesh.

Updates:

The most common network modes that I use are host and bridge. With host network mode, container exposes ports on the interface of the host machine. Containers talk to each other via that interface. With bridge network, containers have their own namespace of networking separate from the one from the interface of the hosts, with a bridge getting the two networks connected.

Reference

Docker Deep dive