Field Notes on Digi Hunch

The Leanest Web and Email Hosting

Thu, 27 Feb 2025 22:14:11 -0500

This site has been quiet for a while. During this time, I migrated the hosting platform again, and refactored email solution. This post, is another note about how I finally came to the most cost-effective web and email solution for a small business, with a solid security posture, and at nearly no cost.

For background, I have had this website for more than five years now, and revamped it a couple times. I have been with Amazon Lightsail at an already low cost. I didn’t have a lot of emails. So I have been relying on the email forwarding feature from my old domain registrar. There have been a few challenges. The content has grown a lot. The speed gradually slows down. WordPress needs reinstall. The outgoing emails get flagged as unverified.

WordPress Hosting

Worried about too many plugins, I debated about changing content management platform. Unfortunately, it’s not easy. There are some alternatives but none has all the features from all the plugins that I’ve been using. Even if they do, it would be too much work: I’d have to migrate the code snippets, highlights, etc, find similar themes and verify the new theme and new plugins work together. Despite of the criticism, WordPress today is still a predominant content management platform. So I stick to it.

Staying with WordPress isn’t maintenance-free. When PHP version has major update once in a while, I still had to re-install and migrate to a new server. Since I’ll exercise migration anyways, why not also shop around for a new virtual private server. Surprisingly I learned that Google Cloud Platform has an always-free tier including a decent size virtual machine.

	Amazon Lightsail 7USD/mo	GCP Free tier
Class	micro	e2-micro
CPU	2 vCPUs	2 vCPUs
Memory	1 GB	1 GiB
Disk	40GB SSD	30 GB (standard persistent disk HDD)
Data Transfer	1TB	1GB egress

The vCPU in different platforms do not represent exactly the same compute capacity, but should be generally comparable. The egress data transfer usage was below 1GB on Lightsail so the new service is sufficient. By going with GCP’s free tier, the only downgrade is the disk speed.

The GCP platform provides Bitnami package image and it’s fairly simple to spin up a new VM. The setup is as simple as Amazon Lightsail and I don’t have to deal much with networking. It doesn’t work out to be completely free but the net cost is negligible. I also picked the Nginx-based WordPress image because I had wasted a lot of time dabbling with Apache which I never use elsewhere.

WordPress Migration

With the new server up in a few minutes the next step is migration. The All-in-One WP Migration and Backup plugin (by ServMask) is the Swiss army knife. There are some pitfalls though.

First, the plugin has a size limitation in the free version. Luckily I can bypass it with changes to the PHP and Nginx configurations. For PHP configuration, change the line in /opt/bitnami/php/etc/php.ini, to upload_max_filesize = 512M. For the other change on Nginx as the reverse proxy, in /opt/bitnami/nginx/conf/nginx.conf, under http configuration, add:

client_max_body_size 512M;

The second pitfall is trickier. I have to temporarily host the new site, either by an public IP, or by a different DNS name (such as new.digihunch.com), so I can log in and verify the functions. Once I log on to the new site, the new DNS name or IP will start to be written to the database. I’m not sure which exact service does that. But once I put the new site on the production domain name (i.e. www.digihunch.com), the database still contains many entries referencing the temporary DNS name. This causes many insidious broken links.

There is a plugin for this kind of issue: Better Search Replace by WP Engine. The plugin goes through all tables in the database and allows you to find and replace strings. It also supports dry-run mode and I managed to fix tens of thousands of bad references. In addition, I managed to change username using a plugin called Easy Username Updater.

Another important activity after migration, is to ensure the MFA by WordFence plugin continues to work, and re-scan the entire file system.

CloudFlare Security

With less egress data and an HDD, the VPS in GCP is slower. I need to speed it up with CDN. It is hard to resist the freebies by CloudFlare. I’ve already been using it for security. Now it’s time to dig deeper.

CloudFlare greatly simplifies TLS certificates. It manages your certificate automatically. On the other side, it issues an origin certificate (and key, with long expiry) for communication with origin server. In full encryption mode, all I need to do is deploy origin certificate to Nginx. There is no manual renewal effort.

Note this setup requires CloudFlare to manage DNS so I just transferred my domain to CloudFlare. Therefore I consolidate domain registrar, certificate vendor and CDN to one vendor. CloudFlare can manage DNS record in proxy mode, to hide the public IP of the origin server from clients. I can also control the security group of the origin server to only open port 443 to CloudFlare IPs.

Another annoying but necessary configuration is the redirect of zone apex and http requests. Specifically most websites needs the following redirects:

If request from client is http://www.digihunch.com, enforce https;
If request is https://digihunch.com, redirect to https://www.digihunch.com;
If request is http://digihunch.com, enforce https, then redirect following rule #2.

I used to manage the these redirects in the reverse proxy (Apache). Now that I changed to Nginx, I offload these settings to CloudFlare, instead of configuring another reverse proxy again. For #1, I simply turned on “Always use HTTPS” and “Automatic HTTPS Rewrites” under edge certificates. For #2, I need a CNAME for zone apex to alias to www, along with a redirect rule. As a result, there is no need to ever open port 80 on the VPS, just to let the reverse proxy redirect URL.

CloudFlare Cache Rules

As to caching, I had to turn on a couple Cache rules to enable the caching. Cache hit was at about 50% and there are rooms to go higher if I enable reserve cache. Once cache is on, there are other considerations, such as expiring the cache, and by passing the cache during server deployment. CloudFlare allows you to purge cache by rules. There is also a Development Mode that temporarily allows you to bypass all cache for testing. With cache enabled, and the WordPress plugin activated, the page load time for recent posts appear to be faster than before.

One challenge is that the admin bar of WordPress went into the CDN and is served to visitors. To get rid of this behaviour I have two catch rules:

A cache-everything rule at order 1, to make everything eligible for cache;
A bypass rule at order 2 to bypass cache conditionally. The condition expression reads:

(http.cookie wildcard "wp-.*") or (http.cookie wildcard "wordpress_logged_in_*") or (http.cookie wildcard "wordpress.*")

To make use of the cache space. It is important to have small image sizes. Last year I added many AI generated feature images, most of which are 2MB in PNG format. I have to convert many PNG images into webp format, which only takes a fraction of the space and is widely accepted by most browsers today.

Sometimes there are other annoyances when it comes to speeding up page loading. For example, I noticed that with integration between Google Tag Manager and Microsoft Clarity, the home page issues a call to https://www.clarity.ms/tag/tag-id-xyz. This call blocks the rendering of a big chunk of home page by a minute! I didn’t notice it because my Brave browser’s ad-block removes that call. Nonetheless, the issue has been there for any visitor without built-in ad-block in their browser. The lesson learnt is that you always test with standard browsers, or at least in Brave browser with shields down.

SMTP service

I have been using email forwarding for incoming emails, and Gmail’s feature to send email from a different address or alias, for outgoing emails. This feature works but oftentimes the email gets marked as unverified on the recipient side. That gives it a good chance to appear as spam.

I need similar email forwarding mechanism and sending service. The free tier of CloudFlare goes a long way. Receiving is simply about creating Email routing rules, with a few MX and TXT records on DNS created and managed by CloudFlare. These records ensure Cloudflare routing rules capture all emails coming to the domain.

For outgoing emails, I have to get rid of the alias-based mechanism by Gmail. I need a proper SMTP server but CloudFlare doesn’t offer one. Resend seems like a popular choice with a good amount of free tier usage. However I’m hesitant to introduce another single-purpose platform so I just resort to Amazon SES for a full-feature, low-cost SMTP service. I do have to request production access, stating the server is for transactional email. This allows the SMTP service to arbitrary recipient address.

The other benefit of having my own SMTP server is to enable email integration in WordPress, which enables other important features such as web form, WordFense security alert and admin password reset by email.

Sending Reputation

To keep outing email from being marked as spam, it is important to understand what mechanisms are at play (SPF, DKIM, DMARC) to maintain sending reputation. At minimum, we configure SPF, DKIM and DMARC. SPF (Sender Policy Framework) identifies which mail servers are allowed to send mail on behalf of your custom MAIL FROM domain through a DNS TXT record that is used by DNS. The receiving mail server checks the SPF record of the sender’s domain to see if the email came from an authorized server. For example, when I use the SMTP server by Amazon SES, and customize the MAIL FROM field to mail.digihunch.com, I need an MX and a TXT record for mail.digihunch.com to tell receivers that it authorized amazonses.com to send email on its behalf.

DKIM (DomainKeys Identified Mail) is an email authentication method used to verify that an email message was sent by an authorized mail server and that the message content hasn’t been altered in transit. When an email is sent, the sending mail server generates a unique digital signature (based on the content of the email) using a private key. This signature is added to the email header. DNS Lookup: The receiving mail server looks up the sender’s domain in DNS to find the corresponding public key for DKIM authentication. Amazon SES configuration requires three TXT records to store the sender’s key information.

With SPF and DKIM configurations, the Amazon SES setup complies with DMRAC authentication protocol, making it less likely to be flagged as spam. The email comes off as mailed by amazonses.com and signed by digihunch.com.

In addition, there is Brand Indicators for Message Identification (BIMI) protocols that enables logo for your email on the recipient’s inbox. I managed to set up BIMI with SES, without Verified Mark Certificate (VMC), a protocol that requires evidence of ownership of the logo.

Although SPF, DKIM and DMARC cannot guarantee the email is not marked as junk, this is the best thing you can do. There are other techniques with diminishing return on effort.

Summary

This post summarizes my journey to land on the leanest web and email hosting solutions for small business. After five years of improvement, the tech stack that brings me low cost, solid security posture and lightening-fast speed, consists of: virtual machines from Google Cloud Platform (or Lightsail from AWS), web security and CDN by CloudFlare with Email supported by Gmail and Amazon SES (or Resend).

Initially with GCP, I tried to stick to the free tier but I did notice the standard persistent disk struggling, especially when I run system scan from WordFence. To overcome that I upgraded the disk to balanced disk (SSD backed) out of the free tier. There might be even cheaper alternatives but the marginal value isn’t worth the effort. Also, I configured the caching in CloudFlare for read, adjusted the home page, and schedule IO-intensive activities such as bi-weekly snapshot to quiet hours. The combination gave the site a 100 score in the CloudFlare speed testing.

Debating between count and for_each in Terraform

Tue, 27 Aug 2024 22:36:39 -0400

In Terraform, we often have to create an array of resources of the same type but similar attribute values. For code reusability, manageability and for DRY principle, it’s better to use loop. Terraform HCL supports loop via the use of meta-argument. Currently, there are two options to drive a loop: count and for_each .

Problem with `count` loop

The book Terraform Up and Running (Chapter 5 Terraform Tips and Tricks) regards count as Terraform’s oldest, simplest and most limited iteration construct. One of the big limitations is the shifting of index if the length of resource array changes. The point comes with a good example:

variable "user_names" {
 description = "Create IAM users with these names" 
 type = list(string)
 default = ["neo", "trinity", "morpheus"]
}
# The Example from the book Terraform Up and Running
resource "aws_iam_user" "example" { 
 count = length(var.user_names)
 name = var.user_names[count.index]
}

As you execute Terraform apply, three IAM users will be created, with the plan looking like:

# aws_iam_user.example[0] will be created
+ resource "aws_iam_user" "example" {
 + name = "neo"
 (...) 
 }
# aws_iam_user.example[1] will be created
+ resource "aws_iam_user" "example" {
 + name = "trinity"
 (...) 
}
# aws_iam_user.example[2] will be created
+ resource "aws_iam_user" "example" {
 + name = "morpheus"
 (...) 
}

Then if you remove “trinity” from the variable user_names, and run terraform plan, the plan would look like:

Terraform will perform the following actions:
 # aws_iam_user.example[1] will be updated in-place
 ~ resource "aws_iam_user" "example" {
 id = "trinity"
 ~ name = "trinity" -> "morpheus"
 }
 # aws_iam_user.example[2] will be destroyed
 - resource "aws_iam_user" "example" {
 - id = "morpheus" -> null
 - name = "morpheus" -> null
 }
Plan: 0 to add, 1 to change, 1 to destroy.

In this plan, instead of deleting the second user, it renames the second user and deletes the third user. While the plan matches the code logic, it is often an unwanted result, considering the resource could be one that many other resources depends on, such as a subnet.

This is a good example of the problem with count. Terraform identifies each resource in the generated list of resource by position(index) . When the length changes, the index shifts. If you remove an item from the middle of the list, Terraform will delete every resource after the deleted item, then re-create all the resources that come after the deleted one. As a consequence, you may loose availability or even worse, lose data.

Embrace `for_each` loop

If we modify the example above to use for_each, the code looks like:

variable "user_names" {
 description = "Create IAM users with these names" 
 type = list(string)
 default = ["neo", "trinity", "morpheus"]
}
# The Example from the book Terraform Up and Running
resource "aws_iam_user" "example" 
{ 
 for_each = toset(var.user_names) 
 name = each.value
}

This results in the creation of three IAM users. If you remove the “trinity” user from the middle of the input collection and apply, the plan looks like this:

Terraform will perform the following actions:
 # aws_iam_user.example["trinity"] will be destroyed
 - resource "aws_iam_user" "example" {
 - arn = "arn:aws:iam::123456789012:user/trinity" -> null
 - name = "trinity" -> null
 }
Plan: 0 to add, 0 to change, 1 to destroy.

The plan suggests that Terraform will delete the very resource that was taken out from the middle of the input collection and no existing resources in the array are impacted.

Note that in the code snippet above, we use function toset() to convert the input list to a set (ordered and de-duped list of string). This is because we can only loop over a set or map when creating an array of resource. If the array of resource being created have another attribute whose value needs to be individualized, we can loop over a map and store the individualized attribute values as key-value pairs.

A few pages down, the book discusses an important limitation for both count and for_each. The length of the resource array that you are creating with count or for_each meta-argument must not be computed from other resources. Terraform must be able to compute count and for_each during the plan phase, before any resources are created or modified. The length of the resource array can be from hardcoded values, data sources, or even a list of other resources to create in the same file, so long as the length can be determined during the plan, instead of not being computed from other resource outputs.

A real-life example with classic pattern

The book then touches on another advantage of for_each: the ability to create multiple inline blocks within a resource. The guide from Hashicorp documentation also has a section on when to use for_each Instead of count, with a similar example. The section merely mentions when to use count in the opening sentence: If your instances are almost identical, count is appropriate.

That makes for_each sound like a no-brainer, after reading all the literatures about this topic. In my experience with a specific use case at the beginning, count feels more efficient. The example from the book is too simplistic. To better compare the two options, I need a realistic example. Let’s consider this use case where, after creating a VPC, I need to create the followings:

one NAT gateway for each availability zone (each NAT Gateway maps to one subnet and one allocation ID)
one public subnet for each availability zone
one public IP allocation in each availability zone

We can summarize the relationships between resources in the following diagram:

I deliberately pick this example because they are self-contained. So are all code snippets in this post. The example also demonstrate a classic relation between resources that we can find everywhere in infrastructure automation. Here’s another example off the bat:

create an array of aws_subnet, each has a subnet_id attribute;
create an array of aws_route_table, each has a reout_table_id attribute;
now, create an array of aws_route_table_association, each referencing one aws_subnet (by subnet_id) and one aws_route (by route_table_id);

If we address the NAT gateway example, we’re good with many other resources that shares the same relation pattern. In the next section, we’ll first implement the NAT gateway example, using count loop.

Implementation using `count`

The use case exemplifies the pattern where we have multiple types of resources related to each other. We need a loop in each type of resources, resulting in multiple arrays of different resource types. Moreover, the elements in the array for aws_nat_gateway has 1-to-1 mappings with both the array for aws_subnet, and the array for aws_eip.

With count, I created Terraform code with everything in a single main.tf file for the convenience of illustration, like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}
variable "public_subnets_cidr_list" {
 type = list(any)
 default = ["147.206.0.0/22", "147.206.4.0/22"]
 #default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"]
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

resource "aws_subnet" "public_subnets" {
 count = length(var.public_subnets_cidr_list)
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = var.public_subnets_cidr_list[count.index]
 map_public_ip_on_launch = true
 availability_zone = data.aws_availability_zones.this.names[count.index]
}

resource "aws_eip" "nat_eips" {
 count = length(var.public_subnets_cidr_list)
}

resource "aws_nat_gateway" "nat_gws" {
 count = length(var.public_subnets_cidr_list)
 subnet_id = aws_subnet.public_subnets[count.index].id
 allocation_id = aws_eip.nat_eips[count.index].id
 depends_on = [aws_internet_gateway.internet_gw]
}

The intent is to create subnet, public IP and NAT gateway for two availability zones. I also want to add one more AZ in the future and have the code to handle the addition gracefully. To add the new AZ, I uncomment line 10 and comment out line 9. The plan after this code change looks like this:

Terraform will perform the following actions:

 # aws_eip.nat_eips[2] will be created
 + resource "aws_eip" "nat_eips" {
 (...) 
 }

 # aws_subnet.public_subnets[2] will be created
 + resource "aws_subnet" "public_subnets" {
 + availability_zone = "us-east-1c"
 + cidr_block = "147.206.8.0/22"
 + id = (known after apply)
 (...) 
 }

 # aws_nat_gateway.nat_gws[2] will be created
 + resource "aws_nat_gateway" "nat_gws" {
 + allocation_id = (known after apply)
 + subnet_id = (known after apply)
 (...) 
 }

Plan: 3 to add, 0 to change, 0 to destroy.

The plan creates a set of resources required for the new availability zone without touching any existing resource, which is expected.

Why do I only focus on the use case of adding a new subnet in new AZ, and not deleting or modifying CIDR on an existing subnet? That’s because we rarely do that with production. We rarely remove the use of an availability zone. Nor do we modify the CIDRs on an existing subnet. In fact, AWS SDK does not even have an API to change CIDRs on a subnet or a VPC. In our infrastructure operation, we make such decisions upfront so they are immutable once provisioned. We simply don’t need to consider all the possible CRUD actions on a resource.

So, the count loop does just the job. Now, what about for_each?

Implementation with `for_each`: first attempt

Since for_each takes a set or map, I have to make some adjustment. My first attempt looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}

variable "public_subnets_cidr_list" {
 type = list(any)
 default = ["147.206.0.0/22", "147.206.4.0/22"] # 2 AZ
 #default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"] # 3 AZ
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

locals {
 subnet_config = [
 for i in range(length(var.public_subnets_cidr_list)) : {
 cidr = var.public_subnets_cidr_list[i]
 az = data.aws_availability_zones.this.names[i]
 }
 ]
}

resource "aws_subnet" "public_subnets" {
 for_each = { for idx, rec in local.subnet_config : idx => rec }
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = each.value.cidr
 map_public_ip_on_launch = true
 availability_zone = each.value.az
 tags = { Name = "PUBLIC-SUBNET" }
}

resource "aws_eip" "nat_eips" {
 for_each = toset(var.public_subnets_cidr_list)
 tags = { Name = "NATEIP" }
}

data "aws_subnets" "public_subnets" {
 filter {
 name = "tag:Name"
 values = ["PUBLIC-SUBNET"]
 }
 depends_on = [aws_subnet.public_subnets]
}

data "aws_eips" "nat_eips" {
 filter {
 name = "tag:Name"
 values = ["NATEIP"]
 }
 depends_on = [aws_eip.nat_eips]
}

locals {
 nat_gw_config = [
 for i in range(length(var.public_subnets_cidr_list)) : {
 subnet_id = data.aws_subnets.public_subnets.ids[i]
 alloc_id = data.aws_eips.nat_eips.allocation_ids[i]
 }
 ]
}

resource "aws_nat_gateway" "nat_gws" {
 for_each = { for idx, rec in local.nat_gw_config : idx => rec }
 subnet_id = each.value.subnet_id
 allocation_id = each.value.alloc_id
 depends_on = [aws_internet_gateway.internet_gw]
}

Note that I have to create a couple of data resources (nat_eips and public_subnets) and local variables (subnet_config and nat_gw_config) in order build the required map data structures and feed them to the for_each parameters.

After Terraform apply, let’s edit public_subnets_cidr_list with the additional subnet CIDR for the 3rd AZ. The plan looks like this:

Terraform will perform the following actions:

 # data.aws_eips.nat_eips will be read during apply
 # (depends on a resource or a module with changes pending)
 <= data "aws_eips" "nat_eips" {
 + allocation_ids = (known after apply)
 (...) 
 }

 # data.aws_subnets.public_subnets will be read during apply
 # (depends on a resource or a module with changes pending)
 <= data "aws_subnets" "public_subnets" {
 + id = (known after apply)
 + ids = (known after apply)
 (...) 
 }

 # aws_eip.nat_eips["147.206.8.0/22"] will be created
 + resource "aws_eip" "nat_eips" {
 + allocation_id = (known after apply)
 (...) 
 }

 # aws_nat_gateway.nat_gws["0"] must be replaced
-/+ resource "aws_nat_gateway" "nat_gws" {
 ~ allocation_id = "eipalloc-0ffe32519d20b9b7f" # forces replacement -> (known after apply) # forces replacement
 ~ subnet_id = "subnet-0b4b202056c75bb0a" # forces replacement -> (known after apply) # forces replacement
 (...) 
 # (1 unchanged attribute hidden)
 }

 # aws_nat_gateway.nat_gws["1"] must be replaced
-/+ resource "aws_nat_gateway" "nat_gws" {
 ~ allocation_id = "eipalloc-0ff1d32fb271b5cf4" # forces replacement -> (known after apply) # forces replacement
 ~ subnet_id = "subnet-09b6486fbe44e9795" # forces replacement -> (known after apply) # forces replacement
 (...) 
 # (1 unchanged attribute hidden)
 }

 # aws_nat_gateway.nat_gws["2"] will be created
 + resource "aws_nat_gateway" "nat_gws" {
 + allocation_id = (known after apply)
 + subnet_id = (known after apply)
 (...) 
 }

 # aws_subnet.public_subnets["2"] will be created
 + resource "aws_subnet" "public_subnets" {
 + availability_zone = "us-east-1c"
 + cidr_block = "147.206.8.0/22"
 (...) 
 }

Plan: 5 to add, 0 to change, 2 to destroy.

Wait a second, I expect the template to create a new subnet, a new elastic IP and a new NAT gateway in that new AZ. But why does it plan to delete the two existing NAT gateways and recreate two? This doesn’t make for_each an appealing option at all.

Apart from the interruptive plan, there are also other problems. First, since the aws_subnet resources requires cidr_block and availability_zone values, I have to build a map (subnet_config) for its resource array to consume. Similarly, I have to build a second map (nat_gw_config) to create resource array for aws_nat_gateway, which requires subnet_id and allocation_id. This map takes more work to build. Because of the 1-to-1 relationship between subnet_id and alloc_id, I have to fetch the values from two data sources (line 47-61), use a common index (line 63-70). Can I neat it up and combine two maps into one? Not really. Because the second map (nat_gw_config) uses a data source depending on the subnets, which depends on the first map (subnet_config). Trying to combine the maps causes circular dependency!

Also, the additions of data sources makes the code less readable. As Marcel L pointed out in his post, two cons with for_each are: complexity and requiring a map (to store multiple attribute values). Now we seem to have one more: it may cause unintended deletions

Is `for_each` a bad idea?

Let’s find out why for_each could destroy two existing NAT gateways.

Notice that I built the map nat_gw_config by looping through the list of variable public_subnets_cidr_list. After apply, we appended it one more string at the end, without changing the existing order. However, the devil lies in the order of the string lists returned from the data sources. By printing this map, we found that the originally value before the AZ addition is:

index	alloc_id	subnet_id
0	eipalloc-0ffe32519d20b9b7f	subnet-0b4b202056c75bb0a
1	eipalloc-0ff1d32fb271b5cf4	subnet-09b6486fbe44e9795

Based on this, NAT Gateway with index 0 is created with eipalloc-***b7f and subnet-***b0a. NAT Gateway with index 1 is created with eipalloc-***cf4 and subnet-***795. After we add the third AZ, and apply the run, the new map, with a new alloc_id and a new subnet_id looks like this:

index	alloc_id	subnet_id
0	eipalloc-0b70460721596e33f (new)	subnet-0b4b202056c75bb0a
1	eipalloc-0ffe32519d20b9b7f	subnet-0335071ced2dc9922 (new)
2	eipalloc-0ff1d32fb271b5cf4	subnet-09b6486fbe44e9795

There are two factors at play. When the data sources return the ids (data.aws_subnets.public_subnets.ids and data.aws_eips.nat_eips.allocation_ids), the return is sorted. It doesn’t matter whether the order alphabetical or the opposite. Because in any given order, the randomly generated new ID, can fall anywhere in the list. In this particular result, the new alloc_id falls at the beginning, and the new subnet_id falls in the middle. As a result, NAT Gateway with index 0 and 1 are both changed. Therefore they have to be destroyed and replaced.

All these come from having to build a map. The values of each object in the map come from two different data sources. The values are not predetermined and contain a random part. When we add more AZ, the entire map get shuffled, leading to deletion of existing resources. Yikes.

Implementation with `for_each`: second attempt

The first draft of this post drew some ideas on Reddit. One redditor pointed out that the snippet above with for_each isn’t the optimal way. With some tricks to we can manage the map so that it maintain relative order if we have to add new AZ. The strategy is:

Avoid using data sources to retrieve attribute values
Use a unique key to identify objects in the map;
Directly look up from the resource by the unique key

We’re able to do #2 and #3 because when a resource has the for_each argument set, the resource itself becomes a map of objects. We can then locate that resource by the key. We can determine what that key is so long as it uniquely identifies the resource. Below is the revised code snippet with for_each:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}

variable "public_subnets_cidr_list" {
 type = list(any)
 default = ["147.206.0.0/22", "147.206.4.0/22"] # 2 AZ
 #default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"] # 3 AZ
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

locals {
 subnet_config = {
 for cidr in var.public_subnets_cidr_list : md5(cidr) => {
 cidr = cidr
 az = data.aws_availability_zones.this.names[index(var.public_subnets_cidr_list, cidr)]
 }
 }
}

resource "aws_subnet" "public_subnets" {
 for_each = local.subnet_config
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = each.value.cidr
 map_public_ip_on_launch = true
 availability_zone = each.value.az
}

resource "aws_eip" "nat_eips" {
 for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) => cidr }
}

locals {
 nat_gw_config = {
 for cidr in var.public_subnets_cidr_list : md5(cidr) => {
 subnet_id = aws_subnet.public_subnets[md5(cidr)].id
 alloc_id = aws_eip.nat_eips[md5(cidr)].allocation_id
 }
 }
}

resource "aws_nat_gateway" "nat_gws" {
 for_each = local.nat_gw_config
 subnet_id = each.value.subnet_id
 allocation_id = each.value.alloc_id
 depends_on = [aws_internet_gateway.internet_gw]
}

In this example, I use the MD5 hash of CIDR as the unique identifier key to ensure we have a consistent mapping between allocation id and subnet id. When a new AZ is created, the new allocation-subnet id pair will have its own new key. The unique key can be any identifier (even the CIDR itself) as long as it is unique and we do not change the selection of unique key after the first apply.

One more shot with `for_each`

The code snippet above got rid of data sources, but still have to leverage two local values (subnet_config and nat_gw_config) as helpers. Are they absolutely necessary?

Not really. The Terraform documentation has a page about References to Values, where it states:

If the resource has the count argument set, the reference’s value is a list of objects representing its instances.
If the resource has the for_each argument set, the reference’s value is a map of objects representing its instances.

In other words, using for_each with a map as input, we’re also creating a map as output, which is the resource array itself. The key is the same as the input map. Therefore, we can reuse the key. I know that sounds too abstract. Here’s the code refined:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}

variable "public_subnets_cidr_list" {
 type = list(any)
 #default = ["147.206.0.0/22", "147.206.4.0/22"] # 2 AZ
 default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"] # 3 AZ
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

resource "aws_subnet" "public_subnets" {
 for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) => {
 cidr = cidr
 az = data.aws_availability_zones.this.names[index(var.public_subnets_cidr_list, cidr)]
 }
 }
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = each.value.cidr
 map_public_ip_on_launch = true
 availability_zone = each.value.az
}

resource "aws_eip" "nat_eips" {
 for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) => cidr }
}

resource "aws_nat_gateway" "nat_gws" {
 for_each = aws_subnet.public_subnets
 subnet_id = aws_subnet.public_subnets[each.key].id
 allocation_id = aws_eip.nat_eips[each.key].allocation_id
 depends_on = [aws_internet_gateway.internet_gw]
}

Voila. I use md5 of the CIDR as the key again, first to create both aws_subnet and aws_eip. I also followed the example of chaining for_each between resource types. This way, when creating aws_nat_gateway, I can reference an instance in each resource array by the same key. Chaining for_each is very handy. But admittedly, it takes several iterations for me to get there. The code is neater, but not as straightforward to read due to the list/map comprehension.

Conclusion

I came across a team where the code review guideline favours for_each strongly. I see where that comes from after reading the book. But I don’t find count to be evil. That triggered my initiative to dive deep into this topic.

In this post we brought up a classic pattern of relationship between resources, and examined several ways to implement them using count and for_each. Using count can be straightforward but carries the risk of index shifting if additional element is added in the middle of the resource array. On the other hand, for_each is more powerful, but it requires some crafting with the Python-style list/map comprehension.

My recommendation is, start with a holistic look at the types of resources to create with loop, and how they are related with each other. Go with count if index shifting isn’t a risk. For example, when you need to create one instance of a resource conditionally. Otherwise, use for_each loop if the team is comfortable with the list/map comprehension. In some cases where we need to conditionally create several instances of the same resource, we can use a technique such as:

for_each = variable.disabled ? {} : data.any_resource.map
for_each = variable.disabled ? toset([]) : data.any_resource.list

In fact, the recommendation from AWS Terraform best practice is highly in favour of for_each.

Test Open ID Connect Flows Locally

Mon, 05 Aug 2024 13:24:34 -0400

Earlier this year, I had to integrate an application with an identity provider. Both claim to be compliant with Open ID Connect. But when they don’t get along, I must find out where it breaks to determine which party isn’t compliant. Therefore, I had to really get to the transaction-level details. As a result, I was eager to find out a way to test Open ID Connect Flows locally. Considering that I discussed OIDC in several past articles and use cases (authenticating to kube-api server, Kubernetes workload, ROSA), I feel it is important to be able to test Open ID Connect flows locally.

Architecture

What makes local testing difficult is that there are several parties involved and they act in different roles. Take Authorization Code Flow as an example, the main actors are:

User Agent: the browser;
Authorization Server: an identity store, also referred to as identity provider;
Resource server: an HTTP server that returns protected resource;
OIDC client application: the component that communicates with identity store on behalf of the resource server;

Apart from the bowser (obviously running on my laptop), we can group the other actors in different patterns. Here I illustrate some options below:

In essence, we need an OIDC client app and an authorization server. And I find the most simplistic test architecture to be pattern #2 (without reverse proxy) on the client side, and pattern #1 on the server side. In the next section let’s discuss why I prefer this test architecture.

Choice of Tools

The principle of this lab is that we can focus on the OIDC flows itself and simplify every other aspects as much as we can.

The authorization server needs to access my client app. As a result, if I use a public authorization server, such as Azure or Google, I’d have to host my client app on a public IP with domain name as well. This creates churns. To see how much hassle this can involve, review my article Istio External Authentication lab. Instead, I need a tool to host the authorization server on my laptop. And yes, it’s KeyCloak. It is a well-renewed open-source project for identity and access management. It is also the upstream project of RedHat SSO. Another reason is it operates on PostgreSQL database which is a common relational database technology with release in Docker images.

On the client side, we can have our web service with built-in OIDC capability, which usually requires development work in a language with OIDC library. Alternatively, if the client app lacks such capability, we would address this capability in a different component on the client side. It can either be a standalone, purpose-built proxy, or a generic reverse proxy with OIDC capability. For this, I examined a few options. Nginx and Traefik have OIDC support for a fee in their Enterprise product. Apache has a module mod_auth_oidc for free but it requires building the plug-in on my own for Mac platform. Eventually I landed on the purpose-built option, using the oauth2-proxy open source project. This project can act as both actor #3 (with a minimal HTTP server) and actor #4 (OIDC client). So it also saves me from hosting a separate web server. Also, I have used it in the past in the Istio External Authentication lab.

In reality, each message in the OIDC flows must be TLS encrypted. In local testing though we don’t really care. Similarly we don’t necessary need a reverse proxy if it plays no role in the OIDC flow. Both Keycloak (with PostgreSQL) and oauth2-proxy are released in Docker images. As a result, we are ready to roll with only three tools: the browser on local host, Keycloak and oauth2-proxy in the Docker daemon, which provides great portability.

To get started, let’s have two fictitious domains, web.digihunch.com, and keycloak.digihunch.com. In the Docker compose manifest, I name the services based on their hostname so that they can reference each other from within the container network namespace. To access the service from the host, I force the DNS resolution to localhost in /etc/hosts on my Mac, and make sure to declare the same host port in the port mapping (4180 for dummy web service; and 8080 for Keycloak).

Configuration

I created a Docker compose file as below, to set up my test:

services:
 web.digihunch.com:
 container_name: oauth2-proxy
 image: quay.io/oauth2-proxy/oauth2-proxy:latest
 command:
 - --http-address
 - 0.0.0.0:4180
 environment:
 OAUTH2_PROXY_COOKIE_SECRET: NYZaClZinINKwxNGzEDeFGh64W6tmq1eB6uHQPa4S5o 
 OAUTH2_PROXY_CLIENT_ID: dh-user-client
 OAUTH2_PROXY_CLIENT_SECRET: pmkwBjkVesrj7fw1MY7h5s9e3cmAKXgc 

 OAUTH2_PROXY_PROVIDER: oidc 
 OAUTH2_PROXY_OIDC_ISSUER_URL: http://keycloak.digihunch.com:8080/realms/digihunch-users

 OAUTH2_PROXY_PASS_ACCESS_TOKEN: true 
 OAUTH2_PROXY_EMAIL_DOMAINS: '*' 
 OAUTH2_PROXY_REDIRECT_URL: http://web.digihunch.com:4180/oauth2/callback
 OAUTH2_PROXY_PROVIDER_DISPLAY_NAME: DHCKC

 OAUTH2_PROXY_COOKIE_CSRF_EXPIRE: '5m'
 OAUTH2_PROXY_COOKIE_CSRF_PER_REQUEST: true
 OAUTH2_PROXY_COOKIE_SECURE: false # Needed for HTTP connection

 #OAUTH2_PROXY_UPSTREAMS: file:///var/www/static/#/home/ # serve page at /home path
 OAUTH2_PROXY_UPSTREAMS: static://202 
 volumes:
 - ./config/oauth2-proxy.cfg:/etc/oauth2-proxy.cfg
 # - ./config/www:/var/www/static/
 ports: 
 - 4180:4180
 networks:
 - oidc_network
 restart: unless-stopped
 depends_on:
 - keycloak.digihunch.com
 
 postgres-db:
 image: postgres
 container_name: postgresdb
 restart: always
 shm_size: 128mb
 ports:
 - 5432:5432
 networks:
 - oidc_network
 volumes:
 - ./data/pgdata:/var/lib/postgresql/data
 environment:
 - POSTGRES_USER=master
 - POSTGRES_PASSWORD=masterpass
 - POSTGRES_DB=keycloak

 keycloak.digihunch.com:
 image: quay.io/keycloak/keycloak
 command: start
 environment: # Based on Hostname:v2 https://www.keycloak.org/docs/latest/upgrading/#migrating-to-25-0-0
 KC_HOSTNAME: http://keycloak.digihunch.com:8080
 #KC_HOSTNAME_ADMIN: For simplicity, no separate management URL or port
 KC_HOSTNAME_BACKCHANNEL_DYNAMIC: true
 KC_HTTP_ENABLED: true ## Otherwise HTTPS is the enforced by default.
 KC_HEALTH_ENABLED: true
 KEYCLOAK_ADMIN: admin 
 KEYCLOAK_ADMIN_PASSWORD: kcadminpass
 KC_DB: postgres
 KC_DB_URL: jdbc:postgresql://postgres-db/keycloak
 KC_DB_USERNAME: master
 KC_DB_PASSWORD: masterpass 
 ports:
 - 8080:8080
 networks:
 - oidc_network
 restart: always
 depends_on:
 - postgres-db

networks:
 oidc_network:
 enable_ipv6: false
 driver: bridge

For OAuth2-Proxy, the official image for oauth2-proxy is distroless. So if you have to troubleshoot its container file system, you need to access it via an assistant container:

docker run --rm -it --name debugger --privileged --pid container:oauth2-proxy --network container:oauth2-proxy busybox sh
# to see target container's file system, go to: ls -l /proc/1/root/

Alternatively, we can use the one that Bitnami releases but be vary of some nuances.

In the Keycloak part, I use environment variables and had to watch out for the recent changes on hostname v2. We should first start up the Keycloak service. From http://keycloak.digihunch.com:8080, we can login using the credential specified in the environment variables, then we can create a client app:

Create a new realm (dropdown-> Create realm) with name digihunch-users
Switch to this realm from the dropdown, create a couple users (e.g. dhadmin@www.digihunch.com), and set password.
Create a group (e.g. myadmin) and join the user to the group
Under the same realm, create a client, with OpenID Connect as type, client ID being dh-user-client. turn on client authentication (without Direct access grants)
Save the client for now and grab the client secret. Note that the OIDC discovery document (http://keycloak.digihunch.com:8080/realms/digihunch-users/.well-known/openid-configuration) should come online.
Update the docker compose file:
- The value for OAUTH2_PROXY_CLIENT_SECRET is from the client secret;
- The value for OAUTH2_PROXY_OIDC_ISSUER_URL should be http://keycloak.digihunch.com:8080/realms/digihunch-users;
- The value for OAUTH2_PROXY_CLIENT_ID is dh-user-client;
Restart all services including the web. Log on to keycloak and go back to the client configuration in the realm. Under settings. Put in valid redirect URIs as http://web.digihunch.com:4180/oauth2/callback and save the client.

Now, let’s start an private browser session, and browse to http://web.digihunch.com:4180/. The browser should redirect you to keycloak’s login page. Once log in is successful, it should redirect you to the static response with 202 code as the manifest configured.

Summary

In this post, I laid out the steps to test the login for OIDC authorization code flow locally as a starting point. For a bullet-proof solution, I recommend taking a look at KeyCloak administration guide. For example, we typically disable the master realm for security.

There is a similar test setup on Otka blog but I simplified all the aspects that I regard as distractors. There are many other flows that can be tested. However, some of the testing still requires client pattern #1 if we need to initiates an activity from the client application.

IAM Roles for any workload

Sun, 14 Jul 2024 23:53:49 -0400

Background

A few month back a client of mine wanted to use GitLab pipeline to deploy infrastructure on AWS with Terraform. The key question is how to authenticate the Terraform process running in the pipeline to AWS with temporary credential. Having worked it out on GitHub, my proposal at time was to add OIDC provider to represent the GitLab runner.

After a few months, they told me that they are self-hosting their GitLab instance. The idea above was based on exposing an identity provider document on the public Internet, which the client is unable to do.

Now, I have an idea: IAM Roles Anywhere.

IAM Role Introduction

Many 101 tutorials asks beginners to create standalone IAM users (or group) with IAM policies directly attached. For programatic access they also include creating a pair of access key and secret access key and pass them along to an external application. The keys are long term credentials, and worse, never expires. The leakage of these long-term credentials had been such a headache that AWS strongly discourage the use of long term credentials. You can feel the discouragement when trying to create an access key through the web console, or by the banners on top of the documentation page about how to do so.

The recommendation is use temporary security credentials. In the context of AWS that means IAM roles. The users must assume an IAM role by issuing an API call, and the Security Token Service (STS) grants temporary credential in response.

This diagram has several variations. For example, the request can be AssumeRole, AssumeRoleWithSAML and AssumeRoleWithWebIdentity, depending on whether and how the user info is federated with external identity store. The returned response, a triplet of three values, makes the temporary credential that we should use in any secure environment. They must be renewed before expiry. This model works not only for human identity (e.g. SAML integration, OIDC integration, cross-account access) but also for workload identity (e.g. EC2 instance profile, Lambda execution role, ECS task role, etc).

Another good example is IAM Role for Service Account (IRSA), where a web identity represents a Kuberentes Service Account to gain role credential using the AssumeRoleWithWebIdentity API. In this post however, I’d like to explore more about the IAM role for EC2 instance profile.

The IMDS service

For EC2 instance, we all know that we can associate an IAM role as the instance profile and grant the process using AWS SDK running on the instance with permissions associated with the IAM role. At a lower level, this relies on the IMDS (Instance Metadata Service) running on the instance.

If an EC2 instance’s profile points to an IAM role, a process running on the instance using AWS SDK will also need to get the triplet from the STS. It is summarized in this diagram:

The IMDS is a service available on one of the link-local IP address (169.254.169.254) on the EC2 instance. Requests made to this IP address are not routed elsewhere. The Instance Metadata Service (IMDS) is a means for the cloud service provider’s virtualization layer to share information with the processes on the operating system of a virtual machine. It responds with information related to the instance itself, such as the subnets, IAM role, instance ID, AMI ID, security group. The instance metadata also includes user data script for cloud init process to consume, and most relevantly, the role credential for the instance. This also requires that the IMDS service to have connectivity to the STS endpoint, either via interface endpoint or over the Internet. The AssumeRole calls are logged in CloudTrail.

All major cloud vendors (AWS, Azure and GCP) uses the IMDS mechanism, and this mechanism obviously draws the attention of bad actors. I find some good articles on this here and here.

IAM Role for EC2 Workload

For EC2 instances at AWS, the initial IMDS v1 was introduced in 2012 and allows a GET method to fetch instance metadata. The IMDS v1 is subject to attacks such as SSRF (Server-side request forgery). In 2019 AWS introduced IMDS v2 which tackles those vulnerabilities. As of date, the recommendation is to use IMDSv2. Here is an example of how to fetch instance metadata, including the credential:

# Grab a token
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
# Get top-level instance metadata information
curl http://169.254.169.254/latest/meta-data/profile -H "X-aws-ec2-metadata-token: $TOKEN"
# Get the name of the role
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials
# Get the credential for the role session
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/InstanceProfileRoleName

These commands emulate how the SDK library fetches the credentials to assume the instance profile role. You can also find similar commands on the documentation. However, there isn’t much details about how the instance metadata service interacts with the STS service, except a general statement:

These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes before the expiration of the old credentials.

When creating a new EC2 instance, make sure that the instance metadata option has http_endpoint enabled, to enable the IMDS service. Also set http_tokens to required, which would run IMDSv2 exclusively. With that setup, the application does not have to mana to use an SDK version that supports IMDSv2. Another metadata option is http_put_response_hop_limit, with default of 1. This limits the number of hops in the metadata request. If the process runs from a Docker container with bridge networking mode, set it to 2 or the process cannot even secure a token.

The diagram above illustrates this scenario with two hops.

IAM Role Anywhere Concept

As this point, we know that the process of workload assuming an IAM role, is essentially using SDK to gain role credentials from instance metadata. In addition to using SDK and instance metadata, AWS also supports using X.509 certificate to gain role credentials. As a result, workload no longer needs AWS SDK, and it doesn’t rely on instance metadata from an EC2 instance. This mechanism is known as IAM Role Anywhere, and it greatly expands the use cases for IAM Role.

To make this work, we first have to provide a certificate authority (CA) to AWS as a trust anchor. It can be any X.509 CA including AWS Private CA. The IAM Roles Anywhere will allow any end-entity endorsed by this trust anchor, to assume an IAM role as specified. We also need to create a profile, in which we can add IAM policies directly, or link to an IAM roles with a trust policy for service principal rolesanywhere.amazonaws.com.

To gain role credential, the requestor must provide both the private key, and its end-entity certificate. The certificate proofs the endorsement of the CA as Role’s trust anchor specifies. The private key proofs the requestor’s identity. The requestor uses the aws_signing_helper utility to request role credentials.

The utility is compatible with the credential_process feature in AWS config, which passes the returned role credentials to the AWS config profile for AWS CLI or SDK running on external virtual machine.

IAM Role Anywhere Lab

Let’s tweak the three commands from this old post of mine to create the test materials: a self-signed CA and a certificate signed by the CA:

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Health Certificate Authority' -addext basicConstraints=critical,CA:TRUE,pathlen:1 -addext keyUsage=keyCertSign

cat > ext.cnf <<EOF
[v3_leaf]
keyUsage = digitalSignature
basicConstraints=CA:false
EOF

openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=*.digihunch.com'

openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt -extfile ext.cnf -extensions v3_leaf

I tweak them to add the X.509 extensions to meet the requirement for signature validation. We need the following files from the output.

ca.crt -> the certificate of the CA. We provide this file as the trust anchor
server.crt -> the certificate of the server, we need it as the end-entity certificate
server.key -> we need to present this file to proof identity of the requestor

First, we go to AWS console and create a new trust anchor. Copy the content of ca.crt as the certificate. Then we can create a profile with an IAM role, with the trust policy looking like this example. Then we can request the role credential with one command:

aws_signing_helper credential-process --certificate server.crt --private-key server.key --trust-anchor-arn $TRUST_ANCHOR_ARN --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN

Moreover, if the workload supports AWS SDK or can use CLI but not an EC2 instance, we can bake this in the AWS profile on the external machine:

[profile myprofile]
output = json
credential_process = aws_signing_helper credential-process --certificate /path/server.crt --private-key /path/server.key --trust-anchor-arn $TRUST_ANCHOR_ARN --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN

We can even configure this in any pipeline as code to allow deployment from a non-AWS pipeline.

Summary

In summary, apart from native AWS services, an IAM role can trust the following types of principals:

Native IAM identity such as an IAM user or an IAM group
Authenticated identity from SAML identity provider that IAM is configured to trust
Authenticated identity from OIDC identity provider that IAM is configured to trust
Validated identity endorsed by a Certificate Authority that IAM designate as a trust anchor

The first type is rarely used because few organizations uses AWS IAM as their identity store. Most organizations have their identity store with federation capability via SAML. On the other hand, a lot of modern applications adopts identity stores with OIDC compliance. Now with IAM Role Anywhere, any entity with X.509 identity can also assume an IAM role. It works with any CI/CD pipeline, whether it is self-hosted. Also, it is now more important to keep the keys safe.

Managing EC2 instances across accounts with Ansible

Mon, 27 May 2024 14:00:22 -0400

I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is consistency in the VM management, over time, and across platforms (AWS, on-prem, Azure, etc). Even for AWS shops, in the last few years many enterprises have adopted AWS landing zone with the multiple AWS account prescriptive pattern. However AWS Systems Manager still lacks integration with AWS Organization (except for a few non-core capabilities). This creates the demand of managing EC2 instances across AWS accounts. In this post, we propose a secure method to manage a fleet of EC2 instances from multiple AWS accounts, using Systems Manager . It also enables connectivity from an Ansible control node.

Prerequisites

This proposal ties a few CLI tools together, including AWS CLI, SSH, Ansible, etc. It also requires the cloud engineer to understand how they work. I’ll start with the choice of the tools.

Above, I discussed the benefit of Ansible. Since Ansible operates on SSH, we’ll still have to use SSH tools. Even though SSM agent provides a way to connect to EC2 instance without requiring an RSA key pair, we still need SSH since it is a well-established industry standard (RFC4253) and the foundation of Ansible. These two technologies are not mutually exclusive. In fact, the SSM agent provides a secure enhancement to the operation with SSH. Traditionally, on each EC2 instance we’d have to run SSHD services which opens TCP port 22 (or alternative TCP port as configured). For authentication we favour key pair over password but the open port is still an attack surface vulnerable to brute force and DDoS attacks. For EC2 instances on private networks there is no reachability to the instance’s SSH port unless the bastion is also in a connected network.

As I cover in a post, the SSM Session Manager comes in handy. The SSM agent operates from the instance and communicate outbound to AWS backend. Since the SSM agent runs under a privileged user on the OS, you can perform OS-level commands through SSM. Further, AWS developed a Session Manager plugin with AWS CLI, allowing AWS CLI as a proxy command when making an SSH connection. Therefore SSM enables SSH connection without requiring port 22 to be open. In addition we’ll need to use RSA key pair as required for SSH, which is also an improvement to the security posture.

That explains the dependent tools. On the Ansible control node, apart from Ansible itself, we need the latest version of AWS CLI with the Session Manager plugin, we need to configure AWS CLI properly to connect to EC2 instances across multiple AWS accounts.

Configure AWS CLI

This section discusses how to configure AWS CLI. I have a couple of handy aliases for productivity but they are not essential. For example, I often need to check the IAM identity making the call, and I often need to list out all profiles configured. So I added the following two entries in the ~/.aws/cli/alias file:

[toplevel]
whoami = sts get-caller-identity --no-cli-pager --output yaml
profile = configure list-profiles

With that I have an alias to check IAM identity and available profiles. Then we can start configuring the profiles for CLI (in the file ~/.aws/config). Since we’ll be working with multiple AWS accounts, we have to manage multiple CLI profiles, which implies that:

We better use the --profile switch to explicitly specify profile being used, instead of relying on the AWS_PROFILE environment variable;
As a security best practice, we should not configure profiles with long-term IAM credential in the config file;
We must ensure the CLI doesn’t prompt for log-in every time we switch profile

To satisfy #3 there are many ways but we’ll discuss two: using cross-account IAM role, and using AWS SSO.

Bonus point if you enable auto-complete for AWS CLI.

Configure AWS CLI Profiles

With cross-account IAM role, the idea, is that the client start with one IAM identity, and use that IAM identity to assume roles on several other accounts. The configuration looks like this:

[profile jump_account]
credential_process = /opt/bin/awscreds-custom --username helen

[profile target_account_1]
role_arn = arn:aws:iam::123456789011:role/OrganizationAccountAccessRole
source_profile = jump_account

[profile target_account_2]
role_arn = arn:aws:iam::123456789012:role/OrganizationAccountAccessRole
source_profile = jump_account

In this example, you start with an validated identity in the jump account, then assume a privileged IAM role named OrganizationAccountAccessRole on the target accounts. Typically such IAM roles are pre-configured (e.g. in an multi-account landing zone) with appropriate trust policy to allow principals from the jump account. Once you’re validated as the IAM identity in the jump account, then you can use profiles for target accounts without being prompted for credentials again.

If your have configured IAM Identity Center for the multi-account environment, consider an alternative approach using sso login. The configuration usually looks like this:

[sso-session sso]
sso_start_url = https://myorg.awsapps.com/start/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile target_account_1]
sso_session = sso
sso_account_id = 123456789011
sso_role_name = AWSAdministratorAccess

[profile target_account_2]
sso_session = sso
sso_account_id = 123456789012
sso_role_name = AWSAdministratorAccess

This is often used by human users with SSO credential. In this example, to authenticate the sso session, start with command “aws sso login” . Then you can use all profiles by specifying --profile switch without having to log in again.

Configure SSH to EC2 via SSM

The EC2 instance must connect to Systems Manager endpoint, before one can SSH to the instance using the plugin. Once connected, you should find the instance in Fleet Manager. For this to happen, there are a few conditions. First, the instance must be able to reach the endpoint, either via public Internet, or via VPC interface endpoints if in a private subnet. Second, the instance profile must contain an IAM role with appropriate permissions. We can use AWS managed policy AmazonSSMManagedInstanceCore in the role. In addition, if we record the SSM session to an S3 bucket with encryption the instance profile must have permission to use the encryption key.

SSM agent uses the IAM role. The agent runs as a service on Linux or Windows machines. Many AWS managed AMIs come with SSM agent pre-installed. If that is not the case, you’d install the agent in your own AMI, or in user data which requires downloading the installer. With these configuration you’d be able to connect to the instance via SSM. We can use AWS CLI SSM command, or AWS web console to start an SSH session. To use SSH CLI utility, we install the session manager plugin along with AWS CLI on the SSH client machine. We also specify a public key for EC2 instance and run ssh command with the private key. The SSH configuration needs a configuration such as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

With this entry, you may directly SSH by instance ID (usually starting with i-* and mi-*), and the specified Proxy Command with SSM session document AWS-StartSSHSession will be invoked.

Configure Ansible Inventory

We can SSH to an instance (without port 22 open, on top of SSM) using the method above. Similarly, we can also configure Ansible to connect to the instance, without port 22. The inventory configuration looks like this:

mytest:
 hosts:
 instance1:
 ansible_host: i-00aabbffcc7755221
 ansible_user: ubuntu
 ansible_ssh_common_args: -o ProxyCommand="aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1"
 instance2:
 ansible_host: i-eedd88ff66aa22442
 ansible_user: ubuntu
 ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ProxyCommand="sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --profile target_account_2 \""

Note I used two patterns for ansible_ssh_common_args that are similar. Both works. Each entry references its own profile. This is necessary because Ansible does not have the knowledge which instance belongs to which profile’s account.

Another way to get Ansible connect to instances is to use the connection plugin community.aws.aws_ssm, by specifying ansible_connection: aws_ssm (e.g. as host variable) and other required variables (e.g. profile). This method does not require SSH channel but it requires an S3 bucket, and hence IAM permission on the controller node.

Ansible supports dynamic inventory, in three ways: inventory source file (with existing plugin), custom inventory plugin, and inventory script (in Python). Take source file as an example for EC2, add the followings as the content of aws_ec2.yaml:

plugin: amazon.aws.aws_ec2

# Attach the default AWS profile
aws_profile: target_account_1

compose:
 ansible_host: instance_id
 ansible_user: "'ubuntu'"
 ansible_ssh_common_args: "'-o ProxyCommand=\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1 \"'"

Then we can display the rendered inventory list, and Ansible-ping the instances.

ansible-inventory -i aws_ec2.yaml --list -y | less
ansible all -i aws_ec2.yaml -m ping

For more flexibility, for the composed variables, we could use jinja2 expression to generate the value. In both ways, we produce an inventory source per profile using the built-in inventory plugin aws_ec2. For greater flexibility, such as consolidating instances from all accounts into a single inventory, consider writing your own inventory script, or even own inventory plugin.

Conclusion

In this post we propose a way to manage instances across AWS accounts. Two main challenges are establishing the communication channel (SSH on top of SSM) and generating inventory data in Ansible. Some AWS services can generate inventory data, such as resource data sync in Systems Manager, or using AWS Config Aggregator. It is unfortunate that neither way produces the inventory data in a format that is directly compatible with Ansible inventory. Therefore, you might have to create a custom Ansible dynamic inventory script (in Python) that reads from the inventory data from AWS Config Aggregator (which supports AWS Organization). The side benefit of this script is that it is usually faster than the built-in aws_ec2 inventory plugin.

WordPress Security Basics

Fri, 17 Nov 2023 12:02:00 -0400

Background

In 2019, I moved this site to WordPress hosted on an Amazon Lightsail instance. There were few visits at that time so I lived with the single-server architecture. The website traffic has since been in steady growth but I have been too busy to catch up with the WordPress security setup. In July 2023, a malware impacted this site as well as the web traffic. It took me several months to fix a few related issues but the traffic still has not fully recovered. This post is about the lessons.

The Incident

I first noticed the issue when I clicked on links to my web page from Google result and got redirected to some spam site. It did not happen 100% of time, but it is annoying enough. In the mean time, from Google search analytics I noticed traffic volume going up with a lot of traffic going to URLs that I did not recognize or create. Somehow these URLs have a lot of clicks and impression counts. These are signs of artificial traffic.

Obviously the site was hacked. The first thing to determine is whether the server access was compromised. From the audit log (/var/log/auth.log and auth.log.gz) I can see a lot of brute force attempts to connect but fortunately none was successful. That also prompt me to change the default SSH port and use ECDSA key pair. Since the OS access is safe, the hack happens at the WordPress level. I suspected the sideloaded plugins from a few days ago. So I immediately removed all sideloaded plugins.

The attack is called malicious redirect. The plugin puts creepy pages in WordPress directory without my awareness and direct user traffic via my website. To clean up the damage, I looked into my WordPress directories at /opt/bitnami/wordpress and found many suspicious signs:

There are directories with weird names, such as rexall-vitalmin, or q4lee3, etc
In each of those directories there was an index.php file and .htaccess file;
Those directories also have other files which look like red herrings;
All those files have the same date time (from July 6);

Other directories to look at are /bitnami/wordpress/wp-content/plugins, where I noticed two directories (named gokyfozaxy and q199n071) that are not accounted for; and /bitnami/wordpress/wp-content/themes/, which contains unknown directories.

Clean up and hardening

In one of the .htaccess file I noticed segments of mojibake (garbled texts). I first tried to manually remove those files, but the problems stayed. Because the malicious redirect did not happen consistently on every single click, I sometime had false impression that the problem went away.

However, the challenges with manual cleaning are: 1. there are too many bad files (.htaccess and php files containing mojibake segments); 2. some existing files are impacted with mojibake segments too. I found a free plugin called WordFence to scan the file directory for malicious chagnes, and delete the bad files or bad segments. I also tried a paid scanner (Malcare) which found an bad file in /bitnami/wordpress/wp-content/themes/. However, it also blocked my site so I removed Malcare right away. Using the combination of WordFence and Malcare appears to have cleared up the offending files. After restarting apache, the bad URLs are no longer redirecting to spam sites.

This time, I decided to harden the WordPress system given the evidence of brute force attack at different point of entries. At OS level, I mentioned the changes to SSH daemon configuration. At WordPress level, I used WordFence to run several scans for problems and improve the security posture, including enabling MFA for the admin user. I also noticed a few unrecognized wordpress users and used wordpress CLI to delete those and other unused users.

The wordpress.org website has some general guidance on what to do when a site is hacked, and a general guidance on hardening WordPress.

Back Links

Another clean up work I had to do is dealing with back links. Back links are URLs from other sites that references this site. There are several situations:

If it’s a made-up URL, then it returns 404. In my case, these are URLs that stopped working once I cleaned up my server from the incident. However, the sources are still using these bad URLs. They are bad back-links;
If it’s a legit URL, look at if it’s hot linking, such as another site directly access an image from my site. These are bad back-links;
If it’s legit URLs, and the referrer site has a good domain authority score. These are likely to be good back links

Generally, it is painful to deal with bad back links because I’m not in control. I used a few free backlink checker tools (e.g. Links report on Google Search Console, SEOMATOR, SEMRush free) and found a lot of spammy sites that I had to request Google to disavow. Otherwise, they may negatively impact the search performance.

Repercussions

In the next few months, my pages are no longer a stop for their redirect. However, web request for those invalid URLs keep coming. The bad pages are still in Google’s cache. There are a lot of page request with 404 return code, and we consider this an HTTP flood. The problem now is that the amount of 404 return code is impacting how my site ranks in search engine. To make it worse, the amount of these requests with invalid URL increase since August.

To fix this, there are two measures. First, in Google search console, I have to tell Google to remove those URLs from its cache. I have identified a number of prefix patterns, and submitted a request for each URL pattern. It takes google a day to have them cleared.

After that, the bad request will no longer come from Google users clicking on bad URL. In my case, the requests did not reduce significantly, suggesting that most of the requests come from bots. Therefore I had to figure out a way to prevent those bad request hitting my server, which is a typical web application firewall requirement.

Looking for such a solution for my WordPress Security I landed on Cloudflare. Cloudflare is pretty user-friendly with an easy-to-understand reference architecture. When I started, Cloudflare can import my DNS records, and guided me to change my name servers so I delegate my DNS management it. When I first move to Cloudflare the website gives ERR_TOO_MANY_REDIRECTS. I ended up having to go to SSL/TLS and set encryption mode to Full (strict) to get rid of this error. I also have to re-configure email forwarding as a result of name server change.

CloudFlare

Even for a self-hosted single-server site, it is very beneficial to place an Application Firewall upfront for WordPress security. I find Cloudflare to be a very useful service that provides everything else you need to host the web site. For example, it contains a domain registry itself. It manages DNS and allows email forwarding. In addition, it helps generate TLS certificate etc. The free tier covers everything for a small website, with the Application Firewall as the core feature. Within the free tier I can have these features:

domain registrar and name servers (not for free but at a reasonable cost)
SSL certificate (not for free but at a reasonable cost)
Request event tracking
redirect rule: zone apex to www, and /status to uptime status page
return code 409 for obsolete URLs (using routes and workers)
email routing and forwarding
WAF rules (path, parameter, rate, etc)
DDoS protection and Bot Fight mode
hot-linking prevention (i.e. other sites references images on your site directly)

I am still exploring features for CloudFlare. One stunning feature is routes and workers. Essentially you can serve a function in response to HTTP request at a specific route. This is particularly useful in scenarios where it is not straightforward to add web pages on the backend server. For example, I want requests with certain paths to return HTTP code 490 and do not want to mock with the WordPress server, we can make use of CloudFlare worker.

Lessons Learned

For WordPress security, never use suspicious plugins. Keep an additional layer of defense in WordPress such as Wordfense. It helps block malicious traffic that went through the first layer. It also helps configure MFA for administrators.

AWS Systems Manager is an Omnipotent Hodgepodge

Sun, 29 Oct 2023 21:32:49 -0400

Introduction to Systems Manager

AWS Systems Manager addresses a lot of SysOps requirements for configuration management, including server automation. In this domain, there is another AWS service called OpsWorks. However, with OpsWorks Stack, OpsWorks Chef and OpsWorks Puppet all coming EOL in 2024, the entire OpsWorks service is mostly deprecated. By partnering with leaders such as Chef and Puppet, OpsWork services represent the era when AWS needed to mirror the configuration management capability on premise, in an effort to convince customers migrating to the cloud. Today, AWS Systems Manager has evolved to fill a lot of gaps around configuration management for servers in the cloud.

Although AWS Systems Manager sounds like a single service. It consists a collection of many seemingly disparate capabilities that serves similar requirements around configuration management. In fact, many of the Systems Manager capabilities are built on top of a couple of what I call core capabilities, such as Session Manager, RunCommand, Automation. This post will review these core capabilities and how Systems Manager employs them to expand with other capabilities.

SSM Agent and Session Manager

What enables all other capabilities is the SSM agent installed on the EC2 instances. The agent running as a systemctl task by ssm-user on EC2 instances. Most of AMIs come with this agent pre-installed. It stores the logs in /var/log/amazon/ssm/. This agent works with an instance profile with a role with the AmazonSSMManagedInstanceCore managed policy, in order to communicate with AWS Systems Manager (ssm.<region>.amazonaws.com) backend. Because of that, you also need to provide a network path to the backend endpoint, either via Internet, or interface endpoint.

This communication also allows an IAM user to connect to an instance’s shell. A common use case is for private instance that do not have Internet access but do have access to SSM backend endpoint. In a previous post I discussed using Session Manager to replace a bastion host to connect to EKS nodes.

When launching an instance using an AMI with SSM pre-installed, the SSM agent should launch after all the config sets from Cloudformation Init are finished. As a result, the Cloudformation Init script is not able to communicate with SSM backend via the agent, unless you install and start SSM agent first on your own, in CloudFormation Init. To troubleshoot SSM, it is important to review its logs.

Through Systems Manager Hybrid Activation, the SSM agent can also work on virtual machines out of AWS and report back to with SSM backend. This gives on-prem servers the identities (instance tags, instance profiles) required for Systems Manager to manage them as if they were EC2 instances. As a result, extend Systems Manager capabilities to on-prem fleet (requiring advanced instances tier).

Types of SSM Documents

There are several types of document that SSM uses, including:

Command Document
Automation Document
Package Document
Session Document
Policy Document
Change Calendar Document

The AWS documentation has a table on what they each are for. Here I’ll focus on three types of documents: Command Document, Automation Document and Session Document.

The Command Document is for the RunCommand capability. It executes on EC2 instances usually performing tasks relating to the operating system or application. I think of a Command Document as an Ansible Playbook that consists of Ansible tasks. We can author Command document that runs configuration steps using plugins, such as aws:downloadContents, aws:runShellScript, etc. This feature directly competes with Ansible. To troubleshoot why a command fail on an instance, check the file ssm-document-worker.log in the ssm agent log directory. Each log entry should have a command ID as reference.

The Automation Document (aka runbooks) is for the Automation Capability. You can define sequence of actions for automation. There are many pre-defined actions such as executing AWS API calls (aws:executeAwsApi), run commands (aws:runCommand), or executing a Lambda function. Therefore a runbook requires an IAM role (Automation Role). The schema of action sequence (YAML or JSON) looks very similar to an Ansible playbook. The web console comes with an UI to visualize the action sequence but most of the time I’d rather .

Session Document is for Session Manager capability. AWS Systems Manager Session Manager uses Session documents to determine which type of session to start, such as a standard session, a port forwarding session, or a session to run an interactive command. In most cases, automation developers do not need to create their own Session document, because the pre-built ones are sufficient:

AWS-PasswordReset
AWS-StartInteractiveCommand
AWS-StartPortForwardingSession
AWS-StartPortForwardingSessionToSocket
AWS-StartSSHSession

In my experience, I use the AWS-StartSSHSession and AWS-StartPortForwardingSession documents most often. To establish SSH connection for forward port to connecting host for Remote Desktop session.

To author your own document, reference the schema correctly and use the latest SSM agent. However, I would explore if any existing shared document in the library already covers what you need. For example, the command document AWS-JoinDirectoryServiceDomain help join a Windows server to a managed Active Directory domain. The command document AWS-RunPatchBaseline is used by Systems Manager Patch Manager capability to check and apply operating system patches. They include steps for Windows, MacOS and Linux instances. The automation runbook AWS-AttachIAMToInstance helps you add IAM role to an EC2 instance.

RunCommand and Automation

The Run Command capability run on top of SSM agent. You can specify one or more target instances. You also specify other options such as command parameters, rate control and where the output goes. This capability allows an IAM user to run command directly on the OS of an instance (using an OS user ssm-agent) and centrally keep track of those command runs on the AWS side. The most common commands to run on the OS are packaged into Command Documents. There is even a Command Document that allows you to run a pre-built Ansible playbook.

Another way this capability is extremely helpful, is that we can reduce the load of cloud init process. Traditionally, we put a log of logics in the user data script for the cloud init process to execute. The cloud-init mechanism comes from Linux OS and the execution of the user data script is not very transparent to troubleshoot. You have to check the cloud-init-output log from the OS. The use of the UserData script should be reserved for establishing communication with CloudFormation endpoint and SSM endpoint. From there, other automation tasks should be done using SSM capabilities (e.g. State Manager) for better manageabilities.

Take an example of joining a newly provisioned Windows server to a domain. If we do this in user data script, we will have a few problems. First, we can only tell success/fail state from logs in the OS. Second, if an OS user inadvertently removed the instance from domain, there is no mechanism to capture that. If we use Systems Manager’s RunCommand capability, along with State Manager association, the AWS management console will be able to tell whether domain joining is successful, and the association can detect when the instance is removed from domain, report this finding as out of compliance, and remediate the issue. We’ll discuss State Manager in more detail in the next section.

As part of automation, we often have to invoke AWS API calls, which happens outside of any target VMs. The Automation capability of Systems Manager is for this scenario. You can orchestrate your API calls using Automation runbooks. These automation steps do not execute on any target EC2 instance, so they do not rely on SSM agent. However, it needs its own IAM role to perform API tasks. This capability saves you from having to run API calls by creating a new Shell environment to run AWS CLI, or from your own Lambda function using the boto3 SDK library.

When we combine Automation and RunCommand capabilities, we can perform most of the automation orchestration steps. They are the core capabilities that further enable a variety of other Systems Manager capabilities.

Maintenance Window and State Manager

Maintenance window is a very straight forward capability to schedule RunCommand activity with a cron or rate expression. You can specify target by instance tags, define one or more tasks, and define a window of activity and at what point prior to the end of Windows should the agent stop performing more activities (cutoff). Each task can be a type of a RunCommand command, Step Function, Lambda function and automation runbooks.

State Manager is a similar capabilities with a lot of feature overlap with Maintenance Window. State Manager operates on the concept of associations. An association connects target instances to command document or automation runbook to execute. Similar to Maintenance Window, you can specify a schedule expression, document parameters and instance tags. State Manager was brought in to combat configuration drift. The associated document should consist of idempotent scripts so that a State Manager association can repeatedly execute these documents to ensure compliance.

Maintenance Window is more about scheduling one or more tasks. On the State Manager side however, association failure by default will be reported as out of compliance. This is useful in scenarios such as keeping a Window instance in the domain, or keeping SSM agent up to date. You can choose either capability for many common setups but they have subtle differences. For example, for Patch management, you can use State Manager to detect missing patches and report compliance, and Maintenance Window to actually apply the missing patches. In fact, there is a document page on choosing between State Manager and Maintenance Windows to distinguish their best use cases.

Fleet Manager and Inventory

Fleet Manager presents a centralized view for all instances for users to perform common administration tasks, such as exploring file systems and logs, admin users and groups, manage registry and events on Windows instances, check processes and performance metrics. It also gives shortcuts to patch nodes, run commands, start session, etc. I think of Fleet Manager as a minimalist configuration management UI. It is not as sophisticated as those from Ansible Tower or Puppet but it comes at no additional cost.

A very useful feature of Fleet Manager is to run a web-based remote desktop to connect to Windows Instances. This saves the need for a bastion host as long as the instances have SSM connection. You will need the RSA private key to decrypt the Administrator password, which I would not recommend. If the Windows server is on a domain, you can enter your domain credential via Fleet Manager. If the users logged in via IAM identity center, Fleet Manager also has the login option for them via SSO using IAM Identity Center identity. When a user logs in this way, Fleet Manager uses RunCommand capability to execute AWSSSO-CreateSSOUser document against the server to create a local admin user.

Another aspect of configuration management is the inventory management. Unlike in Ansible, the term inventory in the context of Systems Manager refers to the metadata of instances, which includes installed applications, AWS components, network configurations, instance details, services, Windows registry and roles, etc. The full list of what is part of metadata is in the document and you can even define your own inventory item. To gather inventory data, we can makes use of a State Manager association to execute the AWS-GatherSoftwareInventory document. Once we set up the association, the agents will report inventory data back to Systems Manager. More importantly, we can create Resource Data Sync objects to write inventory data (along with compliance data) to S3 buckets, allowing downstream applications to consume. A common use case is to run Athena query against those bucket and produce QuickSight dashboard.

Patch Manager and Compliance

The Patch Manager also operates on State Manager associations. The automation runbook is AWSRunPatchBaseline, where you can just scan for missing patches or install them as well. The SSM document can run on all three platforms (Windows, Linux and MacOS) and determines which patches are missing relative to a the Patch Baseline. There should be at least one default Patch baseline. Each OS (e.g. Ubuntu, Debian, Amazon Linux, etc) classifies patches differently, and a patch baseline is a configuration that defines whether a patch is approved based on operating system and their classifications. The automation document also allows you to override the patch baseline. When executing the document to scan for patches, it records patch compliance information using the PutInventory API command. When using the document to install patches, you can run the document from a Maintenance Window and specify whether you need to reboot the target instance if required.

The compliance capability reports compliance status for instances. By default there are two types of compliance: association and patch. The association compliance detects whether a state manager association is failed on certain instances. The patch compliance, as just mentioned, checks whether patches are up to date relative to the specified patch baseline. You can also define custom compliance item (with put-compliance-items API) but the documentation isn’t clear on what exactly it can achieve and where on the instance does it pull the compliance status. From the example in put-compliance-items, custom compliance type seems to check the installation of additional software package in the inventory.

Other capabilities

Amongst the other capabilities, the one I use the most often is parameter store, which is a way to store a variable for different services to consume.

In the domain of change management, the change manager is a mini change management system. Organization can use it to manage their change process such as approvals. More importantly, you can fire automation runbook from change manager and tie it back to the change control item. Change calendar allows you to block changes during specific period. Both of them are organization level capabilities.

When it comes to operations management, the Incident Manager capability allows you to create response plan for incidents. Response plan can execute runbook actions once an incident is logged. It also helps you notify the on-call incident response team. On the other hand, OpsCenter capability allows you to create OpsItem, which also includes a way to execute runbook. The OpsData can aggregate to Explorer, which is a centralized dashboard for operations data. The Explorer, OpsCenter and Incident Manager capabilities can operate at organization level.

These capabilities around change management and operations management come nowhere close to full-fledged ITSM solutions such as ServiceNow or SMAX. However, they have the ability to trigger runbooks and natively integrate with other AWS services.

There is also a quick setup capability which uses pre-baked CloudFormation template to configure other services. For Patch manager the current recommendation is to use quick setup to configure patch policy.

Summary

Systems Manager has so many capabilities that I cannot cover everything in a single post. Here is a good walk-through. Some capabilities like session manager, fleet manager and state manager, are extremely helpful. However, in my opinion, there are two problems with grouping all these capabilities under Systems Manager. First, With too many different capabilities, this service lacks focus, which makes it difficult to learn. Second, some capabilities have overlap with other capabilities, or another AWS services, which also makes it confusing. I try to sort out how these capabilities enable each other in the diagram below:

This diagram may not be 100% accurate but it demonstrates the dependencies and can assist troubleshooting. For example, when compliance is missing data, check the execution history of run command. It also illustrates the key role of SSM agent as the underlying enabler of most of the other capabilities.

Overall, Systems Manager is extremely powerful. You can try to replace your server management solutions (e.g. Ansible, Chef and Puppet) with Systems Manager configurations. With a good understanding of its capabilities, you can build your fleet automation in an efficient and scalable way.

Authentication to kube-apiserver via OIDC

Fri, 28 Jul 2023 09:20:00 -0400

Background

There are many benefits of using OIDC to authenticate to kube-api server, especially with multiple clusters that need consistent log-in experience. With the last post on how OIDC Authorization Code Flow works, now I will discuss options with authentication for kubectl to connect to kube API server.

To start, let’s look at the anatomy of kubeconfig file. The full schema is in the documentation. Looking at my kubeconfig file, there are three sections:

clusters: each entry specifies a cluster’s name, server address and certificate authority data (in base64 encoding or a file location).
users: each entry specifies a username. Some users are identified with client key and certificate. Some specify a command to provide client authentication. Refer to the authentication strategies.
contexts: each entry links a user to a cluster

Therefore, the key to use OIDC integration, is to use command to provide client authentication.

Vanilla Kubernetes

The documentation on authenticating has a diagram on how to use OpenID Connect tokens. The diagram does not give details on how access_token and id_token were obtained. So it could be any OIDC flow (Authorization Code Flow, Implicit Flow, etc) as we have discussed.

Although the instruction does not mandate which OIDC flow to use, we should use Authorization Code Flow in this architecture. The API server needs to trust the OIDC issuer, and the document covers how to configure API server.

In the diagram, step 2 and step 3 are required by kubectl itself does not perform these activities. All kubectl does is carry the JWT token in the Authorization Bearer. Vanilla Kubernetes does not provide a solution for OIDC integration. It only provides some instructions and we still need some helper scripts to glue all these instruction steps together.

There are many open-source project for this purpose. For example, Jetstack has kube-oidc-proxy and Int128 developed kubelogin. Other projects such as k8s-auth-client, k8s-oidc-helper, and gangway are no longer being updated. The kubelogin project remains influential. It has a clear diagram too:

From the diagram we can see kubelogin proposes authorization code flow.

Also, one design concern with Kubernetes control plane is the placement of endpoint. From this diagram we can see that even if the cluster endpoint is on private subnet, OIDC integration should still work. The control plane (specifically kube-apiserver) initiates outbound connection to OIDC Provider. There is no inbound connection to it from the OIDC provider.

In the kubelogin setup, the redirect URI is set to localhost:8080 because it stands up a server on the same host where browser is running. The browser can always resolve localhost. For a full configuration steps, Okta has this blog post on how to use kubelogin as helper, and Okta as Authorization Server to authenticate kubectl via OIDC.

Step-by-step with kubelogin

We’ll go through an example with int128/kubelogin because it works with any Kubernetes flavour, including managed Kubernetes services. It is also fairly simple. The instruction covers a few types of Authorization Servers (Google Identity Platform, KeyCloak, Dex with GitHub, Okta and Ping Identity). I’ll take KinD cluster as an example and use Azure AD as Authorization Server.

First, we’ll register an App in Azure Portal. Go to “App Registrations” and “New registration”. Give it a name “kubeoidc” and set Redirect URI to “Web” with URL “localhost:8000”. Click on Register.

The next page shows the details for this app. The Application (client) ID is important for next steps. Click on “Add a certificate or secret”, then “New client secret”, put in expiry and description. The secret value is generated and displayed on the next page, which is important for our next step.

We also need to find out the issuer URL. From the App page above, click on “Endpoints” and find out the URL from field “OpenID Connect metadata document”. My metadata document URL looks like:

https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0/.well-known/openid-configuration

The OIDC Issuer URL is the part before .well-known. In this case, it is:

https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0

Now we have collected what we need for the next few steps: ClientID, ClientSecret and OIDC Issuer URL. We can then create the KinD cluster, and reference ClientID and IssuerURL in the cluster configuration:

cat << EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
 - role: control-plane
 kubeadmConfigPatches:
 - |
 kind: ClusterConfiguration
 apiServer:
 extraArgs:
 oidc-issuer-url: https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0
 oidc-client-id: 99999e88-e777-6666-c5c5-c444444d3d22
 - role: worker
 - role: worker
 - role: worker
EOF

kind create cluster --config=kind-config.yaml

The command will create cluster and configure kubeconfig file with a user named kind-kind as admin, which isn’t what we need. Now we use kubelogin helper command:

kubectl oidc-login setup \
 --oidc-issuer-url=ISSUER_URL \
 --oidc-client-id=YOUR_CLIENT_ID \
 --oidc-client-secret=YOUR_CLIENT_SECRET

This command will act as the client in the OIDC flow and prompts you to log in to Azure in a Browser. Once logged on, it gives you the next few commands to run. Since we’ve already created a cluster with the Issuer URL and Client ID, we can skip creating cluster, and run the steps to:

bind the Azure user to a ClusterRole
set up the kubeconfig locally with a user oidc, which needs to execute the oidc-login command

Lastly, we can test the oidc user with kubectl --user=oidc get nodes. We can also set the context to use oidc user by default. Voila.

From this example, we learned how to configure OIDC integration for any Kubernetes distros. The steps that we need to take are:

On the cluster side, we customize kube-api-server flag with OIDC provider info. Specifically, we provided Issuer URL and client ID in this example. In the OIDC Flow diagram, this step establish a trust from the Resource Server (K8s cluster) to the OIDC provider.
On the kubectl side, kubectl itself cannot fulfill all the duties of a client app in the OIDC Flow. It needs a helper script and we’ve made friend with int128/kubelogin.
On the Identity Store side, we expect it to be an OIDC-compliant Authorization Server. Otherwise, we consider using Dex as a broker in between.

When it comes to managed service, many allows us to customize the OIDC related flags for kube-api-server. Let’s look at how some managed services get this to work.

Azure Kubernetes Service

I find the kubectl authentication with AKS highly opinionated in its documentation. The recommendation is using Azure Active Directory as identity store but I don’t find it work with other OIDC providers.

To work with Azure Active Directory, you would configure the cluster and specify Azure role-based access control with the Group UUID in Azure AD. On the client side, you will need to install Azure’s kubelogin utility. Do not confuse it with int128’s kubelogin, which is for any cluster. This kubelogin is just for Azure. Once installed, you can use az-cli command to update your kubeconfig file, which call this utility from kubeconfig. To examine details about client configuration and AAD integration, check out the Terraform template in the azure directory of my cloudkube project.

The document only covers Azure AD integration and I tried to find if there’s a way to integrate with third-party OIDC providers. Unfortunately I have no luck. ChatGPT points me to a page about enabling OIDC provider but it is in the context of workload identity and it does not allow you to customize the issuer. So it’s completely irrelevant. Because you cannot customize OIDC issuer, etc, it simply won’t work with any third-party OIDC provider. Sure enough, most of Azure’s client use Azure AD anyways.

Azure makes it streamlined to configure OIDC integration of AKS with Azure AD, its own identity store. To my disappointment, it is currently not possible to integrate with third-party OIDC provider for authentication at cluster endpoint.

ROSA

RedHat OpenShift on AWS (ROSA) is a. However it reflects how OpenShift configures third-party identity provider.

Let’s first create a cluster off custom VPC and private endpoint. This is covered in my previous post about ROSA. After the cluster creation, we’re at the point where we can run oc command against cluster endpoint from Bastion host because it is a private cluster. However, being a private cluster is irrelevant to how we configure OIDC integration.

We’ll use Azure AD again as OIDC provider. So we need to register an app the same way as I did above in the kubelogin example. We need to have Client ID, Secret and OIDC issuer URL. For RedirectURI, go to your OpenShift console, and under Cluster, click on the cluster name → access control → identity providers → select OpenID. Note the page clearly states that this is Authorization Code Flow, and the OAuth Callback URL is provided. Use it to regiter App in Azure, and fill in the page with Client ID, Secret and OIDC issuer URL. Also fill in other fields accordingly and click on Add. Now you should have it configured!

Identity Providers for OpenShift cluster

The next step is trying to login. From Bastion host, run oc login and it will give me an URL to use. The URL contains the cluster endpoint, which resolvable from the Bastion Host itself. However I need a Browser session here, so I have to run Bastion host as SOCKS5 proxy and tell Chrome on my MacBook to use it:

~ open /Applications/Google\ Chrome.app --args --proxy-server="socks5://localhost:1080"

The browser session redirects to Azure AD for log in. Once completed the webpage will display a token that I can use with oc login command. Run this command with token from Bastion, I’m logged in:

oc login --token=sha256~3ZT5JGWELOcBzfohftEm9D2UwoOVFvATASuZk3_uxps --server=https://api.dhc.62q3.p1.openshiftapps.com:6443
oc whoami
oc get no

At this point, if I run oc whoami, I get the user name. However, this user cannot do anything. This is because it is not associated with a role yet. You grant more permission to this user: go back to OpenShift console, Clusters → ClusterName → Access Control → Cluster Roles and Access → Add user. Here you can map the user name to a role (let’s say ClusterAdmin). Then this user will have its priviledge:

The whole Flow works with private cluster, because the redirect URI is resolvable from the Bastion host. If you chose to expose cluster endpoint publicly (not recommended), you can perform the above steps directly from your MacBook or Laptop.

So the ROSA experience has been smooth. Unlike kubectl, the oc utility can act as the Client App in Authorization Code Flow. The other part of the configuration such as client secret and issuer URL are made in OpenShift console. Good job!

Elastic Kuberentes Service

EKS allows you to specify OIDC issuers from console or CLI to set up third-party OIDC configuration. There is a blog post from Okta on this, which works for private clusters. In the instruction, the author first manually created kubeconfig file with int128 kubelogin, and then bind ClusterRole with the user.

The blog post is very detailed. Instead of repeating it, I would like to discuss two SSO models available in AWS. I summarize them as below:

Both are SSO models for EKS. Model 1 (IAM Identity Center) is home grown as AWS using IAM Identity Center (formerly AWS SSO). Users start with an IAM principal (AWS construct) and use the aws-auth config map to tie them to Kubernetes groups. This AWS blog post and this support article are based on the IAM Identity Center model. On the other hand, Model 2 (Direct OIDC) is the vanilla Kubernetes approach. It takes group claim from OIDC identity token. The Okta blog post is based on this model.

The IAM identity center model works with SAML compliant identity providers, oftentimes Active Directory, although there seems to be a plan to support OIDC-compliant identity provider as well in the future. Even if it was supported today, I’d still prefer model 2 for working with any OIDC compliant identity provider because it’s simple. Why not leverage K8s’ native capability? For identity providers that do not support OIDC natively, or does not issue group claim (e.g. Google workspace), as we discussed, we can also consider alternatives such as Dex as identity broker. However, this model comes handy when a pipeline user with IAM role needs to authenticate into EKS.

One should always go for the direct OIDC model, if the upstream identity provider supports OIDC. The provider itself can even be AWS Cognito User pool. Cognito User Pool itself supports federated identity but again, I would directly connect EKS cluster to the OIDC compatible identity provider, instead of going through Cognito User Pool. As a result, the only use case where Cognito user pool is absolutely necessary, is when we need the Cognito user pool itself as the identity provider, as we have in CloudKube‘s eks directory.

Summary

Configuring OIDC provider for cluster endpoint can be confusing and we need to understand how OIDC flows work. I dived into OIDC in the previous post and in this post, I explained how to get it to work with vanilla Kubernetes. I summarized the three requirements and looked at the OIDC provider option with some managed services. Then I went through OIDC options for AKS, ROSA and EKS.

Connect kubectl to private Kubernetes cluster in EKS and AKS

Sat, 10 Jun 2023 19:31:00 -0400

Managed Kubernetes services give user a cluster endpoint and a number of worker nodes, with the choice. For each access, users have the choice of making them publicly available, or keeping them on private networking. In my opinion, any deployment beyond personal hobbies, should use Kubernetes private cluster, with both cluster endpoint and worker nodes on private subnet. There is no reason to expose computing nodes or Kubernetes management traffic publicly.

For worker nodes, it is fairly easy to put VMs on private network, but many companies still have the cluster endpoint exposed publicly. There are usually two reasons. First, their CI/CD agent is hosted somewhere else on the Internet (instead of on private network with private connectivity to Kubernetes cluster) and need to access Kubernetes cluster endpoint. Second, when the cluster needs to connect with third-party identity provider as OIDC provider, a two-way communication is needed.

There is a classic pattern of using a public bastion host (jump box), with a bastion host on the public subnet, routable to the private endpoint of managed Kubernetes service. Clients then connect to the bastion host via port 22 on a public IP address. The authentication is based on SSH key pair, or worse, password. The port forwarding (aka SSH tunnelling) capability enables all the magics.

Exposing a jump box in the public subnet with RSA key authentication is still not favourable. In this post, I’ll examine some secure patterns to connect to private endpoint with improved security posture.

AWS options

There are two problems. First, how to establish connectivity to the Bastion host in a private subnet. Second, how to use the Bastion host to proxy traffic to the cluster endpoint also in private subnet. To the first problem, there are two potential solutions: SSM Session Manager, and EC2 Instance Connect (EIC) with EIC endpoint (EICE).

SSM Session Manager was introduce in 2018. It runs an agent on the EC2, which initiates a connection to the SSM endpoint on the AWS side. This connection enables not only Session Manager, but also other Systems Managers (SSM) services such as Fleet Manager, Patch Manager and State Manager. The problem that session manager originally addresses is server management.

AWS launched EC2 Instance Connect (EIC) in 2019, and EIC Endpoint (EICE) in 2023. EIC addresses the problem with managing SSH key pairs at scale. It dynamically generates an SSH key pair for server access, based on IAM permission. However, it still requires an instance to have its SSH port publicly accessible. With EICE, it is no longer a requirement. In the diagram, EICE is placed in a private subnet, allowing EICE service to reach private instances at their SSH port.

Here is a comparison of the two:

	EC2 Instance Connect (EIC) with EIC Endpoint	SSM Session Manager
Location of Bastion host	Private Subnet.	Private Subnet
Need Ingress Port	Yes. Port 22 must open to the endpoint.	No. SSM agent initiate outbound connection from the instance
Traffic Path	AWS CLI → AWS EIC ES → EICE→EC2 Inst	AWS CLI → AWS SSM ES → SSM ← EC2 Inst
Authentication	AWS IAM and ephemeral SSH key when using AWS CLI directly AWS IAM and long-term SSH key when using SSH proxy command	AWS IAM and long-term SSH key when using AWS CLI directly or SSH proxy command
Work with OpenSSH	Yes	Yes
Cost	There is no additional cost for using EIC.	No additional cost, unless private SSM Endpoint.

Let’s take a look at each option.

EC2 Instance Connect

To use EIC, pick an AMI that has it pre-installed and ensure instance profile has correct policy, as the document states here. AWC CLI will make use of local OpenSSL client. So make sure there connection at port 22 is open. To make it work with EC2 instance on a private subnet, create an EC2 Instance Connect Endpoint on the VPC, and ensure that the security group of EC2 allows port 22 from the Endpoint. Run this command:

$ aws ec2-instance-connect ssh --instance-id i-00ea30a6e02db33fe

The command above simply generates a key pair internally, add the public key to the server side, and connect with SSH from the client side. The command takes you to an SSH session. Checking ps -ef | grep ssh on the client machine, you can see the full parameter of SSH, including the location of the ephemeral private key.

However, if you use AWS CLI open-tunnel as proxy command to ssh, then you’d still have to use the key pair used to create the EC2 instance. As suggested at the bottom of this blog post, the command is:

$ ssh ec2-user@[INSTANCE] -i [SSH-KEY] -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id %h'

This is a bummer, because with native SSH tool you do not get the primary benefit of EIC – ephemeral key pair.

SSM Session Manager

Now let’s look at SSM session manager. Similarly, it needs an agent installed and IAM role configured. You can connect to from web console but more importantly, from AWS CLI:

$ aws ssm start-session --target i-0531b19bec8ad022d

This command takes you to an SSH session with user ssm-user, without starting an OpenSSH client process locally. User do not have to manage key pair. There is also a document about using this command as proxy command, which uses an SSM document. I have one of the SSH config entry as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

This allows me to directly SSH to client using OpenSSL client (e.g. ssh i-0531b19bec8ad022d) by Instance ID. With this, I also need to specify my own OS user and matching private key.

I know I will use the OpenSSH client a lot from pipelines because it is very powerful. In both options, I have to live with managing key pairs myself. With SSM session manager’s proxy command, the instance does not need port 22 to open, which is a great advantage, in terms of security and operation. SSM Session Manager is a winner.

SOCKS5 proxy for kubectl

Either SSM Session Manager or EIC with EICE enables an SSH tunnel with key encryption between client (a local computer or a pipeline agent). On top of the SSH tunnel, we can build a SOCKS5 proxy. Kubernetes document has a good page on how to do this. I managed to get this to work with a few gotchas.

To put this in practice, I first created a VPC stack with a bastion host using terraform template from my vpc-base project. The terraform output will give the next set of commands to run to create a private cluster, using a manifest rendered from the file private-cluster.yaml.tmpl:

# cd aws_vpc
# terraform init
# terraform plan
# terraform apply
# ... run the given command ...
# envsubst < private-cluster.yaml.tmpl | tee | eksctl create cluster -f -

# Run this from a remote host without access to cluster endpoint.
# Run terraform apply and terraform output contains the variables needed for the next steps

# the command below may take 15 minutes to create a private cluster

eksctl create cluster -f private-cluster.yaml 

aws eks update-kubeconfig --name private-cluster

At this point, the kubeconfig file has been updated, but kubectl (from Internet or on-prem) is unable to connect to cluster endpoint (on private network). In order to

BASTION_SECURITY_GROUP_ID=$(terraform output -raw bastion_sg_id)
CLUSTER_SECURITY_GROUP_ID=$(aws eks describe-cluster --name private-cluster --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text)

# In Cluster Endpoint's security group, open up port 443 to Bastion host
aws ec2 authorize-security-group-ingress --group-id $CLUSTER_SECURITY_GROUP_ID --source-group $BASTION_SECURITY_GROUP_ID --protocol tcp --port 443

# Test with connecting to Bastion host with ssh i-0750643179667a5b6, assuming .ssh/config file is configured as above. From the bastion host, you can test:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api 
# if you get an HTTP response, even an error code 403, the bastion host has TCP connectivity to cluster endpoint

# then we can start an SSH session as a SOCKS5 proxy on the remote host
ssh -D 1080 -q -N i-0750643179667a5b6

# add > /dev/null 2>&1 & to push it to background, or use ctrl+z after running the command
# to validate that the SOCKS5 proxy is working, you can run the same curl command with a proxy parameter:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api --proxy socks5://localhost:1080

# you can instruct kubectl to use the SOCKS5 proxy with the following environment variable
export HTTPS_PROXY=socks5://localhost:1080

kubectl get node

# alternatively, add "proxy-url: socks5://localhost:1080" below server attribute in ~/.kube/config file.

There are some pitfalls to watch for. On the remote host both ssh command and kubectl command implicitly uses AWS CLI. Therefore, make sure the profile and IAM role are correctly configured. For example, if SSM agent requires one IAM role, and kubectl is created with another IAM role, then make sure AWS CLI assumes the correct IAM role using environment variables, and use “aws sts get-caller-identity” to validate the IAM identity being used.

What about AKS in Azure

I touched on this in my Azure notes in 2021 and did a research again. Unfortunately, options are still fairly limited.

The first option is to use a managed service called “Azure Bastion”, which requires public IP and a dedicated subnet with the exact name of AzureBastionSubnet, as well as some additional requirement. I’m not impressed with these requirement because it is meant to be a managed service. The other option, is essentially to DIY a JumpBox. The idea is the same: put the jumpbox in a public subnet, which is routable to private subnets. When you need to connect to private VMs, get to the jumpbox first.

Apart from having to put the bastion VM on a public subnet, the pattern that we discussed above involving SOCKS5 proxy still works. Exposing a bastion host isn’t ideal but it still reduces attack surface significantly, comparing to exposing the cluster endpoints of all Kubernetes API servers.

Summary

Many immature Kubernetes configurations exposes private endpoint publicly. Having cluster endpoint in private subnet greatly improves security posture. In my opinion, there are very few situations where cluster endpoint must exposed publicly. Having private endpoint should be mandatory for all Kubernetes cluster. In the next post, I also cover how to create a ROSA cluster with private endpoint.

Kubernetes with Multiple CPU Architectures 2 of 2 – Node and Workload

Sat, 20 May 2023 01:34:00 -0400

The most common server CPU architectures today are amd64 (aka x86_64) and arm64. Although AMD developed the former first, Intel names it as x86_64 (or x64 for short). In terms of compatibility, they are the same. In general, arm64 architecture consumes less power and therefore mobile systems first favour it. Its power efficiency now drives a trend towards computing infrastructure. For example, Apple’s MacBook moved to M1 processor in 2020. Since 2018, Amazon’s Graviton processor has entered the third generation. In 2022, Azure also brought Ampere Altra processor, and GCP introduced ARM based VMs. Less power consumption ultimately leads to less computing cost.

I can only see more workloads gradually move to servers with ARM architecture. With Kubernetes, we will most likely have a fleet of computing node consisting of hybrid CPU architectures. We can take a look at what the arm64 adoption entails for workloads on Kubernetes.

ARM64 architecture

Graviton processor is on 64-bit Arm Neoverse cores, targeting for optimizing cloud-native workloads. Currently at AWS, the majority of arm64 instances use Graviton2 processor. This AWS blog posted the news about Graviton3-based general purpose (m7g) and memory-optimized (r7g) EC2 instances. At the bottom, there is a chart that compares the performance of Graviton3 with Graviton2, x86 and M6g instances. We can expect that in a few months the services that supports Graviton2 processor to start supporting Graviton3 processor.

In the serverless landscape, you can specify CPU architecture for Lambda function. If your runtime supports arm64 architecture, you enjoy up to 34% price performance improvement according to this post. In late 2021, AWS Fargate for ECS also started to support Graviton2 Processor with arm64 workload. As to Fargate for EKS, it has not supported Graviton2 processor as of yet, but is on track.

As to Kubernetes, I’ve discussed how to get container registries to support platform-specific images. So we can assume image registries all support OCI format image index(aka fat manifest), which points platform-specific images for arm64 and amd64. In this post, I’ll focus on the node and workload, with EKS as an example. Since control plane is a managed service, we will focus on the worker node, where the Pods are running.

Worker Node

The cloudkube project uses Terraform to build our test EKS cluster. One of the node groups consists of the new m7g.large instance (Graviton3 processor). For this new node group, the AMI type must be AL2_ARM_64, so it picks up an EKS optimized AMI for arm64 during node provisioning. The IAM role of each node has SSM policy so we can use session manager and pre-installed SSM agents to connect to each node. One the m7g node, I would like to check a few things:

The node CPU
The containerd package
The kubelet executable.

They all should be for the right CPU architecture, as the following commands clarifies:

$ lscpu | grep -i arch
Architecture: aarch64
$ yum list | grep containerd
containerd.aarch64 1.6.6-1.amzn2.0.2 @amzn2extra-docker
containerd-stress.aarch64 1.6.8-1.amzn2.0.1 amzn2extra-docker
$ file -b $(which kubelet)
ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked (uses shared libs), BuildID[sha1]=5c7a059f13f8bece4ce30f3357d57631c28bdde2, for GNU/Linux 3.7.0, stripped

We can also check image pulling with containerd. Let’s check what is the correct image first, by examining the image index with manifest-tool:

$ manifest-tool inspect digihunch/colorapp:v0.1

The index in the response tells us that:

the digest of the whole image index starts with 0fa335;
the manifest digest for the arm64 variant starts with 7479df;
the manifest digest for the amd64 variant start with 1bd198;

Now if we use ctr utility to pull image (“sudo ctl pull image”), we can see the correct digest for arm64:

The default behaviour of “ctr image pull” is to pull a platform-specific image, overridable with --platform or --all-platforms. On an amd64 node, I get the corresponding results as well.

Workload

Let’s ensure all workloads in the Namespaces are using correct images. We’ll deploy the colorapp, and then examine that along with some system Pods.

For example, DaemonSet aws-node has one pod per node. To verify the distribution, we can get to pods’ command shell and check CPU architecture with uname command:

$ kubectl -n kube-system get po -l app.kubernetes.io/name=aws-node -o name | xargs -I{} kubectl -n kube-system -c aws-node exec {} -- uname -m

The command above verifies that pods scheduled to arm64 nodes correctly. It does not however, proof that the arm64-specific image is being used. I find it pretty tricky to validate a container is using intended image on arm64 node. I have not found a working kubectl command. There is a plausible tag named imageID under container status. For kube-proxy Pod in kube-system namespace, there are two values. However, for colorapp pods, there is only one value with different format, even though they are scheduled to nodes of both architectures.

$ kubectl -n kube-system get po -l k8s-app=kube-proxy -o yaml | grep 'imageID:' | sort | uniq
 imageID: sha256:04beb3b811d345722d689a70a30bafa27e0edd412613bee76c3648b024b25744
 imageID: sha256:b9b6705d4ad6be861f0e98b7325e5106715ef21a82692f7e8a005a280f159518
$ kubectl -n default get po -l app=color -o yaml | grep 'imageID:' | sort | uniq
 imageID: docker.io/digihunch/colorapp@sha256:0fa335fdbcc3b644d57c8debe075775b19011985b6342adfb430e7011456d12e

This issue reports such inconsistency and the issue unfortunately did not get attention. The reporter also asks to have sha256 of the actual image. However, the Kubernetes developers regard this as an CRI issue. Currently we cannot tell exactly which image is used.

I figured out a workaround, by getting on the node and dump the image on the node:

$ sudo ctr -n k8s.io image list
$ sudo ctr -n k8s.io image export /tmp/x.tar docker.io/digihunch/colorapp@sha256:0fa335fdbcc3b644d57c8debe075775b19011985b6342adfb430e7011456d12e

In the export tar file review the manifest.json file which contains layer digests. We should find these layer digests match those of the platform-specific image’s.

Utilities

Since we can ensure that Pod running on a node can always pull the correct platform-specific image, we do not need to worry about Helm chart. We just need to make sure our container registry references an index digest that points to images of multiple architecture. For the same reason, we do not need to worry about pod autoscaling.

When it comes to node autoscaling, all node should support have kubernetes.io/arch and kubernetes.io/os labels (e.g. Karpenter). However, we generally prefer to expand the arm64 node group since it is cheaper. With cluster autoscaler, we can use priority based expander. With Karpenter, we can set weight so that the provisioner for arm64 node group carries higher weight.

Scheduling

With multi-arch image, the container runtime will pick up the right version of image. From deployment perspective, we do not worry about the difference between nodes in CPU architectures. However, in some use cases, we still want to schedule certain Pods to nodes with one CPU architecture over the other. I call these platform-specific workload.

We mainly needs to control scheduling behaviour. There are two mechanisms, node affinity, and taints & tolerations.

Node Affinity is based on node labeling. From the well-known labels, annotations and taints, all Kubernetes distribution should label their nodes with the kubernetes.io/arch and kubernetes.io/os labels. The value for arch is either arm64 or amd64. When we add a node affinity of requiredDuringSchedulingIgnoredDuringExecution type to Pods, scheduler takes matchExpressions under nodeSelectorTerms into consideration, when placing Pods to Nodes.

When a Pod has lots of nodeSelectorTerms, it can be brain twisting to sort through the logic. In that case we can use Taints and Tolerations. The idea is that once we taint a node, the scheduler will not schedule any Pod to the Node, unless the Pod has a matching Toleration.

In this post, the author customized the bootstrap script so the node provisioning process automatically taints arm64 nodes with arch=arm64:NoSchedule. Otherwise, we can manually taint a node:

$ kubectl get no -o wide # and check KERNEL-VERSION column, taint the ones with aarch64
$ kubectl taint nodes ip-147-207-3-164.us-west-2.compute.internal arch=arm64:NoSchedule

This can be a very useful technique when you’re not sure if every workload image are capable of multi-arch, and you want to avoid scheduling any Pods without tolerations on the arm64 nodes. A Pod cannot get scheduled on those nodes until you confirm their container images, and add corresponding tolerations.

Summary

Given the power efficiency, a lot of workload will gradually migrate to arm64 architecture. However, software will take a while to get ready. For example, hyperkit has not supported M1 processor and I still cannot use it on newer MacOS for Minikube. Hybrid architecture is here to stay and we need to have an end-to-end examination of our supply chain.

Kubernetes with Multiple CPU Architectures 1 of 2 – Container Image

Sat, 15 Apr 2023 12:17:00 -0400

While working on a multi-arch Kubernetes cluster, I came across quite a few issues with image logistics and decided to put these issues in a separate post here. The goal is to supply images with multi-arch support in a standard way. Therefore it is not advisable to rely only on arbitrary image tags to distinguish CPU architecture. If that was the case, each team may use different tag keys and values. The image consumers would have to build custom logics to pull the right image. The standard way is to use Manifest List (Docker’s term), or Image Index (OCI’s term) on the image registry, to announce the multi-platform support of the images in the repository, and to ensure that a container runtime can pick the appropriate index entry that matches the local node’s architecture and platform.

Introducing Image Index

For image index/manifest, there have been three standards, with two currently active:

Docker Image Manifest V2, Schema 1: deprecated in 2019
Docker Image Manifest V2, Schema 2: supports attribute for platform specific image
OCI Image Specification: supports attribute for platform specific image

In 2016, Docker contributed the Docker V2 Image specification as the basis of the OCI image specification. Therefore, the two current active formats (V2.2 and OCI) still look alike today. They are both JSON document and shares many attribute names. However, I take OCI as the standard since it is adopted in CNCF (e.g. containerd, quay.io). Most of the registries (e.g. ECR, ACR and even Docker Hub) support both formats, and you can tell the format by requesting manifest document. When working with Kubernetes we should expect OCI format index even though the Docker Manifest List v2.2 will most likely be compatible.

When examining an image manifest we should look for the following structure:

Manifest list and Manifest, source: https://www.opensourcerers.org/2020/11/16/container-images-multi-architecture-manifests-ids-digests-whats-behind/

In the Docker lingo, a “fat manifest” is simply a list of manifests, equivalent to image index in OCI terms. A manifest document consists of several attributes in sha256 digest, as the diagram shows. There are four types of digests and they should not be mixed:

index-sha256: for the index or fat manifest
manifest-sha256: for the manifest of a single container image
config-sha256: for the config section
layer-sha256: for the image layer files

If two images share a layer, then from their respective manifests, we should be able to find a layer with the same sha256 digest. If an index (fat manifest) references other manifest, we should also find that from the sha256 digest. When you reference an image, you should use the sha256 digest for the entire index or manifest list.

To view image index, “docker inspect” is insufficient. It is executed within Docker daemon, against the image pulled based on the CPU architecture of Docker daemon’s server. So let’s look at some tools to check index from remote registry.

Tools to view image index

The most popular tool is Docker’s experimental manifest inspect command. However, it is still not mature. For example, we have to use Docker CLI version 23.0.0 or later with OCI compatibility issue fixed. Prior to v23.0.0 (Feb 2023), Docker CLI were not able to correctly display a list of OCI-format manifests, and it simply says “no such manifest”. Even after 23.0.0, I still find it clunky. For example, it does not display the sha256 digest of the OCI image index itself (the digest on the far left of the diagram above).

To troubleshoot the reason, I borrowed some idea from this post, and have my shell script as below, to check the index for my colorapp image:

#!/bin/sh

ref="${1:-digihunch/colorapp:v0.1}"
sha="${ref#*@}"
if [ "$sha" = "$ref" ]; then
 sha=""
fi
wosha="${ref%%@*}"
repo="${wosha%:*}"
tag="${wosha##*:}"
if [ "$tag" = "$wosha" ]; then
 tag="latest"
fi
apio="application/vnd.oci.image.index.v1+json"
apiol="application/vnd.oci.image.manifest.v1+json"
apid="application/vnd.docker.distribution.manifest.v2+json"
apidl="application/vnd.docker.distribution.manifest.list.v2+json"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
 | jq -r '.token')
curl -H "Accept: ${apio}" -H "Accept: ${apiol}" -H "Accept: ${apid}" -H "Accept: ${apidl}" \
 -H "Authorization: Bearer $token" \
 -w '\nResponseCode:%{http_code}\nResponseHeader:\n%{header_json}\n' \
 -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}"

It appears that the sha256 digest of the OCI image index itself is provided in the response header, instead of response payload. So docker manifest tool misses the header!

Luckily, there are some alternatives, such as skopeo or manifest-tool. I wasn’t able to get the former to work with OCI index. The latter displays my OCI index in a pretty format and I was able to view a few other indexes that I wasn’t able to with Docker manifest. So I have a good impression of it. So in summary, for the four tools to view image index, my recommendations are:

Docker manifest: still glitch as of 2023 but pretty widespread
manifest-tool: pretty solid, the output is pretty format, requires manual install
skopeo: not straightfoward to use. easy to install
Self-scripting in bash: only for troubleshooting

For the rest of this post, I’ll however continue to use docker manifest, given its popularity and availability. When it fails to display an index, I’ll try manifest-tool.

Inspecting Image Index

With docker manifest command (v23.0.1), let’s take a peak at three images:

docker manifest inspect --verbose osimis/orthanc:22.12.2
docker manifest inspect --verbose public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0
docker manifest inspect --berbose ubuntu:23.04

In the JSON document return by the first command (osimis/orthanc image), we see the following structure:

{
 "Ref": "docker.io/osimis/orthanc:22.12.2",
 "Descriptor": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "digest": "sha256:20413096878fb56bf8d09af08cf4055993dbcf507526f0561b26fc4d0ed7affc",
 "size": 11227,
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 "Raw": "......",
 "SchemaV2Manifest": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "schemaVersion": 2,
 "config": {
 "mediaType": "application/vnd.docker.container.image.v1+json",
 "digest": "sha256:9de20d8a006c6377b85dba9f817d47048982bd0f15fac7daacb64f42060d4b6d",
 "size": 16518
 },
 "layers":[
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "digest": "sha256:025c56f98b679f70b7a54241917e56da7b59ab9d2defecc6ebdb0bf2750484bb",
 "size": 31412852
 },
 ......
 ......
 ......
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "digest": "sha256:a24cf4c71e6977b692bbf517eba7bf6f454d41ceab24c1b2694c3303bc718a1c",
 "size": 174739
 }
 ]
 }
}

We can see that the return is a single manifest. The Descriptor key suggests that it is built for amd64 architecture. The mediaType, along with SchemaV2Manifest and schemaVersion, suggest that it is a Docker Manifest (v2.2) format. Other attributes are summarized here. In the response from the second command(public.ecr.aws/amazonlinux/amazonlinux), we see the structure below:

[
 {
 "Ref": "public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0@sha256:260907696498cbf078abc2f3428bf8d19faf77cded5d5459900997a1bc29903d",
 "Descriptor": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "digest": "sha256:260907696498cbf078abc2f3428bf8d19faf77cded5d5459900997a1bc29903d",
 "size": 529,
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 "Raw": "......",
 "SchemaV2Manifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "config": {
 "mediaType": "application/vnd.docker.container.image.v1+json",
 "size": 1478,
 "digest": "sha256:d27c2e45784db13b0b2bc89a52be6661aa1d53bd25c070b41626768c9c563c3d"
 },
 "layers": [
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "size": 62386320,
 "digest": "sha256:d78505e615251c4f4af6eaa9507b67917d263d23551dcc5a1eed3c012d32a54d"
 }
 ]
 }
 },
 {
 "Ref": "public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0@sha256:7fb3183b38e1a9859374a343e72dc43731aeccaf26507da94ebc310067f39fed",
 "Descriptor": {
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "digest": "sha256:7fb3183b38e1a9859374a343e72dc43731aeccaf26507da94ebc310067f39fed",
 "size": 529,
 "platform": {
 "architecture": "arm64",
 "os": "linux",
 "variant": "v8"
 }
 },
 "Raw": "......",
 "SchemaV2Manifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 "config": {
 "mediaType": "application/vnd.docker.container.image.v1+json",
 "size": 1493,
 "digest": "sha256:a1ea533a0632c6501d7848c7ed481e8fb0398c3277c0d9fddf0b0fdcd5731c09"
 },
 "layers": [
 {
 "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
 "size": 64003805,
 "digest": "sha256:71343c2791199c6e2c19c308cff6493497a02f57e225c11405e1934dc7428b3c"
 }
 ]
 }
 }
]

Comparing this structure with the first return, we noticed that the return is a list of two manifests, each with its own platform architecture. So the amazonlinux image has multi-architecture capability. It can serve as base image for custom images for both architectures. On the “image tags” tab of registry page, we can see a list of published tags. Each is tied to either a single image manifest, or a manifest list:

Each image tag is associated with a single manifest or a manifest list (aka “fat manifest”)

When you click on “image manifest”, you can see both “Image manifest media type” and “Artifact media type” values. When you click on “manifest list”, you see the “Image manifest media type” value, because the “fat manifest” does not point to a single artifact.

Now, let’s review the third command (ubuntu:23.04) response:

[
 {
 "Ref": "docker.io/library/ubuntu:23.04@sha256:52293638ba652a2e8f9e1c1cfcc905839b1f2a9e671ddcc9bf77909b6bf527d0",
 "Descriptor": {
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "digest": "sha256:52293638ba652a2e8f9e1c1cfcc905839b1f2a9e671ddcc9bf77909b6bf527d0",
 "size": 424,
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 "Raw": "......",
 "OCIManifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "config": {
 "mediaType": "application/vnd.oci.image.config.v1+json",
 "size": 2299,
 "digest": "sha256:beb2152822b716b4deac2996f16bc84db0a14b7cbc549579635590438f9c0e1d"
 },
 "layers": [
 {
 "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
 "size": 26638886,
 "digest": "sha256:db781b8aed497363312ef32499cbfac28821e0494db7f0cadc4e716853e02a12"
 }
 ]
 }
 },
 {
 "Ref": "docker.io/library/ubuntu:23.04@sha256:0c8e3367a3fe9b703c759e1c148c5809df1a2734f8f37529bd11fbcfd34b1d1c",
 "Descriptor": {
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "digest": "sha256:0c8e3367a3fe9b703c759e1c148c5809df1a2734f8f37529bd11fbcfd34b1d1c",
 "size": 424,
 "platform": {
 "architecture": "arm64",
 "os": "linux",
 "variant": "v8"
 }
 },
 "Raw": "......",
 "OCIManifest": {
 "schemaVersion": 2,
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "config": {
 "mediaType": "application/vnd.oci.image.config.v1+json",
 "size": 2316,
 "digest": "sha256:eb2d2fb228861107934403e776544a3f516bc7123a1275d52f1992bada8e94d6"
 },
 "layers": [
 {
 "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
 "size": 25802344,
 "digest": "sha256:29d183ded65aecf549f39ef891c21feb9034b5b10f341533b4af297bb5c60bb8"
 }
 ]
 }
 },
 ......,
 ......,
 ......
]

I readapted the response document for simplicity. Notice that the document is also a “fat manifest” except that the mediaType, OCIManifest and schemaVersion keys suggest that it is an OCI format. The image supports more platforms (combination of OS and CPU architectures). The attributes for OCI index is available here.

Single-platform image Build

At the beginning of the supply chain, we build platform-specific image with CI/CD jobs. Traditionally, the docker build process can only produce images for the platform where the build command run. For the process to work in multiple platforms, we’d need multiple build agents of different platforms. Each agent runs build process and pushes artifact to the registry (with “docker push”). At the end we create a fat manifest that combines the images for all platforms, with “docker manifest create” command. A blog post on Docker from April 2020 refers to this as the “hard way”.

Since then, Docker introduced a new client Docker Buildx, a CLI plugin that extends the docker command with the full support of the features provided by BuildKit builder toolkit. One such feature is the ability to produce multi-platform images in one command run. There is also a push switch that helps you generate manifest behind the scene. In my own testing, I am able to build image for both amd64 and arm64 on MacOS (M1).

Multi-platform image build

Take the colorapp Python application as a simple example, I followed this guide to generate corss-CPU-architecture build. First, we can examine if we already have a build instance that supports our desired platform and whether that is already selected:

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
vibrant_hypatia * docker-container
 vibrant_hypatia0 unix:///var/run/docker.sock running v0.11.3 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default docker
 default default running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux docker
 desktop-linux desktop-linux running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

In the example above, I’ve already got one named vibrant_hypatia that supports linux/arm64 and linux/amd64 and it is selected. If that’s not the crease, we can create a new build instance and tell Docker to use it. Then

$ docker buildx create --use

Then we can run the actual build command. In the build command we specify the platforms and tags. We also specify –push so we can push the entire binary at the same time.

$ docker buildx build --platform linux/amd64,linux/arm64 --push -t digihunch/colorapp:v0.2 ./colorapp

Note that in this single command, the building for both platforms is completed and pushed with the correct OCI-compliant manifest. There is no need to build them separately and work separately on the manifest. We can use the manifest command from last section to verify the image manifest.

The behaviour of defaulting to OCI-compliant image index is a change in the buildx version 0.10 in Jan 2023. With —provenance=false, we can control manifest format with oci-mediatypes=true. In buildx 0.10, the default value for provenance changed from false to true, which always makes OCI the image manifest format. This change of default, along with the glitch of “docker manifest” prior to v23.0.0, had pretty big impact and raised confusions (such as this bug report from Ubuntu, and the issue in this post). However, since I take OCI-compliant format as the standard, I do not have a problem with this change. We can verify the image manifest with:

$ docker manifest inspect --verbose digihunch/colorapp:v0.1. ## docker CLI version > 23

Platform-specific image

Our Dockefile looks like this, with a base image.

FROM public.ecr.aws/amazonlinux/amazonlinux:2
RUN yum update -y && yum install -y python3 && yum clean all && rm -rf /var/cache/yum
COPY serve.py ./
RUN chmod +x ./serve.py
CMD ["python3", "-u", "./serve.py"]

To build platform-specific images, we need the base image(amazonlinux) to support multi-arc too, so that the build process picked the correct platform specific image as base. We can verify this is the case by looking at the sha256 digest of the first layer of each colorapp variant. They are d78505 for the amd64 image, and 71343c for the arm64 image. We can find the same layer digest from the amazonlinux manifest.

We now have mutli-architecture images, as well as the index. Docker Hub shows both digests with their architecture, although unlike ECR it does not show whether the tag is a manifest list or a manifest:

In real life, our supply chain usually has multiple levels of base images or multiple base images. It is important to ensure platform-specific images are available by examining their manifests.

In the Dockerfile, we consider it a best practice to reference base image by digest instead of by tag. We just need to make sure the digest actually points to an image index (manifest list), with each manifest points to the platform-specific image.

Summary

Now we’ve build the first part of our supply chain with arm64 capability. We have build platform specific images that can use their own platform-specific base images. We’ve also created an OCI-compliant image manifest to announce the image supports multi-architecture. Next, we will examine how containerd on different CPU architectures consumes the platform-specific image.

DICOM testing over TLS

Sat, 18 Feb 2023 01:05:00 -0400

I have two open-source projects to deploy a medical imaging application on different platforms. In both of them, I define DICOM validation scenario, and provide steps to test DICOM traffic with TLS. The steps have been working well, until a recent change in Envoy broke the testing, and led me to revisit the test scenario.

In a nutshell, I changed from dcm4che to dcmtk binary builds. I’ve also expanded the test case from a self-signed server certificate to one involving a self-signed CA. Although this test is about TLS for DICOM traffic, the principles apply to any traffic at TCP level. If you just need instruction for DICOM validation on Orthweb or Korthweb projects, skip the Background section.

Background

To test DICOM traffic with command line tool, I was investigating between dcm4che and dcmtk. Both are open-source projects with builds for multiple platforms. My DICOM test is as simple as a C-Echo command and a C-Store command using the tool, with TLS enabled. Once they work, other DICOM commands usually work as well.

I have been primarily using dcm4che as I was familiar with its previous version from my old job. For example, I can issue a C-ECHO with TLS using storescu command:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123!

To turn that test into a C-Store test, simply add a DCM file as input:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! MY.DCM

The C-Store output tracks each DIMSE command and return codes. Note that in the command, we specify –tls12 as the version, and with –tls-aes switch we enabled AES or 3DES encryption. We also specified a file for trust store and password to the trust store.

This is when I first frowned over dcm4che. The dcm4che utility is a Java-based program we have to take an extra step of turning certificate into Java trust store. Different JVM versions may also cause different behaviours in the test. What later prompted me to switch to dcmtk is that with dcm4che I came across a weird error since Envoy proxy version 1.23, which impacted both Orthweb (Envoy proxy) and Korthweb (Istio Ingress).

Moreover, dcmtk is available as a HomeBrew package, Ubuntu package, and Debian package.

The test case, data and the tool

We can install dcmtk utility simply with brew install dcmtk, and then we need its echoscu and storescu commands with correct TLS options. The DICOM data I used for testing is a CT exam available for download here.

Before mocking with dcmtk‘s TLS options, we first need to understand what would be a good test. Previously I have used a single self-signed certificate on server. It is an over-simplified scenario that is far from a real-life certificate chain, and also does not test client certificate. If I also self-sign the client certificate, the client and server certificates are signed by entirely different parties and have no trust relationship, making it an invalid test case for client certificate.

For a full-blown testing with TLS, we should have two levels of CA as below:

The chart represents a typical hierarchy of three-level certificate authorities. Sometimes we need simplicity in our testing, and it is reasonable to simplify the diagram to the following, with one CA that issues certificate for both client and server:

Our DICOM validation testing will be based on both approaches depending on the project and deployment option. For Orthweb project and the Helm-chart driven option in Korthweb, we have one level of CA. For the GitOps and manual option in Korthweb, we have two levels of CA. When configuring testing, it is important to have the diagram above in mind.

In addition, we should also be aware of the limitation of the testing. As a personal project I will not pay for the certificates. I have to self-sign the certificate of the CA so there is no way to derive trust on this CA from another level. As a result, we must tell the client and server to trust the CA. The steps to create the needed certificates are:

Generate a key pair for Test CA. Generate the certificate for Test CA by self-signing its own public key
Generate a key pair for the (DICOM) server. Generate the certificate for the server by signing its public key with Test CA’s private key
Generate a key pair for the (DICOM) client. Generate the certificate for the client by signing its public key with Test CA’s private key

Because the client now also has its certificate, we can test with and without client certificate, using the following dcmtck switches:

-d (shorthand for –debug): print out detailed DICOM communication log. For succinct output, use -v (shorthand for –verbose) instead.
+tla (shorthand for –anonymous-tls): enable anonymous TLS (without client certificate)
+tls (shorthand for –enable-tls): enable full TLS (with client certificate), followed by client key and certificate files
-rc (shorthand for –require-peer-cert): –require-peer-cert, require peer (server) certificate
+cf (shorhand for –add-cert-file): –add-cert-file, add server certificate so client can trust it.

For C-ECHO, the testing commands with and without client certificate look like:

$ echoscu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112
$ echoscu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112

For C-STORE, the testing commands with and without client certificate look like:

$ storescu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm
$ storescu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm

As to how to create the certificates, in the post Creating X.509 TLS certificate in Kubernetes, I discussed different ways to create certificates for testing, including using openssl. In the next two sections, I will discuss them in further details.

Orthweb Test

The orthweb project runs Docker containers on an EC2 instance. In the cloud-init script of the EC2 instance, we self-sign a test CA with openssl. Then we create key and certificate for server and client respectively:

IssuerComName=issuer.orthweb.digihunch.com
ClientComName=dcmclient.orthweb.digihunch.com
ServerComName=ca-central-1.compute.amazonaws.com

openssl11 req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$IssuerComName/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ServerComName/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:$IssuerComName -keyout /tmp/server.key -out /tmp/server.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ClientComName/emailAddress=orthweb@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

When simulating DICOM activities from the client side, we supply the three files (ca.key, client.crt and client.key) to the echoscu and storescu executables, to issue DIMSE commands on top of TLS from client side, as shown in the previous section.

Korthweb Test

Currently, the Korthweb project deploys to Kubernetes cluster in three approaches, including two types of ingress controllers: Istio CRD (manual and GitOps deployment options) and Traefik CRD (Helm Chart driven deployment options). With both approaches, it is fairly straightforward to validate the HTTPS port. We export the CA certificate and run a curl command such as:

curl -HHost:web.orthweb.com -k -X GET https://web.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert ca.crt

Note that by default, the curl command adds SNI (server name indication) extension by default to its TLS ClientHello Message (even without -HHost switch). It acts like a modern browser.

On the ingress controller side, most ingress controllers use SNI to drive request routing (because Host field in payload is encrypted). For example, Traefik Proxy has HostSNI matching rule. With Istio, the document states that: TLS implies the connection will be routed based on the SNI header to the destination. With DICOM traffic the Ingress also expects the client to make use of SNI extension in the TLS ClientHello message. The ingress supports multiple sites so the SNI even has an impact of which TLS certificate the ingress serves to the client. We can use openssl to examine which certificate an ingress serves. For example:

openssl s_client -showcerts -connect dicom.orthweb.com:11112 -servername dicom.orthweb.com < /dev/null
openssl s_client -showcerts -connect dicom.orthweb.com:11112 < /dev/null

The second command without -servername switch constructs an ClientHello message without SNI. The ingress may not have a clue of what certificate to serve, depending on its own implementation of TLS protocol. We can also force TLS version with a switch such as -tls1_2.

When it comes to open-source DICOM client, neither dcm4che or dcmtk puts SNI in the TLS request. This created some limitation with my testing. Luckily I do not have multiple routing destinations for now so I only need to direct all DICOM traffic to a service. When I use Istio ingress, I was able to set hosts to “*” so the Ingress does not care missing SNI extension in the client request. With Traefik proxy, I had to set sniStrict to false, and also forgo client certificate check.

The workaround is different per ingress implementation. Even worse, depending on what the available workaround can achieve, the testing steps vary as well.

For example, I perform DICOM validation (Istio ingress) with the steps below:

# bhs: generate client key pair
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=dcmclient.bhs.orthweb.com/emailAddress=dcmclient@www.digihunch.com -keyout bhs.client.key -out bhs.client.csr

# bhs: export intermediate CA credentials
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.key}' | base64 -d > bhs.int.ca.key
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > bhs.int.ca.crt

# bhs: get intermediate CA to sign client cert 
openssl x509 -req -sha256 -days 365 -in bhs.client.csr -CA bhs.int.ca.crt -CAkey bhs.int.ca.key -set_serial 01 -out bhs.client.crt

# bhs: validate web request (without client certificate)
curl -HHost:web.bhs.orthweb.com -k -X GET https://web.bhs.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert bhs.int.ca.crt

# bhs: validate DICOM c-echo request (with client certificate)
echoscu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112

# bhs: validate DICOM c-store request (with client certificate)
storescu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 DICOM_CT/0001.dcm

On the other hand, for Traefik ingress, I have to use anonymous TLS without client certificate:

echoscu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112
storescu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 DICOM_CT/123.dcm

The complexities with different test paths are consequences of the missing SNI capability in both DICOM toolkits. Unfortunately, the developers of the two DICOM tools are not aware of these consequences. I tried to contact dcmtk about this and will see what happens.

TLS profile

Another setting to pay close attention to is the security profile for TLS communication. These profiles define the behaviours of dcmtk when it establishes TLS connection. The dcmtk has the following security profiles:

–profile-bcp195-nd (+py default): Non-downgrading BCP 195 TLS Profile
–profile-bcp195 (+px): BCP 195 TLS Profile
–profile-bcp195-ex (+pz): Extended BCP 195 TLS Profile
–profile-aes (+pa): AES TLS Secure Transport Connection Profile (retired)
–profile-null (+pn): Authenticated unencrypted communication (retired, was used in IHE ATNA)

The two at the bottom have been retired. The current profiles are all based on BCP195. BCP (best current practice) are sub-series of the corresponding RFC document series. The current revision of DICOM standard discusses bcp195-nd, bcp195 and bcp195-ex profiles in DICOM standard chapter PS 3.15 (B.9-B.11). For example, bcp195-nd requires that:

Implementation shall not negotiate TLS 1.0 or 1.1
Client and server shall prefer strict TLS configuration (as opposed to startTLS)
Ciphers that should be supported.
Recommend port 2762

These BCP profiles were incorporated into DICOM standard since 2018 and are all based on BCP195. BCP 195 states in section 3.6

3.6. Server Name Indication
TLS implementations MUST support the Server Name Indication (SNI)
extension defined in Section 3 of [RFC6066] for those higher-level
protocols that would benefit from it, including HTTPS. However, the
actual use of SNI in particular circumstances is a matter of local
policy.
Rationale: SNI supports deployment of multiple TLS-protected virtual
servers on a single address, and therefore enables fine-grained
security for these virtual servers, by allowing each one to have its
own certificate.

So technically, missing SNI is considered incompliant.

Summary

I find myself switching between dcmtk and dcm4che back and forth in the past. This time, I spent some time hoping to settle with the better tool this time. The effort is insightful but not fruitful. It is unfortunate to realize that neither supports SNI so I had to compromise the feature of my deployment. Hopefully one of those tools will catch up.

GitHub Action Gotchas

Fri, 27 Jan 2023 01:15:02 -0400

I started with GitHub Actions a couple years ago. Recently I came across a few interesting use cases while I was trying to setup Terraform workflow with GitHub actions. These use cases prompted me to make use some new features in GitHub Action. So I put them in a post here.

Runners can assume IAM Role in AWS

In many scenarios we want to execute AWS CLI command from GitHub action. Also, executables such as terraform inherits credential from AWS CLI. The credential should be a temporary role-based credential instead of an IAM user based on access keys.

There is a GitHub Action called configure-aws-credentials-for-github-actions that can help configure GitHub runner using OIDC identity provider (since Nov 2021 v1.6.0). With the action, the GitHub runner can assume an IAM role as an IAM user (with access key), or using a web identity.

For a GitHub runner to have a web identity thereby assume an IAM role, we should configure OIDC provider in AWS. We can do that from AWS console (i.e. under IAM), or using CloudFormation code. Below is a snippet as an example:

Resources:
 GitHubOIDC:
 Type: AWS::IAM::OIDCProvider
 Properties:
 Url: https://token.actions.githubusercontent.com
 ClientIdList: 
 - sts.amazonaws.com
 ThumbprintList:
 - 6938fd4d98bab03faadb97b34396831e3780aea1

Then from the configured OIDC provider, we can obtain a thumbprint. GitHub action gives the thumbprint here. In AWS, we configure an IAM role whose AssumeRolePolicyDocument will reference the thumbprint. Here is an example. In the condition section of AssumeRolePolicyDocument, we can also specify a specific GitHub repository so that only Actions from that repository can assume the IAM role with their web identities.

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v1-node16
 with:
 role-to-assume: ${{ vars.IAM_ROLE_ARN }}
 aws-region: ${{ vars.AWS_REGION }}

This way, we map a GitHub runner’s web identity to an IAM role with a step using the Action above. We also filter what GitHub org and what repo can trigger actions that assumes the role, by the condition clause in the role statement. If the step fails, we can look at CloudTrail on the AWS side for causes. Look for entries with AssumeRoleWithWebIdentity as Event Name.

Reusable workflows

For better reusability of Action steps, GitHub introduced reusable workflows (generally available since Nov 2021). It is particularly helpful when we need to run a workflow for different environments. The reusable workflows files (YML) can be placed in separate repositories, and be reference as such. This allows enterprises to centralize the management of reusable workflows.

I have been using the act project to emulate GitHub action locally on MacBook. As of Jan 2023, act does not support reusable workflow.

With the split between caller and reusable workflows, we have a new challenge of passing secrets and variables between them. It is not straightforward and GitHub documentation needs improvement to get the documentation clear. Also because the word “environment” is used in different contexts, it is ambiguous and therefore difficult to Google relevant information.

Passing variables

First, there are several types of variables in GitHub action:

Environment variable: declared under env keyword in a workflow. To use environment variable, use the env context. For example: ${{ env.MY_VARIABLE }}
Configuration variable: introduced in Jan 2023, configuration variables are defined at repository, environment and organization levels. To use configuration variable, use vars context, and ensure the workflow job specifies a value for environment attribute.
Secrets: GitHub also calls it Environment secret when defined at environment level. It works the same way as a configuration variable because it is also specific to an environment. The content is not viewable once set.

The reason GitHub action makes this so confusing, is that on one page, its documentation distinguishes between environment variable and configuration variable:

On another page, the document refers to configuration variables at environment level as environment variable:

It seems that “configuration variable” is too new for GitHub to refine its documentation as of January. This semantical confusion gave me a hard time investigating how to pass “Environment variable” to reusable workflows. I will stick to the meaning on the first page to distinguish environment variable and configuration variable at environment level.

Passing environment variable isn’t straightforward. In this discussion thread, people discussed how inconvenient it is. I used the workaround in this comment, where I had to create a job for the sake of storing variable values to output.

Pass secret is easier. This is an insightful blog post (Dec 2021) about passing secret to reusable workflow. The attempt 3 in the post works for me. First, we pass the value of environment to the reusable workflow as an input, then at job level specify the environment with the value. Then in the jobs we can reference secrets as ${{ secrets.NAME }}. The job will pick up the secret based on the correct environment.

It appears that since May 2022, GitHub introduced secrets: inherit keyword to address this. However, the method above still works for configuration variable.

Authentication of GitHub Actions

By default, a GitHub action can access the code repository that triggers the action and no other repositories (with GITHUB_TOKEN). However, in many cases we need to access external repositories. For example, terraform init command from a GitHub action implicitly calls git clone to pull module code from external repositories. A GitHub workflow may also reference a workflow file from external repositories.

The question is how to authenticate GitHub workflow to access external repo. This post has a thorough discussion. We may create a Personal Access Token and pass it to set-git-credentials action. We are essentially sharing a personal credential (and repo access) with a GitHub action, which is not a good practice.

The proper way to solve this problem, is to create a separate GitHub App and grant the access only the repo that the workflow needs to access. The GitHub App will generate a private key. Then we supply the private key to workflow-application-token-action so the workflow can act as the GitHub App, thereby access the external repos. The post has more details in the GitHub App section. Suppose we have terraform get command to clone external repo, the actions may look like this:

 - name: HashiCorp - Setup Terraform
 uses: hashicorp/setup-terraform@v2

 - name: Get RepoReader App Token
 id: get_repo_reader_token
 uses: peter-murray/workflow-application-token-action@v2
 with:
 application_id: ${{ vars.REPO_READER_APPLICATION_ID }}
 application_private_key: ${{ secrets.REPO_READER_PRIVATE_KEY }}

 - name: Cache Git Creds
 uses: de-vri-es/setup-git-credentials@v2
 with:
 credentials: https://x-access-token:${{ steps.get_repo_reader_token.outputs.token }}@github.com/

 # Terraform Get implicitly calls git clone which uses the credential cached as above
 - name: Terraform Get 
 run: terraform get

Another benefit of using GitHub App is that the token is a short-lived credential that expires as the job is finished, whereas a PAT will expire on a preset date. In this use case we can think of GitHub App as a service account with minimized privilege to read a short list of repos.

Final words

I came across this reddit post against pipeline use and there are some legit points. For example, the user is frustrated with the limitation with variable passing and unclear documentation. The post wasn’t specific to GitHub action but I do share some frustration while working with GitHub Actions.

On a positive note, since I first used GitHub actions, it has evolved quite a bit with new features, although the documentation is somewhat lagging. It is still very helpful as all of these are free to personal use. I look forward to more interesting features.

Optimize CPU and Memory for Kubernetes Pod

Fri, 13 Jan 2023 11:47:00 -0400

When optimizing workload performance, it is important to understand how on earth operating system allocates CPU and memory to processes. This helps understand how to set resource limit Kubernetes Pod in an optimal way.

CPU resource assignment

The OS distributes CPU resource to processes by the unit of time share of CPU time. Most of the time, many processes with CPU instructions (machine code) are waiting in the Job queue, for their share of CPU time in order to execute their instructions. As soon as CPU becomes idle, the CPU scheduler selects a process from the ready queue to run next:

Ideally, OS should schedule CPU in a way that it should not waste any CPU cycle. It should also minimizes waiting time and response time of processes. At a high level, there are two types of CPU scheduling:

Preemptive: OS allocate CPU resources to a process for only a limited period of time and then takes those resources back. It could interrupt a running process to execute a higher priority process.
Non-preemptive: New processes are executed only after the current executing process has completed its execution.

Here is more information about preemptive and non-preemptive scheduling.

CPU is compressible resource in Linux

In the Linux world, all scheduling is preemptive. We also call it kernel preemption. As the wikipedia entry states: the scheduler is permitted to forcibly perform a context switch (on behalf of a runnable and higher-priority process) on a driver or other part of the kernel during its execution, rather than co-operatively waiting for the driver or kernel function (such as a system call) to complete its execution and return control of the processor to the scheduler when done.

The Linux scheduler implements a number of scheduling policies, which determine when and for how long a thread runs on a particular CPU core. The scheduling policies in RHEL include real time policies such as SCHED_FIFO and SCHED_RR where processes have a sched_priority value in the range of 1 (low) to 99 (high); and normal policies such as SCHED_OTHER, SCHED_BATCH and SCHED_IDLE, where sched_priority (specified as 0) is not used in scheduling decisions.

It is important to understand preemptive CPU scheduling on Linux. When OS allocate CPU resource to a process for one time slot, it is not committed to the same process for the next time slot. The OS reserves the ability to revoke the next CPU use and re-assign it for processes of higher priority.

Because of this, we regard CPU as a compressible resource. The compressible characteristic impacts how we optimize CPU utilization for a process, including setting CPU request and limit for Kubernetes workload.

Memory is non-compressible resource

A few years ago, I discussed how to calculate memory usage. A process requests memory from OS using memory allocation functions (the malloc family), and return memory to OS using free functions. The design of Linux OS knows that processes have a tendency to request more memory than they use, which causes under-utilization. In combat against under-utilization, the Linux OS supports memory overcommitment (on by default), allowing processes to request more memory than what is available. The processes have access to virtual memory space and the OS may swap some pages out to disks. The overcommitment mechanism also prevents processes from crashing due to insufficient memory assignment. The kernel can also OOM kill a process when the entire system is in a crisis.

Memory is non-compressible resource. When OS assigns memory pages to a process, the process has to right to keep those pages, until the OS takes them away. Unlike assigning CPU cycles, the assignment of memory pages to processes does not have an expiry time. This is the non-compressible characteristic of memory assignment.

CPU limit and requests for Kubernetes workload

It was considered best practice to set request and limit for memory and CPU. However, knowing CPU is compressible resource and memory isn’t, we should re-consider this practice. In short, for CPU, we should set request only, without setting limit. For memory, we should set limit to exactly the same as request.

A process has different level of demands for CPU at different times. Depending on the activity in the process, the level of demand can even be spiky. If there is a lot of iowait, it may not need a lot of CPU. But when there are lots of computing-bound activities, the program is CPU-thirsty as it is programmed to do more. The last thing we want is to throttle the CPU use for a process in such legit situations. When throttling happens, the process does not get sufficient time share of CPU time. At the platform level, we can’t control when the Pod (process) gets busy. The best thing it can do, is trying to fit more CPU time shares to this process when it becomes CPU thirsty. When we apply a limit of CPU in workload setting, we are potentially throttling the CPU use for a process at the times it needs more CPU time shares, which is counter-productive.

We should still configure CPU request, so that kube-scheduler factors it in when scheduling multiple Pods to a Node. The CPU request alone ensures the number of Pods are not excessive. This is the only thing we can do about controlling CPU assignment for Pods. We should also monitor CPU throttling.

Memory limit and request for Kubernetes workload

Memory is not compressible, therefore we should set both limit and request to the same value. We set memory request so that kube-scheduler has an idea assigning Pods. We set the limit so that no single Pod takes more memory than its fair share. Unlike CPU, once a Pod takes more memory than its fair share, the platform will have to be aggressive to reclaim it back, which may impacts the running of the Pod (process). In contrast, CPU scheduler never guarantees the assignment of CPU time share to a Pod beyond the end of the current CPU cycle.

When we’re setting memory limit and request with different values, we’re sending a confusing signal. We’re inviting Pods to use more memory than they requested. This increases the chance of memory shortage at the node level, and hence the need to OOM kill a Pod.

Horizontal autoscaling and Cluster Autoscaling

The native HPA is metrics-based. As I previously discussed, neither CPU nor memory metrics are good indicators of time to scale. A process or a Pod may have a temporary high demand of CPU purely due to how programmers write the code. Even if we followed the best practices as above, I would still not regard CPU and memory metrics as a reliable indicator to drive auto scaling. If a service is a potential point of congestion, we should use a queue in front and the queue size is almost always a much better indicator of the timing to scale.

As to cluster autoscaler, on it FAQ, it says flat out that you should NOT use a CPU usage based scaling mechanism. I guess this is for a similar reason (compressibility). As discussed, when a Pod is pending for schedule for too long, it emits and event that drives the cluster autoscaler.

Summary

When I first worked on Kubernetes workload I did not give this much thought and proposed the use of CPU limit. As of January 2023 I still find static code analysis tools that requires CPU limit for Pods in the check (e.g. CKV_K8S_11 on Checkov), which leads me to investigate the issue further, and noticed more voices advocating the correct use of resource limit (such as this post) in 2022. For existing deployments, it is worth a review the resource limit configuration.

EKS impression

Fri, 23 Dec 2022 18:18:19 -0400

I’ve worked on a few AKS projects previously. Since I joined AWS I wanted to put aside some time to check out EKS (Elastic Kubernetes Service). Here in this post, I put down my first impression on EKS, and also share my Terraform template in cloudkube project to create an EKS cluster.

Similar to AKS, EKS exposes API endpoint and the control plane components are hidden from AWS users. When creating EKS cluster it does not create the underlying VPC and subnets. Therefore, you have create an existing VPC and at least two subnets ahead of time, and specify them during EKS creation. Bear in mind that there is a list of requirement for the VPC and subnets.

In the cluster, the CNI that EKS officially supports is Amazon VPC CNI plugin. It is available as an add-on. Similar to Azure CNI, each Pod gets its own IP address. In addition, EKS supports other compatible CNI plugins such as Calico, Cilium, Weave Net and Antrea.

Computing Nodes in EKS

There are three modes to address computing capacity: self-managed nodes, EKS managed node groups and AWS Fargate. The documentation has a comparison table.

With self-managed nodes, users create EC2 instances separately and then register them to the control plane. The instances must use the same IAM role and AMI. You can use Auto Scaling groups of Bottlerocket (AWS-sponsored purpose-built Linux distro for container host) nodes. The self-managed node option is mostly for AWS outpost customers who bring in their own computing capacity from data centre.

If you provision computing capacity from AWS, it makes sense to assign EKS managed node groups when creating EKS cluster. We can turn on Cluster Autoscaler, a Kubernetes construct to manage the auto scaling of node groups. Sometimes we want to have more than one node groups. For example, to build a multi-architecture cluster, we need one node group with amd64 nodes and the other with arm64 nodes (e.g. instances with Graviton processor). In general, arm-based CPU delivers better performance with less power consumption and the industry is slowly moving towards more arm-based CPU architecture.

Fargate is what I call managed computing service for EKS. With Fargate you do not need to tweak Cluster Autoscaler to self-manage computing capacity. The Fargate documentation has a long list of considerations. For example, Pods must match a Fargate profile (here‘s an example) at the time that they’re scheduled to run on Fargate. So we need to build Fargate profile and Pod labelling properly. Also, Fargate does not support DaemonSet. Another big consideration is that Fargate does not support non-VPC CNI. In my opinion these are pretty significant limitations. Many workloads (system-level or application-level) would need Daemonset (e.g. kube-proxy, some CNI or CSI drivers, Dynatrace monitoring).

The pro of Fargate is the serverless computing model. The construct of a Fargate profile isn’t complicated. You just specify subnets, namespace and labels. However, the downside is the long list of considerations. Some teams may consider these restrictions too much. The other overhead is the need to manage Fargate profile to ensure all Pods are scheduled somewhere.

To me, using Fargate alone impairs portability of workload. The good thing is that Fargate and Managed Node Group are not mutually exclusive on a cluster. In most cases, we can go partially serverless, and reap the benefits of both of them.

Node AutoScaling

For workloads that don’t have a matching Fargate profile, we have to figure out node autoscaling ourselves. I touched on Cluster Autoscaler in “Autoscaling on Kubernetes Platform“. CA works on AWS as well and is triggered upon a Pod coming to unschedulable status in Scheduler. There is some limitations though. For example, CA interacts with Autoscaling Group (instead of EC2 instances directly). When it determines it’s time to scale up, it bumps up the desired capacity by one at a time in the Autoscaling group. The configurations in Autoscaling group may also be at play and CA do not have direct control. For example, the “scaling cooldown“. The pool of nodes is homogenous as per the pre-configured launch template and CA has no control. If a Pod requires a different type of node (e.g. ARM64 CPU, spot instance, etc), then we’d first have to create a node group with the desired node type. Moreover, in the worst cases, one-at-a-time scale-up does not meet the increase of demand driven by Pod increases, causing nuances such as racing conditions.

Because the Cluster Autoscaler doesn’t really deal with the nodes themselves, this kind of integration is clunky and slow. Nearly half of Kubernetes customers on AWS report that configuring cluster auto scaling using the Kubernetes Cluster Autoscaler is challenging and restrictive, according to this blog post. As a result, AWS launched an open-source cluster autoscaler project, Karpenter. Karpenter first only supported EKS but now the support includes other CSPs. For EKS, Karpenter directly interact with different types of EC2 instances.

Karpenter makes node scaling work in a more cloud-native manner. In the presence of unschedulable Pods, Karpenter bypasses the Kubernetes scheduler and works directly with the Cloud provider, to launch the minimal compute resources needed to fit those Pods and immediately binds the Pods to the newly provisioned Nodes without waiting for scheduler. As Pods are removed or rescheduled to other nodes, Karpenter looks for opportunities to terminate under-utilized nodes. Karpender defines a CR called Provisioner to specify node provisioning configuration, such as instance size, zone, CPU architecture, etc. It is a manifest that describes a node group so the node scaler is aware of all the available node types. You can have multiple Provisioners for different needs, just like node groups. The Provisioner CR can also set TTL for empty Nodes, such that once a Node has no pods other than DaemonSet, Karpenter will terminate the Node on TTL expiry.

Karpenter’s idea is similar to the idea of AutoPilot cluster in GKE. The new EKS workshop has an section on how to set up CA and Karpenter in practice.

Identity Management for EKS

For IAM, we need to be concerned with three aspects. The management traffic to the cloud service, the management traffic for Kubernetes cluster and business traffic.

Traffic type	AWS	Azure
I. Cloud Service Endpoint (Management Traffic for Cloud Service)	AWS IAM identity	Azure RBAC
II. Kubernetes API (Management Traffic for K8s Cluster)	IAM mapping or OIDC	Azure RBAC (implementation of OIDC)
III. Business traffic	Up to Kubernetes Ingress	Up to Kubernetes Ingress

For business traffic (type III), it is all up to the Ingress. I’ve written another article on managing ingress traffic on Kubernetes platforms. We interact with cloud service endpoint (type II) with either AWS CLI or Terraform, to create any object, including resources needed for a cluster. This is generally how we work with cloud service, not specific to Kubernetes. Usually the IAM identity assumes another IAM role, which empowers it with a lot of permissions.

For access to Kubernetes API (type III), EKS supports OIDC and IAM mapping. AWS documentation refers to this as “Cluster Authentication“. There is one special scenario where your identity for type II access inherits your identity for type I access. As the document puts:

When you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s role-based access control (RBAC) configuration in the Amazon EKS control plane. This IAM entity doesn’t appear in any visible configuration, so make sure to keep track of which IAM entity originally created the cluster.

This special scenario (I call it the “implicit master user“) allows us to perform critical activities on the cluster, such as creating IAM mapping, or OIDC configuration.

The above addressed how AWS resource access Kubernetes resource. On the other hand, to address how a Kubernetes resource access AWS resources, we need IRSA (IAM Roles for Service Account). We have a service account in Kubernetes and map it to an IAM role.

AppMesh

AppMesh is AWS’ Envoy based service-mesh offering supporting Kubernetes cluster, ECS service and even EC2 instance. AppMesh’s control plane is a managed AWS service, with a controller running on the Kubernetes cluster. To install AppMesh on the cluster:

On the EKS cluster, install AppMesh Controller using Helm
Associate the cluster with IAM OIDC provider
Create an IAM role for the appmesh-controller service account

After these steps, you can create a mesh using CloudFormation, Terraform, etc. The data plane (Envoy proxy) can run on Kubernetes (as sidecar). Traffic between control plane and data plane can go through private link (Interface VPC endpoint) for added security. Like Istio, AppMesh enables mTLS. For observability, you can export Envoy metrics with Prometheus. Coupled with XRay, AppMesh also supports distributed tracing.

AppMesh uses a different set of CRDs than Istio. Key CRDs are:

Mesh: represents an entire service mesh. At mesh level you can configure Egress filter (to allow or deny external traffic) and set IP version (v4 vs v6)
VirtualGateway: a CRD that represents an Ingress in to the Mesh. A virtual gateway allows resources that are outside of your mesh to communicate to resources that are inside of your mesh. A virtual gateway references Envoy proxy deployment by podSelector. It references GatewayRoutes by namespaceSelector, and optionally gatewayRouteSelector. You also specify listeners in the manifest to reference Envoy proxy Service (LoadBalancer Type).
GatewayRoute: A gateway route is attached to a virtual gateway and routes traffic to an existing virtual service. If a route matches a request, it can distribute traffic to a target virtual service. In the manifest, you specify a list of httpRoute, each with matching condition and action. In the action section you can specify virtualService as target.
VirtualService: an abstraction of a real service provided by a virtual node directly or indirectly by means of a virtual router. Dependent services call your virtual service by its virtualServiceName, and those requests are routed to the VirtualNode or VirtualRouter that is specified as the provider for the VirtualService.
VirtualRouter: Virtual routers handle traffic for virtual services. In a virtual router manifest, you can define Route to direct incoming requests to virtual nodes as target.
VirtualNode: A virtual node acts as a logical pointer to a particular task group (i.e. ECS service, Kubernetes deployment). It represents a Service in the AppMesh. In the manifest, you reference Pods by podSelector, specify listeners for any inbound traffic that your virtual node expects, and specify serviceDiscovery for your task group.

You can configure those Custom Resources using YAML manifests (and check the API reference a lot). Alternatively, you can configure them from AWS CLI or AWS console. The console will help you visualize what can be configured. For further details on how these CRs play together, there is a workshop for AppMesh.

EKS cluster using Terraform

Officially, there is an EKS blueprint project for provisioning EKS cluster in Terraform.

I also keep my own Terraform code in the AWS directory of cloudkube project. It works out to be a little more complex than my Terraform template to create Azure Kubernetes Cluster (Azure directory). Because I had to create Cognito resources with initial credential to allow users to connect to cluster without using the implicit master account.

Below is the diagram of the processes.

Create EKS cluster with Terraform module

The template configures kubectl access on a Bastion host, which assumed the same role that our IAM user uses to create the Kubernetes cluster. Therefore, the IAM role is the master identity.

Note that the IAM user (power-user) has very powerful permissions. Usually it is ideal to assign lots of permission to IAM Roles (temporary credential) instead of IAM user (long-term credential). So the role chaining would look like:

The IAM user that Terraform uses has no permission other than assuming a “PowerUser” role
The PowerUser role trusts the IAM user. It also has the permission to assume the “EKS-Manager” role
The EKS-Manager role trusts PowerUser’s role session.

However, the role chaining scenario above is not currently supported in Terraform.

I use a Bastion host because the cluster endpoint is on private subnet. The bastion host is on a public subnet. However, if we do not like public subnet and public IP, we can place the bastion host on a private subnet, and use SSM system manager agent with SSH tunnel plugin to have SSH access to private bastion host.

Summary

I first came across this article about EKS and its awfulness and then decided to check out EKS. I’m not sure all points are still valid but it’s generally real-life experiences. There are also many peripheral services, such as AMP (AWS Managed Prometheus), AMG (AWS Managed Grafana), ADOT (AWS Distro for Open Telemetry), AppMesh (Another Envoy-based Service Mesh, easier to manage than Istio but less Powerful), with a lot to explore.

Istio Operation Gotchas

Sat, 19 Mar 2022 11:09:00 -0400

In this post I discuss a few aspects when putting istio in operation.

Installation

Istio installation can be confusing, due to architectural and guideline changes as well as renaming of operator CRDs since its release, and especially since 2020. This left lots of information outdated on the web, adding to Istio’s perceived complexity.

Currently, the recommended installation methods are istioctl or Helm. Using Istio Operator manifest (with out istioctl) is discouraged. As to the Helm chart installer, it was once deprecated (around early 2020), but was later re-introduced in the fall of 2021. This back-and-forth had caused some aversion. As of Nov 2021, their re-introduced Helm charts version dropped alpha tag. The Helm repo consists of separate Helm charts, for Istio CRD (base), control plane (istiod), each gateway and CNI respectively. A typical deployment therefore requires multiple Helm Releases (example here). Istio document still considers Helm support as alpha, so I assume the most reliable method to install Istio is istioctl.

The istioctl utility can be used with many options to customize Istio install. For example, we can supply a YAML declaration input (to -f switch) to customize installation behaviours. The YAML file declares a CRD. Two different CRDs have been used: IstioOperator and IstioControlPlane. According to this blog and this post, since Istio 1.5 in early 2020, we’re supposed to IstioOperator CRD exclusively. The IstioControlPlane CRD is left only for legacy support. As stated in the current documentation:

The istioctl command supports the full IstioOperator API via command-line options for individual settings or for passing a yaml file containing an IstioOperator custom resource (CR).

With IstioOperator CRD, we still have a number of options to tweak the install behaviours. Here is a summary of potential options:

use IstioOperator API via IstioOperator CRD (without using “values” or overlay fields)
specify an attribute value in argument, including a pre-built profile e.g. –set meshConfig.accessLogFile=/dev/stdout, –set profile=demo
use K8sObjectOverlay by using “k8s/overlays/patches” field in IstioOperatorCRD
use Helm API by using “values” field in IstioOperatorCRD

The key-value specified in –set switch overrides the same key-value supplied in the IstioOperator CRD. So option 2 overrides option 1. The value for profile can also be empty if you’d rather start from scratch. However too many –set switches makes the command wordy so we should build our own IstioOperator CRD

For option 4, the document hyper-links Helm API to a section from version istio 1.4, and I appears to exist only for legacy (pre-2020 Helm support) compatibility. Option 3 (K8sObjectOverlay) would be helpful when a field cannot be conveniently customized with option 1 and we have to patch the object like in Kustomization.

So the most practical approach is IstioOperator CRD for per-component customization, potentially with K8sObjectOverlay. No matter which option, istioctl compiles the installation manifest before applying it against Kubernetes API. This manifest can be previewed using “istioctl manifest” command, so that you can take a look before installation.

Thank you Isito for so much confusion just to land on a working installation method. Below is the content of az-istio-operator.yaml file that I use for my installation:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
 name: istio-install-customization
spec:
 profile: default
 hub: docker.io/istio
 tag: 1.13.1
 revision: 1-13-1
 namespace: istio-system
 meshConfig:
 accessLogFile: /dev/stdout
 outboundTrafficPolicy:
 mode: "REGISTRY_ONLY"
 components:
 pilot:
 k8s:
 hpaSpec:
 maxReplicas: 7
 minReplicas: 1
 nodeSelector:
 beta.kubernetes.io/os: linux
 ingressGateways:
 - name: istio-ingressgateway
 #namespace: istio-ingress
 enabled: true
 label:
 istio: ingressgateway
 k8s:
 hpaSpec:
 maxReplicas: 11
 minReplicas: 1
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "my-lb-subnet"
 egressGateways:
 - name: istio-egressgateway
 enabled: true

I can preview the install, run the install and validate installation status:

$ istioctl manifest generate -f az-istio-operator.yaml
$ istioctl install -f az-istio-operator.yaml -y --verify
$ istioctl verify-install -f az-istio-operator.yaml
$ kubectl -n istio-system get IstioOperator installed-state-istio-install-customization-1-13-1 -o yaml | less

Note that in the IstioOperator declaration I marked the revision. This is helpful when I run multiple versions of control plane (e.g. during upgrade). I can check revisions of istiod with:

$ istioctl x revision list

We can delete installed istio components

$ istioctl x uninstall -f az-istio-operator.yaml
$ istioctl x uninstall --revision 1-11-5

In practice, it is helpful to use separate operators each for a different component (pilot, ingressGateways, egressGateways). This makes maintenance and upgrade easier.

Debugging

The documentation has a page for common problems that one needs to be familiar with. It covers not only problems, but also steps to troubleshoot each kind of problem (e.g. authorization policy).

Although istioctl is pretty confusing as an installation tool, it is a good utility for many troubleshooting activities. We should probably add its path to PATH environment variable and add the export command (e.g. export PATH=”$HOME/istio/bin:$PATH“) to .zshrc or .bashrc. Istioctl has a few useful subcommands, some of which are only available as experimental and therefore needs to be following an x. Some common commands are given below:

To analyze Istio problems:

$ istioctl analyze -n istio-system

To look at proxy configuration of an Envoy instance at different levels, use proxy-config sub-command or pc for shorthand:

$ istioctl proxy-config <clusters|listeners|routes|endpoints|bootstrap|log|secret|stats>
$ istioctl pc cluster deploy/istio-ingressgateway -n istio-system # see what Envoy cluster an ingress gateway knows about
$ istioctl proxy-config log deploy/httpbin --level "rbac:debug"
$ istioctl pc log <pod_name> -n <namespace> --level connection:debug
$ istioctl pc secret -n istio-system deploy/istio-ingressgateway # check certificates loaded to a gateway
$ istioctl proxy-config listeners deploy/istio-ingressgateway -n istio-system # query envoy listener configuration
$ istioctl pc routes deploy/istio-ingressgateway -n istio-system --name http.8080 # query envoy route configuration

The last two commands set logging level to the specified workload. If we want to set logging level at mesh level, we can use these commands:

$ istioctl admin log --level authorization:debug
$ istioctl admin log

To look at synchronization status of each envoy in the mesh, use proxy-status sub-command, or ps for shorthand:

$ istioctl ps # ensure data plane is in sync

If an item for a workload shows STALE instead of SYNCED, it means that the configuration has not been pushed from control plane to that instance of Envoy proxy. Check if the Istio configuration change is valid. If the system is newly installed and there is no ingress or egress gateway resources declared, the RDS column for ingress or egress may show “NOT SENT”. The far right column displays the version of istiod connected.

To describe applied istio config:

$ istioctl describe <pod|service>
$ istioctl describe po workload1 -n my-workload # detect misconfigurations on workload

To view dashboard:

$ istioctl dashboard <envoy|grafana|prometheus>

To check authorization policy on a Pod,

$ istioctl x authz check mypod -n workload

To validate istio configuration in a file:

$ istioctl validate -f resource_authorization_policy.yaml

Sometimes we need to turn on access logging just on the envoy proxy on the Gateway Pod. In that case, we will need to apply Istio’s Envoy filter object. This filter is applied to Pods labelled as gateway. It patches the existing filter chain with the additional defined in the manifest:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
 name: hcm-gw-access-log
 namespace: istio-system
spec:
 workloadSelector:
 labels:
 istio: ingressgateway
 configPatches:
 - applyTo: NETWORK_FILTER
 match:
 context: GATEWAY
 listener:
 filterChain:
 sni: demo.digihunch.com
 filter:
 name: "envoy.filters.network.http_connection_manager"
 patch:
 operation: MERGE
 value:
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
 access_log:
 - name: envoy.access_loggers.file
 typed_config:
 "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
 path: /dev/stdout
 format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

The object above enables access logging on the Gateway Pods only, without impacting other Envoy proxies.

Multi-tenancy

In large enterprises, Istio is usually installed by a platform/operation team. The entire platform is shared by multiple application teams. Istio should not be seen as a responsibility of a single party. It is necessary to break down the Istio CRDs and define the responsibility of each CRD. For example:

Gateways are placed in the istio-system namespace (or a dedicated istio-ingress namespace in some case). Platform team can decide whether a Gateway is considered a shared infrastructure, or each tenant (application team) uses their own Gateway. In the servers/hosts field of Gateway declaration, add namespace before host to suggest that the routing behaviour of that host must be defined in a certain namespace.
Virtual Services can be managed in two models as well. They can be managed individually and are all placed in each tenant namespace. Alternatively in a shared responsibility model, Virtual Service can be created in istio-system namespace, and the processing of each match can be delegated to a Virtual Service in each tenant namespace, using the Delegate feature of Virtual Service. Another field that can help with multi-tenancy is the “gateways” field, you can specify a value of mesh to indicate the virtual service is available to the entire mesh.
Destination Rules are usually placed in each Tenant’s workspace
Peer Authentication also depends on the configuration. If a mesh level configuration is enforced, it is easier for the Platform Team to manage it and this should be the setup for new clusters. If for historical reasons Peer Authentication is enforced per tenant namespace, it can be delegated to each application team. It can make communication troubleshooting more complex.
Control Plane and observability workloads such as Kiali are usually the responsibility of Platform team.
Request Authentication and Authorization Policy should be the responsibility of individual application team.

In a multi-tenancy management model, Istio related resource should be subject to admission control policy, as well as scrutiny by the security and platform teams. Google Anthos has a good page on the constraint templates for Istio resources.

As to restricting traffic between namespaces, apart from the measures from this previous post, we can also use Sidecar CRD to restrict outbound traffic. As its documentation states:

Sidecar describes the configuration of the sidecar proxy that mediates inbound and outbound communication to the workload instance it is attached to. By default, Istio will program all sidecar proxies in the mesh with the necessary configuration required to reach every workload instance in the mesh, as well as accept traffic on all the ports associated with the workload. The Sidecar configuration provides a way to fine tune the set of ports, protocols that the proxy will accept when forwarding traffic to and from the workload. In addition, it is possible to restrict the set of services that the proxy can reach when forwarding outbound traffic from workload instances.

The documentation page also includes two examples. The first is a sidecar at the mesh level that restricts outbound traffic to the same namespace that the sidecar is in, and the istio-system namespace:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: istio-config
spec:
 egress:
 - hosts:
 - "./*"
 - "istio-system/*"

The second example overrides the mesh level default above, and allows egress traffic to three specified namespaces:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
 name: default
 namespace: prod-us1
spec:
 egress:
 - hosts:
 - "prod-us1/*"
 - "prod-apis/*"
 - "istio-system/*"

As to inbound control, we can expose a Virtual Service to other namespaces by using the exportTo field to specify which other namespaces the Virtual Service should be exported to. If no namespaces are specified then the virtual service is exported to all namespaces by default.

Sometimes we want a virtual services to expose to both outside of the mesh via Ingress, and within the mesh, and we hope to use the same hostname. For this requirement, we can use the ServiceEntry CRD. ServiceEntry enables adding additional entries into Istio’s internal service registry.

Service Proxy – from Nginx to Envoy

Wed, 09 Mar 2022 21:36:00 -0400

Update (Nov 20, 2022): 1. Envoy’s configuration schema can be hard to get used to. It is lacking examples because the documentation is mostly generated. Use its examples directory to find real-life configuration examples. 2. the Envoy implementation in the example project has been reverted in favour of Nginx.

Envoy proxy is the underlying technology for Istio, as well as a number of other service mesh products, such as AppMesh (AWS), Consul (Hashicorp) and OpenServiceMesh (Azure). Most of the capabilities of Isito is ultimately provided by Envoy proxy. Envoy has a page outlining its differences with similar technologies. I decided to take a look into Envoy by replacing Nginx with it.

Rate limiting and Circuit Breaker

In most SDLC, it is application developers that create backend APIs or server applications. Most developers specializes in application features, and cannot fathom all the nuances with the TCP/IP network stack. Nginx allows them to push network concerns (non-business features) to a dedicated proxy to handle the dynamics in network connection. Nginx can be configured as both a reverse proxy (handling incoming connection on behalf of the process) and a forward proxy (handling outgoing connection on behalf of the process). This is the prototype of sidecar pattern, an important idea behind service mesh. For example, when a sudden increase in connection to the server-side application, the server process could be either unresponsive (refer to “the queuing knee“, and Little’s Law), or OOM killed. When such interruptions are not automatically recovered, a downtime is caused. This traditionally requires some congestion control strategy for TCP/IP queue but two features provided by a network proxy can help circumvent this situation: rate limiting, and circuit breaking. Rate limiting keeps more requests above threshold from entering the queue. Circuit breaker releases downstream pressure by cutting out existing in-queue request. Nginx added both over the years but Circuit breaker still remains a premium feature exclusive to Nginx Plus. Envoy on the other hand has them free when it was launched.

Envoy also supports other advanced traffic management such as traffic shaping, and mirroring. It is on top of those features that Istio introduces its own abstraction such as virtual service, destination rules to its users. In that sense, we can think of Istio as a configurator (control plane) for Envoy proxy (data plane), similar to Ansible to Nginx proxy instances.

Dynamic Configuration via API

I used Nginx previously with traditional environment and loved its flexibility. As the system grows, I started to feel the pain of management overhead. With one of the production system, there were 25 + instances of Nginx each running on a VM and I managed configuration files with Ansible. Ansible pushes out configuration files and triggers a reload from each Nginx instance. In the cloud-native era where Pods are ephemeral, this kind of overhead would snowball to an unmanageable level. Envoy was designed for cloud-native applications, with all these kinds of problems in mind. Envoy has dynamic configuration. The majority of the configurations can be pulled from xDS API, or file system. Updating configuration drains connections gracefully without runtime having to reload the file. The idea of centrally managing Nginx instances with Ansible, also evolved into the concept of control plane.

TLS origination

In the Orthweb project, I used Nginx to proxy TLS and HTTP traffic, and performed TLS termination on both ports. This is know as TLS offloading. The traffic between the proxy and the upstream service takes place in the clear, even though they do not travel across different network interfaces in most cases. For a true end-to-end encryption, it is helpful to also encrypt the traffic between proxy and upstream server. This requires the capability of securing TCP traffic to upstream server. With Nginx, the ability to secure HTTP traffic to upstream server is offered in open-source. The ability to secure TCP traffic is available in Nginx Plus, or with self-compiled binary. In Envoy, both are available using the UpstreamTlsContext configuration.

More pros

In another post, I also discussed Nginx as a LDAP proxy to front services such as Kibana and Nifi. It requires a proxy service (ldap-auth in this case), to defer auth to third party. Envoy has this capability using a filter with extension for external authorization. Istio also exposes this capability, an enabler for the configuration proposed in my previous post. Envoy also uses WebAssembly for its extensibility.

Another useful feature is protocol detection. It can use filters to detect protocol (TLS or regular TCP) and route traffic to predefined destination.

Performance wise, this benchmark from 2018 ran a comparison among the popular options where Envoy leads by a margin.

Observability (logging, metrics and tracing) are well supported in Envoy. User can configure format of logs that takes effect immediately. There are many metrics that works with Prometheus and they are expandable using filters. On the tracing side, Envoy supports integration with jaeger, zipkin and datadog.

Basics of Envoy

The configuration of Envoy is more involving. There is an Envoy course by Tetrate, as well as two blog entries for envoy 101: Envoy as gateway proxy and File-based dynamic configuration. Another good way to get started is the Sandboxes projects, which covers a number of different areas of configuration. The admin port (by default at port 9901. Istio’s default is 15000) provides helpful information. If we need to turn on debug on some features, we can do so with curl:

curl -X POST http://localhost:9901/logging?client=debug

Stats are exposed at the same port:

curl -X GET http://localhost:9901/stats

When packets are received at a listener, the packet is first processed by listener filters. Then, depending on filter match, one or more network filter chains will further process the packet, including further actions, as illustrated below:

Envoy supports dynamic configuration, which uses a set of discovery services (xDS) APIs. Some of the important xDS APIs include:

LDS (Listener Discovery Service) – allows you to add listeners dynamically while Envoy is running
RDS (Route Discovery Service) – allows you to dynamically update routes for HTTP connection managers
CDS (Cluster Discovery Service) – allows you to update cluster definitions dynamically
EDS (Endpoint Discovery Service) – allows you to add or remove endpoints dynamically
Secret DS

The relation can be illustrated in this diagram below:

This post from Tetrate has more examples.

Nginx to Envoy

For the advantages of Envoy, I decided to migrate from Nginx to Envoy on my Orthweb project. Using Envoy as service proxy is not where Envoy is mostly used (as sidecar), but it is how Envoy was originally used at Lyft to replace ELB in 2015.

The original Nginx configuration was referenced in this old blog post. The Envoy setup also covers both TCP (DICOM) and HTTP (HTTPS) traffic. For HTTP traffic, it also encrypts the traffic to upstream. Below is what it looks like:

admin:
 address:
 socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
 listeners:
 - name: https_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 443
 filter_chains:
 - filters:
 - name: envoy.filters.network.http_connection_manager
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
 codec_type: AUTO
 stat_prefix: ingress_http
 route_config:
 name: local_route
 virtual_hosts:
 - name: app
 domains:
 - "*"
 routes:
 - match:
 prefix: "/"
 route:
 cluster: service-https
 http_filters:
 - name: envoy.filters.http.router
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 - name: dicomtls_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 11112
 filter_chains:
 - filters:
 - name: envoy.filters.network.tcp_proxy
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
 stat_prefix: downstream_cx_total
 cluster: service-dicomtls
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 validation_context:
 allow_expired_certificate: true
 trusted_ca: {"filename": "/etc/ssl/certs/site.pem"}
 require_client_certificate: false
 clusters:
 - name: service-https
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-https
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 8042
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
 - name: service-dicomtls
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-dicomtls
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 4242
layered_runtime:
 layers:
 - name: static_layer_0
 static_layer:
 envoy:
 resource_limits:
 listener:
 https_listener:
 connection_limit: 1000
 overload:
 global_downstream_max_connections: 5000

From Nginx to Envoy, to achieve nearly the same functionalities, it takes 100 lines of configuration instead of less than 30. The configuration also appears more abstract, which is one of the cons of Envoy to consider before the migration.

Update from Jan 2025 – To align with what most uses in Orthanc community, the project switched back to using Nginx. The envoy configuration is kept here.

Istio External Authorization via OIDC

Fri, 25 Feb 2022 01:29:00 -0400

Istio service mesh allows application developers to offload non-core features to infrastructure layer. We explored authentication and authorization with Istio in a basic lab. In this post we continue to explore its capabilities with OIDC integration. This capability is made available thanks to the CUSTOM action in authorization policy, supported since the release of 1.9. It enables any workload on Istio to integrate with an external IAM solution.

The rest of this post, provides the step-by-step instruction to configure OIDC integration, based on Istio’s External Authorization use case. My work is influenced by two blog posts from jetstack and elastisys on similar topic, with my own additions, simplifications and clarifications. In this lab I use my own DNS hostname demo1.digihunch.com and to follow along you need to bring your own hostname as well. The prerequisites of this lab is summarized as below:

An identity provider: we use Google in the lab but it can be anything with OIDC capabilities (e.g. Azure AD, Facebook). Read my old post for more details on OIDC.
A public IP address for Cluster’s Ingress service: Since we use Google as identity provider on the Internet, we need our service to be exposed on the Internet because the OIDC integration involves a two-way HTTP redirect.
Public DNS resolution to the service’s public IP: during authentication, the identity provider will call back with JWT. So we need our hostname demo1.digihunch.com resolved to the service’s public IP address. This is also required in other scenarios. For example, cert-manager needs to automatically configure X.509 certificate using Let’s Encrypt.
An application workload: this is the tenant workload without its own IAM capability. We use a dummy workload base on a hello-world image, for simplicity.

The files required in this lab is in istio-oidc repo under the envoy directory. It is also important to understand the Authentication Code flow in OIDC authorization code flow to make sense of the integration between OIDC provider and Istio on our platform. For example, the client application needs a one-time registration with identity provider. Let’s start with this configuration.

Configure Identity Provider

We need a Google account with a GCP project and log on to APIs & Services to register our application. At the console, from the left side bar, click on “OAuth consent screen” and create an App. Provide the followings:

Step 1. OAuth consent screen. Under App Information, provide App name and User support email.
Step 1. OAuth consent screen. Under App domain, provide Application home page
Step 1. OAuth consent screen. Under Authorized domains, provide an Authorized domain.
Step 3. Test users. Add the emails for Test users.

The summary page looks like this:

OAuth Consent Screen

Then on the sidebar, click on Credentials. We create new credentials for OAuth 2.0 Client ID. Select Web application and add “https://demo1.digihunch.com/oauth2/callback” as an Authorized redirect URI. The credential created includes a Client ID and a Client secret, which can be downloaded as a JSON file. Here is my screen:

API Credential

Note that the redirect URIs cannot be localhost. Because the browser will consume the redirect URI and it has no clue where localhost is. It must resolve to the public IP of the ingress.

We need to keep the Client ID and Client secret for later use. According to Google’s documentation, there is a discovery document from which we can understand the keys in their Open ID configuration. We need the value of two keys (issuer and jwks_url) to use later in the configuration.

Configure Infrastructure Services

My real-quicK-cluster project has a few different ways to quickly bring up a cluster. For this lab, I use Azure CLI command to create a simple, three-node cluster:

$ az aks create \
 -g AutomationTest \
 -n orthCluster \
 --node-count 3 \
 --enable-addons monitoring \
 --generate-ssh-keys \
 --vm-set-type VirtualMachineScaleSets \
 --network-plugin azure \
 --network-policy calico \
 --tags Owner=MyOwner

Then we configure kubeconfig credential to connect to the cluster, so that kubectl and helm CLI commands to connect to the newly created cluster.

$ az aks get-credentials --resource-group AutomationTest --name orthCluster

Then we will install the infrastructure services and workload. We can start with loading environment variables for use later:

OIDC_DISCOVERY=$(curl "https://accounts.google.com/.well-known/openid-configuration")
OIDC_ISSUER_URL=$(echo $OIDC_DISCOVERY | jq -r .issuer)
OIDC_JWKS_URI=$(echo $OIDC_DISCOVERY | jq -r .jwks_uri) 
COOKIE_SECRET=$(openssl rand -base64 32 | tr -- '+/' '-_')
WEB_HOST="demo1.digihunch.com"
CLIENT_ID="ThisIsTheClientIDFromGoogle"
CLIENT_SECRET="ThisIsTheClientSecretFromGoogle"

For WEB_HOST, CLIENT_ID, CLIENT_SECRET, you need to bring your own values. The COOKIE_SECRET value is randomly generated. The variables OIDC_ISSUER_URL and OIDC_JWKS_URI are parsed from Google OpenID configuration and they should remain static. We can validate the value with echo commands. Also, to use helm v3 later, we need to add the repositories we need. These steps can be skipped if they have been performed in the client environment:

$ echo $OIDC_ISSUER_URL
https://accounts.google.com
$ echo $OIDC_JWKS_URI
https://www.googleapis.com/oauth2/v3/certs
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo add istio https://istio-release.storage.googleapis.com/charts
$ helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
$ helm repo update

Now, with Helm ready, we can install Cert Manager, Istio CRD, Control Plane, Gateways as well as OAuth2-Proxy using Helm. We run each of the following commands from the envoy directory, where the required files are stored:

$ helm install cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --create-namespace \
 --version v1.7.1 \
 --set installCRDs=true
$ helm install -n istio-system istio-base istio/base --create-namespace
$ helm -n istio-system install istiod istio/istiod -f istiod-values.yaml --wait
$ helm -n istio-system install istio-ingress istio/gateway -f ingress-gateway-values.yaml
$ kubectl -n istio-system get po
$ kubectl create ns oauth2-proxy && kubectl label ns oauth2-proxy istio-injection=enabled
$ helm install -n oauth2-proxy \
 --version 6.0.1 \
 --values oauth2-proxy-values.yaml \
 --set config.clientID=$CLIENT_ID \
 --set config.clientSecret=$CLIENT_SECRET \
 --set config.cookieSecret=$COOKIE_SECRET \
 --set extraArgs.oidc-issuer-url=$OIDC_ISSUER_URL \
 --set extraArgs.whitelist-domain=$WEB_HOST \
 oauth2-proxy oauth2-proxy/oauth2-proxy
$ kubectl -n oauth2-proxy get pods -l app=oauth2-proxy

Note that the istiod installation step uses a value file that includes extensionProviders. This is where we tell Istio to connect to external authorization provider:

meshConfig:
 accessLogFile: /dev/stdout
 extensionProviders:
 - name: oauth2-proxy
 envoyExtAuthzHttp:
 service: oauth2-proxy.oauth2-proxy.svc.cluster.local
 port: 4180
 includeRequestHeadersInCheck:
 - cookie
 headersToUpstreamOnAllow:
 - authorization
 headersToDownstreamOnDeny:
 - set-cookie

When we install istio ingress gateway, we need to expose port 80 as well as 443. Port 80 is used in ACME protocol for certificate configuration. We also installed oauth2-proxy with some configurations from oauth2-proxy-values.yaml as well as some argument set imperatively.

Once configuration is successful, we should be able to confirm that oauth2-proxy service is running. We should also be able to tell the External IP address of the Ingress:

$ kubectl -n oauth2-proxy get svc
$ kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Now we can go to our DNS configuration portal, to populate the DNS A-record for demo1.digihunch.com with this IP address:

Add DNS A-record

It may take several minutes to a couple of hours, for the A-record update/creation to take effect, depending on the TTL. Sit tight and query DNS with nslookup command, until it is set. In the next step, we will need successful DNS resolution for certificate configuration.

Install workload and configure Certificate

Now we install the workload, along with the necessary constructs such as Gateway CRD, Virtual Service, and certificates. We can use kubectl kustomize command to preview the changes, then apply the changes with -k:

$ kubectl kustomize demo
$ kubectl apply -k demo

When you have your own DNS hostname, modify the kustomization.yaml accordingly before applying. This command uses Kustomization to apply numerous YAML manifests in the demo directory, including the following activities:

Create a Namespace named demo
Create Deployment and Service
Set up PeerAuthentication for the mesh
Configure Ingress class, ClusterIssuer and Certificate using cert-manager
Configure Virtual Service and Gateway

This configures everything we need and we need to verify from several aspects. The demo Service is a ClusterIP service exposed within the cluster at port 80. Then we check the status of certificate:

$ kubectl -n istio-system get certificate
$ kubectl -n istio-system describe certificate demo

The column of READY should have a value of True. Check the logs from cert manager Pods if that’s not the case. Common reasons include:

The automatic ACME validation is just not ready
The automatic ACME validation is still waiting for DNS resolution
The Istio ingress gateway port 80 is not open for ACME validation
The let’s encrypt server applies rate limiting

Note that for the ACME server, we can use staging server’s URL or productions. The former provisions a certificate that may come off as insecure as the CA is not fully trusted by browser. The latter doesn’t have this issue, but the server applies rate limiting. For more details of how validation with ACME work, check out my previous post.

Once the certificate is applied, our website is open to any visitor with HTTPS access:

Then we will use Request Authentication and Authorization Policy to tighten up the access by requiring visiting user to log in with Google identity.

RequestAuthentication and AuthorizationPolicy

We configure a CUSTOM action in the AuthorizationPolicy, and specified the provider by name oauth2-proxy. The YAML manifests for RequestAuthentication and AuthorizationPolicy looks like this:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
 name: istio-ingressgateway
 namespace: istio-system
spec:
 jwtRules:
 - issuer: https://accounts.google.com
 jwksUri: https://www.googleapis.com/oauth2/v3/certs
 selector:
 matchLabels:
 app: istio-ingressgateway
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: istio-ingressgateway
 namespace: istio-system
spec:
 action: CUSTOM
 provider:
 name: oauth2-proxy
 rules:
 - {}
 selector:
 matchLabels:
 app: istio-ingressgateway

Note that we cannot apply Request Authentication and Authorization Policy before we confirm that the Certificate has been created. This is because when letsencrypt use HTTP resolver to provision certificate with ACME protocol, the request from letsencrypt should get to the validating URI without being asked to provide a Google identity. So do not perform this step yet before confirming the previous step:

$ kubectl apply -f oidc-auth.yaml

Now launch your browser in incognito mode and browse to our URL (https://demo1.digihunch.com) and you will be re-redirected to log on to Google (accounts.google.com). Once logged on, you will have access to the site.

If you run into errors, check the oauth2-proxy log from Pod stdout, which should give a reason of 4xx errors. If oauth2-proxy log indicates no activity, confirm if the request has been forwarded to the proxy. Check the service object of the proxy and make sure it is exposed to the correct port, as indicated in the meshConfig. To check mesh config, examine the configmap named istio in the namespace of istio-system. To check if Istio’s authorization is unable to speak with oatuh2-proxy, inspect the log of istiod Pod.

To clean up the lab, remove the app registration from Google, and then delete the cluster from Azure:

$ az aks delete -g AutomationTest -n orthCluster

In this lab we used a very open AuthorizationPolicy. We can polish it up with more conditions such as:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: istio-ingressgateway
 namespace: istio-system
spec:
 action: CUSTOM
 provider:
 name: oauth2-proxy
 rules:
 - when:
 - key: request.auth.audiences
 values:
 - $CLIENT_ID
 selector:
 matchLabels:
 app: istio-ingressgateway

Under the rules section, we can explore the claims from the JWT and make creative use of it to achieve our specific goals for IAM.

Bottom line

In this lab, we registered our app with Google and obtained client ID and secret for it to identify our application. We configured oauth2-proxy as our IAM application, carrying the credential validated by Google. We also configured Istio to delegate authorization to oauth2-proxy as external authorization provide, making up an entire OIDC integration.

The third party service (IdP) we integrate with is GCP, which natively supports OIDC protocol. However, if the third party IdP does not support OIDC natively, such as Active Directory via LDAP, then we would need one more proxy to sit between OAuth2 and the external IdP service. One good choice is the Dex connector. Another more complex yet powerful alternative connector is Key Cloak. Consider Dex or KeyCloak as an OIDC proxy, when:

The external IdP does not support OIDC; or
We choose not to use the OIDC capability of the external IdP because we want to managed OIDC centrally in our own proxy
The external identity provider does support OIDC, but the configuration is managed by a different team (e.g. IAM team) other than the platform team

AuthorizationPolicy is the key piece in this integration, and it is executed at the http filter in envoy sidecar proxy. It can be done with alternatives to OAuth2-Proxy such as the authservice project. A slightly different approach to implement OIDC integration is to use leverage extensibility of WebAssembly (a sandboxing technology to extend Envoy), which is suggested in this example for WasmPlugin. However, Wasm is still considered experimental.

Istio Authentication and Authorization

Sat, 05 Feb 2022 23:04:00 -0400

Applications running on Kubernetes platform seeks to offload common non-business features to the platform. Istio helps Kubernetes bridge that gap. It can enforce mTLS communication, which is known as Peer Authentication. It can help with two other things with the use of JWT token: when a web request presents a JWT token, it can validate whether it is authentic. Then, it can use the claims in JWT token to drive authorization decision on whether the specific request is allowed or denied. Both will use Istio CRDs.

Peer Authentication

Istio can enforce mTLS for TCP traffic between Pods. According to its documentation, enforcing mTLS at mesh level is as simple as applying a Peer Authentication resource to the root-level namespace:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: default
 namespace: istio-system
spec:
 mtls:
 mode: STRICT

The role of mTLS is so Pods can validates each other’s identity and then encrypt the TLS traffic in between. Each workload must first have an identity and Envoy proxy addressed this issue by adopting SPIFFE framework. It gives each workload an identity in the format of <TRUST_DOMAIN>/ns/<NAMESPACE>/sa/<SERVICE_ACCOUNT>. For example: spiffe://cluster.local/ns/myapp-dev/sa/default. It is also important to understand that only Pods with injected Envoy sidecar have SPIFFE workload identity and therefore is able to speak in mTLS. For new services, this is usually not an issue. For migrating workload without sidecar, a Pod without sidecar may connect with one in the mesh (with sidecar) if the mtls mode is PERMISSIVE in Peer Authentication. Otherwise, the connect is reset at layer 4 with the following error:

curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56

Therefore, it is advisable to start with PERMISSIVE mode for a precautionary migration of workload to mTLS. With mTLS all effective at the mesh level, there is no need to natively configure TLS between services.

The SPIFFE identity used in PeerAuthentication can also be used in Request Authorization as rule conditions. I will discuss request authentication before request authorization. To understand request authentication, let’s first warm up on JWT.

JSON Web Token

JSON Web Token (JWT, RFC 7519) is a format to carry JSON payload with optional signature and/or encryption. It can be thought of as a document (in JSON format) with signature for web servers to exchange information. The signature portion makes it friendly for document consumers to validate the authenticity. It is also URL-safe, and thereby adopted in web-browser SSO context, to pass identity of an authenticated user between and identity provider and a service provider.

JWT enables token-based authentication, a significant improvement from traditional session-based authentication. The traditional session-based authentication can be illustrated as below:

session-based authentication

This authentication model has major drawbacks. First, a mechanism to validate the authenticity of Cookie is missing. Second, the server has to keep the session information, making itself not stateless, unless a state store such as memcached is introduced.

token-based authentication

In token-based authentication such as using JWT, a token is issued. The authenticity of the token are validated before the server provides data, and it can be validated by any backend server.

The payload of JWT consists of claims, which are statements about an identity (such as name, role, email). There are custom claims as well as standard reserved claims, such as iss (issuer), sub (subject), aud (audience), iat (issued at time), exp (expiration time), and jti (JWT ID). When a program produces a JWT, it turns the raw payload into standardize payload by adding the required reserved claims and may sort the claims alphabetically.

The JWT consists of three parts with a period as delimiter:

The third part is a signature in the format of JWS (JSON Web Signature, RFC 7515) for the JWT consumer to validate its authenticity. The first and second parts, as you can tell, are the claims in the document. Their base64 encoding can be decoded with no effort and should therefore be considered exposed. Although JWT addresses the authenticity of information, it does not intend to address the confidentiality of the payload at HTTP layer. The payload should not carry sensitive information and should always be used with secure HTTPS port. To tackle this issue, there is JWE (JSON Web Encryption, RFC 7516) which is an implementation similar to JWT which also encrypts the payload.

Some IAM protocols are built on top of JWT. For example, the OpenID Connect specification also defines a set of standard claims that it uses while still allow custom claims.

Request Authentication

Istio can perform request authentication using its CRD. It is important to distinguish request authentication and user authentication. In user authentication, the identify provider typically looks up an identity store and compares password hash results to check whether the identity of the visiting user is authentic or not. This is outside of Istio’s capability but many off-the-shelf solution excels at it, such as Azure AD. Once the user’s identity is validated by identity provider, and a JWT is issued for downstream service providers to consume. Istio’s CRD can front the service provider and validate that the presented JWT is authentic. It authenticates the identity of a request (as truly issued by the trusted issuer without being tampered). This process does not involve checking user’s identity, even though user’s identity could be stored in the payload by the JWT issuer.

Istio uses the RequestAuthentication CRD to perform this function. The JWT issuer signs with its private key and stores the signature in the JWT. When it is presented to Istio, Istio’s RequestAuthentication CRD needs the public key of the issuer in order to validate the JWT. The public key usually comes in as a JWK (JSON Web Key, RFC7517), a format convertible to and from PEM format. The JWK can be provided either inline in the RequestAuthentication’s YAML manifest, or via a URI.

Below is an example of a basic RequestAuthentication declaration:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
 name: httpbin
 namespace: foo
spec:
 selector:
 matchLabels:
 app: httpbin
 jwtRules:
 - issuer: "issuer-foo"
 jwksUri: https://example.com/.well-known/jwks.json

In this example (from the documentation), the jwtRule requires that the issuer be issuer-foo, and the JWK (containing public key) is provided by a given URI address. Istio will pass the authentication once the signature in the presented JWT is verified with the JWK.

Authorization Policy

Istio’s Authorization Policy by itself can operate at both TCP or HTTP layers and is enforced at the envoy proxy. The result is an ALLOW or DENY decision, based on a set of conditions at both levels. If the traffic is HTTP then you should consider use some HTTP level information as it provides a lot more flexibility. Even when operating at HTTP layer, AuthorizationPolicy does not have to work in conjunction with RequestAuthentication. The rules can use path, methods, etc to drive an authorization decision, for example:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: authz-policy-orthanc
spec:
 selector:
 matchLabels:
 app: orth
 action: ALLOW
 rules:
 - from:
 - source:
 namespaces: ["istio-system","orthweb"]
 to:
 - operation:
 methods: ["GET","POST","PUT","HEAD","DELETE"]
 ports: ["8042"]
 - from:
 - source:
 namespaces: ["istio-system","orthweb"]
 to:
 - operation:
 ports: ["4242"]

The claims in the JWT payload can also be used to drive authorization decision, as exemplified in the Istio documentation, by using a when keyword in a rule and specifying the claim as a key:

 when:
 - key: request.auth.claims[iss]
 values: ["https://accounts.google.com"]

The when clause requires that the iss claim in the JWT must carry a specific value in order to ALLOW the HTTP request. While the claims in JWT is just an additional factor to drive authorization decision, using authenticated information to drive authorization decision makes the overall workflow more secure, and should therefore be used when applicable.

When using AuthorizationPolicy CRD, keep in mind:

Use correct selectors so it only applies to workloads labelled as such. Without selector the policy takes effect in the entire namespace which is a recipe for issues
When multiple policies (each with multiple rules) are applied to the same workload, be aware of the policy precedence. Troubleshooting may get tricky.
For an AuthorizationPolicy to use HTTP fields in rules, it first needs to identify the traffic as HTTP. To tell Authorization Policy to treat the traffic as HTTP, we need to understand Protocol Selection. In a nutshell, we can name the port as http or http-* in the Service manifest of the workload. Otherwise all HTTP-based rules will be missed.

For troubleshooting, we can check authorization policies effective on a Pod with:

$ istioctl x authz check orthanc-6c9679d8c7-2ttlf -n dev-orthweb

This returns the effective policies but does not necessarily indicate which rule is matched when a request is denied or allowed. To find out further information, you will need to follow Istio FAQ to set RBAC logging to debug, and then monitor the log in the istio-proxy sidecar.

Apart from HTTP fields, path, authenticated claims in JWT, Istio Authorization can also integrate with an Open Policy Agent (OPA) to drive actions, in advanced use cases.

Bottom line

While Istio itself does not perform user authentication, its support of JWT in RequestAuthentication allows a workload to integrate with external identity provider. This capability, along with creative use of claims in JWT, also empowers authorization capability.

From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms

Wed, 29 Dec 2021 22:50:16 -0400

Update: also read my other article here on the different generations of ingress technologies.

In my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.

The word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article.

Background

In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node’s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node’s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.

Kubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.

Requirement

Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:

The ability to proxy TCP traffic over an arbitrary TCP port
The ability to terminate TLS/SSL encryption for TCP traffic

The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I’m not confident.

I also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:

Product	Requirement #1	Requirement #2
F5 driven Nginx Ingress Controller	Supported	There have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.
HA Proxy Ingress controller	Supported	There’s no mention of this in documentation.
Kong’s Kubernetes Controller (with TCPIngress CRD)	Supported	Documentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.
Traefik Lab’s IngressRoute (with IngressRouteTCP CRD)	Supported	Documentation seems to suggest that it is supported by IngressRoute.

My general impression is that Req #2 isn’t very popular so the providers either don’t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP’s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.

Introduction to Gateway

In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:

This sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to “Route” resource type.

This conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The diagram above stolen from Gateway API website illustrate the management model for each resource.

Kubernetes Gateway Implementations

The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.

Because of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification.

For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API’s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong’s Kubernetes Ingress Controller follows the same path.

The Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.

Until the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.

Istio Gateway implementation

As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.

First, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I’ve compared three major service mesh technologies as below:

Linkderd	Istio	Consul
– lightweight – uses linkerd2-proxy – the original service mesh – no Gateway implementation (deal breaker in my use case)	– feature rich – uses Envoy proxy – complex but getting better	– initially a service discovery and distributed key-value store – lacks observability features but getting better – uses Envoy proxy

It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in.

Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.

Istio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).

In addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:

Here is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called “Ingress”, nor a resource type named “Ingress Gateway”, although they may be loosely used to refer to “Gateway resources configured to manage ingress traffic”.

Istio Gateway Installation

There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm.

With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:

install CRDs using the base chart
install istiod using the istiod chart. Here is an example of values provided to Helm installer.
install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition.

This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.

AKS Lessons Learned 2 of 2

Sat, 18 Dec 2021 01:18:00 -0400

Even though Azure Kubernetes Service (AKS) is a managed service, building a cluster is not trivial. For help resources, I would start with the webinar “Configure Your AKS cluster with Confidence” from April 2021, which focuses on a set of working best practices (convention over configuration) but obviously not every recommendation suits every use case. For a deeper technical tour, the John Savill’s Technical Training channel has good videos (from 2020) on AKS overview, high availability and networking. Lastly, there is also an AKS checklist to remind you of the implementation details to consider.

All the references aside, I need to write down some gotchas from my implementation experience in the last two month.

Identity and Access Management

AKS is a special type of Azure resource in the sense that it manages other Azure services on user’s behalf. Therefore the access management needs to consider several aspects:

Access Type	Mechanisms involved	Example
User access Kubernetes API	Azure AD, Azure RBAC and Kubernetes RBAC. – Azure AD is for authentication – Azure RBAC for Kubernetes – Kubernetes RBAC	A user connects to Kube-API server using kubectl
AKS access other Azure resource	There are several identities that represents different components of AKS. For example, the AKS cluster, the node agent pool, and each add-on. The AKS cluster can be represented as a service principal, or managed identity (system assigned or user assigned). The node agent pool can be represented as a managed identity	AKS cluster connects to a VNet in a different resource group. (requiring cluster’s identity to have network contributor role on the target network resource group) AKS node agent pulls images from ACR (requiring the node agent pool’s identity to have ArcPull role on the target ACR)
Pod access other Azure resource	AAD-Pod Managed Identity	Business workload connects to managed database service such as PostgreSQL on Azure.
Pod access Kubernetes API	Access Kubernetes API using Service Account. This issue is solved completely by Kubernetes native mechanisms. Roles and ClusterRoles define permissions. RoleBindings and ClusterRoleBindings associates Service Accounts with permissions	Workload access ConfigMap, Secret etc.

In the first access type, for RBAC with user to access Kubernetes API, there is an overlap between Azure RBAC and Kubnernetes RBAC. Azure RBAC has four built-in roles and three of them (reader, writer, admin) are namespaced. When you use Azure CLI to manage to assign one of those roles, the rolebinding and cluster rolebinding record stored in etcd will be updated accordingly.

RBAC mechansim	Use case
Azure RBAC for Kubernetes	Manage RBAC programmatically using Azure CLI, or infrastructure as code
Kubernetes RBAC	Manage RBAC declaratively with more granularity for all types of Kubernetes resources including CRD

For ease of operation it is advised to use Kubernetes RBAC whenever possible. Azure RBAC is still used for RBAC at the level of Azure resource but not at the level of Kubernetes resource.

In the second access type, AKS cluster may use managed identity or service principal. Azure’s recommendation is managed identity over service principal. Managed Identity is a wrapper around Service Principal with less overhead. Managed Identity can be system assigned (created at the time of cluster creation), or user assigned (can be created ahead of time by Azure administrator and imported to the cluster’s context).

The second access type can be further broken down because there are several components in AKS that uses their own identities. I list the related managed identities as below:

Name	Purpose	BYO identity with Terraform
cluster identity	This identity represents the cluster	Specify in identity block in kubernetes_cluster resource
agent pool identity	This identity represents kubelet running in the agent pool	Specify in kubelet_identity block.
addon: azurepolicy	This identity represents azure policy addon to access the policy	N/A
addon: omsagent	This identity represents OMS agent to access monitoring etc	Specify in oms_agent_identity block
addon: secret	This identity represents to the secret addon, to access AKV	Specify in secret_identity block
addon: ingress gateway	This identity represents the ingress application gateway	ingress_application_gateway_identity block

By default, the system creates a new managed identity for each of the required identity above. For simplicity with identity management, we may create a managed identity and use it for all the occasions where an identity is needed and user assigned (BYO) identity is supported.

In the “az aks show” command return (a JSON document), the identity section (root level) reports the cluster identity, the identityProfile section (root level) reports the agent pool (kubelet) identity. Other identities such as omsagent, are reported in their own child document.

Node Networking

In Azure, a subnet can span across multiple availability zones. Therefore an AKS cluster can put its nodes on a single subnet with nodes evenly distributed across three AZs for high availability. The AZ of each node is indicated in the node label, and can be displayed with kubectl command.

Within a single AZ, a good practice to minimize latency between nodes is to place the nodes in a proximity placement group (PPG). However, only a single PPG can be associated with a node group. You can’t have three PPGs, one in each AZ, for a single subnet.

Pod Networking

The default Pod networking model is kubenet, which involves overlay network. Pod-to-Pod traffic across nodes requires Network Address Translation (NAT). To overcome this performance tax, Azure introduces Azure CNI which gives each Pod an routable IP address from the VNet’s CIDR. This requires advanced IP planning to prevent IP exhaustion. A risk introduced in Azure CNI is that all Pods are exposed on the V-net, which needs to be protected by Network Security Group and/or outbound firewall.

DNS

On the DNS side, when AKS cluster integrate with an external node network, it may create weird issues that are hard to troubleshoot. Another example is with DNS. If the V-Net uses an external DNS server (which is common for enterprises with hybrid network to use an on-premise DNS server), then the cluster creation failed with time-out with misleading error messages (for example, this comment). This is because the DNS name of the newly created cluster is not resolvable within the V-NET, which points to the on-prem DNS server. The fix to that is:

Use a BYO DNS zone (in Azure) for AKS cluster creation;
The AKS cluster will publish the A-record to the zone. To allow this to happen, the AKS cluster’s managed identity needs to have DNS contributor permission for the zone;
Configure the on-prem DNS for conditional forwarding to the DNS zone

This fix will allow AKS to resolve its name and therefore confirm its own creation. Here is a good blog about the DNS zone options.

Another potential issue introduced with the use of on-prem DNS server, is the resolution of single-label hostname of the nodes. This is not just an issue in the context of AKS. It is a generic issue with VMs running on a V-Net pointing to on-prem DNS, as explained in detail here.

In this situation, we should use the fully qualified hostname instead of single-label hostname. The fully qualified hostname with DNS suffix can help the on-prem server to configure conditional forwarding. For example, when the DNS suffix is *.internal.cloudapp.net, then forward it to Azure’s virtual internal DNS server 168.63.129.16 which can resolve the hostname.

If only the Pods need to resolve those FQDNs, then we can configure Core-DNS with conditional forwarding, which will take effect only at the cluster level without the need for changing the on-prem DNS. The Core-DNS configuration looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
 name: coredns-custom # this is the name of the configmap you can overwrite with your changes
 namespace: kube-system
data:
 cloudapp.override: | # you may select any name here, but it must end with the .override file extension
 log
 rewrite continue {
 name regex ^(.*[0-9]{7}-vmss[0-9]{6})$ {1}.internal.cloudapp.net
 answer name ^(.*)\.internal\.cloudapp\.net$ {1}
 }
 forward internal.cloudapp.net 168.63.129.16
 cloudapp.server: |
 internal.cloudapp.net:53 {
 errors
 log
 cache 10
 forward . 168.63.129.16
 }

Alternatively, use Pod DNS policy so that the Pod can use an external DNS server.

Initial Service Account

When a cluster is created, an Azure AD user or group can be assigned as cluster administrator. For a CI/CD pipeline to interact with the newly created cluster, a service account in Kubernetes is needed. Suppose we use Terraform to create the AKS cluster, we can create such service account automatically with the Kubernetes provider. This requires that the Terraform execution environment to have network access to the cluster. If the AKS cluster is located in a private network, then the agent where Terraform CLI runs should also be on the network. Alternatively, use Terraform Enterprise hosted in an environment with access to the cluster’s network.

Integration with Azure KeyVault

Azure Key Vault can store several types of secrets, key value pair, X509 keys and certificate. When AKV is integrated with an AKS cluster, the Kubernetes workload can access the secrets as mounted volumes, using CRD named SecretProviderClass. Further, they can be presented as Kubernetes Secret, using a Pod to sync between mounted content and Secret. AKV has three types of entries: key, certificate and secret (key-value). The certificate entry requires both key and certificate are stored, with optional certificate chain. In my opinion this is an over design. Unless we need Azure to manage the certificate (e.g. rotation) I would simply use secret to store my own X509 key and certificate.

AKS Lessons Learned 1 of 2

Sat, 04 Dec 2021 02:11:06 -0400

In general, troubleshooting Kubernetes is tricky. That is because one has to get in and out of pods. I took two days to troubleshoot some networking issues with private AKS cluster. For the amount of tricks I had to employ, I need to take some notes.

The issue

After writing the Terraform code, I used the following dummy service to test the private AKS cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: aks-helloworld-one
spec:
 replicas: 1
 selector:
 matchLabels:
 app: aks-helloworld-one
 template:
 metadata:
 labels:
 app: aks-helloworld-one
 spec:
 containers:
 - name: aks-helloworld-one
 image: neilpeterson/aks-helloworld:v1
 ports:
 - containerPort: 80
 env:
 - name: TITLE
 value: "Welcome to Azure Kubernetes Service (AKS)"
---
apiVersion: v1
kind: Service
metadata:
 name: aks-helloworld-one
spec:
 type: LoadBalancer
 ports:
 - port: 80
 selector:
 app: aks-helloworld-one

The expected behaviour, is that the service object will tell cloud API to provision a load balancer, with public IP listing at port 80. I should be able to curl to the IP address and connect to the site in the Pod. However, I was not able to. On the bastion host, I was able to curl to nodePort of the node address. But anything on public IP does not work, no matter where I ran curl from. This feels like a basic issue, but is quite annoying because the native troubleshooting tool for Azure Load Balancer is horrible. In and out of a bunch of components named “insights”, “diagnostic log”, or “Metrics”, I can’t simply find a way to trace whether it received an HTTP request. Most of the information I was able to see was irrelevant or useless.

The approach

The hard way to troubleshooting infrastructure as code, is configuration comparison approach: revert to a baseline configuration, and see if the expected function works. Then from the baseline, change one configuration at a time and see where it starts to break. This approach is very time consuming, and AKS cluster as a relatively large resource, with numerous attributes, takes this effort to extreme.

The baseline configuration I started with is:

az aks create -g AutomationTest -n orthCluster --generate-ssh-keys --node-count 1 --tags Owner=MyOwner

With this baseline, I simply use kubectl to apply the YAML file above. Then I can tell that the port is working. With a good start point, I started to apply one change at a time and repeat the test. I ran into a snug when I’m using the following configuration:

az aks create -g AutomationTest -n orthCluster --generate-ssh-keys --node-count 1 --tags Owner=MyOwner --enable-private-cluster --network-plugin azure --network-policy calico

With the cluster created from the command above, the variable introduced is –enable-private-cluster. This puts the cluster on a private network. I cannot connect to the cluster via a public endpoint anymore, and thus have to figure out some tricks to run the kubectl commands. I had to play with the Command Run feature of AKS cluster because I don’t have a bastion host when using AZ CLI command. The Command Run feature would not allow me to use any file from bastion host. So i had to create my test objects, the Deploy and the Service objects all by imperative commands. The equivalent commands I worked out is:

kubectl create deployment aks-helloworld-one --image=neilpeterson/aks-helloworld:v1 --replicas=1 --port=80
kubectl expose deploy aks-helloworld-one --port 80 --target-port 80 --type='LoadBalancer'

Then I realized a limitation with Command Run feature: it only supports basic command switches and doesn’t like switches such as –replicas. So I used the following commands:

az aks command invoke -g AutomationTest -n orthCluster -c "kubectl get no"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl create deployment aks-helloworld-one --image=neilpeterson/aks-helloworld:v1"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl get deploy"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl expose deploy aks-helloworld-one --port 80 --target-port 80 --type=LoadBalancer"
az aks command invoke -g AutomationTest -n orthCluster -c "kubectl get svc"

This trick allows me to continue with the testing eliminate Azure CNI and Calico policy as the cause. Testing after each cluster creation is painful because the cluster creation can take 10 minutes.I had to temporarily minimize the size of the cluster to speed up provisioning.

I finally came to the point that I can reproduce the issue using TF template. I realized that when I set the vnet_subnet_id attribute of azurerm_kubernetes_cluster’s default_node_pool, the problem came back. That’s the smoking gun that the node subnet is the issue.

The Network Security Group on Node Subnet

The node subnet has an associated network security group. I discovered that once I add an allow rule for port 80 to the security group, the curl test will work. I also noticed the security group rule change will take 60 sec to come to effect and load balancer will also take 60 sec to warm up.

This confuses me because port 80 is only listened by the load balancer and not by any of the nodes. It’s most likely when public load balancer is used the load balancer is placed on the node subnet. According to this note:

Inbound, external traffic flows from the load balancer to the virtual network for your AKS cluster. The virtual network has a Network Security Group (NSG) which allows all inbound traffic from the load balancer. This NSG uses a service tag of type LoadBalancer to allow traffic from the load balancer.

The packet coming from external source can travel up to the VNet, but it was blocked at the NSG of node subnet.

Lessons Learned

We always need to have some dummy service ready to test what we need. We can use nginx dummy service like:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-nginx
spec:
 selector:
 matchLabels:
 run: my-nginx
 replicas: 2
 template:
 metadata:
 labels:
 run: my-nginx
 spec:
 containers:
 - name: my-nginx
 image: nginx
 ports:
 - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: my-nginx
 labels:
 run: my-nginx
spec:
 type: LoadBalancer
 ports:
 - port: 80
 protocol: TCP
 selector:
 run: my-nginx

As discussed above, it’s also important to have a Bastion host able to access the control plane when the AKS cluster is private. Azure touts about CloudShell (and its ability to run in specified V-Net) but it’s pretty useless in troubleshooting. CloudShell sessions run inside of Kubernetes cluster and lacks common network troubleshooting tool such as nc. Azure has a managed service for Bastion but it requires a subnet with the exact name of AzureBastionSubnet.

We will explore more issues in the next post.

From Microservice to Service Mesh

Thu, 25 Nov 2021 00:31:19 -0400

We all know what microservice is now but how does service mesh assist with microservice development.

Microservice

Microservice as an architecture was firstly conceptualized in this article by Martin Fowler in 2014. It covers the pros (strong module boundaries, independent deployment, technology diversity) and cons (dealing with distributed system, eventual consistency, operational complexity). The reality is, many teams develops their product with the microservice architectural pattern.

The implementation of microservice architecture involves a lot of programming patterns, and tools. The creation of these patterns and tools are usually done in a separate dedicated project so developers can focus on business logics. When building software, developers only need to interact with libraries and frameworks. Libraries (e.g. log4j) provides dependencies, and developers needs to write code to call the libraries. On the other hand, frameworks (e.g. Spring, Flask) not only provides tools, but also implements a pattern. It addresses a set of common problems such as authentication, expose http service, logger and database connectivity. Once set up, the framework will call the code that developers write (unlike in libraries).

Spring – an example of Microservice framework

When it comes to microservice, a well-known appliction framwork is the Spring framework. It solves problems such as:

Application context and dependency injection (for Inversion of Control, or IOC)
Database access and transaction management
Expose rest APIs (using spring MVC)

There’s an entire ecosystem of projects under Spring framework. This framework is a huge system requiring a lot of configuration efforts. This is where Spring Boot helps.

Spring Boot makes it easy to create stand-alone, production-grade Spring based applications that you can just run. It features the “convention over configuration” paradigm to save programmers from boiler plate configuration. SpringBoot gives you a standalone application ready to run without complicated deployment steps.

Managing configuration in property files does not scale in the time of microservice. Spring Cloud provides configuration as a service (in line with everything else microservice framework). It doesn’t necessarily have to be hosted in the cloud. It is comparable to Apachee Zookeeper, Etcd (distributed key value store), Hashicorp Consul and Netflix OSS (Eureka, Ribbon, Hystrix). You can pull from Git repo. The mission of Spring Cloud is to eliminate boilerplate associated with distributed systems problems for Spring Boot applications.

Many developers use Spring Boot along with Spring Cloud to build microservices. This page contains a diagram for such architecture.

In this architecture, Spring cloud helps with service discovery, traffic routing, circuit-breaking, distributed tracing and monitoring. It can also act as API gateway (in place of Nginx).

API Gateway and API Management

Microservices relies on API. Let’s distinguish API gateway, and API management (this long post has some good information).

API Gateway is a microserivce pattern. The idea is a single point of entry for all clients. The API gateway either proxy an incoming request to the appropriate service, or it may fan out a request to multiple services. The other important aspect is the API gatway can expose a different API for reach client.

API Gateway Pattern

A variation of this pattern is the Backends for frontends pattern, where it defines a separate API gateway for each kind of client. The API Gateway may authenticate user and pass an Access Token containing information about the user to the services. It may use a circuit breaker to invoke services. To summarize, the key functions of an API Gateway in this pattern is:

Unified entry point for multiple API implementations
Protocol transformation
Request morphing
Client specific logics

This page from Azure has a good comparison between API Gateway pattern vs direct connection between client and microservice. Note that API Gateway can also refers to API Gateway product, which implements the functions above. For example

Spring Cloud Gateway
Solo.io Gloo
Netflix Zuul

API management acts as a proxy for an existing API implementations. Typical functions include:

AuthN and AuthZ
Service Discovery Ingreation
Load balancing (e.g. L7 path based routing)
Logging, tracing (track user), correlation
Response Caching
Retry policies, circuit breaker, QoS
Enforce policy
Track usage and monetization
metrics (duration)
rate limiting and throttling
Request morphing (header, query string and claims transformation)
IP whitelisting

API management are usually implemented as tightly controlled shared infrastructure owned by either a “platform team”, “integration team”, or other API infrastructure teams. Examples of API management product (including SaaS) are:

Google Cloud Apigee
Mulesoft
Kong

In real world, people often use API management produce and API gateway product interchangeably.

Service Mesh

Many dub Service Mesh the next generation of Microservice. So what is the relationship between microservice and service mesh. The Microservice architectural pattern creates the need for API gateway pattern. To address this pattern, the API Gateway products first emerged. Service mesh emerged later. Service mesh and API gateway have a common set of features.

In this presentation (a tale of two frameworks) from 2018 (early days of service mesh), two teams discussed microservice (spring cloud) and service mesh (istio) approaches. There is a slide about when to use which. Many teams since have moved to Service Mesh for feature richness. This is a case study from 2021.

There are a number of service mesh technologies, such as Consul, Isito and Linkerd. Here is a comparison chart. Even though all those projects are open-source, there is some competition already. Linkerd is the first to bring up the concept of service mesh in this blog. It also purportedly has better performance than Istio. However, it does not use Envoy proxy. Istio is good at marketing. It has higher adoption rate and is feature rich. However, Google did not donate Istio project to CNCF as many expected. Instead, it created its own governing body, the Open Usage Commons. The Istio is not an open-governance project, which potentially diverge from CNCF in the future [1]. Hashicorp Consul initially was built for service discovery and distributed key/value store. It supports Kubernetes and VM. However, it still lacks observability features.

Apart from the three major technologies, other players tries to push for standardization of service mesh. The most influential initiative is the SMI (service mesh interface), pushed by Microsoft. The idea is a separation of standard and implementation, so late players will have a chance. OpenServiceMesh is Microsoft’s reference implementation of SMI. The SMI is something to watch for but it remains pretty week thus far. Google’s platform has Anthos Service Mesh which is a commercial distribution based on Istio. AWS has its own AppMesh technology, also using Envoy proxy.

[1] Update from Apr 25, 2022 – Istio applied to become CNCF project.

Infrastructure deployment in Terraform 2/2

Fri, 05 Nov 2021 01:05:00 -0400

In a previous post, I introduced Terraform Cloud and covered how to use AWS profiles with Terraform. This time I explored some alternatives to Terraform Cloud, in the context of Azure. I use Scalr as an example of multi-cloud management platform. I will also discuss some issues I’ve came across while managing permissions and variables for Terraform.

Scalr

Scalr is a multi-cloud management platform. I first used it in January but since then it seemed to focus on being a collaboration platform for Terraform. It organizes deployment by environments and workspaces. Accounts in the free tiers is allowed to have one Environment. You will also need to configure (cloud) providers and VCS providers. Once configured, it is important to link a cloud provider with an Environment. Each workspace inside of an Environment can be associated with a VCS provider. In the case of Terraform, this limits a workspace with a single cloud provider.

Permission with Azure

I have a resource group (e.g. named AutomationTest) under a subscription. My account has Contributor role of this resource group. To run Terraform, I could login to Azure as my own account on my environment using AWS CLI. Terraform will pick up the session from Azure CLI and execute as my user. However, it is recommended to run Terraform as a separate own entity. This would allow me to run Terraform template from Scalr, or Terraform Cloud. It is also a good practice for Terraform to use a separate account than a regular user account. There are a number of ways to do this as suggested on the guides for Terraform azurerm provider, including:

Authenticating via a Service Principal and a Client Secret
Authenticating via a Service Principal and a Client Certificate
Authenticating via Managed Identity
Authenticating via the Azure CLI, only recommended when running Terraform locally.

I chose the first option and followed the instruction, using the following CLI command to create the service principal:

az ad sp create-for-rbac -n tf-sp --role="Contributor" --scopes="/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/resourceGroups/AutomationTest"

The command returns a few attributes (client ID, tenant ID, secret) that I needed to configure cloud providers in Scalr. The service principal will also be visible under App Registrations in Azure. Once configured I needed to link the provider to an Environment, for Scalr to make an connection to Azure. Otherwise, the Scalr run will return the following Error:

Terraform error

Once provider linking is completed, Scalr automatically populate required environment variables in the workspace. They show up as “Shell” variables under VARIABLES tab.

Under the Terraform tab are input variables that you wish to put in for Terraform template to pick up. Then you can run the template. This works well until I came across a permission issue when I added azurerm_role_assignment resource in Terraform template. What I was trying to do is something like this:

resource "azurerm_role_assignment" "admin_assignment" {
 scope = var.rbac_aks_id
 role_definition_name = "Azure Kubernetes Service RBAC Admin"
 principal_id = var.rbac_principal_object_id
}

And whenever at this line, the following error returned:

Apparently the code 403 indicates Azure doesn’t think the Terraform Service Principal has the privilege to perform Microsoft.Authorization/roleAssignments action. The reason dates back to the way I created service principle above, where I specified contributor role for resource group. However, contributor as a built-in role does not include the permission to assign roles in Azure RBAC. To address this issue, I needed a custom role, named TerraformContributor, with the following definition:

 {
 "assignableScopes": [
 "/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/resourceGroups/AutomationTest"
 ],
 "description": "Grants full access to manage all resources, but does not allow you to manage assignments in Azure Blueprints, or share image galleries.",
 "id": "/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/providers/Microsoft.Authorization/roleDefinitions/637824aa-52ae-42f6-a24e-26b2a443afdf",
 "name": "637824aa-52ae-42f6-a24e-26b2a443afdf",
 "permissions": [
 {
 "actions": [
 "*"
 ],
 "dataActions": [],
 "notActions": [
 "Microsoft.Blueprint/blueprintAssignments/delete",
 "Microsoft.Compute/galleries/share/action",
 "Microsoft.Blueprint/blueprintAssignments/write"
 ],
 "notDataActions": []
 }
 ],
 "roleName": "TerraformContributor",
 "roleType": "CustomRole",
 "type": "Microsoft.Authorization/roleDefinitions"
 }

Compare this with the JSON statement of built-in contributor role, the exclusion of actions for Microsoft.Authorization are removed. The creation can be completed with CLI command “az role definition create” in the subscription, or use Azure portal. Once the role is created, create a new service principal using the

az ad sp create-for-rbac -n tf-sp --role="TerraformContributor" --scopes="/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/resourceGroups/AutomationTest"

This solution is suggested on this thread.

Composite type for Input variable

Sometimes we want to define an input variable that describes a hierarchy of attributes on a resource. A good example would be Azure Kubernetes service. We can use either object or map as the variable type. The example below has a variable of each type.

variable "cluster_detail" {
 description = "AKS cluster"
 type = object({
 resource_group = string,
 cluster_name = string,
 kubernetes_version = string,
 node_subnet = object({
 subnet_name = string
 vnet_name = string
 resource_group = string
 }),
 pod_subnet = object({
 subnet_name = string
 vnet_name = string
 resource_group = string
 }),
 lb_subnet = object({
 subnet_name = string
 vnet_name = string
 resource_group = string
 }),
 ad_admin_group_object_ids = list(string)
 })
}

variable "common_tags" {
 description = "common tags"
 type = map(any)
 default = {
 tagA = "valueA"
 tagB = "valueB"
 }
}

When using object as the type, the default value needs to define all fields.

In this example where node pool configuration is exposed, we can see how using variables with hierarchy helps template user customize infrastructure specification.

Parsing Map as Input variable

To find out the best multi-cloud management platform, I tested a few of them. I find it inconsistent when I have an input variable of the map type. With Terraform Cloud, I have to specify the variable to be parsed as HCL, and the value has to be:

{"Environment" = "Dev", "Owner" = "info@digihunchtest.com"}

With Scalr, I also have to specify the variable to be parsed as HCL, and the value can be either the one above, or the one below:

{"Environment":"Dev","Owner":"info@digihunchtest.com"}

So Scalr is more flexible in parsing maps. Apart from Scalr and Terraform, I also tested env0 but I gave up after an hour. I could not specify to parse a variable as HCL. They need to work harder on this. Neither was I able to figure out the right syntax as plain variable. I did not test Atlantis or SpaceLift.

Enterprise Deployment

When deploying code to enterprises with their own network environment, Scalr supports running a self-hosted agent inside of the Enterprise network. This is also supported by Terraform Cloud (manage in cloud, execution in enterprise network). This is very useful when the execution machine needs to access the resource created in the enterprise environment.

A good example, is using Terraform’s Azure provider to provision an AKS cluster on the corporate network. Then use Terraform’s kubernetes provider to connect to the newly created cluster and create some Kubernetes object such as service account, as illustrated in this blog post.

Azure Deets

Mon, 25 Oct 2021 23:13:52 -0400

Both Azure and AWS are leading players in public cloud. AWS developed a lot of SMB customer in technology. Azure attracted many enterprises from their on-prem customers. In addition to clienteles, their models to manage resources in the cloud are also different in several aspects. While this post is by no means a comprehensive comparison, it serves as a refresher on how Azure is different from AWS in cloud engineering.

Subscription and Resource Group

Both subscription and resource group are means to manage resources in group. Subscription is associated with a credit card and groups resource financially. Resource group groups resources logically. A subscription can have multiple resource groups, as illustrated here in the hierarchy map.

Azure CLI

There are two CLI tools: Azure PowerShell for PowerShell users, and Azure CLI for Linux users. The distinction is not clear, because PowerShell can also run on multiple platforms such as MacOS and Linux. On the other hand, Linux Bash can run on Windows (e.g. using WSL2). The different lies more in the command nomenclature. For example, to list subscriptions, the PowerShell Cmdlets reads:

Get-AzSubscription

The Azure CLI comes much cleaner:

az account list

The command and argument naming in Azure CLI align with Linux commands. I prefer Azure CLI for its succinctness. When looking up VM skus, we can use this command:

az vm list-skus -l eastus2

The return is a JSON document. In advanced use cases, we can filter the result by using jq utility. Alternatively, we can filter the result and groom the output with jmespath query, just like AWS CLI. For example, in eastus2 region, we look for instances that:

Has virtualMachines as resource Type
Has AcceleratedNetworking enabled
Has EncryptionAtHostSupported enabled
Has PremiumIO enabled

For each of the result, we print out:

number of vCPUs
Memory size

For this use case we will have to provide a JMESPath query as below, then output the result as a table:

az vm list-skus -l eastus2 --query "[?resourceType=='virtualMachines' && capabilities[?name=='AcceleratedNetworkingEnabled' && value=='True'] && capabilities[?name=='EncryptionAtHostSupported' && value=='True'] && capabilities[?name=='PremiumIO' && value=='True']].{Name:name,vCPUs:capabilities[?name=='vCPUs'].value|[0],MemoryGB:capabilities[?name=='MemoryGB'].value|[0]}" --output table

The result looks like this:

I use variations of the command above very often to find out the best instance for AKS nodes.

Infrastructure as Code

The native infrastructure as code option is ARM (Azure Resource Manager) template in JSON format. It is extremely wordy and perhaps why Azure later developed Bicep as the second generation of IaC tool. Terraform has a provider for Azure as well. For comparison among ARM, Terraform and Bicep, I have written a blog post for Slalom build covering more details.

Networking

Here is a great post comparing Azure Network with AWS.

At a high level, Azure Virtual Network (or VNet) is the equivalent of VPC in Amazon. Likewise, peering can be configured between VNets. As to subnet, Azure is different because there is no conceptual distinction between public subnet and private subnet. In AWS, public subnet is subnet attached with an Internet Gateway via a network route. So “private” or “public” are in terms of outbound traffic. On the other side, Azure does not distinguish between private or public subnet. Resources connected to a VNet have access out to the Internet by default. As to inbound traffic, you can make a VM available on Internet by giving it a public IP (same as AWS). You can make it available to other VNet, by configuring a service endpoint. Customers typically need custom routes to redirect outbound traffic (e.g. through firewall). In VPC, subnets are mapped to availability zones one-to-one, whereas in Azure, a subnet may traverse multiple availability zones.

With Azure, it is also important to understand difference between Azure service endpoint and Azure private endpoint:

Azure service endpoint: provides connectivity to Azure services over n optimized route over the Azure backbone network. Traffic will leave your VNet.
Azure private endpoint: a NIC that uses private IP from your VNet. This NIC connects you privately and securely to a service powered by Azure Private Link. By enabling a private endpoint, you’re bringing the service into your VNet.

On security group, we can associated a network security group with a network interface, or with a subnet. In contrast, in AWS, a security group can only be associated with an instance’s network interface.

Azure Bastion and Jump Box

There is a managed service called Azure Bastion. It is a SSH/RDP proxy fully managed as PaaS. However, its use case is virtual machines. It cannot be used to access other services. For example, if you create a private AKS cluster, then you need a command terminal to access the API server. This is not what Azure Bastion can do. Instead, you either need a virtual machine in the AKS network as jump box.

The alternative is an Azure Cloud Shell, which will require storage but can be configured to be placed inside of a V-Net. However, Azure Cloud Shell is not running inside of a full-fledged Linux operating system. You cannot install commands.

To create a bastion host, e.g. without public IP address, use the following CLI command:

az vm create -n MyBastion -g AutomationTest --image UbuntuLTS --subnet suitable-porpoise-node-subnet --vnet-name suitable-porpoise-vnet --ssh-key-values ~/.ssh/id_rsa.pub --authentication-type ssh

If the bastion host is needed with a public Ip, configure the network security group accordingly. The AZ CLI command above will create a VM, with an OS user named after the command line terminal user.

IAM

Azure AD is a managed identity service. Here is the difference between Active Directory and Azure AD.

Azure RBAC is a mechanism for authorization. Just like IAM policies, Azure RBAC enforces permissions using role assignment, which consists of:

security principal ( user, group, service principal, or managed idenity)
role definition: defines what actions is allowed and what is not allowed
scope: the object of the action

To enforce RBAC, one needs to create role assignment objects, each specifying principal, role, and scope.

Service Principle and Managed Identity

The two concepts may appear confusing. I find this article a great reference to demystify them. The takeaway is: service principle is the equivalent of service account in old Active Directory. Managed identity is a service principle automatically managed by a resource. Managed identity can be user assigned or system assigned.

Managed Identity is a “wrapper” around a service principal. It is automatically created and automatically rotated.

Azure DevOps

I think of Azure DevOps (ADO) of a managed pipeline implementation, with a repository (just like BitBucket), a board to manage tickets (similar to JIRA), Wiki (just like Confluence), Artifactory. The Pipelines is the part that’s similar to Jenkins. ADO calls a build pipeline a Pipeline, and a release/deployment pipeline a Release. A pipeline and a release are fundamentally the same but they are used in different ways. A pipeline’s input is usually the code repository, and the output is artifact. A release’s input is usually an artifact, and it connects to infrastructure in different environments.

ADO has its own ecosystem for plugins, managed under Visual Studio marketplace. Many extensions are open-source. If you are not happy with an extension, you can publish your own extension to market place.

Logging and Monitoring

Azure Monitor manages metrics, logs and alerts. To further analyze logs, create a log analytics workspaces, where you can run Kusto queries. You can create a workbook and embed Kusto queries into visual objects on the workbook.

Storage

Azure manages storage resources under storage account. The resource classes include Blob (object), File, Queue, Table and Disk (block). They have a few acronyms on redundancy levels.

Redundancy Option	Data Copy	Access level
LRS – Locally redundant	synchronously copy your data three times within the AZ in the primary region.	Write is acknowledged after three synchronous writes.
ZRS – Zone-redundant	synchronously copy your data across three AZs in the primary region. your data is still accessible for both read and write even if one AZ becomes unavailable.	Write is acknowledged after three synchronous writes. If an AZ becomes unavailable, Azure undertakes networking updates (e.g. DNS re-pointing). Application may perceive a blip where re-try policies may help.
GRS – Geo-redundant	LRS in primary region + asynchronously copy your data to a single AZ in the secondary region + LRS in secondary region	your data in the secondary region isn’t available for read or write access unless there is a failover to the secondary region. for read access to the secondary region, configure your storage account to use RA-GRS (read-access geo-redundant storage) If the primary region becomes unavailable, you can choose to fail over to the secondary region. After the failover has completed, the secondary region becomes the primary region, and you can again read and write data.
GZRS – Geo-zone-redundant	ZRS in primary region + asynchronously copy your data to a single AZ in the secondary region + LRS in secondary region	your data in the secondary region isn’t available for read or write access unless there is a failover to the secondary region. for read access to the secondary region, configure your storage account to use RA-GZRS (read-access geo-zone-redundant storage) If the primary region becomes unavailable, you can choose to fail over to the secondary region. After the failover has completed, the secondary region becomes the primary region, and you can again read and write data.

The disaster recovery and failover happens at storage account level.

Local multi-node cluster – Minikube, MicroK8s and KinD

Tue, 14 Sep 2021 11:18:00 -0400

In this post we compare Minikube, MicroK8s and KinD as different approaches to build multi-node cluster locally.

Is Docker desktop bad?

In the previous post about docker desktop as a single-node Kubernetes cluster setup, I touched on the deprecation of docker-shim. Now that CRI beats OCI as the standard for container runtime, the docker runtime will no longer be supported by Kubernetes. Also deprecated is docker-shim, the temporary interface that had make Docker runtime work in Kubernetes. This was announced in December 2020, and is coming through in Kubernetes 1.23, expected Oct 2021. However, docker desktop still uses docker runtime in it’s single-node Kubernetes cluster. This essentially renders itself a non-compliant Kubernetes environment.

Docker desktop still has great value for application developers. If your role is development, spending a lot of time coding business logics and need an easy-to-use container runtime on your laptop, Docker desktop is a good choice. The recent moves by the company seems to suggest that this is the business they are targeting now. On the other hand, if your roles are deployment, automation, orchestration, cloud native etc and you are looking for a playground, most likely you do need a runtime compliant to Kubernetes CRI. Docker desktop is not a CNCF-certified project anymore, and it is not your choice.

Alternatives

There are a number of alternatives, the most well-known ones are Minikube, MicroK8s, KinD and K3s with K3d. This presentation from CNCF in 2020 covers a lot of details about these technologies. I’ll try to add my opinion.

K3s is Rancher Lab’s lightweight Kubernetes distribution that supports multi-node cluster as well as different runtimes (e.g. containerd). It is not straightforward to setup, and k3d is an command-line wrapper to make it easy to install K3s cluster. K3s was accepted as a CNCF project but only at Sandbox maturity level, so it is not my choice.

The other three: Minikue, MicroK8s and KinD are all certified CNCF project. I will further discuss how to choose among them. These projects are technologies that takes different approach to address the challenges with deploying multiple nodes in local environment (e.g. my laptop).

The challenge with running a Kubernetes cluster with multiple nodes locally is how to manage these nodes. They are separate virtual resources that need to be isolated from computing perspective, and connected as a cluster. This is typically the use case of a Type II hypervisor, or alternatively, it can also be implemented with container technology. This layer of technology (referred to as drivers) makes a big difference.

Minikube

Minikube supports multiple drivers. Depending on your platform (Windows, Linux, or MacOS), the preferred driver is different. Refer to the document here for preferred driver, and this blog post for more instructions. In addition to the documents, here some notes from my personal experience:

On MacOS, Minikube lists Docker as preferred driver. I disagree with that. If you have no other reason to install Docker, then I would recommend hyperkit as the preferred driver. Hyperkit can be installed with a simple Homebrew command. For two reasons I do not recommend Docker as the driver of Minikube. First, it requires a separate installation of Docker Desktop, which includes a built-in instance of hyperkit on its own. This isn’t neat. Second, I often need Metal LB add-on with Minikube for testing Kubernetes Ingress. With Minikube on Docker, the Ingress ports are not exposed to MacOS’s. Therefore you cannot directly visit websites spun up on Minikube. This is a known issue for a while due to limitation on docker bridge with Mac. Some reported an ugly workaround with docker-mac-net-connect but I never got it to work.
On Windows native environment, the preferred driver is hyper-V. The Minikube cli command have to run from Windows PowerShell.
On WSL2, Minikube doesn’t play well, regardless of driver. The hyperkit driver won’t work (it is designed for MacOS only). The kvm2 driver would require a KVM2 hypervisor. However, WSL2 itself is a VM on top of hypervisor, as explained here. If KVM2 driver works it would require nested virtualization so I doubt it will ever be supported. As for Docker on WSL2 as driver, Minikube has it as an experimental feature, and requires configuring cgroup to allow setting memory. I am not confident with it.

To me, Minikube is the tool for MacOS (I have Intel processor). On MacOS, we first need to install minikube and hyperkit with home brew.

We can then start a kubernetes cluster, with minikube in a single command. I noticed a process on my MacBook called dnscrypt-proxy that conflicts with hyperkit DNS server when starting minikube. I had to remove dnscrypt-proxy (part of Cisco Umbrella Roaming Client) in order to get minikube to work, as this thread suggests. You can find out by running:

sudo lsof -i :53

If dnscrypt-proxy is running, find out the application by PID and remove the application. Otherwise there will be issues. Check out this section on the document. The commands that I use to start multi-node cluster is:

minikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2 --disk-size=150g --nodes 3
kubectl get po -A
kubectl describe node minikube|grep Runtime

Node administration is simple. To enable dashboard, simply run “minikube dashboard”. To SSH to a node, simply do “minikube ssh -n <node_name>”. In order to stop the node and delete cluster, run “minikube stop && minikube delete”.

There are some addons in minikube, for example, efk, gvisor, istio, metrics-server. To list add-ons, and enable metrics-server, for example, run:

minikube addons list
minikube addons enable metrics-server

When creating cluster, instead of specifying the cluster imperatively, the configuration (e.g. driver, container runtime, cpu, memory, number of nodes, etc) can be stored as a profile with -p switch. Like other Minikube configuration information, Minikube profiles are stored in ~/.minikube under the profile directory.

Minikube also has a page that benchmarks the performance of these technologies, where it presents itself as the most performant.

Minikube, KinD, k3d and microK8s performance

MicroK8s

MicroK8s is developed by Canonical. It can use either Multipass or LXD container as driver. Multipass can configure Ubuntu VMs using cloud-init. It supports multiple hypervisor backends as well but hyperkit is the default on MacOS, Hyper-V on Windows, and KVM on Linux.

MicroK8s supports multi-node configuration across multiple machines. That is, nodes can span across multiple physical machines. This is more powerful than Minikube where multiple nodes are on the same physical machine. It brings MicroK8s additional use cases such as edge and IoT devices.

With that capability comes the extra step to configure a MicroK8s cluster. You will need to manually join a node to a cluster because the new node is potentially located on a different machine, and you execute the command from the new machine. On the other hand, with Minikube you simply specify the number of nodes desired in a command or profile.

Snap is the native package manager to install MicroK8s, making GNU Linux (e.g. Ubuntu) the native platform. It also supports MacOS and Windows. MicroK8s does not rely on Docker (unlike KinD and Minikube with Docker as driver), and uses containerd as runtime.

MicroK8s comes with its own packaged version of kubectl, and you use that with “microk8s kubectl” command, which is not convenient. You can configure your host kubectl to point to the MicroK8s cluster, as an extra step.

Compared to the other two technologies, MicroK8s is more powerful in the sense that the cluster is build on nodes across multiple machines. However, it takes more step to configure even for a multi-node, single-machine environment. Refer to this post for the steps.

KinD

KinD is similar to Minikube with Docker as driver. It is more restricted than Minikube considering Docker is the only driver it supports. This makes it a requirement to have Docker installed locally.

Although KinD uses Docker to run nodes, it does not use Docker as its container runtime. Therefore it remains as compliant environment.

Another advantage of KinD is it supports Docker on WSL2 very well. Simply install KinD on WSL2 and start Docker. This blog post contains the steps required to install KinD vs Minikube on WSL2. There is a comparison table in the conclusion section that highlights the fact that it is much easier to install KinD with WSL2 than to install Minikube.

However, there are currently some known limitations with Docker desktop for Windows (including on WSL2). One is the absence of docker0 bridge. This means on Windows you cannot route traffic to the containers.

For cluster specification, KinD can configure a cluster declaratively using YAML file for example, the kind-config.yaml contains the following snippet:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
 - role: control-plane
 - role: worker
 - role: worker
 - role: worker
networking:
 disableDefaultCNI: true

We can bring up a cluster with a command:

kind create cluster --config=kind-config.yaml

The command will also configure the kubectl context so we can check node with kubectl command. The file is in my real-quicK-cluster repo.

Conclusion

After reviewing the technologies that back up multi-node kubernetes cluster for my role, I find that Minikube with hyperkit is my favourite for MacOS. On WSL2, I prefer to use KinD. Since I do not use Windows native environment or Ubuntu on my laptop, I cannot make recommendations. However I would start with Minikube (with hypverv or kvm2 as driver).

Update July 2022: When the test workload involves persistent storage, KinD is a better choice. When the test workload involves load balancer. Minikube is a better choice.

As to storage provisioner, Minikube with storage-provisioner addon uses k8s.io/minikube-hostpath. KinD uses rancher.io/local-path. When I have to test workload with persistent storage (e.g. PostgreSQL with Crunchy pgo), I realized Minikube have permission issues with persistent volume, as discussed here as an issue with multiple nodes. The issue has been open since Aug 2021.

For Load Balancer, Minikube has metallb as an addon and I can configure it within a bash script conveniently. With KinD, I’d have to configure that in a few steps with both kubectl and Docker CLI commands and I was not able to connect to the load balancer by IP even after following the steps. So I tend to just use Minikube to test workload requiring load balancer and service mesh.

I find myself switch between Minikube and KinD on my MacBook depending on the test workload.

Creating X.509 TLS certificate for workload on Kubernetes

Sun, 29 Aug 2021 23:19:00 -0400

In deployment automation, I often had to create self-signed X.509 certificate for testing TLS traffic into Kubernetes. Sometimes self-signed, sometimes signed by a CA. This post summarized the approaches I’ve taken.

Create self-signed certificate with OpenSSL

Traditionally, this is done in three OpenSSL commands:

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Health Certificate Authority'
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=*.orthweb.com'
openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

I have an older post to cover the basics of cryptography in TLS certificate and PKI. In the three commands above, the first produces a private key and self-signed certificate for a CA. The second creates a private key and a CSR for the web site. The third one uses the CA’s signing private key to sign the CSR from the website. The output is the certificate for the website.

Workloads running in Kubernetes typically consume certificates stored in Kubernetes Secret. The cons of this approach is that it usually requires an extra step to import the certificate files into Kubernetes Secret. For example:

kubectl create -n orthweb secret generic orthweb-cred --from-file=tls.key=server.key --from-file=tls.crt=server.crt --from-file=ca.crt=ca.crt

Note, people use the term self-signed certificate loosely. It sometimes means literally a certificate that is self-signed, like the one generated above. Sometimes, I had to self-sign a CA, then use the CA to sign one certificate for the server and one for the client. Because the application being tested requires that the client and server’s certificates both under the same CA. This would involve a few more commands, for example:

# Self-sign a CA
openssl req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=issuer.digihunch.com/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

# Generate a CSR for server
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=server.digihunch.com/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com -keyout /tmp/server.key -out /tmp/server.csr

# Use the self-signed CA to issue a certificate to the server
openssl x509 -req -sha256 -days 3650 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt -extfile <(echo subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com)

# Generate a CSR for clietn
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=client.digihunch.com/emailAddress=client@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr

# Use the self-signed CA to issue a certificate to the client
openssl x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

In the example above, it is important to note that even though the server’s CSR contains subject alternative name (SAN), I still have to specify the SAN again when signing the certificate for the server.

Similar to OpenSSL there are other toolkits such as CFSSL that supports specifying configuration files. However, the steps in Shell command are generally not always easy to automate.

Create self-signed certificate with Helm

Moving to the context of workload deployment in Kubernetes, running openSSL command isn’t always a viable option. For example, generating a certificate in the middle of deployment using a Helm Chart.

In Helm, template functions is for this purpose. In my Korthweb project I used genSignedCert to create self-signed certificate and then store the key, certificate and CA certificate as Kubernetes Secret:

{{- $dbtlscert := genSignedCert .Values.dbtls.certCommonName nil (list .Values.dbtls.certCommonName) 365 $ca }}
apiVersion: v1
kind: Secret
metadata:
 name: {{ .Values.dbtls.certCommonName | quote }}
 namespace: {{ $.Release.Namespace | quote }}
type: kubernetes.io/tls
data:
 tls.crt: {{ $dbtlscert.Cert | b64enc | quote }}
 tls.key: {{ $dbtlscert.Key | b64enc | quote }}
 ca.crt: {{ $ca.Cert | b64enc | quote }}
{{- end }}

The cons of this approach is that the syntax is not straightforward. As indicated in Helm documentation: Helm Chart templates are written in the Go template language, with the addition of 50 or so add-on template functions from the Sprig library and a few other specialized functions. While we talk about the “Helm template language” as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates. Many resources on Go templates may be helpful as you learn about templating.

Create self-signed certificate with Cert-Manager

The Cert Manager project is very popular to produce X.509 certificates directly in Kubernetes secret. We can install cert manager using Helm:

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.3 --set installCRDs=true
kubectl get pods -n cert-manager
kubectl get crd | grep cert-manager.io

Alternatively, FluxCD’s documentation on Kustomization dependency uses Cert Manager as an example. It is a good way of installing cert-manager if you have GitOps pattern.

Creating self-signed certificate for website is fairly simple. It starts with bootstrapping a CA issuer. Take the manifest below as an example. When creating the first certificate, make sure to specify isCA=true, so it stores the signing private key along with its own certificate in the ca-secret. Then use the newly created CA as issuer to create the X.509 certificate for the website.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: selfsigned-issuer
spec:
 selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: my-ca
 namespace: orthweb
spec:
 isCA: true
 commonName: my-ca
 secretName: ca-secret
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: selfsigned-issuer
 kind: ClusterIssuer
 group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: my-ca-issuer
 namespace: orthweb
spec:
 ca:
 secretName: ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: orthweb-cert
 namespace: orthweb
spec:
 commonName: orthweb.com
 secretName: orthweb-secret
 duration: 2160h
 renewBefore: 72h
 subject:
 organizations:
 - digihunch
 dnsNames:
 - web.orthweb.com
 - dcm.orthweb.com
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: my-ca-issuer
 kind: Issuer
 group: cert-manager.io

The site certificate is directly stored in Kubernetes Secret as specified in the secretName field. To fetch the certificate text, we need to decode the secret entry, for example:

kubectl -n orthweb get secret orthweb-secret -o jsonpath='{.data.ca\.crt}' | base64 -d

Note that the example above uses ECDSA algorithm with size 256 for private key and certificate. It requires that the TLS client to support ECDSA algorithm as well. For more supportability, you can use RSA algorithm (2048 or 4096 size).

In addition to creating self-signed certificate, Cert Manager supports a number of other issuer types. For example, the support of ACME issuer type enables integration with Let’s Encrypt.

Cert Manager can secure Kubernetes Ingress resources with a sub-component called ingress-shim. It is configured via annotation on the Ingress resource.

Cert Manager

Create CA-signed certificate manually

For a certificate signed by a CA, there are may paid options, from manual, to self-help, to automated. The classic manual way is using OpenSSL, generating key, CSR. The CA takes CSR to sign a X.509 certificate returned to the website administration.

Many CA websites charges for a fee and makes it easy. For example, this site currently uses certificate from SSLs.com. Apart from the fee-for-cert option, there is a website named “SSL for free“, a CA with free option for 90-day single-domain, non-wildcard certificate and we can request it simply on their website, with proof of domain ownership.

The other popular free option is Let’s Encrypt, which also employs ACME protocol. The protocol requires ACME challenges to be satisfied in order to proof domain ownership. There are a few types of challenges:

HTTP-01 challenge
DNS-01 challenge
TLS-SNI-01 challenge
TLS-ALPN-01 challenge

I have used the HTTP-01 and DNS-01 challenges. The DNS-01 challenge requires adding TXT records to DNS configuration. The HTTP-01 challenge requires adding a DNS A-record to resolve to the server, then two URIs with pre-defined value.

When I first set up this site I used certbot (the client program for letsencrypt) to create certificate every 90 days from the wordpress server, following this guide, including solving DNS-01 challenges.

Create CA-signed certificate automatically with cert manager and letsencrypt

With Kubernetes, cert-manager has the ability to integrate with let’s encrypt for full automation. Here is a good blog post on this. Domain verification is still required but it can be done automatically. We first need to register an A record that resolves host name to the Ingress IP to enable this automation. The domain ownership validation may use the ACME protocol. This should also work on private networks with private DNS and ACME protocol using a private boulder server.

Take domain name demo1.digihunch.com for example, if ingress exposes a public IP address which the domain name resolves to, then we can configure certificate with the following manifest:

kind: IngressClass
metadata:
 name: istio
spec:
 controller: istio.io/ingress-controller
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt
spec:
 acme:
 privateKeySecretRef:
 name: letsencrypt
 server: https://acme-staging-v02.api.letsencrypt.org/directory
 solvers:
 - http01:
 ingress:
 class: istio
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: demo
spec:
 dnsNames:
 - demo1.digihunch.com
 issuerRef:
 kind: ClusterIssuer
 name: letsencrypt
 secretName: demo-tls

This example uses Istio as ingress controller but the method works regardless of the controller technology behind Ingress. In the ClusterIssuer object, we’re telling it to use the staging server from letsencrypt. We also specify http01 as challenge type, and that the ingress type is istio. In the Certificate object, we provided dnsName and specified ClusterIssuser. We also tell it to store the credentials to a secret named demo-tls.

When we apply the resources above, the ClusterIssuer connects to letsencrypt server via ACME protocol. Since the DNS name already resolves to the Public IP that the ingress is hosting, the ClusterIssuer configures the required Ingress, Services and Pods accordingly so the token to satisfy the challenge is presented at the designated URI. Instead of a staging server, we can also use production ACME server for production deployment. Note that the production ACME endpoint has a stricter rate limit.

When the ACME validation is in progress, it is important to ensure that port 80 is open and there is no other mechanism (such as routing rule, authorization requirement, mandatory redirect to 443) that blocks access from letsencrypt server.

Bottom line

Cert Manager is deployed in Kubernetes, supporting a variety of issuer types. As a Kubernetes-native tool, it is a no-brainer for Kubernetes workload for X.509 certificate. Compared with using template function in Helm, it is not dependent on template function and the syntax is consistent (YAML). Compared with OpenSSL or other binary tools, it is easy to integrate with the platform.

Single-node Kubernetes cluster – docker desktop

Sun, 22 Aug 2021 00:24:00 -0400

While there are many tools to set up single-node Kubernetes cluster (e.g. minikube, MicroK8s, kind, or k3s with the k3d wrapper), docker-desktop has a significant advantage: it comes with Docker installation, on MacOS, or on Windows. It is installed simply by enabling the option “Enable Kubernetes”. It can be blown away and reset in a heartbeat (with the button “Reset Kubernetes Cluster”). For its versatility, docker-desktop is a great development environment.

However, there are always nuances, which motivates me to write this blog. I wanted to note down what is on earth different about Docker-desktop, because the instructions for applications might differ slightly between single-node cluster on MacOS/Windows and the “real” multi-node cluster.

I will start with a deep dive into the docker-desktop architecture, then we’ll go through the steps to install some common applications with Kubernetes.

Docker-Desktop on MacOS

There are a number of open-source and proprietary projects involved to bring docker-desktop to implementation. Let’s begin with the following five:

Hypervisor Framework: Apple’s APIs on MacOS that allows you to interact with virtualization technologies in user space.
bhyve: A type-2 hypervisor initially written for FreeBSD (and was contributed to FreeBSD in May 2011).
xhyve: A port of bhyve project to MacOS with integration via Apple’s Hypervisor Framework. The Hypervisor Framework allows xhyve to run entirely in userspace. It is sometimes loosely referred to as xhyve/bhyve hypervisor, and is optimized for lightweight virtual machines and container deployment.
HyperKit is an open-source toolkit on macOS based on xhyve. HyperKit is lightweight and therefore allows you to embed hypervisor capabilities in your application. The hypervisor component in HyperKit is based on xhyve/bhyve. HyperKit is designed to be interfaced with higher-level components such as the VPNKit and DataKit. Docker-desktop and MiniKube are built on HyperKit.
LinuxKit is a toolkit for building custom minimal, immutable and purpose-build Linux distributions. It supports several well-known hypervisor platforms, such as HyperKit, Hyper-V, qemu and VMware. LinuxKit started as an internal project in Docker Inc and is now managed as a Moby Project.

Hyperkit is installed as part of docker desktop. The process can be found with ps command:

ps -Af | grep hyperkit

Or in the activity monitor:

Docker related processes

Docker-deskop is essentially a LinuxKit virtual machine (as defined here). It runs containerd process inside of the virtual machine. This is an older article about this architecture. If Kubernetes is enabled, the virtual machine is also installed with kubelet, the agent process running on each Kubernetes node.

Since MacOS is not the direct host of the containers, there is no way to map MacOS file system to container’s as you can with a Docker/Kubernetes host. Prior to Docker 20.10, there used to be a trick to indirectly access host volume from MacOS terminal. It has stopped working according to this issue but workarounds are provided here. This post proposes some good alternatives to access the file system of LinuxKit VM. For example, use netcat:

 user@LinVM # nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock

/ # cat /etc/os-release
cat /etc/os-release
PRETTY_NAME="Docker Desktop"
/ # cat /etc/kubernetes/current-version
cat /etc/kubernetes/current-version
kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"archive", BuildDate:"2021-06-18T05:24:26Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Typing in this terminal session feels clunky. According to this thread, we can connect to the LinuxKit VM with tty and sane auto completion, using the command below:

stty -echo -icanon && nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock && stty sane

There are some other alternatives, using privileged Docker containers:

docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh

The following command uses a smaller image:

docker run -it --rm --privileged --pid=host justincormack/nsenter1

As with Kubernetes, to access the file system on the node is via a privileged container, you can follow the tips from Azure, identify node name, and debug against the node using a special container:

kubectl debug node/docker-desktop -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11

Note that the root directory on the host is mounted to container’s file system as /host. This mapping renders a lot of symbolic link as dangled, even though they are actually not on the host file system.

Docker-Desktop on Windows

Docker works with Linux kernel. There have been a couple of efforts to run Linux virtual machine on Windows. For example, Hyper-V backend, and Windows Subsystem Linux (WSL) backend.

Traditionally, Docker on Windows was implemented with Hyper-V as the hypervisor. A LinuxKit distro is running on the Hypver-V VM, provider Linux kernel capabilities. Docker refers to containers running in this architecture as “Windows Containers”, which is a misnomer in my opinion.

The first release of WSL provides a Linux-compatible kernel interface and runs a GNU user space on top of the interface. Neither the Linux kernel code, or a hypervisor is involved. The user space contains GNU Bash shell, command language, command-line tools and interpreters. The absence of Linux kernel in WSL, makes it useless for Docker setup. At that time The Hypver-V backend was still the only option to host docker container during the first version of WSL. This post has a diagram of Docker on Windows with Hyper-V backend.

WSL2 comes with a real Linux Kernel (also on top of Hyper-V), making WSL2 a better alternative than the legacy Hyper-V as the backend of Docker on Windows. It can be turned on as the screenshot shows above. The rest of this post assumes WSL2 as backend. In this setup, we run a Bootstrapping distro independent of the WSL2 Linux distro, although both inside of the lightweight Linux Utility VM. Below is the diagram:

The innovative component is the lightweight Linux Utility VM. It is called a VM, but very different from the traditional sense of VM such as VirtualBox or VMware. Traditional VM is isolated from host OS, slow to boot and has large memory footprint. The lightweight Utility VM on the other hand, is integrated with host OS, super fast to boot (i.e. ~1 second), and comes with small memory footprint. It is not turned on until needed. The VM runs both a WSL2 Linux Kernel and GNU/Linux usermode (known as “distribution”, for example, Ubuntu). When an end-user say WSL2, s/he most likely refers to the distribution. Similarly, the so called “docker-desktop on Windows with WSL2 backend”, is also managed as two WSL2 distros: the bootstrapping distro (docker-desktop) and the data store distro (docker-desktop-data).

The detailed components in the Bootstrapping distro is in the second diagram in this post, which has a detailed discussion. With this architecture, you don’t even need the WSL2 Linux Distro for Docker desktop to function. You can even run docker CLI command from Windows PowerShell without any Linux distro (although this is implemented only for backward compatibility and not recommended anymore) . In the following session, we first list out the WSL2 distros. Notice that the docker-desktop distro is not the default. We then connect to the distro using -d switch. Last, we run docker info from windows user space.

PS C:\WINDOWS\system32> wsl -l -v
 NAME STATE VERSION
* Ubuntu Running 2
 docker-desktop Running 2
 docker-desktop-data Running 2
PS C:\WINDOWS\system32> wsl -d docker-desktop
WINLAPTOP:/mnt/host/c/WINDOWS/system32# cd ~
WINLAPTOP:~# printenv|grep DIST
WSL_DISTRO_NAME=docker-desktop
WINLAPTOP:~# exit
PS C:\WINDOWS\system32> docker info
Client:
 Context: default
 Debug Mode: false
 Plugins:
 buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
 compose: Docker Compose (Docker Inc., v2.0.0-beta.6)
 scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 93
 Running: 80
 Paused: 0
 Stopped: 13
 Images: 28
 Server Version: 20.10.7
 Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
 userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
 seccomp
 Profile: default
 Kernel Version: 5.10.16.3-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 12.32GiB
 Name: docker-desktop
 ID: WHDE:PJF3:HVFC:AZJA:EDKH:VUZR:RRUJ:HHXX:TDV5:4UJG:XY4E:PK4F
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
 127.0.0.0/8
 Live Restore Enabled: false

The wsl -d command as illustrated above is a good way to connect to the docker-desktop distro. The alternative to get to the distro is via privileged container (Docker) or helper pod (Kubernetes), which is the same as in Docker-desktop on MacOS. Refer to the section above for specific steps.

Application Install on docker desktop

The followings are my notes to install commonly used applications in Docker Desktop with Kubernetes. They work on both MacOS or WSL2, requiring Kubernetes enabled.

Metric server

The metric server is provided in the official repository. Releases are publish here, which provides the installation step as follows:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml

However, there is some issues when deploying it on MacOS, the deployment will fail due to certificates not matching the hostname. To fix the issue, it is recommended to download the yaml file (components.yaml), and edit the file by adding –kubelet-insecure-tls to the args section of the container named metrics-server. This is sufficient to fix the issue. Some people are not comfortable with port 443 being insecure TLS, and would rather change the port to 4443. This is completely unnecessary but if that’s the case, make sure the named port for https is also updated to 4443.

Once metric server has been installed, the following two commands should return meaningful results:

kubectl top no
kubectl top po

This should also allow application (Pods) to query for cluster resource usage. When working with single-node cluster on MacOS or WSL2, multiple Pods might come up with a single command and the memory can be easily over-subscribed. The two commands above allows you to check and make adjustment on the node configuration (in Docker preference).

Dashboard

Similar to metric server, the dashboard is kept in the official repo, in the path of aio/deploy/recommended.yaml. Here is the published release, where the instruction says:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml

However, this is not directly applicable either because the login page requires token or kubeconfig. We need to be able to bypass that. To do so, download the yaml file (recommended.yaml), and add parameter –enable-skip-login to the args section for the container named kubernetes-dashboard.

To display the login page properly, we need to start the proxy using this command:

kubectl proxy

The login page will then be available at this URL. The URL reflects the namespace and service name. On the login page, the skip button will be available.

Rancher

The official installation guide of Rancher 2.5x recommends RKE Kubernetes. If you prefer not to run a separate cluster on MacOS, you can install it on docker desktop (with Kubernetes enabled). The installation steps require Helm 3 and are completed in three helm commands.

Install Nginx Ingress controller using Helm, following the three commands here. Alternatively, you can apply the rendered template as posted here. The controller will later be used by the ingress that Rancher’s chart creates.
Follow the steps on this page to install Rancher, even though the page does not say it applies to docker desktop. If you do not have TLS certificate, the Rancher helm chart can generate one for you, using cert-manager.
The installation exposes rancher application on port 443 of the MacBook, and the cert is issued to “rancher.my.org” by default. To access it, add “127.0.0.1 rancher.my.org” to the host file.

To back out from the steps above, just uninstall with helm. For example:

helm uninstall rancher -n cattle-system
helm uninstall cert-manager -n cert-manager
helm uninstall ingress-nginx

Jenkins

Similar to Rancher, Jenkins instruction assumes minikube cluster instead of docker desktop.

helm repo add jenkinsci https://charts.jenkins.io
helm repo update
kubectl create namespace jenkins
helm install jenkins -n jenkins -f https://raw.githubusercontent.com/jenkinsci/helm-charts/main/charts/jenkins/values.yaml jenkinsci/jenkins
kubectl --namespace jenkins port-forward svc/jenkins 8080:8080

To find out the default password for admin user:

kubectl -n jenkins get secrets jenkins -o jsonpath={.data.jenkins-admin-password} | base64 -D

To uninstall Jenkins:

helm uninstall jenkins -n jenkins

Container Runtime

Although I seem to be a proponent of docker desktop thus far, this post would be incomplete to not discuss what is missing with docker desktop. One key difference between docker desktop and minikube is the container runtime being used. Docker desktop uses docker as the runtime, and it does not support other runtime as of now. Minikube allows user to choose runtime, including containerd.

Because of this difference, Kubernetes nodes with Docker as runtime and with containerd as runtime place pod log files in different locations. To find out the runtime, use the following command:

kubectl describe node <node_name> | grep Runtime

If the runtime is docker, the stdout of container is placed in /var/lib/docker/containers/<sha>/. If the runtime is containerd, the stdout log of pods are stored in /var/log/containers/. This is important to know when you configure log shipping and needs to get stdout from node. The log path used in containerd is the standard path in compliance with Container Runtime Interface (CRI) so you should develop log shipping solution based on that.

Docker’s refusal to comply to CRI also caused Kubernetes to stop supporting it as container runtime as of Dec 2020. For more background, refer to this and this. Here is also an article with great diagrams on the removal of docker-shim.

Bottom line

Docker-desktop is a great tool for a quick single-node Kubernetes environment. As of docker 20.10, docker-desktop still uses docker as runtime. This limits its use case to development only.

If you need a CRI compliant environment, docker-desktop is not a good choice. We will discuss alternatives in the next post.

Infrastructure deployment in Terraform 1/2

Wed, 11 Aug 2021 21:44:00 -0400

Terraform is an excellent Infrastructure-as-Code (IaC) tool based on Hashicorp Configuration Language (HCL). Compared to JSON or YAML based declarative templates (e.g. CloudFormation and ARM), HCL is more concise, thanks to the flexibility of HCL. On the other hand, HCL is not as flexible as general purpose languages. For that sake, I see HCL as semi-declarative IaC. This post is my notes about best practices with Terraform development, from the context of AWS, but also applies to other cloud platforms.

Complex Types

There are three primitive types (string, number and bool) that forms collection types and structural types. Here are some common ones:

list: element may repeat, and order is maintained:

["orange", "banana", "orange", "apple"]

set: elements are unique and unordered

["apple", "banana", "orange"]

tuple: each element has its own type

["a", 15, true]

object: defined by a schema with named attributes each with its own type

{
 name = "John"
 age = 52
}

list of object

[
 {
 alloc_id = "0b7271a3219bc1fc2"
 subnet_id = "0c02af76c2c3e46fa"
 },
 {
 alloc_id = "0440c334c48d4247f"
 subnet_id = "02652e69fa2a71de8"
 }
]

map of string

{
 property = "foo"
 attribute = "bar"
}

map of object

{
 objkey1 = {
 alloc_id = "0b7271a3219bc1fc2"
 subnet_id = "0c02af76c2c3e46fa"
 }
 objkey2 = {
 alloc_id = "0440c334c48d4247f"
 subnet_id = "02652e69fa2a71de8"
 }
}

Whenever applicable, Terraform converts types implicitly or explicitly. For example, when a list or tuple is converted to set, all elements are converted to string and duplicates are removed. Object and map are very similar. Map of string can be converted to object if the attributes comply with the schema. Additional attributes not in the schema are discarded.

HCL Types is similar to Python

Although being totally different beasts, the complex types between HCL and Python are similar, to the point I suspect the HCL design is influenced by Python. I summarize the similarities as such:

Python Types	list []	tuple ()	set {}	dict {}
Terraform Types	list []	tuple []	set []	map {}
ordered	Y You can access item by index	Y	N you cannot access an item by index or key; however you can loop over all itmes	N key-value pair that allows you to access item by key
changeable (mutable)	Y	N You cannot update, add or remove items	Y Add or remove only. no change to existing elements	Keys must remain unique or the values get overwritten; Values are mutable
allow duplicate elements	Y	Y	N	Keys must be unique; values don’t have to

In Python, list and tuple allow elements of mixed types but in IaC like Terraform we mostly don’t need mixed types. In Terraform, an object is a map without a defined type. In most situations, lists and tuples behave identically, as do maps and objects.

Another area of similarity is with the comprehension of list and dict/maps. In Python for example,

## Supposed you need to create a list:
lst=[]
for x in range(10):
 lst.append(x**2)

print(lst)

### that can be simplified as the following to create the list:
lst = [x**2 for x in range(10)]

## You can even add contidion
even_numbers=[num for num in range(10) if num%2==0]

## You can introduce function calls:
words = ["hello", "world", "python", "list", "comprehension"]
lengths = [len(word) for word in words]

## You can even combine two lists
lst1=[1,2,3,4]
lst2=['a','b','c','d']

pair=[[i,j] for i in lst1 for j in lst2]

print(pair)

## With dict, it's similar
evens={x:x**2 for x in range(10) if x%2==0}
print(evens)

In Terraform, we use similar techniques:

[for s in var.list : upper(s)] # build a tuple/list from a list

[for k, v in var.map : length(k) + length(v)] # build a list from a map

{for s in var.list : s => upper(s)} # build a map from a list

[for s in var.list : upper(s) if s != ""] # build a tuple/list from a list with condition

Note that the documentation of Terraform doesn’t explicitly call them out as comprehensions. However, it’s exactly the same idea as comprehensions in Python. Even the range() function exists both in Python and Terraform.

Modularization

Modules allows you to group related resources together. They can also be re-used and called by other modules. It is fairly straightforward to create a module:

put the resource declarations into a sub-directory
define input and output in the directory

However, the introduction of module complicates the directory structure and variable referencing, which is important to take into account before starting creating modules. This guideline has further discussion about when to create a module. I re-wrote the terraform templates in Orthweb project to leverage modularization wherever possible, but there is still some stand-alone resource (e.g. random_id) not belonging to any module.

To reference resources across modules, you need to import those resources (using data source) from within the module. There are a couple of ways. You may pass the argument of data source as input variable, or you can leverage the filter capability of data source. Let’s look at one example of each mechanism.

In the example below, we import a subnet by subnet id:

data "aws_subnet" "private_subnet" {
 id = var.private_subnet_id
}

In the example below, we import a subnet by filtering from all subnets in the VPC by tag:

data "aws_subnet" "private_subnet" {
 vpc_id = var.vpc_id
 filter {
 name = "tag:Name"
 values = ["Private"]
 }
}

There are pros and cons of each approach. A module with mechanism 1 is more transferrable across different environment, because the ID of subnet is explicitly provided. However, authors needs to manage those explicit variables with code. Mechanism 2 fetches target resources with filter. It depends on a well-implemented tagging policy in the resource farm.

The Terraform Registry (since 2017) contains a lot of pre-built modules for each backend platform (e.g. AWS). If you find any module that can be used in your project, the module repo can be referenced directly by Git repository URL. You should be aware of the risk of this practice though. Many platforms are keen to publish modules for their platform. Anyone can publish their own modules to the community as well.

You can quickly generate module documentation with terraform-docs.

Local Execution

Local execution is the basic workflow mode which is mostly seen with very small collaboration team. In this mode, the developer executes terraform binary (Terraform CLI) from their workstation (e.g. Laptop). The Terraform CLI converts code into API calls to interface cloud provider. The most frequently used commands (from terraform directory) are:

terraform init
terraform plan
terraform apply

The init command initializes the working directory. The plan command figures out the delta between code and infrastructure. It outlines the changes it is about to make. The apply command commits the change. The documentation of Terraform CLI commands is here.

Terraform keeps track of the infrastructure it manages in state file. This article explains the purpose of state. State management collaboration difficult with local execution because the state file by default is created in the working directory on user’s workstation. Although the state file can be configured to be stored in a shared location such as S3, it still requires a mechanism to lock the state in a multi-developer collaboration.

In large operations, the same code base in Terraform, is usually used to created several different sets of infrastructures, for example, in different geographic regions. So it is a 1-to-many relationship between the code repo and the infrastructure state. To further complicate things, each state might have been deployed using different revisions of the code. To overcome that challenge, Terraform introduced the concept of workspace, which is essentially an instance of state describing a particular group of infrastructure being managed by the same source code. When there are many workspaces, it becomes tricky to manage them with CLI commands.

State management is a major challenge that needs to be solved for team collaboration in local execution workflow. Each state must use the same revision of Terraform code. You can use Git in combination as a workaround to that limitation but the point is you cannot tie a workspace to a commit with the workspace commands.

In some enterprise environment, the execution is from a VM (e.g. ADO agent on-premise) without Internet access, which poses another challenge. First, we need to pre-load required providers manually. The enterprise needs a proxy solution to safely download packages from Hashicorp website. One good option is Nexus Repository, with both open-source and pro supports. It is a full-function artifactory repo that can host helm repo, apt repo, yum repo, etc. Second, we also need to configure Terraform so it picks up providers locally. Managing plugins without Internet access requires understanding of the order in which Terraform tries to load plugins during initialization.

Remote Execution

In remote execution, the code is executed in Terraform Enterprise or Terraform Cloud. Both are remote web servers. The difference is that Terraform Enterprise is self-hosted service, requiring IT specialist to install and maintain Terraform Enterprise. Terraform Cloud on the other hand, is a managed SaaS service. The pricing model includes a free plan for small number of users.

Terraform workspace configuration

In Terraform Enterprise or Cloud, the remote execution is organized in workspaces. You need to create an organization, and then create workspace under the organization in order to execute code. With each workflow, you can specify version control system (VCS) and subdirectory, to tell the workspace where to fetch Terraform code from. The workspace also allows you to define secrets and variables specific to the workspace. When you execute a workspace plan, the secrets and variables are passed from workspace to the execution logic.

Terraform Workspace Variable configuration

You will also need to design the Terraform code in a way to work seamlessly with the secrets and variables loaded from the workspace. The variable declaration in code should match the definition in workspace. There are already a number of variables that came in handy. Check out this guide.

Each execution is referred to as a “run”, with its own run id. A workspace involves may runs, which may succeed or fail. Each run pulls a specific commit of the source repository, and goes through stages such as plan, and apply. The UI from each run result list out the status of each result, in a very easy to read format.

The state data is persisted in the web server as they were generated. Therefore the collaborator does not need to worry about managing state with CLI tools. If there are files that you do not want picked up by the execution engine, their locations can be added to a file .terraformignore. Refer to this guide.

AWS profile

Local execution still has a lot of use cases in enterprises such as testing with temporary resources. A common challenge is authentication. As discussed, Terraform CLI picks up identity information from AWS CLI and authenticates its way into the backend to run API calls against. So AWS CLI must be configured correctly with the sufficient permission to provision resources. On the other hand, enterprises usually offload IAM to an identity store, such as AzureAD, Okta, etc. Putting those together, the pattern of authentication and authorization usually looks like this:

User logs on via SSO (e.g. SAML). The validation response gives a name of an IAM role.
Upon successful authentication, user takes the IAM role. The role does not have any capability, except for assuming a second IAM role.
The second IAM role (the functional role) grants user the permission to do its business.

The steps above, can be carried out in AWS console, or with AWS cli using assume-role command. However, when we put Terraform in the picture, it becomes a little involving because the credential information is updated whenever the functional role is assumed, and the assume-role command takes a pretty long argument.

To skip typing the long command every time, there are some handy tools, such as aws-azure-login. An even better tool that works with a variety of identity stores is saml2aws. The tool allows you to configure identity backend, assume the functional role, and update credential information in aws credential file, all with a single command. The AWS CLI configuration reads:

[default]
region = us-east-1
output = json
cli_history = enabled
cli_pager =
role_session_name = functional_operation

[profile function_user]
source_profile = default
role_session_name = functional_operation
role_arn = arn:aws:iam::9998887766:role/admin-access
region = us-east-1

In Terraform provider, we need to tell it to assume that role as well:

provider "aws" {
 region = "us-east-1"
 assume_role {
 role_arn = "arn:aws:iam::9998887766:role/admin-access"
 session_name = "terraform"
 }
}

This will ensure Terraform assumes appropriate role before doing its job.

AWS EC2 SSH Key Pair

RSA key authentication for SSH should be used for Linux Instances. When creating an EC2 instance, we give it our public key so we can then later authenticate through SSH. If the key is already stored in AWS, we just need to tell EC2 the name of the key, in the key_name property. If the code is likely to be executed from several different places by different users, then we can write the code so it picks up public key from user’s workstation (~/.ssh/id_rsa.pub). Here is an example:

variable "local_pubkey_file" {
 type = string
 default = "~/.ssh/id_rsa.pub"
}
data "local_file" "pubkey" {
 filename = pathexpand(var.local_pubkey_file)
}
resource "aws_key_pair" "user-pubkey" {
 key_name = "runner-pubkey"
 public_key = data.local_file.pubkey.content
}
resource "aws_instance" "bastion" {
 instance_type = "t2.micro"
 key_name = aws_key_pair.user-pubkey.key_name
 ......
}

For remote execution, we can even add an option to pass public key in as variable, to override the key file variable. For an example, check out my orthweb project.

To upload files to EC2 instance from Terraform execution environment, we can use the file provisioner with ssh as connection type.

Git Branching Strategy

Sun, 07 Mar 2021 19:20:42 -0400

I have been in two discussions about Git branching strategy in different organizations. Too many concepts! So I open this post to jot down the lineage of common branching strategies to help organizations develop their branching policies. In terms of reference, there is a lot from Atlassian documentation. In addition, I also find this one article a good resource.

Centralized workflow (no branching)

In Centralized Workflow, the team uses a central repository to serve as the single-point-of-entry for all changes to the project. The default branch is master, and all changes are committed to this branch. This workflow does not require any other branches beside master. Local changes may conflict with upstream commits, and conflict needs to be resolved.

This workflow is usually seen in teams transitioning from SVN, with very basic skill level. This workflow may also be adopted in teams working on configuration management instead of source code. Centralized workflow is great for small teams. The conflict resolution process can form a bottleneck as the team scales in size.

Feature Branch Workflow

Instead of directly committing on their local master branch, developers create a new branch every time they start work on a new feature. Feature branches should have descriptive names (e.g. issue#112). Feature branches are pushed to the central repository so that they can be shared to other developers without touching any official (master) code.

To get feedback on the new feature branch, create a pull request in a repository management solution (e.g. Bitbucket Cloud, Bitbucket Server). Before merge, you may have to resolve merge conflicts if others have made changes to the master branch of repo. This is to make sure your local master is synchronized with the upstream master. When your pull request is approved and conflict free, you can merge your branch to master branch.

feature branch workflow

The Git Feature Branch Workflow is a composable workflow that can be leveraged by other high-level Git workflows. Git Feature Branch Workflow is branching model focused, instead of release focused. The Git Feature Branch Workflow can be incorporated into other workflows. The Gitflow, and Git Forking Workflows traditionally use a Git Feature Branch Workflow in regards to their branching models.

Gitflow Workflow

First published in 2010 by Vincent Driessen. Gitflow defines a strict branching model designed around the project release. This provides a robust framework for managing larger projects. In addition to Feature Branch Workflow, Gitflow workflow assigns very specific roles to different branches and defines how and when they should interact.

Instead of a single master branch, this workflow uses two branches to record the history of the project. The master branch stores the official release history, and the develop branch serves as an integration branch for features. It is also convenient to tag all commits in the master branch with a version number.

This workflow is operated in the following ways:

A develop branch is created from master
Feature branches are created from develop. When a feature is complete, with PR reviewed, it is merged into the develop branch. Features branches
Once develop has acquired enough features for a release (or a predetermined release date is approaching), we fork a release branch off of develop. Creating this branch starts the next release cycle, so new features can be added to develop after this point. On the release branch itself, only bug fixes, documentation generation, and other release-oriented tasks should go in this branch. Once ready to ship, the release branch gets merged into master and tagged with a version number. In addition, it should also be merged back into develop, which may have progressed since the release was initiated.
Maintenance or hotfix branches are used to quicly patch production releases. Hotfix branches are a lot like release branchs and feature branches except they’re based on master instead of develop. As soon as the fix is complete, it should be merged into both master and develop (or the current release branch), and master should be tagged with an updated version number.

Gitflow workflow

There is a Git extension named git-flow to provide high-level repository operations for this Workflow, such as start a release, finish a release, start a hotfix, finish a hotfix.

Gitflow is ideally suited for projects that have a scheduled release cycle and for the DevOps best practice of continuous delivery. It ensures that the master branch reflects what is deployed (e.g. in production). However, it is quite complex and have a steep learning curve for organizations. It also runs long-lived branches, which is considered bad from CI/CD perspective. Branches are by definition to isolate and hide changes, whereas continuous integration is about exposing changes early on and frequently. In that sense, the Gitflow branching model and CI/CD are mutually exclusive ideas.

In 2020, Vincent Driessen added a note at the beginning of his article on Gitflow:
If your team is doing continuous delivery of software, I would suggest to adopt a much simpler workflow (like GitHub flow) instead of trying to shoehorn git-flow into your team.
If, however, you are building software that is explicitly versioned, or if you need to support multiple versions of your software in the wild, then git-flow may still be as good of a fit to your team as it has been to people in the last 10 years.

Driessen’s notes also points to some simple alternatives. On the other hand, the Atlassian tutorial on Gitflow has described Gitflow workflow as a legacy (since Aug 2021 based on web archive). It points out at the beginning that Gitflow has fallen in popularity in favor of trunk-based workflows.

GitHub flow, Trunk-based development (TBD) and GitLab flow

It is operationally expensive to manage multiple mainlines in Git flow workflow, with both source control and release in the picture. Some simple alternatives have been evolved, with single mainline, for example, GitHub flow and trunk-based development. They differ in where the release is performed from.

In the GitHub flow, release is performed from branch before being merged back to master (trunk).

GitHub flow

In trunk-based development, release is not performed until the feature branch has been merged to the trunk (master). In trunk-based development, feature branches are supposed to be short-lived. It is a common practice among DevOps teams, since it streamlines merging and integration phases.

Trunk-based development

Trunk-based development has gained some momentum in recent years, due to its DevOps friendliness. This is a website that advocates it and here’s a DZone article about it.

Gitlab Flow

In response to the shortcomings of GitHub flow and Gitflow, Gitlab introduced its own proposal of branching strategy, known as Gitlab flow. The most distinctive aspect is the environment branches. In Gitlab flow, you run multiple long-lived branches, each of them representing an environment. The typical steps are as follows:

You create short-lived feature branches, and merge them often to the master.
Every developer starts from master and targets master. Other branches are merged from previous lower environment branches.
You can deploy a new version to production, by merging master into the production branch.
If you need to know what code is in production, you can check out the production branch to see.

Gitlab flow

You only need to work with release branches, if you need to release software code to the outside world. Here are some best practices in GitLab flow.

Forking Workflow

Forking workflow is fundamentally different. The key steps are as follows:

A developer ‘forks’ an ‘official’ server-side repository. This creates their own server-side copy. This is their personal public repository, and no other developer are allowed to push to it.
The new server-side copy is cloned to their local system. This forms an environment dedicated to this developer.
With the local clone, developer needs to create the upstream remote manually using “git remote add upstream” command. This allows the developer keep the local repository up-to-date as the official project progresses.
A new local feature branch is created. Developer commits to the new local branch, and pushes to their own copy of repository on server.
Developer files a pull request from the new branch (in own copy of repository) to the ‘official’ repository. The project maintainer knows that an update is ready to be integrated. The PR also serves as a discussion thread.
The PR gets approved for merge and is merged into the original server-side repository.

This workflow has other names, such as fork-and-branch workflow, and is commonly used in GitHub for managing open-source projects. However, this should not be confused with the aforementioned GitHub flow.

Conclusion

In order to fully support distributed source control, Git abstract version control problems into concepts such as commit, branch, etc. This makes discussion about Git workflow and branching strategy difficult due to the conceptual hurdles and organization differences. We covered choices of Git branching strategy in this post. As Vincent Driessen commented in his original Gitflow posting, panaceas don’t exist. We should consider the context (e.g. team size, Git skill level, etc) to determine the best Git branching strategy.

Host legacy application in Docker 2 of 2

Thu, 22 Oct 2020 17:54:00 -0400

My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months…

Use Case

I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process. When it is up it listens to more than 70 TCP ports for different business services. Here is a simplified list:

Application service	TCP port to bind
Business service A	8030
Business service B	8040
Business service C	8050
……	…

TCP port requirement

The application also communicates with database and search engine on the same server. Since I am building a training environment where multiple instances of our application needs to run on a single server host. All these instances of application share the same underlying database and search engine services. With multiple instances, additional constraints are introduced. For example:

Each instance requires more than 120 configuration files. A small number of them defines what ports the process binds to. The rest of configuration files are the same across all instances.
The OS needs to host 6 processes of the same application all running at the same time;
The OS does not allow multiple processes to bind to a single TCP port (duh!);
It is extremely labourious to change the path for application to read configuration files from. This bad configuration also breaks the upgrade process going forward.

From the statements of constraints, I determine that we need a mechanism to bring running application process into an isolated environment. This is exactly the definition of container and a perfect use case for docker. The following table represents an example of how the multiple instances can be orchestrated.

OS	Container ID	Application Service	container port	published port
Host CentOS	Container 1 (Instance #1)	Business Service A	8030	9301
		Business Service B	8040	9401
		Business Service C	8050	9501
	Container 2 (Instance #2)	Business Service A	8030	9302
		Business Service B	8040	9402
		Business Service C	8050	9502
	Container 3 (Instance #3)	Business Service A	8030	9601
		Business Service B	8040	9602
		Business Service C	8050	9603

This way of orchestration allows the different instances of applications to share as much configuration files as possible, so that each process thinks that they bind to TCP ports (8030, 8040, 8050, etc), by taking advantage of Docker’s ability to map ports for publishing.

Below is an example of the docker compose file:

version: '3.6'
services:
 dapp1:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp1
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9301:8030 # BUSINESS SERVICE A
 - 9401:8040 # BUSINESS SERVICE B
 - 9501:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:cf
 environment:
 - INSTANCE_TAG=dapp1 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp1/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp1/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp2:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp2
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9302:8030 # BUSINESS SERVICE A
 - 9402:8040 # BUSINESS SERVICE B
 - 9502:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d0 
 environment:
 - INSTANCE_TAG=dapp2 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp2/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp2/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
 dapp3:
 image: docker.digihunch.com/dapp:${DAPP_VER}
 container_name: dapp3
 entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
 ports:
 - 9601:8030 # BUSINESS SERVICE A
 - 9602:8040 # BUSINESS SERVICE B
 - 9603:8050 # BUSINESS SERVICE C
 mac_address: 2c:1f:4e:c5:9e:d1 
 environment:
 - INSTANCE_TAG=dapp3 
 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
 networks:
 - vcnet
 volumes:
 - /opt/dapp/etc:/opt/dapp/etc:ro
 - ./instances/dapp3/dapp.lic:/opt/dapp/etc/dapp.lic:ro
 - ./instances/dapp3/variables:/opt/dapp/etc/variables:ro
 deploy:
 resources:
 limits:
 cpus: '0.5'
 memory: ${DAPP_MEM:-4096M}
 reservations:
 memory: ${DAPP_MEM:-4096M}
 tty: true
networks:
 vcnet:
 driver: bridge
 driver_opts:
 com.docker.network.enable_ipv6: "false"

In this compose file, the environment variables are stored in .env file in the same directory and if they are not declared, the default is specified (syntax: ${VAR:-default}).

Helper scripts

The docker commands are fairly long so I had to organize them into several helper scripts. For example:

docker-entrypoint.sh: this script is the ENTRYPOINT script for container. It is responsible for:
- Initialization work that cannot be done in Dockerfile, such as setting environment variable
- Launch the application, including pointing log file to stdout
- Adding host entry for host.docker.internal to /etc/hosts, as a workaround to this issue with Docker on Linux
build_image.sh: this script makes the image build process smoother
- check if image to build already exist, and ask permission to delete the existing image if so;
- build the image with Dockerfile, and create directory structure for Dockerfile to use during COPY instruction
start_dapp_all.sh: this script starts all containers using docker-compose up and also add required iptables rules. We need to edit PREROUTING rules in IP tables to allow traffic between host NIC interface and the docker bridge interface, created each time service is up, as pointed out in previous post.
stop_dapp_all.sh: this script removes the relevant iptables rules and stop all containers using docker-compose. Note that when deleting routing rules by number, start from the highest rule number and work your way down, since each deletion will cause the rules to be re-numbered.

Permission

The container uses a non-root user to run application (e.g. with su dhunch -c “command” from entry point script to run application as dhunch user), because the legacy application uses the same (non-root) user to perform its actions, and it is generally not advised to use root user. To ensure consistency, we need to create the dhunch user in container (in Dockerfile) so it’s uid and gid aligns with those of the host. The file and directory on the host to be access by the process in container also needs to allow dhunch user to read and write. Otherwise, entry point script will fail.

In the docker-compose file, we mount a file or a directory on the host to the container, and specify 😮 if it is read only mount, under volumes. We can alternatively use bind mount (check here for comparison). In either case, we need to keep in mind of the permission – owner alignment. For example, we have the following mount statement under volumes:

– /var/lib/dapp/dcontainer/archive:/var/lib/dapp/dhost/archive

We also need the entire directory hierarchy accessible to dhunch user. To configure this correctly, we need to create the entire directory hierarchy and set proper owner to it. Here is the comparison between the bad configuration and good configuration:

	Dockerfile instruction for container	Permission issue during mount by docker-compose
Bad config	RUN mkdir -p /var/lib/dapp && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was not created until mount time and it is created implicitly with root as owner (since there is no user section in docker-compose, so root as default is used). The application running as dhunch user in container will have permission issue going into dcontainer directory after mount.
Good config	RUN mkdir -p /var/lib/dapp/dcontainer/archive && chown -R dhunch:dhunch /var/lib/dapp	The directory “dcontainer” was already created with proper permission prior to mount and the main application process running as dhunch user will not have permission issue.

For application process running as dhunch, it also needs to write logs to stdout, so the result can be viewed from outside the container using docker logs command. The docker-entrypoint.sh script makes this happen by:

su dhunch -c "ln -sf /dev/stdout $DHUNCH_LOG_DIR/dhunch.log"

However, this command itself will run into permission issues. To fix, we need to add user dhunch to tty group (e.g. in Dockerfile as it’s needed on every container):

usermod -a -G tty dhunch

For application process to write to a shared volume on host (e.g. NFS), we can either allow access through volume mapping, or for performant access, mount the NFS share directly to container with proper driver.

Java application

For Java applications, only use the needed package (openjdk, openjdk-devel, openjdk-headless) as the Docker image size must be kept as small as possible. The headless package is for non-UI components, the devel package is for development stuff.

It is also worth-noting that the upper limit of heap size (Xmx) should be set based on the reserved memory of container (specified under docker-compose under resource limit and reservation). If heap is larger than container’s available memory, OOM will be triggered and the container will be killed. This article has some good explanation on this.

Automated Deployment Pipeline 3 of 3

Wed, 14 Oct 2020 17:27:00 -0400

Background

We have previously covered a pipeline example with Jenkins calling Ansible to leverage OpenSSH configuration and Ansible inventory. We also discussed a use case with declarative pipeline.

In this posting, I will provide another advanced example, built on declarative pipeline. The pipeline file will be pulled from Git repository. Also, the script is executed on a remote agent, instead of the Jenkins master server. The reason this example is important, is that:

Jenkinsfile is now version controlled (pipeline as code);
Service script (e.g. python) is also version controlled from a central repository;
Computing resource is provided by a remote agent. Since the script is pulled before running, the agent is still fungible;
The result from service script execution is archived (similar to the way build artifact is stored in Jenkins) in master.

The architecture of this pipeline now becomes the followings:

Configure Agent

Jenkins has a plugin called SSH Build Agent, that allows you to configure a Linux agent, communicating with Jenkins master in SSH. For Window agent, it uses JNLP to communicate with master, which is outside of our scope of discussion. As I touched on in previous post, Jenkins uses its own implementation of SSH protocol to achieve this. This means that it cannot re-use the configurations in ~/.ssh/config and thus the ability to do SSH chaining is eliminated. This is incompatible with our architecture so I have to register Jenkins agent using a different launch method “Launch agent via execution of command on the master”. The execution of command on master can still leverage OpenSSH config file. In order to do so, we must copy the agent.jar file to the remote agent first (URL is ${JENKINS_URL}/jnlpJars/agent.jar). Then use SSH command to call the jar file from agent (aka slave) machine. You may add some java argument for troubleshooting. Below is an example configuration for the node.

I shall also note that if the remote agent is a different operating system where the path of bash might be different, then you need to include the directory of bash executable in PATH environment variable. That can be done as in the screenshot above (PATH=${PATH:/usr/bin}). If this is incorrect, you might run into issues when running sh step in Jenkins pipeline. Here is an article about this. To translate that page, the symptom of this issue includes:

Error from pipeline execution that says the following, which is very generic:

process apparently never started in /home/dhunch/jenkins/workspace/site-remote-job@tmp/durable-b997d26c 
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

Job status shows:

hudson.AbortException: script returned exit code -2
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:659)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:605)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:549)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Threaddump (only available during execution) shows:

Thread #6
 at DSL.sh(awaiting process completion in /home/dhunch/jenkins/workspace/site-remote-job/durable-b997d26c; recurrence period: 9543ms; check task scheduled; cancelled? false done? false)
 at WorkflowScript.run(WorkflowScript:9)
 at DSL.script(Native Method)

This issue is tricky because none of the symptoms above make you think about the environment variable defined for the agent, and that the issue does not occur until you actually execute a Jenkins pipeline, usually well after the node agent is registered, and only impacts shell step (sh). It is recommended to test this with sh steps in pipeline.

Configure Repository

Our example job executes service script pulled from SCM, on a remote agent. Then the master pulls the result file to itself. The git repository can be set in Jenkins job, where you specify that the script named Jenkinsfile (from the repo) is the pipeline file that needs to be executed.

This is fairly simple, but what makes the situation more complex is the following few requirements:

it’s the agent node that needs git (installed) and clone to repo;
the master does not need (and should not attempt to) clone from repo;
the master needs to do its job without having to pull SCM;

We need to be able to specify whether each step needs to pull from SCM. The following pipeline syntax shows how this is done:

def agent_dir = 'initial_value'
pipeline {
 agent none
 options { skipDefaultCheckout(true) }
 stages {
 stage('Execute Job') {
 agent {
 label 'remote-agent-customer1'
 }
 options { skipDefaultCheckout(false) }
 steps {
 echo 'Executing job on node'
 sh 'whoami && pwd'
 script {
 agent_dir = sh(returnStdout: true, script: 'echo -n ${WORKSPACE}')
 // the variable should not include carriage return
 }
 sh "echo ${agent_dir}"
 }
 }
 stage('Pull Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 echo 'Pulling job below'
 sh "echo ${agent_dir}"
 sh "scp dhunch@site1:\"${agent_dir}\"/result.csv ./"
 }
 }
 }
}

In this example pipeline file, we declare agent none for the pipeline, then an agent for each specific stage. we also specify the option skipDefaultCheckout as true at the step where pulling from SCM is not needed. This allows us to finish job with multiple agent, and only pull from SCM as needed. This snippet also exemplifies how to declare a variable, assign it from stdout from one agent, and persist the value across ensuing stages.

Store Result

The reason we run pipeline jobs on this remote agent is because it is sitting in customer network and has local direct access to data. So this is perfect for situation such as data analytical jobs, which access to database on local network and store result. We need to pull the result file back to agent and make it available on Jenkins. This is so similar to “archive artifact” task (commonly seen in CI process) that we can simply use its plugin to achieve what we need. Before archive artifact, we need to pull it to local (master), as shown in the example code above. After that, we need another stage to archive the result.

 stage('Archive Result') {
 agent {
 label 'master'
 }
 options { skipDefaultCheckout(true) } // no need to pull scm to this agent
 steps {
 archiveArtifacts artifacts: '*.csv',onlyIfSuccessful: true,fingerprint: true
 }
 }

I believe there is plugins to compress artifacts as well.

Conclusion

At the end I’d like to reiterate my perception about Jenkins. It is a very generic and adaptive automation platform that originally evolved from use cases in build automation. Due to this original root, many components in Jenkins are named around Continuous Integration use cases, such as the “build” button, and the “archiveArtifacts” step. These misnomers underplays what Jenkins can potentially do in continuous deployment or other automation scenarios. It is important for automation engineers to understand Jenkins components and plugins, through their functionalities and not by the name, and therefore make creative use of Jenkins as automation engine in all scenarios.

An alternative to this proposed pipeline would be Ansible Tower, a commercial project based on open-source Ansible, but with nice UI support. Ansible Tower is Ansible oriented, and it does not have everything that Jenkins can do. It should still be a decent alternative given the proposes pipeline uses Ansible a lot.

Automated Deployment Pipeline 2 of 3

Tue, 06 Oct 2020 22:05:00 -0400

In this posting, we continue to discuss Jenkins’ ability to automate deployment routines. Jenkins supports freestyle project out of the box, as well as Pipeline with several plugins. Freestyle project allows user to specify multiple steps on UI. This does not scale well when your entire process involves many steps. As explained on Jenkins’ website:

While standard Jenkins “freestyle” jobs support simple continuous integration by allowing you to define sequential tasks in an application lifecycle, they do not create a persistent record of execution, enable one script to address all the steps in a complex workflow, or confer the other advantages of pipelines.
In contrast to freestyle jobs, pipelines enable you to define the whole application lifecycle. Pipeline functionality helps Jenkins to support continuous delivery (CD). The Pipeline plugin was built with requirements for a flexible, extensible, and script-based CD workflow capability in mind.

So although freestyle projects are easy to set up, and can do technically everything that Jenkins pipeline can do, the major advantage of Jenkins Pipeline is the ability to manage multiple-step as code, and version control the pipeline-as-code. Here is some more information.

We will start with a freestyle project to understand Jenkins’ ability and then advance to building pipelines.

Freestyle projects

As mentioned, Jenkins calls a task a “build”, and the build can be triggered in a variety of ways:

Time schedule (with or without parameter)
Remotely via API
On completion of other projects
Poll SCM for changes
Commit to SCM

There are also several ways to execute a job:

Execute Shell command, batch command or groovy script
Invoke Ansible adhoc command, playbook or vault
Conditional on specified boolean value, file existence, etc

After the job one can specify post-build jobs, for example:

start other build projects
notification of various means
publishing result file

In deployment, it is common task to execute a task over SSH, using SSH command or invoke Ansible command with Ansible plugin. The former fits simple command line tasks. For example:

The Ansible plugin is good for more steps and more complicated inventory hierarchies. This post includes an example of an inventory involving multiple layers. The other limitation with SSH command is lack of a straightforward configuration to escalate privilege and run remote command. On the other hand, Ansible addressed this with become method. Below is a screenshot

In order to use Ansible command, you will also need to specify where the Ansible binary in Global Tool Configuration. You will also need to store vault credential in Jenkins credential store so it’s not being prompted.

Jenkins Pipeline

Jenkins pipeline allows one to describe actions in a pipeline in groovy Domain Specific Language (DSL). There are two styles of pipelines:

Declarative pipeline is identified by a block named ‘pipeline’, it is relatively new and supports the pipeline-as-code concept. It can be stored as Jenkinsfile in code repository or edited in Jenkins’ UI.
Scripted pipeline is identified by a block named ‘node’, it is the conventional format and can only be edited in Jenkins’ UI.

To fully understand the two styles of pipelines you also need to know declarative programming and imperative programming. Read this instruction for a better explanation. Here is another good one that focus on their differences.

Jenkins’ plugin provides UI components for user to input information (e.g. Invoke Ansible Command), now with Jenkins pipelines, plugins can provide step functions in order to help Jenkins user. This page from Jenkins lists the most common ones, many of which requires plugin installation.

One example of step function is SSH Pipeline Steps. It allows one to issue SSH connection from Jenkinsfile. However, in our deployment scenarios, it has some limitations:

Unable to match a group of host by specific pattern
Cannot use host name aliases
SSH tunneling is not supported
No means of privilege escalation

Out of these 1 through 3 are due to the fact that SSH step function does not use the OpenSSH configuration on the machine. The jenkins-ssh-slaves plugin uses trilead SSH2 implementation written in Java. Only OpenSSH implementation uses ~/.ssh/config file. This creates problem whenever SSH tunneling is needed. For example, to register a remote slave node via SSH tunneling, we need to select “Launch agent via execution of command on the master” instead of “Launch agent via SSH” as launch method. Here is an instruction and below is what it looks like:

Similarly, if we use ssh-agent plugin as tool for deployment, we cannot use any configuration made by OpenSSH. Therefore Ansible in Jenkins Pipeline is a better tool for deployment because it can use OpenSSH.

Pipelines with Ansible

In this section I demonstrate the use of Ansible playbook and adhoc command in pipeline through two examples.

In example 1, the job pulls authorized key file from SCM, and then use a playbook from SCM, to push the key file to all servers in the specified inventory.

Here is the playbook file for example 1:

---
# Example:
# ansible-playbook -l all -i ~/ansible/inventories/bh.yml push-key.yml --ask-vault-pass
- name: push key to target
 hosts: all
 tasks:
 - name: sync key to host
 copy:
 src: "{{authorized_keys_src}}"
 dest: "/home/dhunch/.ssh/authorized_keys"
 force: yes
 mode: 0600

Here is the pipeline script (note ampersand is mistakenly displayed as & in the box below):

import java.net.URLEncoder;
 
pipeline {
 agent any
 
 options {
 skipDefaultCheckout(true)
 }
 environment {
 BITBUCKET_CREDS = credentials('bitbucket') 
 //BITBUCKET_CREDS_USR and BITBUCKET_CREDS_PSW are set
 BITBUCKET_CREDS_USR = "${BITBUCKET_CREDS_USR}"
 BITBUCKET_CREDS_PSW = URLEncoder.encode("${BITBUCKET_CREDS_PSW}", "UTF-8") 
 // if password contains special character we need to url encode it. e.g. @->%40
 }
 stages {
 stage('Start') {
 steps {
 echo "Starting pipeline ..."
 deleteDir()
 }
 }
 stage('Prep') {
 steps{
 wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: [[password: "${BITBUCKET_CREDS_PSW}", var: 'RANDOM']]]) {
 // MaskPasswordsBuildWrapper requires Mask Passwords Plugin and is to mask specific string in console output. 
 // Otherwise BITBUCKET_CREDS_PSW will display in the clear
 sh "git init && git config core.sparsecheckout true"
 sh "git remote add origin https://$BITBUCKET_CREDS_USR:${BITBUCKET_CREDS_PSW}@bitbucket.org/vendorcompoany/configmanagerepo.git"
 sh "echo 'public_keys/*' >> .git/info/sparse-checkout"
 echo "Downloading key file"
 sh "git pull --depth=1 origin master" 
 }
 }
 }
 
 stage('Deploy to Site 1') {
 steps {
 echo '> Deploying to Site 1 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site1.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 1 ...'
 }
 }
 
 stage('Deploy to Site 2') {
 steps {
 echo '> Deploying to Site 2 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site2.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 2 ...'
 }
 }
 
 stage('Deploy to Site 3') {
 steps {
 echo '> Deploying to Site 3 ...'
 ansiblePlaybook (
 installation: 'Ansible on Mac',
 playbook: '${WORKSPACE}/public_keys/push-key.yml',
 inventory: '~/ansible/inventories/site3.yml',
 vaultCredentialsId: 'ansible-vault-pass',
 extraVars: [authorized_keys_src: "$WORKSPACE/public_keys/authorized_keys",]
 )
 echo '> Deployed to Site 3 ...'
 }
 }
 }
 post {
 always {
 cleanWs()
 }
 }
}

In this example, we pull a sub-directory from git repo. We use two tricks to minimize amount of traffic. First, we use git pull with depth=1 so only the required recent commits are pulled, not the entire history. Second we use sparse checkout to get result from a sub-directory, not the entire repo.

When connecting to repo, password is required. We pull the credential from Jenkins’ credential store and they are masked by default. However, the password must be converted to URL string when used in git remote add origin. Otherwise if the password contains special character the URL will not work.

This modification to password brings about another challenge, password masking during the job execution. We will have to build our own wrapper function using class MaskPasswordsBuildWrapper in order to mask any variable.

In example 2, we wrap an Ansible step function to call adhoc command to check version on all hosts. Here is the pipeline script:

def HunchVersionQuery(pattern, siteinventory){
 step([
 $class: 'AnsibleAdHocCommandBuilder',
 ansibleName: 'Ansible on Mac',
 inventory: [$class: 'InventoryPath',path: siteinventory],
 hostPattern: pattern,
 module: 'shell',
 command: 'cat /etc/*release',
 forks: 1,
 vaultCredentialsId: 'ansible-vault-pass'
 ]);
}

pipeline {
 agent any

 stages {
 stage('Start') {
 steps {
 echo 'Staring Pipeline'
 }
 }
 stage('Query Site1') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site1.yml')
 }
 }
 stage('Query Site2') {
 steps { HunchVersionQuery('*app','~/ansible/inventories/site2.yml')
 }
 }
 stage('Query Site3') {
 steps {
HunchVersionQuery('*app','~/ansible/inventories/site3.yml')
 }
 }
 }
}

Because Ansible plugin does not provide a warpper function for adhoc command step, we will have to build our own wrapper function in the above code.

Automated Deployment Pipeline 1 of 3

Wed, 30 Sep 2020 22:04:00 -0400

The business case

You launched a software application. You installed it on two customer sites. You support the application mostly by SSH to customer server and run Bash commands, or slightly better, Bash scripts.

The product is a hit to the market. You hired 20 support specialists in a customer service department. The dream client came through: an enterprise giving you a fleet of 100 servers to deploy your application on.

More staff, more business, more installations, more incidents, but the same old command driven steps. Problems:

Non-standard support procedures. Every one takes notes and everyone’s notes are slightly different.
Information sharing among team members are ad hoc, and at high level.
Post-mortem discussion is driven by memory and command fragments, instead of evidence end-to-end

If that looks like your organization, chances are you also suffer from some secondary damages over the long term, such as:

Downtime resolutions rely on the knowledgeable few
Documentation helps. But it never catches up to the latest version of application
Lack of auditing of commands during support

I propose an automation scheme to existing support and deployment practice. This automation scheme combines a suite of common technologies, such as Bash, Python, Ansible, OpenSSH and Jenkins. The automation allows the department to, either fully or partially, operationalize the steps in support and deployment, and eventually shift towards agile practice.

Bash, Python and Ansible

Bash script is based on shell command, perfect for running critical system tasks such as volume management. When it turns into a script, it can be cumbersome, especially with complex data structure. Python, as a tool for system administration, is a good complement to that.

Python 2 comes with most Linux distributions, and is also a dependency of other built-in tools such as YUM. Python3 can be installed easily from default YUM repositories. Both Python2 and Python3 can exist on the same operating system, although new module development are now shifted to Python3. Python’s syntax is very simple and offers object-oriented programming ability. Moreover, there is an entire open-source community behind Python, which offers modules in every aspect of IT (for example, Datastax has a driver module for connecting to Cassandra). Those modules are installed with PIP tool, or PIP3 for python3.

Both bash and Python executes on local machine. To run them on remote servers over SSH. You want to have a list of target hosts, and specify which one to execute the script against. This is where Ansible comes in handy. Ansible is superior in the following aspects:

Free and open-source, with commercial alternative (Towers);
Inventory management (inventory);
Desire state engine (roles)

Ansible is built on Python and is agent-less. Connectivity to remote host is done via secure shell so it can take advantage of existing SSH configurations. Job execution on the target machine is done through Python. With Python you can also develop custom module in Ansible. For some use cases in customer support with Ansible, refer to my two previous postings about Ansible at scale.

Jenkins

The tools above forms a package for automation. The issue is that all of them are command-line based. Any task that requires Ansible requires the IT professional craft up long command, such as running playbook, executing a role, or ad-hoc command. This is inconvenient when a task needs to be done during an incident. Such tasks also require trained professional with the relevant skills.

These tasks can be stored in, or initiated by Jenkins. Although Jenkins is well known for build automation in continuous integration, it is automation engine for any command-line based IT tasks. The button to start such tasks in Jenkins UI is called “Build”, which is also a misnomer that underplays Jenkins’ versatility: building application from source code is just one of the many IT tasks that involves multiple long running commands. In this and next article we introduce Jenkins as an engine for deployment automation.

The infrastructure architecture is diagramed as above, and with the connection across Internet, the target hosts must be hardened properly in the following aspects:

Connectivity to remote host is via SSH chaining, through an SSH proxy;
Root login must be disabled for remote session or by password;
Service user may be shared, but must be authenticated by individual RSA key pair;
Service user connected remotely needs to escalate privilege by su if needed;

I want to make a theoretical distinction between our topic here and continuous deployment. We simply focus on the technical side of deployment automation. Essentially automating a few bash scripts. On the other hand, a continuous deployment process is an extension to an existing continuous integration pipeline, with the vision to streamline the process end-to-end from code commit to production rollout. Implementing CI/CD pipelines should be approached as an organizational program rather than an individual technical initiative. Here is a good technical overview on CI/CD pipeline with Jenkins and Ansible.

Security

The security mechanism of this system is based on OpenSSH because the connectivity between servers are through SSH chaining. RSA key authentication must be used in order to encrypt traffic with password-less login. Connection to an SSH host can be done through a proxy server. Below is an example of SSH configuration:

Include customer1.config
Include customer2.config

Host *
 IdentityFile ~/.ssh/id_rsa
 ServerAliveInterval 60
 ServerAliveCountMax 3
 Compression yes
 ControlPersist 3h
 ControlPath ~/.ssh/sockets/%r@%h-%p

Host gateway
 Hostname support.digihunch.com
 User jdoe
 Port 2223

Host customer-server-0
 Hostname 192.168.201.12
 User support
 ProxyCommand ssh -W %h:%p gateway

Open SSH configuration file (~/.ssh/config) needs to be configured properly with useful host names and aliases. To prevent the config files from growing too long, include statement can be used to reference other configuration file (available with OpenSSH v7.3 sp1 and up).

The host names (as well as aliases) listed in SSH configuration can be directly referenced in Ansible inventory, allowing Ansible (and Jenkins) to reference site by alias and connect to target host through proxy.

Plugins

Jenkins has a community that develops a variety of plugins, which makes Jenkins the most powerful automation platform. Here are some examples of useful plugins:

Audit Trail: output job execution history to file or Elasticsearch;
Credentials: stores credentials in Jenkins;
Pipeline: build declarative (new) or scripted (old) pipeline for Jenkins jobs;
Simple Theme: just a theme but allows console output to be dark (using CSS);
Job Configuration History: job configuration audit;
Mask password: mask variables (including password) from console output
Ansible: invokes ad-hoc commands and playbooks
SSH agent, SSH pipeline steps, SSH credentials: features related in SSH in Jenkins pipelines.
Purge Job History: purge all of build history, or purge by time and number of old builds.
Parameterized Scheduler: schedule to run a job and provide parameter
Workspace cleanup: clean up workspace when invoked.

In the next article, we will go over some common job configurations.

Spark, Cassandra and Python

Tue, 15 Sep 2020 16:24:09 -0400

In this post we touch briefly on Apache Spark as a cluster computing framework that supports a number of drivers to pipe data in, and that its stunning performance thanks much to resilient distributed dataset (RDD) as its architectural foundation. In this hands-on guide, we expand on how to configure Spark, and use Python to connect to Cassandra data source.

Spark supports Sala, Java and Python shells. I’m not familiar with Scala but I have had Python background and know it’s importance in big data processing. One key data structure with big data processing in Python is Pandas data frame. Spark has the ability to map its own data frame to Pandas data frame.

Spark also needs a third party connector to connect to Cassandra. This connector is provided by Datastax in this open-source project called spark-cassandra-connector. The Github page includes a README with compatibility matrix, which is very important to understand before any configuration works. However, the Github is only the source code repository for anyone to build the project themselves. An alternative source of the dependency is this page from Maven repository. When running Spark we can simply reference that page URL as dependency.

Suppose we install spark onto CentOS, we download and unzip this package to somewhere such as user directory (~). Assuming we already have Open JDK 1.8 installed, when we run spark binary, it places cache and jar files in ~/.ivy2, potentially we need to manually move the following dependencies to ~/.ivy2/jars:

org.codehaus.groovy_groovy-json-2.5.7.jar
com.github.jnr_jffi-1.2.19.jar
org.codehaus.groovy_groovy-2.5.7.jar

These jar files are available for download from Maven’s repository as well if you wish provide them as package dependencies. We have two flavours of interactive shells to connect to Spark: the Scala shell (spark-shell) and python shell (PySpark)

Scala Shell

We can enter the default scala shell by

$ ./bin/spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 --conf spark.cassandra.connection.host=10.10.10.151 --verbose

During the start, note a stdout line that says:

Spark context Web UI available at http://spark-host:4040

Then we can open that tcp port on iptables and view that job in browser. From within scala shell we can test connectivity to Cassandra with the following commands:

>>> val new_exam = spark.read.format("org.apache.spark.sql.cassandra").options(Map("table" -> "new_exam","keyspace" -> "examarchive")).load()

Python Shell

Python Shell (aka PySpark) brings Python shell which is known to many engineers from system admin or development background. By default, python 2 will be used. To specify python version, set some environment variables before we start pyspark with cassandra connector package specified:

$ export PYSPARK_PYTHON=python3
$ export PYSPARK_DRIVER_PYTHON=python3
$ export SPARK_HOME=/home/dhunch/spark-2.4.6-bin-hadoop2.7
$ export PATH=$SPARK_HOME/bin:$PATH
$ ./bin/pyspark --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 --conf spark.cassandra.connection.host=10.10.10.151

Once you’re in the interactive shell, you can start with loading required python libraries, and test your connectivity:

>>> from pyspark import SparkContext, SparkConf
>>> from pyspark.sql import SQLContext

>>> load_options = { "table": "new_exam", "keyspace": "examarchive"}
>>> df=spark.read.format("org.apache.spark.sql.cassandra").options(**load_options).load()
>>> df.show()
>>> df.write.csv('/tmp/mycsv.csv')

>>> #df.registerTempTable("ne")
>>> df.createTempView("ne")

>>> tw1=sqlContext.sql("select count(*) from ne")
>>> tw1.show()

>>> qrdf2=sqlContext.sql("select study_key, image_count from ne where current_exam_version=exam_version")
>>> qrdf2.write.csv('/tmp/tw2')

Note that the load method returns type pyspark.sql.dataframe.DataFrame, which is already a distributed data structure. So there is no need to parallelize it with parallelize() method. As of Spark 2.0, we are supposed to use createTempView() method instead of the old registerTempTables() method. Read this for further information.

Python Application

With interactive shell you run one or several commands at a time. We can build a python script and submit the whole script as an application. This is an example command:

./bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 sample.py

Note that the sample.py script name must be provided after –packages switch. Otherwise, you will get an error saying missing dependency (Failed to find data source: org.apache.spark.sql.cassandra). In the script, we can manipulate the data from Cassandra with greater flexibility. For example, we can map one field to several fields. For example, if one of the fields stores an XML document, the script can drill down the XML tree structure parse out values at different levels of child nodes, into separate data base columns. Here is an example of python script where we register a custom UDF declared in python and apply it to some existing columns to build new columns:

#! /usr/bin/python3
# To submit this script as an application to spark:
# ./bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 examstat.py
# Note that the script name must be placed after --packages 

import sys,datetime,re
import xml.etree.ElementTree as ET
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, SparkSession
from pyspark.sql.functions import udf 
from pyspark.sql.types import StringType,StructType,StructField

cluster_seeds=['dest_cass_host']

def pTrimExamCode(raw_code):
 return 'NULL' if raw_code is None or raw_code=='None' else str(raw_code).replace(',','').rstrip('\r\n')

def is_valid_date(date_str):
 isValidDate=bool(re.match("^(19|20)\d\d(0[1-9]|1[012])(0[1-9]|[12][0-9]|3[01])$",date_str))
 if isValidDate:
 try:
 datetime.datetime(int(date_str[:4]),int(date_str[4:6]),int(date_str[6:8]))
 except ValueError:
 isValidDate=False
 return isValidDate

def pPullTags(study_key,raw_xml_field):
 ns={"vc":"http://medical.nema.org/mint"}

 StudyDateTag='None'
 StudyDescriptionTag='None'
 try:
 if raw_xml_field is not None:
 summary_tree=ET.fromstring(str(raw_xml_field)) # str function outputs 'None' or null object
 xml_find_res=summary_tree.find("vc:attributes/vc:attr[@tag='00080020']",ns)
 if xml_find_res is not None: StudyDateTag=str(xml_find_res.attrib.get('val'))
 
xml_find_res=summary_tree.find("vc:attributes/vc:attr[@tag='00081030']",ns)
 if xml_find_res is not None: StudyDescriptionTag=str(xml_find_res.attrib.get('val'))
 except:
 print("----------------------->>>>>>>>> examstat: error parsing metadta for study_key "+study_key)
 
 return (StudyDateTag,StudyDescriptionTag)

# custom StructType for the output tuple
XMLExtractType=StructType([
 StructField("StudyDate",StringType(),False),
 StructField("StudyDescription",StringType(),False)])

if __name__ == "__main__":
 sparkSession=SparkSession.builder \
 .appName('examstat') \
 .config('spark.cassandra.connection.host',','.join(cluster_seeds)) \
 .master('local[*]') \
 .getOrCreate()
 load_options = {"table": "new_exam", "keyspace": "examarchive"}
 sqlContext=SQLContext(sparkSession)

 # pyspark.sql.dataframe.DataFrame is already a distributed data structure. No need to parallelize it.
 df0=sqlContext.read.format('org.apache.spark.sql.cassandra').options(**load_options).load()
 df0.createTempView("new_exam")

 # pyspark.sql.functions.udf(python function,output type)
 sparkSession.udf.register("uTrimExamCode",udf(pTrimExamCode,StringType()))
 sparkSession.udf.register("uPullTags",udf(pPullTags,XMLExtractType))

 # use custom UDFs uTrimExamCode and uPullTags to calculate new columns and remove dups and deleted studies
 df1=sqlContext.sql("select study_key as StudyKey,uTrimExamCode(exam_id) as ExamCode,image_count as ImgCnt,Total_pixel_data_size as PixelSize, uPullTags(study_key,metadata_summary) as XMLExtract, metadata_summary from new_exam where exam_version=current_exam_version and is_deleted=False")
 df1.createTempView("uniq_study")
 
 # map the four fields in XMLExtract to separate columns. we take this as separate step as we don't want uPullTags to execute multiple times in previous step 
 df2=sqlContext.sql("select StudyKey,ExamCode,ImgCnt,PixelSize,XMLExtract.StudyDate as StudyDate,XMLExtract.StudyDescription as StudyDescription from uniq_study")
 df2.createTempView("uniq_study_stat")

 # Run analytical query
 df3=sqlContext.sql("SELECT ExamCode, round(avg(PixelSize)/1024/1024) as avg_size_mb, round(sum(PixelSize)/1024/1024/1024,2) as total_size_gb,count(StudyKey) as study_count FROM uniq_study_stat GROUP BY ExamCode order by study_count desc")
 #data frames are lazily loaded and processing not started until the following call
 df3.write.csv('/tmp/examstat_'+datetime.datetime.now().strftime("%m%d%H%M%S"))

It is important to understand the concept of lazy evaluation in Spark RDD here. The execution of function to RDD does not start until an action is triggered (eg. show method, or write method). Spark maintains the record of which operation is being called through DAG (directed acyclic graph). Such record is referred to as a transformation. We need to understand whether each RDD method is a transformation, or an action so we know whether it will be lazily evaluated (here’s more information).

This is a major difference between Apache Spark and Hadoop MapReduce. With MapReduce, developer spend a lot of time in minimizing the number of MapReduce passes. It happens by clubbing the operations together.

Host legacy application in Docker 1 of 2

Fri, 04 Sep 2020 16:24:00 -0400

This is my notes from containerizing a legacy application with Docker compose. We have to run multiple instances of our application because we’re unable to secure additional VMs for this single-VM education environment. The application is target of containerization, because it requires mass reconfiguration (around TCP port) to run multiple instances of the application. We want to use the same application configuration file for multiple containers, and map the TCP port to different groups of ports on the host, leveraging port mapping in Docker. On the other hand, the auxiliary services are not being containerized, such as Cassandra database and ElasticSearch because they can be shared for multiple application instances. In other words, we use Docker to isolate processes of the same application.

Prepare environment

The CentOS server needs to have docker-ce (through YUM) as well as docker-compose (direct download). They can be installed this way:

$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
$ sudo yum install docker-ce docker-ce-cli containerd.io

$ curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo systemctl start docker

Our Docker registry is not publicly available. So we need to port the Docker image we need to remote server and load it into the local registry. We first examine the registry locally:

$ curl -XGET https://admin:password@docker.digihunch.com/v2/dhunch/tags/list | python -m json.tool

Once we identify the image, we export it to a tar file:

$ docker save docker.digihunch.com/dhunch > dhunch_image.tar

SCP the file to remote server and load it locally:

$ docker load -i /home/dhunch/dhunch_image.tar
$ docker image ls

We need to distinguish these commands:

docker save: saves an (non-running) image with all layers to file
docker export: saves a running or paused container to file
docker import: import the contents from a tarball to create a filesystem image, most used with docker export
docker load: load an image from a tar archive or STDIN, most used with docker save

Build docker-compose file

I need to cater to the customer environment with a newly create docker-compose file. The customer environment includes specific storage and networking configurations. Docker compose’s official documentation is here. We repeat the following commands for our troubleshooting:

$ docker-compose up -d
$ docker-compose exec -it dhunch1 bash
$ docker container ls

Once we start the container, the status might go unhealthy after it starts. The documentation explains two reasons you’re seeing an unhealthy container:

a single run of the command takes longer than the specified timeout
health check fails; the health check command will retry a number of times before it declares the container as unhealthy.

In our case, It is most likely because it does not pass a built-in health check mechanism. We need to understand where the health check was defined. There are four ways to enable health check:

Dockerfile instruction when building the image
Docker run command
Docker-compose or docker stack yaml file
Docker service

With #1, unfortunately, you can’t reverse engineer an image and view the Dockerfile that were used to built it and review the health check statement. What you can do is check docker events, or inspect the container, and go to the log files as specified under logPath section in the inspection result and look for HealthCheck section. We determined it is the case, then we can disable, or override the built-in healthcheck command from image, with a statement in docker compose.

For network interface, docker compose also allows us to specify MAC address for each container with mac_address keyword (for license key). MAC address generator are available on the internet.

EntryPoint vs CMD

The difference between EntryPoint and CMD is very important when launching container. Some literature also mentions RUN, which is only used when building a new layer of images so it is not relevant here (in the context of launching a container from image). EntryPoint and CMD has similar functionalities both allowing you to specify a command to run. The difference is whether they can be overwritten by command line arguments that user provide to docker-compose or docker run in an ad-hoc manner. As their names suggests, EntryPOINT means what is specified under it must be executed as it launches into the container, regardless of any adhoc commands. On the other hand, CMD is just an entry to save users from typing in a command every time they run docker compose or docker run. Should user prefer a different command, it can be provided as an explicit argument and it will be respected overwriting the pre-defined CMDentry in Dockerfile or command entry in docker-compose.yml.

Both CMD and EntryPoint supports shell and exec forms. More details here.

Choice of Networking

With single-host deployment, the containerized application needs to communicate with other existing, non-containerized service on host, such as database or elastic search. If docker uses host network, the container shares interface with the host and it does not have its own IP address. Host network removes isolation between container and host. This allows container to run the application that was licensed to the host based on MAC address. There is also no port mapping from container to host network. Container simply uses port on host, and is subject to the availability of TCP/UDP port on host.

We will have to use bridge network here. We can force MAC address the app container, and pre-generate license. For container to communicate with a service on host, through bridge network, there are two problems to address:

Container knows the IP of the host (layer-3 connectivity, ping);
Making host service available to container (layer-4 connectivity, telnet);

Docker creates its own interface for bridge network. If it’s an unnamed network, i.e. not explicitly declared under networks section in docker compose, then interface docker0 is used. If it’s a named network, then an interface name starting with br- is used.

The first problem is easier to address, we simply needs to IP address of the host on the interface. We can validate by pinging from container to host. Docker can also use host.docker.internal to reference the host. Unfortunately, this stopped working for linux since 18.09.3.

It is reportedly to be fixed in 20.04 and until it is available, we may add it to manual dns. The following command outputs the entry to add to /etc/hosts in container.

# ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}'

To do this automatically in docker compose, we need some tricks:

Store the Host IP in host environment variable ( use an export command)
Use compose to pass host environment variable to container environment variable
Have the container write its environment variable to /etc/hosts

The compose file will contain a line like this:

services:
 myenv1:
 image: alpine
 command: >
 sh -c "apk update &&
 echo $$HostDNSLine >> /etc/hosts &&
 bash"
 #network_mode: bridge
 environment:
 - HostDNSLine=${HOSTDNSREC}

Note ampersand might be mistakenly displayed as & in the above. Then we run it with the following:

# export HOSTDNSREC=$(echo 1.2.3.4 host.docker.internal) && docker-compose up

The second problem is harder to address because the service on host may not bind to docker’s interface. Some services such as ssh bind to all interfaces on host and you can telnet to port 22 with any IP address the host is associated with. This is however not the case for most other services, such as Cassandra or Elastic Search. They typically only bind to main interface, such as ens192, or eth0, and not to the docker interface. In order to make the service available to container, we either need to bind these services to the docker interface, or use iptables rules as an alternative.

Suppose it is a named network and Docker’s interface name is br-90ae024d5324, and the service on host listens to port 9042, we will need the following two commands from host:

# sysctl -w net.ipv4.conf.br-90ae024d5324.route_localnet=1

# iptables -t nat -A PREROUTING -p tcp -i br-90ae024d5324 --dport 9042 -j DNAT --to-destination 127.0.0.1:9042

Note that docker compose can configure to run sysctl in container but not from host. If there are multiple ports, we can turn this into a shell script:

#!/bin/bash
tcp_port_list="9200 9042 8302 8303 8304 8305 8306"

if_name=$(basename -a /sys/class/net/* | grep ^br- | head -1)
echo enable route localnet on interface $if_name
sysctl -w net.ipv4.conf.$if_name.route_localnet=1
for tcp_port in $tcp_port_list; do
 echo open host tcp port $tcp_port to interface $if_name
 iptables -t nat -A PREROUTING -p tcp -i $if_name --dport $tcp_port -j DNAT --to-destination 127.0.0.1:$tcp_port
done

echo $(ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po 'inet \K[\d.]+' | awk '{print $1 " host.docker.internal"}')

On the other hand, binding service to multiple interfaces usually require some re-configuration on the service itself. For example, if it is Elastic Search, we need to update [network.host] entry in elasticsearch.yml to include multiple IP addresses. For Cassandra, we need to update rpc_address to 0.0.0.0 or set rpc_interface in cassandra.yml.

Integration with storage

The application in the container need to store files to storage available to host, whether it is an NFS share or a block disk. We can use volume mapping with Docker compose, to map a path in container to a path presented to host as persistent volume. At this step, we might run into permission issues.

By default, containers initializes as root (uid=1) within the container, and the entrypoint script launches application as root. When application writes to persistent volume, files are written as root user. In the legacy non-container setup, we expect the application to write file as dhunch user. Moreover, NFS volume will not allow writing files as root (if the server has root squash configured). To address this, there are two approaches:

launch container as a regular user
launch container as root user, then have the entrypoint script launch application as regular user (dhunch)

For approach 1, we need to tell Docker to launch container as a regular user by specify the uid and gid for container to run application. We can specify the following envrionment variable in the compose yaml:

user: ${CURRENT_UID}

Then we assign the environment variable before running docker-compose:

# export CURRENT_UID=$(id -u dhunch):$(id -g dhunch) &amp;&amp; docker-compose up

This allows container to initialize as the regular user. However, if the entry point script needs to perform activities that requires root permission within the container, it will fail. For example, a regular user in container will not be able to update /etc/hosts;

With approach 2, we do not specify user in docker compose so container initializes as root. Then the entry point script launches application as regular user. For example, use su command before launch Java:

su dhunch -c "
exec java \
 -Xms512M -Xmx8192M \
 -Djava.io.tmpdir=$APP_HOME/var/tmp \
 -server \
 -XX:CompileCommandFile=$APP_HOME/etc/hotspot_compiler \
 -jar $APP_HOME/lib/jar/jruby-complete-*.jar \
 --1.9 \
 $APP_HOME/lib/rubybin/runapp.rb
"

Before doing this, we need to first create user dhunch within container, and the uid and gid must match those of the host. So that when container picks up dhunch user, it converts it to the correct uid.

groupadd -g 1011 dhunch
useradd -m -c 'regular user' -u 1011 -g 1011 dhunch

To further understand how uid and gid work, here are two posts with more information.

This user ownership setup will also work for NFS. To configure NFS, we need some extra client-side configurations in the container, as well as a special volume driver for NFS. Refer to this post.

Java Garbage Collection

Fri, 07 Aug 2020 23:19:17 -0400

Tuning the garbage collector is the most important thing that can be done to improve the performance of a Java application. GC is typically caused when the JVM decides GC is necessary, specifically when:

a minor GC will be triggered when the new generation is full;
a full GC will be triggered when the old generation is full;
a concurrent GC (if applicable) will be triggered when the heap starts to fill up

OpenJDK has three collectors suitable for production, with different performance characteristics. In order to study the GC behaviours in application, it is important to turn on GC logging. The detailed step is different in JDK 8 and JDK 11 (read about java version here).

Java developers don’t need to manage life cycle of objects explicitly as the JVM automatically fress the object. In order to track objects that are still in use, it is insufficient to count references to objects. Instead, the JVM must periodically search the heap for unused objects. Once it finds unused objects, the JVM frees the memory occupied by those objects. It also needs to compact the memory to prevent memory fragmentation. The performance of GC is dominated by these basic operations (finding unused objects; freeing up their memory; compacting the heap), no matter which collector is used. Some algorithms delay compaction until absolutely necessary, some compact entire sections of the heap at a time, and some compact the heap by relocating small amounts of memory at a time. These different approaches are why different algorithms have different performance characteristics.

Java programs are typically heavily multithreaded, and the garbage collector itself often runs multiple threads too. We refer to the application logic threads as mutator threads, since they are mutating objects as part of the application logic. When GC threads track object references or move objects around in memory, they must make sure application threads are not using those objects on the move. This introduces a pause when all application threads are stopped (known as stop-the-world pauses), which generally has the greatest impact on the performance of an application. Minimizing those pauses is one important consideration when tuning GC.

Garbage collectors are generational

Most garbage collectors work by splitting the heap into generations. These are called the old (or tenured) generation, and the young generation, which is further divided into sections known as eden and survivor spaces, with eden taking up the vast majority of the young generation. The rationale for having separate generations is that many objects are used for a very short period of time in the real life of application programming. Garbage collector is designed to take advantage of this. Objects are first allocated in the young generation, which is a subset of the entire heap. When the young generation fills up, the garbage collector will stop all the application threads and empty out the young generation. Objects that are no longer in use are discarded, and objects that are still in use are moved elsewhere. This operation is called a minor GC or a young GC. Common GC algorithms have stop-the-world pauses during collection of the young generation.

Heap Generation

This design has two performance advantages. First, Cleaning up young generation as a only a portion of the entire heap causes shorter pause than cleaning up the entire heap. Second, by moving used objects to survivor spaces or old generation, and discarding unused objects, compatction is achieved.

With used objects moved to the old generation, eventually it woo will fill up, and the JVM will needt o find any objects within the old generation that are no longer in use to discard. This is where GC algorithms have their biggest differences. The simpler alghorithms stop all application threads, find the unused objects, free their memory, and then compact the heap. This process is called a full GC, and it generally causes a relatively long pause for the application threads.

On the other hand, sophisticated alghrithms are able to find unused objects while application threads are running. These algorithms are called concurrent collectors, or low-pause collectors. A concurrent collector typically allows an application to experience fewer and shorter pauses. The biggest trade-off here is the overall CPU required by the sophisticated algorithms.

The three main algorithms

Serial GC is the simplest and the default for single core host (e.g. client-class machine, single-processor VM or Docker container). The serial collector uses a single thread to process the heap. It will stop all application threads as the heap is processed (for either a minor or full GC). During a full GC, it will fully compact the old generation. The serial collector is enabled by using the -XX:+UseSerialGC flag.

The throughput collector (aka parallel collector) is the default collector for any 64-bit machine with two or more CPUs. The throughput collector uses multiple threads to collect the young generation, which makes minor GCs much faster than when the serial collector is used. This uses multiple threads to process the old generation as well. The throughput collector stops all application threads during both minor and full GCs, and it fully compacts the old generation during a full GC. Since it is the default in most situations where it would be used, it needen’t be expliticly enabled. To enable it where necessary, use the flag -XX:+UseParallelGC

The G1 GC (or garbage first garbage collector) uses a concurrent collection strategy to collect the heap with minimal pauses. It is the default collector in JDK 11 and later for 64-bit JVMs on machines with two or more CPUs. G1 GC divides the heap into regions, but it still considers the heap to have two generations. Some of those regions make up the young generation, and the young generation is still collected by stopping all application threads and moving all objects that are alive into the old generation or the survisor spaces, using multiple threads. In G1 GC, the old generation is processed by background threads that don’t need to stop the application threads to perform most of their work.

In G1 GC, the old generation is processed by background threads that don’t need to stop the application threads to perform most of their work. Because the old generation is divided into regions, G1 GC can clean up objects from the old generation by copying from one region into another, which means that it compacts the heap during normal processing. This helps keep G1 GC heaps from becoming fragmented.

The trade-off for avoiding the full GC cycles is CPU time; the multiple background threads G1 GC uses to process the old generation requires CPU cycles available at the same time the application threads are running. G1 GC is enabled by specifying the flag -XX:+UseG1GC. It is the default in JDK 11, and functional in JDK 8 as well, with some performance feature missing.

In all cases, GC is caused when the JVM decices GC is necessary; a minor GC will be triggered when the new generation is full; a full GC will be triggered when the old generation is full, or a concurrent GC (if applicable) will be triggered when the heap starts to fill up. Java also provides a mechanism for applications to force a GC to occur: the System.gc() method, although it is always a bad idea to call that method explicitly because it triggers a full GC which hangs the application threads. This method can be disabled by including -XX:+DisableExplicitGC in the JVM arguments.

As to choosing GC algorithm, the rule of thumb is that G1 GC is the better choice. However, in JDK 8, the ability of G1 GC to avoid a full GC is also a key consideration. In this case we may need to choose betwen serial collectors and throughput collectors, based on the number of CPUs on the machine.

The serial collector makes sense when running CPU-bound applications on a machine with a single CPU, even if that single CPU is hyper-threaded. The throughput collector makes sens on multi-CPU machines running jobs that are CPU bound. Even for jobs that are not CPU bound, the throughput collector can be the better choice if it does relatively few full GCs or if the old generation is generally full.

Basic GC tuning

Sizing the heap

If the heap is too small, the program will spend too much time performing GC and not enough time performing application logic. On the contrary, a very large heap will increase the time spent in GC pauses, even thought the pauses occur less frequently. It is also potentially dangerous due to interaction with memory swap. If a Java program with a 12 GB heap is running on a system where swap is enabled, the OS may handle it by keeping 8GB of the heap in RAM and 4GB on disk. The JVM does not know about this because swapping is handled by the OS. The JVM will happily fill up all 12GB of heap it has been told to use. This can cause a sever performance penalty when OS swaps data from disk to RAM. Worse, the one time this swapping is guaranteed to occur is during a full GC, when the JVM must access the entire heap. Swapping during full GC makes the pause an order of magnitude longer.
So heap size (total for all JVMs) should never exceed the amount of physical memory on the machine. Size of heap is controled by two values (Xms as initial value and Xmx as maximum value). Having an initial and maximum size for the heap allows the JVM to tune its behaviour depending on the workload. If the JVM sees that it is doing too much GC, with the initial heap size, it will continually increase the heap until the JVM is doing the “correct” amount of GC, or until the heap hits its maximum size.

A good rule of thumb is to size the heap so that it is 30% occupied after a full GC. To calculate this, start your application and push it to high load. Then connect to the application with jconsole, force a full GC, and observe how much memory is used when the full GC completes.

Sizing the generations

The JVM must also decide how much of the heap to allocate respectively to they young generation and old generation. THe JVM usually does this automatically and usually does a good job in determining the optimal ratio. In some cases you might hand-tune these values.
In general, if there is a relatively larger young generation, young GC pause times will increase, but the young generation will be collected less often, and fewer objects will be promoted into the old genration. But on the other hand, older generation will be relatively smaller and fill up more frequently and do more full GCs. The command-line flag to tune the generation sizes are:
-XX:NewRatio=N
-XX:NewSize=N
-XX:MaxNewSize=N
-Xmn N
The size of initial young generation is determined by initial heap size and new ratio:
Initial Young Gen Size = Initial Heap Size / (1 + NewRatio)
The young generation will grow in tandem with the overall heap size, but it can also fluctuate as a percentage of the total heap (based on the initial and maximum size of the young generation). Adaptive sizing controls how the JVM alters the ratio of young genration to old gneeration within the heap. It should be kept enabled in general. For finely tuned heaps, adaptive sizing can be disabled for a small performance boost.

Sizing the metaspace

When the JVM loads classes, it must keep track of certain metadata about those classes. This occupies a separate heap space called the metaspace. In older JVMs this was handled by a different implementation called permgen. To end users, the metaspace is opaque. It does not hold the actual instance of the class. The objects are held in the regular heap. Information in the metaspace is used only by the compiler and JVM runtime, and the data it holds is referred to as class metadata.
Tuning the metaspace is fairly rare these days because the default values for the size of metaspace are very generous. It is sized dynamically based on an initial size (-XX:MetaspaceSize=N) and will increase as needed to a maximum size (-XX:MaxMetaspaceSize=N).
Resizing the metaspace requires a full GC, so it is an expensive operation. If there are a lot of full GCs during the startup of a program (as it is loading classes), it is often because permgen or metaspace is being resized, so increasing the initial size is a good idea to improve startup in that case.

Controlling Parallelism

All GC alghorithms except the serial collector use multiple threads. The number of these threads is controlled by the -XX:ParallelGCThreads=N flag. Bacuase these GC operations stop all application threads from executing, the JVM attempts to use as many CPU resources as it can in order to minimize the pause time. By default, that means the JVM will run one thread for each CPU on a machine, up to eight. Once that threashold has reached, the JVM adds new thread for only every 1.6 CPus. Sometimes this number is too large relative to the heap size and hand tuning is needed.

Reference: Java Performance by Scott Oaks

This post also contains some helpful information, where the original Oracle white paper about GC was cited. Further than GC, this website from Oracle describes more about JVM.

Ansible at scale 2 of 2

Mon, 25 May 2020 22:08:54 -0400

Template (with Jinja2) and files

In an Ansible role, we can use files or templates to achieve similar results for configuration files. If the configuration file is the same across all targets then we can place it in files directory to push out. If the content of configuration file varies depending on the cluster size, we use Jinja2 template. For example, when you configure zookeeper configuration, a first entry may require total number of nodes in the cluster, a second entry may require the hostname of the server itself; and a third entry may require a comma separated line with hostnames of all nodes in the cluster. This is a typical use case of Jinja template.

We need to make sure Jinjas version is above 2.11.2 (as of May 2020) because older version such as 2.7.2 has known issues with namespaces. To check version and then upgrade Jinja2, we need to use pip:

pip show Jinja2
pip install -U Jinja2

The Ansible template module takes Jinja2 file as input and delivers result file on target host. Note that if the template references host variables from Ansible playbook, then you need to gather facts about host. This means you will have to use a basic playbook like below instead of adhoc command.

A basic playbook to test Jinja2 template is:

- hosts: '{{ansible_limit}}'
 gather_facts: yes
 tasks:
 - template:
 src: cassandra_xml.j2
 dest: /tmp/cassandra.xml

Although Jinja2 offers a lot of flexibility with loop and if-else statement, it is just a templating language and not a programming language. It requires some tricks to achieve what you may otherwise easily do with programming language. One example is persisting a variable outside of a loop. As per the document, it is not possible to set variables inside a block and have them show up outside of it. This also applies to loops. The only exception to that rule are if statements which do not introduce a scope. To achieve that, you would have to use namespace, for each loop where you need to access the variable afterwards from outside of the loop.

{% block db_cluster_config_nobackup %}
{% set ns=namespace(nodeid=0) %}
{% for host in groups[my_db_group]|sort %}
 <var name="DBHost{{ns.nodeid+1}}" value="{{hostvars[host].inventory_hostname}}" />
{% set ns.nodeid=ns.nodeid+1 %}
{% endfor %}
 <var name="DBClusterHosts" value="{% for i in range(ns.nodeid) %}
${DBHost{{i+1}}}{% if not loop.last %},{% endif %}
{% endfor %}" />
{% endblock cass_cluster_config_nobackup %}

For the same reason, you might as well clearly define the start and end of each block in order to not run into trouble with scoping behaviours of variables. These limitations makes Jinja2 template not easy to read and may take several rounds of playbook runs to troubleshoot.

Handler vs conditional task

Sometimes you only want to run a task when its previous task results a change. There are two ways to achieve this: conditional task and handler.

With conditional task, we register the result of previous task to a variable, and execute the ensuing tasks conditionally based on assessment of the variable. We’d have to specify the condition for each of the subsequent tasks that needs to execute conditionally. These tasks, if condition is met, can execute immediately after the first task that registers the variable.

The alternative is through an Ansible mechanism called handler. Handler implements a series of tasks in a separate yaml file in the handers directory under the role. In the triggering task we need to notify the handler. The tasks in the hander will fire if the triggering task returns “changed” in its result. Handler is a great way to shorten the length of task or Playbook. However, we need to understand several subtleties with regard to handlers:

Although handler is notified during a task run, it is not fired until the end of each block of tasks in a play. They are not immediately fired after triggering task.
A handler will only execute once at the end of play, even if it was notified multiple times by different tasks during the play run.
Handler tasks are executed in the order of declaration, not in the order of notification.

In summary, Ansible’s notification handling mechanism is asynchronous, once-only, and out of sequence. The points above are illustrated in the following playbook:

---
- hosts: ghdocker
 tasks:
 - name: CopyFile3
 copy:
 src: ~/ansible/file3.txt
 dest: /tmp/file3.txt
 notify:
 - handler3
 - handlergeneral
 - name: CopyFile2
 copy:
 src: ~/ansible/file2.txt
 dest: /tmp/file2.txt
 notify:
 - handler2
 - handlergeneral
 - name: CopyFile1
 copy:
 src: ~/ansible/file1.txt
 dest: /tmp/file1.txt
 notify:
 - handler1
 - handlergeneral
 - debug: msg="end of play!"
 handlers:
 - name: handler1
 debug: msg="file1.txt has been copied."
 - name: handler2
 debug: msg="file2.txt has been copied."
 - name: handler3
 debug: msg="file3.txt has been copied."
 - name: handlergeneral
 debug: msg="A file has been copied"

Here is the output of the playbook run:

PLAY [ghdocker] ******************************************************************************

TASK [Gathering Facts] ******************************************************************************
ok: [ghdocker]

TASK [CopyFile3] ******************************************************************************
changed: [ghdocker]

TASK [CopyFile2] ******************************************************************************
changed: [ghdocker]

TASK [CopyFile1] ******************************************************************************
changed: [ghdocker]

TASK [debug] ******************************************************************************
ok: [ghdocker] => {
 "msg": "end of play!"
}
RUNNING HANDLER [handler1] ******************************************************************************
ok: [ghdocker] => {
 "msg": "file1.txt has been copied."
}

RUNNING HANDLER [handler2] ******************************************************************************
ok: [ghdocker] => {
 "msg": "file2.txt has been copied."
}

RUNNING HANDLER [handler3] ******************************************************************************
ok: [ghdocker] => {
 "msg": "file3.txt has been copied."
}

RUNNING HANDLER [handlergeneral] ******************************************************************************
ok: [ghdocker] => {
 "msg": "A file has been copied"
}
PLAY RECAP ******************************************************************************
ghdocker : ok=9 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Handler is a good way to keep idempotency. For example, Ansible does not have a way to import a yum .repo file to create a repo. We have to take two steps:

use get_url module to download the repo file (e.g. to /tmp),
use shell module to call yum-config-manager.

The problem is these two steps are not idempotent. If you repeat them, it will attempt to import the same repo file again. A little trick here is to use force=no option on get_url so it does not attempt to download if the file is already present in target directory. Then notify a handler to import repo file so the shell command is only called if there is a change.

The task looks like this:

- name: download repo file
 get_url:
 url: https://download.docker.com/linux/centos/docker-ce.repo
 dest: /tmp/docker-ce.repo
 mode: '0755'
 force: no
 notify:
 - Add docker repository

The handler looks like this:

- name: Add docker repository
 shell: yum-config-manager --add-repo=/tmp/docker-ce.repo

The handler is only fired when it is notified after get_url module returns changed in its result. Running the task again will not cause it to attempt to add the same repo again.

Note that when you use command or shell module, Ansible typically reports changed status. If this is not desired (e.g. you don’t want it to notify handler all the time), this behaviour can be overridden with changed_when parameter. You can specify conditions to meet in order to consider the shell/command module to have a changed result. Here is an example.

Ansible commands

In operation, our engineer needs to run a command on a group of servers. I encourage the use of Ansible adhoc command whenever possible. I recommend start with the following two commands:

ansible-inventory --graph
ansible all -m ping

The ping module triggers an “Ansible ping” to targets in the specified group. Over the years, Ansible community developed many helpful modules, such as yum, yum_repository, apt_rpm, uri, synchronize, fine, copy, etc and many can be used instead of bash command. However, sometimes, the expected Ansible module is either unavailable or missing function. For example, Ansible’s uri module cannot replace curl command with the following switches:

curl -s -XGET http://{{inventory_hostname}}:8080/objects/{{object_id}}/binary/all -o /dev/null -w '%{response_code} %{size_download} %{time_total} %{speed_download}\n' | awk '{if ($1==200) print "size="$2/1048576"MB,time="$3"s,speed="$4/1048576"MB/s"; else if($1==404) print "Cannot find object {{object_id}}"; else print "Unknown error. Code "$1 " when retrieving object {{object_id}}";}'

To leverage all these curl options, we still need to use the shell module in Ansible to call the command in shell.

Other helpful Ansible commands include ansible-pull for pulling playbooks from VCS repo, and ansible-console for interactive adhoc command execution.

Tags and extra variables

Both tags(-t) and extra variables (-e) are great ways to achieve flow control in playbooks. You can specify to run tasks with certain tags or skip tasks with certain tags. Extra variables can override the default variables from the host or the group. Both are great tools to improve re-usability of a Playbook.

Speed up execution

To speed up execution of Ansible tasks, there are several ways. For example, we can disable fact gathering by default so it only gathers fact if explicitly specified. This can be set in gathering=explicit under defaults section of ansible configuration file. If you have to gather facts, you may cache the facts using the following:

[defaults]
gathering = smart
fact_caching_timeout = 86400
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_fact_cache

Other than caching, Ansible allows you to select from several execution strategies for playbook. The linear strategy introduces configurable parallelization per task. The free strategy introduces parallelization per play.

linear (by default): Up to the fork limit of hosts will execute each task at the same time and then the next series of hosts until the batch is done, before going on to the next task. This mode ensures the progress is synchronized at each task.
free: as specified above, this is preferred when there is no need to coordinate the progress between each host target. It is a “free run” for each host all the way till the end of the playbook.
debug: essentially linear strategy except that the progress is controlled by an interactive debug session

The fork limit, with a conservative default of 5, can be adjusted in Ansible configuration. The execution strategy can be either specified in Ansible configuration, or specified per play. For example, the following snippet sets the strategy to free for the current play:

---
- hosts: all
 strategy: free
 tasks:
...

Ansible documentation also mentions some play-level keywords to control execution. The serial keyword, is one of them. It can be set along with any strategy above, and it introduces the effect of hosts batching. The value can be a single number, a percentage, or even a list of numbers (if size for each batch is different). Note that the batch size should not exceed the fork limit. This is particularly useful in rolling upgrades. For example:

---
- name: test play
 hosts: webservers
 serial: "30%"

With the parallelization capacity outlined above, a potential concern is some heavy-lifting task may consume a lot of resources, if being executed for all hosts at the same time. Luckily, Ansible has a task/block level keyword throttle, which “de-parallelize” the multi-host progress at a particular task, or block. Here is an example provided by Ansible documentation:

tasks:
- command: /path/to/cpu_intensive_command
 throttle: 1

If there are long running tasks, we can specify async and poll values so Ansible leaves a task running and check back later. For example, the following task allows Ansible to move on and check back every 5 seconds, if the task takes longer than 45 seconds, it is considered failed:

---
 - hosts: all
 remote_user: root
 tasks:
 - name: simulate long running task for 15 sec, wait for up to 45 sec, poll every 5 sec
 command: /bin/sleep 15
 async: 45
 poll: 5

Python Version

The recommendation is to use Python3 for any new development because there is no dependency. If there is no preference specified, Ansible tries to find out the appropriate interpreter and it can be seen in the response of ansible ping module. You can also force the interpreter by providing additional parameter ansible_python_interpreter. To change default interpreter, specify interpreter_python in ansible.cfg. For example:

[defaults]
inventory=~/ansible/inventories/site.yml
library=~/ansible/library/
vault_password_file = ~/ansible/.vault_key
host_key_checking = False
display_skipped_hosts = False
retry_files_enabled = False
interpreter_python=/usr/bin/python3

[privilege_escalation]
become_method=sudo

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=1h
pipelining = True

My open issues

I have some minor details that I have not been able to address, after a lot of time googling around. So I have to leave them for future reference.

If an Ansible playbook involves multiple plays (i.e. each with their own host), there is no way to persist a variable across different plays. A dumb alternative is to make all the variables to use available for every single host (under all directory).

In Jinja2 template, if I need to access the group of a target host (as defined in inventory), and the target belongs to multiple groups, I cannot filter to match the group I need.

Ansible at scale 1 of 2

Sun, 17 May 2020 19:38:34 -0400

The Ansible In Depth white paper outlines Ansible’s use cases in four categories:

Configuration management
Application deployment
Orchestration: for coordinating a multi-machine process such as interacting with load balancer and rolling cluster upgrade
As-needed task execution: ad-hoc tasks on large number of hosts

At work, my original automation scheme involves several Ansible Playbooks that started off simple but have been sprawling ever since. I have to spend some time to revamp the Ansible code base, following the best practices from the official documentation. The goal is to:

Reduce the number of Playbooks;
Improve code re-usability (by Ansible roles);
Increase portability across different customer environment;
Improve security;
Re-organize the directory so that more team members can contribute to different parts of it.

The Ansible code base is used by our customer service engineers, some of whom are ingrained with the established way they have been using certain Playbooks in their daily tasks. This requires me, throughout the development initiative, to ensure a consistent interaction between them and their Playbook commands.

Inventory and variables

We cannot guarantee that every custom environment are identical, but what we can do is make sure that for a new environment, the only change to make is inventory and variables. This is where we can strike a balance between portability and customization. No changes should be made to tasks, roles or Playbooks when the Ansible directory is deployed at a different customer environment.

If the total number of servers to manage are under 100, all can be listed in a single inventory file, and specify the only inventory file as default so the -i switch is not required for every Ansible command run. If there are more than 100 servers, it is advisable to separate them out into several inventory files in YAML, each less than 200 lines. You will have to specify inventory file with -i each time you run Ansible command.

The inventory may contain a hierarchy of groups, in order to facilitate command calls to specific groups of servers. For example:

all:
 children:
 prod:
 children:
 prod_app:
 children:
 prod_app_dc1:
 hosts:
 apphost01:
 apphost03:
 zk_id: 1
 apphost05:
 apphost07:
 zk_id: 2
 apphost09:
 apphost11:
 zk_id: 3
 apphost13:
 prod_app_dc2:
 hosts:
 apphost02:
 apphost04:
 zk_id: 1
 apphost06:
 apphost08:
 zk_id: 2
 apphost10:
 apphost12:
 zk_id: 3
 apphost14:
 vars:
 has_app: yes
 has_nginx: yes
 has_db: no
 prod_db:
 children:
 prod_db_dc1:
 hosts:
 dbhost01:
 dbhost03:
 dbhost05:
 dbhost07:
 dbhost09:
 dbhost11:
 vars:
 clustered: yes
 bk_node: dbhost11
 prod_db_dc2:
 hosts:
 dbhost02:
 dbhost04:
 dbhost06:
 dbhost08:
 dbhost10:
 dbhost11:
 vars: 
 clustered: yes
 bk_node: dbhost12
 vars:
 has_app: no
 has_nginx: no
 has_db: yes
 vars:
 ansible_become_pass: '{{site_prod_root_pw}}'
 test:
 children:
 test_dc1:
 hosts:
 tapphost01:
 test_dc2:
 hosts:
 tapphost02:
 vars:
 ansible_become_pass: '{{site_test_root_pw}}'
 has_app: yes
 has_nginx: yes
 has_db: yes
 vars:
 ansible_become: yes
 ansible_become_method su
 ansible_become_user: root

User of Ansible Playbook can use -l to specify a pattern that matches a single or multiple groups, such as prod_db_dc*. In the above example, the password is not stored in clear text. They should reference a variable from a separate file encrypted by ansible-vault.

From Playbooks to roles

As a refresher from the white paper, a single “task” in Ansible is essentially a module call with parameters. A “play” consists of a series of tasks (defined under “tasks” section) all to execute on a specified host (defined under “hosts” section). A Playbook consist of one or several plays, as shown in this example. In reality though, a Playbook usually contains only one play. Even that one play can grow to an unmanageable length, as complexity increases over time. This is where we need to change our approach towards scalability and manageability.

The Ansible community advocates the use of roles in place of Playbooks. The concept of Ansible “role” seems fairly abstract and confusing at the beginning. The word “role” pictures a static server state, whereas our existing Playbooks are full of actions (think of shell scripts). How would one convert an action list into static states? After some thought, I came to the understanding that roles should be thought of as desired end state. Yes, the end state is static, but that’s all we care about. This is essentially the whole idea of Ansible’s desired state configuration: you start from the end state and leave it to modules to complete what needs to be done to reach that state. The concept of role perfectly reflects how Ansible wants you to think about solving an infrastructure problem – stop thinking about what you need to do. Instead, think about what you ultimately want, start from the desired state and work backwards.

In our own setup, the best practice turned out to be: if the Playbook involves a single play with less than 5 tasks, just stick to Playbook. We don’t get rid of Playbooks just for the sake of it. Otherwise, if a Playbook has grown to more than 5 tasks, we need to think about our desired state, and either implement a new role, or incorporate it into an existing role. This is the time we have to transition from the Playbook oriented thinking to the role oriented thinking. Each role directory can include a task sub-directory with main.yml that references the rest of the tasks. Each role can define its own role-related variables. If there’s a lot in common between two roles, we can even have a common role with or without its main.yml.

A simplified version of our Ansible directory structure looks like this:

├── deploy-app.yml
├── deploy-db.yml
├── inventories
│   ├── group_vars
│   │   ├── all
│   │   │   ├── all.yml
│   │   │   └── vault_all.yml
│   │   ├── prod_dc1_db.yml
│   │   ├── prod_dc2_db.yml
│   │   ├── test_dc1_db.yml
│   │   └── test_dc2_db.yml
│   ├── host_vars
│   └── site_inventory.yml
├── roles
│   ├── common
│   │   ├── files
│   │   └── tasks
│   │   ├── log.yml
│   │   ├── skip_self.yml
│   │   └── validate_path.yml
│   ├── db_conf
│   │   ├── defaults
│   │   │   └── main.yml
│   │   ├── files
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── meta
│   │   │   └── main.yml
│   │   ├── README.md
│   │   ├── tasks
│   │   │   ├── main.yml
│   │   │   ├── start_db.yml
│   │   │   ├── stop_db.yml
│   │   │   └── update_cluster_var.yml
│   │   ├── templates
│   │   │   ├── myid.j2
│   │   │   ├── db_properties.j2
│   │   │   └── zookeeper_properties.j2
│   │   └── vars
│   │   └── main.yml
│   └── app_conf
│   ├── defaults
│   │   └── main.yml
│   ├── files
│   ├── handlers
│   │   └── main.yml
│   ├── meta
│   │   └── main.yml
│   ├── README.md
│   ├── tasks
│   │   ├── bk_app_conf.yml
│   │   ├── empty_app_conf.yml
│   │   ├── main.yml
│   │   ├── push_app_conf.yml
│   │   ├── start_app.yml
│   │   ├── stop_app.yml
│   │   ├── tar_app_conf.yml
│   │   ├── untar_app_conf.yml
│   │   └── update_cluster_var.yml
│   ├── templates
│   │   └── dbref_xml.j2
│   └── vars
│   └── main.yml
├── service-app.yml
└── service-db.yml

Variables specific to a group of hosts or individual hosts can be included in different yml files. When the entire directory is moved to a different customer environment, our engineers will need to update the inventory and variable files. The task, roles and Playbooks should build their logics using those variables.

Vault

Our previous implementation of Ansible Playbook stores sudo password base64 encoded and use no_log to avoid displaying values. Now we move those to encrypted variable yml file using ansible-vault. We reference the value to encrypt as regular variable:

ansible_become_pass: '{{passtoencrypt}}'
ansible_become_method: sudo
ansible_become: yes

Then we run the following:

ansible-vault create vault_all.yml

This prompt for a key, and once you type in the key it opens a text editor where we can store the real password. For example:

passtoencrypt: MyP@ssw0rd4real!

Use the text editor to save file. The file is now saved encrypted and must be open with correct key (aka vault password). If we call Ansible Playbook with –ask-vault-pass switch then the Playbook will prompt for key input, or use include_vars to include variable from vault file (example). If we want to even skip this, we can store the key in a file and reference them from vault_password_file in ansible.cfg

Ansible Vault has more commands to edit or view the encrypted variables in the documentation.

Optimize connection

OpenSSH 5.6 and later supports multiplexing where multiple SSH sessions share a TCP connection. This can be turned on so that the following SSH connections save the time of TCP handshake. This can be configured in ansible configuration file under ssh_connection. Below is an example of this option with ControlPersist=1h. So the TCP connection is torn down after 1 hour.

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=1h

The other option we can leverage is pipelining. Ansible takes three steps to execute a task:

build a python script based on module used
copy the python script to remote host
execute the python script on the remote host

If pipelining is turned on, the python script is passed in along with the SSH session, this would save a roundtrip and increase performance. Pipelining can be configured under ssh_connection in Ansible configuration file:

[ssh_connection]
pipelining = True

In the example below we can see by pipelining we cut the number of connection in half:

# with pipelining
[ghunch@control-host ~]$ ansible remote-host -vvvv -m ping | grep EST
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch


# without pipelining
[ghunch@control-host ~]$ ansible remote-host -vvvv -m ping | grep EST
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch
<remote-host> ESTABLISH SSH CONNECTION FOR USER: ghunch

Note that if we use sudo command, then we need to disable requiretty in /etc/sudoers on the remote host.

Custom Module

It’s fairly straightforward to build a custom module in Ansible. Just place the module file (modulename.py) in inventory directory and use it as you would with regular Ansible module. The module file needs to be completed in Python with certain return value. Before creating custom module, you should look for existing modules to avoid re-inventing the wheel. You may also need to determine whether you simply need to run a python script on target host (with Ansible’s script module), or you really need an Ansible module. The former is procedural, and the latter focus on desired state. Custom module is more used in proprietary development.

Balloon steals memory from virtual machines

Thu, 07 May 2020 19:49:00 -0400

This article is my experience with memory balloon on virtual machine.

I came across an ElasticSearch server (ESXi guest with 32GB physical memory) where the main process keeps dying of OOM. Even worse, after the OOM event, the free memory left is about 10G and Elastic Search cannot start because its JVM is set with -Xms16g in /etc/elasticsearch/jvm.options. So I need to address the OutOfMemory error.

To understand what triggered OOM, we can use dmesg or just check /var/log/message, where the memory snapshot by OOM killer is displayed during the kernel panic:

Apr 26 04:18:15 elastichost kernel: kworker/7:1 invoked oom-killer: gfp_mask=0x200d2, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: kworker/7:1 cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 7 PID: 13968 Comm: kworker/7:1 Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Workqueue: events_freezable vmballoon_work [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc05f1f84>] vmballoon_work+0x454/0x6ff [vmw_balloon]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22b9d4f>] process_one_work+0x17f/0x440
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bade6>] worker_thread+0x126/0x3c0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1c31>] kthread+0xd1/0xe0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2974c37>] ret_from_fork_nospec_begin+0x21/0x21
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c1b60>] ? insert_kthread_work+0x40/0x40
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3649243 inactive_anon:439380 isolated_anon:0#012 active_file:480 inactive_file:693 isolated_file:0#012 unevictable:0 dirty:2 writeback:0 unstable:0#012 slab_reclaimable:77845 slab_unreclaimable:10219#012 mapped:10086 shmem:9596 pagetables:16727 bounce:0#012 free:50116 free_pcp:238 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122768kB min:5972kB low:7464kB high:8956kB active_anon:471052kB inactive_anon:471040kB active_file:232kB inactive_file:460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:584kB shmem:300kB slab_reclaimable:23908kB slab_unreclaimable:3872kB kernel_stack:656kB pagetables:6440kB unstable:0kB bounce:0kB free_pcp:288kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:214 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61804kB min:61576kB low:76968kB high:92364kB active_anon:14125920kB inactive_anon:1286480kB active_file:1688kB inactive_file:2312kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:8kB writeback:0kB mapped:39760kB shmem:38084kB slab_reclaimable:287472kB slab_unreclaimable:36988kB kernel_stack:5680kB pagetables:60468kB unstable:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:698 all_unreclaimable? no
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 407*4kB (UEM) 506*8kB (UEM) 431*16kB (UM) 364*32kB (M) 272*64kB (UM) 179*128kB (UM) 99*256kB (UM) 35*512kB (M) 11*1024kB (M) 2*2048kB (M) 0*4096kB = 123164kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15576*4kB (UM) 39*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62616kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 44652 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 34162 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884045, delete 10845872, find 3294122/3678936
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 12996 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 83 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 50 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 47 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 113 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 39 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 132 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 4770 100 482 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6332 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 30 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3994129 15027 457539 0 java
Apr 26 04:18:15 elastichost kernel: [16264] 495 16264 18032 0 31 162 0 controller
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16070 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16264 (controller) total-vm:72128kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:15 elastichost kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Apr 26 04:18:15 elastichost kernel: java cpuset=/ mems_allowed=0
Apr 26 04:18:15 elastichost kernel: CPU: 2 PID: 16265 Comm: java Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Apr 26 04:18:15 elastichost kernel: Call Trace:
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2961e41>] dump_stack+0x19/0x1b
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295c86a>] dump_header+0x90/0x229
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa2301052>] ? ktime_get_ts64+0x52/0xf0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23581ef>] ? delayacct_end+0x8f/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba4e4>] oom_kill_process+0x254/0x3d0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b9f8d>] ? oom_unkillable_task+0xcd/0x120
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ba036>] ? find_lock_task_mm+0x56/0xc0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23bad26>] out_of_memory+0x4b6/0x4f0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa295d36e>] __alloc_pages_slowpath+0x5d6/0x724
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23c1105>] __alloc_pages_nodemask+0x405/0x420
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa240df68>] alloc_pages_current+0x98/0x110
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b6347>] __page_cache_alloc+0x97/0xb0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23b8fa8>] filemap_fault+0x298/0x490
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484d0e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa22c2dc0>] ? wake_bit_function+0x40/0x40
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffc0484f0c>] xfs_filemap_fault+0x2c/0x30 [xfs]
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e444a>] __do_fault.isra.59+0x8a/0x100
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e49fc>] do_read_fault.isra.61+0x4c/0x1b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23e93a4>] handle_pte_fault+0x2f4/0xd10
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa23ebedd>] handle_mm_fault+0x39d/0x9b0
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f5e3>] __do_page_fault+0x203/0x500
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296f915>] do_page_fault+0x35/0x90
Apr 26 04:18:15 elastichost kernel: [&lt;ffffffffa296b758>] page_fault+0x28/0x30
Apr 26 04:18:15 elastichost kernel: Mem-Info:
Apr 26 04:18:15 elastichost kernel: active_anon:3607073 inactive_anon:480522 isolated_anon:0#012 active_file:8 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:75170 slab_unreclaimable:10131#012 mapped:2070 shmem:9592 pagetables:16696 bounce:0#012 free:50006 free_pcp:72 free_cma:0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122560kB min:5972kB low:7464kB high:8956kB active_anon:471000kB inactive_anon:471076kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:292kB shmem:288kB slab_reclaimable:22832kB slab_unreclaimable:3752kB kernel_stack:672kB pagetables:6408kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:215 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163
Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61572kB min:61576kB low:76968kB high:92364kB active_anon:13957292kB inactive_anon:1451012kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:0kB writeback:4kB mapped:7988kB shmem:38080kB slab_reclaimable:277848kB slab_unreclaimable:36756kB kernel_stack:5664kB pagetables:60376kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:500 all_unreclaimable? yes
Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0
Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 393*4kB (UEM) 490*8kB (UEM) 453*16kB (EM) 369*32kB (UEM) 269*64kB (M) 180*128kB (M) 98*256kB (M) 35*512kB (M) 12*1024kB (UM) 1*2048kB (M) 0*4096kB = 122148kB
Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15440*4kB (UM) 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61768kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 26 04:18:15 elastichost kernel: 42975 total pagecache pages
Apr 26 04:18:15 elastichost kernel: 33308 pages in swap cache
Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884233, delete 10846914, find 3294127/3678944
Apr 26 04:18:15 elastichost kernel: Free swap = 0kB
Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB
Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM
Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly
Apr 26 04:18:15 elastichost kernel: 193436 pages reserved
Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 4016 100 6314 0 systemd-journal
Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd
Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad
Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd
Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 78 32 88 -900 dbus-daemon
Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind
Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy
Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 46 15 41 0 irqbalance
Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager
Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 42 18 41 0 systemd-logind
Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService
Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 91 58 246 0 vmtoolsd
Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd
Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 36 27 140 0 ntpd
Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 130 62 970 0 snmpd
Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 1215 100 485 0 rsyslogd
Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6230 100 781 0 sh-metricbeat
Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat
Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd
Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned
Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin
Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd
Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 27 18 129 0 crond
Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty
Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node
Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd
Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash
Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3993992 15027 457699 0 java
Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16291 (java) score 512 or sacrifice child
Apr 26 04:18:15 elastichost kernel: Killed process 16070 (java) total-vm:188490524kB, anon-rss:15975968kB, file-rss:0kB, shmem-rss:0kB
Apr 26 04:18:16 elastichost systemd: elasticsearch.service: main process exited, code=killed, status=9/KILL
Apr 26 04:18:16 elastichost systemd: Unit elasticsearch.service entered failed state.
Apr 26 04:18:16 elastichost systemd: elasticsearch.service failed.

The line “Free swap = 0kB” suggest the trigger of OOM is out of swap. So does ElasticSearch contribute to the swap shortage?

ElasticSearch was the main process on the VM and I noticed that the OS did not disable swap, which does not align with the best practice from ElasticSearch community. In order to find out whether Elastic Search is pushed to use swap, we can get the process ID:

pidof java

ElasticSearch happens to be the only Java based process and the PID is 2283, the following command shows the swap usage by this process:

cat /proc/2283/status | grep VmSwap

If it shows a non-zero value, then ElasticSearch is using swap and you should expect some performance issues. We definitely should disable swap on ElasticSearch but does that solve the problem? Probably not because we didn’t address what caused the memory pressure in the first place. Disabling swap most likely makes ElastciSearch last longer before something else such as shortage of available memory triggers OOM.

In search for the source of memory shortage, I checked the result of free command and the top command. The free command (-mh) shows the following:

 total used free shared buff/cache available
Mem: 31G 20G 10G 117M 243M 10G
Swap: 2.0G 249M 1.8G

I tried to follow my example from a previous post to make sense of the memory reads. When I monitor process with top and watch for RSS column, I cannot identify a single process that even takes more than a few hundred megabytes. I’ve downloaded a tool smem from epel repo, and the result of “smem -kt” suggest that the total RSS is about 372.2M:

So there is about 19.6GB of memory usage unaccounted for. There is something that takes this much memory in the VM and haunting round even after OOM killer!

Inspired by this post, I was able to identify the culprit, that is the memory balloon. We don’t have access to the host but from the Guest OS, we can tell by vmware-toolbox-cmd (need to install yum package open-vm-tools):

vmware-toolbox-cmd stat balloon

The result displays a whopping 20807 MB as memory balloon! This needs to be sent to PaaS vendor for investigation but it is likely a result of memory over-allocation/over-commitment at host level, as well as the setup where the memory of guest is not reserved. Looking at the original log snippet, the line with “events_freezable vmballoon_work” is also an indicator of balloon causing the OOM.

Hypervisor needs memory ballooning to reclaim memory from guest. Since the guest OS does not expect the amount of physical memory to change, hypervisor has to maintain the illusion that the guest has its fixed amount of physical memory. The hypervisor first computes the amount of memory that needs to reclaim, then it leverage some low-level mechanism such as a balloon driver (a pseudo-device driver) installed on guest. The driver communicates with hypervisor and is told to allocate or de-allocate memory. If the driver is told to allocate memory to host, it tells the guest OS to pin the allocated pages into physical memory so they are locked and the physical memory available to guest OS is decreased. All these low level mechanisms explains why it is hard to account for memory consumption based on process running in Guest OS.

Memory balloon is a dynamic process and the hypervisor adjusts the size of balloon. However, the hypervisor, the guest OS and the application process (ElasticSearch in this case) may enter a nuance interaction:

The hypervisor reclaims memory from guest OS (ballooning);
Guest OS panicked with OOM exception;
OOM picks a process to kill, based on oom_score. Unfortunately, the true culprit (balloon) is not visible to guest OS, hence exempted from being considered to kill.
By killing the application process (usually the main application for VM), a fair chunk of guest memory is freed up;
the balloon then became more aggressive on the guest as seeing more memory becomes “available”;
the application on guest then does not have minimum memory on guest OS to start;

How to solve this problem? There are several things to consider. On the hypervisor, keep the entire or part of the guest memory reserved for start of application. Tune the OOM scoring so the non-critical application get killed. Implement application daemon, etc.

Common local Git operations

Wed, 08 Apr 2020 11:54:00 -0400

This is a summary of concepts in common Git operations. We will discuss brach, merge, rebase, cherrypick, stash and reset. Then we’ll discuss pull, fetch, and push.

Commit, Branch and HEAD

When you run “git commit”, the following happens:

Git checksums each subdirectory, and stores them as a tree object (file path and name) and blob object (file content) in Git repository;
Git creates a commit object that has the metadata and a pointer to the root project tree; or if this is not the first commit, the pointer will point to the commit immediately before it

The operations above should form a chain of commit. It can be a long chain and may diverge into branches. In Git semantics however, a branch is simply a lightweight, movable pointer to one of the commits. The default branch name in Git is master. A Git repository may contain multiple branches and the name master itself does not suggest any privilege. There is also a special pointer called HEAD, which indicates the branch you are currently working on. So branch is essentially a pointer to a commit; HEAD is essentially a pointer to a branch. “git checkout” can switch branch that HEAD points to.

Basic Merge

One type of basic merge simply moves branch pointer from one commit to another (along the same chain) without creating any commit. Here is a diagram before basic merge:

Before basic merge, Hotfix branch is based on master

The following command performs basic merge:

$ git checkout master
$ git merge hotfix
Updating f42c576..3a0874c
Fast-forward
 index.html | 2 ++
 1 file changed, 2 insertions(+)

Then Git simply moves the pointer (named master) forward. There is no divergent work to move together, hence no chance of merge conflict. This type of basic merge is also called “fast-forward” merge.

After basic merge, master is fast-forwarded to hotfix

The other type of merge involves reconciling divergent work together, which may or may not involve conflict. Suppose this is the commit tree to start with:

Three snapshots used in a typical merge

The following commands perform the merge:

$ git checkout master
Switched to branch 'master'
$ git merge iss53
Merge made by the 'recursive' strategy.
index.html | 1 +
1 file changed, 1 insertion(+)

Instead of just moving the branch pointer forward, Git creates a new snapshot that results from this three-way merge and automatically creates a new commit that points to it. This is referred to as a merge commit, and is special in that it has more than one parent.

A merge commit

Now that your work is merged in, you have no further need for the iss53 branch. You can close the issue in your issue-tracking system, and delete the branch:

$ git branch -d iss53

If commits from two respective branches changes the same file in different ways, then there is a merge conflict. In this case, Git cannot just create a merge commit. Instead it asks the user to resolve the conflict first. You have to choose either side of the change, or just merge the content yourself. At this point, if you introduce a change that does not appear in any parent, it is referred to as an evil merge.

Beyond the basic merge, there are more sophisticated merge conflict resolution tools covered in advanced merging.

Merge and Rebase

There are two ways to integrate changes from one branch to another. Merge and rebase. Suppose your commit chain diverge into a master branch and a feature branch. Merging (from feature to master) takes the content of feature branch and integrate it with master branch.

$ git checkout master
$ git merge feature

When you rebase a feature branch onto master, you move the base of the feature branch to master branch’s ending point.

$ git checkout feature
$ git rebase master

After merge, you are still on the same branch. The commits from other branch are integrated into the branch that you are already on. There is no change in any existing commits (history). After rebase, your base will be moved to a different branch, along with the commits that you have made in the previous branch (since the diverge). In other words, by re-playing those commits on a different branch, it changed history.

merge vs rebase

The chart above is stolen from this article, which does a better job explain in detail the difference, pros and cons of merge and rebase. Merge does create a “merge commit”, and a git history full of merges can be cluttered. Rebase does not create an extra commit but since it changes the history of a branch, it has impact to other collaborators. It can be done in an interactive way (with -i switch). The golden rules of rebasing is covered in this article. One of the principles is that never perform a rebase on a public branch.

git operations

Cherrypick

In a cherrypick operation, the current branch does not change. You simply pick interested commits from other branches to re-apply to your current branch. You may pick a single or a series of commits from other branch. These commits are not “moved” to your current branch. They remain intact. They are just re-played as new commit to current branch. Unlike rebase, there is no re-writing of history, hence not as dangerous.

Reset and Stash

Suppose you are working on a part of a project and it starts getting messy. There has been an urgent bug that needs your immediate attention. It is time to save your changes and switch branches. If you are okay to give up your uncommitted work, you may perform a reset, in one of the three modes covered in a previous article.

But most likely, you don’t want to do a commit of half-done work. The solution is git stash. Stashing is handy if you need to quickly switch context and work on something else but you’re mid-way through a code change and aren’t quite ready to commit. In the most basic workflow, you need to run this command to save your uncommitted (but staged) work. As soon as you stash your change, the working directory is clean with all uncommitted local changes saved elsewhere. You can perform any other Git operations, such as change branch. When you’re ready to resume, you may pop the stash. Here is an example:

$ git add .
$ git stash
$ git checkout correctbranch
$ git stash pop

Instead of pop, you can also use apply to keep the changes in working directory.

$ git stash apply

More details are on this page from Bitbutket.

Fetch and Pull

A git fetch simply downloads blob data from remote so the .git directory comes in sync with the server. It does not attempt to update the local working directory. If there is staged or uncommitted local changes, fetch will not impact them. A git pull is essentially git fetch followed by git merge. In addition to downloading blob data, it also updates local working directory. Therefore, there is a chance of merge conflict when the same file has been modified locally. Git will usually guide you through the merge conflict by flagging the conflict area in the file and let you decide the survival changes. For example:

#! /usr/bin/env ruby

def hello
<<<<<<< HEAD
 puts 'hola world'
=======
 puts 'hello mundo'
>>>>>>> mundo
end

hello()

You will be prompted in an editor session to reconcile the conflict. Once the file is saved, you will also need to do a “merge commit”, before you can pull again.

Push

Git push is the opposite of pull, where you merge local branch to the remote. (There is no opposite of fetch because there is no point to merge to remote without updating working directory, no collaborator works on the working directory on the server after all). If the local branch has fallen out of sync with the remote, there is a chance of merge error during git push. To minimize the chance of a merge during push, we can run a git pull before and reconcile any potential conflict locally. This is known as a pre-merge.

Visualizer

I came across a great visualizer of commit chain here. In the command panel type some git command and it will print the commit graph for you

Clean up Git repository

Sat, 26 Oct 2019 20:33:00 -0400

A BitBucket repo has a hard limit of 2GB in size, and soft limit of 1GB. This is not expandable as per Bitbucket and contributors will start receiving warnings once soft limit is reached. We can tell the usage of a repo from the landing page of the repo in BitBucket.

Git is a distributed version control system for source code management, which implies the followings:

It is intended for source code, or configuration code; but not for storing build artifacts, or installers;
Git remembers every single commit, including the ones associated with large files;
Even a contributor deletes a large file (“git rm filename”) after commit, the large file is only removed from the HEAD. The historical commit still stores the file. After all, the whole point of version control is to survive crazy deletion.
distributed means that those large files will be pulled down to contributors laptop (waste everybody’s space although up to 2G:);

With all these implications, shrinking the size of a repo isn’t as straightforward as just removing large files from current commit. We’d have to rewrite the commit history. Here are the steps we should take once repo size grows over the soft limit.

Clean up remote orphaned branches

Removing these branches (remotes/origin/branchname) per se does not free up space. It simplifies the branch structure, leaving /remote/origin/HEAD the only branch left to cleanse for the rest of the steps.

# git push origin --delete branchname

Remove useless files in current commit (HEAD)

In this step we remove useless files in current commit. Again we should not expect much space freed because all file committed previously, even deleted, are still stored. They are just now showing up in the working directory. For this step, we can create a separate local dir on Mac:

mkdir -p /Users/digihunch/repo-cleanup
cd /Users/digihunch/repo-cleanup

Now within the new directory, we create a bare repo and then the full repo:

git clone --mirror https://gh@bitbucket.org/digihunch/source.git
git clone https://gh@bitbucket.org/digihunch/source.git

Then we dive into the full repo and identify the large files:

find . -type f -size +1000k -exec ls -lh {} \; |awk '{print $9":" $5}'

We can run “git rm ” against the files identified as too large or deletable. Then commit and push to remote repo. This removes large files from current commit.

Remove large file and the relevant commits in the history

As previously mentioned, we have to re-write the history so history forget about the large files. After this step, the historical commits that large files are associated with will all be deleted. Compare the two charts below to understand what the effect is:

We can use the bare repo created in the last step, with “git filter-branch” tool to cleanse the branch tree. Some advocate as a faster third party tool BFG Repo-Cleaner as a faster, third-party alternative but I usually lean towards native tool. This article explains the command switches.

git filter-branch -f --tree-filter "rm -rf \large_file.zip" --prune-empty -- --all

After this steps the repo should be cleansed. According to this guide from BitBucket, we still need to contact their support to run a garbage collection for us in order to see the size change. It even takes time for the size to be reflected after garbage collection. This reference also does great job explaining what we need to do.

Other contributors re-sync history

It is important to understand that the step above modifies history. Although the commit hash did not change, they are assigned with different commit-ids and you can tell from the commit history where it displays former commit id.

This activity only affects remote repository. Each contributor’s local repository still stores the old commits and should be sync’ed with the remote origin by deleting the entire repo and run “git clone” again. Although not welcomed by every individual contributors, but it is a necessary evil and better approached with explicit instruction.

Because this activity takes higher risks, changes each commit, involves vendor support and requires activities by each contributor, the support team should focus on preventing this from happening instead of fixing it.

Configure pre-commit hook as a preventive measure

As we have more Ansible tasks related, working directory becomes complicated and sometimes contributors accidentally committed large unwanted files (and pushed into the remote repo). Down the road, the best practice is to prevent contributors from committing junks.

The best spot to detect this should be a pre-receive hook on the server side, which is only available with self-hosted Bitbucket Server. Unfortunately, this is not a viable option for Bitbucket cloud. Our best bet is client-side pre-commit hook, in which a script performs size check when contributors run “git commit”. The purpose is to fail the commit if total file size is over the limit (20M), and the hook itself should be version controlled as well. Compared to (server side) pre-receive hook, the drawback of (client side) pre-commit hook is it requires initial client configuration. The upside is it captures large files before commit.

This hook can be a shell script as simple as this:

commitsizelimit=20
stagedfilelist=$(git diff --name-only --cached)
stagedfilecnt=`echo "$stagedfilelist"| sed '/^\s*$/d' |wc -l`
if [[ $stagedfilecnt -gt 0 ]]; then
 totalcommitsize=$(du -cm $stagedfilelist | tail -1 | cut -f 1)
 # Redirect output to stderr.
 exec 1>&2

 if [[ $totalcommitsize > $commitsizelimit ]]; then
 echo "Warning: Total size of all files in staging area is "$totalcommitsize"MB, exceeding the limit of "$commitsizelimit"MB."
 echo " To list files by size, run 'du -ch \$(git diff --name-only --cached)'"
 echo " To drop large ones from staging area with 'git rm -f filename'"
 echo " To bypass this limit, use 'git commit --no-verify'"
 exit 1
 fi
fi

In the repo we will have a .githook directory to store hooks (e.g. ~/source/.githooks/pre-commit) and point to the hooks directory using the following command:

git config core.hooksPath .githooks

Personal Vim cheatsheet

Mon, 07 Oct 2019 17:52:00 -0400

This is my personal cheatsheet as intermediate Vim user so I skipped the ones that I consider basic. All the commands listed are used in command mode for fast editing.

Command execution

In command mode, use colon to start ex command. Here are some examples of ex commands:

e: edit file
g: global command
q: quit
w: write
s: substitute

The commands (g)lobal and (s)ubstitute are heavily used in string manipulation. The rest are summarized here:

:e!	reload current file discarding all unsaved changes
:e newfile.txt	open file newfile.txt for editing
:e .	load current directory
:w!	force write (if permission allows)
shift + zz	equivalent to :wq!
shift + zq	equivalent to :q!

Operator

.	repeat last operation
fx	find next character x on the same line

Text Editing

y for yank(copy), i for inside, a for around, d for delete, w for word, p for paragraph or paste. Examples:

yyp	copy current line and insert after
diw	delete the entire word where the cursor sits in (dw deletes from cursor to end of word; db deletes from cursor to beginning of word)
shift + V	select entire line in visual mode (v selects character in visual mode)
2>	visual mode: indent twice on all selected lines edit mode: indent once for 2 lines
3<	visual mode: outdent three times on all selected lines edit mode: outdent once for 3 lines
di”	delete everything between the double quotes surrounding the cursor (exclusive); use c instead of d to finish the same effect with insert mode
di>	delete everything between < and > surrounding the cursor (exclusive); use (c)hange instead of (d)elete to finish the same effect with insert mode
dit	delete everything between tags. e.g. <xml>contenttodelete</xml>
dip	delete the entire paragraph
da’	delete everything between the single quote surrounding the cursor (inclusive); use c instead of d to finish the same effect with insert mode
da}	delete everything between { and } surrounding the cursor (inclusive); use c instead of d to finish the same effect with insert mode
dt.	delete all characters until the next .
i	insert at cursor location
shift + I	move cursor to first non-blank character of line and start in insert mode
a	insert at the location next to cursor
shift + A	move cursor to last non-blank character of line and start in insert mode
0	move to beginning of line. ^ moves to first non-blank character in the line. $ moves to the end of line

Note: wherever d is used in this table, c can be used instead for the same effect but switch to editing mode at the end.

String Manipulation

The general patterns are:

[range]g/pattern/cmd
[range]s/match/replacement/option

If range is not specified, it applies to current line only! To specify the whole file, use % range. You may also specify line range such as “10,20”. Here are some examples:

:%s/bacon/lettuce	For every line of the file, replace the first occurrence of bacon in each line to lettuce
:%s/bacon/lettuce/g	For every line of the file, replace all occurrences of bacon to lettuce
:s/bacon/lettuce	For current line, replace the first occurrence of bacon to lettuce
:s/bacon/lettuce/gi	For current line, replace all occurrences of bacon to lettuce, case insensitive
:g/bacon/d	delete all lines that contain pattern ‘bacon’
:g!/lettuce/d	delete all lines that do not contain pattern ‘lettice’; or use :v/lettuce/d instead
:g/^\s*$/d	delete all blank lines. \s* represents zero or more white spaces

Bookmarking

ma	mark cursor line as bookmark a
`a	jump to cursor position at line a
‘a	jump to beginning of line a
`.	jump to last line where change occurred
“	jump back

Insert Mode

Ctrl + N

Auto complete

Edit and run

While tmux and screen can help split screen in Bash, we sometimes need to split a bash screen to run a quick command when we’re already in vim. This can be done with some simple commands.

:term	Open up a terminal above vim. You can also spell :ter or :terminal
Ctrl+W; Ctrl+W	Press Ctrl+W twice can help you toggle between the terminal and vim buffer
Ctrl+D	Close the terminal

This can be very helpful when you are debugging code and need it run repeatedly. You don’t need to exit vim just to run a command and come back. Note that while you’re in terminal, you can’t use Ctrl+W as a shortcut key to backspace a word. Use Alt + Delete instead.

Git Explained 2 of 2

Tue, 18 Jun 2019 17:06:00 -0400

This is a continuation from Git Explained 1 of 2 where the fundamental concepts are covered. In this article we introduce some tools for customization and maintenance.

As for Git configuration, there are two files to dictate your Git configuration. ~/.gitconfig and .git/config in project directory. Running `git config –list –show-origin` shows all config entries and where they are from. For example you can custom your Git hooks location. Neither of the two files are being version controlled, so the configuration is only effective in the client environment,

Server side Git hooks

Git implementation supports server side hooks (pre-receive, update, post-receive). They are bash scripts placed in .git/hooks with specific names, fired upon event occurrence. Exit code of 1 from the scripts fails the event. Since server side Git hook consumes server resources, many repository hosting vendors (e.g. BitBucket Cloud) do not support it. You will need to enable it in self-hosted servers (e.g. BitBucketServer).

Client side Git hooks

Since server side hooks are not widely supported in every vendor, client-side Git hooks is good alternative places to implement functions such as code style check, commit size check, etc

The default directory for hooks is .git/hooks/ under the project directory is not version controlled and not easy to share with the team. If the hook needs to be shared among project contributors, we can place hooks files in .githooks/ under project directory. This will make the hook files version controlled. In addition, we need to point the hooks to this directory in configuration, by running `git config core.hooksPath .githooks` from project directory.

Web hooks

Web hooks can be thought of as an event notification mechanism. It is a common feature provided by VCS repository hosting providers. If a certain type of event occurs to the repo, web hook will fire an RESTful API call. The HTTP Endpoint, authentication secret and event payload are pre-configured in the repo settings. Web hooks are commonly supported by Git-based repo implementation, such as BitBucket cloud, GitHub, GitLab or AWS CodeCommit. It can also be enabled in self hosted Git repo. Web hook is a powerful tool to drive downstream event, such as Jenkins to start building the code. The major difference between server side hook and web hook is web hook is RESTful API driven, whereas server side hook is executing a script.

Pipelines

Some repo hosting vendor also provide a feature named pipeline. BitBucket has Pipeline as a CI/CD tool, AWS has AWS Code Pipeline, and GitLab offers CI/CD pipeline as well. These pipelines are usually in the form of a YAML file in the repo with a special name. The YAML spell out the steps to perform along the pipeline.

Squash Commits

For small projects I was in the habit of committing to main branch. I often need to squash a number of commits into one to “clean up”. Usually a Pull Request (e.g. in GitHub) or Merge Request (e.g. in GitLab) have such option during approval. We can squash a few commits with git command as well (suppose we want to squash the most recent 16 commits):

git rebase -i HEAD~16
git push origin +main

The command will open text editor to allow you to mark what to do with each commit. You can mark all except one commit as squash. For the commit to keep, mark it as pick. Then save the text editor. Git rebase will perform the squash for you. However, since this is a rebase, do not do this if there are other collaborators working on the same branch.

Cleanse a repository

Take BitBucket cloud for example, the size of a remote repo has a non-expandable hard limit of 2GB, and a soft limit of 1GB. Once the soft limit is reached, a warning will be displayed on Bitbucket’s landing page as well as when contributors pushes changes. Once the hard limit is reached, the entire repo will turn read-only mode.

Space consumption can be caused by accidental committing of large file. As covered in the previous post, files are stored as blob objects in .git directory. If a file was deleted by `git rm` command, it simply means it is de-referenced from the next commit and on. After all, Git as a distributed version control system, has the ability to magically restore the deleted file when we want. The cost of that magic, is that deleted file is permanently stored in the repo, in the form of blob object, although not present in the working directory. It consumes space not only in remote repo, but also in the local repo of each contributor.

The purpose of Git repository is to store source code which are fairly small texts. However if a contributor pushed in large files, it can be tricky to cleanse a Git repo on the remote side. Here is some guidelines:

Before cleansing, identify large unwanted files in current working directory, delete them with `git rm` and then commit this change in master branch.
Consolidate branches (e.g. delete useless remote branches with `git push origin –delete branch_name`). This step itself does not free up space in remote repo but it simplifies the branches;
Remove large blob objects and commit objects that reference them. This step essentially is re-writing the commit history of repo. Given the risk, it is recommended to perform this step from a separate local project directory with bare repo only without working directory. This article presents some good advices and illustrations. The command recommended is `git filter-branch -f –tree-filter “rm -rf \large_file.zip” –prune-empty — –all` ; and if that is too slow here is an alternative named BFG Repo-Cleaner.
After this step, each commit will have a new hash id. For the repo size to reduce, either wait until the next garbage collection on the server side, or request cloud vendor to run a manual garbage collection.

Set up automation with Ansible

Wed, 22 May 2019 22:11:00 -0400

Ansible is very flexible automation tools with many benefits. The free version is command-line based and here is an example to set it up.

Environment setup

Ansible 2.8 is required or some command may not work.
Ansible files (including playbooks, tasks and inventory files) are all located in /home/glowing/ansible
Default inventory file needs to be referenced in Ansible configuration /etc/ansible/ansible.cfg. This ensures ansible or ansible-playbook command can pick up hosts or host patterns without requiring inventory file through -i on every execution. Here is what the inventory config looks like in ansible.cfg:

[defaults]
 
# some basic default values...
inventory = /etc/ansible/hosts,/home/glowing/ansible/inventories/glowing_inventory.yml
 
host_key_checking = False

As best practice, servers involved should be able to ssh to each other on RSA key authentication. This can be achieved by adding a separate authorized keys file and reference it from /etc/ssh/sshd_config, at the line starting with AuthorizedKeysFile, and separated with the file name of existing authorized keys with a space. In this way, you may keep public keys of human user in one authorized key file and the public keys of machines in the other.
Build inventory file in ~/ansible/inventories/glowing_inventory.yml. The inventory file can declare some variables to use across hosts. If password is involved (e.g. synchronize module still requires password), it can be stored here with base64 encoded. Here is an example of glowing_inventory.yml

---
all:
 vars:
 ansible_user: glowing
 gh_sudo_pass: qGS0bWVuu3Jr
 gh_dir: /opt/glowing/etc
 tmp_dir: /tmp
 
 children:
 dc1_front_end:
 hosts:
 e9a-ghfe01:
 e9a-ghfe03:
 e9a-ghfe05:
 e9a-ghfe07:
 e9a-ghfe09:
 
 dc2_front_end:
 hosts:
 e9a-ghfe02:
 e9a-ghfe04:
 e9a-ghfe06:
 e9a-ghfe08:
 e9a-ghfe10:
 
 dc1_back_end:
 hosts:
 e9a-ghbe01:
 e9a-ghbe03:
 e9a-ghbe05:
 
 dc2_back_end:
 hosts:
 e9a-ghbe02:
 e9a-ghbe04:
 e9a-ghbe06:
 
 dc1_database:
 hosts:
 e9a-ghdb01:
 e9a-ghdb03:
 e9a-ghdb05:
 
 dc2_database:
 hosts:
 e9a-ghdb02:
 e9a-ghdb04:
 e9a-ghdb06:

That is the basic steps to set up Ansible. Now we can run adhoc commands. The command below allows me to copy a file from executing server to all destination servers that match a pattern:

# ansible 'dc*_back_end:!'`hostname -s` -e "file_name={{gh_dir}}/test.zip" -m copy -a "src={{file_name}} dest={{file_name}}"

In this command:

variable {{gh_dir}} is declared in the inventory file. It must be referenced by placing variable name between double curly bracket;
An extra variable {{file_name}} is declared at run time because this is the dynamic part of the command;
This adhoc ansible command uses copy module. With copy module, the src anddest files are in the same absolute path here so we use this variable to save some typing;
reference to the host support wildcard such as dc*_back_end;
hostname -s returns the host name of the server where the adhoc command is run
:! excluds the running host from being matched as destination server. this is in case that running machine is already in the dc*_back_end group, where copy source and destination are identical

Here is another example for deleting a file from destination servers:

# ansible dc2_database -e "fn=/tmp/file_to_delete" -m file -a "path={{fn}} state=absent"

If we can run adhoc command, then we can start writing some playbook and roles. Below is a simple playbook test-conn.yml to ping each server:

---
 - name: measure mint retrieval time from {{ ansible_limit }}
 hosts: '{{ ansible_limit }}'
 serial: 1
 order: sorted
 gather_facts: no
 ignore_errors: yes
 tasks:
 - name: measure time
 command: curl -s -XGET http://{{ inventory_hostname }}:8080/index.html -o /dev/null
 delegate_to: localhost
 register: curlout
 no_log: true
 - name: display result
 debug:
 msg: "Time to load webpage from {{ inventory_hostname }} is {{ curlout.delta }}."

Then you can run that playbook with the following command:

# ansible-playbook -l dc2_front_end test-conn.yml

Ansible automation is essentially editing yaml files for playbooks. Writing playbook involves a lot of module interaction and one needs to follow best practices. Understanding Ansible roles can help reuse some code.

In addition here are some common playbook keywords you should be familiar with:

serial
order
gather_facts
ignore_errors
when
run_once
local_action
register

Here are common modules to know

set_fact
state
touch
fail
synchoronize
lineinfile

Redhat Firewall configuration: from iptables to firewalld

Fri, 12 Oct 2018 17:56:00 -0400

Tools to manage firewall

Packet filter rules in Linux Kernel is managed by an user-space application named iptables in CentOS and RedHat. Since CentOS 7, firewalld is introduced as an alternative to iptables. Firewalld can be installed and executed as a systemd service, and it is supposed to replace iptables. This article describes how to configure both.

There are several advantages in firewalld. One is the support of zones. Here are some useful information. Also, iptables involves three different services for IPv4(iptables), IPv6(ip6tables), and software bridging (ebtables), whereas firewalld only involves a single service to manage all three. Firewalld allows user to add or remove rules/ports from running firewall, without restarting firewall. Unless you have specific reason to use iptables, always use firewalld service to manage firewall. Here is an instruction to firewalld service. In this posting however, we will be focusing on iptables to understand firewall managment. We also go through an example of opening a TCP port.

How does iptables work

When working with iptables, it is important to understand that its related concepts (tables->chains->rules->criteria and targets) and how the order of rules plays a factor. There are five independent tables, each contains a number of chains, either built-in or user-defined. Administrators mostly deals with built-in chains in filter and nat tables. The five tables are:

filter: If -t isn’t specified, this is the default table. It contains built-in chains:
- INPUT: for packet destined to local sockets
- FORWARD: for packets being routed through the box
- OUTPUT: for locally-generated packets
nat: this table is consulted when a packet that creates a new connection is encountered. It has three built-in chains:
- PREROUTING: for altering packets as soon as they come in
- OUTPUT: for altering locally generated packets before routing
- POSTROUTING: for altering packets as they are about to go out
mangle: this table is used for specialized packet alternation, with five built-in chains (since kernel 2.4.18): PREROUTING and OUTPUT, INPUT, FORWARD, and POSTROUTING
raw: this table is mainly for configuring exceptions from connection tracking with two built-in chains: PREROUTING and OUTPUT
security: for Mandatory Access Control (MAC) networking rules, with three built-in chains: INPUT, OUTPUT, and FORWARD.

Under the table (e.g. filter, nat), each chain (e.g. INPUT, OUTPUT, etc) consists of list of firewall rules. Each rule is made up of two parts defined for the packets:

Criteria: if the packet does not match the criteria, the next rule in the chain is examined; if it does match, then the next rule is specified by the value of the target.
Target: what to do if criteria is met. The target can be:
- user-defined chain,
- one of the target described in iptables-extensions, or
- in most cases, one of the special values ACCEPT, DROP or RETURN
  - ACCEPT – let the packet through
  - DROP – drop the packet on the floor
  - RETURN – stop traversing this chain, and resume at next rule in the previous (calling) chain

The rules, defined in each chain under their tables, can be found in file /etc/sysconfig/iptables. You can find tables (prefix with asterisk *), chains (prefix with colon :), rules under their chains and a statement COMMIT after each table. The iptables process flow illustrates how a packet interact with all these rules under different chains and tables defined in this file:

iptables Process Flow

Although this big picture looks formidable, an administrator commonly only deals with the green and purple blocks (filter and nat), with the big picture in mind. Here is an example of /etc/sysconfig/iptables file from a newly installed system:

# Generated by iptables-save v1.4.21 on Fri Sep 11 23:15:32 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [132:17200]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Fri Sep 11 23:15:32 2017

The rule simply allows SSH traffic. This file will be loaded up on every reboot (specifically, restart of iptables service). So if you have made some changes to rules and you want the change picked up on reboot. The rules should be saved to this file:

$ sudo iptables-save > /etc/sysconfig/iptables

Other than saving rule for reboot, if you simply want to edit the rules (e.g. order of rules is incorrect), you can save the rules to file, modify the file and restore the rule from file:

$ sudo iptables-save > ~/iptables.txt
$ sudo vi ~/iptables.txt
$ sudo iptables-restore &lt; ~/iptables.txt

Here is some further reading about iptables architecture.

Anatomy of a rule

The man page for iptables species the following synopsis:

iptables [-t table] {-A|-C|-D} chain rule-specification
rule-specification = [matches...] [target]
match = -m matchname [per-match-options]
target = -j targetname [per-target-options]

So when you append (-A), delete (-D), insert (-I) or replace (-R) a rule, you need to specify rule specification. The man page further explains that the following parameters make up a rule specification:

protocol (-p): the protocol of the rule of the packet to check. value can be tcp, udp, icmp, all or any name defined in /etc/protocols.
match (-m): specifies the name of a match to use and is followed by match options. The match refers to an extension module that tests for a specific property. Those extension modules are documented in the man page of iptables-extensions. You may specify -m multiple times for different match names, which together make up the condition under which a target is invoked. Matches are evaluated first to last as specified. We often use extensions tcp and state. According to iptables-extensions man page, we can specify –dport followed by port number for the tcp extension, and –state followed by value such as NEW or ESTABLISHED for the state extension.
jump (-j): specifies the target of the rule, such as ACCEPT, REJECT or DROP.
source and destination (-s and -d): source and destination IP address or masks. Hostname will work but not recommended since resolution is needed.
inbound and outbound interface (-i and -o): name of interface via which the packet was received and is going to be sent.
goto (-g): processing should continue in a user specified chain
Other parameters: -4/–ipv4, -6/–ipv6, -c/–set-counters, -f/–fragment

When we run iptables command to view rules, we need to specify the table (e.g. filter, nat, etc) followed by -S or –list-rules:

$ iptables -t nat -S

If you do not specify -t switch, the default (-t filter) is applied. Be aware that in this case, you’re only seeing rules under filter table, and not all rules under tall tables!

In the result, for example one line from command “iptables -S” may say:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 9200 -j ACCEPT

The interpretation: appending a rule to INPUT chain of filter table (implicitly specified). The protocol is tcp. The first match extension is state, and the state value shall be NEW. The second match extension is tcp, and the dport value shall be 9200. If the packet is a match, then the target (action) is ACCEPT.

Managing rules

As mentioned earlier, rules can be dumped to any file or /etc/sysconfig/iptables, in which the rules are assessed in order. Below is a real life iptables file with a nat table as well.

# Generated by iptables-save v1.4.21 on Wed Jan 15 13:58:39 2017
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [4:208]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7000:7001 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7199 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9042 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9160 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 161 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 162 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

*nat
:PREROUTING ACCEPT [1:328]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398
-A PREROUTING -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398
-A OUTPUT -o lo -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398
-A OUTPUT -o lo -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398
COMMIT
# Completed on Wed Jan 15 13:58:39 2017

In this example, the nat table defines traffic forwarding: traffic arriving at TCP port 2392 and 2393 are forwarded to port 2398; outgoing traffic to port 2392 and 2393 are also redirected to port 2398. These rules do not overlap each other so the rules probably don’t matter.

On the other hand, the tcp filter table lists the rules to open certain TCP and UDP ports. Its block starts with a couple accepting rules and ends with a couple reject rules (regardless of protocols or ports). This is a good way to close a chain of rules with security. However, if you need to add additional rules to open more TCP ports, the new rule should not be appended after the reject rules at the bottom since the order matter here!

Correct way to open a TCP port

It’s a common task for developers to open a TCP port simply for the purpose of bring up a web service and make it accessible to client. If we simply add a new rule to existing list, for example:

# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9870 -j ACCEPT
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

# systemctl reload iptables

You will notice that the rule is appended to the end of INPUT block, below the INPUT REJECT rule, which will never take effect.

To address this, you can use iptables-save and iptables-restore to export, edit to correct order and reload the rule, as illustrated above, instead of using iptables command to modify the rule directly.

# iptables-save > /tmp/rule.list
# vi /tmp/rule.list
# iptables-restore < /tmp/rule.list
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

Alternatively, you could use some advanced iptables command switches to add the new rule to certain line number with –line-number switch. Here is more information.

Log shipping through ELK

Wed, 19 Sep 2018 17:26:51 -0400

A common devops task is build logging pipeline with ELK stack (Elasticsearch, Logstash, Kibana). Suppose the application is written in Java and currently use log4j’s RollingFileAppender to generate log files locally. We can use log4j’s socket appender to write to Logstash, which further pushes the log stream to Elasticsearch cluster.

In this model, failure to push a log line results in the following in log4j output:

2018-09-06 11:05:39,778,ERROR,stderr - [AsyncAppender-Dispatcher-Thread-672] log4j:WARN Detected problem with connection: java.net.SocketException: Broken pipe (Write failed)

The logstash log displays the socket exception as well:

[2018-09-06T10:12:21,935][DEBUG][logstash.inputs.log4j ] Accepted connection {:client=>"192.168.111.56:58118", :server=>"0.0.0.0:4560"}
[2018-09-06T10:12:21,963][DEBUG][logstash.pipeline ] filter received {"event"=>{"method"=>"?", "thread"=>"676774870@qtp-1804103302-6", "message"=>"Unable to resolve session ID from SessionKey [org.apache.shiro.web.session.mgt.WebSessionKey@71da9a4]. Returning null to indicate a session could not be found.", "priority"=>"DEBUG", "type"=>"log4j", "path"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "@timestamp"=>2018-09-06T15:12:21.950Z, "file"=>"?:?", "@version"=>"1", "host"=>"192.168.111.56:58118", "logger_name"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "class"=>"?", "timestamp"=>1536246741950}}
[2018-09-06T10:12:21,963][DEBUG][logstash.pipeline ] output received {"event"=>{"method"=>"?", "thread"=>"676774870@qtp-1804103302-6", "message"=>"Unable to resolve session ID from SessionKey [org.apache.shiro.web.session.mgt.WebSessionKey@71da9a4]. Returning null to indicate a session could not be found.", "priority"=>"DEBUG", "type"=>"log4j", "path"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "@timestamp"=>2018-09-06T15:12:21.950Z, "file"=>"?:?", "@version"=>"1", "host"=>"192.168.111.56:58118", "logger_name"=>"org.apache.shiro.session.mgt.DefaultSessionManager", "class"=>"?", "timestamp"=>1536246741950}}
[2018-09-06T10:12:22,041][DEBUG][logstash.inputs.log4j ] Closing connection {:client=>"192.168.111.56:58118", :exception=>java.io.InvalidObjectException: Object type java.util.Hashtable is not allowed.}

The troubleshooting isn’t very straightforward. So its alternative is preferred. The alternative is to keep the existing RollingFileAppender as well as the local log files, but use a filebeat agent for each application node. Here is the diagram:

ELK Log Shipping Pipeline

Filebeat is a very light agent to be installed with the application on the same server or container.
Streams of text information congregate to logstash nodes. These log stash service pushes the converged stream to ElasticSearch cluster
ElasticSearch cluster ingest the stream for indexing
Kibana is responsible for viewing.

This is an example architecture with many potential variations. For example, log stash service may be deployed on the same hosts with ElasticSearch cluster. If there aren’t many application nodes, filebeat may directly push its output to ElasticSearch. This architecture provides a lot of flexibility and scalability.

With log4j format, two challenges to address are:

identifying multi-line entry in the log
mapping sections of a log entry into fields in Elastisearch

For example, the following log entry reflects both challenges above:

2018-09-06 11:52:18,022,DEBUG,service.dataportal.web.retrieve - [687285925@qtp-564051174-3350] creating streaming output request 9868594 and requid=fb9a2cad-e24b-4eb8-ad62-b027184e7b9b
2018-09-06 11:52:18,054,ERROR,org.glassfish.jersey.server.ServerRuntime$Responder - [687285925@qtp-564051174-3350] An I/O error has occurred while writing a re
sponse message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.mortbay.jetty.EofException
 at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135)
 at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662)
 at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395)
 at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280)
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
 at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
 at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.Server.handle(Server.java:326)
 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926)
 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
 at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
 at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
 at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.mortbay.jetty.EofException
 at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:789)
 at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:568)
 at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1010)
 at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.flush(ResponseWriter.java:330)
 at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:287)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.flush(WriterInterceptorExecutor.java:305)
 at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79)
 at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.spi.ContentEncoder.aroundWriteTo(ContentEncoder.java:137)
 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
 at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
 ... 40 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
 at java.net.SocketOutputStream.socketWrite0(Native Method)
 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
 at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
 at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368)
 at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:122)
 at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:723)
 ... 58 more

To address multi-line entry we need to tell filebeat how to identify the start of a line through its multiline.pattern configuration option:

- type: log
 enabled: true
 paths:
 - /var/log/dhunch/app.log
 multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'
 multiline.negate: true
 multiline.match: after
 fields:
 product:
 name: dhunchapp
 log:
 type: app_log
 content: diagnostic

When filebeat pushes log entries to logstash, there will be an additional “time” column added, which reflects the time the log is ingested. The entire log4j message line is all put in a single field called “message”, this message field contains the timestamp from application, the thread, class, logging level and the actual diagnostic message. These are not mapped to separate sections making it difficult to search in Elasticsearch.

To address section mapping, the configuration is made on the logstash side. We will take advantage of a plugin called grok filter. The function of grok filter here, is to parse the original “message” field, map different sections into separate columns, respectively called logtime, loglevel, logclass, logthread and logmsg. The grok filter expression is as below:

filter {
 grok {
 match => { "message" => "%{TIMESTAMP_ISO8601:logtime},%{LOGLEVEL:loglevel},%{NOTSPACE:logclass} - \[%{DATA:logthread}\] %{GREEDYDATA:logmsg}" }
 }
 date {
 match => [ "logtime" , "yyyy-MM-dd HH:mm:ss,SSS" ]
 timezone => "America/Chicago"
 target => "@timestamp"
 }
 mutate {
 replace => [ "message" , "%{logmsg}" ]
 }
 mutate {
 remove_field => [ "logmsg" ]
 }
}

The date plugin indicates to Elasticsearch to treat this field as datetime with specified time zone. The mutate plugin below essentially renames the “logmsg” column to “message”, which allows Elasticsearch to understand this field as log message.

cron and anacron in RedHat Linux (How logrotate works)

Tue, 15 May 2018 14:11:00 -0400

Cron and anacron

We all know cron is a job scheduler. Many admin uses crontab to manage scheduled task. It is also important to know that crontab works at different levels as well, as well as the distinction between cron and anacron. They are similar, but different, managed by different sets of files.

Below is a brief description of how cron works from this article.

After Cron starts, it searches its spool area to find and load crontab files into the memory. It additionally checks the /etc/crontab and or /etc/cron.d directories for system crontabs.
After loading the crontabs into memory, Cron checks the loaded crontabs on a minute-by-minute basis, running the events which are due.
In addition to this, Cron regularly (every minute) checks if the spool directory’s modtime (modification time) has changed. If so, it checks the modetime of all the loaded crontabs and reloads those which have changed. That’s why we don’t have to restart the daemon when installing a new cron job.

Basically, in cron, you specify a particular time at which a job will run. These jobs are managed by files in /var/spool/cron/ directory. In this directory, each file is named by the username that owns the crontab file. These files shall not be edited directly by respective users. Instead, they are edited by crontab by each user. To understand the syntax, one can refer to RedHat document for automating system tasks. Note that you can specify periodical jobs here with special syntax. For example, */5 at the minute slot indicates every five minutes.

Running cron jobs can be allowed or disallowed for different users. For this purpose, use the /etc/cron.allow and /etc/cron.deny files. If the cron.allow file exists, a user must be listed in it to be allowed to use cron If the cron.allow file does not exist but the cron.deny file does exist, then a user must not be listed in the cron.deny file in order to use cron. If neither of these files exists, only the super user is allowed to use cron.

In addition, there is a system-wide crontab file in /etc/crontab, in which you need to not only specify tasks, but also the user to run those tasks. By default, the schedule in this file is empty.

The limitation of cron is it assumes the servers are up all the time, if a script misses the schedule while the server is down, it will not be executed when the server comes back up. This is where anacron comes in handy. Although anacron can only be used by superuser, it doesn’t expect system to be running 24×7. If a job is scheduled at a time system is down, it starts the job when system comes back up.

Both cron and anacron are run by systemd service named crond.service. Although they require different packages installed (cronie vs cronie-anacron). They are also managed by different sets of files as explained below:

package	file or directory	purpose	example
cronie	/var/spool/cron/	this directory accommodates files that represents cron jobs for each individual users.	if a file named digihunch contains a valid line, it means that Linux user digihunch has a scheduled task for the time specified.
cronie	/etc/crontab	This file keeps system-wide cronjob entries. Each line needs to sepcify users.	if a line specifies schedule, user and command, it means that at the scheduled time, that user will execute the command.
cronie-anacron	/var/spool/anacron/	This directory accommodates files such as cron.daily, in which a timestamp is kept to indicate last execution time.	if cron.daily in this directory reads 20180418, it indicates last daily execution time stamp is 20180418
	/etc/anacrontab	This file tells anacron where in the file system to go for directories for periodical jobs.	Example: `#period in days delay in minutes job-identifier command 1 5 cron.daily nice run-parts /etc/cron.daily 7 25 cron.weekly nice run-parts /etc/cron.weekly @monthly 45 cron.monthly nice run-parts /etc/cron.monthly` The file usually also indicates RANDOM_DELAY and START_HOURS_RANGE
	/etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/	These directories store script files that anacron needs to execute at different intervals. this is configured in /etc/anacrontab	if script logrotate is present in /etc/cron.daily/, it means the script is to be executed daily

An anacron example: logrotate

Rotating logs is a common task in Linux that can be done by logrotate. To understand how this works, first, make sure cronie-anacron package is installed and crond.service is up. Then examine the /etc/anacrontab file:

# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days delay in minutes job-identifier command
1 5 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
@monthly 45 cron.monthly nice run-parts /etc/cron.monthly

This indicates that daily, weekly and monthly jobs are active. Go into /etc/cron.daily/, and examine script logrotate:

#!/bin/sh

/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
 /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

This indicates that logrotate loads configuration from /etc/logrotate.conf, the man page of logrotate explains how this configuration works, along with an example. If you have any custom application where the log file needs rotated, it can be configured in this file.

CentOS – Remove Swap safely

Sat, 14 Apr 2018 16:15:43 -0400

If the default installation has swap on, you will see it from block device list:

[root@server /]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 500G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 499G 0 part
 ├─centos-root 253:0 0 50G 0 lvm /
 ├─centos-swap 253:1 0 7.9G 0 lvm [SWAP]
 └─centos-home 253:2 0 441.1G 0 lvm /home
sr0 11:0 1 1024M 0 rom
[root@server /]# free -h
 total used free shared buff/cache available
Mem: 7.6G 217M 7.2G 11M 206M 7.2G
Swap: 7.9G 0B 7.9G

Many installations require swap to be off for performance reasons (although some advocate turning off swappiness of the application, instead of removing swap partition from operating system, which is a separate topic). This can be turned off by a simple command:

swapoff -a

Then if you run free command after rebooting, you will see the Swap is set to 0G. Great. But lsblk and df commands still shows the space being used by swap. This disk space is not being used at all.

[root@c7v-vitvcast02 ~]# free -m
 total used free shared buff/cache available
Mem: 7802 237 7380 11 185 7333
Swap: 0 0 0
[root@c7v-vitvcast02 ~]# lvdisplay
 --- Logical volume ---
 LV Path /dev/centos/swap
 LV Name swap
 VG Name centos
 LV UUID 7zHtvT-zGYn-sNBc-JRlT-nTfU-O9hN-r07y3J
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:21 -0500
 LV Status available
 # open 0
 LV Size <7.88 GiB
 Current LE 2016
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:1

 --- Logical volume ---
 LV Path /dev/centos/home
 LV Name home
 VG Name centos
 LV UUID 4XMTa7-uR44-qH0u-Oag4-PDyK-Z1BC-3oUSiz
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:22 -0500
 LV Status available
 # open 0
 LV Size <441.12 GiB
 Current LE 112926
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:2

 --- Logical volume ---
 LV Path /dev/centos/root
 LV Name root
 VG Name centos
 LV UUID tkMCM4-cWxV-FaAC-hgJx-r1vS-DiY5-fyZvjt
 LV Write Access read/write
 LV Creation host, time localhost, 2018-04-07 10:03:22 -0500
 LV Status available
 # open 1
 LV Size 50.00 GiB
 Current LE 12800
 Segments 1
 Allocation inherit
 Read ahead sectors auto
 - currently set to 8192
 Block device 253:0

We can then remove the line for swap from /etc/fstab, this will keep the OS from mounting it upon reboot. However, the logical volume for swap is there, as a partition that’s not exposed.

If you need to reclaim space from swap partition, delete the logical volume:

lvremove /dev/centos/swap
Do you really want to remove active logical volume centos/swap? [y/n]: y
 Logical volume "swap" successfully removed

Now, using command lvdisplay or lvs, the swap space is not displayed anymore. So I rebooted the server in the hopes that the swap space is not being presented and everything is hunky-dory. But no… the server doesn’t boot up at all. It took a long time at the splash screen only to enter dracut, where /boot isn’t present.

On further reading, this is because the grub file is still referencing /dev/centos/swap somewhere. I need to somehow get to the grub file and fix that before it can reboot again. Specifically, I pressed “e” at the menu, which gave me an opportunity to edit grub file. From there I removed section for swap. Then I saved it with “Ctrl + X” (as suggested on the screen) so the OS booted into normal mode. Alternatively, I entered single user mode in order to get to the point I can edit the grub file.

Once booted, I have to make change to the actual grub file at /boot/grub2/grub.cfg so the change persists. In the file, locate the lines starting with linux16 and remove the section “rd.lvm.lv=centos/swap”. There are two appearances of this section on my machine. Then the server will boot just normally.

In summary, the clean way to turn off swap on CentOS involves the following steps:

turn off swap
Remove swap mount from /etc/fstab (and umount)
remove the logical volume for swap
remove reference to swap from grub

Out of those steps, 3 and 4 are needed to reclaim swap spaces. They must be done together. If you rebooted the server before step 4. The server will not be able to boot complaining about missing swap.

TCPdump and Wireshark configuration

Wed, 28 Feb 2018 18:30:06 -0400

This article explains how to troubleshoot TCP packet from Linux (CentOS) and Windows with TCP dump and wireshark. Both are important tools for troubleshooting. If you are troubleshooting a Windows server and have access to it to install Wireshark then there is nothing to worry about. Even if the server to troubleshoot is a Linux one with proper desktop (KDE/GNOME), you may still install the Wireshark UI on it and work from the server. If the server is Linux without any UI, this is where this article is trying to help because you need to run tcpdump on the server and somehow download the capture to your local computer for analysis.

If you work off of a MacOS, and need to capture in real time from a Linux server without a desktop (KDE/GNOME), then the best bet is to run tcpdump remotely from the server and pipe the result into Wireshark. This would require root access to the server. Tcpdump will require libpcap and tcpdump packages. Then from MacBook you can run:

# ssh root@remote-server "tcpdump -w - -s0 -pi eth0 dst port 443 or src port 443"|wireshark -k -i -

This will pipe the tcpdump result into Wireshark session in Mac in real time with a delay.

If you work off a Windows computer where plink.exe is available, you can run the following command if you know the root password:

C:\tools\plink.exe -l root -pw rootpassword 192.168.117.12 -P 22 "tcpdump -w - -s0 -pi eth0 dst port 9042" |"C:\Program Files\Wireshark\Wireshark.exe" -k -i -

Both tricks above assumes that you have direct root log-in to the server, by RSA key or password. It is because running tcpdump requires root access on the server. It is not a good security practice to run tcpdump with a non-root user because it needs to scan the interface.

-s: snap length in bytes. Setting to 0 is making it use default 65535
-i: specify the interface to listen on. e.g. eth0 or ens192
-p: no-promiscuous mode. this option asks tcpdump to not put interface in promiscuous mode
-w: write the raw packets to file rather than parsing and printing them out. a hyphen indicates standard output here.
-Z: drops the privileges of root and changes ownership to the specified user

If you do not have direct root login access, but you can log in as a different user and su to root, you may run this once you are on root user:

echo "###Capture Begin: $(date '+%Y %b %d %H:%M:%S')" && tcpdump dst port 1524 or src port 1524 -s 0 -i eth0 -w "/tmp/cap.$(date +%Y%m%d_%H%M%S).cap" -Z linuser && echo "###Capture End: $(date '+%Y %b %d %H:%M:%S')" && ls -ltr /tmp/cap*.cap

To stop capture, you can use Ctrl-C but make sure that is passed to the server terminal or you will leave a zombie tcpdump process

Field Notes on Digi Hunch

The Leanest Web and Email Hosting

WordPress Hosting

WordPress Migration

CloudFlare Security

CloudFlare Cache Rules

SMTP service

Sending Reputation

Summary

Debating between count and for_each in Terraform

Problem with count loop

Embrace for_each loop

A real-life example with classic pattern

Implementation using count

Implementation with for_each: first attempt

Is for_each a bad idea?

Implementation with for_each: second attempt

One more shot with for_each

Conclusion

Test Open ID Connect Flows Locally

Architecture

Choice of Tools

Configuration

Summary

IAM Roles for any workload

Background

IAM Role Introduction

The IMDS service

IAM Role for EC2 Workload

IAM Role Anywhere Concept

IAM Role Anywhere Lab

Summary

Managing EC2 instances across accounts with Ansible

Prerequisites

Configure AWS CLI

Configure AWS CLI Profiles

Configure SSH to EC2 via SSM

Configure Ansible Inventory

Conclusion

WordPress Security Basics

Background

The Incident

Clean up and hardening

Back Links

Repercussions

CloudFlare

Lessons Learned

AWS Systems Manager is an Omnipotent Hodgepodge

Introduction to Systems Manager

SSM Agent and Session Manager

Types of SSM Documents

RunCommand and Automation

Maintenance Window and State Manager

Fleet Manager and Inventory

Patch Manager and Compliance

Other capabilities

Summary

Authentication to kube-apiserver via OIDC

Background

Vanilla Kubernetes

Step-by-step with kubelogin

Azure Kubernetes Service

ROSA

Elastic Kuberentes Service

Summary

Connect kubectl to private Kubernetes cluster in EKS and AKS

AWS options

EC2 Instance Connect

SSM Session Manager

SOCKS5 proxy for kubectl

What about AKS in Azure

Summary

Kubernetes with Multiple CPU Architectures 2 of 2 – Node and Workload

ARM64 architecture

Worker Node

Workload

Utilities

Scheduling

Summary

Kubernetes with Multiple CPU Architectures 1 of 2 – Container Image

Problem with `count` loop

Embrace `for_each` loop

Implementation using `count`

Implementation with `for_each`: first attempt

Is `for_each` a bad idea?

Implementation with `for_each`: second attempt

One more shot with `for_each`