infrastructure as code on Digi Hunch

Debating between count and for_each in Terraform

Tue, 27 Aug 2024 22:36:39 -0400

In Terraform, we often have to create an array of resources of the same type but similar attribute values. For code reusability, manageability and for DRY principle, it’s better to use loop. Terraform HCL supports loop via the use of meta-argument. Currently, there are two options to drive a loop: count and for_each .

Problem with `count` loop

The book Terraform Up and Running (Chapter 5 Terraform Tips and Tricks) regards count as Terraform’s oldest, simplest and most limited iteration construct. One of the big limitations is the shifting of index if the length of resource array changes. The point comes with a good example:

variable "user_names" {
 description = "Create IAM users with these names" 
 type = list(string)
 default = ["neo", "trinity", "morpheus"]
}
# The Example from the book Terraform Up and Running
resource "aws_iam_user" "example" { 
 count = length(var.user_names)
 name = var.user_names[count.index]
}

As you execute Terraform apply, three IAM users will be created, with the plan looking like:

# aws_iam_user.example[0] will be created
+ resource "aws_iam_user" "example" {
 + name = "neo"
 (...) 
 }
# aws_iam_user.example[1] will be created
+ resource "aws_iam_user" "example" {
 + name = "trinity"
 (...) 
}
# aws_iam_user.example[2] will be created
+ resource "aws_iam_user" "example" {
 + name = "morpheus"
 (...) 
}

Then if you remove “trinity” from the variable user_names, and run terraform plan, the plan would look like:

Terraform will perform the following actions:
 # aws_iam_user.example[1] will be updated in-place
 ~ resource "aws_iam_user" "example" {
 id = "trinity"
 ~ name = "trinity" -> "morpheus"
 }
 # aws_iam_user.example[2] will be destroyed
 - resource "aws_iam_user" "example" {
 - id = "morpheus" -> null
 - name = "morpheus" -> null
 }
Plan: 0 to add, 1 to change, 1 to destroy.

In this plan, instead of deleting the second user, it renames the second user and deletes the third user. While the plan matches the code logic, it is often an unwanted result, considering the resource could be one that many other resources depends on, such as a subnet.

This is a good example of the problem with count. Terraform identifies each resource in the generated list of resource by position(index) . When the length changes, the index shifts. If you remove an item from the middle of the list, Terraform will delete every resource after the deleted item, then re-create all the resources that come after the deleted one. As a consequence, you may loose availability or even worse, lose data.

Embrace `for_each` loop

If we modify the example above to use for_each, the code looks like:

variable "user_names" {
 description = "Create IAM users with these names" 
 type = list(string)
 default = ["neo", "trinity", "morpheus"]
}
# The Example from the book Terraform Up and Running
resource "aws_iam_user" "example" 
{ 
 for_each = toset(var.user_names) 
 name = each.value
}

This results in the creation of three IAM users. If you remove the “trinity” user from the middle of the input collection and apply, the plan looks like this:

Terraform will perform the following actions:
 # aws_iam_user.example["trinity"] will be destroyed
 - resource "aws_iam_user" "example" {
 - arn = "arn:aws:iam::123456789012:user/trinity" -> null
 - name = "trinity" -> null
 }
Plan: 0 to add, 0 to change, 1 to destroy.

The plan suggests that Terraform will delete the very resource that was taken out from the middle of the input collection and no existing resources in the array are impacted.

Note that in the code snippet above, we use function toset() to convert the input list to a set (ordered and de-duped list of string). This is because we can only loop over a set or map when creating an array of resource. If the array of resource being created have another attribute whose value needs to be individualized, we can loop over a map and store the individualized attribute values as key-value pairs.

A few pages down, the book discusses an important limitation for both count and for_each. The length of the resource array that you are creating with count or for_each meta-argument must not be computed from other resources. Terraform must be able to compute count and for_each during the plan phase, before any resources are created or modified. The length of the resource array can be from hardcoded values, data sources, or even a list of other resources to create in the same file, so long as the length can be determined during the plan, instead of not being computed from other resource outputs.

A real-life example with classic pattern

The book then touches on another advantage of for_each: the ability to create multiple inline blocks within a resource. The guide from Hashicorp documentation also has a section on when to use for_each Instead of count, with a similar example. The section merely mentions when to use count in the opening sentence: If your instances are almost identical, count is appropriate.

That makes for_each sound like a no-brainer, after reading all the literatures about this topic. In my experience with a specific use case at the beginning, count feels more efficient. The example from the book is too simplistic. To better compare the two options, I need a realistic example. Let’s consider this use case where, after creating a VPC, I need to create the followings:

one NAT gateway for each availability zone (each NAT Gateway maps to one subnet and one allocation ID)
one public subnet for each availability zone
one public IP allocation in each availability zone

We can summarize the relationships between resources in the following diagram:

I deliberately pick this example because they are self-contained. So are all code snippets in this post. The example also demonstrate a classic relation between resources that we can find everywhere in infrastructure automation. Here’s another example off the bat:

create an array of aws_subnet, each has a subnet_id attribute;
create an array of aws_route_table, each has a reout_table_id attribute;
now, create an array of aws_route_table_association, each referencing one aws_subnet (by subnet_id) and one aws_route (by route_table_id);

If we address the NAT gateway example, we’re good with many other resources that shares the same relation pattern. In the next section, we’ll first implement the NAT gateway example, using count loop.

Implementation using `count`

The use case exemplifies the pattern where we have multiple types of resources related to each other. We need a loop in each type of resources, resulting in multiple arrays of different resource types. Moreover, the elements in the array for aws_nat_gateway has 1-to-1 mappings with both the array for aws_subnet, and the array for aws_eip.

With count, I created Terraform code with everything in a single main.tf file for the convenience of illustration, like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}
variable "public_subnets_cidr_list" {
 type = list(any)
 default = ["147.206.0.0/22", "147.206.4.0/22"]
 #default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"]
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

resource "aws_subnet" "public_subnets" {
 count = length(var.public_subnets_cidr_list)
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = var.public_subnets_cidr_list[count.index]
 map_public_ip_on_launch = true
 availability_zone = data.aws_availability_zones.this.names[count.index]
}

resource "aws_eip" "nat_eips" {
 count = length(var.public_subnets_cidr_list)
}

resource "aws_nat_gateway" "nat_gws" {
 count = length(var.public_subnets_cidr_list)
 subnet_id = aws_subnet.public_subnets[count.index].id
 allocation_id = aws_eip.nat_eips[count.index].id
 depends_on = [aws_internet_gateway.internet_gw]
}

The intent is to create subnet, public IP and NAT gateway for two availability zones. I also want to add one more AZ in the future and have the code to handle the addition gracefully. To add the new AZ, I uncomment line 10 and comment out line 9. The plan after this code change looks like this:

Terraform will perform the following actions:

 # aws_eip.nat_eips[2] will be created
 + resource "aws_eip" "nat_eips" {
 (...) 
 }

 # aws_subnet.public_subnets[2] will be created
 + resource "aws_subnet" "public_subnets" {
 + availability_zone = "us-east-1c"
 + cidr_block = "147.206.8.0/22"
 + id = (known after apply)
 (...) 
 }

 # aws_nat_gateway.nat_gws[2] will be created
 + resource "aws_nat_gateway" "nat_gws" {
 + allocation_id = (known after apply)
 + subnet_id = (known after apply)
 (...) 
 }

Plan: 3 to add, 0 to change, 0 to destroy.

The plan creates a set of resources required for the new availability zone without touching any existing resource, which is expected.

Why do I only focus on the use case of adding a new subnet in new AZ, and not deleting or modifying CIDR on an existing subnet? That’s because we rarely do that with production. We rarely remove the use of an availability zone. Nor do we modify the CIDRs on an existing subnet. In fact, AWS SDK does not even have an API to change CIDRs on a subnet or a VPC. In our infrastructure operation, we make such decisions upfront so they are immutable once provisioned. We simply don’t need to consider all the possible CRUD actions on a resource.

So, the count loop does just the job. Now, what about for_each?

Implementation with `for_each`: first attempt

Since for_each takes a set or map, I have to make some adjustment. My first attempt looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}

variable "public_subnets_cidr_list" {
 type = list(any)
 default = ["147.206.0.0/22", "147.206.4.0/22"] # 2 AZ
 #default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"] # 3 AZ
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

locals {
 subnet_config = [
 for i in range(length(var.public_subnets_cidr_list)) : {
 cidr = var.public_subnets_cidr_list[i]
 az = data.aws_availability_zones.this.names[i]
 }
 ]
}

resource "aws_subnet" "public_subnets" {
 for_each = { for idx, rec in local.subnet_config : idx => rec }
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = each.value.cidr
 map_public_ip_on_launch = true
 availability_zone = each.value.az
 tags = { Name = "PUBLIC-SUBNET" }
}

resource "aws_eip" "nat_eips" {
 for_each = toset(var.public_subnets_cidr_list)
 tags = { Name = "NATEIP" }
}

data "aws_subnets" "public_subnets" {
 filter {
 name = "tag:Name"
 values = ["PUBLIC-SUBNET"]
 }
 depends_on = [aws_subnet.public_subnets]
}

data "aws_eips" "nat_eips" {
 filter {
 name = "tag:Name"
 values = ["NATEIP"]
 }
 depends_on = [aws_eip.nat_eips]
}

locals {
 nat_gw_config = [
 for i in range(length(var.public_subnets_cidr_list)) : {
 subnet_id = data.aws_subnets.public_subnets.ids[i]
 alloc_id = data.aws_eips.nat_eips.allocation_ids[i]
 }
 ]
}

resource "aws_nat_gateway" "nat_gws" {
 for_each = { for idx, rec in local.nat_gw_config : idx => rec }
 subnet_id = each.value.subnet_id
 allocation_id = each.value.alloc_id
 depends_on = [aws_internet_gateway.internet_gw]
}

Note that I have to create a couple of data resources (nat_eips and public_subnets) and local variables (subnet_config and nat_gw_config) in order build the required map data structures and feed them to the for_each parameters.

After Terraform apply, let’s edit public_subnets_cidr_list with the additional subnet CIDR for the 3rd AZ. The plan looks like this:

Terraform will perform the following actions:

 # data.aws_eips.nat_eips will be read during apply
 # (depends on a resource or a module with changes pending)
 <= data "aws_eips" "nat_eips" {
 + allocation_ids = (known after apply)
 (...) 
 }

 # data.aws_subnets.public_subnets will be read during apply
 # (depends on a resource or a module with changes pending)
 <= data "aws_subnets" "public_subnets" {
 + id = (known after apply)
 + ids = (known after apply)
 (...) 
 }

 # aws_eip.nat_eips["147.206.8.0/22"] will be created
 + resource "aws_eip" "nat_eips" {
 + allocation_id = (known after apply)
 (...) 
 }

 # aws_nat_gateway.nat_gws["0"] must be replaced
-/+ resource "aws_nat_gateway" "nat_gws" {
 ~ allocation_id = "eipalloc-0ffe32519d20b9b7f" # forces replacement -> (known after apply) # forces replacement
 ~ subnet_id = "subnet-0b4b202056c75bb0a" # forces replacement -> (known after apply) # forces replacement
 (...) 
 # (1 unchanged attribute hidden)
 }

 # aws_nat_gateway.nat_gws["1"] must be replaced
-/+ resource "aws_nat_gateway" "nat_gws" {
 ~ allocation_id = "eipalloc-0ff1d32fb271b5cf4" # forces replacement -> (known after apply) # forces replacement
 ~ subnet_id = "subnet-09b6486fbe44e9795" # forces replacement -> (known after apply) # forces replacement
 (...) 
 # (1 unchanged attribute hidden)
 }

 # aws_nat_gateway.nat_gws["2"] will be created
 + resource "aws_nat_gateway" "nat_gws" {
 + allocation_id = (known after apply)
 + subnet_id = (known after apply)
 (...) 
 }

 # aws_subnet.public_subnets["2"] will be created
 + resource "aws_subnet" "public_subnets" {
 + availability_zone = "us-east-1c"
 + cidr_block = "147.206.8.0/22"
 (...) 
 }

Plan: 5 to add, 0 to change, 2 to destroy.

Wait a second, I expect the template to create a new subnet, a new elastic IP and a new NAT gateway in that new AZ. But why does it plan to delete the two existing NAT gateways and recreate two? This doesn’t make for_each an appealing option at all.

Apart from the interruptive plan, there are also other problems. First, since the aws_subnet resources requires cidr_block and availability_zone values, I have to build a map (subnet_config) for its resource array to consume. Similarly, I have to build a second map (nat_gw_config) to create resource array for aws_nat_gateway, which requires subnet_id and allocation_id. This map takes more work to build. Because of the 1-to-1 relationship between subnet_id and alloc_id, I have to fetch the values from two data sources (line 47-61), use a common index (line 63-70). Can I neat it up and combine two maps into one? Not really. Because the second map (nat_gw_config) uses a data source depending on the subnets, which depends on the first map (subnet_config). Trying to combine the maps causes circular dependency!

Also, the additions of data sources makes the code less readable. As Marcel L pointed out in his post, two cons with for_each are: complexity and requiring a map (to store multiple attribute values). Now we seem to have one more: it may cause unintended deletions

Is `for_each` a bad idea?

Let’s find out why for_each could destroy two existing NAT gateways.

Notice that I built the map nat_gw_config by looping through the list of variable public_subnets_cidr_list. After apply, we appended it one more string at the end, without changing the existing order. However, the devil lies in the order of the string lists returned from the data sources. By printing this map, we found that the originally value before the AZ addition is:

index	alloc_id	subnet_id
0	eipalloc-0ffe32519d20b9b7f	subnet-0b4b202056c75bb0a
1	eipalloc-0ff1d32fb271b5cf4	subnet-09b6486fbe44e9795

Based on this, NAT Gateway with index 0 is created with eipalloc-***b7f and subnet-***b0a. NAT Gateway with index 1 is created with eipalloc-***cf4 and subnet-***795. After we add the third AZ, and apply the run, the new map, with a new alloc_id and a new subnet_id looks like this:

index	alloc_id	subnet_id
0	eipalloc-0b70460721596e33f (new)	subnet-0b4b202056c75bb0a
1	eipalloc-0ffe32519d20b9b7f	subnet-0335071ced2dc9922 (new)
2	eipalloc-0ff1d32fb271b5cf4	subnet-09b6486fbe44e9795

There are two factors at play. When the data sources return the ids (data.aws_subnets.public_subnets.ids and data.aws_eips.nat_eips.allocation_ids), the return is sorted. It doesn’t matter whether the order alphabetical or the opposite. Because in any given order, the randomly generated new ID, can fall anywhere in the list. In this particular result, the new alloc_id falls at the beginning, and the new subnet_id falls in the middle. As a result, NAT Gateway with index 0 and 1 are both changed. Therefore they have to be destroyed and replaced.

All these come from having to build a map. The values of each object in the map come from two different data sources. The values are not predetermined and contain a random part. When we add more AZ, the entire map get shuffled, leading to deletion of existing resources. Yikes.

Implementation with `for_each`: second attempt

The first draft of this post drew some ideas on Reddit. One redditor pointed out that the snippet above with for_each isn’t the optimal way. With some tricks to we can manage the map so that it maintain relative order if we have to add new AZ. The strategy is:

Avoid using data sources to retrieve attribute values
Use a unique key to identify objects in the map;
Directly look up from the resource by the unique key

We’re able to do #2 and #3 because when a resource has the for_each argument set, the resource itself becomes a map of objects. We can then locate that resource by the key. We can determine what that key is so long as it uniquely identifies the resource. Below is the revised code snippet with for_each:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}

variable "public_subnets_cidr_list" {
 type = list(any)
 default = ["147.206.0.0/22", "147.206.4.0/22"] # 2 AZ
 #default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"] # 3 AZ
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

locals {
 subnet_config = {
 for cidr in var.public_subnets_cidr_list : md5(cidr) => {
 cidr = cidr
 az = data.aws_availability_zones.this.names[index(var.public_subnets_cidr_list, cidr)]
 }
 }
}

resource "aws_subnet" "public_subnets" {
 for_each = local.subnet_config
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = each.value.cidr
 map_public_ip_on_launch = true
 availability_zone = each.value.az
}

resource "aws_eip" "nat_eips" {
 for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) => cidr }
}

locals {
 nat_gw_config = {
 for cidr in var.public_subnets_cidr_list : md5(cidr) => {
 subnet_id = aws_subnet.public_subnets[md5(cidr)].id
 alloc_id = aws_eip.nat_eips[md5(cidr)].allocation_id
 }
 }
}

resource "aws_nat_gateway" "nat_gws" {
 for_each = local.nat_gw_config
 subnet_id = each.value.subnet_id
 allocation_id = each.value.alloc_id
 depends_on = [aws_internet_gateway.internet_gw]
}

In this example, I use the MD5 hash of CIDR as the unique identifier key to ensure we have a consistent mapping between allocation id and subnet id. When a new AZ is created, the new allocation-subnet id pair will have its own new key. The unique key can be any identifier (even the CIDR itself) as long as it is unique and we do not change the selection of unique key after the first apply.

One more shot with `for_each`

The code snippet above got rid of data sources, but still have to leverage two local values (subnet_config and nat_gw_config) as helpers. Are they absolutely necessary?

Not really. The Terraform documentation has a page about References to Values, where it states:

If the resource has the count argument set, the reference’s value is a list of objects representing its instances.
If the resource has the for_each argument set, the reference’s value is a map of objects representing its instances.

In other words, using for_each with a map as input, we’re also creating a map as output, which is the resource array itself. The key is the same as the input map. Therefore, we can reuse the key. I know that sounds too abstract. Here’s the code refined:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


provider "aws" {}

variable "vpc_cidr_block" {
 type = string
 default = "147.206.0.0/16"
}

variable "public_subnets_cidr_list" {
 type = list(any)
 #default = ["147.206.0.0/22", "147.206.4.0/22"] # 2 AZ
 default = ["147.206.0.0/22", "147.206.4.0/22", "147.206.8.0/22"] # 3 AZ
}

data "aws_availability_zones" "this" {}

resource "aws_vpc" "base_vpc" {
 cidr_block = var.vpc_cidr_block
}

resource "aws_internet_gateway" "internet_gw" {
 vpc_id = aws_vpc.base_vpc.id
}

resource "aws_subnet" "public_subnets" {
 for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) => {
 cidr = cidr
 az = data.aws_availability_zones.this.names[index(var.public_subnets_cidr_list, cidr)]
 }
 }
 vpc_id = aws_vpc.base_vpc.id
 cidr_block = each.value.cidr
 map_public_ip_on_launch = true
 availability_zone = each.value.az
}

resource "aws_eip" "nat_eips" {
 for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) => cidr }
}

resource "aws_nat_gateway" "nat_gws" {
 for_each = aws_subnet.public_subnets
 subnet_id = aws_subnet.public_subnets[each.key].id
 allocation_id = aws_eip.nat_eips[each.key].allocation_id
 depends_on = [aws_internet_gateway.internet_gw]
}

Voila. I use md5 of the CIDR as the key again, first to create both aws_subnet and aws_eip. I also followed the example of chaining for_each between resource types. This way, when creating aws_nat_gateway, I can reference an instance in each resource array by the same key. Chaining for_each is very handy. But admittedly, it takes several iterations for me to get there. The code is neater, but not as straightforward to read due to the list/map comprehension.

Conclusion

I came across a team where the code review guideline favours for_each strongly. I see where that comes from after reading the book. But I don’t find count to be evil. That triggered my initiative to dive deep into this topic.

In this post we brought up a classic pattern of relationship between resources, and examined several ways to implement them using count and for_each. Using count can be straightforward but carries the risk of index shifting if additional element is added in the middle of the resource array. On the other hand, for_each is more powerful, but it requires some crafting with the Python-style list/map comprehension.

My recommendation is, start with a holistic look at the types of resources to create with loop, and how they are related with each other. Go with count if index shifting isn’t a risk. For example, when you need to create one instance of a resource conditionally. Otherwise, use for_each loop if the team is comfortable with the list/map comprehension. In some cases where we need to conditionally create several instances of the same resource, we can use a technique such as:

for_each = variable.disabled ? {} : data.any_resource.map
for_each = variable.disabled ? toset([]) : data.any_resource.list

In fact, the recommendation from AWS Terraform best practice is highly in favour of for_each.

Orchestrate Landing Zone with Landing Zone Accelerator on AWS

Fri, 22 Sep 2023 23:05:04 -0400

As a continuation to the last post, we explore the Landing Zone Accelerator on AWS (LZA) as an orchestration tool in this post. LZA borrows a lot from the ASEA, an accelerator project to deploy the security reference architecture (SRA). LZA is a multi-purpose project that consists of both the orchestration engine (the accelerator itself) and a few reference architectures (as configuration files).

Comparison with Control Tower

First, let’s sort out how LZA is related to Control Tower. Control Tower’s main functionalities are available as an AWS service, with some customization capabilities available as a standalone solution on top of the service, as I discussed in the last post. Unlike Control Tower, LZA as a whole is a standalone solution. Luckily, the installation of the solution itself is highly automated.

I see LZA both as an extension of Control Tower, and as a complement to Control Tower. It is an extension of Control Tower because LZA can co-exist with Control Tower. We can configure LZA to enable Control Tower and use its Account Factory to provision new accounts (alternatively but not recommended, we can opt out of Control Tower and manage account creation on our own). I also see LZA as a complement to Control Tower because it comes with full end to end automation scheme for networking infrastructure and most of the services involved. This is missing in Control Tower, which leaves it with users to provision networking infrastructure in the customization.

Thanks to the infrastructure automation capability, even if you do not have a strong regulatory requirement, there are still good reason to go with LZA for its low-code automation capability. Below is a table that summarizes the differences:

Control Tower	Landing Zone Accelerator
– Multi-account management tool – Governance layer – Customization Framework to bring your own infrastructure automation	– can manage Control Tower – low-code automation engine for infrastructure automation and service deployment based on CDK – reference configurations based on common industry profiles and regulatory requirements

Comparison between Control Tower and Landing Zone Accelerator

As the name suggests, LZA is an accelerator so there is no expectation of its user knowing how to program infrastructure as code. However, it still expects its users to know YAML very well. Knowing how CloudFormation and CDK works can greatly help the users troubleshoot deployment.

Reference architectures in LZA

The input of LZA is configuration as code in YAML format. The LZA repository comes with a number of sample configurations to implement some industry-based best practices. The reference architectures currently include:

General best practices reference configuration: for clients other than the categories below;
Government customers: US Gov Cloud (FedRAMP compliant, on aws-us-gov partition), US State and Local Government, China (on aws-cn partition), Canada Federal (CCCS compliant) TSE-SE (Highly Trusted Secure Enclave Sensitive Edition) on commercial partition for governments, national security, defence, and law enforcement customers reference architecture;
Election: for election customers including elections agencies, committees and campaigns;
Healthcare: for healthcare customers. However, the document does not mention HIPAA compliance or anything related to the HIPAA Reference Architecture;
Finance and Taxation: for tax workload to secure Federal Tax Information (FTI) data;
Education: for education industry customers.

Many of these reference architecture shares a few common traits in the networking design. Take the CCCS reference as an example, the networking involves the followings:

Workload VPCs: consisting of a number VPCs for production and test environments;
Shared services VPC: hosting common services such as pipelines, Active Directories, etc
Endpoint VPCs: centrally hosting interface endpoints
Perimeter VPCs: acting as ingress, egress and inspection VPCs.

The Perimeter VPC hosts firewalls (either AWS Network Firewall or NGFW appliances behind Gateway Load Balancers). All the VPCs are centrally managed in an AWS network account, and are shared to other accounts using Resource Access Manager. The reference architecture document keeps the details of this architecture, which was derived from the security reference architecture (SRA).

Special Purpose VPCs

I consider the non-workload VPCs as special purpose VPCs. The shared services VPC is the most straight-forward. The Endpoint VPC is the most standardized. It is used to centrally host VPC interface endpoints for security and cost reasons. Unlike Gateway endpoint which is only available for S3 and DynamoDB, interface endpoint carries a standing charge and therefore should be consolidated. In addition, since interface endpoints are based on interfaces, we can centrally control the security group and interface policy.

To integrate the endpoint VPC, not only do we need to create those interface endpoint. We also need to account for routing (using Transit Gateway route tables) and name resolution. For name resolution, we need to create a Route53 private hosted zone for each DNS name, such as ec2.us-east-1.amazonaws.com and associate them with each workload VPC. Note that the interface endpoints DNS name may not always follow the same format. See the exceptions in my old post. Also note that this would create a lot of associations (between Private Hosted Zone for each Interface endpoint and each workload VPC). For example, 20 workload VPC with 30 private hosted zones will create 600 associations. To overcome this, use Route53 profile (introduced in April 2024).

Another special purpose VPC is the perimeter VPC. This VPC vary greatly between customers because of different requirement and historical preferences. One of the key design areas is the placement of NGFW, which is discussed in this post.

LZA Orchestration Engine

The installation process may feel complex at the beginning because we have to first install the pipeline to that installs the pipeline. The initial setup consists the following steps:

CloudFormation installs the installer. We start with a CloudFormation template to deploy the LZA installer itself. It deploys resources such as CodePipeline (AWSAccelerator-Installer) and CodeBuild project (AWSAccelerator-InstallerProject). These resources are in the INSTALLER circle in the diagram below;
The installer installs the accelerator core. In the LZA installer, the CodePipeline (AWSAccelerator-Installer) and CodeBuild project (AWSAccelerator-InstallerProject) drive the installation of the LZA. The input is the official LZA GitHub and we need a GitHub token for this step. The output is the actual LZA orchestration engine, including CodePipeline (AWSAccelerator-Pipeline) and CodeBuild (AWSAccelerator-BuildProject and AWSAccelerator-ToolkitProject). The user may specify their own GitHub repo as the configuration repo. Otherwise, a CodeCommit repo will be created. The LZA resources are shown in the CORE circle in the diagram below;
The acceleration core configures the landing zone. The LZA orchestration engine deploys actual resources in the landing zone, with the CodeCommit repo (aws-accelerator-config) or the specified GitHub repo as input.

If we enable Control Tower with LZA, we should first log in to management account and configure Landing Zone with Control Tower. we can also create (and register) the required OUs and accounts from Control Tower. Then we can deploy Landing Zone Accelerator with default configuration.

After the initial setup, we will need to iterate over the aws-accelerator-config repo. We implement our landing zone design in YAML configuration following the schema documentation. Changes in the configuration repo will trigger the pipeline (aka LZA’s orchestration engine) to redo step 3, whereas step 1 and step 2 are performed only once. The duration of step 3 is significantly longer than the first two steps.

Pitfalls

If LZA manages Control Tower, it expects existing OUs registered in Control Tower or it will report error. For account, LZA can create accounts listed in the manifest but not yet created. However, with the lengthy account vendor process for multiple account we run the risk of task time out in the pipeline.

During the installation, some account may run into quota limit. For example, the Networking Account usually have more than five VPCs whereas the quota is 5 VPCs per region per account. We need to increase the quota on those accounts.

The full deployment usually creates some SCPs. However, if we ever need to re-deploy a configuration, some steps steps might be blocked by certain SCPs. Attempts to temporarily detach SCPs from OUs, or modify SCPs often get reverted. The cause is an EventBridgeRule in us-east-1 region named RevertScpChangesModifySc. The rule should be disabled temporarily to perform the troubleshooting activity. We can do this with the following steps:

Disable the EventBridgeRule RevertScpChangesModifySc , which is only present in us-east-1 region;
Detach SCPs and note down what are detached, one OU at a time;
Go to the failed CF stack in the region, delete the failed stacks (after turning off termination protection);
Rerun the pipeline step from where it failed. This time it should go past the failure to the end, if SCP is the cause as per our assumption;
Re-attach SCPs. Suppose Security and Infrastructure OUs share one group of SCPs, and Dev, Test, and Prod OUs share a different group of SCPs;
Re-enable the EventBridgeRule;

Even with the EventBridgeRule RevertScpChangesModifySc disabled, when you re-run LZA deployment pipeline, the Accounts step will re-attach the SCPs using the AWSAccelerator-AccountsStack in the management account in us-east-1 region.

In general, how SCP works with organization structure is something to be very careful about, especially when the hierarchy consists of multiple layers of OUs. It is important to keep in mind, that deny statements in SCP take effect down the hierarchy, where as allow statements only affects the immediate child account of the OU where the SCP is attached to, as per the evaluation logic. As a result, an SCP with allow * statement (in the LZA-AWSFullAWSAccess managed policy) must be applied to Root, every OU at each level, and every account, for LZA to function.

In addition, there are some hard limits for SCP. Each SCP has a size limit of 5120 characters, and each OU can attach a limit of 5 SCPs.

Challenges

Powered by CDK, LZA automates the creation of a lot of resources. The configuration files uses the deploymentTargets attribute to allow users to specify to which accounts or OUs the declared resources will be deployed to.

Supporting many resources is a double-edge sword. Because the accelerator needs to go through every aspect of a landing zone, it is very slow to run. The accelerator pipeline may take as long as 40 minutes without any change to the configuration code. This is extremely slow if you just want to make some small changes in the configuration (e.g. update route table, add IAM role).

Even though LZA supports many resources, it’s not flexible with every resource. For example, today we can deploy IAM roles using RoleSet. However, in the trust policy of the IAM role you can only specify a two types of principals under the assumedBy attribute: account and service types. On the other hand a trust policy can support many other types of principals such as another IAM role.

AZ Mapping

Another important ability that LZA does not support is consistent AZ mapping across accounts. (Correction: this is now supported in LZA v1.5 as of Oct 2023). In some example LZA configurations, we deploy VPCs across multiple accounts using two or three availability zones referenced by their logical ID, such as us-east-1a and us-east-1b. However, AWS maps logical ID to physical ID and the mapping may be different in each AWS account. Using the same logical ID cannot guarantee the physical AZ are the same across account. As of LZA 1.5, the ability to reference physical ID in availabilityZone is supported in LZA configuration file.

Summary

I’ve spent a lot of time on LZA recently. It is extremely powerful. LZA streamlined the landing zone deployment process with configuration as code. It also allows users to customize their landing zone towards their own architectural needs and compliance requirement. For example, you can declare arbitrary SSM parameters in each account.

On the down side, the LZA deployment is time consuming through the pipelines. It tries to automate too many aspects of the infrastructure, which makes itself quite a complex project. Expect lots of changes in each new version.

The idea of being a low-code solution is to make it simple for end users but it often sacrifices flexibility. For example, if you want to create an IAM role in each new account that references the Management account ID, it is not possible until such feature is implemented in the accelerator. When the accelerator pipeline fails, it still requires deep CloudFormation knowledge to troubleshoot.

Infrastructure deployment in Terraform 1/2

Wed, 11 Aug 2021 21:44:00 -0400

Terraform is an excellent Infrastructure-as-Code (IaC) tool based on Hashicorp Configuration Language (HCL). Compared to JSON or YAML based declarative templates (e.g. CloudFormation and ARM), HCL is more concise, thanks to the flexibility of HCL. On the other hand, HCL is not as flexible as general purpose languages. For that sake, I see HCL as semi-declarative IaC. This post is my notes about best practices with Terraform development, from the context of AWS, but also applies to other cloud platforms.

Complex Types

There are three primitive types (string, number and bool) that forms collection types and structural types. Here are some common ones:

list: element may repeat, and order is maintained:

["orange", "banana", "orange", "apple"]

set: elements are unique and unordered

["apple", "banana", "orange"]

tuple: each element has its own type

["a", 15, true]

object: defined by a schema with named attributes each with its own type

{
 name = "John"
 age = 52
}

list of object

[
 {
 alloc_id = "0b7271a3219bc1fc2"
 subnet_id = "0c02af76c2c3e46fa"
 },
 {
 alloc_id = "0440c334c48d4247f"
 subnet_id = "02652e69fa2a71de8"
 }
]

map of string

{
 property = "foo"
 attribute = "bar"
}

map of object

{
 objkey1 = {
 alloc_id = "0b7271a3219bc1fc2"
 subnet_id = "0c02af76c2c3e46fa"
 }
 objkey2 = {
 alloc_id = "0440c334c48d4247f"
 subnet_id = "02652e69fa2a71de8"
 }
}

Whenever applicable, Terraform converts types implicitly or explicitly. For example, when a list or tuple is converted to set, all elements are converted to string and duplicates are removed. Object and map are very similar. Map of string can be converted to object if the attributes comply with the schema. Additional attributes not in the schema are discarded.

HCL Types is similar to Python

Although being totally different beasts, the complex types between HCL and Python are similar, to the point I suspect the HCL design is influenced by Python. I summarize the similarities as such:

Python Types	list []	tuple ()	set {}	dict {}
Terraform Types	list []	tuple []	set []	map {}
ordered	Y You can access item by index	Y	N you cannot access an item by index or key; however you can loop over all itmes	N key-value pair that allows you to access item by key
changeable (mutable)	Y	N You cannot update, add or remove items	Y Add or remove only. no change to existing elements	Keys must remain unique or the values get overwritten; Values are mutable
allow duplicate elements	Y	Y	N	Keys must be unique; values don’t have to

In Python, list and tuple allow elements of mixed types but in IaC like Terraform we mostly don’t need mixed types. In Terraform, an object is a map without a defined type. In most situations, lists and tuples behave identically, as do maps and objects.

Another area of similarity is with the comprehension of list and dict/maps. In Python for example,

## Supposed you need to create a list:
lst=[]
for x in range(10):
 lst.append(x**2)

print(lst)

### that can be simplified as the following to create the list:
lst = [x**2 for x in range(10)]

## You can even add contidion
even_numbers=[num for num in range(10) if num%2==0]

## You can introduce function calls:
words = ["hello", "world", "python", "list", "comprehension"]
lengths = [len(word) for word in words]

## You can even combine two lists
lst1=[1,2,3,4]
lst2=['a','b','c','d']

pair=[[i,j] for i in lst1 for j in lst2]

print(pair)

## With dict, it's similar
evens={x:x**2 for x in range(10) if x%2==0}
print(evens)

In Terraform, we use similar techniques:

[for s in var.list : upper(s)] # build a tuple/list from a list

[for k, v in var.map : length(k) + length(v)] # build a list from a map

{for s in var.list : s => upper(s)} # build a map from a list

[for s in var.list : upper(s) if s != ""] # build a tuple/list from a list with condition

Note that the documentation of Terraform doesn’t explicitly call them out as comprehensions. However, it’s exactly the same idea as comprehensions in Python. Even the range() function exists both in Python and Terraform.

Modularization

Modules allows you to group related resources together. They can also be re-used and called by other modules. It is fairly straightforward to create a module:

put the resource declarations into a sub-directory
define input and output in the directory

However, the introduction of module complicates the directory structure and variable referencing, which is important to take into account before starting creating modules. This guideline has further discussion about when to create a module. I re-wrote the terraform templates in Orthweb project to leverage modularization wherever possible, but there is still some stand-alone resource (e.g. random_id) not belonging to any module.

To reference resources across modules, you need to import those resources (using data source) from within the module. There are a couple of ways. You may pass the argument of data source as input variable, or you can leverage the filter capability of data source. Let’s look at one example of each mechanism.

In the example below, we import a subnet by subnet id:

data "aws_subnet" "private_subnet" {
 id = var.private_subnet_id
}

In the example below, we import a subnet by filtering from all subnets in the VPC by tag:

data "aws_subnet" "private_subnet" {
 vpc_id = var.vpc_id
 filter {
 name = "tag:Name"
 values = ["Private"]
 }
}

There are pros and cons of each approach. A module with mechanism 1 is more transferrable across different environment, because the ID of subnet is explicitly provided. However, authors needs to manage those explicit variables with code. Mechanism 2 fetches target resources with filter. It depends on a well-implemented tagging policy in the resource farm.

The Terraform Registry (since 2017) contains a lot of pre-built modules for each backend platform (e.g. AWS). If you find any module that can be used in your project, the module repo can be referenced directly by Git repository URL. You should be aware of the risk of this practice though. Many platforms are keen to publish modules for their platform. Anyone can publish their own modules to the community as well.

You can quickly generate module documentation with terraform-docs.

Local Execution

Local execution is the basic workflow mode which is mostly seen with very small collaboration team. In this mode, the developer executes terraform binary (Terraform CLI) from their workstation (e.g. Laptop). The Terraform CLI converts code into API calls to interface cloud provider. The most frequently used commands (from terraform directory) are:

terraform init
terraform plan
terraform apply

The init command initializes the working directory. The plan command figures out the delta between code and infrastructure. It outlines the changes it is about to make. The apply command commits the change. The documentation of Terraform CLI commands is here.

Terraform keeps track of the infrastructure it manages in state file. This article explains the purpose of state. State management collaboration difficult with local execution because the state file by default is created in the working directory on user’s workstation. Although the state file can be configured to be stored in a shared location such as S3, it still requires a mechanism to lock the state in a multi-developer collaboration.

In large operations, the same code base in Terraform, is usually used to created several different sets of infrastructures, for example, in different geographic regions. So it is a 1-to-many relationship between the code repo and the infrastructure state. To further complicate things, each state might have been deployed using different revisions of the code. To overcome that challenge, Terraform introduced the concept of workspace, which is essentially an instance of state describing a particular group of infrastructure being managed by the same source code. When there are many workspaces, it becomes tricky to manage them with CLI commands.

State management is a major challenge that needs to be solved for team collaboration in local execution workflow. Each state must use the same revision of Terraform code. You can use Git in combination as a workaround to that limitation but the point is you cannot tie a workspace to a commit with the workspace commands.

In some enterprise environment, the execution is from a VM (e.g. ADO agent on-premise) without Internet access, which poses another challenge. First, we need to pre-load required providers manually. The enterprise needs a proxy solution to safely download packages from Hashicorp website. One good option is Nexus Repository, with both open-source and pro supports. It is a full-function artifactory repo that can host helm repo, apt repo, yum repo, etc. Second, we also need to configure Terraform so it picks up providers locally. Managing plugins without Internet access requires understanding of the order in which Terraform tries to load plugins during initialization.

Remote Execution

In remote execution, the code is executed in Terraform Enterprise or Terraform Cloud. Both are remote web servers. The difference is that Terraform Enterprise is self-hosted service, requiring IT specialist to install and maintain Terraform Enterprise. Terraform Cloud on the other hand, is a managed SaaS service. The pricing model includes a free plan for small number of users.

Terraform workspace configuration

In Terraform Enterprise or Cloud, the remote execution is organized in workspaces. You need to create an organization, and then create workspace under the organization in order to execute code. With each workflow, you can specify version control system (VCS) and subdirectory, to tell the workspace where to fetch Terraform code from. The workspace also allows you to define secrets and variables specific to the workspace. When you execute a workspace plan, the secrets and variables are passed from workspace to the execution logic.

Terraform Workspace Variable configuration

You will also need to design the Terraform code in a way to work seamlessly with the secrets and variables loaded from the workspace. The variable declaration in code should match the definition in workspace. There are already a number of variables that came in handy. Check out this guide.

Each execution is referred to as a “run”, with its own run id. A workspace involves may runs, which may succeed or fail. Each run pulls a specific commit of the source repository, and goes through stages such as plan, and apply. The UI from each run result list out the status of each result, in a very easy to read format.

The state data is persisted in the web server as they were generated. Therefore the collaborator do not need to worry about managing state with CLI tools. If there are files that you do not want picked up by the execution engine, their locations can be added to a file .terraformignore. Refer to this guide.

AWS profile

Local execution still has a lot of use cases in enterprises such as testing with temporary resources. A common challenge is authentication. As discussed, Terraform CLI picks up identity information from AWS CLI and authenticates its way into the backend to run API calls against. So AWS CLI must be configured correctly with the sufficient permission to provision resources. On the other hand, enterprises usually offload IAM to an identity store, such as AzureAD, Okta, etc. Putting those together, the pattern of authentication and authorization usually looks like this:

User logs on via SSO (e.g. SAML). The validation response gives a name of an IAM role.
Upon successful authentication, user takes the IAM role. The role does not have any capability, except for assuming a second IAM role.
The second IAM role (the functional role) grants user the permission to do its business.

The steps above, can be carried out in AWS console, or with AWS cli using assume-role command. However, when we put Terraform in the picture, it becomes a little involving because the credential information is updated whenever the functional role is assumed, and the assume-role command takes a pretty long argument.

To skip typing the long command every time, there are some handy tools, such as aws-azure-login. An even better tool that works with a variety of identity stores is saml2aws. The tool allows you to configure identity backend, assume the functional role, and update credential information in aws credential file, all with a single command. The AWS CLI configuration reads:

[default]
region = us-east-1
output = json
cli_history = enabled
cli_pager =
role_session_name = functional_operation

[profile function_user]
source_profile = default
role_session_name = functional_operation
role_arn = arn:aws:iam::9998887766:role/admin-access
region = us-east-1

In Terraform provider, we need to tell it to assume that role as well:

provider "aws" {
 region = "us-east-1"
 assume_role {
 role_arn = "arn:aws:iam::9998887766:role/admin-access"
 session_name = "terraform"
 }
}

This will ensure Terraform assumes appropriate role before doing its job.

AWS EC2 SSH Key Pair

RSA key authentication for SSH should be used for Linux Instances. When creating an EC2 instance, we give it our public key so we can then later authenticate through SSH. If the key is already stored in AWS, we just need to tell EC2 the name of the key, in the key_name property. If the code is likely to be executed from several different places by different users, then we can write the code so it picks up public key from user’s workstation (~/.ssh/id_rsa.pub). Here is an example:

variable "local_pubkey_file" {
 type = string
 default = "~/.ssh/id_rsa.pub"
}
data "local_file" "pubkey" {
 filename = pathexpand(var.local_pubkey_file)
}
resource "aws_key_pair" "user-pubkey" {
 key_name = "runner-pubkey"
 public_key = data.local_file.pubkey.content
}
resource "aws_instance" "bastion" {
 instance_type = "t2.micro"
 key_name = aws_key_pair.user-pubkey.key_name
 ......
}

For remote execution, we can even add an option to pass public key in as variable, to override the key file variable. For an example, check out my orthweb project.

To upload files to EC2 instance from Terraform execution environment, we can use the file provisioner with ssh as connection type.

AWS CDK example in Typescript – provision an AWX server

Sat, 19 Dec 2020 17:14:00 -0400

This post provides an example of using AWS CDK in Typescript.

Ansible Tower and AWX

We have used open-source Ansible extensively in the past. While the automation is convenient, the lack of UI makes it not as suitable as a team collaboration tool. One way to allow team collaboration with open-source Ansible, is to use Jenkins to glue the components together, as discussed in the Automated Deployment Pipeline series. In this setup, the open-source Ansible remains command-line driven, with Jenkins building up the command, rather than a human user. There are many upsides in this configuration, but it is not built specifically for Ansible. Ansible is agent-less, and can be run from any host. This sounds appealing and can work well in smaller server fleet. However, since it requires some configuration on the controlling host for Ansible to function properly, it become unnecessary to configure Ansible environment on every single host (e.g. production). A typically environment only has Ansible environment configured on the bastion host. This brings the need for a dedicated controller server to drive all Ansible tasks.

Ansible Tower is Red Hat’s commercial enhancement to the open source Ansible, providing web-based console, REST API and other services such as Role-based Access Control (RBAC). Managing Ansible via REST API is still somewhat involving but this also enables other open-source contributions to simplify the use of API. For example Tower CLI allows you to use Ansible Tower with simplified command. Ansible Tower has an open-source upstream project called AWX, maintained by Red Hat. AWX is essentially a preview release of Ansible Tower without commercial support. AWX can serve as an engine for all Ansible related task. AWX server is essentially an Ansible control server. AWX, or Ansible Tower, also brings several concepts on top of Ansible:

Job template: defines how an Ansible playbook should be executed, including details such as machine credential, project, inventory, and playbook file.
Job: the actual execution of job template
Project: connects Ansible Tower to source control such as BitBucket. It is tied to a Git repository and a branch within that repository

To deploy AWX on EC2 instances, there is a reference deployment by AWS. However, it is provided as CloudFormation template and appears to be outdated (from 2018). In our project (late 2020, named ansible tower lab, or dubbed as “atlab”), we provide the infrastructure in AWS CDK (written in typescript), to provision the AWX environment.

The goal is that once the configuration is completed, you can run ansible ping against a target EC2 instance. The steps are as automated as possible. However, a number of key steps are purposefully left manual for learning purpose, such as the installation of AWX on EC2 instance.

Infrastructure as Code

In previous posting, I created infrastructure as code in AWS CDK with Python, so I decided to change to typescript in this project, with the assumption it is just a matter of syntax mapping. However, I underestimated the transition to a new language I never learned before. A fuzzy understanding of little details such as when to use let a=4 vs this.a=4, may produce elusive errors that takes hours to troubleshoot. I would therefore strongly recommend reading the basic syntax guide for typescript, before getting started.

In typescript, this example project provides an implementation of configuring autoscaling groups, including cloud init, user data, etc on AWS. Other than the language, everything else is very similar to the project in this post, which was developed in Python. Also, note that the project directory structure varies slightly based on the language being used.

If you are absolutely new to AWS cdk, start with this app. It is beyond the scope of this post, to cover extensively the installation and environment configuration of AWS CDK.

In the provision process for Bastion host, the cloudformation init script pulls a specific version from AWX repository, then makes slight modification. User will need to install it manually. Note that AWX can be installed on three types of platforms:

OpenShift
Kubernetes
Docker Compose

All are documented in their README file. For simplicity in this project, the installation is on standalone docker compose. This is the default mode so there is no need to modify the inventory file.

The code repo

The repository is version controlled here. To run the project, you need to have aws cli environment, then install the required packages including node js, and npm packages such as aws cdk. Once configured, validate with command:

cdk ls

This should display the stacks available. Use cdk deploy to deploy each stack.

When BastionStack is deployed, dependent packages should be installed with user data and cloud init. You will just need to SSH on to the server to manually install AWX, as explained in the instruction, to manually install AWX:

ansible-playbook -i inventory install.yml

Then you can browse to the server (at port 80 by default). Before the log-in page for the first time, the AWX will upgrade itself, with the following screen presented:

Now log on with default credential (in README.md), you will have the UI for AWX:

AWX Web Console

From here, you can edit inventory by adding the host. Or use the helper script (~/awxcompose-helper.sh) from bastion host to create a new inventory (named Private Instance Inventory), and populate it with the hosts in the stack. The helper script does so by querying aws resource, and isssue rest API calls to AWX. After executing the script, you can see a new inventory, and the Private instance inventory should contain all hosts in the stack:

Automatically populated inventory

You can then run Ansible ping against the host to validate connectivity. Note that during inventory creation, the ansible_user is already set to ec2-user (by the helper script):

Ping result

Some technical details

The initialization process on Bastion host creates an RSA key pair, stores the public key to AWS, for the upcoming private instances to uses. It keeps the private key locally in order to make outgoing SSH connection to the private instances. To ensure connectivity between AWX and private instances, there are a couple of (bash) helper scripts involved. Both reflects some technical details that I had to work through.

awxcompose-helper.sh: the initialization process pulls AWX installation file from git repo. The installation process will build a docker-compose file in ~/.awx/awxcompose, based on a template (~/awx-*/installer/roles/local_docker/templates/docker-compose.yml.j2). When user tells AWX to connect to private instance, the connection was made out of a docker container (instead of from the OS of bastion host), we need this script to map SSH key file from host to container, by modifying the template file. Without this helper, outgoing SSH connection will fail with error (Permission denied (publickey,gssapi-keyex,gssapi-with-mic)). This script is invoked in the cloud init process without requiring manual execution.
awxinvt-helper.sh: once the private stack is up and the installation has completed, we need to add the hosts to AWX inventory. This script gets the instance ID and IP addresses of the private instances, and uses Rest API calls to create inventory and populate it with hosts. Ansible has multiple ways of authentication. This script uses the non-stateful basic authentication with each curl command requiring credential. Ansible Rest API guide is provided here and be wary of the convention where URI must end with a slash.

This project is just a start of AWX on AWS CDK project using Typescript. In real life scenarios, there are some work to do to make this even more automated. For example, use cfn-hup service to monitor changes of private stack, and therefore update inventories accordingly.

AWS CDK example in Python – provision Kubernetes Nodes

Thu, 03 Dec 2020 21:14:00 -0400

There are two mechanisms to initialize instances in AWS. Cloud init and CloudFormation Init. Both are widely used and we discuss each of them in this posting. Then we will give an example of using AWS CDK in Python.

Cloud-Init

Cloud-Init is a service originally built for Ubuntu, as a bootstrapping utility to customize a Linux VM as it boots for the first time. It has evolved to be an industry standard multi-distribution method for cross-platform (public or private) cloud instance initialization, or even bare-metal installation. In cloud-init you can install packages and write files, or configure users and security. Because cloud-init is called during the initial boot process, there are no additional steps or required agents to apply your configuration.

Cloud-Init uses UserData, which is part of instance metadata. With AWS, you can pass two types of user data to Amazon EC2: shell scripts and cloud-init directives.

The cloud-config files are text files encoded in base64, with more details covered in the documentation here. cloud-init also works across distributions. For example, you don’t use apt-get install or yum install to install a package. Instead you can define a list of packages to install. cloud-init automatically uses the native package management tool for the distro you select.

CloudFormation Init

The cloudformation init mechanism does not only initialize instance, it also provides a mechanism for the resource being created to communicate with other resources. It allows an instance to emit signal to a different resource (via cfn-signal). It can also monitor changes to external resource and invoke local action (using cfn-hup with hooks). CloudFormation Init requires several components to work together:

The cloudformation resource should have metadata. The metadata must have a key AWS::CloudFormation::Init in which configsets are declared.
The UserData must use helper script (cfn-init) to invoke configuration jobs
The UserData can use helper script (cfn-signal) to signal with a CreationPolicy or WaitCondition (of the same or different resource), so you can synchronize other resources in the stack when the prerequisite resource or application is ready.
The cfn-hup service on the instance can be configured, to check for updates to metadata and execute custom hooks when changes are detected.

Comparison

While there are overlaps between the functionalities of Cloud Init and CloudFormation Init, the major difference is the latter support extended features (signal, update, etc); whereas the former is vendor neutral. The table below summarized some the differences:

	Cloud Init	CloudFormation Init
Works on	Linux OS distribution	CloudFormation resource, in combination with cfn helper scripts
Usecase	Initialization only	Both initialization and resource update
Trigger	cloud-init systemd service	Initial: from UserData Update: by cfn hook
Adoption	Multiple cloud vendors and bare-metal system	AWS cloud instances
Action Playbook	/var/lib/cloud/ Instance Metadata -> User Data, encoded in base 64	CloudFormation Resource -> Metadata section -> AWS::CloudFormation::Init -> configSets and configs
Log file and stdout	/var/log/cloud-init.log /var/log/cloud-init-output.log	/var/log/cfn-init.log /var/log/cfn-init-cmd.log

Comparison between cloud-init and cfn-init

AWS Cloud Development Toolkit (CDK)

Traditionally, AWS CloudFormation uses template in YAML or JSON for resource declaration. As the size of system grows, the amount of resource involved grows quickly and the size of such declaration file may grow beyond manageable. Nested stacks and export of output are mechanisms designed to combat the template sprawling, but to a very limited extent. Two reasons it is hard to control template size are:

In declarative statements, each line carries very small piece of information. Without flow controls such as if-else, loops, object oriented structure, the level of code reusability is very low;
Some auxiliary resources (such as AWS::EC2::VPCGatewayAttachment) must be declared explicitly, even though they are insignificant to the stack functionality

To address these challenges, AWS introduced AWS CDK (cloud development tookkit), which supports multiple languages (JavaScript, TypeScript, Python, Java, and C#). The CDK was natively developed in TypeScript, which is supposed to be the preferred development language. A tutorial is provided here, with detailed API documentation here.

To install aws cdk and create a hello world project, follow this example.

An Example in Python

I have created an example for AWS CDK in Python. The purpose is to create some EC2 instance to complete a lab for Kubernetes (without using managed EKS service). The example provisions the followings:

VPC, a public and private subnets, Internet and NAT gateways;
Relevant security groups and permissions
Bastion host, public instances in public subnet
Private instances in private subnet, with public route through NAT gateway

The private instances forms a cluster for Kubernetes lab. We will use kubespray to initialize these instances. During the bootstraping, we download kubespray, install ansible, etc.

Here is the code repo for this example. With CloudFormation only, the single template could go well beyond 1000 lines. With CDK, the code are organized into several different python files, each representing a stack. The stacks can be stood up with command:

cdk deploy vpc-stack
cdk deploy security-stack
cdk deploy bastion-stack
cdk deploy private-stack

Although the documentation in Python is available, there are generally not a lot of examples built out on the Internet. The pypi site provides some Python specific examples for each module (e.g. core and aws-ec2). Given these libraries are available for only 2 years (since 2018), many advocates TypeScript as the language. However, I have implemented some CloudFormation init, used helper script, and UserData in this example, without running into any language specific issues.It should be noted that the EC2 instance by default will call cfn-init. So there is no need to explicitly run cfn-signal or cfn-init from user data in python code (example). This can be verified in file /var/lib/cloud/instances/<instance-id>/user-data.txt which automatically includes the following lines:

# fingerprint: e1b32ead13878deb
(
 set +e
 /opt/aws/bin/cfn-init -v --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456 -c config_set_1,config_set_2
 /opt/aws/bin/cfn-signal -e $? --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456
 cat /var/log/cfn-init.log >&2
)

In addition to Python, AWS CDK also supports other languages. In the next post, we will discuss use of CDK in Typescript.