Update 2023: the practice outlined in this post has been outdated. This post is for archive ony.
We use CloudFormation to deploy infrastructure including private and public subnets, as well as EC2 instances. It is a good practice to place as many instances as possible in private subnet and access those only from Bastion Host in the public subnet. For smaller systems it is even common to use the NAT instance as Bastion host because both are typically placed in the public subnet.
When provisioning a bastion host (jump box), an administrator can upload his public key and specify it as key name. This allows the administrator SSH access right at the birth of the instance. Private instances are only to be accessible from the bastion host and we cannot preload them with the same public key because then the administrator would have to transfer their private key to the bastion host, which isn’t secure.
In order to open SSH access to bastion host, we recommend that the bastion host generate an SSH key pair of its own during the bootstrapping process. This can be done through AWS cli.
Above is the diagram, in which a key pair needs to be created during the provisioning of Public Subnet. The private key needs to be assigned as SSH client private key (~/.ssh/id_rsa with 400 permission) and the public key to be distributed to private instances, by referencing the correct name in their resource section in the CloudFormation template. The name of the key follows the format of instance_name-instance_id-key, so it can be easily referenced from CloudFormation template as well as from within bash.
Every time the template is launched a new key pair will be created for the new virtual private cloud. This sample code is shared on my Github page. The key is a series of commands to strip out the private key from cli response:
MyInstID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`; MyName=`aws ec2 describe-instances --instance-ids $MyInstID | jq -r ".Reservations[].Instances[].Tags[] | select(.Key==\"Name\") |.Value"`;KeyPairName=$MyName-$MyInstID-key;echo Creating KeyPair $KeyPairName;aws ec2 create-key-pair --key-name $KeyPairName | jq -r ".KeyMaterial" > ~/.ssh/id_rsa;chmod 400 ~/.ssh/id_rsaThis applies to solutions where more than one private subnets are involved.