firewall on Digi Hunch

Firewall Deployment Patterns

Sat, 16 Nov 2024 16:24:15 -0400

The Hub-and-Spoke topology is the most common topic in the discussion for building cloud infrastructure design. This topology appeared in both AWS and Azure design papers and had been around as a very important option in physical networking design.

The AWS whitepaper Building a Scalable and Secure Multi-VPC AWS Network Infrastructure has thorough discussion on the topology. This topology often feature a Transit Gateway as the hub. In addition to workload VPCs, the network topology often includes some special-purpose VPCs, such as interface endpoints VPC, or shared tooling VPCs. One of the special-purpose VPC is the inspection VPC. It is a key design area to suit the need of inspection and traffic management for the business and the design may vary a lot depending on the available inspection tools such as a Firewall appliance.

Inspection Requirements

The most common situation with an enterprise is connecting with on-prem networking. Options including Direct Connect, site-to-site IPsec or SD-WAN overlay. The business decides whether and at what level they would like to inspect the traffic between on-prem and their VPCs. Here is an example.

Connectivity	Inspection Requirement
Between Workload VPCs (East-West)	No inspection
Between a workload VPC and a special-purpose VPC	Normal Inspection
Ingress Traffic from Internet to Workload VPC	Deep Packet Inspection
Egress Traffic from Workload VPC to Internet	Deep Packet Inspection
Between Workload VPC and on-prem networking over Direct Connect	Normal Inspection
……

With a normal inspection, the firewall appliance only checks the information in the packet’s header, such as the source and destination IP addresses, port number, etc. With deep packet inspection, the appliance examins a larger range of metadata as well as the data in each packet. DPI provides a more effective mechanism to perform network packet filtering and find otherwise hidden threats. It is however an expensive operations from a performance standpoint. Ultimately the business makes the call but it is important to identify ALL connectivity scenarios in this phase and explicitly document the decision and rationales. They can choose from an NGFW product or the Network Firewall service from AWS, depending on capability required.

Inspection Architecture

At minimum, inspection is required for ingress and egress traffic to and from workload VPC. The design must account for both routing and inspection. Many would use the same VPC for ingress/egress traffic and for inspection.

It is also possible to separate these two purposes into two different dedicated VPCs: an inspection VPC that hosts firewall services or appliances, and an ingress/egress VPC that directs traffic from and to the Internet but we must route the traffic to the inspection appliance. If all traffic to be inspected has to be routed through the Transit Gateway both ways, the cost would be high. In 2020 AWS introduced Gateway Load Balancer (GWLB) to address this use case.

The recommended pattern using GWLB allows you to place firewall appliance and a GWLB in one VPC, and place the GWLB endpoint (GWLBE) in a different VPC. The connectivity between GWLBE and GWLB is backed by HyperPlane, a technology that also enables other endpoint service such as PrivateLink. The connectivity between GWLB and the appliance take place with Geneve encapsulation. This pattern places any appliance behind an endpoint, so long as the appliance supports Geneve protocol.

The GWLB technology enables a number of inspection patterns based on distributed ingress paths, as summarized in this document. Distributed ingress/egress means each workload VPC can have their own Internet Gateway and NAT gateways. They must configure their route table so as to send the traffic via GWLBEs to inspection appliances. In general, I recommend this pattern over the centralized ingress/egress patterns where only the inspection VPC can take ingress traffic from Internet Gateway. The Network architectures for ingress traffic inspection presentation from 2021 ReInvent covered this topic as well, especially about the scaling benefit of distributed ingress.

Firewall deployment patterns

The firewall deployment pattern available differ between vendors and the requirements. Since the GWLB pattern places appliances behind the GWLB, the appliances rely on Geneve traffic that GWLB forward over. Some vendors may argue that this pattern keeps the NGFW product from performing other tasks that do not support Geneve traffic. One example is Network Address Translation. The native NAT gateway services is very expensive (consider fck-nat as an alternative for NAT). Many clients want to use the NAT feature of the NGFW product. The architecture therefore has to be adjusted in favour of centralized egress. Review this post about one-arm mode and two-arm mode.

If we have to go with central ingress/egress anyways, there are still numerous options. Take FortiGate for example, while the GWLB pattern of deployment is supported, other available options include:

Traditional pattern with multiple interfaces across different subnets in the inspection VPC (L3 mode)
Integration with Transit Gateway using Transit Gateway Connect Attachment
Integration with Transit Gateway using Transit Gateway VPN Attachment

I regard the first option as traditional because it does not directly integrate with Transit Gateway and it is very similar to how we deploy them in a physical networking environment. Fortigate refers to it as L3 (NAT/route) mode. In this mode the Firewall appliance can also influence network routing.

The second and the third options are similar except for different types of Transit Gateway attachments are used. The reason to directly integrate with Transit Gateway is so that the Transit Gateway can route the traffic for inspection therefore no need for a Gateway Load Balancer, and thus no dependency on the firewall features supporting Geneve.

The second option builds a GRE (Generic Routing Encapsulation) tunnel over a Transit Gateway Connect attachment as the transport tunnel, and uses BGP to exchange routes between the Transit Gateway and the appliance. It treats the firewall instances as SD-WAN appliance and has performance benefit. The third option uses VPN attachment with the main benefit of encryption if it is part of compliance requirement.

Rules

The rule configuration for Firewall is critical to the operation of the entire multi-VPC network configuration. Unfortunately, there is no standard with the rule syntax across majore NGFW vendors, leading to challenges for customers to swap vendors. Most flavours of rules have common elements such as

Action (Pass, drop, or alert)
Source and Destination
Protocol and Port

A very common open-source firewall rule syntax is the Suricata-compatible format. One important adopter is the AWS Network Firewall, which supports both stateful and stateless rule groups. With stateful rule group, there are two options for how the Suricata engine evaluates rules.

With “Action Order” option, Suricata engine evaluates the rules in the order of: pass, drop, reject and alert. You can use the priority attribute to influence evaluation;
With strict order, the rules are evaluated in the order of the rule definition;

It is important to be aware of the rule evaluation order since it impacts the firewall behaviour deeply.

Summary

In the networking infrastructure design, ingress and egress routing are the most critical one-way door decision. This decision must account for both routing and inspection. While there are many options, we usually start with capturing the key requirements. In this post we reviewed how to approach the requirement, a key technology Gateway Load Balancer and some firewall deployment patterns with FortiGate as an example. The approach is similar for other NGFW vendors, such as Palo Alto, Check Point or Cisco Secure Firewall.

WordPress Security Basics

Fri, 17 Nov 2023 12:02:00 -0400

Background

In 2019, I moved this site to WordPress hosted on an Amazon Lightsail instance. There were few visits at that time so I lived with the single-server architecture. The website traffic has since been in steady growth but I have been too busy to catch up with the WordPress security setup. In July 2023, a malware impacted this site as well as the web traffic. It took me several months to fix a few related issues but the traffic still has not fully recovered. This post is about the lessons.

The Incident

I first noticed the issue when I clicked on links to my web page from Google result and got redirected to some spam site. It did not happen 100% of time, but it is annoying enough. In the mean time, from Google search analytics I noticed traffic volume going up with a lot of traffic going to URLs that I did not recognize or create. Somehow these URLs have a lot of clicks and impression counts. These are signs of artificial traffic.

Obviously the site was hacked. The first thing to determine is whether the server access was compromised. From the audit log (/var/log/auth.log and auth.log.gz) I can see a lot of brute force attempts to connect but fortunately none was successful. That also prompt me to change the default SSH port and use ECDSA key pair. Since the OS access is safe, the hack happens at the WordPress level. I suspected the sideloaded plugins from a few days ago. So I immediately removed all sideloaded plugins.

The attack is called malicious redirect. The plugin puts creepy pages in WordPress directory without my awareness and direct user traffic via my website. To clean up the damage, I looked into my WordPress directories at /opt/bitnami/wordpress and found many suspicious signs:

There are directories with weird names, such as rexall-vitalmin, or q4lee3, etc
In each of those directories there was an index.php file and .htaccess file;
Those directories also have other files which look like red herrings;
All those files have the same date time (from July 6);

Other directories to look at are /bitnami/wordpress/wp-content/plugins, where I noticed two directories (named gokyfozaxy and q199n071) that are not accounted for; and /bitnami/wordpress/wp-content/themes/, which contains unknown directories.

Clean up and hardening

In one of the .htaccess file I noticed segments of mojibake (garbled texts). I first tried to manually remove those files, but the problems stayed. Because the malicious redirect did not happen consistently on every single click, I sometime had false impression that the problem went away.

However, the challenges with manual cleaning are: 1. there are too many bad files (.htaccess and php files containing mojibake segments); 2. some existing files are impacted with mojibake segments too. I found a free plugin called WordFence to scan the file directory for malicious chagnes, and delete the bad files or bad segments. I also tried a paid scanner (Malcare) which found an bad file in /bitnami/wordpress/wp-content/themes/. However, it also blocked my site so I removed Malcare right away. Using the combination of WordFence and Malcare appears to have cleared up the offending files. After restarting apache, the bad URLs are no longer redirecting to spam sites.

This time, I decided to harden the WordPress system given the evidence of brute force attack at different point of entries. At OS level, I mentioned the changes to SSH daemon configuration. At WordPress level, I used WordFence to perform several levels of scans for problems and and improved posture such as admin user’s MFA. I also noticed a few unrecognized wordpress users and used wordpress CLI to delete those and other unused users.

The wordpress.org website has some general guidance on what to do when a site is hacked, and a general guidance on hardening WordPress.

Back Links

Another clean up work I had to do is dealing with back links. Back links are URLs from other sites that references this site. There are several situations:

If it’s a made-up URL, then it returns 404. In my case, these are URLs that stopped working once I cleaned up my server from the incident. However, the sources are still using these bad URLs. They are bad back-links;
If it’s a legit URL, look at if it’s hot linking, such as another site directly access an image from my site. These are bad back-links;
If it’s legit URLs, and the referrer site has a good domain authority score. These are likely to be good back links

Generally, it is painful to deal with bad back links because I’m not in control. I used a few free backlink checker tools (e.g. Links report on Google Search Console, SEOMATOR, SEMRush free) and found a lot of spammy sites that I had to request Google to disavow. Otherwise, they may negatively impact the search performance.

Repercussions

In the next few months, my pages are no longer a stop for their redirect. However, web request for those invalid URLs keep coming. The bad pages are still in Google’s cache. There are a lot of page request with 404 return code, and we consider this an HTTP flood. The problem now is that the amount of 404 return code is impacting how my site ranks in search engine. To make it worse, the amount of these requests with invalid URL increase since August.

To fix this, there are two measures. First, in Google search console, I have to tell Google to remove those URLs from its cache. I have identified a number of prefix patterns, and submitted a request for each URL pattern. It takes google a day to have them cleared.

After that, the bad request will no longer come from Google users clicking on bad URL. In my case, the requests did not reduce significantly, suggesting that most of the requests come from bots. Therefore I had to figure out a way to prevent those bad request hitting my server, which is a typical web application firewall requirement.

Looking for such a solution for my WordPress Security I landed on Cloudflare. Cloudflare is pretty user-friendly with an easy-to-understand reference architecture. When I started, Cloudflare can import my DNS records, and guided me to change my name servers so I delegate my DNS management it. When I first move to Cloudflare the website gives ERR_TOO_MANY_REDIRECTS. I ended up having to go to SSL/TLS and set encryption mode to Full (strict) to get rid of this error. I also have to re-configure email forwarding as a result of name server change.

CloudFlare

Even for a self-hosted single-server site, it is very beneficial to place an Application Firewall upfront for WordPress security. I find CloudFlare are very useful service that provides everything else you need to host the web site. For example, it contains a domain registry itself. It manages DNS and allows email forwarding. In addition, it helps generate TLS certificate etc. The free tier covers everything for a small website, with the Application Firewall as the core feature. Within the free tier I can have these features:

domain registrar and name servers (not for free but at a reasonable cost)
SSL certificate (not for free but at a reasonable cost)
Request event tracking
redirect rule: zone apex to www, and /status to uptime status page
return code 409 for obsolete URLs (using routes and workers)
email routing and forwarding
WAF rules (path, parameter, rate, etc)
DDoS protection and Bot Fight mode
hot-linking prevention (i.e. other sites references images on your site directly)

I am still exploring features for CloudFlare. One stunning feature is routes and workers. Essentially you can serve a function in response to HTTP request at a specific route. This is particularly useful in scenarios where it is not straightforward to add web pages on the backend server. For example, I want requests with certain paths to return HTTP code 490 and do not want to mock with the WordPress server, we can make use of CloudFlare worker.

Lessons Learned

For WordPress security, never use suspicious plugins. Keep an additional layer of defense in WordPress such as Wordfense. It helps block malicious traffic that went through the first layer. It also helps configure MFA for administrators.

Connect kubectl to private Kubernetes cluster in EKS and AKS

Sat, 10 Jun 2023 19:31:00 -0400

Managed Kubernetes services give user a cluster endpoint and a number of worker nodes, with the choice. For each access, users have the choice of making them publicly available, or keeping them on private networking. In my opinion, any deployment beyond personal hobbies, should use Kubernetes private cluster, with both cluster endpoint and worker nodes on private subnet. There is no reason to expose computing nodes or Kubernetes management traffic publicly.

For worker nodes, it is fairly easy to put VMs on private network, but many companies still have the cluster endpoint exposed publicly. There are usually two reasons. First, their CI/CD agent is hosted somewhere else on the Internet (instead of on private network with private connectivity to Kubernetes cluster) and need to access Kubernetes cluster endpoint. Second, when the cluster needs to connect with third-party identity provider as OIDC provider, a two-way communication is needed.

There is a classic pattern of using a public bastion host (jump box), with a bastion host on the public subnet, routable to the private endpoint of managed Kubernetes service. Clients then connect to the bastion host via port 22 on a public IP address. The authentication is based on SSH key pair, or worse, password. The port forwarding (aka SSH tunnelling) capability enables all the magics.

Exposing a jump box in the public subnet with RSA key authentication is still not favourable. In this post, I’ll examine some secure patterns to connect to private endpoint with improved security posture.

AWS options

There are two problems. First, how to establish connectivity to the Bastion host in a private subnet. Second, how to use the Bastion host to proxy traffic to the cluster endpoint also in private subnet. To the first problem, there are two potential solutions: SSM Session Manager, and EC2 Instance Connect (EIC) with EIC endpoint (EICE).

SSM Session Manager was introduce in 2018. It runs an agent on the EC2, which initiates a connection to the SSM endpoint on the AWS side. This connection enables not only Session Manager, but also other Systems Managers (SSM) services such as Fleet Manager, Patch Manager and State Manager. The problem that session manager originally addresses is server management.

AWS launched EC2 Instance Connect (EIC) in 2019, and EIC Endpoint (EICE) in 2023. EIC addresses the problem with managing SSH key pairs at scale. It dynamically generates an SSH key pair for server access, based on IAM permission. However, it still requires an instance to have its SSH port publicly accessible. With EICE, it is no longer a requirement. In the diagram, EICE is placed in a private subnet, allowing EICE service to reach private instances at their SSH port.

Here is a comparison of the two:

	EC2 Instance Connect (EIC) with EIC Endpoint	SSM Session Manager
Location of Bastion host	Private Subnet.	Private Subnet
Need Ingress Port	Yes. Port 22 must open to the endpoint.	No. SSM agent initiate outbound connection from the instance
Traffic Path	AWS CLI → AWS EIC ES → EICE→EC2 Inst	AWS CLI → AWS SSM ES → SSM ← EC2 Inst
Authentication	AWS IAM and ephemeral SSH key when using AWS CLI directly AWS IAM and long-term SSH key when using SSH proxy command	AWS IAM and long-term SSH key when using AWS CLI directly or SSH proxy command
Work with OpenSSH	Yes	Yes
Cost	There is no additional cost for using EIC.	No additional cost, unless private SSM Endpoint.

Let’s take a look at each option.

EC2 Instance Connect

To use EIC, pick an AMI that has it pre-installed and ensure instance profile has correct policy, as the document states here. AWC CLI will make use of local OpenSSL client. So make sure there connection at port 22 is open. To make it work with EC2 instance on a private subnet, create an EC2 Instance Connect Endpoint on the VPC, and ensure that the security group of EC2 allows port 22 from the Endpoint. Run this command:

$ aws ec2-instance-connect ssh --instance-id i-00ea30a6e02db33fe

The command above simply generates a key pair internally, add the public key to the server side, and connect with SSH from the client side. The command takes you to an SSH session. Checking ps -ef | grep ssh on the client machine, you can see the full parameter of SSH, including the location of the ephemeral private key.

However, if you use AWS CLI open-tunnel as proxy command to ssh, then you’d still have to use the key pair used to create the EC2 instance. As suggested at the bottom of this blog post, the command is:

$ ssh ec2-user@[INSTANCE] -i [SSH-KEY] -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id %h'

This is a bummer, because with native SSH tool you do not get the primary benefit of EIC – ephemeral key pair.

SSM Session Manager

Now let’s look at SSM session manager. Similarly, it needs an agent installed and IAM role configured. You can connect to from web console but more importantly, from AWS CLI:

$ aws ssm start-session --target i-0531b19bec8ad022d

This command takes you to an SSH session with user ssm-user, without starting an OpenSSH client process locally. User do not have to manage key pair. There is also a document about using this command as proxy command, which uses an SSM document. I have one of the SSH config entry as:

host i-* mi-*
 ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
 User ec2-user
 IdentityFile ~/.ssh/id_rsa

This allows me to directly SSH to client using OpenSSL client (e.g. ssh i-0531b19bec8ad022d) by Instance ID. With this, I also need to specify my own OS user and matching private key.

I know I will use the OpenSSH client a lot from pipelines because it is very powerful. In both options, I have to live with managing key pairs myself. With SSM session manager’s proxy command, the instance does not need port 22 to open, which is a great advantage, in terms of security and operation. SSM Session Manager is a winner.

SOCKS5 proxy for kubectl

Either SSM Session Manager or EIC with EICE enables an SSH tunnel with key encryption between client (a local computer or a pipeline agent). On top of the SSH tunnel, we can build a SOCKS5 proxy. Kubernetes document has a good page on how to do this. I managed to get this to work with a few gotchas.

To put this in practice, I first created a VPC stack with a bastion host using terraform template from my vpc-base project. The terraform output will give the next set of commands to run to create a private cluster, using a manifest rendered from the file private-cluster.yaml.tmpl:

# cd aws_vpc
# terraform init
# terraform plan
# terraform apply
# ... run the given command ...
# envsubst < private-cluster.yaml.tmpl | tee | eksctl create cluster -f -

# Run this from a remote host without access to cluster endpoint.
# Run terraform apply and terraform output contains the variables needed for the next steps

# the command below may take 15 minutes to create a private cluster

eksctl create cluster -f private-cluster.yaml 

aws eks update-kubeconfig --name private-cluster

At this point, the kubeconfig file has been updated, but kubectl (from Internet or on-prem) is unable to connect to cluster endpoint (on private network). In order to

BASTION_SECURITY_GROUP_ID=$(terraform output -raw bastion_sg_id)
CLUSTER_SECURITY_GROUP_ID=$(aws eks describe-cluster --name private-cluster --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text)

# In Cluster Endpoint's security group, open up port 443 to Bastion host
aws ec2 authorize-security-group-ingress --group-id $CLUSTER_SECURITY_GROUP_ID --source-group $BASTION_SECURITY_GROUP_ID --protocol tcp --port 443

# Test with connecting to Bastion host with ssh i-0750643179667a5b6, assuming .ssh/config file is configured as above. From the bastion host, you can test:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api 
# if you get an HTTP response, even an error code 403, the bastion host has TCP connectivity to cluster endpoint

# then we can start an SSH session as a SOCKS5 proxy on the remote host
ssh -D 1080 -q -N i-0750643179667a5b6

# add > /dev/null 2>&1 & to push it to background, or use ctrl+z after running the command
# to validate that the SOCKS5 proxy is working, you can run the same curl command with a proxy parameter:
# curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api --proxy socks5://localhost:1080

# you can instruct kubectl to use the SOCKS5 proxy with the following environment variable
export HTTPS_PROXY=socks5://localhost:1080

kubectl get node

# alternatively, add "proxy-url: socks5://localhost:1080" below server attribute in ~/.kube/config file.

There are some pitfalls to watch for. On the remote host both ssh command and kubectl command implicitly uses AWS CLI. Therefore, make sure the profile and IAM role are correctly configured. For example, if SSM agent requires one IAM role, and kubectl is created with another IAM role, then make sure AWS CLI assumes the correct IAM role using environment variables, and use “aws sts get-caller-identity” to validate the IAM identity being used.

What about AKS in Azure

I touched on this in my Azure notes in 2021 and did a research again. Unfortunately, options are still fairly limited.

The first option is to use a managed service called “Azure Bastion”, which requires public IP and a dedicated subnet with the exact name of AzureBastionSubnet, as well as some additional requirement. I’m not impressed with these requirement because it is meant to be a managed service. The other option, is essentially to DIY a JumpBox. The idea is the same: put the jumpbox in a public subnet, which is routable to private subnets. When you need to connect to private VMs, get to the jumpbox first.

Apart from having to put the bastion VM on a public subnet, the pattern that we discussed above involving SOCKS5 proxy still works. Exposing a bastion host isn’t ideal but it still reduces attack surface significantly, comparing to exposing the cluster endpoints of all Kubernetes API servers.

Summary

Many immature Kubernetes configurations exposes private endpoint publicly. Having cluster endpoint in private subnet greatly improves security posture. In my opinion, there are very few situations where cluster endpoint must exposed publicly. Having private endpoint should be mandatory for all Kubernetes cluster. In the next post, I also cover how to create a ROSA cluster with private endpoint.

IPVS, iptables and kube-proxy

Tue, 24 Nov 2020 13:17:00 -0400

This is an overview of the underlying technologies that drives load balancing. It covers LVS, Netfilter, iptables, IPVS and eventually kube-proxy.

LVS (Linux Virtual Server)

One of the ways to implement software load balancing is via LVS (Linux Virtual Server), as previously discussed. The diagram below shows the LVS framework, with IPVS as the fundamental technology:

The major work of the LVS project is to develop advanced IP load balancing software (IPVS), application-level load balancing software (KTCPVS), cluster management components. KTCPVS implements application-level load balancing inside the Linux kernel (still under development). IPVS is an advanced IP load balancing software implemented inside the Linux kernel. The IPVS code was already included into the standard Linux kernel 2.4 and 2.6.

Netfilter

Both IPVS and iptables (the technology behind Linux firewall, discussed here) are based on netfilter, a packet-filtering framework provided by the Linux kernel. In this section, we will discuss them all together, starting with Netfilter and then discuss how iptables and IPVS uses netfilter.

Netfilter allows various networking-related operations to be implemented in the form of customized handlers, by offers various functions and operations for packet filtering, network address translation, and port translation, which provide the functionality required for directing packets through a network and prohibiting packets from reaching sensitive locations within a network. Netfilter represents a set of hooks inside the Linux kernel, allowing specific kernel modules to register callback functions with the kernel’s networking stack. Those functions, usually applied to the traffic in the form of filtering and modification rules, are called for every packet that traverses the respective hook within the networking stack.

Iptables

The kernel modules named ip_tables, ip6_tables, arp_tables (the underscore is part of the name), and ebtables comprise the legacy packet filtering portion of the Netfilter hook system. They provide a table-based system for defining firewall rules that can filter or transform packets. The tables can be administered through the user-space tools iptables, ip6tables, arptables, and ebtables. Notice that although both the kernel modules and userspace utilities have similar names, each of them is a different entity with different functionality.

When a network packet is received on a network device, it first passes through the Prerouting hook. This is where the routing decision takes place. The kernel decides whether the packet is destined for a local process (e.g., a listening socket on a server in this system) or whether to forward it (system operates as a router). In the first case, the packet passes the Input hook and is then handed over to the local process. If the packet is destined to be forwarded, it traverses the Forward hook and then a final Postrouting hook before being sent out on a network device. For packets that are generated locally (e.g., by a client or server process that likes sending things out), they must first pass the Output hook and then the Postrouting hook before being sent out on a network device.

The aforementioned hooks exist independently for the IPv4 and IPv6 protocols. Thus, IPv4 and IPv6 packets each traverse their own hooks. There are also other hooks for ARP packets and for Bridging. And all the hooks exist independently within each network namespace. Additionally, there is an ingress hook for each network device. The list goes on… More explanations are from here and here.

IPVS

In LVS, IPVS is also based on netfilter framework, but works only on INPUT chain, by registering ip_vs_in hook function, to process request. IPVS (aka layer-4 switching) runs on a host at the front of a cluster of real servers. It directs requests for TCP/UDP based servers to the real server, while ensuring the resonse from (one or several) real server appears to the client as if they were all from a virtual service on a sigle IP address. It is based on in-kernel hash tables. The userspace utility is ipvsadm.

When the client request reaches the kernel space of load balancer, it arrives at PREROUTING chain. Route will determine whether the request packet is for the local host or not, based on the destination address of the packet. The packet is sent to INPUT chain if it is. The ip_vs_in function is hooked to LOCAL_IN and will examine the packet. If it finds a matching IPVS rule, it will (bypass INPUT chain) directly trigger POSTROUTING chain, skipping iptables rules.vThis is discussed in detail here.

IPVS supports 8 load balancing algorithms (round robin, weighted round robin, least-connection, weighted least connection, locality-based least-connection, locality-based least-connection with replication, destination-hashing, and source-hashing) and 3 packet-forwarding methods (NAT, tunneling and direct routing).

The main difference between iptables and IPVS, is iptables includes a number of tables, each with a number of chains, each further involves a number of rules. The total number of rules is large. The packet is assessed against many of such rules. For the same reason, the order of the rule matters. IPVS on the other hand, leverages hash table, with a complexity of O(1), or O(n) in the worst case scenarios. They vary significantly in the efficiency of packet filtering and forwarding, especially when the rules gets complicated. Iptable also presents more latency when adding or removing rules as more rules are involved. This presentation includes some quantitative comparison.

KubeProxy

In Kubernetes architecture, KubeProxy takes care of load balancing. Kube-proxy can run in three modes: userspace, iptables and IPVS.

userspace proxy mode

The userspace mode is old and inefficient. The packet is compared against iptables rule and then forwarded to a pod named kube-Proxy, which operates as an application to forward packet to backend pods.

iptables proxy mode

The iptables mode is better since it uses the kernel feature of iptables, which is fairly mature. kube-proxy manages iptables rule based on the service yaml of Kubernetes.

IPVS proxy mode

With the comparison between iptables and IPVS earlier, we can expect that iptables operations slow down dramatically in large scale cluster. Therefore IPVS based kubeproxy was brought up. This presentation illustrated the differences.

In this post we discussed load balancing technologies from ipvs to iptables and then to kube-proxy, which is used in Kubernetes nodes.

Redhat Firewall configuration: from iptables to firewalld

Fri, 12 Oct 2018 17:56:00 -0400

Tools to manage firewall

Packet filter rules in Linux Kernel is managed by an user-space application named iptables in CentOS and RedHat. Since CentOS 7, firewalld is introduced as an alternative to iptables. Firewalld can be installed and executed as a systemd service, and it is supposed to replace iptables. This article describes how to configure both.

There are several advantages in firewalld. One is is the support of zones. Here are some useful information. Also, iptables involves three different services for IPv4(iptables), IPv6(ip6tables), and software bridging (ebtables), whereas firewalld only involves a single service to manage all three. Firewalld allows user to add or remove rules/ports from running firewall, without restarting firewall. Unless you have specific reason to use iptables, always use firewalld service to manage firewall. Here is an instruction to firewalld service. In this posting however, we will be focusing on iptables to understand firewall managment. We also go through an example of opening a TCP port.

How does iptables work

When working with iptables, it is important to understand that its related concepts (tables->chains->rules->criteria and targets) and how the order of rules plays a factor. There are five independent tables, each contains a number of chains, either built-in or user-defined. Administrators mostly deals with built-in chains in filter and nat tables. The five tables are:

filter: If -t isn’t specified, this is the default table. It contains built-in chains:
- INPUT: for packet destined to local sockets
- FORWARD: for packets being routed through the box
- OUTPUT: for locally-generated packets
nat: this table is consulted when a packet that creates a new connection is encountered. It has three built-in chains:
- PREROUTING: for altering packets as soon as they come in
- OUTPUT: for altering locally generated packets before routing
- POSTROUTING: for altering packets as they are about to go out
mangle: this table is used for specialized packet alternation, with five built-in chains (since kernel 2.4.18): PREROUTING and OUTPUT, INPUT, FORWARD, and POSTROUTING
raw: this table is mainly for configuring exceptions from connection tracking with two built-in chains: PREROUTING and OUTPUT
security: for Mandatory Access Control (MAC) networking rules, with three built-in chains: INPUT, OUTPUT, and FORWARD.

Under the table (e.g. filter, nat), each chain (e.g. INPUT, OUTPUT, etc) consists of list of firewall rules. Each rule is made up of two parts defined for the packets:

Criteria: if the packet does not match the criteria, the next rule in the chain is examined; if it does match, then the next rule is specified by the value of the target.
Target: what to do if criteria is met. The target can be:
- user-defined chain,
- one of the target described in iptables-extensions, or
- in most cases, one of the special values ACCEPT, DROP or RETURN
  - ACCEPT – let the packet through
  - DROP – drop the packet on the floor
  - RETURN – stop traversing this chain, and resume at next rule in the previous (calling) chain

The rules, defined in each chain under their tables, can be found in file /etc/sysconfig/iptables. You can find tables (prefix with asterisk *), chains (prefix with colon :), rules under their chains and a statement COMMIT after each table. The iptables process flow illustrates how a packet interact with all these rules under different chains and tables defined in this file:

iptables Process Flow

Although this big picture looks formidable, an administrator commonly only deals with the green and purple blocks (filter and nat), with the big picture in mind. Here is an example of /etc/sysconfig/iptables file from a newly installed system:

# Generated by iptables-save v1.4.21 on Fri Sep 11 23:15:32 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [132:17200]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Fri Sep 11 23:15:32 2017

The rule simply allows SSH traffic. This file will be loaded up on every reboot (specifically, restart of iptables service). So if you have made some changes to rules and you want the change picked up on reboot. The rules should be saved to this file:

$ sudo iptables-save > /etc/sysconfig/iptables

Other than saving rule for reboot, if you simply want to edit the rules (e.g. order of rules is incorrect), you can save the rules to file, modify the file and restore the rule from file:

$ sudo iptables-save > ~/iptables.txt
$ sudo vi ~/iptables.txt
$ sudo iptables-restore &lt; ~/iptables.txt

Here is some further reading about iptables architecture.

Anatomy of a rule

The man page for iptables species the following synopsis:

iptables [-t table] {-A|-C|-D} chain rule-specification
rule-specification = [matches...] [target]
match = -m matchname [per-match-options]
target = -j targetname [per-target-options]

So when you append (-A), delete (-D), insert (-I) or replace (-R) a rule, you need to specify rule specification. The man page further explains that the following parameters make up a rule specification:

protocol (-p): the protocol of the rule of the packet to check. value can be tcp, udp, icmp, all or any name defined in /etc/protocols.
match (-m): specifies the name of a match to use and is followed by match options. The match refers to an extension module that tests for a specific property. Those extension modules are documented in the man page of iptables-extensions. You may specify -m multiple times for different match names, which together make up the condition under which a target is invoked. Matches are evaluated first to last as specified. We often use extensions tcp and state. According to iptables-extensions man page, we can specify –dport followed by port number for the tcp extension, and –state followed by value such as NEW or ESTABLISHED for the state extension.
jump (-j): specifies the target of the rule, such as ACCEPT, REJECT or DROP.
source and destination (-s and -d): source and destination IP address or masks. Hostname will work but not recommended since resolution is needed.
inbound and outbound interface (-i and -o): name of interface via which the packet was received and is going to be sent.
goto (-g): processing should continue in a user specified chain
Other parameters: -4/–ipv4, -6/–ipv6, -c/–set-counters, -f/–fragment

When we run iptables command to view rules, we need to specify the table (e.g. filter, nat, etc) followed by -S or –list-rules:

$ iptables -t nat -S

If you do not specify -t switch, the default (-t filter) is applied. Be aware that in this case, you’re only seeing rules under filter table, and not all rules under tall tables!

In the result, for example one line from command “iptables -S” may say:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 9200 -j ACCEPT

The interpretation: appending a rule to INPUT chain of filter table (implicitly specified). The protocol is tcp. The first match extension is state, and the state value shall be NEW. The second match extension is tcp, and the dport value shall be 9200. If the packet is a match, then the target (action) is ACCEPT.

Managing rules

As mentioned earlier, rules can be dumped to any file or /etc/sysconfig/iptables, in which the rules are assessed in order. Below is a real life iptables file with a nat table as well.

# Generated by iptables-save v1.4.21 on Wed Jan 15 13:58:39 2017
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [4:208]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7000:7001 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7199 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9042 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9160 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 161 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 162 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

*nat
:PREROUTING ACCEPT [1:328]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398
-A PREROUTING -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398
-A OUTPUT -o lo -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398
-A OUTPUT -o lo -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398
COMMIT
# Completed on Wed Jan 15 13:58:39 2017

In this example, the nat table defines traffic forwarding: traffic arriving at TCP port 2392 and 2393 are forwarded to port 2398; outgoing traffic to port 2392 and 2393 are also redirected to port 2398. These rules do not overlap each other so the rules probably don’t matter.

On the other hand, the tcp filter table lists the rules to open certain TCP and UDP ports. Its block starts with a couple accepting rules and ends with a couple reject rules (regardless of protocols or ports). This is a good way to close a chain of rules with security. However, if you need to add additional rules to open more TCP ports, the new rule should not be appended after the reject rules at the bottom since the order matter here!

Correct way to open a TCP port

It’s a common task for developers to open a TCP port simply for the purpose of bring up a web service and make it accessible to client. If we simply add a new rule to existing list, for example:

# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9870 -j ACCEPT
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

# systemctl reload iptables

You will notice that the rule is appended to the end of INPUT block, below the INPUT REJECT rule, which will never take effect.

To address this, you can use iptables-save and iptables-restore to export, edit to correct order and reload the rule, as illustrated above, instead of using iptables command to modify the rule directly.

# iptables-save > /tmp/rule.list
# vi /tmp/rule.list
# iptables-restore < /tmp/rule.list
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

Alternatively, you could use some advanced iptables command switches to add the new rule to certain line number with –line-number switch. Here is more information.