Story
When I was doing a research to design AWS EKS clusters from the ground up. Networking aspect was a part of my considerations. One problem was that how many IP addresses I need in a long run. While it had no simple answer, instead I could tell you that how to save your IP addresses.
Prerequsites
The technique I am going to introduce is based on AWS CNI plugin (Amazon VPC Container Network Interface plugin for Kubernetes), which is also the default CNI of your EKS cluster.
From the offical documentation, Using this plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network. This is why your pods can communicate to computing resources outside your clusters, or even in other VPCs.
And let me roughly explain the real-world scenerio:
Each node group provisons a group of EC2 instances. Each instance has multiple ENIs, and one ENI contains multiple IP addresses. AWS CNI plugin helps you to manage these IP addresses and associate to your EKS pods.
EKS features
Assign address prefixes to your ENI
By default, the number of IP addresses available to assign to pods is based on the number of IP addresses assigned to Elastic network interfaces, and the number of network interfaces attached to your Amazon EC2 node, i.e. AWS CNI plugin assigns IP addresses to ENIs.
EKS offers your another options, let AWS CNI plugin to assign / 28 IP address prefixes to ENIs.
Let me show you a simple example.
Consider that a ENI has ten slots:
- If you use the default mode, you can assign 10 IP addresses to it.
- If you use the prefixes mode, you can assign 10 /28 prefix blocks to it, i.e. 10 * 16 IP addresses = 160 IP addresses.
If you find out your instances have not fully utilized, you have a choice to optimize the pod-to-instance ratio, such that you can create fewer instances.
If it does not sound useful to you, prefixes mode has an addtional advantage. You can make less API calls to configure the network interfaces and IP addresses necessary for pod connectivity.
In the situation of rapidly scaling, API throttling might happen. Your EKS will stuck in a dead loop when it scales too quick.
Your pods need IP addresses, AWS VPC CNI will call APIs to allocate IP addresses for your pods. However API throttling prevent CNI to allocate IP addresses, then AWS VPC CNI keeps sending API calls ....
In the default mode, you are assigning a single IP addresses each time. Now you can assign 16 IP addresses each time. While it is not 100% correct, you can roughly assuming that it saves 16x API calls.
CNI custom networking
By default, when new network interfaces are allocated for pods, AWS CNI plugin uses the security groups and subnet of the node's primary network interface, i.e. ** both pods and nodes are created in the same subnet.**
Normally, we use RFC 1918 CIDR ranges (10.0.0.0/8, 172.16.0.0/12 and 192.16.0.0/16) to allocate our IP addresses.
However, many of you might not know, EKS supports additional IPv4 CIDR blocks in the 100.64.0.0/10 and 198.19.0.0/16 ranges.
The trick is simple. You can create your nodes in RFC 1918 CIDR ranges and then create your pods in 100.64.0.0/10 and 198.19.0.0/16 ranges. You can much more IP addresses to allocate now.
Please be reminded that 100.64.0.0/10 and 198.19.0.0/16 are not common IP ranges. Proper routing needs to be configured if you want the pods created in 100.64.0.0/16 and 192.19.0.0/16 to be able to have outbound connections to RFC 1918 CIDR ranges through VPC peering, transit gateway peering.
Why I emphasis outbound connections? Here is another trick, you can place load balancers in RFC-1918 ip ranges, and forward traffic to non RFC-1918 ip ranges pods. This way your clients only need to have route to your load balancers, without need to know anything of your pods.
Conclusion
Depends on your use cases, you can enable either or both Assign address prefixes to your ENI and CNI custom networking feature.
I also encourage you to understand what AWS VPC CNI plugin does if you are not familiar it before. It can help you to design EKS cluster in a better way.
If you are interested to use these features, AWS documentation has a very detailed explanation, which I have attached below. You should also checkout considerations and prerequisites in the documentation.
Reading
cni-increase-ip-addresses
cni-custom-network
Using additional IPv4 IP ranges
eni-max-pods
Top comments (0)