Amazon Elastic Kubernetes Service (EKS)

3 MINUTE READ

Big picture

Enable Calico in EKS managed Kubernetes service.

Value

EKS has built-in support for Calico, providing a robust implementation of the full Kubernetes Network Policy API. EKS users wanting to go beyond Kubernetes network policy capabilities can make full use of the Calico Network Policy API.

You can also use Calico for networking on EKS in place of the default AWS VPC networking without the need to use IP addresses from the underlying VPC. This allows you to take advantage of the full set of Calico networking features, including Calico’s flexible IP address management capabilities.

How to

Install EKS with Amazon VPC networking

The geeky details of what you get:

Policy
Calico
IPAM
AWS
CNI
AWS
Overlay
No
Routing
VPC Native
Datastore
Kubernetes
?

To enable Calico network policy enforcement on an EKS cluster using the AWS VPC CNI plugin, follow these step-by-step instructions: Installing Calico on Amazon EKS

As discussed here, AWS CNI provisions multiple ENIs per node as the number of pods on the node increases. AWS CNI will add entries for the primary ENI into the main routing table, and will then create routing tables for each additional ENI, starting at index 2. Additionally, if VLANs are being used, it appears that AWS CNI will use tables from 100 onwards. By default, Felix considers routing table indexes from 1-250 to be under its control, and hence will remove the routing tables created by AWS CNI. This can cause loss of connectivity between pods if they are not on the primary ENI.

Note: The following steps will result in loss of connectivity between some pods. It is recommended to only make such changes during a maintenance window. To ensure that AWS CNI and Felix manage separate ranges of routing tables, you must do the following:

  1. Configure Felix to manage a routing table range which is distinct from the range used by AWS CNI:
     kubectl patch felixconfiguration default --type='merge' -p '{"spec": {"routeTableRange":{"min": 65, "max": 99}}}'
    
  2. Delete any routing rules and tables in the range 1-64 as they could be damaged or incomplete

  3. Kill all the aws-node pods, which will force AWS CNI to recreate its routing rules and tables.

Install EKS with Calico networking

The geeky details of what you get:

Policy
Calico
IPAM
Calico
CNI
Calico
Overlay
VXLAN
Routing
Calico
Datastore
Kubernetes
?

Note: Calico networking cannot currently be installed on the EKS control plane nodes. As a result the control plane nodes will not be able to initiate network connections to Calico pods. (This is a general limitation of EKS’s custom networking support, not specific to Calico.) As a workaround, trusted pods that require control plane nodes to connect to them, such as those implementing admission controller webhooks, can include hostNetwork:true in their pod spec. See the Kuberentes API pod spec definition for more information on this setting.

For these instructions, we will use eksctl to provision the cluster. However, you can use any of the methods in Getting Started with Amazon EKS

Before you get started, make sure you have downloaded and configured the necessary prerequisites

  1. First, create an Amazon EKS cluster without any nodes.

    eksctl create cluster --name my-calico-cluster --without-nodegroup
    
  2. Since this cluster will use Calico for networking, you must delete the aws-node daemon set to disable AWS VPC networking for pods.

    kubectl delete daemonset -n kube-system aws-node
    
  3. Now that you have a cluster configured, you can install Calico.

    kubectl apply -f https://docs.projectcalico.org/manifests/calico-vxlan.yaml
    
  4. Finally, add nodes to the cluster.

    eksctl create nodegroup --cluster my-calico-cluster --node-type t3.medium --node-ami auto --max-pods-per-node 100
    

    Tip: Without the --max-pods-per-node option above, EKS will limit the number of pods based on node-type. See eksctl create nodegroup --help for the full set of node group options.

Next steps

Required

Recommended