Amazon Elastic Kubernetes Service (EKS)
Big picture
Enable Calico in EKS managed Kubernetes service.
Value
EKS has built-in support for Calico, providing a robust implementation of the full Kubernetes Network Policy API. EKS users wanting to go beyond Kubernetes network policy capabilities can make full use of the Calico Network Policy API.
You can also use Calico for networking on EKS in place of the default AWS VPC networking without the need to use IP addresses from the underlying VPC. This allows you to take advantage of the full set of Calico networking features, including Calico’s flexible IP address management capabilities.
Before you begin…
Note: Calico makes use of the Kubernetes Container Storage Interface (CSI) to support various types of volumes. The necessary drivers required for CSI to function correctly in EKS clusters will no longer be present by default in clusters running Kubernetes 1.23. Please see the following link to ensure your cluster is configured correctly based on the version of Kubernetes being used in your cluster: AWS EBS CSI driver
How to
Install EKS with Amazon VPC networking
The geeky details of what you get:
Kubernetes network policies are implemented by network plugins rather than Kubernetes itself. Simply creating a network policy resource without a network plugin to implement it, will have no effect on network traffic.
The Calico plugin implements the full set of Kubernetes network policy features. In addition, Calico supports Calico network policies, providing additional features and capabilities beyond Kubernetes network policies. Kubernetes and Calico network policies work together seamlessly, so you can choose whichever is right for you, and mix and match as desired.
How Kubernetes assigns IP address to pods is determined by the IPAM (IP Address Management) plugin being used.
The AWS IPAM plugin dynamically allocates small blocks of IP addresses to nodes as required, using IP addresses from the underlying VPC (Virtual Private Cloud). The AWS IPAM plugin is used in conjunction with the Amazon VPC CNI plugin to provide VPC native pod networking.
The CNI (Container Network Interface) plugin being used by Kubernetes determines the details of exactly how pods are connected to the underlying network.
The AWS Amazon VPC CNI and IPAM plugins provide pods with IP addresses from the underlying VPC (Virtual Private Cloud) to provide a VPC-Native pod network. The AWS VPC is used to route pod traffic between nodes, and understands which pod IP address are located on which nodes. This avoids the need for an overlay, and typically has good network performance characteristics.
In addition, pod IPs are understood by the broader AWS network, so for example, VMs outside of the cluster can connect directly to any pod without going via a Kubernetes service if desired.
Operating without using an overlay provides the highest performance network. The packets that leave your pods are the packets that go on the wire.
For completeness, in contrast, with an overlay network, packets between pods on different nodes are encapsulated using a protocol such as VXLAN or IPIP, wrapping each original packet in an outer packet that uses node IPs, and hiding the pod IPs of the inner packet. This can be done very efficiently by the Linux kernel, but it still represents a small overhead, which you might want to avoid if running particularly network intensive workloads.
The underlying cloud VPC (Virtual Private Cloud) is used to route pod traffic between nodes, and understands which pod IP address are located on which nodes. This avoids the need for an overlay, and typically has good performance characteristics.
In addition, pod IPs are understood by the broader cloud network, so for example, VMs outside of the cluster can connect directly to any pod without going via a Kubernetes service if desired.
Calico stores the operational and configuration state of your cluster in a central datastore. If the datastore is unavailable, your Calico network continues operating, but cannot be updated (no new pods can be networked, no policy changes can be applied, etc.).
Calico has two datastore drivers you can choose from
- etcd - for direct connection to an etcd cluster
- Kubernetes - for connection to a Kubernetes API server
The advantages of using Kubernetes as the datastore are:
- It doesn’t require an extra datastore, so is simpler to install and manage
- You can use Kubernetes RBAC to control access to Calico resources
- You can use Kubernetes audit logging to generate audit logs of changes to Calico resources
For completeness, the advantages of using etcd as the datastore are:
- Allows you to run Calico on non-Kubernetes platforms (e.g. OpenStack)
- Allows separation of concerns between Kubernetes and Calico resources, for example allowing you to scale the datastores independently
- Allows you to run a Calico cluster that contains more than just a single Kubernetes cluster, for example, bare metal servers with Calico host protection interworking with a Kubernetes cluster or multiple Kubernetes clusters.
Calico’s flexible modular architecture supports a wide range of deployment options, so you can select the best networking and network policy options for your specific environment. This includes the ability to run with a variety of CNI and IPAM plugins, and underlying networking options.
The Calico Getting Started guides default to the options most commonly used in each environment, so you don’t have to dive into the details unless you want to.
You can click on any deployment option to learn more.
To enable Calico network policy enforcement on an EKS cluster using the AWS VPC CNI plugin, follow these step-by-step instructions: Installing Calico on Amazon EKS
Install EKS with Calico networking
The geeky details of what you get:
Kubernetes network policies are implemented by network plugins rather than Kubernetes itself. Simply creating a network policy resource without a network plugin to implement it, will have no effect on network traffic.
The Calico plugin implements the full set of Kubernetes network policy features. In addition, Calico supports Calico network policies, providing additional features and capabilities beyond Kubernetes network policies. Kubernetes and Calico network policies work together seamlessly, so you can choose whichever is right for you, and mix and match as desired.
How Kubernetes assigns IP address to pods is determined by the IPAM (IP Address Management) plugin being used.
The Calico IPAM plugin dynamically allocates small blocks of IP addresses to nodes as required, to give efficient overall use of the available IP address space. In addition, Calico IPAM supports advanced features such as multiple IP pools, the ability to specify a specific IP address range that a namespace or pod should use, or even the specific IP address a pod should use.
The CNI (Container Network Interface) plugin being used by Kubernetes determines the details of exactly how pods are connected to the underlying network.
The Calico CNI plugin connects pods to the host networking using L3 routing, without the need for an L2 bridge. This is simple and easy to understand, and more efficient than other common alternatives such as kubenet or flannel.
An overlay network allows pods to communicate between nodes without the underlying network being aware of the pods or pod IP addresses.
Packets between pods on different nodes are encapsulated using VXLAN, wrapping each original packet in an outer packet that uses node IPs, and hiding the pod IPs of the inner packet. This can be done very efficiently by the Linux kernel, but it still represents a small overhead, which you might want to avoid if running particularly network intensive workloads.
For completeness, in contrast, operating without using an overlay provides the highest performance network. The packets that leave your pods are the packets that go on the wire.
Calico routing distributes and programs routes for pod traffic between nodes using its data store without the need for BGP. Calico routing supports unencapsulated traffic within a single subnet, as well as selective VXLAN encapsulation for clusters that span multiple subnets.
Calico stores the operational and configuration state of your cluster in a central datastore. If the datastore is unavailable, your Calico network continues operating, but cannot be updated (no new pods can be networked, no policy changes can be applied, etc.).
Calico has two datastore drivers you can choose from
- etcd - for direct connection to an etcd cluster
- Kubernetes - for connection to a Kubernetes API server
The advantages of using Kubernetes as the datastore are:
- It doesn’t require an extra datastore, so is simpler to install and manage
- You can use Kubernetes RBAC to control access to Calico resources
- You can use Kubernetes audit logging to generate audit logs of changes to Calico resources
For completeness, the advantages of using etcd as the datastore are:
- Allows you to run Calico on non-Kubernetes platforms (e.g. OpenStack)
- Allows separation of concerns between Kubernetes and Calico resources, for example allowing you to scale the datastores independently
- Allows you to run a Calico cluster that contains more than just a single Kubernetes cluster, for example, bare metal servers with Calico host protection interworking with a Kubernetes cluster or multiple Kubernetes clusters.
Calico’s flexible modular architecture supports a wide range of deployment options, so you can select the best networking and network policy options for your specific environment. This includes the ability to run with a variety of CNI and IPAM plugins, and underlying networking options.
The Calico Getting Started guides default to the options most commonly used in each environment, so you don’t have to dive into the details unless you want to.
You can click on any deployment option to learn more.
Note: Calico networking cannot currently be installed on the EKS control plane nodes. As a result the control plane nodes will not be able to initiate network connections to Calico pods. (This is a general limitation of EKS’s custom networking support, not specific to Calico.) As a workaround, trusted pods that require control plane nodes to connect to them, such as those implementing admission controller webhooks, can include
hostNetwork:truein their pod spec. See the Kubernetes API pod spec definition for more information on this setting.
For these instructions, we will use eksctl to provision the cluster. However, you can use any of the methods in Getting Started with Amazon EKS
Before you get started, make sure you have downloaded and configured the necessary prerequisites
-
First, create an Amazon EKS cluster without any nodes.
eksctl create cluster --name my-calico-cluster --without-nodegroup -
Since this cluster will use Calico for networking, you must delete the
aws-nodedaemon set to disable AWS VPC networking for pods.kubectl delete daemonset -n kube-system aws-node -
Now that you have a cluster configured, you can install Calico.
-
Install the operator.
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml -
Configure the Calico installation.
kubectl create -f - <<EOF kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: kubernetesProvider: EKS cni: type: Calico calicoNetwork: bgp: Disabled EOF
-
Install the Calico manifest.
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico-vxlan.yaml -
Configure Calico to disable AWS src/dst checks.
kubectl -n kube-system set env daemonset/calico-node FELIX_AWSSRCDSTCHECK=Disable
-
Finally, add nodes to the cluster.
eksctl create nodegroup --cluster my-calico-cluster --node-type t3.medium --max-pods-per-node 100Tip: Without the
--max-pods-per-nodeoption above, EKS will limit the number of pods based on node-type. Seeeksctl create nodegroup --helpfor the full set of node group options.
Next steps
Required
Recommended