Install Calico networking and network policy for on-premises deployments
5 MINUTE READ
Big picture
Install Calico to provide both networking and network policy for self-managed on-premises deployments.
Value
Calico networking and network policy are a powerful choice for a CaaS implementation. If you have the networking infrastructure and resources to manage Kubernetes on-premises, installing the full Calico product provides the most customization and control.
Features
This how-to guide uses the following Calico features:
calico/node
Typha
Concepts
Calico manifests
Calico provides manifests for easy customization. Each manifest contains the necessary resources for installing Calico on each node in your Kubernetes cluster. You may want to customize the Calico manifests before installing Calico on nodes.
The Kubernetes API datastore is the recommended datastore for Kubernetes workloads.
Note: The etcd database is not recommended for new installs. However, it is an option if you are running Calico as the network plugin for both OpenStack and Kubernetes.
Install Calico on nodes
Based on your datastore and number of nodes, select a link below to install Calico.
Note: The option, Kubernetes API datastore, more than 50 nodes provides scaling using Typha daemon. Typha is not included for etcd because etcd already handles many clients so using Typha is redundant and not recommended.
If you are using pod CIDR 192.168.0.0/16, skip to the next step. If you are using a different pod CIDR with kubeadm, no changes are required - Calico will automatically detect the CIDR based on the running configuration. For other platforms, make sure you uncomment the CALICO_IPV4POOL_CIDR variable in the manifest and set it to the same value as your chosen pod CIDR.
Customize the manifest as necessary.
Apply the manifest using the following command.
kubectl apply -f calico.yaml
The geeky details of what you get:
Policy
Calico
Calico Policy
Kubernetes network policies are implemented by network plugins rather than Kubernetes itself. Simply creating a network policy resource without a network plugin to implement it, will have no effect on network traffic.
The Calico plugin implements the full set of Kubernetes network policy features. In addition, Calico supports Calico network policies, providing additional features and capabilities beyond Kubernetes network policies. Kubernetes and Calico network policies work together seamlessly, so you can choose whichever is right for you, and mix and match as desired.
IPAM
Calico
Calico IPAM
How Kubernetes assigns IP address to pods is determined by the IPAM (IP Address Management) plugin being used.
The Calico IPAM plugin dynamically allocates small blocks of IP addresses to nodes as required, to give efficient overall use of the available IP address space. In addition, Calico IPAM supports advanced features such as multiple IP pools, the ability to specify a specific IP address range that a namespace or pod should use, or even the specific IP address a pod should use.
CNI
Calico
Calico CNI
The CNI (Container Network Interface) plugin being used by Kubernetes determines the details of exactly how pods are connected to the underlying network.
The Calico CNI plugin connects pods to the host networking using L3 routing, without the need for an L2 bridge. This is simple and easy to understand, and more efficient than other common alternatives such as kubenet or flannel.
Overlay
IPIP
IPIP Overlay
An overlay network allows pods to communicate between nodes without the underlying network being aware of the pods or pod IP addresses.
Packets between pods on different nodes are encapsulated using IPIP, wrapping each original packet in an outer packet that uses node IPs, and hiding the pod IPs of the inner packet. This can be done very efficiently by the Linux kernel, but it still represents a small overhead, which you might want to avoid if running particularly network intensive workloads.
For completeness, in contrast, operating without using an overlay provides the highest performance network. The packets that leave your pods are the packets that go on the wire.
Routing
BGP
BGP Routing
BGP (Border Gateway Protocol) is used to dynamically program routes for pod traffic between nodes.
BGP is a standards-based routing protocol used to build the internet. It scales exceptionally well, and even the largest Kubernetes clusters represent a tiny amount of load compared to what BGP can cope with.
Calico can run BGP in three modes:
Full mesh - where each node talks BGP to each other, easily scaling to 100 nodes, on top of an underlying L2 network or using IPIP overlay
With route reflectors - where each node talks to one or more BGP route reflectors, scaling beyond 100 nodes, on top of an underlying L2 network or using IPIP overlay
Peered with TOR (Top of Rack) routers - in a physical data center where each node talks to routers in the top of the corresponding rack, scaling to the limits of your physical data center.
Datastore
Kubernetes
Kubernetes Datastore
Calico stores the operational and configuration state of your cluster in a central datastore. If the datastore is unavailable, your Calico network continues operating, but cannot be updated (no new pods can be networked, no policy changes can be applied, etc.).
Calico has two datastore drivers you can choose from
etcd - for direct connection to an etcd cluster
Kubernetes - for connection to a Kubernetes API server
The advantages of using Kubernetes as the datastore are:
It doesn’t require an extra datastore, so is simpler to install and manage
You can use Kubernetes RBAC to control access to Calico resources
You can use Kubernetes audit logging to generate audit logs of changes to Calico resources
For completeness, the advantages of using etcd as the datastore are:
Allows you to run Calico on non-Kubernetes platforms (e.g. OpenStack)
Allows separation of concerns between Kubernetes and Calico resources, for example allowing you to scale the datastores independently
Allows you to run a Calico cluster that contains more than just a single Kubernetes cluster, for example, bare metal servers with Calico host protection interworking with a Kubernetes cluster or multiple Kubernetes clusters.
?
Calico Deployment Options
Calico’s flexible modular architecture supports a wide range of deployment options, so you can select the best networking and network policy options for your specific environment. This includes the ability to run with a variety of CNI and IPAM plugins, and underlying networking options.
The Calico Getting Started guides default to the options most commonly used in each environment, so you don’t have to dive into the details unless you want to.
You can click on any deployment option to learn more.
Install Calico with Kubernetes API datastore, more than 50 nodes
Download the Calico networking manifest for the Kubernetes API datastore.
If you are using pod CIDR 192.168.0.0/16, skip to the next step. If you are using a different pod CIDR with kubeadm, no changes are required - Calico will automatically detect the CIDR based on the running configuration. For other platforms, make sure you uncomment the CALICO_IPV4POOL_CIDR variable in the manifest and set it to the same value as your chosen pod CIDR.
Modify the replica count to the desired number in the Deployment named, calico-typha.
We recommend at least one replica for every 200 nodes, and no more than
20 replicas. In production, we recommend a minimum of three replicas to reduce
the impact of rolling upgrades and failures. The number of replicas should
always be less than the number of nodes, otherwise rolling upgrades will stall.
In addition, Typha only helps with scale if there are fewer Typha instances than
there are nodes.
Warning: If you set typha_service_name and set the Typha deployment replica
count to 0, Felix will not start.
Customize the manifest if desired.
Apply the manifest.
kubectl apply -f calico.yaml
The geeky details of what you get:
Policy
Calico
Calico Policy
Kubernetes network policies are implemented by network plugins rather than Kubernetes itself. Simply creating a network policy resource without a network plugin to implement it, will have no effect on network traffic.
The Calico plugin implements the full set of Kubernetes network policy features. In addition, Calico supports Calico network policies, providing additional features and capabilities beyond Kubernetes network policies. Kubernetes and Calico network policies work together seamlessly, so you can choose whichever is right for you, and mix and match as desired.
IPAM
Calico
Calico IPAM
How Kubernetes assigns IP address to pods is determined by the IPAM (IP Address Management) plugin being used.
The Calico IPAM plugin dynamically allocates small blocks of IP addresses to nodes as required, to give efficient overall use of the available IP address space. In addition, Calico IPAM supports advanced features such as multiple IP pools, the ability to specify a specific IP address range that a namespace or pod should use, or even the specific IP address a pod should use.
CNI
Calico
Calico CNI
The CNI (Container Network Interface) plugin being used by Kubernetes determines the details of exactly how pods are connected to the underlying network.
The Calico CNI plugin connects pods to the host networking using L3 routing, without the need for an L2 bridge. This is simple and easy to understand, and more efficient than other common alternatives such as kubenet or flannel.
Overlay
IPIP
IPIP Overlay
An overlay network allows pods to communicate between nodes without the underlying network being aware of the pods or pod IP addresses.
Packets between pods on different nodes are encapsulated using IPIP, wrapping each original packet in an outer packet that uses node IPs, and hiding the pod IPs of the inner packet. This can be done very efficiently by the Linux kernel, but it still represents a small overhead, which you might want to avoid if running particularly network intensive workloads.
For completeness, in contrast, operating without using an overlay provides the highest performance network. The packets that leave your pods are the packets that go on the wire.
Routing
BGP
BGP Routing
BGP (Border Gateway Protocol) is used to dynamically program routes for pod traffic between nodes.
BGP is a standards-based routing protocol used to build the internet. It scales exceptionally well, and even the largest Kubernetes clusters represent a tiny amount of load compared to what BGP can cope with.
Calico can run BGP in three modes:
Full mesh - where each node talks BGP to each other, easily scaling to 100 nodes, on top of an underlying L2 network or using IPIP overlay
With route reflectors - where each node talks to one or more BGP route reflectors, scaling beyond 100 nodes, on top of an underlying L2 network or using IPIP overlay
Peered with TOR (Top of Rack) routers - in a physical data center where each node talks to routers in the top of the corresponding rack, scaling to the limits of your physical data center.
Datastore
Kubernetes
Kubernetes Datastore
Calico stores the operational and configuration state of your cluster in a central datastore. If the datastore is unavailable, your Calico network continues operating, but cannot be updated (no new pods can be networked, no policy changes can be applied, etc.).
Calico has two datastore drivers you can choose from
etcd - for direct connection to an etcd cluster
Kubernetes - for connection to a Kubernetes API server
The advantages of using Kubernetes as the datastore are:
It doesn’t require an extra datastore, so is simpler to install and manage
You can use Kubernetes RBAC to control access to Calico resources
You can use Kubernetes audit logging to generate audit logs of changes to Calico resources
For completeness, the advantages of using etcd as the datastore are:
Allows you to run Calico on non-Kubernetes platforms (e.g. OpenStack)
Allows separation of concerns between Kubernetes and Calico resources, for example allowing you to scale the datastores independently
Allows you to run a Calico cluster that contains more than just a single Kubernetes cluster, for example, bare metal servers with Calico host protection interworking with a Kubernetes cluster or multiple Kubernetes clusters.
?
Calico Deployment Options
Calico’s flexible modular architecture supports a wide range of deployment options, so you can select the best networking and network policy options for your specific environment. This includes the ability to run with a variety of CNI and IPAM plugins, and underlying networking options.
The Calico Getting Started guides default to the options most commonly used in each environment, so you don’t have to dive into the details unless you want to.
You can click on any deployment option to learn more.
Install Calico with etcd datastore
Note: The etcd database is not recommended for new installs. However, it is an option if you are running Calico as the network plugin for both OpenStack and Kubernetes.
If you are using pod CIDR 192.168.0.0/16, skip to the next step. If you are using a different pod CIDR with kubeadm, no changes are required - Calico will automatically detect the CIDR based on the running configuration. For other platforms, make sure you uncomment the CALICO_IPV4POOL_CIDR variable in the manifest and set it to the same value as your chosen pod CIDR.
In the ConfigMap named, calico-config, set the value of etcd_endpoints to the IP address and port of your etcd server.
Tip: You can specify more than one etcd_endpoint using commas as delimiters.
Customize the manifest if desired.
Apply the manifest using the following command.
kubectl apply -f calico.yaml
The geeky details of what you get:
Policy
Calico
Calico Policy
Kubernetes network policies are implemented by network plugins rather than Kubernetes itself. Simply creating a network policy resource without a network plugin to implement it, will have no effect on network traffic.
The Calico plugin implements the full set of Kubernetes network policy features. In addition, Calico supports Calico network policies, providing additional features and capabilities beyond Kubernetes network policies. Kubernetes and Calico network policies work together seamlessly, so you can choose whichever is right for you, and mix and match as desired.
IPAM
Calico
Calico IPAM
How Kubernetes assigns IP address to pods is determined by the IPAM (IP Address Management) plugin being used.
The Calico IPAM plugin dynamically allocates small blocks of IP addresses to nodes as required, to give efficient overall use of the available IP address space. In addition, Calico IPAM supports advanced features such as multiple IP pools, the ability to specify a specific IP address range that a namespace or pod should use, or even the specific IP address a pod should use.
CNI
Calico
Calico CNI
The CNI (Container Network Interface) plugin being used by Kubernetes determines the details of exactly how pods are connected to the underlying network.
The Calico CNI plugin connects pods to the host networking using L3 routing, without the need for an L2 bridge. This is simple and easy to understand, and more efficient than other common alternatives such as kubenet or flannel.
Overlay
IPIP
IPIP Overlay
An overlay network allows pods to communicate between nodes without the underlying network being aware of the pods or pod IP addresses.
Packets between pods on different nodes are encapsulated using IPIP, wrapping each original packet in an outer packet that uses node IPs, and hiding the pod IPs of the inner packet. This can be done very efficiently by the Linux kernel, but it still represents a small overhead, which you might want to avoid if running particularly network intensive workloads.
For completeness, in contrast, operating without using an overlay provides the highest performance network. The packets that leave your pods are the packets that go on the wire.
Routing
BGP
BGP Routing
BGP (Border Gateway Protocol) is used to dynamically program routes for pod traffic between nodes.
BGP is a standards-based routing protocol used to build the internet. It scales exceptionally well, and even the largest Kubernetes clusters represent a tiny amount of load compared to what BGP can cope with.
Calico can run BGP in three modes:
Full mesh - where each node talks BGP to each other, easily scaling to 100 nodes, on top of an underlying L2 network or using IPIP overlay
With route reflectors - where each node talks to one or more BGP route reflectors, scaling beyond 100 nodes, on top of an underlying L2 network or using IPIP overlay
Peered with TOR (Top of Rack) routers - in a physical data center where each node talks to routers in the top of the corresponding rack, scaling to the limits of your physical data center.
Datastore
etcd
etcd Datastore
Calico stores the operational and configuration state of your cluster in a central datastore. If the datastore is unavailable, your Calico network continues operating, but cannot be updated (no new pods can be networked, no policy changes can be applied, etc.).
Calico has two datastore drivers you can choose from:
etcd - for direct connection to an etcd cluster
Kubernetes - for connection to a Kubernetes API server
The advantages of using etcd as the datastore are:
Allows you to run Calico on non-Kubernetes platforms (e.g. OpenStack)
Allows separation of concerns between Kubernetes and Calico resources, for example allowing you to scale the datastores independently
Allows you to run a Calico cluster that contains more than just a single Kubernetes cluster, for example, bare metal servers with Calico host protection interworking with a Kubernetes cluster or multiple Kubernetes clusters.
For completeness, the advantages of using Kubernetes as the datastore are:
It doesn’t require an extra datastore, so is simpler to install and manage
You can use Kubernetes RBAC to control access to Calico resources
You can use Kubernetes audit logging to generate audit logs of changes to Calico resources.
?
Calico Deployment Options
Calico’s flexible modular architecture supports a wide range of deployment options, so you can select the best networking and network policy options for your specific environment. This includes the ability to run with a variety of CNI and IPAM plugins, and underlying networking options.
The Calico Getting Started guides default to the options most commonly used in each environment, so you don’t have to dive into the details unless you want to.
You can click on any deployment option to learn more.
If you are using the default BGP networking with full-mesh node-to-node peering with no encapsulation, go to Configure BGP peering to get traffic flowing between pods.
If you are unsure about networking options, or want to implement encapsulation (overlay networking), see Determine best networking option.