Self-managed Kubernetes in Microsoft Azure
Big picture
Use Calico with a self-managed Kubernetes cluster in Microsoft Azure.
Value
Managing your own Kubernetes cluster (as opposed to using a managed-Kubernetes service like AKS), gives you the most flexibility in configuring Calico and Kubernetes. Calico combines flexible networking capabilities with “run-anywhere” security enforcement to provide a solution with native Linux kernel performance and true cloud-native scalability.
Concepts
aks-engine is an open-source tool for creating and managing Kubernetes clusters in Microsoft Azure. It is the core technology for Microsoft’s Azure Kubernetes Service (AKS), but allows you to manage the cluster yourself.
Before you begin…
- Install kubectl
- Install Azure CLI tools
How to
There are many ways to install and manage Kubernetes in Azure. This guide shows how to use aks-engine to deploy a cluster with Azure’s CNI plugin for networking and Calico for network policy enforcement. The advantage of this approach is that pods are assigned IP addresses associated with Azure Network Interfaces on worker nodes. The IPs come from the VNET network pool and therefore do not require NAT to access resources outside the Kubernetes cluster. However, there are other options that may work better for your environment.
aks-engine for Azure networking and Calico network policy
Install aks-engine on your workstation.
Before deploying, customize your cluster definition to use Calico for network policy. Add or modify the kubernetesConfig section to include the following (see the aks-engine documentation for other Kubernetes configuration settings).
"kubernetesConfig": {
"networkPlugin": "azure",
"networkPolicy": "calico"
}
Or, start with this example cluster definition with these value already set, and customize to meet your needs.
Then, follow the ask-engine documentation to deploy your cluster, passing your cluster definition to aks-engine deploy via the -m flag.
The geeky details of what you get:
Kubernetes network policies are implemented by network plugins rather than Kubernetes itself. Simply creating a network policy resource without a network plugin to implement it, will have no effect on network traffic.
The Calico plugin implements the full set of Kubernetes network policy features. In addition, Calico supports Calico network policies, providing additional features and capabilities beyond Kubernetes network policies. Kubernetes and Calico network policies work together seamlessly, so you can choose whichever is right for you, and mix and match as desired.
How Kubernetes assigns IP address to pods is determined by the IPAM (IP Address Management) plugin being used.
The Azure IPAM plugin dynamically allocates small blocks of IP addresses to nodes as required, using IP addresses from the underlying VNET (Virtual Network). The Azure IPAM plugin is used in conjunction with the Azure CNI plugin to provide VPC native pod networking.
The CNI (Container Network Interface) plugin being used by Kubernetes determines the details of exactly how pods are connected to the underlying network.
The Azure CNI and IPAM plugins provide pods with IP addresses from the underlying Azure VNET (Virtual Network) to provide a VPC-Native pod network. The Azure VNET is used to route pod traffic between nodes, and understands which pod IP address are located on which nodes. This avoids the need for an overlay, and typically has good network performance characteristics.
In addition, pod IPs are understood by the broader AWS network, so for example, VMs outside of the cluster can connect directly to any pod without going via a Kubernetes service if desired.
Operating without using an overlay provides the highest performance network. The packets that leave your pods are the packets that go on the wire.
For completeness, in contrast, with an overlay network, packets between pods on different nodes are encapsulated using a protocol such as VXLAN or IPIP, wrapping each original packet in an outer packet that uses node IPs, and hiding the pod IPs of the inner packet. This can be done very efficiently by the Linux kernel, but it still represents a small overhead, which you might want to avoid if running particularly network intensive workloads.
The underlying cloud VPC (Virtual Private Cloud) is used to route pod traffic between nodes, and understands which pod IP address are located on which nodes. This avoids the need for an overlay, and typically has good performance characteristics.
In addition, pod IPs are understood by the broader cloud network, so for example, VMs outside of the cluster can connect directly to any pod without going via a Kubernetes service if desired.
Calico stores the operational and configuration state of your cluster in a central datastore. If the datastore is unavailable, your Calico network continues operating, but cannot be updated (no new pods can be networked, no policy changes can be applied, etc.).
Calico has two datastore drivers you can choose from
- etcd - for direct connection to an etcd cluster
- Kubernetes - for connection to a Kubernetes API server
The advantages of using Kubernetes as the datastore are:
- It doesn’t require an extra datastore, so is simpler to install and manage
- You can use Kubernetes RBAC to control access to Calico resources
- You can use Kubernetes audit logging to generate audit logs of changes to Calico resources
For completeness, the advantages of using etcd as the datastore are:
- Allows you to run Calico on non-Kubernetes platforms (e.g. OpenStack)
- Allows separation of concerns between Kubernetes and Calico resources, for example allowing you to scale the datastores independently
- Allows you to run a Calico cluster that contains more than just a single Kubernetes cluster, for example, bare metal servers with Calico host protection interworking with a Kubernetes cluster or multiple Kubernetes clusters.
Calico’s flexible modular architecture supports a wide range of deployment options, so you can select the best networking and network policy options for your specific environment. This includes the ability to run with a variety of CNI and IPAM plugins, and underlying networking options.
The Calico Getting Started guides default to the options most commonly used in each environment, so you don’t have to dive into the details unless you want to.
You can click on any deployment option to learn more.
Other options and tools
Calico networking
You can also deploy Calico for both networking and policy enforcement. In this mode, Calico uses a VXLAN-based overlay network that masks the IP addresses of the pods from the underlying Azure VNET. This can be useful in large deployments or when running multiple clusters and IP address space is a big concern.
Unfortunately, aks-engine does not support this mode, so you must use a different tool chain to install and manage the cluster. Some options:
- Use Terraform to provision the Azure networks and VMs, then kubeadm to install the Kubernetes cluster.
- Use Kubespray
Terraform
Terraform is a tool for automating infrastructure provisioning using declarative configurations. You can also go as far as automating the install of Docker, kubeadm, and Kubernetes using Terraform “provisioners.” See the Terraform documentation for more details.
kubeadm
kubeadm is a command line tool for bootstrapping a Kubernetes cluster on top of already-provisioned compute resources, like VMs in a cloud or bare metal hosts. Unlike aks-engine which handles provisioning cloud resources, installing Kubernetes, and installing Calico, kubeadm only handles the second step of installing Kubernetes. You should proceed to install Calico after completing kubeadm install.
Kubespray
Kubespray is a tool for provisioning and managing Kubernetes clusters with support for multiple clouds including Azure. Calico is the default networking provider, or you can set the kube_network_plugin variable to calico. See the Kubespray docs for more details.
Next steps
Required
Recommmended