Self-managed Kubernetes in DigitalOcean (DO)
Big picture
This tutorial creates a self-managed Kubernetes cluster (1 Master, 2 Worker nodes) using Calico networking in DigitalOcean.
Value
Managing your own Kubernetes cluster (as opposed to using a managed-Kubernetes service like EKS), gives you the most flexibility in configuring Calico and Kubernetes. Calico combines flexible networking capabilities with “run-anywhere” security enforcement to provide a solution with native Linux kernel performance and true cloud-native scalability.
Concepts
Kubernetes Operations (kops) is a cluster management tool that handles provisioning cluster VMs and installing Kubernetes. It has built-in support for using Calico as the Kubernetes networking provider.
Note: Kops support for DigitalOcean is currently in the early stages of development and subject to change.
More information can be viewed at this link.
Before you begin…
How to
There are many ways to install and manage Kubernetes in DO. Using Kubernetes Operations (kops) is a good default choice for most people, as it gives you access to all of Calico’s flexible and powerful networking features. However, there are other options that may work better for your environment.
The geeky details of what you get:
Kubernetes network policies are implemented by network plugins rather than Kubernetes itself. Simply creating a network policy resource without a network plugin to implement it, will have no effect on network traffic.
The Calico plugin implements the full set of Kubernetes network policy features. In addition, Calico supports Calico network policies, providing additional features and capabilities beyond Kubernetes network policies. Kubernetes and Calico network policies work together seamlessly, so you can choose whichever is right for you, and mix and match as desired.
How Kubernetes assigns IP address to pods is determined by the IPAM (IP Address Management) plugin being used.
The Calico IPAM plugin dynamically allocates small blocks of IP addresses to nodes as required, to give efficient overall use of the available IP address space. In addition, Calico IPAM supports advanced features such as multiple IP pools, the ability to specify a specific IP address range that a namespace or pod should use, or even the specific IP address a pod should use.
The CNI (Container Network Interface) plugin being used by Kubernetes determines the details of exactly how pods are connected to the underlying network.
The Calico CNI plugin connects pods to the host networking using L3 routing, without the need for an L2 bridge. This is simple and easy to understand, and more efficient than other common alternatives such as kubenet or flannel.
An overlay network allows pods to communicate between nodes without the underlying network being aware of the pods or pod IP addresses.
Packets between pods on different nodes are encapsulated using IPIP, wrapping each original packet in an outer packet that uses node IPs, and hiding the pod IPs of the inner packet. This can be done very efficiently by the Linux kernel, but it still represents a small overhead, which you might want to avoid if running particularly network intensive workloads.
For completeness, in contrast, operating without using an overlay provides the highest performance network. The packets that leave your pods are the packets that go on the wire.
BGP (Border Gateway Protocol) is used to dynamically program routes for pod traffic between nodes.
BGP is a standards-based routing protocol used to build the internet. It scales exceptionally well, and even the largest Kubernetes clusters represent a tiny amount of load compared to what BGP can cope with.
Calico can run BGP in three modes:
- Full mesh - where each node talks BGP to each other, easily scaling to 100 nodes, on top of an underlying L2 network or using IPIP overlay
- With route reflectors - where each node talks to one or more BGP route reflectors, scaling beyond 100 nodes, on top of an underlying L2 network or using IPIP overlay
- Peered with TOR (Top of Rack) routers - in a physical data center where each node talks to routers in the top of the corresponding rack, scaling to the limits of your physical data center.
Calico stores the operational and configuration state of your cluster in a central datastore. If the datastore is unavailable, your Calico network continues operating, but cannot be updated (no new pods can be networked, no policy changes can be applied, etc.).
Calico has two datastore drivers you can choose from
- etcd - for direct connection to an etcd cluster
- Kubernetes - for connection to a Kubernetes API server
The advantages of using Kubernetes as the datastore are:
- It doesn’t require an extra datastore, so is simpler to install and manage
- You can use Kubernetes RBAC to control access to Calico resources
- You can use Kubernetes audit logging to generate audit logs of changes to Calico resources
For completeness, the advantages of using etcd as the datastore are:
- Allows you to run Calico on non-Kubernetes platforms (e.g. OpenStack)
- Allows separation of concerns between Kubernetes and Calico resources, for example allowing you to scale the datastores independently
- Allows you to run a Calico cluster that contains more than just a single Kubernetes cluster, for example, bare metal servers with Calico host protection interworking with a Kubernetes cluster or multiple Kubernetes clusters.
Calico’s flexible modular architecture supports a wide range of deployment options, so you can select the best networking and network policy options for your specific environment. This includes the ability to run with a variety of CNI and IPAM plugins, and underlying networking options.
The Calico Getting Started guides default to the options most commonly used in each environment, so you don’t have to dive into the details unless you want to.
You can click on any deployment option to learn more.
Generate your DigitalOcean API token
An API token is needed by kops for the CRUD (Create, Read, Update and Delete) operations necessary for resources in your DigitalOcean account. Use this link to generate your API token and then export it as an environment variable.
export DIGITALOCEAN_ACCESS_TOKEN=<API_ACCESS_TOKEN>
Create an object storage
DigitalOcean provides an S3 compatible storage API that Kops uses object storage to save your cluster status. You should create a Space using this link and export it.
export KOPS_STATE_STORE=do://<your-space-name>
export S3_ENDPOINT=<ENDPOINT>
Note: Using FQDN for
S3_ENDPOINTcauses an error. If your Space FQDN ismy-test-space.nyc3.digitaloceanspaces.comjust exportmy-test-spaceas<your-space-name>.
Generate an API key for object storage
Access to object storage requires an API key. Follow this tutorial and generate your keys then export them as environment variables.
export S3_ACCESS_KEY_ID=<ACCESS_KEY_ID>
export S3_SECRET_ACCESS_KEY=<SECRET_ACCESS_KEY>
Enable kops alpha feature
Enable alpha feature support using KOPS_FEATURE_FLAGS environment variable.
export KOPS_FEATURE_FLAGS="AlphaAllowDO"
Create your cluster
Kops supports various options that enables you to customize your cluster the way you like.
- Add Calico to your cluster using
--networking=calico. - Kops requires an external DNS server in order to create a cluster, by adding
.k8s.localsuffix to--name=option you generate a gossip DNS to bypass this requirement.
You can view a complete list of options supported by kops in this link.
kops create cluster --cloud=digitalocean --name=calico-demo.k8s.local \
--networking=calico --master-zones=nyc1 --zones=nyc1 \
--master-count=1 --api-loadbalancer-type=public \
--node-size=s-1vcpu-2gb --image=ubuntu-20-04-x64 --yes
You can further customize the Calico install with options listed in the kops documentation.
Cleanup
If you wish to remove resources created by this tutorial
kops delete cluster calico-demo.k8s.local --yes
Use the DigitalOcean web UI to remove the API tokens and Space you created.
Next steps
Required
Recommended