K3s multi-node install
Big picture
This tutorial gets you a multi node K3s cluster with Calico in approximately 10 minutes.
Value
K3s is a lightweight implementation of Kubernetes packaged as a single binary.
The geeky details of what you get:
Kubernetes network policies are implemented by network plugins rather than Kubernetes itself. Simply creating a network policy resource without a network plugin to implement it, will have no effect on network traffic.
The Calico plugin implements the full set of Kubernetes network policy features. In addition, Calico supports Calico network policies, providing additional features and capabilities beyond Kubernetes network policies. Kubernetes and Calico network policies work together seamlessly, so you can choose whichever is right for you, and mix and match as desired.
How Kubernetes assigns IP address to pods is determined by the IPAM (IP Address Management) plugin being used.
The Calico IPAM plugin dynamically allocates small blocks of IP addresses to nodes as required, to give efficient overall use of the available IP address space. In addition, Calico IPAM supports advanced features such as multiple IP pools, the ability to specify a specific IP address range that a namespace or pod should use, or even the specific IP address a pod should use.
The CNI (Container Network Interface) plugin being used by Kubernetes determines the details of exactly how pods are connected to the underlying network.
The Calico CNI plugin connects pods to the host networking using L3 routing, without the need for an L2 bridge. This is simple and easy to understand, and more efficient than other common alternatives such as kubenet or flannel.
An overlay network allows pods to communicate between nodes without the underlying network being aware of the pods or pod IP addresses.
Packets between pods on different nodes are encapsulated using VXLAN, wrapping each original packet in an outer packet that uses node IPs, and hiding the pod IPs of the inner packet. This can be done very efficiently by the Linux kernel, but it still represents a small overhead, which you might want to avoid if running particularly network intensive workloads.
For completeness, in contrast, operating without using an overlay provides the highest performance network. The packets that leave your pods are the packets that go on the wire.
Calico routing distributes and programs routes for pod traffic between nodes using its data store without the need for BGP. Calico routing supports unencapsulated traffic within a single subnet, as well as selective VXLAN encapsulation for clusters that span multiple subnets.
Calico stores the operational and configuration state of your cluster in a central datastore. If the datastore is unavailable, your Calico network continues operating, but cannot be updated (no new pods can be networked, no policy changes can be applied, etc.).
Calico has two datastore drivers you can choose from
- etcd - for direct connection to an etcd cluster
- Kubernetes - for connection to a Kubernetes API server
The advantages of using Kubernetes as the datastore are:
- It doesn’t require an extra datastore, so is simpler to install and manage
- You can use Kubernetes RBAC to control access to Calico resources
- You can use Kubernetes audit logging to generate audit logs of changes to Calico resources
For completeness, the advantages of using etcd as the datastore are:
- Allows you to run Calico on non-Kubernetes platforms (e.g. OpenStack)
- Allows separation of concerns between Kubernetes and Calico resources, for example allowing you to scale the datastores independently
- Allows you to run a Calico cluster that contains more than just a single Kubernetes cluster, for example, bare metal servers with Calico host protection interworking with a Kubernetes cluster or multiple Kubernetes clusters.
Calico’s flexible modular architecture supports a wide range of deployment options, so you can select the best networking and network policy options for your specific environment. This includes the ability to run with a variety of CNI and IPAM plugins, and underlying networking options.
The Calico Getting Started guides default to the options most commonly used in each environment, so you don’t have to dive into the details unless you want to.
You can click on any deployment option to learn more.
Before you begin
- Make sure you have a linux host that meets the following requirements
- x86-64 processor
- 1CPU
- 1GB Ram
- 10GB free disk space
- Ubuntu 16.04 (amd64), Ubuntu 18.04 (amd64), Ubuntu 20.04 (amd64)
Note: K3s supports ARM processors too, this tutorial was tested against x86-64 processor environment. For more detail please visit this link.
How to
Initializing master instance
K3s installation script can be modified by environment variables. Here you are providing some extra arguments in order to disable flannel
, disable k3s default network policy and change the pod ip CIDR.
Note: Full list of arguments can be viewed at this link.
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --cluster-cidr=192.168.0.0/16" sh -
Note: If 192.168.0.0/16 is already in use within your network you must select a different pod network CIDR by replacing 192.168.0.0/16 in the above command.
Enable remote access to your K3s instance
In order to setup remote access to your cluster first ensure you have installed kubectl
on your system.
Note: If you are not sure how to install kubectl in your OS visit this link.
K3s stores a kubeconfig file in your server at /etc/rancher/k3s/k3s.yaml
, copy all the content of k3s.yaml
from your server into ~/.kube/config
on the system that you like to have remote access to the cluster.
Add extra nodes to K3s cluster
In order to add additional nodes to your cluster you need two piece of information.
K3S_URL
which is going to be your main node ip address.K3S_TOKEN
which is stored in/var/lib/rancher/k3s/server/node-token
file in main Node (Step 1). Execute following command in your node instance and join it to the cluster.
Note: Remember to change
serverip
andmytoken
.
curl -sfL https://get.k3s.io | K3S_URL=https://serverip:6443 K3S_TOKEN=mytoken sh -
Install Calico
Install the Calico operator and custom resource definitions.
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml
Note: Due to the large size of the CRD bundle,
kubectl apply
might exceed request limits. Instead, usekubectl create
orkubectl replace
.
Install Calico by creating the necessary custom resource. For more information on configuration options available in this manifest, see the installation reference.
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml
Note: Before creating this manifest, read its contents and make sure its settings are correct for your environment. For example, you may need to change the default IP pool CIDR to match your pod network CIDR.
Install Calico by using the following command.
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml
Note: You can also view the YAML in a new tab.
You should see the following output.
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
Check the installation
- Confirm that all of the pods are running using the following command.
NAMESPACE NAME READY STATUS RESTARTS AGE
tigera-operator tigera-operator-c9cf5b94d-gj9qp 1/1 Running 0 107s
calico-system calico-typha-7dcd87597-npqsf 1/1 Running 0 88s
calico-system calico-node-rdwwz 1/1 Running 0 88s
kube-system local-path-provisioner-6d59f47c7-4q8l2 1/1 Running 0 2m14s
kube-system metrics-server-7566d596c8-xf66d 1/1 Running 0 2m14s
kube-system coredns-8655855d6-wfdbm 1/1 Running 0 2m14s
calico-system calico-kube-controllers-89df8c6f8-7hxc5 1/1 Running 0 87s
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-9hn9z 1/1 Running 0 23m
kube-system local-path-provisioner-6d59f47c7-drznc 1/1 Running 0 38m
kube-system calico-kube-controllers-789f6df884-928lt 1/1 Running 0 23m
kube-system metrics-server-7566d596c8-qxlfz 1/1 Running 0 38m
kube-system coredns-8655855d6-blzl5 1/1 Running 0 38m
-
Confirm that you now have two nodes in your cluster with the following command.
kubectl get nodes -o wide
It should return something like the following.
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k3s-master Ready master 40m v1.18.2+k3s1 172.16.2.128 <none> Ubuntu 18.04.3 LTS 4.15.0-101-generic containerd://1.3.3-k3s2 k3s-node1 Ready <none> 30m v1.18.2+k3s1 172.16.2.129 <none> Ubuntu 18.04.3 LTS 4.15.0-101-generic containerd://1.3.3-k3s2
Congratulations! You now have a multi node K3s cluster equipped with Calico and Traefik.
Next steps
- Try running the Kubernetes Network policy demo to see live graphical view of network policy in action