Bài đăng phổ biến

Showing posts with label calico CNI. Show all posts
Showing posts with label calico CNI. Show all posts

Thursday, September 8, 2022

Setup cluster kubernetes (k8s) by kubeadm with calico on Production enviroment .

References :

  1. Creating a cluster with kubeadm

  2. GitHub - mmumshad/kubernetes-the-hard-way: Bootstrap Kubernetes the hard way on Vagrant on Local Machine. No scripts.

  3. Install Kubernetes from Scratch [2] - Provisioning infrastructure for Kubernetes

  4. Install Calico networking and network policy for on-premises deployments

  5. Release v3.18.4 · projectcalico/calicoctl ( calicoctl )

  6. Overlay networking


I. Prerequisite : Run in all worker node .

  1. Setup docker( or any other container runtime ) container on all node

Install Docker Engine on CentOS )

2. Letting iptables see bridged traffic on all node:

1cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf 2br_netfilter 3EOF 4 5cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf 6net.bridge.bridge-nf-call-ip6tables = 1 7net.bridge.bridge-nf-call-iptables = 1 8net.ipv4.ip_forward = 1 9EOF 10sudo sysctl --system

3. Change cgroup of docker to systemd :

1sudo mkdir /etc/docker 2cat <<EOF | sudo tee /etc/docker/daemon.json 3{ 4 "exec-opts": ["native.cgroupdriver=systemd"], 5 "log-driver": "json-file", 6 "log-opts": { 7 "max-size": "100m" 8 }, 9 "storage-driver": "overlay2" 10} 11EOF

then restart docker

4. Setup kubelet,kubeadm,kubectl

1cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo 2[kubernetes] 3name=Kubernetes 4baseurl=\$basearch 5enabled=1 6gpgcheck=1 7repo_gpgcheck=1 8gpgkey= 9exclude=kubelet kubeadm kubectl 10EOF 11 12# Set SELinux in permissive mode (effectively disabling it) 13sudo setenforce 0 14sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config 15yum install -y kubelet-1.21.0 kubeadm-1.21.0 kubectl-1.21.0 --disableexcludes=kubernetes 16systemctl enable kubelet.service

5. Create file with content like that :

# vi /etc/NetworkManager/conf.d/99-unmanaged-devices.conf

1[keyfile] 2unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico

run below command to apply :

1systemctl restart NetworkManager

II. Demo installation environment:

  • 1 Load balancer : ( (vip: )

  • 3 server master ( ,,

  • 1 server worker node ( )

III. Setup load balancer

  • Setup keepalived ( another guide )

  • Setup haproxy ( another guide )

main content of file : haproxy.cfg


1frontend fe-apiserver 2 bind 3 mode tcp 4 option tcplog 5 default_backend be-apiserver 6backend be-apiserver 7 mode tcp 8 option tcplog 9 option tcp-check 10 balance roundrobin 11 12 server master-1 check fall 3 rise 2 13 server master-2 check fall 3 rise 2 14 server master-3 check fall 3 rise 2 15 16 17


IV. Setup masternode :

Have a lot of to setup master node If you guys want to learn to install each component this is follow link to do :

GitHub - mmumshad/kubernetes-the-hard-way: Bootstrap Kubernetes the hard way on Vagrant on Local Machine. No scripts.

In this tutorial i will describe how to install using kubeadm .

ssh into server master01 : run it like this :

kubeadm init --control-plane-endpoint "$LOADBALANCER:6444" --upload-certs --pod-network-cidr --service-cidr

--pod-network-cidr : network for pod --service-cidr: if you want coredns and kubeapi have the same subnet ,you should use the same subnet pod-network-cidr

waiting for kubeadm finish it will return like below:

1To start using your cluster, you need to run the following as a regular user: 2 3 mkdir -p $HOME/.kube 4 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 5 sudo chown $(id -u):$(id -g) $HOME/.kube/config 6 7Alternatively, if you are the root user, you can run: 8 9 export KUBECONFIG=/etc/kubernetes/admin.conf 10 11You should now deploy a pod network to the cluster. 12Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 13 14 15You can now join any number of the control-plane node running the following command on each as root: 16 17 kubeadm join --token 2rgzgq.hzsc9n4tgmyohca4 \ 18 --discovery-token-ca-cert-hash sha256:b37088b13028cb53b05b2e6ac79eaaccd5c53d1030df2df873b96d1150395823 \ 19 --control-plane --certificate-key cf9f8a63ab1d642dc6cdd7e74c12456f0f3e1498bf9a47e0d4c5aeb9a66cfb2a 20 21Please note that the certificate-key gives access to cluster sensitive data, keep it secret! 22As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use 23"kubeadm init phase upload-certs --upload-certs" to reload certs afterward. 24 25Then you can join any number of worker nodes by running the following on each as root: 26 27kubeadm join --token 2rgzgq.hzsc9n4tgmyohca4 \ 28 --discovery-token-ca-cert-hash sha256:b37088b13028cb53b05b2e6ac79eaaccd5c53d1030df2df873b96d1150395823

Then :


1 mkdir -p $HOME/.kube 2 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 3 sudo chown $(id -u):$(id -g) $HOME/.kube/config

run this command to get status of worker node :

1 kubectl get node

to view node status , note: status of master node is still NotReady

Login into master02 và master03 : then run :

1kubeadm join --token 2rgzgq.hzsc9n4tgmyohca4 \ 2 --discovery-token-ca-cert-hash sha256:b37088b13028cb53b05b2e6ac79eaaccd5c53d1030df2df873b96d1150395823 \ 3 --control-plane --certificate-key cf9f8a63ab1d642dc6cdd7e74c12456f0f3e1498bf9a47e0d4c5aeb9a66cfb2a

V. Join worker node

Login into each worker node then run:

1kubeadm join --token 2rgzgq.hzsc9n4tgmyohca4 \ 2 --discovery-token-ca-cert-hash sha256:b37088b13028cb53b05b2e6ac79eaaccd5c53d1030df2df873b96d1150395823



1kubectl get node

status of cluster is still NotReady because we have not yet install network for cluster .

VI. Setup network cho Kubernetes :

have a lot of 3rd support CNI but in this tutorial I choose calico :

login into a master node then run these below command:

1curl -O 2kubectl apply -f calico.yaml 3# kiểm tra xem đã setup xong chưa bằng cmd: 4kubectl -n kube-system get pod -w|grep calico

After done network , Status of cluster should be Ready. :grinning: :

kubectl get node



Calico default is enable ipip encapsulation , I switch to vxlan encapsulation because i think it's stable than ipip encapsulation .

  1. Download calicoctl from : Release v3.18.4 · projectcalico/calicoctl

  2. create a file : xyz.yaml with content like below :

apiVersion: kind: IPPool metadata: name: default-ipv4-ippool spec: cidr: vxlanMode: CrossSubnet natOutgoing: true

  1. run this command to apply: calicoctl apply -f xzy.yaml

  2. check : calico get ippool -o wide