How to deploy kubernetes cluster in your local laptop/desktop?
Deploying kubernetes using minikube is all good. But it doesn’t give you true exposure to the kubernetes cluster world. Its a different system altogether with multiples worker nodes coming into play. How do pods gets scheduled when we have more than one worker nodes? Is there an algorithm which kubernetes apply while selecting nodes? If yes, then can we override to fit our needs? What happens if node goes not with pods running and you have stateful pod running on that node? And what happens when that node comes back up.
You may have found multiple websites that will walk you through the process of k8s cluster setup in your local. Mine may not be any different..or may be. This is just an another attempt of just like any other website to try to make this process smooth and easy to install on your machine.
This is how our end architecture going to look like.
Step 1: Install VirtualBox on your laptop(currently not supported on Mac M1 chip).
I have installed this on both mac and windows and there is really not much difference and it took equal effort and time to do it. Download virtualbox from here -> VirtualBox. Also, I downloaded the vm images from https://www.linuxvmimages.com/ .
Step 2: Download VM image to install on virtualbox
Download linux images from here -> https://www.linuxvmimages.com
I chose minimal installation which is this -> centos-minimal
Minimal is all we need for this setup. You can try other installation and explore, but instructions I am going to have below are tried and tested on centos.
Step 3: Setting up the Virtual Machine before we install the centos with the vm image we downloaded in step 2 above.
Open VirtualBox that we installed and click on ‘New’ to create a new VM.
Once the new window that appears, give a name to the VM. I prefer any name with keyword ‘master’ for easy distinguishing between master and worker nodes we are going to create. Below is what it looked like on my end.
Next select Memory size as shown below. Select 2048MB as minimum to avoid any kind of performance issue. And done wor
Next: Create a new Virtual Machine
Select all the default options and that that will end the setup screen for the vm creation.
Once the VM is created, do not start the machine yet. We have to make few more changes to this VM.
Step 4: Update CPU, Network settings.
Once the VM is created, select this VM on the VirtualBox window and click Settings that will configurations for this VM.
Go to System tab and switch to Processor settings. Update the CPU count to 2 if that is not what it is set to.
Next, go to Network tab while on the same settings window of that VM.
Make sure the Enable Network Adapter option is checked. And then from the dropdown of Attached to, select Bridged Adapter. This will connect your VM to the network to which your host machine is connected. That way, your master and worker nodes will be on same network and can talk to each other. No need to enable any more network adapters. Click OK.
This is how it looks on my machine.
Step 5: Install Centos on the Virtual Machine.
Select the virtual machine on the VirtualBox again and this time click start.
You will now be prompted to select a startup disk to book from. Here point to the cento-os image you downloaded in Step 2.
This will begin the installation of the OS on your machine.
Next, select language of your choice for OS installation.
Next, select the path where the installation needs to happen. Select default settings if you are not sure what to select.
Important step before you click Begin Installation.
Update Network & Host Name settings and make sure you enable to Ethernet option.
This is how it looks on my machine.
Let the installation begin!!
While installation is in progress, make sure you select root password and also create additional user. Setting the root password will help you ssh to the master node from your laptop and execute the commands that I am to list below. You can just copy past those commands on laptop once you ssh.
Step 6: Pre-setup for installing docker and kubernetes.
Next following steps can be executed directly on your this brand new VM. OR, from your laptop using ssh into in this VM. I prefer the ssh method so that I can copy paste the commands on my laptop terminal easily to speedup the process. For this you need to login to your VM once to know the ip address so that you can ssh.
Execute following command into your VM once logged in.
[root@centos-master centos]# ip addr show
This will print the ip address. For me it showed under enp03 inet as below.
Now we are going to ssh into this master node with root user from your laptop. Hope you remember the root password that you set during the installation process above.
Go to your laptop terminal(terminal, iterm, powershell, cmder etc.)
Enter below command. That will prompt you to enter password. Once you enter password, that’s it.
ssh root@192.168.4.142
You are now into master node.
Now on, we are going to execute the commands on your master node via ssh from your laptop,
In order to begin the installation and ensure connectivity to the k8s api server, we need to disable security and firewall on your VM.
Note: Do not do this on your real life environment which you are going to use as dev/test/prod .
Disable the SELinux first. You can do this by updating /etc/selinux/config file to have SELINUX=permissive. This will permanently disable the SELinux.
You can also just execute the below command on your terminal. But this will re-enable the SELinux after reboot. Hence if you need this as a permanent change, follow above step of modifying the /etc/selinux/config file.
[root@centos ~]# setenforce 0
Disabling firewall.
Run below commands.(Reminder, hope your logged into as root or sudo to root for all the commands we have executed so far and also for the next coming commands.)
[root@centos ~]# systemctl disable firewalld && systemctl stop firewalld
Below will be the output after executing the command.
To install a software on unix based we have multiple options. Download the installation file and trigger the installation. OR, setup the yum repo on machine and begin installation. That’s what we are going to do here now. Execute below command from your terminal. This will create /etc/yum.repos.d/kubernetes.repo file with the contents starting from [kubernetes] till gpgkey.
[root@centos-master ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
Step 7: Installing docker and kubernetes.
Installing docker and kubernetes along with other required apps kubeadm, kubectl, etc.)
Execute below command for the installation.
[root@centos-master ~]# yum install -y docker kubelet kubeadm kubectl kubernetes-cni
This will take some time since we are installing multiple apps in one go.
Now we need to enable the docker and kubelet services.
[root@centos-master ~]# systemctl enable docker && systemctl start docker
[root@centos-master ~]# systemctl enable kubelet && systemctl start kubeletI have no idea why we have to disable the below. But was facing issue connecting toe kube api server. After googling, found the below solution.
[root@centos-master ~]# sysctl -w net.bridge.bridge-nf-call-iptables=1
[root@centos-master ~]# echo "net.bridge.bridge-nf-call-iptables=1" > /etc/sysctl.d/k8s.confAgain, kublet was not running on the worker nodes without disabling swap. Hence the below command.
[root@centos-master ~]# swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab
You are almost there. All we have now left with setting up worker nodes and make them join the cluster. Don’t worry, we are not going to repeat all the above steps for worker nodes. We are just going to clone the master-node VM now to create copies of it and make the copied VM’s(with some additional config changes) as worker nodes. In order to clone you need to shutdown this vm.
Btw, just hit execute kubectl and docker commands on your teminal and now you see some output. Which means kubectl was installed successfully.
[root@centos-master ~]# kubectl
kubectl controls the Kubernetes cluster manager.
Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/
Basic Commands (Beginner):
create Create a resource from a file or from stdin
expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes service
run Run a particular image on the cluster
set Set specific features on objects
Basic Commands (Intermediate):
explain Get documentation for a resource
get Display one or many resources[root@centos-master ~]# docker
Usage: docker COMMAND
A self-sufficient runtime for containers
Options:
--config string Location of client config files (default "/root/.docker")
-D, --debug Enable debug mode
--help Print usage
-H, --host list Daemon socket(s) to connect to (default [])
-l, --log-level string Set the logging level ("debug", "info", "warn", "error", "fatal") (default "info")
--tls Use TLS; implied by --tlsverify
--tlscacert string Trust certs signed only by this CA (default "/root/.docker/ca.pem")
--tlscert string Path to TLS certificate file (default "/root/.docker/cert.pem")
--tlskey string Path to TLS key file (default "/root/.docker/key.pem")
--tlsverify Use TLS and verify the remoteStep 8: Create worker nodes
For this step, we are not going to go through the laborious of all the security disabling, setup repo for kubernetes and docker installation etc. Instead we are going to use the clone feature of the VirtualBox and create these worker nodes.
Create total three clones from the master. So now, you should have one master and three worker nodes.
Once the cloning is complete, verify the cpu and network settings are same as the master node. But mac address should be different for all of them.
Step 9: Configuring worker nodes
We need to now setup the hostname for all the cloned worker nodes. We did that for the master node when we enabled the Ethernet adapter on network settings screen during installation of centos.
Login to each of the nodes and execute below commands on the respective worker nodes. Also, while you are logged into the worker nodes, make a note of the ip address of each node since we are going to use it later. Also, do not shutdown the worker nodes after retrieving the ip address. Because we are going to update dns entries on each one of them once we get to know ip address of all of the worker nodes.
You should know by now how to find ip address of a node. If not, scroll up.
Setting centos-worker-1 host
hostnamectl --static set-hostname centos-worker-node1
Setting hostname for centos-worker-2 host
hostnamectl --static set-hostname centos-worker-node2
Setting hostname for centos-worker-3 host
hostnamectl --static set-hostname centos-worker-node3
Also, we need to update the dns settings on each of the nodes including master to that they can reach other using their dns name. For this, we need to need the ip address of each node you noted above.
Now, we need to update the /etc/hosts file of each node including the master node with the ipaddress and hostname of each node. Mine look like below.
192.168.4.142 centos-master.k8s
192.168.4.143 centos-worker-1.k8s
192.168.4.144 centos-worker-2.k8s
192.168.4.145 centos-worker-3.k8sStep 10: Initializing master node with kubeadm
Now we need to initialize the kubernetes master node with the init command as shown below. Make sure you are logged in as root. We will need to copy the last line from the output of the below command.
[root@centos-master ~]#kubeadm initBelow are few last lines from the output for me.
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.4.142:6443 --token nmppyk.m1k85d82evd05788 \
--discovery-token-ca-cert-hash sha256:d07d71529946ffdf991851393383b2ce8239cdf4a337d9cd21450f8aaad4b8cdWe will the last line from the output (obviously it will be different for you). Copy it somewhere.
So far, we have deployed some of the key component needed for kubernetes to run.
But for kubectl is not yet ready to connect to the cluster.
When we run the kubectl related commands, it looks for the config file at ~/.kube/config by default. If the file does not exist at this location, then it looks for the env variable named KUBECONFIG
In our case, we do not have the ~/.kube/config file on our master node, but it exists at /etc/kubernetes/admin.conf
We now have two options now for kubectl commands to run successfully. Either move the admin.config file to ~/.kube folder. OR, create env variable KUBECONFIG and make it point to /etc/kubernetes/admin.conf
I will go with the second option for now. So, lets execute the below command on master node.
[root@centos-master ~]# export KUBECONFIG=/etc/kubernetes/admin.conf
Now, lets execute our first kubectl command to get nodes and pods.
[root@centos-master ~]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-64897985d-2dsnp 0/1 Pending 0 43m
coredns-64897985d-l8jx2 0/1 Pending 0 43m
etcd-centos-master.k8s 1/1 Running 0 43m
kube-apiserver-centos-master.k8s 1/1 Running 0 43m
kube-controller-manager-centos-master.k8s 1/1 Running 0 43m
kube-proxy-zhxtl 1/1 Running 0 43m
kube-scheduler-centos-master.k8s 1/1 Running 0 43m
[root@centos-master ~]#[root@centos-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
centos-master.k8s NotReady control-plane,master 45m v1.23.4Don’t worry if the node says NotReady now. Its because we have not setup the networking for this cluster yet. We will be doing it in next few steps.
Step 11: Let’s make the worker nodes part the cluster.
Remember the last line from the kubeadm init command which I mentioned to make a note of it. We will be using this command now to execute on all the worker nodes. Let’s do that now. I won’t paste the output from each worker nodes.
[root@centos-worker-node1 ~]# kubeadm join 192.168.4.142:6443 --token nmppyk.m1k85d82evd05788 \
> --discovery-token-ca-cert-hash sha256:d07d71529946ffdf991851393383b2ce8239cdf4a337d9cd21450f8aaad4b8cd
[preflight] Running pre-flight checks
[WARNING Hostname]: hostname "centos-worker-node1" could not be reached
[WARNING Hostname]: hostname "centos-worker-node1": lookup centos-worker-node1 on 75.75.75.75:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
Now, lets go back to the master node and try kubectl get nodes and check the output once the worker-node-1 joins.
[root@centos-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
centos-master.k8s NotReady control-plane,master 53m v1.23.4
centos-worker-node1 NotReady <none> 66s v1.23.4Note: Just in case if you get error regarding connecting to the host as below, then you need to try the export KUBECONFIG….. command again.
[root@centos-master ~]# kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?I will now execute the kubeadm join commands on all the nodes.
Now lets go back to master node to check connected notes to the cluster. But also, lets also create config file in .kube folder and copy the contents from /etc/kubernetes/admin.conf.
[root@centos-master ~]# cp /etc/kubernetes/admin.conf ~/.kube/config
[root@centos-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
centos-master.k8s NotReady control-plane,master 68m v1.23.4
centos-worker-node1 NotReady <none> 15m v1.23.4
centos-worker-node2 NotReady <none> 7m5s v1.23.4
centos-worker-node3 NotReady <none> 101s v1.23.4Now lets setup the networking in the cluster so that we can get rid of the NotReady status of the nodes.
If you want to verify that the status of the node is NotReady due to networking, lets try below command.
[root@centos-master ~]# kubectl describe node centos-worker-node1
Let me paste the error screenshot from the above command.
Let’s fix that.
Kubeadm is a plan vanilla way of installing kubernetes. Its not a distribution provided by openshift etc. What that means is we need to install few components/plugins that doesn’t come along with by default. Networking solution is one of them. We are going to install weave net plugin.
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"Now, check the nodes status. It will take few seconds to get the status updated.
[root@centos-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
centos-master.k8s Ready control-plane,master 106m v1.23.4
centos-worker-node1 Ready <none> 54m v1.23.4
centos-worker-node2 Ready <none> 45m v1.23.4
centos-worker-node3 Ready <none> 39m v1.23.4That is it..you have your kubernetes cluster with multiple nods up and running in your local!
One last optional things. If you do not want to manage your cluster from your loptop without ssh into master, copy the ~./kube/config file from master to your local and that’s it.
Let me know under the comments what you think of this documentation.