Kubernetes can be used to manage container based cluster deployment, so hundred of thousands of containers can communicate, discover, and function as expected under the control of kubernetes. Although it sounds complex, but fundamentally it is just a software application managing the running of your application in containers.
1. Installation with kubeadm
There are three main tools for kerbenetes installation and management, they need to be installed on both master and worker nodes with below command
apt-get install -y kubeadm=1.18.1-00 kubelet=1.18.1-00 kubectl=1.18.1-00
1.1 kubeadm: for installation of master node
A regular linux package running on your virtual machines or local boxes's terminal console to install the kubernetes software, it sets up master node's kubernetes cluster manager, and also allow other VM to join the kubernetes as worker node. Before installing kubeadm, docker needs to be installed first on the VM, which is the basic running unit in the kubernetes cluster
//1. update apt-get
sudo -i
apt-get update
apt-get upgrade -y
//install docker
apt-get install -y docker.io
//2. prepare kubernetes repo by adding a new line in the below file
//install vim if it is not installed on the current node
apt-get install -y vim
vim /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
apt-get update
//3 install kubeadm, kubelet, kubectl
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
//4 install network add on
wget https://docs.projectcalico.org/manifests/calico.yaml
//uncomment below lines in the file to used for kubeadm init pod-network-cidr parameter
- name: CALICO_IPV4POOL_CIDR
value: "192.168.0.0/24"
//get ens4 ip address, for example, 10.0.0.6
ip addr show
//add the host ip to host table for k8smaster
root@master: ̃# vim /etc/hosts
10.0.0.6 k8smaster
//create kubeadm-config.yaml file as below at /root folder
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.20.0
controlPlaneEndpoint: "k8smaster:6443"
networking:
podSubnet: 192.168.0.0/24
// run kubeadm init to start cluster, the output has information for how to add other nodes
root@master:~# kubeadm init --config=kubeadm-config.yaml --upload-certs
root@master:~# exit
//allow no root user to manage cluster
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
//apply network addon calico
sudo cp /root/calico.yaml .
kubectl apply -f calico.yaml
//verify master node, and major kube services (calico, coredns, etcd, apiserver, kubeproxy, scheduler) are ready
kubectl get nodes
kubectl -n kube-system get pods
1.2 kubeadm: for installation of worker node
//1. update apt-get and install docker
sudo -i
apt-get update
apt-get upgrade -y
//install docker
apt-get install -y docker.io
//2. prepare kubernetes repo by adding a new line in the below file
//install vim if it is not installed on the current node
apt-get install -y vim
vim /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
apt-get update
//3 install kubeadm, kubelet, kubectl
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
//4 run the below commands on master node to get master ip, for example 10.0.0.6
ip addr show ens4
//use below command to get token from master
sudo kubeadm token list
//if the token is expired, use the below command to generate new one
sudo kubeadm token create
//run the command to get cert hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/ˆ.* //'
//output is (stdin)= fb5f18060d9c9363763da1e0ddf164dd1b41e640204d0e5ab01c366c3f5b944b
//5 go back to worker node, to add master node ip as k8smaster to host table
root@worker: ̃# vim /etc/hosts
//add below line
10.0.0.6 k8smaster
//6 add worker node to cluster
kubeadm join k8smaster:6443 --token xwqbt0.ig282rtsalg84ndw --discovery-token-ca-cert-hash sha256:fb5f18060d9c9363763da1e0ddf164dd1b41e640204d0e5ab01c366c3f5b944b
root@worker: ̃# exit
//6 run below command on master node to confirm the worker node is joined
student@master:~$ kubectl get nodes
1.3 cluster upgrade
1.3.1 First upgrade kubeadm package on master node
kubeadm upgrade plan
//get upgrade information
apt-get install kubeadm=1.19.0
//download new version of kubeadmin on master node
1.3.2 upgrade cluster
kubeadm upgrade apply v1.19.0
//apply the kubeadm upgrade of master node, running the command on master node
1.3.3 upgrade kubelet to new version on master node
apt-get install kubelet=1.19.0
or
apt-get upgrade -y kubelet=1.19.0
//restart kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet
1.3.4 upgrade kubelet on worker node,
//first drain the worker node from master, then run "ssh workernodename" to login to worker node
apt-get upgrade -y kubelet=1.19.0
apt-get upgrade -y kubeadm=1.19.0
kubeadm upgrade node
//the above command also applies to worker node and other control plan (not master control node)
sudo systemctl daemon-reload
sudo systemctl restart kubelet
at last, uncordon the worker node from master node
2. Manage cluster with kubectl
After kubernetes is installed, on the master node's VM console terminal, user can use kubectl to communicate with kubernetes to manage kubernetes cluster, such as, create/delete nodes/pods, get info from nodes/pods, scale up and down. Internally, kuberctl sends http requests to kubernetes API server, which communicates to different services and controllers to be sure all worker nodes are in good state.
To get the version for yaml file on different resource type, run the below command
kubectl explain typeofresource, for example
kubectl explain replicaset
2.1 creating pod
// create a pod with run command
kubectl run mypodname --image dockerimagename
kubectl run mytestpod --image=nginx --dry-run=client -o yaml > mypod.yaml
//get pod info
kubectl get pods
kubectl get pods -o wide
kubectl describe pod podname
//delete pod
kubectl delete pod podname
//use dry run to generate pod create yaml file
kubectl run mytestpod --image=nginx --dry-run=client -o yaml > mypod.yaml
//generated yaml file as below
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: mytestpod
name: mytestpod
spec:
containers:
- image: nginx
name: mytestpod
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
// create a pod with a yaml file
kubectl create -f pod.yml
//apply the content in yaml file
kubectl apply -f pod.yaml
//update pod config
kubectl edit pod podname
//select pod based on label matching
kubectl get pod --selector env=prod,bu=finance,tier=frontend
//generate a yaml file based on exist pod
kubectl get pod mypod -o yaml > mypod.yaml
//create a pod with resource limit and request
apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: app
image: images.my-company.example/app:v4
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
- name: log-aggregator
image: images.my-company.example/log-aggregator:v6
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
2.2 Create Replicaset
Replicaset yaml file's spec section must include pod creating template, selector and replica number fields
kubectl create -f replicaset -f myreplicaset.yaml
kubectl get replicaset -o wide
kubectl delete replicaset myreplicasetname
kubectl replace -f myreplicasetfilename.yaml
kubectl scale replicaset --replicas=10 myreplicasetname
kubectl describe replicaset myreplicasetname
//yaml file to create replicaset
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: frontend
labels:
app: guestbook
tier: frontend
spec:
# modify replicas according to your case
replicas: 3
selector:
matchLabels:
tier: frontend
template:
metadata:
labels:
tier: frontend
spec:
containers:
- name: php-redis
image: gcr.io/google_samples/gb-frontend:v3
2.3 create Deployment
kubectl create deployment --image=nginx mynginx --replicas=4
kubectl create deployment --image=nginx mynginx --dry-run=client -o yaml
kubectl create deployment --image=nginx mynginx --dry-run=client -o yaml > nginx-deployment.yaml
kubectl scale deployment mynginx --replicas=4
//expose deployment as clusterIP on a particular port
kubectl expose deployment mynginx --port 80
kubectl get deployment mydeployment -o yaml
kubectl edit service mynginx
kubectl edit deployment mydeployment
2.4 namespace
kubectl get pods --namespace=dev
kubectl config set-context $(kubectl config current-context) --namespace=dev
kubectl get pods --all-namespaces
2.5 get statistic from pod and node
kubectl top node
kubectl top pod
2.6 get log
kubectl logs -f podname containername
2.7 rollout deployment
Rollout operation is based on the deployment name, the deployment with the same name can be restart pause, resume, undo, etc
//roll back a deployment to previous change
kubectl rollout undo deployment/mydeployment
2.8 command and arg
In pod yaml file, it can uses command and args to replace Docker file's ENTRYPOINT (execution command) and CMD (argument) field
apiVersion: v1
kind: Pod
metadata:
name: command-demo
labels:
purpose: demonstrate-command
spec:
containers:
- name: command-demo-container
image: debian
command: ["printenv"]
args: ["HOSTNAME", "KUBERNETES_PORT"]
In pod yaml file, it can uses command and args to replace Docker file's ENTRYPOINT (execution command) and CMD (argument) field
To execute a shell command in container
kubectl exec mypodname -c mycontainername -i -t -- mybashcommand
//run interactive bash command
kubectl exec mypodname -it -- sh
2.9 ConfigMaps for Environment variables
kubectl create configmap myconfigmap --from-literal=mykey1=myvalue1 --from-literal=mykey2=myvalue2
kubectl create configmap myconfigmap --from-file=mypropertyfile
Using yaml file to create config map
kubectl create -f myconfigmapyamlfile
apiVersion: v1
kind: ConfigMap
metadata:
name: game-config-env-file
namespace: default
resourceVersion: "809965"
uid: d9d1ca5b-eb34-11e7-887b-42010a8002b8
data:
allowed: '"true"'
enemies: aliens
lives: "3"
sample pod yaml file to set environment from config map
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
envFrom:
- configMapRef:
name: special-config
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
- name: SPECIAL_LEVEL_KEY
valueFrom:
configMapKeyRef:
name: special-config
key: special.how
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: env-config
key: log_level
2.10 secret for pod
kubectl create secret generic mysecret --from-literal=mykey1=myvalue1 --from-literal=mykey2=myvalue2
kubectl create secret myconfigmap --from-file=mypropertyfile
kubectl create -f mysecretfile
Generate hash value for secret file
echo -n "myvalue" | based64
decode hash value
echo -n 'myhashedvalue" | base64 --decode
Set environment variable from secret
apiVersion: v1
kind: Pod
metadata:
name: secret-env-pod
spec:
containers:
- name: mycontainer
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password
apiVersion: v1
kind: Pod
metadata:
name: secret-test-pod
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
envFrom:
- secretRef:
name: mysecret
restartPolicy: Never
create docker registry secret
kubectl create secret docker-registry private-reg-cred --docker-username=docker_user --docker-password=dock_password --docker-server=myprivateregistry.com:5000 --docker-email=dock_user@myprivateregistry.com
2.11 certificate management
kubectl get csr
get certificate signing request
based 64 encode the csr and remove \n
cat john.csr | base64 | tr -d "\n"
kubectl certificate approve mycsrname
signing a certificate request and create the signed certificate
kubectl certificate deny mycsrname
deny a certificate signing request
kubectl certificate delete csr myscrname
delete certificate signing request
After the certificate is signed, use the below command to view the cert, the generated cert is in status/certificate field. The content is base 64 encoded, it need to be base64 decoded.
kubectl get csr mycsrname -o yaml
Certificate signing request yaml file
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: john
spec:
groups:
- system:authenticated
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZqQ0NBVDRDQVFBd0VURVBNQTBHQTFVRUF3d0dZVzVuWld4aE1JSUJJakFOQmdrcWhraUc5dzBCQVFFRgpBQU9DQVE4QU1JSUJDZ0tDQVFFQTByczhJTHRHdTYxakx2dHhWTTJSVlRWMDNHWlJTWWw0dWluVWo4RElaWjBOCnR2MUZtRVFSd3VoaUZsOFEzcWl0Qm0wMUFSMkNJVXBGd2ZzSjZ4MXF3ckJzVkhZbGlBNVhwRVpZM3ExcGswSDQKM3Z3aGJlK1o2MVNrVHF5SVBYUUwrTWM5T1Nsbm0xb0R2N0NtSkZNMUlMRVI3QTVGZnZKOEdFRjJ6dHBoaUlFMwpub1dtdHNZb3JuT2wzc2lHQ2ZGZzR4Zmd4eW8ybmlneFNVekl1bXNnVm9PM2ttT0x1RVF6cXpkakJ3TFJXbWlECklmMXBMWnoyalVnald4UkhCM1gyWnVVV1d1T09PZnpXM01LaE8ybHEvZi9DdS8wYk83c0x0MCt3U2ZMSU91TFcKcW90blZtRmxMMytqTy82WDNDKzBERHk5aUtwbXJjVDBnWGZLemE1dHJRSURBUUFCb0FBd0RRWUpLb1pJaHZjTgpBUUVMQlFBRGdnRUJBR05WdmVIOGR4ZzNvK21VeVRkbmFjVmQ1N24zSkExdnZEU1JWREkyQTZ1eXN3ZFp1L1BVCkkwZXpZWFV0RVNnSk1IRmQycVVNMjNuNVJsSXJ3R0xuUXFISUh5VStWWHhsdnZsRnpNOVpEWllSTmU3QlJvYXgKQVlEdUI5STZXT3FYbkFvczFqRmxNUG5NbFpqdU5kSGxpT1BjTU1oNndLaTZzZFhpVStHYTJ2RUVLY01jSVUyRgpvU2djUWdMYTk0aEpacGk3ZnNMdm1OQUxoT045UHdNMGM1dVJVejV4T0dGMUtCbWRSeEgvbUNOS2JKYjFRQm1HCkkwYitEUEdaTktXTU0xMzhIQXdoV0tkNjVoVHdYOWl4V3ZHMkh4TG1WQzg0L1BHT0tWQW9FNkpsYWFHdTlQVmkKdjlOSjVaZlZrcXdCd0hKbzZXdk9xVlA3SVFjZmg3d0drWm89Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
signerName: kubernetes.io/kube-apiserver-client
usages:
- client auth
2.12 cluster config
kubectl config view
view the cluster config file in ~/home/.kube/config
The current-conext element specifies the current context used by kubectl.
To switch to a different context, using the below command:
kubectl config use-context myusername@myclustername
kubectl config --kubeconfig=/mypath/myconfigfile use-context mycontextname
2.13 volume
1. volume has the same lifetime of a pod, it outlives any containers that run within the pod, and the data is preserved across container restarts. When a pod ceases to exist, the volume is destroyed.
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
# this field is optional
type: Directory
2. Persistent Volume
Persistent Volume has a lifecycle independent of any individual pod. It is a resource provisioned by cluster admin. A persistentVolumeClaim is a request by a pod to ask for the persistent volume resource.
persistent volume yaml file sample
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv0003
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: slow
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /tmp
server: 172.17.0.2
PersistentVolumeClaim yaml sample
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myclaim
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 8Gi
storageClassName: slow
selector:
matchLabels:
release: "stable"
matchExpressions:
- {key: environment, operator: In, values: [dev]}
Pod yaml sample to request a persistentVolumeClaim
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
volumeMounts:
- mountPath: "/var/www/html"
name: mypd
volumes:
- name: mypd
persistentVolumeClaim:
claimName: myclaim
StorageClass yaml file sample
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
- debug
volumeBindingMode: Immediate
3. Scheduler
3.1 Scheduler is used to decide which pod runs on which node.
The basic scheduler uses nodeName to select a node with matched name, and run on it.
3.2 taints and toleration
Taint allows a node to define a rule to expel all the pods it does not want, so only those pods with matched toleration can be hosted, in this case, node is rule decider. taint has three effects:
NoSchedule
PreferNoSchedule
NoExecute
//set taint for a node, which means no pod can be schedule onto node1 unless it has a match toleration
kutectl taint nodes node1 app=blue:NoSchedule
Pod with matched toleration can be hosted in the nodes with taint
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
tolerations:
- key: "example-key"
operator: "Exists"
effect: "NoSchedule"
another toleration operation is equal, which has the same result as exists
tolerations:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
3.3 node selection and affinity
Pods selection and affinity let pods decide which node can host it, so it expels all other node it does not want. In this case, pod is decision maker.
3.3.1 pod creation yaml file can contain nodeSelector element which indicates which node can be used to host the pod.
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
nodeSelector:
disktype: ssd
3.3.2 node affinity
Node affinity are configuration which uses complex express in pod definition to match which node can host the pod
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: disktype
operator: In
values:
- ssd
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
3.4 resource request and limit
Pod can specify its resource request and resource limit to let scheduler to decide which node can be used to put the pod
apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: app
image: images.my-company.example/app:v4
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
- name: log-aggregator
image: images.my-company.example/log-aggregator:v6
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
3.5 node maintenance
kubectl drain nodename --ignore-daemonsets --force
Move current pods in the specified node to other nodes, and also mark the specified node as unable to schedule.
kubectl cordon mynode
mark the specified node as unable to schedule.
kubectl uncordon mynodename
mark a node to be able to schedule
Backup cluster data
ETCDCTL_API=3
etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/snapshot-pre-boot.db
restore to a new data folder
etcdctl snapshot restore restorefilepath --data-dir=newdatapath
edit etcd manifest file to set data directory to the new data folder
vim /etc/kubernetes/manifests/etcd.yaml
4. Node
In Kubernetes, a node is a VM or a local computer hosting pods. Kubernetes has two types of nodes: master and worker node (), A node can have multiple pods, and each pod can have multiple containers.
kubectl get nodes
Kubernetes scheduler decides which node will host a particular pod.
Label the nodes for pod selection
kubectl label nodes mynodename labelkey=labelvalue
To execute bash command on a node in cluster, use ssh command.
To execute bash command on a container in a pod, use kubectl exec -it podname -- sh
Master:
master node manages the kubernetes cluster. It contains a component Kube-APIServe to communicate with other management components and all worker nodes.
Master node contains services to allow pods to communicate to each other and to the outside world, master node also contains controllers to manage and upgrade the deployment of pods
Worker:
worker nodes contain one or more pods, a pod contains one or more (docker) containers. A pod has a unique ip address and mounted storage shared by all containers within it.
5. Network configuration:
Kube-Proxy
kube-proxy is used for one pod to access another pod, however, as pod's ip address may change anytime, so calling pod cannot directly connect to target pod using pod's ip address. Instead, the service is used to link the fixed service ip address to dynamic pod ip address. Kube-proxy has a service table to map each service to the current available pod ip address which provides the specific service.
Cluster IP
cluster ip groups the pods with same function and expose them with a single ip address (cluster IP) and port number. When requests sent to the cluster ip and port, it will be redirect to one of matched pods. on the targetPort.
The output of
kubectl describe service myclusteripservice -o wide
includes endpoint field, which is the matched pods' ip address and ports.
//create a service of ClusterIP to expose the pods on a particular port
kubectl expose pod myredis --port=6379 --name=redis-service --dry-run=client -o yaml
Cluster IP is used for internal communication between function providers to function consumers.
yaml file to create cluster ip
apiVersion: v1
kind: Service
metadata:
name: backend
spec:
type: ClusterIP
ports:
- targetPort: 80
port: 81
selector:
app: mybackendapp
NodePort
NodePort is a service running on a worker node which listens on a particular port on the node, and then forward the requests received on that port to one of matched pod. The client request must send to a particular pod's ip address and node port.
The NodePort definition also defines the listening port on the matched pod using targetPort field.
yaml file to create service:
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
type: NodePort
ports:
- targetPort: 80 //listening port on the matched pods for handling external request
port: 88 //service port number to be connected by pods or other services
nodePort: 30008 //listening port on node for accepting external request
selector:
app: myapp //label for marching pod
yaml file for creating matched pod:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
app: myapp //matching label for pods
name: mytestpod
spec:
containers:
- image: nginx
name: mytestpod
Client can send requests from anywhere to node's external ip address and Nodeport (30008) in order to access the pod's listening port 80.
External load balancer
Using external loadbalance with kubernetes can expose services to external with NodePort, the client request will be assigned to different NodePort port to spread the load balance.
Ingress (Internal loadbalancer)
Ingress within kubernetes receives the external requests and distribute the requests to different services' cluster IP based on ingress rule. It is preferred than external loadbalancer.
