Saturday 4 January 2020

Upgrading kubeadm clusters from v1.14 to v1.15 - Part 1

Trying to upgrade the kubenetes cluster, but we need to upgrade minor version to latest with increment of one
first, we shall upgrade master node and later worker nodes.

current version:

k8@master:~$ kubectl get nodes
NAME      STATUS   ROLES    AGE    VERSION
master   Ready    master   215d   v1.14.2
node01     Ready    <none>   214d   v1.14.2
k8@master:~$

Drain master node for maintenance so that no new pods would be scheduled on the master node.

k8@master:~$ kubectl cordon master
node/master cordoned
k8@master:~$

k8@master:~$ kubectl get nodes
NAME      STATUS                     ROLES    AGE    VERSION
master   Ready,SchedulingDisabled   master   215d   v1.14.2
node01     Ready                      <none>   214d   v1.14.2
k8@master:~$

k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:20:34Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$

sudo apt-get upgrade
sudo apt-cache policy kubeadm

you will now need to see the latest kubeadm with upgraded minor version ..  I am looking at v1.15.0

k8@master:~$ sudo apt-cache policy kubeadm
kubeadm:
  Installed: 1.14.2-00
  Candidate: 1.17.0-00
  Version table:
     1.17.0-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.4-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.3-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.2-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.1-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.0-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.15.7-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages <== last patch version
     1.15.6-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages

upgrade control plane node, I would be using the latest patch version for v1.15.7

k8@master:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:3 http://security.ubuntu.com/ubuntu cosmic-security InRelease                                                                
Hit:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease                                                                  
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease                                          
Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done                    
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following additional packages will be installed:
  cri-tools
The following packages will be upgraded:
  cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 225 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 8s (2,005 kB/s)                                                                                                                                                                              
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@master:~$

k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:40:15Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$

k8@master:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
I0104 06:01:59.979160   29689 version.go:248] remote version is much newer: v1.17.0; falling back to: stable-1.15
[upgrade/versions] Latest stable version: v1.15.7
[upgrade/versions] Latest version in the v1.14 series: v1.14.10

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     2 x v1.14.2   v1.14.10

Upgrade to the latest version in the v1.14 series:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.14.2   v1.14.10
Controller Manager   v1.14.2   v1.14.10
Scheduler            v1.14.2   v1.14.10
Kube Proxy           v1.14.2   v1.14.10
CoreDNS              1.3.1     1.3.1
Etcd                 3.3.10    3.3.10

You can now apply the upgrade by executing the following command:

kubeadm upgrade apply v1.14.10

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     2 x v1.14.2   v1.15.7

Upgrade to the latest stable version:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.14.2   v1.15.7
Controller Manager   v1.14.2   v1.15.7
Scheduler            v1.14.2   v1.15.7
Kube Proxy           v1.14.2   v1.15.7
CoreDNS              1.3.1     1.3.1
Etcd                 3.3.10    3.3.10

You can now apply the upgrade by executing the following command:

kubeadm upgrade apply v1.15.7

_____________________________________________________________________


k8@master:~$ sudo kubeadm upgrade apply v1.15.7
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.15.7"
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.15.7"...
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests124065540"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: 461cf48224e9b4057addb8c3f5d64870
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: 9d89927ff1a0d70cf9452b3af5827f19
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
Static pod: kube-scheduler-master hash: 7d6a1cec31a680b45724ee90bd535b49
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.7". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
k8@master:~$

k8@master:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl

kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease        
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease      
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease              
Reading package lists... Done
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)                                                                                                                                                                              
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@master:~$


k8@master:~$ sudo systemctl restart kubelet
k8@master:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Sat 2020-01-04 06:08:20 UTC; 9s ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 4890 (kubelet)
    Tasks: 19 (limit: 4565)
   Memory: 27.6M
   CGroup: /system.slice/kubelet.service
           └─4890 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network


k8@master:~$ kubectl get nodes
NAME      STATUS                     ROLES    AGE    VERSION
master   Ready,SchedulingDisabled   master   215d   v1.15.7
node01     Ready                      <none>   214d   v1.14.2
k8@master:~$

uncordon the master nodes so that any new pods would be scheduled.

k8@master:~$ kubectl uncordon master
node/master uncordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME      STATUS   ROLES    AGE    VERSION
master   Ready    master   215d   v1.15.7  <== cluster has been upgraded !
node01     Ready    <none>   214d   v1.14.2
k8@master:~$
k8@master:~$


+++++++++++++++=   upgrading worker nodes +++++++++++++++

update kubeadm on the worker nodes.

k8@node01:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Get:1 http://security.ubuntu.com/ubuntu cosmic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease                  
Get:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease [88.7 kB]          
Get:5 http://security.ubuntu.com/ubuntu cosmic-security/main i386 Packages [197 kB]                                                  
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]                      
Get:6 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease [74.6 kB]                                        
Get:7 http://archive.ubuntu.com/ubuntu cosmic-updates/main i386 Packages [332 kB]                                                
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages                                            
Get:9 http://security.ubuntu.com/ubuntu cosmic-security/main amd64 Packages [210 kB]                                          
Get:10 http://security.ubuntu.com/ubuntu cosmic-security/main Translation-en [84.1 kB]                                                
Get:11 http://security.ubuntu.com/ubuntu cosmic-security/universe i386 Packages [498 kB]                                                        
Get:12 http://archive.ubuntu.com/ubuntu cosmic-updates/main amd64 Packages [345 kB]                                                            
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages                                                      
Get:13 http://archive.ubuntu.com/ubuntu cosmic-updates/main Translation-en [144 kB]                                                    
Get:14 http://security.ubuntu.com/ubuntu cosmic-security/universe amd64 Packages [501 kB]                                                  
Get:15 http://archive.ubuntu.com/ubuntu cosmic-updates/universe amd64 Packages [697 kB]                                                    
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [274 kB]                                          
Get:16 http://security.ubuntu.com/ubuntu cosmic-security/universe Translation-en [144 kB]                                                      
Get:17 http://security.ubuntu.com/ubuntu cosmic-security/multiverse amd64 Packages [3,744 B]                                                        
Get:18 http://security.ubuntu.com/ubuntu cosmic-security/multiverse i386 Packages [3,904 B]                                                            
Get:19 http://archive.ubuntu.com/ubuntu cosmic-updates/universe i386 Packages [692 kB]                                                                
Get:20 http://archive.ubuntu.com/ubuntu cosmic-updates/universe Translation-en [195 kB]                              
Get:21 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse i386 Packages [3,904 B]        
Get:22 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse amd64 Packages [3,744 B]
Get:23 http://archive.ubuntu.com/ubuntu cosmic-backports/universe i386 Packages [3,992 B]
Get:24 http://archive.ubuntu.com/ubuntu cosmic-backports/universe amd64 Packages [3,996 B]
Fetched 4,597 kB in 3s (1,349 kB/s)            
Reading package lists... Done
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following additional packages will be installed:
  cri-tools
The following packages will be upgraded:
  cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 168 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 4s (4,646 kB/s)  
(Reading database ... 41117 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@node01:~$

Drain worker node so that any new pods won't be scheduled, already existing nodes would be evicted to other nodes.


k8@master:~$ kubectl drain node01 --ignore-daemonsets
node/node01 cordoned
k8@master:~$


k8@node01:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
k8@node01:~$

k8@node01:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl

kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease        
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease      
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease              
Reading package lists... Done
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)                                                                                                                                                                              
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@node01:~$


k8@node01:~$ sudo systemctl restart kubelet
k8@node01:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Sat 2020-01-04 06:24:13 UTC; 4s ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 19127 (kubelet)
    Tasks: 14 (limit: 1134)
   Memory: 31.4M
   CGroup: /system.slice/kubelet.service
           └─19127 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --networ

k8@node01:~$


k8@master:~$ kubectl get nodes
NAME      STATUS                     ROLES    AGE    VERSION
master   Ready                      master   215d   v1.15.7
node01     Ready,SchedulingDisabled   <none>   214d   v1.15.7
k8@master:~$

k8@master:~$ kubectl uncordon node01
node/node01 uncordoned
k8@master:~$

k8@master:~$ kubectl get nodes
NAME      STATUS   ROLES    AGE    VERSION
master   Ready    master   215d   v1.15.7
node01     Ready    <none>   214d   v1.15.7
k8@master:~$

Trying to upgrade the kubenetes cluster, but we need to upgrade minor version to latest with increment of one
first, we shall upgrade master node and later worker nodes.

current version:

k8@master:~$ kubectl get nodes
NAME      STATUS   ROLES    AGE    VERSION
master   Ready    master   215d   v1.14.2
node01     Ready    <none>   214d   v1.14.2
k8@master:~$

Drain master node for maintenance so that no new pods would be scheduled on the master node.

k8@master:~$ kubectl cordon master
node/master cordoned
k8@master:~$

k8@master:~$ kubectl get nodes
NAME      STATUS                     ROLES    AGE    VERSION
master   Ready,SchedulingDisabled   master   215d   v1.14.2
node01     Ready                      <none>   214d   v1.14.2
k8@master:~$

k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:20:34Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$

sudo apt-get upgrade
sudo apt-cache policy kubeadm

you will now need to see the latest kubeadm with upgraded minor version ..  I am looking at v1.15.0

k8@master:~$ sudo apt-cache policy kubeadm
kubeadm:
  Installed: 1.14.2-00
  Candidate: 1.17.0-00
  Version table:
     1.17.0-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.4-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.3-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.2-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.1-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.16.0-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.15.7-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages <== last patch version
     1.15.6-00 500
        500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages

upgrade control plane node, I would be using the latest patch version for v1.15.7

k8@master:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:3 http://security.ubuntu.com/ubuntu cosmic-security InRelease                                                                
Hit:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease                                                                  
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease                                          
Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done                    
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following additional packages will be installed:
  cri-tools
The following packages will be upgraded:
  cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 225 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 8s (2,005 kB/s)                                                                                                                                                                              
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@master:~$

k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:40:15Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$

k8@master:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
I0104 06:01:59.979160   29689 version.go:248] remote version is much newer: v1.17.0; falling back to: stable-1.15
[upgrade/versions] Latest stable version: v1.15.7
[upgrade/versions] Latest version in the v1.14 series: v1.14.10

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     2 x v1.14.2   v1.14.10

Upgrade to the latest version in the v1.14 series:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.14.2   v1.14.10
Controller Manager   v1.14.2   v1.14.10
Scheduler            v1.14.2   v1.14.10
Kube Proxy           v1.14.2   v1.14.10
CoreDNS              1.3.1     1.3.1
Etcd                 3.3.10    3.3.10

You can now apply the upgrade by executing the following command:

kubeadm upgrade apply v1.14.10

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     2 x v1.14.2   v1.15.7

Upgrade to the latest stable version:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.14.2   v1.15.7
Controller Manager   v1.14.2   v1.15.7
Scheduler            v1.14.2   v1.15.7
Kube Proxy           v1.14.2   v1.15.7
CoreDNS              1.3.1     1.3.1
Etcd                 3.3.10    3.3.10

You can now apply the upgrade by executing the following command:

kubeadm upgrade apply v1.15.7

_____________________________________________________________________


k8@master:~$ sudo kubeadm upgrade apply v1.15.7
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.15.7"
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.15.7"...
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests124065540"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: 461cf48224e9b4057addb8c3f5d64870
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: 9d89927ff1a0d70cf9452b3af5827f19
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
Static pod: kube-scheduler-master hash: 7d6a1cec31a680b45724ee90bd535b49
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.7". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
k8@master:~$

k8@master:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl

kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease        
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease      
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease              
Reading package lists... Done
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)                                                                                                                                                                              
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@master:~$


k8@master:~$ sudo systemctl restart kubelet
k8@master:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Sat 2020-01-04 06:08:20 UTC; 9s ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 4890 (kubelet)
    Tasks: 19 (limit: 4565)
   Memory: 27.6M
   CGroup: /system.slice/kubelet.service
           └─4890 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network


k8@master:~$ kubectl get nodes
NAME      STATUS                     ROLES    AGE    VERSION
master   Ready,SchedulingDisabled   master   215d   v1.15.7
node01     Ready                      <none>   214d   v1.14.2
k8@master:~$

uncordon the master nodes so that any new pods would be scheduled.

k8@master:~$ kubectl uncordon master
node/master uncordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME      STATUS   ROLES    AGE    VERSION
master   Ready    master   215d   v1.15.7  <== cluster has been upgraded !
node01     Ready    <none>   214d   v1.14.2
k8@master:~$
k8@master:~$


+++++++++++++++=   upgrading worker nodes +++++++++++++++

update kubeadm on the worker nodes.

k8@node01:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Get:1 http://security.ubuntu.com/ubuntu cosmic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease                  
Get:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease [88.7 kB]          
Get:5 http://security.ubuntu.com/ubuntu cosmic-security/main i386 Packages [197 kB]                                                  
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]                      
Get:6 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease [74.6 kB]                                        
Get:7 http://archive.ubuntu.com/ubuntu cosmic-updates/main i386 Packages [332 kB]                                                
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages                                            
Get:9 http://security.ubuntu.com/ubuntu cosmic-security/main amd64 Packages [210 kB]                                          
Get:10 http://security.ubuntu.com/ubuntu cosmic-security/main Translation-en [84.1 kB]                                                
Get:11 http://security.ubuntu.com/ubuntu cosmic-security/universe i386 Packages [498 kB]                                                        
Get:12 http://archive.ubuntu.com/ubuntu cosmic-updates/main amd64 Packages [345 kB]                                                            
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages                                                      
Get:13 http://archive.ubuntu.com/ubuntu cosmic-updates/main Translation-en [144 kB]                                                    
Get:14 http://security.ubuntu.com/ubuntu cosmic-security/universe amd64 Packages [501 kB]                                                  
Get:15 http://archive.ubuntu.com/ubuntu cosmic-updates/universe amd64 Packages [697 kB]                                                    
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [274 kB]                                          
Get:16 http://security.ubuntu.com/ubuntu cosmic-security/universe Translation-en [144 kB]                                                      
Get:17 http://security.ubuntu.com/ubuntu cosmic-security/multiverse amd64 Packages [3,744 B]                                                        
Get:18 http://security.ubuntu.com/ubuntu cosmic-security/multiverse i386 Packages [3,904 B]                                                            
Get:19 http://archive.ubuntu.com/ubuntu cosmic-updates/universe i386 Packages [692 kB]                                                                
Get:20 http://archive.ubuntu.com/ubuntu cosmic-updates/universe Translation-en [195 kB]                              
Get:21 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse i386 Packages [3,904 B]        
Get:22 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse amd64 Packages [3,744 B]
Get:23 http://archive.ubuntu.com/ubuntu cosmic-backports/universe i386 Packages [3,992 B]
Get:24 http://archive.ubuntu.com/ubuntu cosmic-backports/universe amd64 Packages [3,996 B]
Fetched 4,597 kB in 3s (1,349 kB/s)            
Reading package lists... Done
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following additional packages will be installed:
  cri-tools
The following packages will be upgraded:
  cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 168 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 4s (4,646 kB/s)  
(Reading database ... 41117 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@node01:~$

Drain worker node so that any new pods won't be scheduled, already existing nodes would be evicted to other nodes.

k8@master:~$ kubectl drain node01 --ignore-daemonsets
node/node01 cordoned
k8@master:~$

k8@node01:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
k8@node01:~$

k8@node01:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl

kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease        
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease      
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease              
Reading package lists... Done
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)                                                                                                                                                                         
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@node01:~$

k8@node01:~$ sudo systemctl restart kubelet
k8@node01:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Sat 2020-01-04 06:24:13 UTC; 4s ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 19127 (kubelet)
    Tasks: 14 (limit: 1134)
   Memory: 31.4M
   CGroup: /system.slice/kubelet.service
           └─19127 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --networ

k8@node01:~$

k8@master:~$ kubectl get nodes
NAME      STATUS                     ROLES    AGE    VERSION
master   Ready                      master   215d   v1.15.7
node01     Ready,SchedulingDisabled   <none>   214d   v1.15.7
k8@master:~$

k8@master:~$ kubectl uncordon node01
node/node01 uncordoned
k8@master:~$

finally, we have upgraded kubernetes cluster to v1.15.7. 

k8@master:~$ kubectl get nodes
NAME      STATUS   ROLES    AGE    VERSION
master   Ready    master   215d   v1.15.7
node01     Ready    <none>   214d   v1.15.7
k8@master:~$ 

k8@master:~$ kubectl get deployments
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           63s
k8@master:~$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP                NODE    NOMINATED NODE   READINESS GATES
nginx-7bb7cd8db5-mrzk4   1/1     Running   0          69s   192.168.177.232   knode   <none>           <none>
k8@master:~$



No comments:

Post a Comment