Trying to upgrade the kubenetes cluster, but we need to upgrade minor version to latest with increment of one
first, we shall upgrade master node and later worker nodes.
current version:
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.14.2
node01 Ready <none> 214d v1.14.2
k8@master:~$
Drain master node for maintenance so that no new pods would be scheduled on the master node.
k8@master:~$ kubectl cordon master
node/master cordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready,SchedulingDisabled master 215d v1.14.2
node01 Ready <none> 214d v1.14.2
k8@master:~$
k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:20:34Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$
sudo apt-get upgrade
sudo apt-cache policy kubeadm
you will now need to see the latest kubeadm with upgraded minor version .. I am looking at v1.15.0
k8@master:~$ sudo apt-cache policy kubeadm
kubeadm:
Installed: 1.14.2-00
Candidate: 1.17.0-00
Version table:
1.17.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.4-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.3-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.2-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.1-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.15.7-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages <== last patch version
1.15.6-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
upgrade control plane node, I would be using the latest patch version for v1.15.7
k8@master:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:3 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
cri-tools
The following packages will be upgraded:
cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 225 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 8s (2,005 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@master:~$
k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:40:15Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$
k8@master:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
I0104 06:01:59.979160 29689 version.go:248] remote version is much newer: v1.17.0; falling back to: stable-1.15
[upgrade/versions] Latest stable version: v1.15.7
[upgrade/versions] Latest version in the v1.14 series: v1.14.10
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 2 x v1.14.2 v1.14.10
Upgrade to the latest version in the v1.14 series:
COMPONENT CURRENT AVAILABLE
API Server v1.14.2 v1.14.10
Controller Manager v1.14.2 v1.14.10
Scheduler v1.14.2 v1.14.10
Kube Proxy v1.14.2 v1.14.10
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.14.10
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 2 x v1.14.2 v1.15.7
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.14.2 v1.15.7
Controller Manager v1.14.2 v1.15.7
Scheduler v1.14.2 v1.15.7
Kube Proxy v1.14.2 v1.15.7
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.15.7
_____________________________________________________________________
k8@master:~$ sudo kubeadm upgrade apply v1.15.7
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.15.7"
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.15.7"...
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests124065540"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: 461cf48224e9b4057addb8c3f5d64870
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: 9d89927ff1a0d70cf9452b3af5827f19
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
Static pod: kube-scheduler-master hash: 7d6a1cec31a680b45724ee90bd535b49
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.7". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
k8@master:~$
k8@master:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@master:~$
k8@master:~$ sudo systemctl restart kubelet
k8@master:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2020-01-04 06:08:20 UTC; 9s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 4890 (kubelet)
Tasks: 19 (limit: 4565)
Memory: 27.6M
CGroup: /system.slice/kubelet.service
└─4890 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready,SchedulingDisabled master 215d v1.15.7
node01 Ready <none> 214d v1.14.2
k8@master:~$
uncordon the master nodes so that any new pods would be scheduled.
k8@master:~$ kubectl uncordon master
node/master uncordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7 <== cluster has been upgraded !
node01 Ready <none> 214d v1.14.2
k8@master:~$
k8@master:~$
+++++++++++++++= upgrading worker nodes +++++++++++++++
update kubeadm on the worker nodes.
k8@node01:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Get:1 http://security.ubuntu.com/ubuntu cosmic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Get:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease [88.7 kB]
Get:5 http://security.ubuntu.com/ubuntu cosmic-security/main i386 Packages [197 kB]
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:6 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease [74.6 kB]
Get:7 http://archive.ubuntu.com/ubuntu cosmic-updates/main i386 Packages [332 kB]
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Get:9 http://security.ubuntu.com/ubuntu cosmic-security/main amd64 Packages [210 kB]
Get:10 http://security.ubuntu.com/ubuntu cosmic-security/main Translation-en [84.1 kB]
Get:11 http://security.ubuntu.com/ubuntu cosmic-security/universe i386 Packages [498 kB]
Get:12 http://archive.ubuntu.com/ubuntu cosmic-updates/main amd64 Packages [345 kB]
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Get:13 http://archive.ubuntu.com/ubuntu cosmic-updates/main Translation-en [144 kB]
Get:14 http://security.ubuntu.com/ubuntu cosmic-security/universe amd64 Packages [501 kB]
Get:15 http://archive.ubuntu.com/ubuntu cosmic-updates/universe amd64 Packages [697 kB]
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [274 kB]
Get:16 http://security.ubuntu.com/ubuntu cosmic-security/universe Translation-en [144 kB]
Get:17 http://security.ubuntu.com/ubuntu cosmic-security/multiverse amd64 Packages [3,744 B]
Get:18 http://security.ubuntu.com/ubuntu cosmic-security/multiverse i386 Packages [3,904 B]
Get:19 http://archive.ubuntu.com/ubuntu cosmic-updates/universe i386 Packages [692 kB]
Get:20 http://archive.ubuntu.com/ubuntu cosmic-updates/universe Translation-en [195 kB]
Get:21 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse i386 Packages [3,904 B]
Get:22 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse amd64 Packages [3,744 B]
Get:23 http://archive.ubuntu.com/ubuntu cosmic-backports/universe i386 Packages [3,992 B]
Get:24 http://archive.ubuntu.com/ubuntu cosmic-backports/universe amd64 Packages [3,996 B]
Fetched 4,597 kB in 3s (1,349 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
cri-tools
The following packages will be upgraded:
cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 168 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 4s (4,646 kB/s)
(Reading database ... 41117 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@node01:~$
Drain worker node so that any new pods won't be scheduled, already existing nodes would be evicted to other nodes.
k8@master:~$ kubectl drain node01 --ignore-daemonsets
node/node01 cordoned
k8@master:~$
k8@node01:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
k8@node01:~$
k8@node01:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@node01:~$
k8@node01:~$ sudo systemctl restart kubelet
k8@node01:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2020-01-04 06:24:13 UTC; 4s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 19127 (kubelet)
Tasks: 14 (limit: 1134)
Memory: 31.4M
CGroup: /system.slice/kubelet.service
└─19127 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --networ
k8@node01:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7
node01 Ready,SchedulingDisabled <none> 214d v1.15.7
k8@master:~$
k8@master:~$ kubectl uncordon node01
node/node01 uncordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7
node01 Ready <none> 214d v1.15.7
k8@master:~$
Trying to upgrade the kubenetes cluster, but we need to upgrade minor version to latest with increment of one
first, we shall upgrade master node and later worker nodes.
current version:
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.14.2
node01 Ready <none> 214d v1.14.2
k8@master:~$
Drain master node for maintenance so that no new pods would be scheduled on the master node.
k8@master:~$ kubectl cordon master
node/master cordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready,SchedulingDisabled master 215d v1.14.2
node01 Ready <none> 214d v1.14.2
k8@master:~$
k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:20:34Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$
sudo apt-get upgrade
sudo apt-cache policy kubeadm
you will now need to see the latest kubeadm with upgraded minor version .. I am looking at v1.15.0
k8@master:~$ sudo apt-cache policy kubeadm
kubeadm:
Installed: 1.14.2-00
Candidate: 1.17.0-00
Version table:
1.17.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.4-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.3-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.2-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.1-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.15.7-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages <== last patch version
1.15.6-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
upgrade control plane node, I would be using the latest patch version for v1.15.7
k8@master:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:3 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
cri-tools
The following packages will be upgraded:
cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 225 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 8s (2,005 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@master:~$
k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:40:15Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$
k8@master:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
I0104 06:01:59.979160 29689 version.go:248] remote version is much newer: v1.17.0; falling back to: stable-1.15
[upgrade/versions] Latest stable version: v1.15.7
[upgrade/versions] Latest version in the v1.14 series: v1.14.10
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 2 x v1.14.2 v1.14.10
Upgrade to the latest version in the v1.14 series:
COMPONENT CURRENT AVAILABLE
API Server v1.14.2 v1.14.10
Controller Manager v1.14.2 v1.14.10
Scheduler v1.14.2 v1.14.10
Kube Proxy v1.14.2 v1.14.10
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.14.10
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 2 x v1.14.2 v1.15.7
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.14.2 v1.15.7
Controller Manager v1.14.2 v1.15.7
Scheduler v1.14.2 v1.15.7
Kube Proxy v1.14.2 v1.15.7
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.15.7
_____________________________________________________________________
k8@master:~$ sudo kubeadm upgrade apply v1.15.7
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.15.7"
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.15.7"...
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests124065540"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: 461cf48224e9b4057addb8c3f5d64870
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: 9d89927ff1a0d70cf9452b3af5827f19
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
Static pod: kube-scheduler-master hash: 7d6a1cec31a680b45724ee90bd535b49
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.7". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
k8@master:~$
k8@master:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@master:~$
k8@master:~$ sudo systemctl restart kubelet
k8@master:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2020-01-04 06:08:20 UTC; 9s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 4890 (kubelet)
Tasks: 19 (limit: 4565)
Memory: 27.6M
CGroup: /system.slice/kubelet.service
└─4890 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready,SchedulingDisabled master 215d v1.15.7
node01 Ready <none> 214d v1.14.2
k8@master:~$
uncordon the master nodes so that any new pods would be scheduled.
k8@master:~$ kubectl uncordon master
node/master uncordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7 <== cluster has been upgraded !
node01 Ready <none> 214d v1.14.2
k8@master:~$
k8@master:~$
+++++++++++++++= upgrading worker nodes +++++++++++++++
update kubeadm on the worker nodes.
k8@node01:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Get:1 http://security.ubuntu.com/ubuntu cosmic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Get:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease [88.7 kB]
Get:5 http://security.ubuntu.com/ubuntu cosmic-security/main i386 Packages [197 kB]
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:6 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease [74.6 kB]
Get:7 http://archive.ubuntu.com/ubuntu cosmic-updates/main i386 Packages [332 kB]
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Get:9 http://security.ubuntu.com/ubuntu cosmic-security/main amd64 Packages [210 kB]
Get:10 http://security.ubuntu.com/ubuntu cosmic-security/main Translation-en [84.1 kB]
Get:11 http://security.ubuntu.com/ubuntu cosmic-security/universe i386 Packages [498 kB]
Get:12 http://archive.ubuntu.com/ubuntu cosmic-updates/main amd64 Packages [345 kB]
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Get:13 http://archive.ubuntu.com/ubuntu cosmic-updates/main Translation-en [144 kB]
Get:14 http://security.ubuntu.com/ubuntu cosmic-security/universe amd64 Packages [501 kB]
Get:15 http://archive.ubuntu.com/ubuntu cosmic-updates/universe amd64 Packages [697 kB]
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [274 kB]
Get:16 http://security.ubuntu.com/ubuntu cosmic-security/universe Translation-en [144 kB]
Get:17 http://security.ubuntu.com/ubuntu cosmic-security/multiverse amd64 Packages [3,744 B]
Get:18 http://security.ubuntu.com/ubuntu cosmic-security/multiverse i386 Packages [3,904 B]
Get:19 http://archive.ubuntu.com/ubuntu cosmic-updates/universe i386 Packages [692 kB]
Get:20 http://archive.ubuntu.com/ubuntu cosmic-updates/universe Translation-en [195 kB]
Get:21 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse i386 Packages [3,904 B]
Get:22 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse amd64 Packages [3,744 B]
Get:23 http://archive.ubuntu.com/ubuntu cosmic-backports/universe i386 Packages [3,992 B]
Get:24 http://archive.ubuntu.com/ubuntu cosmic-backports/universe amd64 Packages [3,996 B]
Fetched 4,597 kB in 3s (1,349 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
cri-tools
The following packages will be upgraded:
cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 168 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 4s (4,646 kB/s)
(Reading database ... 41117 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@node01:~$
Drain worker node so that any new pods won't be scheduled, already existing nodes would be evicted to other nodes.
k8@master:~$ kubectl drain node01 --ignore-daemonsets
node/node01 cordoned
k8@master:~$
k8@node01:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
k8@node01:~$
k8@node01:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@node01:~$
k8@node01:~$ sudo systemctl restart kubelet
k8@node01:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2020-01-04 06:24:13 UTC; 4s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 19127 (kubelet)
Tasks: 14 (limit: 1134)
Memory: 31.4M
CGroup: /system.slice/kubelet.service
└─19127 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --networ
k8@node01:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7
node01 Ready,SchedulingDisabled <none> 214d v1.15.7
k8@master:~$
k8@master:~$ kubectl uncordon node01
node/node01 uncordoned
k8@master:~$
finally, we have upgraded kubernetes cluster to v1.15.7.
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7
node01 Ready <none> 214d v1.15.7
k8@master:~$
first, we shall upgrade master node and later worker nodes.
current version:
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.14.2
node01 Ready <none> 214d v1.14.2
k8@master:~$
Drain master node for maintenance so that no new pods would be scheduled on the master node.
k8@master:~$ kubectl cordon master
node/master cordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready,SchedulingDisabled master 215d v1.14.2
node01 Ready <none> 214d v1.14.2
k8@master:~$
k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:20:34Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$
sudo apt-get upgrade
sudo apt-cache policy kubeadm
you will now need to see the latest kubeadm with upgraded minor version .. I am looking at v1.15.0
k8@master:~$ sudo apt-cache policy kubeadm
kubeadm:
Installed: 1.14.2-00
Candidate: 1.17.0-00
Version table:
1.17.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.4-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.3-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.2-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.1-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.15.7-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages <== last patch version
1.15.6-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
upgrade control plane node, I would be using the latest patch version for v1.15.7
k8@master:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:3 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
cri-tools
The following packages will be upgraded:
cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 225 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 8s (2,005 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@master:~$
k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:40:15Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$
k8@master:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
I0104 06:01:59.979160 29689 version.go:248] remote version is much newer: v1.17.0; falling back to: stable-1.15
[upgrade/versions] Latest stable version: v1.15.7
[upgrade/versions] Latest version in the v1.14 series: v1.14.10
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 2 x v1.14.2 v1.14.10
Upgrade to the latest version in the v1.14 series:
COMPONENT CURRENT AVAILABLE
API Server v1.14.2 v1.14.10
Controller Manager v1.14.2 v1.14.10
Scheduler v1.14.2 v1.14.10
Kube Proxy v1.14.2 v1.14.10
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.14.10
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 2 x v1.14.2 v1.15.7
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.14.2 v1.15.7
Controller Manager v1.14.2 v1.15.7
Scheduler v1.14.2 v1.15.7
Kube Proxy v1.14.2 v1.15.7
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.15.7
_____________________________________________________________________
k8@master:~$ sudo kubeadm upgrade apply v1.15.7
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.15.7"
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.15.7"...
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests124065540"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: 461cf48224e9b4057addb8c3f5d64870
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: 9d89927ff1a0d70cf9452b3af5827f19
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
Static pod: kube-scheduler-master hash: 7d6a1cec31a680b45724ee90bd535b49
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.7". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
k8@master:~$
k8@master:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@master:~$
k8@master:~$ sudo systemctl restart kubelet
k8@master:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2020-01-04 06:08:20 UTC; 9s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 4890 (kubelet)
Tasks: 19 (limit: 4565)
Memory: 27.6M
CGroup: /system.slice/kubelet.service
└─4890 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready,SchedulingDisabled master 215d v1.15.7
node01 Ready <none> 214d v1.14.2
k8@master:~$
uncordon the master nodes so that any new pods would be scheduled.
k8@master:~$ kubectl uncordon master
node/master uncordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7 <== cluster has been upgraded !
node01 Ready <none> 214d v1.14.2
k8@master:~$
k8@master:~$
+++++++++++++++= upgrading worker nodes +++++++++++++++
update kubeadm on the worker nodes.
k8@node01:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Get:1 http://security.ubuntu.com/ubuntu cosmic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Get:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease [88.7 kB]
Get:5 http://security.ubuntu.com/ubuntu cosmic-security/main i386 Packages [197 kB]
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:6 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease [74.6 kB]
Get:7 http://archive.ubuntu.com/ubuntu cosmic-updates/main i386 Packages [332 kB]
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Get:9 http://security.ubuntu.com/ubuntu cosmic-security/main amd64 Packages [210 kB]
Get:10 http://security.ubuntu.com/ubuntu cosmic-security/main Translation-en [84.1 kB]
Get:11 http://security.ubuntu.com/ubuntu cosmic-security/universe i386 Packages [498 kB]
Get:12 http://archive.ubuntu.com/ubuntu cosmic-updates/main amd64 Packages [345 kB]
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Get:13 http://archive.ubuntu.com/ubuntu cosmic-updates/main Translation-en [144 kB]
Get:14 http://security.ubuntu.com/ubuntu cosmic-security/universe amd64 Packages [501 kB]
Get:15 http://archive.ubuntu.com/ubuntu cosmic-updates/universe amd64 Packages [697 kB]
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [274 kB]
Get:16 http://security.ubuntu.com/ubuntu cosmic-security/universe Translation-en [144 kB]
Get:17 http://security.ubuntu.com/ubuntu cosmic-security/multiverse amd64 Packages [3,744 B]
Get:18 http://security.ubuntu.com/ubuntu cosmic-security/multiverse i386 Packages [3,904 B]
Get:19 http://archive.ubuntu.com/ubuntu cosmic-updates/universe i386 Packages [692 kB]
Get:20 http://archive.ubuntu.com/ubuntu cosmic-updates/universe Translation-en [195 kB]
Get:21 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse i386 Packages [3,904 B]
Get:22 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse amd64 Packages [3,744 B]
Get:23 http://archive.ubuntu.com/ubuntu cosmic-backports/universe i386 Packages [3,992 B]
Get:24 http://archive.ubuntu.com/ubuntu cosmic-backports/universe amd64 Packages [3,996 B]
Fetched 4,597 kB in 3s (1,349 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
cri-tools
The following packages will be upgraded:
cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 168 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 4s (4,646 kB/s)
(Reading database ... 41117 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@node01:~$
Drain worker node so that any new pods won't be scheduled, already existing nodes would be evicted to other nodes.
k8@master:~$ kubectl drain node01 --ignore-daemonsets
node/node01 cordoned
k8@master:~$
k8@node01:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
k8@node01:~$
k8@node01:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@node01:~$
k8@node01:~$ sudo systemctl restart kubelet
k8@node01:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2020-01-04 06:24:13 UTC; 4s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 19127 (kubelet)
Tasks: 14 (limit: 1134)
Memory: 31.4M
CGroup: /system.slice/kubelet.service
└─19127 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --networ
k8@node01:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7
node01 Ready,SchedulingDisabled <none> 214d v1.15.7
k8@master:~$
k8@master:~$ kubectl uncordon node01
node/node01 uncordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7
node01 Ready <none> 214d v1.15.7
k8@master:~$
Trying to upgrade the kubenetes cluster, but we need to upgrade minor version to latest with increment of one
first, we shall upgrade master node and later worker nodes.
current version:
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.14.2
node01 Ready <none> 214d v1.14.2
k8@master:~$
Drain master node for maintenance so that no new pods would be scheduled on the master node.
k8@master:~$ kubectl cordon master
node/master cordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready,SchedulingDisabled master 215d v1.14.2
node01 Ready <none> 214d v1.14.2
k8@master:~$
k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:20:34Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$
sudo apt-get upgrade
sudo apt-cache policy kubeadm
you will now need to see the latest kubeadm with upgraded minor version .. I am looking at v1.15.0
k8@master:~$ sudo apt-cache policy kubeadm
kubeadm:
Installed: 1.14.2-00
Candidate: 1.17.0-00
Version table:
1.17.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.4-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.3-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.2-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.1-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.15.7-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages <== last patch version
1.15.6-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
upgrade control plane node, I would be using the latest patch version for v1.15.7
k8@master:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:3 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
cri-tools
The following packages will be upgraded:
cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 225 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 8s (2,005 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@master:~$
k8@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:40:15Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
k8@master:~$
k8@master:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
I0104 06:01:59.979160 29689 version.go:248] remote version is much newer: v1.17.0; falling back to: stable-1.15
[upgrade/versions] Latest stable version: v1.15.7
[upgrade/versions] Latest version in the v1.14 series: v1.14.10
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 2 x v1.14.2 v1.14.10
Upgrade to the latest version in the v1.14 series:
COMPONENT CURRENT AVAILABLE
API Server v1.14.2 v1.14.10
Controller Manager v1.14.2 v1.14.10
Scheduler v1.14.2 v1.14.10
Kube Proxy v1.14.2 v1.14.10
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.14.10
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 2 x v1.14.2 v1.15.7
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.14.2 v1.15.7
Controller Manager v1.14.2 v1.15.7
Scheduler v1.14.2 v1.15.7
Kube Proxy v1.14.2 v1.15.7
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.15.7
_____________________________________________________________________
k8@master:~$ sudo kubeadm upgrade apply v1.15.7
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.15.7"
[upgrade/versions] Cluster version: v1.14.2
[upgrade/versions] kubeadm version: v1.15.7
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.15.7"...
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests124065540"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: abc9138b5fe7a4d853cb54b606ef2b35
Static pod: kube-apiserver-master hash: 461cf48224e9b4057addb8c3f5d64870
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: f55e807a968b84be4948aa51916af06f
Static pod: kube-controller-manager-master hash: 9d89927ff1a0d70cf9452b3af5827f19
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-01-04-06-03-39/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-master hash: 9b290132363a92652555896288ca3f88
Static pod: kube-scheduler-master hash: 7d6a1cec31a680b45724ee90bd535b49
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.7". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
k8@master:~$
k8@master:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@master:~$
k8@master:~$ sudo systemctl restart kubelet
k8@master:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2020-01-04 06:08:20 UTC; 9s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 4890 (kubelet)
Tasks: 19 (limit: 4565)
Memory: 27.6M
CGroup: /system.slice/kubelet.service
└─4890 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready,SchedulingDisabled master 215d v1.15.7
node01 Ready <none> 214d v1.14.2
k8@master:~$
uncordon the master nodes so that any new pods would be scheduled.
k8@master:~$ kubectl uncordon master
node/master uncordoned
k8@master:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7 <== cluster has been upgraded !
node01 Ready <none> 214d v1.14.2
k8@master:~$
k8@master:~$
+++++++++++++++= upgrading worker nodes +++++++++++++++
update kubeadm on the worker nodes.
k8@node01:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.15.7-00 && sudo apt-mark hold kubeadm
kubeadm was already not hold.
Get:1 http://security.ubuntu.com/ubuntu cosmic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu cosmic InRelease
Get:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease [88.7 kB]
Get:5 http://security.ubuntu.com/ubuntu cosmic-security/main i386 Packages [197 kB]
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:6 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease [74.6 kB]
Get:7 http://archive.ubuntu.com/ubuntu cosmic-updates/main i386 Packages [332 kB]
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Get:9 http://security.ubuntu.com/ubuntu cosmic-security/main amd64 Packages [210 kB]
Get:10 http://security.ubuntu.com/ubuntu cosmic-security/main Translation-en [84.1 kB]
Get:11 http://security.ubuntu.com/ubuntu cosmic-security/universe i386 Packages [498 kB]
Get:12 http://archive.ubuntu.com/ubuntu cosmic-updates/main amd64 Packages [345 kB]
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Get:13 http://archive.ubuntu.com/ubuntu cosmic-updates/main Translation-en [144 kB]
Get:14 http://security.ubuntu.com/ubuntu cosmic-security/universe amd64 Packages [501 kB]
Get:15 http://archive.ubuntu.com/ubuntu cosmic-updates/universe amd64 Packages [697 kB]
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [274 kB]
Get:16 http://security.ubuntu.com/ubuntu cosmic-security/universe Translation-en [144 kB]
Get:17 http://security.ubuntu.com/ubuntu cosmic-security/multiverse amd64 Packages [3,744 B]
Get:18 http://security.ubuntu.com/ubuntu cosmic-security/multiverse i386 Packages [3,904 B]
Get:19 http://archive.ubuntu.com/ubuntu cosmic-updates/universe i386 Packages [692 kB]
Get:20 http://archive.ubuntu.com/ubuntu cosmic-updates/universe Translation-en [195 kB]
Get:21 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse i386 Packages [3,904 B]
Get:22 http://archive.ubuntu.com/ubuntu cosmic-updates/multiverse amd64 Packages [3,744 B]
Get:23 http://archive.ubuntu.com/ubuntu cosmic-backports/universe i386 Packages [3,992 B]
Get:24 http://archive.ubuntu.com/ubuntu cosmic-backports/universe amd64 Packages [3,996 B]
Fetched 4,597 kB in 3s (1,349 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
cri-tools
The following packages will be upgraded:
cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 168 not upgraded.
Need to get 17.0 MB of archives.
After this operation, 1,663 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-00 [8,776 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.15.7-00 [8,253 kB]
Fetched 17.0 MB in 4s (4,646 kB/s)
(Reading database ... 41117 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.13.0-00_amd64.deb ...
Unpacking cri-tools (1.13.0-00) over (1.12.0-00) ...
Preparing to unpack .../kubeadm_1.15.7-00_amd64.deb ...
Unpacking kubeadm (1.15.7-00) over (1.14.2-00) ...
Setting up cri-tools (1.13.0-00) ...
Setting up kubeadm (1.15.7-00) ...
kubeadm set on hold.
k8@node01:~$
Drain worker node so that any new pods won't be scheduled, already existing nodes would be evicted to other nodes.
k8@master:~$ kubectl drain node01 --ignore-daemonsets
node/node01 cordoned
k8@master:~$
k8@node01:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
k8@node01:~$
k8@node01:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.15.7-00 kubectl=1.15.7-00 && sudo apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Hit:1 http://archive.ubuntu.com/ubuntu cosmic InRelease
Hit:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 224 not upgraded.
Need to get 29.0 MB of archives.
After this operation, 8,398 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.15.7-00 [8,760 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.15.7-00 [20.3 MB]
Fetched 29.0 MB in 10s (3,033 kB/s)
(Reading database ... 113172 files and directories currently installed.)
Preparing to unpack .../kubectl_1.15.7-00_amd64.deb ...
Unpacking kubectl (1.15.7-00) over (1.14.2-00) ...
Preparing to unpack .../kubelet_1.15.7-00_amd64.deb ...
Unpacking kubelet (1.15.7-00) over (1.14.2-00) ...
Setting up kubelet (1.15.7-00) ...
Setting up kubectl (1.15.7-00) ...
kubelet set on hold.
kubectl set on hold.
k8@node01:~$
k8@node01:~$ sudo systemctl restart kubelet
k8@node01:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2020-01-04 06:24:13 UTC; 4s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 19127 (kubelet)
Tasks: 14 (limit: 1134)
Memory: 31.4M
CGroup: /system.slice/kubelet.service
└─19127 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --networ
k8@node01:~$
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7
node01 Ready,SchedulingDisabled <none> 214d v1.15.7
k8@master:~$
k8@master:~$ kubectl uncordon node01
node/node01 uncordoned
k8@master:~$
finally, we have upgraded kubernetes cluster to v1.15.7.
k8@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 215d v1.15.7
node01 Ready <none> 214d v1.15.7
k8@master:~$
k8@master:~$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 1/1 1 1 63s
k8@master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-7bb7cd8db5-mrzk4 1/1 Running 0 69s 192.168.177.232 knode <none> <none>
k8@master:~$
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 1/1 1 1 63s
k8@master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-7bb7cd8db5-mrzk4 1/1 Running 0 69s 192.168.177.232 knode <none> <none>
k8@master:~$
No comments:
Post a Comment