In lightning lab, I got all correct except the cluster upgrade.. Not able to upg . . .

Kubernetes
#1

Mayur Sharma:
In lightning lab, I got all correct except the cluster upgrade… Not able to upgrade the worker node, as kubeadm version 1.19.4 is not found on worker node.
here are the steps I performed,

When I run following to check available version, I see the latest version available is 1.19.4-00

apt update
apt-cache madison kubeadm
  1. Upgraded master’s kubeadm to 1.19.
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.19.4-00 && \
apt-mark hold kubeadm
  1. Here I got the next available version is 1.19.11
sudo kubeadm upgrade plan
  1. I stick to 1.19.4-00 and upgraded the master node
kubeadm upgrade apply v1.19.4-00
  1. drain the master node
sudo kubeadm upgrade node
kubectl drain controlplane --ignore-daemonsets
  1. Upgraded the kubectl and kubelet to 1.19.4 version on master node
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.19.4-00 kubectl=1.19.4-00 && \
apt-mark hold kubelet kubectl
  1. Restarted kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet
  1. Now, the turn is of worker node. Here the kubeadm version is not found.
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.19.4-00 && \
apt-mark hold kubeadm

Output of above command on worker node:
Hit:9 <http://security.ubuntu.com/ubuntu> bionic-security InRelease                                                                  
Hit:10 <http://us.archive.ubuntu.com/ubuntu> bionic-updates InRelease                                          
Hit:11 <http://us.archive.ubuntu.com/ubuntu> bionic-backports InRelease                                        
Err:3 <https://dl.yarnpkg.com/debian> stable InRelease                     
  The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging &lt;[email protected]&gt;
Get:6 <https://packages.cloud.google.com/apt> kubernetes-xenial InRelease [9,383 B]
Err:6 <https://packages.cloud.google.com/apt> kubernetes-xenial InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FEEA9169307EA071 NO_PUBKEY 8B57C5C2836F4BEB
Fetched 17.1 kB in 1s (15.1 kB/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: <https://dl.yarnpkg.com/debian> stable InRelease: The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging &lt;[email protected]&gt;
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: <https://packages.cloud.google.com/apt> kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FEEA9169307EA071 NO_PUBKEY 8B57C5C2836F4BEB
W: Failed to fetch <http://apt.kubernetes.io/dists/kubernetes-xenial/InRelease>  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FEEA9169307EA071 NO_PUBKEY 8B57C5C2836F4BEB
W: Failed to fetch <https://dl.yarnpkg.com/debian/dists/stable/InRelease>  The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging &lt;[email protected]&gt;
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Version '1.19.4-00' for 'kubeadm' was not found
#2

unnivkn:
Hi Step:1 Line-2 should be: 1.19.0-00 apt-get update && apt-get install -y kubeadm=1.19.0-00 &&

#3

unnivkn:
hop you are upgrading from 1.18.0 to 1.19.0

#4

unnivkn:
you have to correct the version in step-5 as well

#5

unnivkn:
same case with step-7

#6

Mayur Sharma:
@unnivkn: okay, so you meant kubeadm version should be upgraded to 1.19.0 not 1.19.4.
Few more follow-up doubts,

  1. why we cannot choose 1.19.4 as it is also available .
  2. kubeadm was successfully upgraded to 1.19.4 on master, but why not on worker node?
    Following commands are interlinked, so I believe if I got the kubeadm version right, then I can apply other accordingly.
#7

unnivkn:
what is mentioned in the question…you have to follow that

#8

unnivkn:
can you paste the question here ?

#9

Mayur Sharma:
sure, let me restart the lab … give me few minutes

#10

Mayur Sharma:
@unnivkn, here is the question

Upgrade the current version of kubernetes from 1.18 to 1.19.0 exactly using the kubeadm utility. Make sure that the upgrade is carried out one node at a time starting with the master node. To minimize downtime, the deployment gold-nginx should be rescheduled on an alternate node before upgrading each node.

it mention exactly 1.19.0, that explains I should choose 1.19.0 and then only it accepts the answer.

But not related to exam question, if we have latest verison of kubeadm like here we have 1.19.4, why we should not choose this?

#11

unnivkn:
if the ask is for upgrade higher version…(say 18 to 19) then the cluster won’t be having 19.1 or higher version. that won’t make sense

#12

unnivkn:
check what version you have now. kget nodes

#13

Mayur Sharma:
As per the ask, we have upgraded and new version is upgraded as expected to 11.19.0

Think of a real life cluster, if we have latest version available, why we cannot choose that.
My doubt is around , why it did not work if I upgraded to 1.19.4 (though I fail in that question)

That too the upgrade failed on worker node, not on master node with the error v.1.19.4 is not found

#14

unnivkn:
you mean you already have 1.19.0 & want to upgrade to 1.19.4 ?

#15

Mayur Sharma:
No, the env had v.18.0 initially. I unknowingly chose the latest version available, i.e. 1.19.4

Was successful on master and it was visible in k get nodes as well.
But failed on worker node with error v.1.19.4 is not found

#16

unnivkn:
ok…in that case in master, your 3rd step is wrong. see this: kubeadm upgrade apply v1.19.0 . Keep in mind master & worker version should be in sync

#17

unnivkn:
its looks totally mess up for me. just follow below steps:

#18

unnivkn:
##master node upgrade
Note: replace x with 0 & use sudo before every command

k get nodes

apt update
apt-cache madison kubeadm

apt-get update &&
apt-get install -y --allow-change-held-packages kubeadm=1.19.x-00

kubeadm version

sudo kubeadm upgrade plan

kubeadm upgrade apply v1.19.0

kubectl drain <cp-node-name> --ignore-daemonsets

apt-get update &&
apt-get install -y --allow-change-held-packages kubelet=1.19.x-00 kubectl=1.19.x-00

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon <cp-node-name>

#19

unnivkn:
###Upgrade worker nodes

##from master:
kubectl drain <node-to-drain> --ignore-daemonsets

ssh node01

apt-get update &&
apt-get install -y --allow-change-held-packages kubeadm=1.19.x-00

sudo kubeadm upgrade node

apt-get update &&
apt-get install -y --allow-change-held-packages kubelet=1.19.x-00 kubectl=1.19.x-00

sudo systemctl daemon-reload
sudo systemctl restart kubelet

##from master
kubectl uncordon <node-to-drain>
kubectl get nodes

#20

Mayur Sharma:
Sorry, that was a typo – I run the below command only

kubeadm upgrade apply v1.19.4

will update my question also.

Also, will run the upgrade again from v1.18.0 to v1.19.4-00, will check