Greeting,
My query is pending since long time and I requested multiple times in SLACK channel as well but so far my issue (configure cluster in VM) is not yet fixed, need your expertise assistance to fix.
I am unable to configure kubernets cluster in my local VM (1 master and 2 worker). I am flowing steps mentioned in course " Kubernetes for the Absolute Beginners - Hands-on" and also I tried some technotes from google and requested couple of time over SLACK channel as well but so far my cluster is not stable.
POD coredns and weave-net-gvtgs are showing in ContainerCreating/CrashLoopBackOff, could you please suggest me the best way to configure cluster in VM.
root@kubemaster:~# kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
coredns-6955765f44-kbxwp 0/1 ContainerCreating 0 32d
coredns-6955765f44-wp7rf 0/1 ContainerCreating 0 32d
etcd-kubemaster 1/1 Running 2 35d
kube-apiserver-kubemaster 1/1 Running 2 35d
kube-controller-manager-kubemaster 1/1 Running 2 35d
kube-proxy-4q9zl 1/1 Running 2 35d
kube-proxy-4tkd8 1/1 Running 3 35d
kube-proxy-vznx2 1/1 Running 3 35d
kube-scheduler-kubemaster 1/1 Running 2 35d
weave-net-8949l 2/2 Running 6 35d
weave-net-gvtgs 1/2 CrashLoopBackOff 89 35d
weave-net-q8f8k 1/2 CrashLoopBackOff 95 35d
root@kubemaster:~#
Your response is highly awaited.
Thank you,
-Subhash
Hi Subhash, all you weave plugins are in a crashloopbackoff state. Can you check if you are able to see the logs of any of these pods: use - kubectl -n kube-namespace logs -f and post the result.
Have you set up networking correctly between the nodes in the virtual box?
Hi Vijin,
Please find the requested details:
root@kubemaster:~# kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
coredns-6955765f44-kbxwp 0/1 ContainerCreating 0 34d
coredns-6955765f44-wp7rf 0/1 ContainerCreating 0 34d
etcd-kubemaster 1/1 Running 3 36d
kube-apiserver-kubemaster 1/1 Running 3 36d
kube-controller-manager-kubemaster 1/1 Running 3 36d
kube-proxy-4q9zl 1/1 Running 3 36d
kube-proxy-4tkd8 1/1 Running 4 36d
kube-proxy-vznx2 1/1 Running 4 36d
kube-scheduler-kubemaster 1/1 Running 3 36d
weave-net-8949l 2/2 Running 9 36d
weave-net-gvtgs 1/2 CrashLoopBackOff 105 36d
weave-net-q8f8k 1/2 CrashLoopBackOff 111 36d
root@kubemaster:~# kubectl -n kube-system logs -f coredns-6955765f44-kbxwp
Error from server (BadRequest): container “coredns” in pod “coredns-6955765f44-kbxwp” is waiting to start: ContainerCreating
root@kubemaster:~# kubectl -n kube-system logs -f coredns-6955765f44-wp7rf
Error from server (BadRequest): container “coredns” in pod “coredns-6955765f44-wp7rf” is waiting to start: ContainerCreating
root@kubemaster:~# kubectl -n kube-system logs -f weave-net-gvtgs
error: a container name must be specified for pod weave-net-gvtgs, choose one of: [weave weave-npc]
root@kubemaster:~# kubectl -n kube-system logs -f weave-net-q8f8k
error: a container name must be specified for pod weave-net-q8f8k, choose one of: [weave weave-npc]
root@kubemaster:~#
I was following absolute beginner course in Kode Kloud but if you want I setup it from the beginning.
My requirement is to have cluster running in my local system’s VM.
Thank you,
-Subhash
There are two containers in the weave net pod, so specify the “eave” container when checking the logs with the -c option.
Without the logs its almost impossible to understand where the issue is.
You can try setting up from the beginning if you want, I suspect the issue is with networking between the nodes but it could be a number of different things.
Hi Vijin,
Please find the log output:
root@kubemaster:~# kubectl -n kube-system logs -f weave-net-q8f8k -c weave
FATA: 2020/01/26 01:23:36.954821 [kube-peers] Could not get peers: Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout
Failed to get peers
root@kubemaster:~# kubectl -n kube-system logs -f weave-net-gvtgs -c weave
FATA: 2020/01/26 01:23:36.158676 [kube-peers] Could not get peers: Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout
Failed to get peers
root@kubemaster:~#
Thank you.
Hi VIjin,
I reinstalled cluster from scratch and still core-DNS is getting failed.
root@kubemaster:~# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5c45f5bd9f-msq7x 1/1 Running 0 11m
kube-system calico-node-mzfwp 1/1 Running 0 11m
kube-system coredns-6955765f44-csffd 0/1 CrashLoopBackOff 6 13m
kube-system coredns-6955765f44-vsn9k 0/1 CrashLoopBackOff 6 13m
kube-system etcd-kubemaster 1/1 Running 0 13m
kube-system kube-apiserver-kubemaster 1/1 Running 0 13m
kube-system kube-controller-manager-kubemaster 1/1 Running 0 13m
kube-system kube-proxy-nzmhr 1/1 Running 0 13m
kube-system kube-scheduler-kubemaster 1/1 Running 0 13m
root@kubemaster:~# kubectl describe pod coredns-6955765f44-csffd -n kube-system
Name: coredns-6955765f44-csffd
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: kubemaster/192.168.1.17
Start Time: Sun, 26 Jan 2020 13:30:23 -0500
Labels: k8s-app=kube-dns
pod-template-hash=6955765f44
Annotations: cni.projectcalico.org/podIP: 192.168.141.2/32
Status: Running
IP: 192.168.141.2
<<>
Events:
Type Reason Age From Message
Warning FailedScheduling 11m (x25 over 13m) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn’t tolerate.
Warning FailedCreatePodSandBox 11m kubelet, kubemaster Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “ff94823c45cbebd7083ec3ff1e6236a752d98bf11ee52cddfad4e45a42f43b6d” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 11m kubelet, kubemaster Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “c4947e526f206dc4ba81871addeaaf18e3f45c51388ea0ee7b1aa717c4f4c8ff” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 10m kubelet, kubemaster Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “56b078118672a9bcbc8a625cee24aabf92894c8acd35c9915aaefcacdfd9573a” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 10m kubelet, kubemaster Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “16ba2be58d8d30a410279caa91c786b625e0359bcd20cb60953fc7bffb040b2d” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 10m kubelet, kubemaster Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “3bb06ce1da7c50ec67bba8075b1a0fc63b28ce4bc3f24fe752c1dd7fc7d88a3f” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 10m kubelet, kubemaster Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “2c68d5c731e6a1f88da509e4829b552b2cd93a23673e5ecbfee29e09f6a6023c” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 10m kubelet, kubemaster Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “9225f2f16d5439274848126ca4bd8669bb7ee11b9cb79c4cb465d20a34889726” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 10m kubelet, kubemaster Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “1a69cec6b623e879329b9087553a9597e3dc195ed7b702c872b5adfaa911dbd6” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 10m kubelet, kubemaster Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “8c8a1f5c2074bdb9daef1b32e010dbd461d59076a6992bced8f1d28e328cbe89” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Normal SandboxChanged 10m (x12 over 11m) kubelet, kubemaster Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 10m (x4 over 10m) kubelet, kubemaster (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “5978e68f4ead2d1204cbd36db0c7e6a868991ab455e5146648265551f1342f18” network for pod “coredns-6955765f44-csffd”: networkPlugin cni failed to set up pod “coredns-6955765f44-csffd_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning BackOff 60s (x48 over 10m) kubelet, kubemaster Back-off restarting failed container
root@kubemaster:~#
root@kubemaster:~# kubectl -n kube-system logs coredns-6955765f44-csffd
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.5
linux/amd64, go1.13.4, c2fd1b2
[FATAL] plugin/loop: Loop (127.0.0.1:38593 → :53) detected for zone “.”, see loop. Query: “HINFO 5761231769039660643.4999974427986932100.”
root@kubemaster:~#
Please close this, I am able to fix it.