Anika Rathi:
This is regarding “Multiple Scheduler”. I have two issues - 1. Once I make the changes in kube-scheduler yaml and apply it. The scheduler “my-scheduler” takes almost 2 minutes to be in RUNNING state. 2. The nginx pod is NEVER scheduled on the my-scheduler though I’ve passed the schedulerName: my-scheduler property in the definition file. What am I missing?
cp /etc/kubernetes/manifests/kube-scheduler.yaml /root/my-scheduler.yaml
controlplane $ vi /root/my-scheduler.yaml
controlplane $ diff /root/my-scheduler.yaml /etc/kubernetes/manifests/kube-scheduler.yaml
18,20c18,19
< - --leader-elect=false
< - --scheduler-name=my-scheduler
< - --port=10280
---
> - --leader-elect=true
> - --port=0
28c27
< port: 10280
---
> port: 10259
42c41
< port: 10280
---
> port: 10259
Nicolas G.:
you also need to add
- --port=10280
- --secure-port= # different from port above
and update all port: bellow that to be the same as --secure-port
Anika Rathi:
@Nicolas G. Thanks for the suggestion but it didn’t work.
diff /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/my-scheduler.yaml
6c6
< component: kube-scheduler
---
> component: my-scheduler
8c8
< name: kube-scheduler
---
> name: my-scheduler
18,19c18,21
< - --leader-elect=true
< - --port=0
---
> - --leader-elect=false
> - --scheduler-name=my-scheduler
> - --port=10208
> - --secure-port=10288
27c29
< port: 10259
---
> port: 10288
32c34
< name: kube-scheduler
---
> name: my-scheduler
41c43
< port: 10259
---
> port: 10288
Nicolas G.:
@Anika Rathi you don’t need to change the component , also make sure there is nothing else running on ports 10208 and 10288
netstat -tlpn | egrep "10208|10288"
Nicolas G.:
the ports you select must be free so they can be used
Anika Rathi:
Yes, the ports are free
controlplane $ netstat -tlpn | egrep "10208|10288"
controlplane $
controlplane $
Changing component name back to kube-scheduler didn’t help either.
Anika Rathi:
Let me know if you need more information.
Gonzalo Acosta:
in my case working well with this configuration of static pod.
Gonzalo Acosta:
vagrant@kmaster:~/cka$ sudo cat /etc/kubernetes/manifests/my-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-scheduler
tier: control-plane
name: my-scheduler
namespace: kube-system
spec:
containers:
- command:
- kube-scheduler
- --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
- --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
- --bind-address=127.0.0.1
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=true
- --port=10282
- --secure-port=0
- --scheduler-name=my-scheduler
image: <http://k8s.gcr.io/kube-scheduler:v1.20.2|k8s.gcr.io/kube-scheduler:v1.20.2>
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10282
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: kube-scheduler
resources:
requests:
cpu: 100m
startupProbe:
failureThreshold: 24
httpGet:
host: 127.0.0.1
path: /healthz
port: 10282
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/kubernetes/scheduler.conf
name: kubeconfig
readOnly: true
hostNetwork: true
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/kubernetes/scheduler.conf
type: FileOrCreate
name: kubeconfig
status: {}
vagrant@kmaster:~/cka$ k get pods -n kube-system my-scheduler-kmaster
NAME READY STATUS RESTARTS AGE
my-scheduler-kmaster 1/1 Running 0 6m15s
vagrant@kmaster:~/cka$
Gonzalo Acosta:
Scheduling pod ok
vagrant@kmaster:~/cka$ k describe pod podsche | tail -10
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 114s my-scheduler Successfully assigned cka/podsche to kworker2
Normal Pulling 113s kubelet Pulling image "nginx"
Normal Pulled 111s kubelet Successfully pulled image "nginx" in 2.150946176s
Normal Created 111s kubelet Created container podsche
Normal Started 111s kubelet Started container podsche
vagrant@kmaster:~/cka$
in my case only for test I disabled the HTTPS and change the liveness and readiness probes.
Anika Rathi:
Yes, the yaml given for /var/answers works fine. But it has many more changes than only the two changes that I’ve done (port, and scheduler-name). I wanted this one to work.
Anika Rathi:
cat /tmp/my-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-scheduler
tier: control-plane
name: my-scheduler
namespace: kube-system
spec:
containers:
- command:
- kube-scheduler
- --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
- --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
- --bind-address=127.0.0.1
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=false
- --scheduler-name=my-scheduler
- --port=10208
- --secure-port=10280
image: <http://k8s.gcr.io/kube-scheduler:v1.19.0|k8s.gcr.io/kube-scheduler:v1.19.0>
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10280
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: my-scheduler
resources:
requests:
cpu: 100m
startupProbe:
failureThreshold: 24
httpGet:
host: 127.0.0.1
path: /healthz
port: 10280
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/kubernetes/scheduler.conf
name: kubeconfig
readOnly: true
hostNetwork: true
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/kubernetes/scheduler.conf
type: FileOrCreate
name: kubeconfig
status: {}
Anika Rathi:
And now it is haunting me… 
Nicolas G.:
This is what I do when I copy the existing scheduler config and it always work:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-scheduler
tier: control-plane
name: my-scheduler # change
namespace: kube-system
spec:
containers:
- command:
- kube-scheduler
- --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
- --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
- --bind-address=127.0.0.1
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=false # change
- --scheduler-name=my-scheduler # add
- --port=12345 # change
- --secure-port=12346 # add
image: <http://k8s.gcr.io/kube-scheduler:v1.19.1|k8s.gcr.io/kube-scheduler:v1.19.1>
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 12346 # change
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: kube-scheduler
resources:
requests:
cpu: 100m
startupProbe:
failureThreshold: 24
httpGet:
host: 127.0.0.1
path: /healthz
port: 12346 # change
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
Nicolas G.:
if you are still having issues, check what the errors are by looking at your scheduler docker logs (docker logs XYZID)
Anika Rathi:
Still the same behavior. It takes about 2 min to come up. And the new pod is not scheduled on it. I’ve made the suggested changes. Below are the logs.
kubectl logs -f my-scheduler -n kube-system
I0129 02:43:13.217187 1 registry.go:173] Registering SelectorSpread plugin
I0129 02:43:13.217233 1 registry.go:173] Registering SelectorSpread plugin
I0129 02:43:13.611180 1 serving.go:331] Generated self-signed cert in-memory
I0129 02:43:14.505366 1 registry.go:173] Registering SelectorSpread plugin
I0129 02:43:14.505389 1 registry.go:173] Registering SelectorSpread plugin
W0129 02:43:14.509557 1 authorization.go:47] Authorization is disabled
W0129 02:43:14.509706 1 authentication.go:40] Authentication is disabled
I0129 02:43:14.509775 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:12345
I0129 02:43:14.514355 1 secure_serving.go:197] Serving securely on 127.0.0.1:12346
I0129 02:43:14.514576 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0129 02:43:14.514688 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0129 02:43:14.514705 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0129 02:43:14.514728 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0129 02:43:14.514736 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0129 02:43:14.514583 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0129 02:43:14.518518 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0129 02:43:14.614882 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0129 02:43:14.615013 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0129 02:43:14.618736 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
Anika Rathi:
kubectl get events | grep -i schedule
5m55s Normal Scheduled pod/nginx Successfully assigned default/nginx to node01
Anika Rathi:
Let me know if you need more information.