Tip: A few commands to debug Issues with Kubelet

sudo systemctl status -l kubelet
kubectl describe node <name>
sudo journalctl -u kubelet | grep ready
sudo systemctl restart docker

kubectl cluster-info dump  --- to detailed cluster info

Tip: Impersonate users on kubectl

We can impersonate users with the --as= and the --as-group= flags.

kubectl auth can-i create pods --as=me

Tip: remove linux files with special characters

ls -ltr

-rw-rw-r-- 1 henryxie henryxie    0 Apr 22 12:14 --header

-rw-rw-r-- 1 henryxie henryxie    0 Apr 22 12:14 -d

 To remove these 2 files

rm -v -- "-d"

rm -v -- "--header"

Tip: Pods are not created while deployment is created

Symptom:

  We have a normal deployment which was working fine. When we test it on a new Kubernetes cluster, the deployment is created well, but the pod is not created. No warning or error messages.

 "kubectl describe deployment"  does not show any hints. Pod security policy check is good, RBAC privilege check is good.

OldReplicaSets:    <none>

NewReplicaSet:     livesqlstg-admin-678df959b4 (0/1 replicas created)

Events:

  Type    Reason             Age   From                   Message

  ----    ------             ----  ----                   -------

  Normal  ScalingReplicaSet  16s   deployment-controller  Scaled up replica set livesqlstg-admin-678df959b4 to 1

Solution:

  The reason is we have resource quota implemented on the namespace. 

 spec:

  hard:

    configmaps: "10"

    limits.cpu: "10"

    limits.memory: 20Gi

    persistentvolumeclaims: "10"

    ....

By doing that, we need an additional resource section in the deployment yaml file.  ie

      resources:

              requests:

                  memory: "10Gi"

                  cpu: "1"

              limits:

                  memory: "10Gi"

                  cpu: "1"

 It would be good for Kubernetes to give users some warnings for that.