How to Enable Debug of Fluentd Daemonset in K8S

Requirement:

   We have many K8S Pods running,  like we have fluentd pod running on each worker node as Daemonset. We need to enable debug mode of fluentd to get more information in the logs.
   We have same requirements for all other applications running in the Pods.  As long as the application accept the parameters to enable debug mode, or put more trace into log files, we should be able to enable it in K8S pods

Solution:

• First we need to find what parameters we can pass to application to enable debug. In fluentd, there are  -v  and -vv  2 parameters to enable debug information output of the fluentd. Please refer fluend office website . 
• We need to get yaml file of the daemonset from kubectl. Same concept if it is deployment or statefulsets. 

kubectl get pod -n <namespace>  <daemonset name>  -o yaml > /tmp/temp.yaml

• Edit this temp.yaml file and find the section which passes parameters to fluentd. In fluentd it is like

 - name: FLUENTD_ARGS

          value: --no-supervisor -q

• Update -q to to be -v  or -vv , like

 - name: FLUENTD_ARGS

          value: --no-supervisor -vv

• Save the temp.yaml and apply it

kubectl apply -f /tmp/temp.yaml

• It won't be effective right away. The config will be stored in Etcd data store. When you delete pods, the new pods will read the latest config and start pods with -vv  parameters .
• Then we can use kubectl logs -n devops <pod name>  see debug infor of the pods.