Sunday, May 12, 2019

Tip to Get Output From A Command Running Inside Pod

Symptom:

  We write a client-go program to run a simple command ie pwd inside a pod.  It runs fine, No error  but I don't see pwd output to the standard OS output. Partial sample code is like

func ExecPodCmd(o *KubeOperations,Podname,Podnamespace string,SimpleCommand []string) error {
       SimpleCommand := []string{"pwd"}
execReq := o.clientset.CoreV1().RESTClient().Post().
    Resource("pods").
Name(Podname).
Namespace(Podnamespace).
SubResource("exec")
    execReq.VersionedParams(&corev1.PodExecOptions{
Command:   PsqlCommand,
Stdin:     true,
Stdout:    true,
Stderr:    true,
}, scheme.ParameterCodec)
exec, err := remotecommand.NewSPDYExecutor(o.restConfig, "POST", execReq.URL())
if err != nil {
return fmt.Errorf("error while creating Executor: %v", err)
}
err = exec.Stream(remotecommand.StreamOptions{
Stdin:  os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
Tty:    false,
})
if err != nil {
return fmt.Errorf("error in Stream: %v", err)
} else {
return nil
}
}

Solution:

It turns out we need to use bash -c to run the simplecommand to get proper os output.
Replace   SimpleCommand := []string{"pwd"} with SimpleCommand := []string{"/bin/sh", "-c", "pwd"}

Error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr

Symptom:

  We would like to run a simple comand ie pwd in a pod via client-go.  It error on below part of code:
err = exec.Stream(remotecommand.StreamOptions{
Stdin:  stdin,
Stdout: stdout,
Stderr: stderr,
Tty:    false,
})

Solution:

  We can simply to use os.Stdin os.Stdout os.Stderr to get the comand output from pod to our standard os output.  Correct code is like:
err = exec.Stream(remotecommand.StreamOptions{
Stdin:  os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
Tty:    false,
})

Friday, May 10, 2019

Error: cannot use "k8s.io/api/core/v1".ConfigMapVolumeSource literal (type "k8s.io/api/core/v1".ConfigMapVolumeSource) as type *"k8s.io/api/core/v1".ConfigMapVolumeSource in field value

Symptom:

Related code:
o.pgreplicamaster.Spec.Template.Spec.Volumes = []corev1.Volume{
{Name: "pgreplica-config",
VolumeSource: corev1.VolumeSource{ 
   ConfigMap: corev1.ConfigMapVolumeSource{
    LocalObjectReference: corev1.LocalObjectReference{Name: o.UserSpecifiedCM},
   },
   },
    
},
}
 Compile Error
cannot use "k8s.io/api/core/v1".ConfigMapVolumeSource literal (type "k8s.io/api/core/v1".ConfigMapVolumeSource) as type *"k8s.io/api/core/v1".ConfigMapVolumeSource in field value

Solution:

    Refer VolumeSource k8s doc. ConfigMap suppose to have pointer  *ConfigMapVolumeSource, not value of  ConfigMapVolumeSource , We need to add &.
Meanwhile we can't mix field value and value , refer stackflow link
Correct code is:
o.pgreplicamaster.Spec.Template.Spec.Volumes = []corev1.Volume{
{Name: "pgreplica-config",
 VolumeSource: corev1.VolumeSource{ 
   ConfigMap: &corev1.ConfigMapVolumeSource{
    LocalObjectReference: corev1.LocalObjectReference{Name: o.UserSpecifiedCM},
   },
   },
 
},
}

Error: spec.volumes[1].configMap.name: Required value, spec.containers[0].volumeMounts[1].name: Not found

Symptom:

  We use partial yaml below to create a statefulset with mount a volume for configmap.It is successful
apiVersion: apps/v1
kind: StatefulSet
......  
    spec:
      volumes:
        - name: pgreplica-config
          configMap:
            name: pgconfigmap
........
When we try to use client-go to do the same thing.
Related codes
o.pgreplicamaster.Spec.Template.Spec.Volumes = []corev1.Volume{
{Name: "pgreplica-config",
 VolumeSource: corev1.VolumeSource{ 
   ConfigMap: &corev1.ConfigMapVolumeSource{
    Items: corev1.KeyToPath{{Key: "name",Path: pgconfigmap}},
   },
   },
    
},
}
We hit error
Error: spec.volumes[1].configMap.name: Required value, spec.containers[0].volumeMounts[1].name: Not found

Solution

Refer k8s doc of ConfigMapVolumeSource
There is LocalObjectReference which we should use, instead of using Items
Update code as below to make it work
o.pgreplicamaster.Spec.Template.Spec.Volumes = []corev1.Volume{
{Name: "pgreplica-config",
VolumeSource: corev1.VolumeSource{
   ConfigMap: &corev1.ConfigMapVolumeSource{
    LocalObjectReference: corev1.LocalObjectReference{Name: "pgconfigmap"},
   },
   },
   
},
}

Example of Updating ConfigMap name in Client-go

Option 1:

o.pgreplicamaster.Spec.Template.Spec.Volumes[0].VolumeSource.ConfigMap.LocalObjectReference = corev1.LocalObjectReference{Name: o.UserSpecifiedCM}

is doing the same thing as

Option 2:

o.pgreplicamaster.Spec.Template.Spec.Volumes = []corev1.Volume{
{Name: "pgreplica-config",
VolumeSource: corev1.VolumeSource{
   ConfigMap: &corev1.ConfigMapVolumeSource{
    LocalObjectReference: corev1.LocalObjectReference{Name: o.UserSpecifiedCM},
   },
   },
 
},
}

fmt.Printf("%#v\n",o.pgreplicamaster.Spec.Template.Spec.Volumes[0].VolumeSource.ConfigMap.LocalObjectReference)

Thursday, May 09, 2019

Go Exercise: How To use yaml Unmarshal to convert yaml into k8s client go data struct

please see code on github link

Go Exercise: How To use decode yaml into k8s client go data struct

Please see code on github link

Some Packages Match Between Ubuntu and Centos

Symptom:

We have requirements to convert docker images based on ubuntu to centos.
There are quite a few packages we need to match ubuntu from centos
After testing, below are the details of we have so far

Solution:

Ubuntu       ---> Centos

apt-get        ---> yum
python-dev ---> python-devel
apt-get install  build-essential ---> yum groupinstall 'Development Tools'
libfreetype6-dev ---> freetype-devel
libpng-dev          ---> libpng-devel
libpq-dev            ---> postgresql-devel
apache2              ---> httpd
libapache2-mod-wsgi  ---> mod_wsgi

Monday, May 06, 2019

SSH via Proxy Socat Connect Tips

Use SOCAT in Linux

ssh -oIdentityFile=VM-PrivateKey.txt -o ServerAliveInterval=5 -o ProxyCommand='socat - "proxy:<proxy server>:%h:%p,proxyport=80"' opc@<hostname or ip address>

Use CONNECT in Git Bash

ssh -oIdentityFile=VM-PrivateKey.txt -o ServerAliveInterval=5 -o ProxyCommand="connect  -H  <proxy server>:80  %h %p"  opc@<hostname or ip address>

Saturday, May 04, 2019

Error: net/http: TLS handshake timeout via kubectl

Symptom:

  When we try to use kubectl logs <pod> or kubectl exec it  <pod> /bin/bash ....etc , we get below error:
.........  net/http: TLS handshake timeout.

While TLS certificates are valid and kubectl get nodes, kubectl cluster-info are working fine

Solution:

Use -v=8 flag to enable more details kubectl rest API call details
We found such HTTP 500 error when kubectl contacts API masterserver

GET https://Your-Master-node:6443/api/v1/namespaces/default/pods/test-deployment-6669d6df59-vdnk5/log
I0424 04:47:05.882800   11526 round_trippers.go:408] Response Status: 500 Internal Server Error in 10100 milliseconds
..
I0424 05:28:52.001101   21195 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get https://10.0.64.2:10250/containerLogs/default/test-deployment-6669d6df59-vdnk5/django: net/http: TLS handshake timeout","code":500}

10.0.64.2 is the private ip of the Node and 10250 is the listening port of kubelet
It turns out TLS error is on kubelet side of the node though TLS certificates are valid
kubectl get nodes ,kubectl cluster-info are fine as apiserver don't need to contact kubelet while kubectl logs needs apiserver to contact kubelet
It could be potential a bug.  We upgrade k8s of worker node to fix it.
Similar github issue link


Go Excercise: Cobra Example

Please see code on github link

Friday, May 03, 2019

Error: cannot load k8s.io/client-go/pkg/api": cannot find module providing package k8s.io/client-go/pkg/api

Symptom:

  When we use client-go api ,there is line of code :
api.Codecs.UniversalDeserializer()
We often import "k8s.io/client-go/pkg/api". It was working. Then suddently we hit this error
Error: cannot load k8s.io/client-go/pkg/api": cannot find module providing package k8s.io/client-go/pkg/api

Solution:

  The reason of this error is due to code baseline has moved to a new location
Instead of import "k8s.io/client-go/pkg/api"
we should use
import "k8s.io/client-go/kubernetes/scheme"

Then change code to use scheme instead of api
scheme.Codecs.UniversalDeserializer()