Wednesday, January 30, 2019

How to Use JQ to Filter Live K8S Event Output

Requirement:

    Per other node ,we can watch live K8S core API output. The output is Json format. It has lots of information.It is abit hard to find useful information from such big volume output. This note is to address that we can use jq ( an excellent Json tool) to filter the live output and get useful , meaningful information we are looking for. More jq details are on jq cookbook

Solution:

   Please refer my other node to prepare the kubectl proxy and curl. So we can use curl to get K8S Core API event output.  The default K8S output is not nicely formatted JSON. To get proper ,pretty-print output on the screen. Below are a few examples how to use JQ for that.

Example 1:
 Display all output without filtering.
(curl  -sN http://127.0.0.1:8080/api/v1/nodes?watch=true) | while read -r watchoutput;do echo "$watchoutput" | jq '.'; done

Example 2:
To get all labels of nodes via filter
(curl -sN http://127.0.0.1:8080/api/v1/nodes?watch=true) | while read -r watchoutput;do echo "$watchoutput" | jq '. | {labels: .object.metadata.labels}'; done

Example 3:
To get conditions (disk pressure, memory pressure...) of nodes via filter
(curl -sN http://127.0.0.1:8080/api/v1/nodes?watch=true) | while read -r watchoutput;do echo "$watchoutput" | jq '. | {conditions: .object.status.conditions}'; done

Example 4:
To get node name from Core API via curl
curl -sN http://127.0.0.1:8080/api/v1/nodes | jq '.items[].metadata.name'

Example 5:
Use select and contain functions of jq to find out what K8S roles of  a person named "john" has
kubectl get clusterrolebindings -o json|jq '.items[] | select(.subjects != null) | select (.subjects[].name| contains("john")).metadata.name'

Friday, January 25, 2019

How To Build CI/CD Pipeline via Wercker in K8S

Requirement

We try to build an sample CI/CD pipeline to increase efficiency of development and deployment.It is also core part of devops methodology. With power of OKE(Oracle Kubernets Engine) and Wercker (Cloud based CI/CD tool Oracle acquired),"We can automatically build, test and deploy their applications, end-to-end, all the way from source to production in a repeatable transparent manner." Quote from Oracle CloudNative Lab Wercker Guide. More details please refer Wercker Doc

Preparation

  • Create Github account link
  • We can use github account Oauth to login Wercker
  • Login Github,create a new private repository in Github. ie iridize-oci-oke-project
  • Login Wercker via github sign-in, authorize wercker to access github .
    • On top right, click "add application"
    • Select user and SCM using the github account
    • Find the new private repository we just created
    • Let Wercker add deployment key for this private repository in github
    • In the first page of wercker, choose language or just create default wercker.yml
    • Copy and paste the wercker.yml into the Github repository
    • You would see a build is triggered in wercker as it detects a change in the repository

Implementation Steps On the Cloud

Create CI/CD for Postgresql Docker Image
  • Generate wercker.yml
box: postgres:9.3
build:
   steps:
      - internal/docker-build:
        dockerfile: Postgres-Dockerfile
        image-name: postgresimage
        
      - internal/docker-push:
        # specify the image to be pushed - this is the one we created earlier
        image-name: postgresimage
        username: $OCIRUSERNAME # Registry username
        password: $OCIRPASSWORD # Registry password
        registry: https://iad.ocir.io/v2
        repository: $IRIDIZEREPO
        tag: $VERSION
  • Copy the Postgres-dockerfile
FROM postgres:9.3
RUN mkdir -p /var/lib/postgresql-static-test/data
ENV PGDATA /var/lib/postgresql-static-test/data
  • Add Application environment variables in app.wercker.com
Application environment variables screenshot
  • Do a test update on Postgres-dockerfile or wercker.yml , the CI/CD pipeline will start automatically. rebuild the docker images and deploy it to OCIR without intervention

Implementation Steps Locally via CLI

  • Download Wercker CLI locally. More details refer doc
    • In linux where has docker running,download CLI via below command
    curl -L https://s3.amazonaws.com/downloads.wercker.com/cli/stable/linux_amd64/wercker -o /usr/local/bin/wercker
    chmod u+x /usr/local/bin/wercker
    
  • Use "wercker build" Please refer more details in wercker doc

Thursday, January 24, 2019

How To Connect Wercker With Oracle Cloud Developer Service

Requirement:

   Wercker is integrated with github well. It also integrates with Oracle Cloud Developer service well
    You have options to use both. See the screenshot

Solution:


  •  Create an instance of Oracle cloud Developer service. Refer link
  • Once the instance is up and running, you can login with your oracle cloud account 
  • Next step is configure wercker to connect to your instance. 
  • Instance url is like https://teststage-testtenancy.developer.ocp.oraclecloud.com/teststage-testtenancy  (no / at the end)
  • username is  your cloud account login name
  • password is your cloud account password
  •  configuration page is like
  • Once it is saved. it means it has connected to Oracle Developer Cloud Service well.Otherwise it will error out
  • Then you can see your projects in wercker



Wednesday, January 23, 2019

Wercker Build panic: runtime error: index out of range

Symptom:

 When we run wercker build to build a new docker image. We hit below error:
panic: runtime error: index out of range
goroutine 1 [running]:
github.com/wercker/wercker/docker.(*DockerBuildStep).configure(0xc4202086e0, 0xc42029a510, 0xf15920, 0x120baa0)
        /go/src/github.com/wercker/wercker/docker/docker_build.go:117 +0x1563

export  POSTGRES_BUILD_ARGES="foo=a  bar=b"
wercker.yml is as:
box: python:2.7
build:
   steps:
      - internal/docker-build:
        dockerfile: Postgres-Dockerfile
        image-name: postgresimage
        build-args: $POSTGRES_BUILD_ARGES

Solution:

 For some reasons, when wercker invoke docker build, $POSTGRES_BUILD_ARGES is not recognized. After I replace it with a string ,it works (build-args: "foo=a  bar=b")

Tuesday, January 22, 2019

Error parsing your wercker.yml: yaml: line 1: mapping values are not allowed in this context

Symptom:

   When we create a wercker.yml in github to build a CI/CD pipeline via app.wercker.com
    It error out :
 Error parsing your wercker.yml: yaml: line 1: mapping values are not allowed in this context

   The wercker.yml file is :
box: python:  2.7
build:
   steps:
      - internal/docker-build:
        dockerfile: Postgres-Dockerfile
        image-name: PostgresImage
        build-args: $POSTGRESS_BUILD_ARGES

Solution:

   This is yaml syntax error , nothing related to wercker . The error is from space between python:  and 2.7
   Remove space and correct one is  python:2.7

Thursday, January 17, 2019

Tip To Upload Files Into OCI Object Storage with Pre-Authenticated Requests

Symptom:

 Sometimes we need to provide many files to customers and customers need to provide files to us.
We can use OCI Object Storage Pre-Authenticated Requests to share data. Please refer more details on  Oracle Blog
    However after I create bucket Pre-Authenticated Requests , copy and paste url into our browser, we get below error

{"code":"NotFound","message":"Not Found"}

Solution:

The object Pre-Authenticated Requests can be writing or reading or both. 
The bucket Pre-Authenticated Requests is for writing purpose only (not reading)
So we need to use curl or other tools to put objects into the bucket. ie

curl https://objectstorage.us-ashburn-1.oraclecloud.com/p/THVqAQC1oIiwmerj6aK1lY5q3xG-rK3iGZvM2b1kq0/n/testtest/b/testdata/o/ --upload-file  mydata.tar

Wednesday, January 16, 2019

How To Deal With Dep Ensure Issue

Symptom:

When we create a new project in golang, we run "dep ensure" and get below error:
dep ensure
Solving failure: No versions of k8s.io/gengo met constraints:
        master: unable to update checked out version: fatal: reference is not a tree: f8a0810f38afb8478882b3835a615aebfda39afa
: command failed: [git checkout f8a0810f38afb8478882b3835a615aebfda39afa]: exit status 128

Solution:

Use "dep ensure -v"  to get debug information and find out which packages have check out issues
In this example, k8s.io/gengo  has issues.

It is very likely  dep cache corrupted on this package when new version of our apps or tools being added 

We need to delete dep cache of k8s.io/gengo

rm -rf  $GOPATH/go/dep/source/https---github.com-kubernetes-gengo

rerun dep ensure


Friday, January 04, 2019

How To Duplicate an Application in K8S to Another K8S Cluster

Please refer github note

How To Install MiniKube Inside Linux

Requirement

  • Make sure your linux  CPU has vmx svm supported. run below to check. If none return, it means your linux does not support refer doc 
 egrep '(vmx|svm)' /proc/cpuinfo
  • Otherwise you may get error like
E0830 20:33:53.017334   30356 start.go:174] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: creating domain: Error defining domain xml:

Install kubectl first

  • setup your proxy on the OS if it is behind firewall
add this into .bash_profile
export http_proxy=http://www-proxy.us.test.com:80/
export https_proxy=http://www-proxy.us.test.com:80/
  • download kubectl from google which is only 1 big file
  • curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
  • chmod +x ./kubectl
  • mv ./kubectl  /usr/bin/kubectl
  • test it to run : kubectl version to see any output. example like 
# kubectl version
    Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
      The connection to the server localhost:8080 was refused - did you specify the right host or port?
      • connection refused error can be ignored
      • refer doc 

      Install KVM2 Driver&Start Libvirtd Service for virutal machine

      • refer doc
      • #yum install libvirt
      • #yum install libvirt-daemon-kvm qemu-kvm
      • # newgrp libvirt
      • # usermod -a -G libvirt $(whoami)
      •  systemctl start  libvirtd.service
      • download driver from google
      # curl -Lo docker-machine-driver-kvm2 https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-kvm2 \
      && chmod +x docker-machine-driver-kvm2 \
      && sudo cp docker-machine-driver-kvm2 /usr/bin/ \
      && rm docker-machine-driver-kvm2

      Install / Start minikube 

      • refer doc
      • download minikube which is only 1 file
      • curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.28.2/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/bin/
       #minikube version
      minikube version: v0.28.2

      • # minikube start   --vm-driver kvm2
      Starting local Kubernetes v1.10.0 cluster...
      Starting VM...
      Downloading Minikube ISO
       160.27 MB / 160.27 MB [============================================] 100.00% 0s
      Getting VM IP address...
      Moving files into cluster...
      Downloading kubeadm v1.10.0
      Downloading kubelet v1.10.0
      Finished Downloading kubelet v1.10.0
      Finished Downloading kubeadm v1.10.0
      Setting up certs...
      Connecting to cluster...
      Setting up kubeconfig...
      Starting cluster components...

      Thursday, January 03, 2019

      How To Add Self-Signed Certificate in Oracle OCI Loadbalancer

      Requirement:

        Sometimes when we create a test environment,  we need to set SSL/TLS while we don't wanna go through the process to get official SSL/TLS CA which takes much time.
        So we can genearete self-signed Certificate and import it into OCI load balancer

      Solution:

        Refer Oracle OCI  official doc
         Use openssl to generate Certificate and Private key ( we don't need CA Authority for test)

      openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout test.key -out test.crt -subj '/CN=test.com' 

      test.key  :  Private Key file
      test.crt : Certificate file

      In OCI load balancer , click "add certificate" button, examples like


      PASSPHASE: can be empty

      Once it is added, we can apply it on OCI load balancer