Error:request.go:598:31: not enough arguments in call to watch.NewStreamWatcher have (*versioned.Decoder) want (watch.Decoder, watch.Reporter)

Symptom:

When we build a go program, we hit such error:
k8s.io/client-go/rest/request.go:598:31: not enough arguments in call to watch.NewStreamWatcher have (*versioned.Decoder) want (watch.Decoder, watch.Reporter)
It appears there are updates on the request.go which have new requirements
See details of changes log  on apimachinary 

Solution:

   We need to avoid to use the latest master branch of the client-go. Instead we can use stable version of client-go. Fortunately go-modules addresses these problems
see github go-modules
So here are the steps to fix it

• $ export GO111MODULE=on
• In your project location, run :  go mod init     ---- create go.mod file
• Go build  cmd/test.go    --- go mod will fetch related files which replace dep ensure
• You will still see the error, that's ok , next step we fix it 
• Edit go.mod and replace client-go with correct version
• In this case  we use :   k8s.io/client-go v0.0.0-20190425172711-65184652c889
• Go build cmd/test.go     ---error would be gone

Error: No route to host while ICMP is working

Symptom:

   We try to access a port 1521 service in a VM which has Oracle Linux 7.6 . Ping (ICMP) is working fine. But we get error when we test port via nc or telnet
Ncat: No route to host.

Solution:

  There are quite a few reasons for that.

• check your subnet security list, make sure ports are open
• use traceroute and dig to check route table is working as expected
• check target VM linux firewalld is up 

In our case, firewalld is up by default, it blocks traffic of  1521. The error message is confusing people "Ncat: No route to host." . It is not about route. It is due to traffic blocked by target host Linux

Fix it via systemctl stop firewalld ;systemctl disable firewalld

Error: cannot refer to unexported name in Golang

Symptom:

  The packages are in the right place and imported ,but when we define variable from a type in another package, it error out when we define myvar a.mytype
Error: cannot refer to unexported name ******

package a
type  mytype struct {
    a int
}

package main
import "a"
func main() {
  var myvar a.mytype
}

Solution:

The reason why mytype can't be exported is due to Golang needs uppercase of first letter of a exported type or functions.  See more details in stackoverflow link

Correct one is
package a
type  Mytype struct {
    a int
}

package main
import "a"
func main() {
  var myvar a.Mytype
}

VirtualBox Guest OS Network Connects To Host VPN

Requirement:

   We have linux guest OS running in virtualbox win 10.  We have vpn running on win 10 host.
We have issues to connect to network in guest OS after VPN started.  After VPN shutdown, it works fine

Solution:

We need to add natdnsresolver1 to the VM

• Set Guest OS to attach to NAT and use adapter to "Paravirutalized network"  
• Shutdown Guest OS
• Run VBoxManage.exe list vms
• Run VBoxManage.exe modifyvm <uuid here or name > --natdnshostresolver1 on
• Start Guest OS
• We can disconnect and reconnect host VPN on the fly, Guest OS can pick up connections.

Error: Package libseccomp was not found in the pkg-config search path in go build

Symptom:

  When we test usage of  "github.com/seccomp/libseccomp-golang" , it always error out as below
package libseccomp was not found in the pkg-config search path.
Perhaps you should add the directory containing `libseccomp.pc'

Solution:

It turns out we need to manually compile this libseccomp library from code as it is not included in normal pkg.

• download released C code from github
• tar zxvf <release tar file>
• ./configure
• make
• make install

Then we have this library included in our OS, the issue is gone

A few Useful Postgresql Tips

Export data:

pg_dump -U testuser -h <db host> -p 5432 <db name > testdump.sql

Import data (plain sql file):

psql -U testuser <db name>  < testdump.sql> testImport.log

Connect to remote DB:

psql -U testuser -h  <db host> -p 5432 <db name>

Connect to remote DB with password

psql "dbname=<db name> user=postgres password=***  host=<db host> port=5432"

Grant privileges:

drop database testdb;
create database testdb;
create role testuser ;
alter role testuser  createdb;
alter role testuser login;
alter role testuser  createrole;

Grant an user to be a superuser:

ALTER USER testuser WITH SUPERUSER;

Check db connections in the Postgresql ie client_addr ,client_hostname

select * from pg_stat_activity  where datname = 'testdb';

Check db parameters ie max_connections

show max_connections;

Create a DB with UTF8 

CREATE DATABASE "teststg"
    WITH OWNER "postgres"
    ENCODING 'UTF8'
    LC_COLLATE = 'en_US.UTF-8'
    LC_CTYPE = 'en_US.UTF-8'
    TEMPLATE template0;

psql: FATAL: Ident authentication failed for user “postgres”

Symptom:

psql connect to postgresql server, get below error:
psql: FATAL: Ident authentication failed for user “postgres”

Solution:

By default , the authentication in pg_hba.conf is "ident"
We need to replace it with "md5" to use password. After that reload postgres

example to allow apps connections and trust local connections :

# "local" is for Unix domain socket connections only
local   all             all                                         trust
# IPv4 local connections:
host    all             all             127.0.0.1/32        trust
# IPv6 local connections:
host    all             all             ::1/128                 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                             trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                     trust
host    all                  all             0.0.0.0/0                  md5

Reserve Source IP via externalTrafficPolicy

Requirement:

We often need to save and check original source IP of clients for audit or analysis.  In K8S , Source NAT is enabled by default for NodePort and LoadBalancer.

Solution:

We can set externalTrafficPolicy = Local to reserve client source IP. More details in K8S source ip doc

Squid Proxy Logs Sample ouput:

External Traffic Policy : not set   (10.244.1.1 is sourced nat IP)
```1554177505.281      0 10.244.1.1 TCP_DENIED/403 4116 CONNECT 140.84.22.11:443 - HIER_NONE/- text/html
1554177510.401      0 10.244.1.1 TCP_DENIED/403 4116 CONNECT 140.84.22.11:443 - HIER_NONE/- text/html```

External Traffic Policy : Local  (132.30.131.49 is client IP)
```1554180756.818      0 132.30.131.49 TCP_DENIED/403 3995 CONNECT 140.84.22.11:443 - HIER_NONE/- text/html
1554180984.270      0 132.30.131.49 TCP_DENIED/403 4104 CONNECT 140.84.22.11:443 - HIER_NONE/- text/html```

Use ConfigMap To Store Http.conf For Http Service Pod

Requirement:

    Lots of applications would have http service ie apache or nginx as the frontend. We often deploy pods of apache or nginx for it.  Take apache httpd as example, we often need to update httpd.conf for rewrite, redirect.....etc. It is ok to build a new docker image to achieve that ,but not efficient .  ConfigMap of K8S can store text and binary files in K8S, can be mounted in the pod. So we can leverage that to update httpd.conf without rebuilding the docker images. We can use same concept for all other config files of different apps ie nginx, ords...etc

Solution:

• Prepare for Dockerfile and your customized httpd.conf file . Example can be found on github repo
• Once new docker image is built , we need to store httpd.conf into configmap via kubectl 

kubectl create configmap httpdconfig --from-file=httpd.conf 

• Prepare for deployment yaml to mount configmap in the pod. Partial yaml file is like

         volumes:
            - name: httpd-config
              configMap:
                 name: httpdconfig
         containers:
           - name: httpd
             image: httpd-configmap:v3
             imagePullPolicy: IfNotPresent
             volumeMounts:
                - name: httpd-config
                  mountPath: /mnt/k8s
             ports:
                - containerPort: 80

• Kubectl command of updating httpd.conf  after we have new version of httpd.conf

kubectl create configmap httpdconfig --from-file=httpd.conf  -o yaml --dry-run | kubectl replace -f -

• Bounce the http pod to let new pod  read the new configmap
• It is the same concept and process for any other apps which have config files ie ORDS, Nginx.....etc
• Configmap supports binary file as well ie wallets  see other note

Warning: 199 APEX "HTTP request but need HTTPS" on Apache Reverse Proxy

Symptom:

  We have APEX and ORDS running on port 8888. We have TLS/SSL enabled on LoadBalancer.  We have reverse proxy configuration for Http and Ords

 ProxyPass "/apex" "http://localhost:8888/apex" retry=60
 ProxyPassReverse /apex http://localhost:8888/apex
 ProxyPreserveHost On

When we apex applications are not verifying HTTPS connections, all are fine. After apex applications start to verify HTTPS connections, error out though we have TLS on Loadblanancer
Warning: 199 APEX "HTTP request but need HTTPS"

Solution:

It turns out issue on type Loadbalancer we created. By default it is on TCP-443, so it is on Transport Layer , it has no idea it is https or http, connections pass to apex application is  TCP connections with port 443. So apex application would not regard it as https.

We need to change Loadbalancer type to HTTP -443 which is Application Layer, in this way, apex application can see it is https, thus the issue is gone.

In OKE service yaml file , we can add below to inform OCI LB to use "HTTP"

service.beta.kubernetes.io/oci-load-balancer-backend-protocol: "HTTP"

Tips for Apache Reverse Proxy

• It is fine from HTTPS --> HTTP  
• Need extra work for HTTP --> HTTPS . SSLProxyEngine --> ON  Apache link  stackoverflow link
• HTTPS --> HTTPS is similar as HTTP --> HTTPS

Error :no available volume zone in Kubernetes

Symptom:

   When we create deployment/statefulset/pod in OKE (Oracle Kubernete Engine), somehow we hit below error:
Warning  FailedScheduling  3s (x7 over 3m)   default-scheduler  0/3 nodes are available: 1 node(s) didn't match node selector, 2 node(s) had no available volume zone.

Solution:

   One of the reasons is the we use OKE auto provision for our block volume storage. It has a constraint that block volume need to be the same AD (availability zone) as VM. In that case the block volume is created in different AD, the pod can't access the block volume
To fix that, we just need to adjust the label to let pod be created in the same AD as block volume.

Go Implementation For smallest-window-in-a-string-containing-all-the-characters-of-another-string

This note is to add Go implementation of below problem
https://practice.geeksforgeeks.org/problems/smallest-window-in-a-string-containing-all-the-characters-of-another-string/0

Golang playground url : https://play.golang.org/p/ao9J2Y4veUt

Github code url  

Go Implementation For generate-binary-string

This note is to add Go implementation of below problem
https://practice.geeksforgeeks.org/problems/generate-binary-string/0

Golang playground url : https://play.golang.org/p/VsJ8RIurpwm

Github code url 

Go Implementation For longest-k-unique-characters-substring

This note is to add Go implementation of below problem
https://practice.geeksforgeeks.org/problems/longest-k-unique-characters-substring/0

Golang playground url :https://play.golang.org/p/fINrPuXtBKA

Github code url 

Go Implementation For remove-b-and-ac-from-a-given-string

This note is to add Go implementation of below problem
https://practice.geeksforgeeks.org/problems/remove-b-and-ac-from-a-given-string/0

Golang playground url : https://play.golang.org/p/Y5msCtkHzkU

Github code url 

Go Implementation For Knapsack with Duplicate Items

This note is to add Go implementation of below problem
https://practice.geeksforgeeks.org/problems/knapsack-with-duplicate-items/0

Golang playground url : https://play.golang.org/p/gRnJl-ssfVM

Github code url 

Go Implementation For find-largest-word-in-dictionary

This note is to add Go implementation of below problem
https://practice.geeksforgeeks.org/problems/find-largest-word-in-dictionary/0

Golang playground url : https://play.golang.org/p/3yIbRbMnnrG

Github code url 

Error : /usr/bin/postgresql-setup initdb no such file or directory

Symptom:

When we build postgresql 9.5 docker image, we
 /usr/bin/postgresql-setup initdb  no such file or directory

Solution:

It is due to initdb was not in default PATH . By default, yum installs initdb at /usr/pgsql-9.5/bin
To fix that we add below line into Dockerfile
RUN ln -s /usr/pgsql-9.5/bin/initdb  /usr/bin/initdb

The full Dockerfile details of Postgresql 9.5 is on github link

Failed to Link Error When Building Postgresql Docker Image

Symptom:

   When we build docker image for Postgresql 9.2  9.5 on Oracle Linux 7, we hit below error

failed to link /usr/share/man/man1/clusterdb.1 -> /etc/alternatives/pgsql-clusterdbman: No such file or directory
failed to link /usr/share/man/man1/createdb.1 -> /etc/alternatives/pgsql-createdbman: No such file or directory
.......

Solution:

It is due to base image Oracle linux does not have such directory (to save space for linux image)   /usr/share/man/man1/ 
Add below to Dockerfile to workaround it
RUN mkdir -p /usr/share/man/man1

The full Dockerfile details of Postgresql 9.5 is on github link

Simple GO Codes Functions

Golang  function to fill a specific default value into an Array
Simple code to achieve that

func filldefault(YourArray []int ) {
   for i := range YourArray { YourArray[i] = -1 }
}

Golang function of max and min
Simple code to achieve that

func max(x, y int) {
   if x > y {
        return x
   } else {
       return y
        }
}

func min(x, y int) {
   if x > y {
        return y
   } else {
       return x
       }
}

Golang  function of contain or find an element in an array
Simple code to achieve that

func contain(t []int, x int) bool {
       for _, n := range t {
        if n == x {
          return true
       }
     }
   return false

}

Dockerfile for Httpd on Oracle Linux 7

Please check details on Github OracleLinux Httpd Dockerfile Repo

Dockerfile for Postgresql9.5 on Oracle Linux 7

Please check details on Github OracleLinux Postgresql 9.5 Dockerfile 

Docker: Failed to get D-Bus connection: Operation not permitted

Symptom:

When we build Postgresql 9.5 docker image on  Oracle Linux 7-slim ( FROM oraclelinux:7-slim)
it error out
Docker: Failed to get D-Bus connection: Operation not permitted

Solution:

It is due to Oracle Linux 7-slim is very slim. It takes systemd function out of the image.  D-bus need systemd to function.  In order to fix it ,we need to use a bit heavy linux image which includes systemd.  To use Oracle Linux 7 would fix it (FROM oraclelinux:7)

The full Dockerfile details of Postgresql 9.5 is on github link