Change the Reclaim Policy of a PersistentVolume In OKE

Symptom:

  By default OKE(Oracle Kubernete Engine)  storageclass is oci which is OCI block volume
  If we don't specify storageclass in yaml file,  OKE would automatically create block volumes as persistent volumes and attach to pods for us which is very convenient.
 However reclaim policy of persistent volumes is "DELETE"
 It means if we delete pv and pvc OKE created, OKE would delete block volumes in OCI as well.

Solution:

  To prevent potential data loss due to reclaim policy " DELETE" , we can update it to be "RETAIN"

kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Please refer kubernete doc for more details

Docker Network Related Issue When We Build Docker Images

Symptom:

  When I ran "docker build -t oracle/database:19c  .  "  , it always error out on yum repo issues as below
https://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/repodata/repomd.xml: [Errno 12] Timeout on https://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/repodata/repomd.xml: (28, 'Resolving timed out after 30540 milliseconds')

Solution:

I checked around proxy settings according to note. I didn't find any issues.
I added proxy settings into Dockerfile like below to inform yum to use proxy
RUN echo "proxy=http://<proxy server IP address>:80" >> /etc/yum.conf
"docker login iad.ocir.io" was working fine.
DNS entries in /etc/resolv.conf  of the container had correct settings
I  restarted docker daemon , didn't work

After checking around, I suspected it was related to docker network
We created a new bridge network in docker via "docker network create  henrynetwork"
In this way, we can force docker build to use new bridge network instead of default one
 "docker build --network=henrynetwork -t oracle/database:19c  .  "
It works.  So far I don't have any clue why default bridge network won't work.  But new bridge network will work around it. 

Tip For Size of Oracle DB Tempfiles

Symptom:

   You will see big difference between ls -l and du -sm  on the same Oracle DB Tempfiles
ie
# ls -l o1_mf_temp_fpq46223_.dbf
-rw-r-----. 1 54321 54321 34358697984 Oct 14 06:21 o1_mf_temp_fpq46223_.dbf
--> 32G
# du -sm o1_mf_temp_fpq46223_.dbf
6       o1_mf_temp_fpq46223_.dbf
-->6M

Reason:

 ls -l  reads the attributions of files, so when the tempfile was created, it was created as 32G. So both allocated and unallocated together will be 32G.
However du is counting the actual number of allocated blocks.The unallocated blocks read as 0, take up no space on disk.
In this way, du is closer to the truth about how much space was used by DB.
ls shows us total size of allocated and unallocated blocks

Tips of How to Understand CustomResourceDefinition in K8S

In Kubernetes world, CustomResourceDefinition (CRD) will play a big role when we extend K8S cluster.  In the future, CRD may include even core K8S components like pod, deployment, statefulset......etc.  see CNCF youtube link
The basic logic is :

• We define what CRD we would like to use , ie mysql-operator in github, postgres-operator.... 

Once they are defined, they will be stored in Etcd.  They are purely text file, We can define whatever we want. K8S just regard them as text config file and put them in Etcd. K8S won't check if the CRD has been implemented or not. It won't error out if CRD is not implemented.  Related kubectl commands:

lubectl get crd <name>  -o yaml    kubectl api-resources   kubectl api-versions

• We implement the business logic via GO

This part will implement how we handle events related to the CRD we define.  K8S will send events to this part, this part will do business logic for the events. K8S cluster is events driven system. All events are put in different queues inside K8S core. 

• We create yaml files to use the CRD we define

We create yaml to generate some events (create, update, delete....)  to K8S,  our 2nd part above will receive these events and do the business logic we implement.

Fortunately kubebuilder  does lots of scaffolding work for us. More details Refer kubebuilder book

How To Put Binary or Text Config Files into K8s ConfigMap

Requirement:

  We often have all kind of config files for our Apps.  From binary wallet files to simple text config files, we need to store them somewhere where apps can access.  It is not a good idea to put these config files inside docker images or PV(persistent volume) as it would make the pod less portable and hard to scale and migrate. Fortunately from K8S v1.10 , Configmap would support both binary and text config files .  Here are some examples how we utilize configmap to achieve that. Mounted configmap are updated automatically. More details please refer kubernetes configmap doc

Solution:

We plan to create 2 configmaps . One for binary files like ewallet and one for text config files  like sysctl.conf, init.ora....

• Use kubectl create configmap walletconfigmap --from-file=ewallet1 --from-file=ewallet2
• Use kubectl create configmap textconfigmap --from-file=sysctl.conf  --from-file=jdbc.xml  --from-file=init.ora  
• Use kubectl get configmap walletconfigmap -o yaml . You would see ewallet would be stored as binaryData

apiVersion: v1

items:

- apiVersion: v1

  binaryData:

    ewallet: sdfsweffewg.....

........

• Mount wallet configmap as a volume in the pod with correct config file path . ie wallet file is on /etc/oracle/tde
• Mount text configmap as a volume in the pod with correct config file path . ie text file is on /opt/oracle/dbs

volumes:

- name: wallet-volume

  configMap:

          name: walletconfigmap 

- name: textconfig-volume

   configMap:

          name: textconfigmap 

Under container section of yaml file

volumeMounts:

- name: wallet-volume

  mountPath: /etc/oracle/tde

- name: textconfig-volume

  mountPath: /opt/oracle/dbs

• In this way, when we into the pod via kubectl exec -it  <pod name> /bin/bash , we would see ewallet1 ewallet2 in /etc/oracle/tde  and  sysctl.conf jdbc.xml and init.ora are in /opt/oracle/dbs.
• If we do some updates on configmap , K8S will sync them periodically.

A Simple F5 Bigip iRule to Rewrite Urls

Rewrite example.com/test/test/mytest.html  --> example.com/test/mytest.html

when HTTP_REQUEST {
    if { [string tolower [HTTP::uri]] starts_with "/test/test" } {
        set uri [string map -nocase {"/test/test" "/test"} [HTTP::uri]]
        HTTP::uri $uri
        }
    }

How to Convert Kubeconfig CA string to Readable CA Format

Symptom:

  In ~/.kube/config  , we have kubeconfig file which has CA string like below. We would like to convert it to a readable CA format.
$ cat config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTk
............
    server: https://1.1.1.1:6443
  name: kubernetes

Solution:

First to use base64 decode the string to PEM
$ echo 'above string' | base64 -d  > /tmp/1.crt
Output is like:
-----BEGIN CERTIFICATE-----
MIIC8jCCAdqgAwIBAgIIb7ih1QetvQ8wDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE
..............
-----END CERTIFICATE-----

Second to use openssl to convert it to readable format
$ openssl x509 -in  /tmp/1.crt  -noout -text
Output is like
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8050362270253694223 (0x6fb8a1d507adbd0f)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Oct 30 08:01:56 2018 GMT
            Not After : Oct 30 08:02:01 2019 GMT
        Subject: O=system:masters, CN=kubernetes-admin
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:

                    00:bd:50:23:56:92:2c:0a........

How To Add OKE Worker Nodes via OCI API

Requirement

  We have Kubernetes cluster running in OKE (Oracle Kubernetes Engine).  We would like to dig deeper to use OCI API to manager the node pool of our K8S cluster. ie we can add a new worker node into our new private subnet. With OCI API, we can have more flexibility to handle worker nodes in the cluster.  In this example, we plan to use curl and raw REST API calls to do that.  Later oci-cli or SDK would support such functions

Solution

• First to install oci-curl .  It is a Bash Shell script. Please refer OCI Doc
• Configure the oci-curl to use correct tenancyId , authUserId , keyFingerprint, privateKeyPath. More details Please refer OCI Credential Doc and  OCI API  details
• Test oci-curl . If you can get result like below , it means it works 

# .   ./oci-curl.sh

# oci-curl containerengine.us-ashburn-1.oraclecloud.com GET "/20180222/clusterOptions/all"

{

  "kubernetesVersions": [

    "v1.10.11",

    "v1.11.5"

  ]

• Prepare oke-nodepool.json The example of json file is like

{

  "compartmentId": "INSERT_YOUR_COMPARTMENT_ID",

  "clusterId": "INSERT_YOUR_CLUSTER_ID",

  "name": "INSERT_YOUR_NODE_POOL_NAME",

  "kubernetesVersion": "INSERT_YOUR_K8S_VERSION",

  "nodeImageName": "INSERT_YOUR_CUSTOM_IMAGE_OCID",

  "nodeShape": "INSERT_YOUR_SHAPE_NAME",

  "nodeMetadata": {

    "INSERT_YOUR_CUSTOM_KEY_1": "INSERT_YOUR_CUSTOM_VALUE_1",

    "INSERT_YOUR_CUSTOM_KEY_2": "INSERT_YOUR_CUSTOM_VALUE_2"

  },

  "initialNodeLabels": [

    {

      "key": "node.info/INSERT_YOUR_SAMPLE_KEY",

      "value": "INSERT_YOUR_SAMPLE_VALUE"

    }

  ],

  "sshPublicKey": "INSERT_YOUR_SSH_PUBLIC_KEY",

  "quantityPerSubnet": INSERT_YOUR_QTY_PER_SUBNET,

  "subnetIds": [

    "",

    "",

    ""

  ]

}

• Create a node pool 

oci-curl containerengine.us-ashburn-1.oraclecloud.com POST ./oke-nodepool.json "/20180222/nodePools"

• Go to the OCI ashburn console , a new nodepool would be created in OKE according to json file.

Error: Couldn't find key username in Secret default/test-stg-secret

Symptom:

  We create test-stg-secret via yaml file below

apiVersion: v1
kind: Secret
metadata:
  name:  test-stg-secret
type: Opaque
stringData:
  config.yaml: |-
    username: test
    password:  test

 However we got "Error: Couldn't find key username in Secret default/test-stg-secret" when we create statefulset. Part of secret usages are:
env:

            - name: TEST_USER
              valueFrom:
                  secretKeyRef:
                     name: test-stg-secret
                     key: username
            - name: TEST_PASSWORD
              valueFrom:
                  secretKeyRef:
                     name: test-stg-secret
                     key: password

Solution:

The reason is due to when we create secret it regards the whole text as one single string as config.yaml

It does not take username or password as key

To workaround it, we need to get decoded string via base64

echo -n 'test' | base64  --->  dGVzdA==

-n will take new line at the end

Then correct yaml file is :

apiVersion: v1kind: Secretmetadata:  name:  test-stg-secret
type: Opaque
data:
    username: dGVzdA==
    password:  dGVzdA==

Tip of Streaming Logs to Stdout for K8S Pods

Requirement:

 In K8S world, we need to stream apps logs output to standard output for analysis purpose. For example, we need to gather access logs information  of our applications , so we need to redirect all access logs data to stdout of Pod.

Solution:

  Find full path of the logs which we would like to stream
  ie /var/log/apache2/access_log    and /var/log/apache2/error_log

Use below ln to link to stdout
ln -sf   /proc/self/fd/1  /var/log/apache2/access_log 
ln -sf  /proc/self/fd/1   /var/log/apache2/error_log

Tip to Get K8S API Server Startup Parameters

If the API server is running in a pod, to find out details of startup parameters of K8S API
$ kubectl get po -n kube-system kube-apiserver -o yaml
.......
- command:
    - kube-apiserver
    - --secure-port=6443
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --insecure-port=0
    - --requestheader-username-headers=X-Remote-User
    - --requestheader-allowed-names=front-proxy-client
    - --service-cluster-ip-range=1.1.1.1/12
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --enable-bootstrap-token-auth=true
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --advertise-address=1.1.1.1
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
    - --allow-privileged=true
    - --requestheader-group-headers=X-Remote-Group
    - --authorization-mode=Node,RBAC
    - --etcd-servers=https://127.0.0.1:2379
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    image: container-registry.oracle.com/kubernetes_developer/kube-apiserver-amd64:v1.10.5

..........

Tip to Check K8S Core API Https Header and Contents via Kubectl

As we know, K8S core API are Https calls. We can use kubectl --v=8 to get  debug details each https call of the core API. With same way, we can other kubectl commands debug info. ie

$ kubectl --v=8 create  -f ship_v1beta1_frigate.yaml
I0203 23:37:32.108940   21540 loader.go:357] Config loaded from file /home/k8suser/.kube/config
I0203 23:37:32.111815   21540 round_trippers.go:383] GET https://1.1.1.1:6443/openapi/v2
I0203 23:37:32.111862   21540 round_trippers.go:390] Request Headers:
I0203 23:37:32.111884   21540 round_trippers.go:393]     Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0203 23:37:32.111906   21540 round_trippers.go:393]     User-Agent: kubectl/v1.10.5+2.0.2.el7 
.................
frigate.ship.example.com "frigate-sample-test" created

Tip of How to Verify CA chain of Kubernetes

Find root CA file of the Kubernetes cluster
Find Client certificate file
They could be in /etc/kubernetes/pki  or  /var/lib/kubelet

# openssl  verify  -CAfile ca.crt  apiserver-kubelet-client.crt
apiserver-kubelet-client.crt: OK

Tip of How To Find When Kubernetes CA Expires

Find the locations of CA files
They could be in /etc/kubernetes/pki  or  /var/lib/kubelet
Use openssl

# openssl x509 -in  ca.crt  -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Oct 30 08:01:56 2018 GMT
            Not After : Oct 27 08:01:56 2028 GMT
        Subject: CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:cd:ab:2d:f3:86:f3:57:28:43:4f:08:93:a6:e7:
                    22:b0:51:73:74:cb:5a:3c:ee:01:2d:ad:1c:a4:97:
                 .........