Tip: Add Resources via Kustomize edit add

kustomize edit add support globbing strings

Add directory
kustomize edit add resource <dir>

Add Yaml files
kustomize edit add resource  ./*.yaml

Tip: List Only Filename in Different Lines

ls  *.yaml  --format single-column

Tip: remote: error: Your commit has been blocked due to certain binary file(s) being oversized or not allowed.

Symptom:

  When we git push a large file into gitlab, we hit this error

remote: error: Your commit has been blocked due to certain binary file(s) being oversized or not allowed.

Solution:

git reset   --soft HEAD~1.   (2 or 3 depends how far to rollback)
git commit -m   "your message"
git pull  (to merge the changes)
git push

Tip: Add Password Into Existing Private Keys

Add Password:

openssl rsa -aes256 -in your-private-key.pem -out your-private-key-encrypted.pem
writing RSA key
Enter PEM pass phrase:   ****
Verifying - Enter PEM pass phrase: ****

Remove Password:

openssl rsa -in  your-private-key-encrypted.pem -out your-private-key.pem
Private key passphrase:  ****

Warning: the ECDSA host key for 'testgitlab.test.com' differs from the key for the IP address 1.1.1.1

To fix it:

ssh-keygen -R 1.1.1.1

Tip: find sed awk egrep

find ./ -type f -exec sed -i -e 's/oldstring/newstring/g' {} \;

find ./ -name *.yaml | while read -r filename; do echo "test" >> "$filename"; done

kubectl get rolebindings  --all-namespaces |egrep 'strings1|strings' |grep psp |awk '{print $2" -n "$2 }'

Goal: convert string from "adc@adc.com  ocid.aa******" to

"

- apiGroup: rbac.authorization.k8s.io 

  kind: User 

  name: ocid.aa*** # adc@adc.com

"

Tip:

Put all strings in 1.txt

cat 1.txt  | while read -r line; do echo $line | awk '{print "- apiGroup: rbac.authorization.k8s.io","\n", " kind: User","\n"," name: "$2" # "$1, "\n"}'; done

Tip: Find List of Changed Files after Git Commit

git diff-tree --no-commit-id --name-only -r ab8776d..a7d508b

ab8776d and  a7d508b are git commit SHA

Kubectl Stop Working After Upgrade Ubuntu

Symptom:

   After we upgrade ubuntu to 20.04, kubectl stop working with OKE (Oracle Kubernetes Engine).  Error is like

$kubectl version
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00007f2d902e0740 (most recent call first):
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Unable to connect to the server: getting credentials: exec: signal: aborted (core dumped)

Solution:

  The issue is related to Kube config of OKE. In OKE, we need to use oci to authenticate users to access OKE control plane. The upgrade of ubuntu somehow breaks the python env that oci cli depends.  To fix it, we need to re-install oracle cloud oci cli. Please refer link

A few simple Tips on Kubernetes

• Core DNS is deployment and Flannel is DaemonSet
• Persistent Volume (PV)  is not under the namespace
• Persistent Volume Claim(PVC) is under the namespace
• Delete PVC will delete PV automatically by default (Oracle Kubernetes Engine). Change retain policy if necessary
• Drain the node before rebooting worker node
• Both TCP and UDP must be open in the worker node subnet.
• If UDP is open after VMs are up and running, we may need to recreate VMs to let docker daemon to work with new settings

Tip:Name or service not known isues in Kubernetes

Symptom:

    We got below error when we try to psql into Postgres in Kubernetes Pods. The error is intermittent.

psql: could not translate host name “test-dbhost” to address: Name or service not known

We use this command to test Kube DNS service, curl -v telnet://10.96.5.5:53

The result is also intermittent and DNS resolution is kind of working but very slow

Solution:

   We have checked Kube-DNS service is up and running.  The core DNS Pods are up and well.
We also found if the pods are in the same worker node, they are working fine. However, if cross the nodes, we hit issues. It seems issues on the node to node communications.

   Finally, we find the TCP is open but UDP is not open in the node subnet. We have to open the UDP.
After UDP is open, the intermittent issues are still existing. It is quite possibly related to docker daemon stuck in the old settings.  We need to rolling restart worker nodes to fix it.

To rolling restart worker node:

1. Assume you have nodes available in the same AD of OKE, kubectl drain will move pv,pvc to the new node automatically for statefulset and deployment
2. kubectl drain <node> --ignore-daemonsets  --delete-local-data
3. reboot the node
4. kubectl uncordon <node>

Tip: Clean Oracle DB Diagnose Home Automatcally

Requirement:

  Oracle DB can generate a huge amount of trace files and fill up the file system. Tired to clean Oracle DB trace files and incident files?

Solution:

SHORTP_POLICY : Retention for ordinary trace files
LONGP_POLICY : Retention for like incident files

adrci> set control (SHORTP_POLICY = 360) ===>15days
adrci> set control (LONGP_POLICY = 2160) ===>90 Days
adrci> show control

Purging Trace files manually:

Following command will manually purge all tracefiles older than 2 days (2880 minutes):
adrci> purge -age 4880 -type trace
adrci> purge -age 129600 -type ALERT ===> purging ALERT older than 90 days
adrci> purge -age 43200 -type INCIDENT ===> purging INCIDENT older than 30 days
adrci> purge -age 43200 -type TRACE ===> purging TRACE older than 30 days
adrci> purge -age 43200 -type CDUMP ===> purging CDUMP older than 30 days
adrci> purge -age 43200 -type HM ===> purging HM older than 30 days
adrci> show tracefile -rt

Crontab to purge files Automatically
00 20 * * * adrci exec="set home diag/rdbms/****;purge -age 4880 -type trace;purge -age 43200 -type INCIDENT;purge -age 43200 -type CDUMP"