When we run
curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64
We get error
curl: (23) Failed writing body (0 != 1369)
Tt is due to "/usr/local/bin/argocd" is on the /usr/local/bin directory which is owned by root user while we use normal user to run curl.
To fix it , change "/usr/local/bin/argocd" to be "/tmp/argocd"
We used to have an HTTP proxy to access Github. It was working fine. When we take off HTTP proxy via "git config --global -e", use "git config --global -l" to confirm it is taken off.
However, it still can't communicate with GitHub. Error like
kex_exchange_identification: Connection closed by remote host fatal: Could not read from remote repository
It is due to we use ssh to communicate with GitHub, while there are extra HTTP proxy settings in ~/.ssh/config file
Host=github.com
ProxyCommand=socat - PROXY:<proxy-server>:%h:%p,proxyport=80
Take them off will fix the issue.
We hit issues that all kubectl command stuck like kubectl get pod...etc
initially, we thought it is a Kubernetes control plane issue but confirmed with the cloud provider, the control plane has some communication issues with the webhook
It turns out the OPA gatekeeper was stuck and cause webhook issues with the control plane.
Workaround:
1. Delete webhook
kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io gatekeeper-validating-webhook-configuration
2. It will stabilize the communications with the control plane
3. Delete and redeploy opa keeper deployment
When we kubectl create secret tls ..., we hit below error
error: failed to load key pair tls: failed to parse private key
It is likely the private key file is encrypted with a passphrase.
Use openssl to unencrypt it and use the new key for kubectl
openssl rsa -in encrypted-private.key -out unencrypted.key
Enter pass phrase for ......
Pods always crashloopbackoff
"kubectl describe pod..." does not give meaningful info, as well as "kubectl get events"
One of the likely reason is related to pod security policy. My situation is the existing pod security policy does not allow Nginx or Apache to run. It does not have
allowedCapabilities:
- NET_BIND_SERVICE
# apache or nginx need escalation to root to function well
allowPrivilegeEscalation: true
So the pods keep crashloopbackoff. To fix it is to add the above into the pod security policy.
There is some confusion about where we should put istio objects. Is it in the istio-system or users namespace?
Here are some tips:
For TLS,mTLS CA, certs, key management in istio, the Kubernetes secrets should be created in the istio-system. Not in users' namespace
Gateway and VirtualService need to be created on the users' namespace
