Error: net/http: TLS handshake timeout via kubectl

Symptom:

  When we try to use kubectl logs <pod> or kubectl exec it  <pod> /bin/bash ....etc , we get below error:
.........  net/http: TLS handshake timeout.

While TLS certificates are valid and kubectl get nodes, kubectl cluster-info are working fine

Solution:

Use -v=8 flag to enable more details kubectl rest API call details
We found such HTTP 500 error when kubectl contacts API masterserver

GET https://Your-Master-node:6443/api/v1/namespaces/default/pods/test-deployment-6669d6df59-vdnk5/log
I0424 04:47:05.882800   11526 round_trippers.go:408] Response Status: 500 Internal Server Error in 10100 milliseconds
..
I0424 05:28:52.001101   21195 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get https://10.0.64.2:10250/containerLogs/default/test-deployment-6669d6df59-vdnk5/django: net/http: TLS handshake timeout","code":500}

10.0.64.2 is the private ip of the Node and 10250 is the listening port of kubelet
It turns out TLS error is on kubelet side of the node though TLS certificates are valid
kubectl get nodes ,kubectl cluster-info are fine as apiserver don't need to contact kubelet while kubectl logs needs apiserver to contact kubelet
It could be potential a bug.  We upgrade k8s of worker node to fix it.
Similar github issue link