How To Fix "server returned HTTP status 403 Forbidden" in Prometheus

Requirement:

   We installed and started Prometheus, however we can't get node metrics.  via /targets  , we find the error " server returned HTTP status 403 Forbidden "

Solution:

    The Prometheus /targets page will show the kubelet job with the error 403 Unauthorized, when token authentication is not enabled. Ensure, that the --authentication-token-webhook=true flag is enabled on all kubelet configurations.
    We need to enable --authentication-token-webhook=true in our kubelet conf
In the Host OS:
cd /etc/systemd/system/kubelet.service.d
vi 10-kubeadm.conf
Add  "--authentication-token-webhook=true"  into  "KUBELET_AUTHZ_ARGS"
After that, it would be like
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --authentication-token-webhook=true"
# systemctl daemon-reload
# systemctl restart kubelet.service

403 error should be gone. More details Refer github doc