kubectl top node doesn't work. Looks like an issue with heapsterError from kubectl with google cloud container engineissue in installing ambassador over kubernetesWhen creating a regional node pool with gcloud. The nodes do not registerkube-dns fails open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directorykubectl top is trying to connect to heapster instead of metrcis serverkubectl cannot authenticate with AWS EKSKubernetes on Windows with NAT - ports mapping doesn't workMissing logs in Stackdriver from Kubernetes deployments using GKENode-to-Node communication doesn't work with Kubernetes with CalicoError: forwarding ports: error upgrading connection: error dialing backend: - Azure Kubernetes Service
Unknowingly ran an infinite loop in terminal
Should my Json storage handle exceptions?
What happens to the Time Stone
How can I support myself financially as a 17 year old with a loan?
Should I replace my bicycle tires if they have not been inflated in multiple years
Father and Son and Grandsons
What actually is the vector of angular momentum?
How would adding a darkvision racial trait to Dragonborn affect balance?
Why wasn't the Night King naked in S08E03?
What is the unit of the area when geometry attributes are calculated in QGIS?
Besides the up and down quark, what other quarks are present in daily matter around us?
To customize a predefined symbol with different colors
SFINAE works with deduction but fails with substitution
How do I tell my manager that his code review comment is wrong?
If prion is a protein. Why is it not disassembled by the digestive system?
What does a yield inside a yield do?
Where can I go to avoid planes overhead?
How to reply this mail from potential PhD professor?
Are we obligated to aspire to be Talmidei Chachamim?
Identifying my late father's D&D stuff found in the attic
Moving the subject of the sentence into a dangling participle
Theorem won't go to multiple lines and is causing text to run off the page
Is Jon mad at Ghost for some reason and is that why he won't acknowledge him?
What happens to matryoshka Mordenkainen's Magnificent Mansions?
kubectl top node doesn't work. Looks like an issue with heapster
Error from kubectl with google cloud container engineissue in installing ambassador over kubernetesWhen creating a regional node pool with gcloud. The nodes do not registerkube-dns fails open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directorykubectl top is trying to connect to heapster instead of metrcis serverkubectl cannot authenticate with AWS EKSKubernetes on Windows with NAT - ports mapping doesn't workMissing logs in Stackdriver from Kubernetes deployments using GKENode-to-Node communication doesn't work with Kubernetes with CalicoError: forwarding ports: error upgrading connection: error dialing backend: - Azure Kubernetes Service
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I have a fresh k8s cluster on gke.
Whenever I run kubectl top node gke-data-custom-vm-6-25-0cbae9b9-hrkc
I get
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
At the same time I have this service:
> kubectl -n kube-system get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend NodePort 10.11.241.20 <none> 80:32688/TCP 59d
heapster ClusterIP 10.11.245.182 <none> 80/TCP 59d
kube-dns ClusterIP 10.11.240.10 <none> 53/UDP,53/TCP 59d
metrics-server ClusterIP 10.11.249.26 <none> 443/TCP 59d
and a pod with heapster is running (and I can see it was restarted a lot of times)
kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
event-exporter-v0.2.3-85644fcdf-kwd6g 2/2 Running 0 16d
fluentd-gcp-scaler-8b674f786-dbrcr 1/1 Running 0 16d
fluentd-gcp-v3.2.0-2fqgl 2/2 Running 0 17d
fluentd-gcp-v3.2.0-47586 2/2 Running 0 17d
fluentd-gcp-v3.2.0-552xm 2/2 Running 0 16d
heapster-v1.6.0-beta.1-fdc7fd478-8s998 3/3 Running 73 16d
However I can see in logs of heapster-nanny container some errors:
> kubectl logs -n kube-system --tail 10 -f po/heapster-v1.6.0-beta.1-fdc7fd478-8s998 -c heapster-nanny
ERROR: logging before flag.Parse: E0418 23:30:10.075539 1 nanny_lib.go:95] Error while querying apiserver for resources: Get https://10.11.240.1:443/api/v1/namespaces/kube-system/pods/heapster-v1.6.0-beta.1-fdc7fd478-8s998: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:10.971230 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:11.972337 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:12.973637 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:13.975024 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:14.976582 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:16.063760 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:27.065693 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0418 23:30:30.077159 1 nanny_lib.go:95] Error while querying apiserver for resources: Get https://10.11.240.1:443/api/v1/namespaces/kube-system/pods/heapster-v1.6.0-beta.1-fdc7fd478-8s998: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0418 23:30:59.778560 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
and also in heapster container
I0423 07:02:10.765134 1 heapster.go:113] Starting heapster on port 8082
W0423 07:16:27.975467 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:16:43.064110 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:20:36.875359 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:20:44.383790 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:22:29.683060 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:22:40.278962 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:31:27.072711 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:31:54.580031 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
How can I fix this?
Any additional info that I should provide?
monitoring kubernetes google-kubernetes-engine metrics
asked Apr 23 at 8:04
Korjavin IvanKorjavin Ivan
1,36221638
add a comment |
I have a fresh k8s cluster on gke.
Whenever I run kubectl top node gke-data-custom-vm-6-25-0cbae9b9-hrkc
I get
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
At the same time I have this service:
> kubectl -n kube-system get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend NodePort 10.11.241.20 <none> 80:32688/TCP 59d
heapster ClusterIP 10.11.245.182 <none> 80/TCP 59d
kube-dns ClusterIP 10.11.240.10 <none> 53/UDP,53/TCP 59d
metrics-server ClusterIP 10.11.249.26 <none> 443/TCP 59d
and a pod with heapster is running (and I can see it was restarted a lot of times)
kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
event-exporter-v0.2.3-85644fcdf-kwd6g 2/2 Running 0 16d
fluentd-gcp-scaler-8b674f786-dbrcr 1/1 Running 0 16d
fluentd-gcp-v3.2.0-2fqgl 2/2 Running 0 17d
fluentd-gcp-v3.2.0-47586 2/2 Running 0 17d
fluentd-gcp-v3.2.0-552xm 2/2 Running 0 16d
heapster-v1.6.0-beta.1-fdc7fd478-8s998 3/3 Running 73 16d
However I can see in logs of heapster-nanny container some errors:
> kubectl logs -n kube-system --tail 10 -f po/heapster-v1.6.0-beta.1-fdc7fd478-8s998 -c heapster-nanny
ERROR: logging before flag.Parse: E0418 23:30:10.075539 1 nanny_lib.go:95] Error while querying apiserver for resources: Get https://10.11.240.1:443/api/v1/namespaces/kube-system/pods/heapster-v1.6.0-beta.1-fdc7fd478-8s998: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:10.971230 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:11.972337 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:12.973637 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:13.975024 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:14.976582 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:16.063760 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:27.065693 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0418 23:30:30.077159 1 nanny_lib.go:95] Error while querying apiserver for resources: Get https://10.11.240.1:443/api/v1/namespaces/kube-system/pods/heapster-v1.6.0-beta.1-fdc7fd478-8s998: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0418 23:30:59.778560 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
and also in heapster container
I0423 07:02:10.765134 1 heapster.go:113] Starting heapster on port 8082
W0423 07:16:27.975467 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:16:43.064110 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:20:36.875359 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:20:44.383790 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:22:29.683060 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:22:40.278962 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:31:27.072711 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:31:54.580031 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
How can I fix this?
Any additional info that I should provide?
monitoring kubernetes google-kubernetes-engine metrics
asked Apr 23 at 8:04
Korjavin IvanKorjavin Ivan
1,36221638
heapter is deprecated and you need to use metric server
– c4f4t0r
Apr 23 at 8:50
there is metric-server is well, but kubectl top doesn't work still.
– Korjavin Ivan
Apr 23 at 9:32
Can you please confirm, if you still observe these errors in your cluster ?
– Nepomucen
Apr 23 at 14:24
yes, I still observe these errors
– Korjavin Ivan
Apr 23 at 18:30
add a comment |
I have a fresh k8s cluster on gke.
Whenever I run kubectl top node gke-data-custom-vm-6-25-0cbae9b9-hrkc
I get
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
At the same time I have this service:
> kubectl -n kube-system get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend NodePort 10.11.241.20 <none> 80:32688/TCP 59d
heapster ClusterIP 10.11.245.182 <none> 80/TCP 59d
kube-dns ClusterIP 10.11.240.10 <none> 53/UDP,53/TCP 59d
metrics-server ClusterIP 10.11.249.26 <none> 443/TCP 59d
and a pod with heapster is running (and I can see it was restarted a lot of times)
kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
event-exporter-v0.2.3-85644fcdf-kwd6g 2/2 Running 0 16d
fluentd-gcp-scaler-8b674f786-dbrcr 1/1 Running 0 16d
fluentd-gcp-v3.2.0-2fqgl 2/2 Running 0 17d
fluentd-gcp-v3.2.0-47586 2/2 Running 0 17d
fluentd-gcp-v3.2.0-552xm 2/2 Running 0 16d
heapster-v1.6.0-beta.1-fdc7fd478-8s998 3/3 Running 73 16d
However I can see in logs of heapster-nanny container some errors:
> kubectl logs -n kube-system --tail 10 -f po/heapster-v1.6.0-beta.1-fdc7fd478-8s998 -c heapster-nanny
ERROR: logging before flag.Parse: E0418 23:30:10.075539 1 nanny_lib.go:95] Error while querying apiserver for resources: Get https://10.11.240.1:443/api/v1/namespaces/kube-system/pods/heapster-v1.6.0-beta.1-fdc7fd478-8s998: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:10.971230 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:11.972337 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:12.973637 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:13.975024 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:14.976582 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:16.063760 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:27.065693 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0418 23:30:30.077159 1 nanny_lib.go:95] Error while querying apiserver for resources: Get https://10.11.240.1:443/api/v1/namespaces/kube-system/pods/heapster-v1.6.0-beta.1-fdc7fd478-8s998: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0418 23:30:59.778560 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
and also in heapster container
I0423 07:02:10.765134 1 heapster.go:113] Starting heapster on port 8082
W0423 07:16:27.975467 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:16:43.064110 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:20:36.875359 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:20:44.383790 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:22:29.683060 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:22:40.278962 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:31:27.072711 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:31:54.580031 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
How can I fix this?
Any additional info that I should provide?
monitoring kubernetes google-kubernetes-engine metrics
asked Apr 23 at 8:04
Korjavin IvanKorjavin Ivan
1,36221638
I have a fresh k8s cluster on gke.
Whenever I run kubectl top node gke-data-custom-vm-6-25-0cbae9b9-hrkc
I get
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
At the same time I have this service:
> kubectl -n kube-system get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend NodePort 10.11.241.20 <none> 80:32688/TCP 59d
heapster ClusterIP 10.11.245.182 <none> 80/TCP 59d
kube-dns ClusterIP 10.11.240.10 <none> 53/UDP,53/TCP 59d
metrics-server ClusterIP 10.11.249.26 <none> 443/TCP 59d
and a pod with heapster is running (and I can see it was restarted a lot of times)
kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
event-exporter-v0.2.3-85644fcdf-kwd6g 2/2 Running 0 16d
fluentd-gcp-scaler-8b674f786-dbrcr 1/1 Running 0 16d
fluentd-gcp-v3.2.0-2fqgl 2/2 Running 0 17d
fluentd-gcp-v3.2.0-47586 2/2 Running 0 17d
fluentd-gcp-v3.2.0-552xm 2/2 Running 0 16d
heapster-v1.6.0-beta.1-fdc7fd478-8s998 3/3 Running 73 16d
However I can see in logs of heapster-nanny container some errors:
> kubectl logs -n kube-system --tail 10 -f po/heapster-v1.6.0-beta.1-fdc7fd478-8s998 -c heapster-nanny
ERROR: logging before flag.Parse: E0418 23:30:10.075539 1 nanny_lib.go:95] Error while querying apiserver for resources: Get https://10.11.240.1:443/api/v1/namespaces/kube-system/pods/heapster-v1.6.0-beta.1-fdc7fd478-8s998: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:10.971230 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:11.972337 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:12.973637 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:13.975024 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:14.976582 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:16.063760 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: getsockopt: connection refused
ERROR: logging before flag.Parse: E0418 23:30:27.065693 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0418 23:30:30.077159 1 nanny_lib.go:95] Error while querying apiserver for resources: Get https://10.11.240.1:443/api/v1/namespaces/kube-system/pods/heapster-v1.6.0-beta.1-fdc7fd478-8s998: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0418 23:30:59.778560 1 reflector.go:205] k8s.io/autoscaler/addon-resizer/nanny/kubernetes_client.go:107: Failed to list *v1.Node: Get https://10.11.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
and also in heapster container
I0423 07:02:10.765134 1 heapster.go:113] Starting heapster on port 8082
W0423 07:16:27.975467 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:16:43.064110 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:20:36.875359 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:20:44.383790 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:22:29.683060 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:22:40.278962 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
W0423 07:31:27.072711 1 manager.go:152] Failed to get all responses in time (got 2/3)
W0423 07:31:54.580031 1 manager.go:107] Failed to get kubelet_summary:10.128.0.49:10255 response in time
How can I fix this?
Any additional info that I should provide?
monitoring kubernetes google-kubernetes-engine metrics
monitoring kubernetes google-kubernetes-engine metrics
asked Apr 23 at 8:04
Korjavin IvanKorjavin Ivan
1,36221638
asked Apr 23 at 8:04
Korjavin IvanKorjavin Ivan
1,36221638
asked Apr 23 at 8:04
Korjavin IvanKorjavin Ivan
1,36221638
asked Apr 23 at 8:04
Korjavin IvanKorjavin Ivan
1,36221638
asked Apr 23 at 8:04
Korjavin IvanKorjavin Ivan
1,36221638
1,36221638
heapter is deprecated and you need to use metric server
– c4f4t0r
Apr 23 at 8:50
there is metric-server is well, but kubectl top doesn't work still.
– Korjavin Ivan
Apr 23 at 9:32
Can you please confirm, if you still observe these errors in your cluster ?
– Nepomucen
Apr 23 at 14:24
yes, I still observe these errors
– Korjavin Ivan
Apr 23 at 18:30
add a comment |
heapter is deprecated and you need to use metric server
– c4f4t0r
Apr 23 at 8:50
there is metric-server is well, but kubectl top doesn't work still.
– Korjavin Ivan
Apr 23 at 9:32
Can you please confirm, if you still observe these errors in your cluster ?
– Nepomucen
Apr 23 at 14:24
yes, I still observe these errors
– Korjavin Ivan
Apr 23 at 18:30
heapter is deprecated and you need to use metric server
– c4f4t0r
Apr 23 at 8:50
heapter is deprecated and you need to use metric server
– c4f4t0r
Apr 23 at 8:50
there is metric-server is well, but kubectl top doesn't work still.
– Korjavin Ivan
Apr 23 at 9:32
there is metric-server is well, but kubectl top doesn't work still.
– Korjavin Ivan
Apr 23 at 9:32
Can you please confirm, if you still observe these errors in your cluster ?
– Nepomucen
Apr 23 at 14:24
Can you please confirm, if you still observe these errors in your cluster ?
– Nepomucen
Apr 23 at 14:24
yes, I still observe these errors
– Korjavin Ivan
Apr 23 at 18:30
yes, I still observe these errors
– Korjavin Ivan
Apr 23 at 18:30
add a comment |
2 Answers
2
active
oldest
votes
Heapster Deprecation
Heapster is a deprecated project and may have problems when running in recent Kubernetes versions.
See Heapster Deprecation Timeline:
| Kubernetes Release | Action | Policy/Support |
|---------------------|---------------------|----------------------------------------------------------------------------------|
| Kubernetes 1.11 | Initial Deprecation | No new features or sinks are added. Bugfixes may be made. |
| Kubernetes 1.12 | Setup Removal | The optional to install Heapster via the Kubernetes setup script is removed. |
| Kubernetes 1.13 | Removal | No new bugfixes will be made. Move to kubernetes-retired organization. |
Since Kubernetes v1.10, the kubectl top relies on metrics-server by default.
CHANGELOG-1.10.md:
- Support metrics API in
kubectl topcommands. (#56206, @brancz)
This PR implements support for the
kubectl topcommands to use the
metrics-server as an aggregated API, instead of requesting the metrics
from heapster directly. If themetrics.k8s.ioAPI is not served by the
apiserver, then this still falls back to the previous behavior.
What you should do:
As of Heapster is deprecated, and you already have a metrics-server deployed, the best option is to use a kubectl version v1.10 or above, as it fetches the metrics from metrics-server.
However, beware of kubectl Version Skew Policy:
kubectlis supported within one minor version (older or newer) of
kube-apiserver
Check your kube-apiserver version before choosing your kubectl version.
answered Apr 24 at 12:44
Eduardo BaitelloEduardo Baitello
17910
New contributor
Eduardo Baitello is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1
Yes, thank you, ugrading kubectl fixed the problem.
– Korjavin Ivan
Apr 25 at 4:15
Thank you for straightening it out
– Nepomucen
Apr 25 at 7:55
add a comment |
I guess your issue might be related to auto-upgrade of your GKE's Master nodes.
Mine got upgraded recently to v1.11.8-gke.6, and during the upgrade, I observed the same intermittent errors inside heapster-nanny container:
(error code: E0418)
For me, the problem no longer persists, and I can safely get the nodes' metrics with kubectl.
edited Apr 24 at 13:09
Eduardo Baitello
17910
answered Apr 23 at 14:24
NepomucenNepomucen
1763
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "2"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f964167%2fkubectl-top-node-doesnt-work-looks-like-an-issue-with-heapster%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Heapster Deprecation
Heapster is a deprecated project and may have problems when running in recent Kubernetes versions.
See Heapster Deprecation Timeline:
| Kubernetes Release | Action | Policy/Support |
|---------------------|---------------------|----------------------------------------------------------------------------------|
| Kubernetes 1.11 | Initial Deprecation | No new features or sinks are added. Bugfixes may be made. |
| Kubernetes 1.12 | Setup Removal | The optional to install Heapster via the Kubernetes setup script is removed. |
| Kubernetes 1.13 | Removal | No new bugfixes will be made. Move to kubernetes-retired organization. |
Since Kubernetes v1.10, the kubectl top relies on metrics-server by default.
CHANGELOG-1.10.md:
- Support metrics API in
kubectl topcommands. (#56206, @brancz)
This PR implements support for the
kubectl topcommands to use the
metrics-server as an aggregated API, instead of requesting the metrics
from heapster directly. If themetrics.k8s.ioAPI is not served by the
apiserver, then this still falls back to the previous behavior.
What you should do:
As of Heapster is deprecated, and you already have a metrics-server deployed, the best option is to use a kubectl version v1.10 or above, as it fetches the metrics from metrics-server.
However, beware of kubectl Version Skew Policy:
kubectlis supported within one minor version (older or newer) of
kube-apiserver
Check your kube-apiserver version before choosing your kubectl version.
answered Apr 24 at 12:44
Eduardo BaitelloEduardo Baitello
17910
New contributor
Eduardo Baitello is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1
Yes, thank you, ugrading kubectl fixed the problem.
– Korjavin Ivan
Apr 25 at 4:15
Thank you for straightening it out
– Nepomucen
Apr 25 at 7:55
add a comment |
Heapster Deprecation
Heapster is a deprecated project and may have problems when running in recent Kubernetes versions.
See Heapster Deprecation Timeline:
| Kubernetes Release | Action | Policy/Support |
|---------------------|---------------------|----------------------------------------------------------------------------------|
| Kubernetes 1.11 | Initial Deprecation | No new features or sinks are added. Bugfixes may be made. |
| Kubernetes 1.12 | Setup Removal | The optional to install Heapster via the Kubernetes setup script is removed. |
| Kubernetes 1.13 | Removal | No new bugfixes will be made. Move to kubernetes-retired organization. |
Since Kubernetes v1.10, the kubectl top relies on metrics-server by default.
CHANGELOG-1.10.md:
- Support metrics API in
kubectl topcommands. (#56206, @brancz)
This PR implements support for the
kubectl topcommands to use the
metrics-server as an aggregated API, instead of requesting the metrics
from heapster directly. If themetrics.k8s.ioAPI is not served by the
apiserver, then this still falls back to the previous behavior.
What you should do:
As of Heapster is deprecated, and you already have a metrics-server deployed, the best option is to use a kubectl version v1.10 or above, as it fetches the metrics from metrics-server.
However, beware of kubectl Version Skew Policy:
kubectlis supported within one minor version (older or newer) of
kube-apiserver
Check your kube-apiserver version before choosing your kubectl version.
answered Apr 24 at 12:44
Eduardo BaitelloEduardo Baitello
17910
New contributor
Eduardo Baitello is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1
Yes, thank you, ugrading kubectl fixed the problem.
– Korjavin Ivan
Apr 25 at 4:15
Thank you for straightening it out
– Nepomucen
Apr 25 at 7:55
add a comment |
Heapster Deprecation
Heapster is a deprecated project and may have problems when running in recent Kubernetes versions.
See Heapster Deprecation Timeline:
| Kubernetes Release | Action | Policy/Support |
|---------------------|---------------------|----------------------------------------------------------------------------------|
| Kubernetes 1.11 | Initial Deprecation | No new features or sinks are added. Bugfixes may be made. |
| Kubernetes 1.12 | Setup Removal | The optional to install Heapster via the Kubernetes setup script is removed. |
| Kubernetes 1.13 | Removal | No new bugfixes will be made. Move to kubernetes-retired organization. |
Since Kubernetes v1.10, the kubectl top relies on metrics-server by default.
CHANGELOG-1.10.md:
- Support metrics API in
kubectl topcommands. (#56206, @brancz)
This PR implements support for the
kubectl topcommands to use the
metrics-server as an aggregated API, instead of requesting the metrics
from heapster directly. If themetrics.k8s.ioAPI is not served by the
apiserver, then this still falls back to the previous behavior.
What you should do:
As of Heapster is deprecated, and you already have a metrics-server deployed, the best option is to use a kubectl version v1.10 or above, as it fetches the metrics from metrics-server.
However, beware of kubectl Version Skew Policy:
kubectlis supported within one minor version (older or newer) of
kube-apiserver
Check your kube-apiserver version before choosing your kubectl version.
answered Apr 24 at 12:44
Eduardo BaitelloEduardo Baitello
17910
New contributor
Eduardo Baitello is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Heapster Deprecation
Heapster is a deprecated project and may have problems when running in recent Kubernetes versions.
See Heapster Deprecation Timeline:
| Kubernetes Release | Action | Policy/Support |
|---------------------|---------------------|----------------------------------------------------------------------------------|
| Kubernetes 1.11 | Initial Deprecation | No new features or sinks are added. Bugfixes may be made. |
| Kubernetes 1.12 | Setup Removal | The optional to install Heapster via the Kubernetes setup script is removed. |
| Kubernetes 1.13 | Removal | No new bugfixes will be made. Move to kubernetes-retired organization. |
Since Kubernetes v1.10, the kubectl top relies on metrics-server by default.
CHANGELOG-1.10.md:
- Support metrics API in
kubectl topcommands. (#56206, @brancz)
This PR implements support for the
kubectl topcommands to use the
metrics-server as an aggregated API, instead of requesting the metrics
from heapster directly. If themetrics.k8s.ioAPI is not served by the
apiserver, then this still falls back to the previous behavior.
What you should do:
As of Heapster is deprecated, and you already have a metrics-server deployed, the best option is to use a kubectl version v1.10 or above, as it fetches the metrics from metrics-server.
However, beware of kubectl Version Skew Policy:
kubectlis supported within one minor version (older or newer) of
kube-apiserver
Check your kube-apiserver version before choosing your kubectl version.
answered Apr 24 at 12:44
Eduardo BaitelloEduardo Baitello
17910
New contributor
Eduardo Baitello is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered Apr 24 at 12:44
Eduardo BaitelloEduardo Baitello
17910
New contributor
Eduardo Baitello is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered Apr 24 at 12:44
Eduardo BaitelloEduardo Baitello
17910
answered Apr 24 at 12:44
Eduardo BaitelloEduardo Baitello
17910
17910
New contributor
Eduardo Baitello is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Eduardo Baitello is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Eduardo Baitello is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1
Yes, thank you, ugrading kubectl fixed the problem.
– Korjavin Ivan
Apr 25 at 4:15
Thank you for straightening it out
– Nepomucen
Apr 25 at 7:55
add a comment |
1
Yes, thank you, ugrading kubectl fixed the problem.
– Korjavin Ivan
Apr 25 at 4:15
Thank you for straightening it out
– Nepomucen
Apr 25 at 7:55
1
1
Yes, thank you, ugrading kubectl fixed the problem.
– Korjavin Ivan
Apr 25 at 4:15
Yes, thank you, ugrading kubectl fixed the problem.
– Korjavin Ivan
Apr 25 at 4:15
Thank you for straightening it out
– Nepomucen
Apr 25 at 7:55
Thank you for straightening it out
– Nepomucen
Apr 25 at 7:55
add a comment |
I guess your issue might be related to auto-upgrade of your GKE's Master nodes.
Mine got upgraded recently to v1.11.8-gke.6, and during the upgrade, I observed the same intermittent errors inside heapster-nanny container:
(error code: E0418)
For me, the problem no longer persists, and I can safely get the nodes' metrics with kubectl.
edited Apr 24 at 13:09
Eduardo Baitello
17910
answered Apr 23 at 14:24
NepomucenNepomucen
1763
add a comment |
I guess your issue might be related to auto-upgrade of your GKE's Master nodes.
Mine got upgraded recently to v1.11.8-gke.6, and during the upgrade, I observed the same intermittent errors inside heapster-nanny container:
(error code: E0418)
For me, the problem no longer persists, and I can safely get the nodes' metrics with kubectl.
edited Apr 24 at 13:09
Eduardo Baitello
17910
answered Apr 23 at 14:24
NepomucenNepomucen
1763
add a comment |
I guess your issue might be related to auto-upgrade of your GKE's Master nodes.
Mine got upgraded recently to v1.11.8-gke.6, and during the upgrade, I observed the same intermittent errors inside heapster-nanny container:
(error code: E0418)
For me, the problem no longer persists, and I can safely get the nodes' metrics with kubectl.
edited Apr 24 at 13:09
Eduardo Baitello
17910
answered Apr 23 at 14:24
NepomucenNepomucen
1763
I guess your issue might be related to auto-upgrade of your GKE's Master nodes.
Mine got upgraded recently to v1.11.8-gke.6, and during the upgrade, I observed the same intermittent errors inside heapster-nanny container:
(error code: E0418)
For me, the problem no longer persists, and I can safely get the nodes' metrics with kubectl.
edited Apr 24 at 13:09
Eduardo Baitello
17910
answered Apr 23 at 14:24
NepomucenNepomucen
1763
edited Apr 24 at 13:09
Eduardo Baitello
17910
edited Apr 24 at 13:09
Eduardo Baitello
17910
edited Apr 24 at 13:09
Eduardo Baitello
17910
17910
answered Apr 23 at 14:24
NepomucenNepomucen
1763
answered Apr 23 at 14:24
NepomucenNepomucen
1763
answered Apr 23 at 14:24
NepomucenNepomucen
1763
1763
add a comment |
add a comment |
Thanks for contributing an answer to Server Fault!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f964167%2fkubectl-top-node-doesnt-work-looks-like-an-issue-with-heapster%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
heapter is deprecated and you need to use metric server
– c4f4t0r
Apr 23 at 8:50
there is metric-server is well, but kubectl top doesn't work still.
– Korjavin Ivan
Apr 23 at 9:32
Can you please confirm, if you still observe these errors in your cluster ?
– Nepomucen
Apr 23 at 14:24
yes, I still observe these errors
– Korjavin Ivan
Apr 23 at 18:30