Otdfbt

In a nutshell, I munged my kubeadm cluster and am trying to start from scratch (I was not able to do a proper tear down of the cluster, which is why I was forced to go this route of kubeadm reset and sudo yum remove kubeadm kubectl kubelet kubernetes-cni kube* minus the removal of the node from the cluster).

I removed all packages (running on RHEL 7) and am now trying to reinstall, but getting the error on startup:

-- Unit kubelet.service has begun starting up.
Nov 13 12:23:50 thalia2.ahc.umn.edu kubelet[31361]: F1113 12:23:50.681719 31361 server.go:190] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to
read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Nov 13 12:23:50 thalia2.ahc.umn.edu systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Nov 13 12:23:50 thalia2.ahc.umn.edu systemd[1]: Unit kubelet.service entered failed state.
Nov 13 12:23:50 thalia2.ahc.umn.edu systemd[1]: kubelet.service failed.
Nov 13 12:24:00 thalia2.ahc.umn.edu systemd[1]: kubelet.service holdoff time over, scheduling restart.
Nov 13 12:24:00 thalia2.ahc.umn.edu systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--

Every post I've seen regarding this error indicates that this file (/var/lib/kubelet/config.yaml) is missing because I did not run kubeadm init, but I am NOT creating a master node - I just want to run this like it was a fresh install as per installing-kubeadm-kubelet-and-kubectl

Am I missing something obvious? I do not remember running kubeadm init on each node before I started the kubelet service in my initial install a couple months back, plus, nowhere in the directions for installing the packages does it say to do a Kubeadm init.

Also of note: On my former master node, I am getting the error after trying a reinstall and start of kubectl:

Get https://134.84.53.162:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dthalia0.ahc.umn.edu&limit=500&resourceVersion=0: dial tcp 134.84.53.162:6443: connect: connection refused
Nov 13 13:40:40 thalia0.ahc.umn.edu kubelet[17689]: W1113 13:40:40.487124 17689 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 13 13:40:40 thalia0.ahc.umn.edu kubelet[17689]: E1113 13:40:40.487404 17689 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNot
Ready message:docker: network plugin is not ready: cni config uninitialized

but, I thought I had removed everything by doing a kubeadm reset?

I did delete the calico folder on my former master node (sudo rm -rf /var/etcd/calico-data), and then did a restart of kubectl and am getting this error now:

Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.447381 29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.547732 29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.648002 29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.748219 29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.849201 29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.950018 29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:38 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:38.050193 29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:38 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:38.150479 29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:38 thalia0.ahc.umn.edu kubelet[29725]: I1113 13:55:38.173736 29725 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 13 13:55:38 thalia0.ahc.umn.edu kubelet[29725]: I1113 13:55:38.180154 29725 kubelet_node_status.go:70] Attempting to register node thalia0.ahc.umn.edu

So, it would appear that there is "residue" from my old cluster. Is there some hidden package out there that a kubeadm reset and a sudo yum remove kubeadm kubectl kubelet kubernetes-cni kube* did not delete?

I thought I had removed everything by doing a kubeadm reset?

regarding to documentation it is not delete everything. it just reverse the changes, so where you delete the packages and just broke all paths, run kubeadm init to reinitialise cluster.

The issue ended up being that the /var directory had gotten full. Cleaned it out and now everything starts as expected.