k8s etcd 运行错误 failed to find plugin “flannel“ in path [/usr/lib/cni]

表现形式为 coredns 一直处于 ContainerCreating 状态

root@master-node:~# kubectl -n kube-system get pod
NAME                                  READY   STATUS              RESTARTS   AGE
coredns-66bc5c9577-bg6r2              0/1     ContainerCreating   0          4m19s
coredns-66bc5c9577-zgrxl              0/1     ContainerCreating   0          4m19s
etcd-master-node                      1/1     Running             0          4m28s
kube-apiserver-master-node            1/1     Running             0          4m28s
kube-controller-manager-master-node   1/1     Running             0          4m28s
kube-proxy-k28lq                      1/1     Running             0          4m20s
kube-scheduler-master-node            1/1     Running             0          4m28s

describe 可以看到一些错误的事件

Events:Type     Reason                  Age                 From               Message----     ------                  ----                ----               -------Warning  FailedScheduling        4m33s               default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.Normal   Scheduled               97s                 default-scheduler  Successfully assigned kube-system/coredns-66bc5c9577-bg6r2 to master-nodeWarning  FailedCreatePodSandBox  96s                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9d01df1e999793e5efa68228832ca7288668866636f933b46db5db7cae5a264d": plugin type="flannel" failed (add): failed to find plugin "flannel" in path [/usr/lib/cni]

检查发现 /usr/lib/cni 目录下确实没有 flannel 插件，查询之后看到 flannel 在 /opt/cni/bin 位置，这个可能就是问题的根源，我并没有明确安装部署 flannel 可能是其他服务无意间下载到了这个目录

root@master-node:/usr/lib/cni# ls
bandwidth  bridge  dhcp  firewall  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sbr  static  tuning  vlan  vrf
r

root@master-node:/opt/cni/bin# ls
bandwidth  bridge  dhcp  dummy  firewall  flannel  host-device  host-local  ipvlan  LICENSE  loopback  macvlan  portmap  ptp  README.md  sbr  static  tap  tuning  vlan  vrf
r

解决方案创建符号链接

sudo ln -sf /opt/cni/bin/* /usr/lib/cni/

如果你没有 cni，还是需要下载的。