rancher使用rke在华为云多网卡的服务器上安装k8s集群问题处理
报错:
问题：
[[network] Host [192.168.0.213] is not able to connect to the following ports: [192.168.0.213:2379]. Please check network policies and firewall rules]
问题：
root@hwy-isms-210-66:~# gotelnet 172.17.210.66 2379
map[2379:failed]
root@hwy-isms-210-66:~# gotelnet 127.0.0.1 2379
map[2379:success]
root@hwy-isms-210-66:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b6f75ff566d5 rancher/rke-tools:v0.1.96 "/docker-entrypoint.…" 6 hours ago Up 6 hours 80/tcp, 0.0.0.0:10250->1337/tcp rke-worker-port-listener
ac3e20c949df rancher/rke-tools:v0.1.96 "/docker-entrypoint.…" 6 hours ago Up 6 hours 80/tcp, 0.0.0.0:6443->1337/tcp rke-cp-port-listener
e106814143a3 rancher/rke-tools:v0.1.96 "/docker-entrypoint.…" 6 hours ago Up 6 hours 80/tcp, 0.0.0.0:2379->1337/tcp, 0.0.0.0:2380->1337/tcp rke-etcd-port-listener
6a866546f8bb rancher/rancher-agent:v2.8.5 "run.sh --server htt…" 6 hours ago Up 6 hours peaceful_albattani
9bbffd35d9a4 rancher/rancher-agent:v2.8.5 "run.sh --server htt…" 6 hours ago Up 6 hours confident_fermi
root@hwy-isms-210-66:~# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500inet 172.18.0.1 netmask 255.255.0.0 broadcast 172.18.255.255ether a6:c3:99:d0:cf:03 txqueuelen 0 (Ethernet)RX packets 3547 bytes 100789 (98.4 KiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 86 bytes 5196 (5.0 KiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500inet 172.17.210.66 netmask 255.255.255.0 broadcast 172.17.210.255ether fa:16:3e:40:01:71 txqueuelen 1000 (Ethernet)RX packets 122941811 bytes 23935288095 (22.2 GiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 127262310 bytes 14351697946 (13.3 GiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500inet 172.17.210.67 netmask 255.255.255.0 broadcast 172.17.210.255ether fa:16:3e:40:01:72 txqueuelen 1000 (Ethernet)RX packets 207177 bytes 17420004 (16.6 MiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 202098 bytes 20182560 (19.2 MiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500inet 172.17.210.68 netmask 255.255.255.0 broadcast 172.17.210.255ether fa:16:3e:40:01:73 txqueuelen 1000 (Ethernet)RX packets 180108 bytes 15241156 (14.5 MiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 248119 bytes 22751922 (21.6 MiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536inet 127.0.0.1 netmask 255.0.0.0loop txqueuelen 1000 (Local Loopback)RX packets 1352589 bytes 102392483 (97.6 MiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 1352589 bytes 102392483 (97.6 MiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0veth13ea56c: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500ether 7a:fc:db:8f:3c:0f txqueuelen 0 (Ethernet)RX packets 59 bytes 3636 (3.5 KiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 73 bytes 4338 (4.2 KiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0veth6b767de: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500ether 7e:17:74:fd:a7:27 txqueuelen 0 (Ethernet)RX packets 3 bytes 126 (126.0 B)RX errors 0 dropped 0 overruns 0 frame 0TX packets 6 bytes 412 (412.0 B)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0vethf9165ed: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500ether f6:46:67:c2:93:2e txqueuelen 0 (Ethernet)RX packets 3 bytes 126 (126.0 B)RX errors 0 dropped 0 overruns 0 frame 0TX packets 9 bytes 538 (538.0 B)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0root@hwy-isms-210-66:~# cat /etc/rc.local
#!/bin/sh -e
# rc.local
# 开机执行的路由配置命令
ip route add default via 172.17.210.1 dev eth0 table 10
ip route add 172.17.210.0/24 dev eth0 table 10
ip rule add from 172.17.210.66 table 10
ip route add default via 172.17.210.1 dev eth1 table 20
ip route add 172.17.210.0/24 dev eth1 table 20
ip rule add from 172.17.210.67 table 20
ip route add default via 172.17.210.1 dev eth2 table 30
ip route add 172.17.210.0/24 dev eth2 table 30
ip rule add from 172.17.210.68 table 30
exit 0
root@hwy-isms-210-66:~#
为啥127.0.0.1 2379可以通，172.17.210.66 2379不通，且从同网段的服务器上是可以telnet通172.17.210.66 2379端口的。该问题的根本原因在于网络路由策略和Docker端口绑定的限制，具体分析如下：
Docker端口绑定限制：
容器端口映射配置为0.0.0.0:2379->1337/tcp，理论上应允许所有IP访问
路由策略冲突：
/etc/rc.local中配置了多网卡策略路由，强制不同源IP走不同路由表
从172.17.210.66发出的流量被ip rule add from 172.17.210.66 table 10强制路由
可能因路由表10缺少到docker0网桥(172.18.0.1/16)的路由导致不通
root@hwy-isms-210-66:~# ip route list
default via 172.17.210.1 dev eth0 proto dhcp metric 100
169.254.169.254 via 172.17.210.1 dev eth0 proto dhcp metric 100
172.17.210.0/24 dev eth0 proto kernel scope link src 172.17.210.66 metric 100
172.17.210.0/24 dev eth1 proto kernel scope link src 172.17.210.67 metric 101
172.17.210.0/24 dev eth2 proto kernel scope link src 172.17.210.68 metric 102
172.18.0.0/16 dev docker0 proto kernel scope link src 172.18.0.1
root@hwy-isms-210-66:~# ip rule list
0: from all lookup local
32763: from 172.17.210.68 lookup 30
32764: from 172.17.210.67 lookup 20
32765: from 172.17.210.66 lookup 10
32766: from all lookup main
32767: from all lookup default
root@hwy-isms-210-66:~#
root@hwy-isms-210-66:~# ip route show table 10
default via 172.17.210.1 dev eth0
172.17.210.0/24 dev eth0 scope link
root@hwy-isms-210-66:~# 
解决方案：
方案1：在路由表10中添加docker0网段路由
ip route add 172.18.0.0/16 dev docker0 table 10