CentOS7二进制安装包方式部署K8S集群之ETCD集群部署
目录
1、集群各节点创建etcd相关目录
(1)、K8s-controller节点
(2)、K8s-node01节点
(3)、K8s-node02节点
2、创建etcd集群服务证书和私钥
(1)、创建etcd集群服务证书签名请求文件
(2)、签发etcd-server证书文件和私钥文件
(4)、分发etcd证书文件至etcd集群各节点
1)分发到K8s-controller节点
2)分发到K8s-node01节点
3)分发到K8s-node02节点
3、下载etcd安装文件
(1)、下载etcd安装文件
(2)、分发etcd安装文件
1)分发到K8s-node01节点
2)分发到K8s-node02节点
4、部署集群各节点etcd服务
(1)、K8s-controller节点
1)解压etcd安装文件
2)创建etcd服务配置文件
3)创建etcd.service文件
(2)、K8s-node01节点
1)解压etcd安装文件
2)创建etcd服务配置文件
3)创建etcd.service文件
(3)、K8s-node02节点
1)解压etcd安装文件
2)创建etcd服务配置文件
3)创建etcd.service文件
5、启动集群各节点etcd服务
(1)、启动K8s-controller节点etcd服务
1)设置开机启动etcd服务
2)启动etcd服务
(2)、启动K8s-node01节点etcd服务
1)设置开机启动etcd服务
2)启动etcd服务
(3)、启动K8s-node02节点etcd服务
1)设置开机启动etcd服务
2)启动etcd服务
6、验证etcd集群状态
(1)、查看etcd集群健康状态
(2)、查看etcd集群LEADER节点
更多精彩博文详见:
《Linux系统应用运维》专栏总目录(持续更新)
《CentOS二进制安装包方式部署K8S集群之系统初始化》
《CentOS7二进制安装包方式部署K8S集群之CA根证书生成》
《CentOS7二进制安装包方式部署K8S集群之ETCD集群部署》
本文主要介绍了在CentOS 7系统下,使用二进制安装包方式部署K8S集群中etcd集群部署的相关内容。本次Kubernetes集群使用分布式键值存储系统etcd进行数据存储,其中etcd采取三节点集群方式部署。为了节约服务器资源,etcd集群节点复用Kubernetes集群节点。
etcd集群节点规划如下:
主机名称 | IP | 部署组件 |
K8s-controller | 192.168.110.150 | etcd-01 |
K8s-node01 | 192.168.110.151 | etcd-02 |
K8s-node02 | 192.168.110.152 | etcd-03 |
1、集群各节点创建etcd相关目录
此处集群各节点分别创建如下目录:
- /K8s/etcd/bin:该目录下为etcd服务相关的可执行文件。
- /K8s/etcd/data:该目录为etcd服务的数据文件目录。
- /K8s/etcd/ssl:该目录下为etcd服务相关的证书文件。
(1)、K8s-controller节点
[root@K8s-controller ~]# mkdir -p /K8s/etcd/{bin,data,ssl}
[root@K8s-controller ~]# ll /K8s/etcd/
(2)、K8s-node01节点
[root@K8s-node01 ~]# mkdir -p /K8s/etcd/{bin,data,ssl}
[root@K8s-node01 ~]# ll /K8s/etcd/
(3)、K8s-node02节点
[root@K8s-node02 ~]# mkdir -p /K8s/etcd/{bin,data,ssl}
[root@K8s-node02 ~]# ll /K8s/etcd/
2、创建etcd集群服务证书和私钥
(1)、创建etcd集群服务证书签名请求文件
- 创建etcd集群服务证书文件存放目录
[root@K8s-controller ~]# mkdir -pv /K8s/cfssl/cert_file/etcd
- 创建etcd集群服务证书签名请求文件etcd-csr.json
[root@K8s-controller ~]# cat > /K8s/cfssl/cert_file/etcd/etcd-server-csr.json <<EOF
{
"CN": "etcd-server",
"hosts": [
"192.168.110.150",
"192.168.110.151",
"192.168.110.152"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Nanning",
"ST": "Guangxi",
"O": "k8s",
"OU": "lbj"
}
]
}
EOF
hosts字段是etcd集群节点IP地址。
(2)、签发etcd-server证书文件和私钥文件
使用etcd集群CA根证书、CA根证书私钥、CA根证书配置文件、etcd-server证书签名请求json文件共同签发etcd-server证书文件和私钥文件。此处会生成三个文件,分别是证书etcd-server.pem、证书私钥etcd-server-key.pem、证书签名请求etcd-server.csr(用于交叉签名或重新签名)。
[root@K8s-controller ~]# cd /K8s/cfssl/cert_file/etcd/
[root@K8s-controller etcd]# cfssl gencert -ca=/K8s/cfssl/cert_file/etcd-ca.pem -ca-key=/K8s/cfssl/cert_file/etcd-ca-key.pem -config=/K8s/cfssl/cert_file/etcd-ca-config.json -profile=etcd /K8s/cfssl/cert_file/etcd/etcd-server-csr.json | cfssljson -bare etcd-server
(4)、分发etcd证书文件至etcd集群各节点
此处分发etcd集群CA根证书文件etcd-ca.pem、etcd集群服务证书文件etcd-server.pem、etcd私钥文件etcd-server-key.pem。
1)分发到K8s-controller节点
[root@K8s-controller ~]# cp /K8s/cfssl/cert_file/etcd/etcd-server.pem /K8s/cfssl/cert_file/etcd/etcd-server-key.pem /K8s/cfssl/cert_file/etcd-ca.pem /K8s/etcd/ssl/
[root@K8s-controller ~]# ll /K8s/etcd/ssl/
2)分发到K8s-node01节点
- K8s-controller节点分发
[root@K8s-controller ~]# scp -P 22 /K8s/cfssl/cert_file/etcd/etcd-server.pem /K8s/cfssl/cert_file/etcd/etcd-server-key.pem /K8s/cfssl/cert_file/etcd-ca.pem root@192.168.110.151:/K8s/etcd/ssl/
- K8s-node01节点查看
[root@K8s-node01 ~]# ll /K8s/etcd/ssl/
3)分发到K8s-node02节点
- K8s-controller节点分发
[root@K8s-controller ~]# scp -P 22 /K8s/cfssl/cert_file/etcd/etcd-server.pem /K8s/cfssl/cert_file/etcd/etcd-server-key.pem /K8s/cfssl/cert_file/etcd-ca.pem root@192.168.110.152:/K8s/etcd/ssl/
- K8s-node02节点查看
[root@K8s-node02 ~]# ll /K8s/etcd/ssl/
3、下载etcd安装文件
(1)、下载etcd安装文件
此处部署的是v3.4.18-linux-amd64版本,可根据实际环境选择相应的版本。
[root@K8s-controller ~]# cd /K8s/etcd/
[root@K8s-controller etcd]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.18/etcd-v3.4.18-linux-amd64.tar.gz
(2)、分发etcd安装文件
1)分发到K8s-node01节点
[root@K8s-controller ~]# scp -P 22 /K8s/etcd/etcd-v3.4.18-linux-amd64.tar.gz root@192.168.110.151:/K8s/etcd/
[root@K8s-node01 ~]# ll /K8s/etcd/
2)分发到K8s-node02节点
[root@K8s-controller ~]# scp -P 22 /K8s/etcd/etcd-v3.4.18-linux-amd64.tar.gz root@192.168.110.152:/K8s/etcd/
[root@K8s-node02 ~]# ll /K8s/etcd/
4、部署集群各节点etcd服务
(1)、K8s-controller节点
1)解压etcd安装文件
- 解压etcd安装文件
[root@K8s-controller ~]# cd /K8s/etcd/
[root@K8s-controller etcd]# tar -xzvf etcd-v3.4.18-linux-amd64.tar.gz
- 复制解压后的可执行文件到etcd的bin目录
[root@K8s-controller etcd]# cp /K8s/etcd/etcd-v3.4.18-linux-amd64/{etcd,etcdctl} /K8s/etcd/bin/
[root@K8s-controller etcd]# ll /K8s/etcd/bin/
- 创建etcdctl可执行文件软链接
[root@K8s-controller etcd]# ln -s /K8s/etcd/bin/etcdctl /usr/bin/etcdctl
[root@K8s-controller etcd]# ll /usr/bin/etcdctl
2)创建etcd服务配置文件
此处创建etcd服务配置文件/K8s/etcd/etcd.conf
[root@K8s-controller ~]# cat > /K8s/etcd/etcd.conf << EOF
#[member]
# ETCD_NAME:节点名称,集群中唯一,此处指集群节点01
ETCD_NAME="etcd-01"
# ETCD_DATA_DIR:当前集群节点的数据目录
ETCD_DATA_DIR="/K8s/etcd/data"
# ETCD_LISTEN_PEER_URLS:集群通信的监听地址
ETCD_LISTEN_PEER_URLS="https://192.168.110.150:2380"
# ETCD_LISTEN_CLIENT_URLS:客户端访问的监听地址
ETCD_LISTEN_CLIENT_URLS="https://192.168.110.150:2379,http://127.0.0.1:2379"#[cluster]
# ETCD_INITIAL_ADVERTISE_PEER_URLS:集群的通告地址
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.110.150:2380"
# ETCD_ADVERTISE_CLIENT_URLS:客户端的通告地址
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.110.150:2379"
# ETCD_INITIAL_CLUSTER:集群的节点地址
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.110.150:2380,etcd-02=https://192.168.110.151:2380,etcd-03=https://192.168.110.152:2380"
# ETCD_INITIAL_CLUSTER_TOKEN:集群Token
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
# ETCD_INITIAL_CLUSTER_STATE:即将加入的集群的当前状态,new是新建的集群,existing表示加入已存在的集群。
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
3)创建etcd.service文件
创建etcd.service文件,通过systemd来管理etcd服务。
[root@K8s-controller ~]# cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target[Service]
Type=notify
EnvironmentFile=/K8s/etcd/etcd.confExecStart=/K8s/etcd/bin/etcd --cert-file=/K8s/etcd/ssl/etcd-server.pem --key-file=/K8s/etcd/ssl/etcd-server-key.pem --peer-cert-file=/K8s/etcd/ssl/etcd-server.pem --peer-key-file=/K8s/etcd/ssl/etcd-server-key.pem --trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --peer-trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --logger=zap
Restart=on-failure
LimitNOFILE=65536[Install]
WantedBy=multi-user.targetEOF
(2)、K8s-node01节点
1)解压etcd安装文件
- 解压etcd安装文件
[root@K8s-node01 ~]# cd /K8s/etcd/
[root@K8s-node01 etcd]# tar -xzvf etcd-v3.4.18-linux-amd64.tar.gz
- 复制解压后的可执行文件到etcd的bin目录
[root@K8s-node01 etcd]# cp /K8s/etcd/etcd-v3.4.18-linux-amd64/{etcd,etcdctl} /K8s/etcd/bin/
[root@K8s-node01 etcd]# ll /K8s/etcd/bin/
- 创建etcdctl可执行文件软链接
[root@K8s-node01 etcd]# ln -s /K8s/etcd/bin/etcdctl /usr/bin/etcdctl
[root@K8s-node01 etcd]# ll /usr/bin/etcdctl
2)创建etcd服务配置文件
此处创建etcd服务配置文件/K8s/etcd/etcd.conf
[root@K8s-node01 ~]# cat > /K8s/etcd/etcd.conf << EOF
#[member]
# ETCD_NAME:节点名称,集群中唯一,此处指集群节点01
ETCD_NAME="etcd-02"
# ETCD_DATA_DIR:当前集群节点的数据目录
ETCD_DATA_DIR="/K8s/etcd/data"
# ETCD_LISTEN_PEER_URLS:集群通信的监听地址
ETCD_LISTEN_PEER_URLS="https://192.168.110.151:2380"
# ETCD_LISTEN_CLIENT_URLS:客户端访问的监听地址
ETCD_LISTEN_CLIENT_URLS="https://192.168.110.151:2379,http://127.0.0.1:2379"#[cluster]
# ETCD_INITIAL_ADVERTISE_PEER_URLS:集群的通告地址
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.110.151:2380"
# ETCD_ADVERTISE_CLIENT_URLS:客户端的通告地址
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.110.151:2379"
# ETCD_INITIAL_CLUSTER:集群的节点地址
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.110.150:2380,etcd-02=https://192.168.110.151:2380,etcd-03=https://192.168.110.152:2380"
# ETCD_INITIAL_CLUSTER_TOKEN:集群Token
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
# ETCD_INITIAL_CLUSTER_STATE:即将加入的集群的当前状态,new是新建的集群,existing表示加入已存在的集群。
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
3)创建etcd.service文件
创建etcd.service文件,通过systemd来管理etcd服务。
[root@K8s-node01 ~]# cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target[Service]
Type=notify
EnvironmentFile=/K8s/etcd/etcd.conf
ExecStart=/K8s/etcd/bin/etcd --cert-file=/K8s/etcd/ssl/etcd-server.pem --key-file=/K8s/etcd/ssl/etcd-server-key.pem --peer-cert-file=/K8s/etcd/ssl/etcd-server.pem --peer-key-file=/K8s/etcd/ssl/etcd-server-key.pem --trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --peer-trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --logger=zap
Restart=on-failure
LimitNOFILE=65536[Install]
WantedBy=multi-user.targetEOF
(3)、K8s-node02节点
1)解压etcd安装文件
- 解压etcd安装文件
[root@K8s-node02 ~]# cd /K8s/etcd/
[root@K8s-node02 etcd]# tar -xzvf etcd-v3.4.18-linux-amd64.tar.gz
- 复制解压后的可执行文件到etcd的bin目录
[root@K8s-node02 etcd]# cp /K8s/etcd/etcd-v3.4.18-linux-amd64/{etcd,etcdctl} /K8s/etcd/bin/
[root@K8s-node02 etcd]# ll /K8s/etcd/bin/
- 创建etcdctl可执行文件软链接
[root@K8s-node02 etcd]# ln -s /K8s/etcd/bin/etcdctl /usr/bin/etcdctl
[root@K8s-node02 etcd]# ll /usr/bin/etcdctl
2)创建etcd服务配置文件
此处创建etcd服务配置文件/K8s/etcd/etcd.conf
[root@K8s-node02 ~]# cat > /K8s/etcd/etcd.conf << EOF
#[member]
# ETCD_NAME:节点名称,集群中唯一,此处指集群节点01
ETCD_NAME="etcd-03"
# ETCD_DATA_DIR:当前集群节点的数据目录
ETCD_DATA_DIR="/K8s/etcd/data"
# ETCD_LISTEN_PEER_URLS:集群通信的监听地址
ETCD_LISTEN_PEER_URLS="https://192.168.110.152:2380"
# ETCD_LISTEN_CLIENT_URLS:客户端访问的监听地址
ETCD_LISTEN_CLIENT_URLS="https://192.168.110.152:2379,http://127.0.0.1:2379"#[cluster]
# ETCD_INITIAL_ADVERTISE_PEER_URLS:集群的通告地址
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.110.152:2380"
# ETCD_ADVERTISE_CLIENT_URLS:客户端的通告地址
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.110.152:2379"
# ETCD_INITIAL_CLUSTER:集群的节点地址
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.110.150:2380,etcd-02=https://192.168.110.151:2380,etcd-03=https://192.168.110.152:2380"
# ETCD_INITIAL_CLUSTER_TOKEN:集群Token
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
# ETCD_INITIAL_CLUSTER_STATE:即将加入的集群的当前状态,new是新建的集群,existing表示加入已存在的集群。
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
3)创建etcd.service文件
创建etcd.service文件,通过systemd来管理etcd服务。
[root@K8s-node02 ~]# cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target[Service]
Type=notify
EnvironmentFile=/K8s/etcd/etcd.conf
ExecStart=/K8s/etcd/bin/etcd --cert-file=/K8s/etcd/ssl/etcd-server.pem --key-file=/K8s/etcd/ssl/etcd-server-key.pem --peer-cert-file=/K8s/etcd/ssl/etcd-server.pem --peer-key-file=/K8s/etcd/ssl/etcd-server-key.pem --trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --peer-trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --logger=zap
Restart=on-failure
LimitNOFILE=65536[Install]
WantedBy=multi-user.targetEOF
5、启动集群各节点etcd服务
此处需要注意的是,至少同时启动两个集群节点,否则会有集群节点连接失败的信息,导致集群启动失败。
当集群无法正常启动时,可以使用命令“journalctl -xeu etcd.service”或者命令“journalctl -u etcd.service”查看启动失败的原因。
(1)、启动K8s-controller节点etcd服务
1)设置开机启动etcd服务
[root@K8s-controller ~]# systemctl enable etcd
2)启动etcd服务
[root@K8s-controller ~]# systemctl start etcd.service
[root@K8s-controller ~]# systemctl status etcd.service
[root@K8s-controller ~]# netstat -tnlp
(2)、启动K8s-node01节点etcd服务
1)设置开机启动etcd服务
[root@K8s-node01 ~]# systemctl enable etcd
2)启动etcd服务
[root@K8s-node01 ~]# systemctl start etcd.service
[root@K8s-node01 ~]# systemctl status etcd.service
[root@K8s-node01 ~]# netstat -tnlp
(3)、启动K8s-node02节点etcd服务
1)设置开机启动etcd服务
[root@K8s-node02 ~]# systemctl enable etcd
2)启动etcd服务
[root@K8s-node02 ~]# systemctl start etcd.service
[root@K8s-node02 ~]# systemctl status etcd.service
[root@K8s-node02 data]# netstat -tnlp
6、验证etcd集群状态
(1)、查看etcd集群健康状态
[root@K8s-controller ~]# ETCDCTL_API=3 etcdctl --cacert=/K8s/etcd/ssl/etcd-ca.pem --cert=/K8s/etcd/ssl/etcd-server.pem --key=/K8s/etcd/ssl/etcd-server-key.pem --endpoints="https://192.168.110.150:2379,https://192.168.110.151:2379,https://192.168.110.152:2379" endpoint health
由上可见,etcd三个集群节点状态均为健康。
(2)、查看etcd集群LEADER节点
[root@K8s-controller ~]# ETCDCTL_API=3 etcdctl -w table --cacert=/K8s/etcd/ssl/etcd-ca.pem --cert=/K8s/etcd/ssl/etcd-server.pem --key=/K8s/etcd/ssl/etcd-server-key.pem --endpoints="https://192.168.110.150:2379,https://192.168.110.151:2379,https://192.168.110.152:2379" endpoint status
由上可见,192.168.110.151节点(K8s-controller)为LEADER节点