03.搭建K8S集群

K8S集群搭建的方式

目前主流的搭建k8s集群的方式有kubeadm、minikube、二进制包三种方式：

kubeadm（本案例搭建方式）

• 是一个工具，用于快速搭建kubernetes集群，目前应该是比较方便和推荐的，简单易用

• kubeadm是Kubernetes 1.4开始新增的特性

• kubeadm init 以及 kubeadm join 这两个命令可以快速创建 kubernetes 集群

minikube

• 一般用于本地开发、测试和学习，不能用于生产环境

• 是一个工具，minikube快速搭建一个运行在本地的单节点的Kubernetes

二进制包

• 在官网下载相关的组件的二进制包，上面的两个是工具，可以快速搭建集群，也就是相当于用程序脚本帮我们装好了集群，前两者属于自动部署，简化部署操作，自动部署屏蔽了很多细节，使得对各个模块感知很少，遇到问题很难排查，如果手动安装，对kubernetes理解也会更全面。

• 目前生产环境的主流搭建方式，已在生产环境验证，kubeadm也可以搭建生产环境，不过kubeadm应该还没有被大规模在生产环境验证

搭建k8s集群

1.关闭防火墙

# 关闭每台服务器的防火墙
sudo systemctl stop firewalld
sudo systemctl disable firewalld
sudo systemctl status firewalld 

2.配置yum源

# 配置yum源
cd /etc/yum.repos.d/
mv CentOS-Base.repo CentOS-Base.repo.back
vi  CentOS-Base.repo
# 添加如下内容
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#[base]
name=CentOS-$releasever - Base - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/os/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/os/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#released updates
[updates]
name=CentOS-$releasever - Updates - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/updates/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/extras/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/centosplus/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/contrib/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/contrib/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#[base]
name=CentOS-$releasever - Base - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/os/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/os/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#released updates
[updates]
name=CentOS-$releasever - Updates - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/updates/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/extras/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/centosplus/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/contrib/$basearch/http://mirrors.aliyuncs.com/centos/$releasever/contrib/$basearch/http://mirrors.cloud.aliyuncs.com/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
# 清理缓存、重新生成缓存、列出yum源
yum clean all
yum makecache
yum repolist
# 安装nfs-utils
yum install -y nfs-utils

3.关闭swap

# 注释掉swap配置，并保存
vim /etc/fstab
# /dev/mapper/centos-swap swap           swap    defaults        0 0

重启服务器

4.设置三台服务器的hostname，并且配置host文件，方便直接使用hostname能够互相访问

# 在master节点
hostnamectl set-hostname k8s-master
# 在node1节点
hostnamectl set-hostname k8s-node1
# 在node2节点
hostnamectl set-hostname k8s-node2
# 在三台服务器分别编辑 /etc/hosts文件，增加DNS解析
# 将下方ip替换为自己的kubernetes节点ip
192.168.32.132 k8s-master
192.168.32.134 k8s-node1
192.168.32.133 k8s-node2

分别在每台服务器上ping hostname，验证是否已经配置成功

5.将桥接的IPV4流量传递到iptables的链

# 在每台服务器上编辑/etc/sysctl.d/k8s.conf，新增如下内容
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
# 使配置生效
sysctl --system
# 在每台服务器上同步时间
yum install ntpdate -y
ntpdate time.windows.com

6.安装docker20.10.7-3

# 安装必要的软件包
yum install -y yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安装 Docker Engine
sudo yum install -y docker-ce-20.10.7-3.el7 docker-ce-cli-20.10.7-3.el7 containerd.io
# 启动 Docker 服务
sudo systemctl start docker
# 设置 Docker 开机自启
sudo systemctl enable docker
# 验证 Docker 安装
docker --version
# 关闭docker cgoup
vim /etc/docker/daemon.json
# 在该json文件中添加如下
{"exec-opts": ["native.cgroupdriver=systemd"],"registry-mirrors": ["https://docker.1panelproxy.com","https://2a6bf1988cb6428c877f723ec7530dbc.mirror.swr.myhuaweicloud.com","https://docker.m.daocloud.io","https://hub-mirror.c.163.com","https://mirror.baidubce.com","https://your_preferred_mirror","https://dockerhub.icu","https://docker.registry.cyou","https://docker-cf.registry.cyou","https://dockercf.jsdelivr.fyi","https://docker.jsdelivr.fyi","https://dockertest.jsdelivr.fyi","https://mirror.aliyuncs.com","https://dockerproxy.com","https://mirror.baidubce.com","https://docker.m.daocloud.io","https://docker.nju.edu.cn","https://docker.mirrors.sjtug.sjtu.edu.cn","https://docker.mirrors.ustc.edu.cn","https://mirror.iscas.ac.cn","https://docker.rainbond.cc"]
}
# 重启docker
systemctl daemon-reload
sudo systemctl restart docker

7.添加kubernet yum源

vim /etc/yum.repos.d/kubernetes.repo[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

8.安装kubeadm、kubelet、kubectl

yum install -y kubelet-1.23.6 kubeadm-1.23.6 kubectl-1.23.6
systemctl enable kubelet

9.部署kubernet master

#在Master节点下执行
# 将192.168.177.171替换为自己的master节点ip地址
kubeadm init \
--apiserver-advertise-address=192.168.177.171 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.23.6 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=all
#安装成功后，复制如下配置并执行
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get nodes

10.加入kubernetes node

#分别在k8s-node1和k8s-node2执行一下命令，加入master节点
# 将192.168.177.171替换为自己的master节点ip地址
# 将eojlre.jrnvoav77jmiz0q0替换为自己的master节点控制台输出的toker
# 将0c5a5878f52074b695103204fdcd17b0758890ed4bcaba89c1462476684e5b1c替换为自己的master节点控制台输出的hash值
kubeadm join 192.168.177.171:6443 --token eojlre.jrnvoav77jmiz0q0 \
--discovery-token-ca-cert-hash sha256:0c5a5878f52074b695103204fdcd17b0758890ed4bcaba89c1462476684e5b1c
#如果初始化的token不小v心清空了，可以通过如下命令获取或者重新申请
#如果token已经过期，就重新申请
kubeadm token create
#token没有过期可以通过如下命令获取
kubeadm token list
#获取-discovery-token-ca-cert-hash值，得到值后需要在前面拼接上 sha256:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt openssl rsa -pubin -outform der 2>/dev/null \

k8s的下列组件未成功下载，需要配置网络

# 在master节点下创建k8s目录，下载calico.yaml文件
curl https://docs.projectcalico.org/manifests/calico.yaml -O
# 修改calico.yaml文件中的CALICO_IPV4POOL_CIDR属性的值为第6步中的--pod-network-cidr值相同

# 删除镜像docker.io/前缀，避免下载过慢导致失败
sed -i 's#docker.io/##g' calico.yaml

11.构建

# 构建应用
kubectl apply -f calico.yaml

结果如下图：

12.测试kubernet集群

# 创建部署nginx
kubectl create deployment nginx --image=nginx
# 暴露端口
kubectl expose deployment nginx --port=80 --type=NodePort
# 查看pod以及服务信息
kubectl get pod,svc
# 尝试使用curl或者浏览器访问服务信息

其他命令及问题

# 查看k8s的状态
systemctl status kubelet
# 查看日志情况
journalctl -xefu kubelet
# 重启k8s
systemctl restart kubelet
# 获取k8s的节点
kubectl get nodes
# 重启docker
systemctl restart docker
# k8s获取组件状态
kubectl get componentstatus
# k8s列出指定命名空间的pod，kubeadm在执行k8s的pod的时候会在kube-system这个命名空间下执行，所以使用kubectl get pod的时候查看不到查看的默认的pod，需要指定这个命名空间
kubectl get pods -n <namespace>
# 查看一个pod详情
kubectl describe po <pod-name> -n <namespace>

若出现如下错误，则需要关闭docker的cgroup

如果查看pod详情时出现如下错误，可能是linux内核需要升级

# 升级linux内核版本
yum list kernel
yum update kernel -y
reboot