基于Rook的Ceph云原生存储部署与实践指南（下）
#作者：任少近
文章目录
- 6Ceph资源对像管理
- 6.1查看services
- 6.2查看Jobs
- 6.3 查看deployments.apps
- 6.4查看daemonsets.apps
- 6.5查看configmaps
- 6.6查看clusterroles.rbac.authorization.k8s.io
- 6.7查看clusterrolebindings.rbac.authorization.k8s.io
- 6.8通过cephclusters.ceph查看OSD池信息
- 7访问ceph
- 7.1Toolbox客户端
- 7.1K8s节点访问ceph
- 7.2暴露端口web访问
- 7.3删除OSD Deployment
- 7.4Ceph的Pool（多租户）创建pool设置pg的数量
- 7.5修改登录密码
- 8安装错误汇总
- 8.1quincy版的Ceph-coomon安装报错
- 9故障处理
- 9.1ceph集群提示daemons have recently crashed, health: HEALTH_WARN
- 9.2osd down
6Ceph资源对像管理
6.1查看services
[root@k8s-master ~]# kubectl -n rook-ceph get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mgr ClusterIP 10.110.141.201 <none> 9283/TCP 13h
rook-ceph-mgr-dashboard ClusterIP 10.103.197.146 <none> 8443/TCP 13h
rook-ceph-mon-a ClusterIP 10.110.163.61 <none> 6789/TCP,3300/TCP 13h
rook-ceph-mon-b ClusterIP 10.100.49.10 <none> 6789/TCP,3300/TCP 13h
rook-ceph-mon-c ClusterIP 10.96.193.162 <none> 6789/TCP,3300/TCP 13h
6.2查看Jobs
[root@k8s-master]#kubectl -n rook-ceph get jobs
NAME COMPLETIONS DURATION AGE
rook-ceph-osd-prepare-k8s-master 1/1 6s 11h
rook-ceph-osd-prepare-k8s-node1 1/1 7s 11h
rook-ceph-osd-prepare-k8s-node2 1/1 7s 11h
rook-ceph-osd-prepare-k8s-node3 1/1 6s 11h
6.3 查看deployments.apps
[root@k8s-master]# kubectl -n rook-ceph get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
csi-cephfsplugin-provisioner 2/2 2 2 12h
csi-rbdplugin-provisioner 2/2 2 2 12h
rook-ceph-crashcollector-k8s-master 1/1 1 1 12h
rook-ceph-crashcollector-k8s-node1 1/1 1 1 12h
rook-ceph-crashcollector-k8s-node2 1/1 1 1 12h
rook-ceph-crashcollector-k8s-node3 1/1 1 1 12h
rook-ceph-mgr-a 1/1 1 1 12h
rook-ceph-mgr-b 1/1 1 1 12h
rook-ceph-mon-a 1/1 1 1 12h
rook-ceph-mon-b 1/1 1 1 12h
rook-ceph-mon-c 1/1 1 1 12h
rook-ceph-operator 1/1 1 1 12h
rook-ceph-osd-0 1/1 1 1 12h
rook-ceph-osd-1 1/1 1 1 12h
rook-ceph-osd-2 1/1 1 1 12h
rook-ceph-osd-3 1/1 1 1 12h
6.4查看daemonsets.apps
[root@k8s-master]# kubectl -n rook-ceph get daemonsets.apps
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
csi-cephfsplugin 4 4 4 4 4 <none> 12h
csi-rbdplugin 4 4 4 4 4 <none> 12h
6.5查看configmaps
[root@k8s-master]# kubectl -n rook-ceph get configmaps
NAME DATA AGE
kube-root-ca.crt 1 13h
rook-ceph-csi-config 1 12h
rook-ceph-csi-mapping-config 1 12h
rook-ceph-mon-endpoints 5 12h
rook-ceph-operator-config 33 13h
rook-config-override 1 12h
6.6查看clusterroles.rbac.authorization.k8s.io
[root@k8s-master # kubectl -n rook-ceph get clusterroles.rbac.authorization.k8s.io
NAME CREATED AT
cephfs-csi-nodeplugin 2023-06-13T13:56:29Z
cephfs-external-provisioner-runner 2023-06-13T13:56:29Z
rbd-csi-nodeplugin 2023-06-13T13:56:29Z
rbd-external-provisioner-runner 2023-06-13T13:56:29Z
rook-ceph-cluster-mgmt 2023-06-13T13:56:29Z
rook-ceph-global 2023-06-13T13:56:29Z
rook-ceph-mgr-cluster 2023-06-13T13:56:29Z
rook-ceph-mgr-system 2023-06-13T13:56:29Z
rook-ceph-object-bucket 2023-06-13T13:56:29Z
rook-ceph-osd 2023-06-13T13:56:29Z
rook-ceph-system 2023-06-13T13:56:29Z
6.7查看clusterrolebindings.rbac.authorization.k8s.io
kubectl -n rook-ceph get clusterrolebindings.rbac.authorization.k8s.io
cephfs-csi-nodeplugin-role ClusterRole/cephfs-csi-nodeplugin
cephfs-csi-provisioner-role ClusterRole/cephfs-external-provisioner-runner
rbd-csi-nodeplugin ClusterRole/rbd-csi-nodeplugin
rbd-csi-provisioner-role ClusterRole/rbd-external-provisioner-runner
rook-ceph-global ClusterRole/rook-ceph-global
rook-ceph-mgr-cluster ClusterRole/rook-ceph-mgr-cluster
rook-ceph-object-bucket ClusterRole/rook-ceph-object-bucket
rook-ceph-osd ClusterRole/rook-ceph-osd
rook-ceph-system ClusterRole/rook-ceph-system
6.8通过cephclusters.ceph查看OSD池信息
如果你使用了 Rook Ceph Operator 来管理 Ceph 集群，还可以查看 Rook 中的自定义资源来获取 OSD 池的信息
[root@k8s-master ~]# kubectl get cephclusters.ceph.rook.io rook-ceph -o yaml
7访问ceph
7.1Toolbox客户端
部署
cd rook/deploy/examples/
kubectl apply -f toolbox.yaml
连接ceph 集群
[root@k8s-master ~]# kubectl -n rook-ceph exec -it rook-ceph-tools-7857bc9568-q9fjk /bin/bash
bash-4.4$ ceph -s
cluster:
id: e320aa6c-0057-46ad-b2bf-5c49df8eba5a
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 13h)
mgr: b(active, since 13h), standbys: a
osd: 4 osds: 4 up (since 13h), 4 in (since 13h)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 45 MiB used, 200 GiB / 200 GiB avail
pgs: 1 active+clean
7.1K8s节点访问ceph
在节点添加ceph.conf keyring
[root@k8s-master]#mkdir /etc/ceph
[root@k8s-master]#cd /etc/ceph
[root@k8s-master]#vi ceph.conf
[global]
mon_host = 10.110.163.61:6789,10.100.49.10:6789,10.96.193.162:6789
[client.admin]
keyring = /etc/ceph/keyring
[root@k8s-master]#vi keyring
[client.admin]
key = AQCGfYhkeMnEFRAAJnW4jUMwmJz2b1dPvdTOJg==
验证
telnet 10.110.163.61 6789 以上三个services地址任一个
添加yum源
[ceph]
name=ceph
baseurl=https://mirrors.aliyun.com/ceph/rpm-quincy/el8/x86_64/
enabled=1
gpgcheck=0
安装ceph-common （安装失败，详情见5.1）
[root@k8s-master]#yum install -y ceph-common
成功可在节点上直接操作如下：
7.2暴露端口web访问
执行rook/deploy/examples/dashboard-external-https.yaml
[root@k8s-master examples]#kubectl apply -f rook/deploy/examples/dashboard-external-https.yaml
rook-ceph-mgr-dashboard-external-https NodePort 10.106.127.224 <none> 8443:31555/TCP
获取密码：
kubectl -n rook-ceph get secrets rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 --decode > rook-ceph-dashboard-password.password
G+LIkJwXQ/E*>/P&DbzB
访问，用户名为admin
https://192.168.123.194:31555/
7.3删除OSD Deployment
如果cluster.yaml中removeOSDsIfOutAndSafeToRemove: true设置为true，则Rook Operator将自动清除Deployment。默认为false。
7.4Ceph的Pool（多租户）创建pool设置pg的数量
以pool为颗粒度，如果不创建/指定，则数据会存放在默认的pool里。创建pool需要设置pg的数量，一般来说每个OSD为100个PG，也可以按照如下规则配置：
若少于5个OSD， 设置pg_num为128。
5~10个OSD，设置pg_num为512。
10~50个OSD，设置pg_num为4096。
超过50个OSD，可以参考pgcalc计算。
Pool上还需要设置CRUSH Rules策略，这是data如何分布式存储的策略。
此外，针对pool，还可以调整POOL副本数量、删除POOL、设置POOL配额、重命名POOL、查看POOL状态信息。
7.5修改登录密码
登录kubectl exec -it rook-ceph-tools-7857bc9568-q9fjk bash
bash-4.4$ echo -n ‘1qaz@WSX’ > /tmp/password.txt
bash-4.4$ ceph dashboard ac-user-set-password admin --force-password -i /tmp/password.txt，以新密码登录。
以admin/1qaz@WSX为用户名密码登录。
8安装错误汇总
8.1quincy版的Ceph-coomon安装报错
原因：aliyuncs上无el7版本的quincy依赖包，只有el8依赖包有quincy版，尝试octopus版同样报错。
--> Finished Dependency Resolution
Error: Package: 2:libcephfs2-17.2.6-0.el8.x86_64 (ceph)
Requires: libstdc++.so.6(GLIBCXX_3.4.21)(64bit)
Error: Package: 2:ceph-common-17.2.6-0.el8.x86_64 (ceph)
Requires: libstdc++.so.6(CXXABI_1.3.11)(64bit)
Error: Package: 2:libcephfs2-17.2.6-0.el8.x86_64 (ceph)
Requires: libstdc++.so.6(GLIBCXX_3.4.20)(64bit)
Error: Package: 2:ceph-common-17.2.6-0.el8.x86_64 (ceph)
Requires: libstdc++.so.6(GLIBCXX_3.4.22)(64bit)
。。。。。
。。。。。
Requires: libstdc++.so.6(GLIBCXX_3.4.20)(64bit)
Error: Package: 2:librgw2-17.2.6-0.el8.x86_64 (ceph)
Requires: libicuuc.so.60()(64bit)
Error: Package: 2:librgw2-17.2.6-0.el8.x86_64 (ceph)
Requires: libstdc++.so.6(GLIBCXX_3.4.21)(64bit)
Error: Package: 2:librgw2-17.2.6-0.el8.x86_64 (ceph)
Requires: libstdc++.so.6(CXXABI_1.3.11)(64bit)
Error: Package: 2:librgw2-17.2.6-0.el8.x86_64 (ceph)
Requires: libthrift-0.13.0.so()(64bit)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
9故障处理
9.1ceph集群提示daemons have recently crashed, health: HEALTH_WARN
bash-4.4$ ceph status
cluster:
id: e320aa6c-0057-46ad-b2bf-5c49df8eba5a
health: HEALTH_WARN
3 mgr modules have recently crashed
services:
mon: 3 daemons, quorum a,b,c (age 23h)
mgr: b(active, since 23h), standbys: a
osd: 4 osds: 4 up (since 23h), 4 in (since 23h)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 45 MiB used, 200 GiB / 200 GiB avail
pgs: 1 active+clean
#查看详细日志信息
bash-4.4$ ceph health detail
HEALTH_WARN 3 mgr modules have recently crashed
[WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed
ceph 的 crash模块用来收集守护进程出现 crashdumps （崩溃）的信息，并将其存储在ceph集群中，以供以后分析。
crash查看一下
bash-4.4$ ceph crash ls
ID ENTITY NEW
2023-06-14T13:56:38.064890Z_75a59d8c-9c99-47af-8cef-e632d8f0a010 mgr.b *
2023-06-14T13:56:53.252095Z_bc44e5d3-67e5-4c22-a872-e9c7f9799f55 mgr.b *
2023-06-14T13:57:38.564803Z_1f132169-793b-4ac6-a3c7-af48c91f5365 mgr.b *
#带*号表示为最新，上面说mgr和osd有异常信息，接下来排查下osd和mgr，看看是不是因为没有归档的原因造成
bash-4.4$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.19519 root default
-5 0.04880 host k8s-master
0 ssd 0.04880 osd.0 up 1.00000 1.00000
-3 0.04880 host k8s-node1
1 ssd 0.04880 osd.1 up 1.00000 1.00000
-9 0.04880 host k8s-node2
3 ssd 0.04880 osd.3 up 1.00000 1.00000
-7 0.04880 host k8s-node3
2 ssd 0.04880 osd.2 up 1.00000 1.00000
通过上面的命令，排查到集群状态是ok，判断crash没有归档，造成误报，接下来进行归档
#第一种方法，适合只有一两个没有归档的
#ceph crash ls
#ceph crash archive <id>
#第二种方法，适合多个归档异常的，我们这边直接执行下面的命令
#ceph crash archive-all
9.2osd down
直接看挂了，看日志。
通过ceph osd tree，发现osd.3 down了。在k8s-node2上。
