当前位置：首页 > news >正文

Kubernetes 构建高可用、高性能 Redis 集群实战指南

news 2025/8/3 12:50:29

1. 部署方案规划

1.1 部署架构图

1.2 前提说明

本实战环境使用 NFS 作为 k8s 集群的持久化存储
Redis 集群所有资源部署在命名空间 chengke 内。

2. 创建 ConfigMap

2.1 创建 ConfigMap

2.1.1 创建 Redis 配置文件

[root@k8s-m1 1]# cat redis-cluster-cm.yaml 
apiVersion: v1
kind: ConfigMap
metadata:name: redis-cluster-config
data:redis-config: |appendonly yesprotected-mode nodir /dataport 6379cluster-enabled yescluster-config-file /data/nodes.confcluster-node-timeout 5000masterauth Chengke2025requirepass Chengke2025

2.1.2 创建资源

[root@k8s-m1 1]# kubectl apply -f redis-cluster-cm.yaml

2.1.3 验证资源

[root@k8s-m1 1]# kubectl get cm
NAME                   DATA   AGE
redis-cluster-config   1      68m

2.2 创建 Redis

本文使用 StatefulSet 部署 Redis 服务，需要创建 StatefulSet 和 HeadLess 两种资源。

2.2.1 创建资源清单

[root@k8s-m1 1]# cat redis-cluster-sts.yaml 
apiVersion: v1
kind: Service
metadata:name: redis-headlesslabels:app.kubernetes.io/name: redis-cluster
spec:ports:- name: redis-6379protocol: TCPport: 6379targetPort: 6379selector:app.kubernetes.io/name: redis-clusterclusterIP: Nonetype: ClusterIP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: redis-clusterlabels:app.kubernetes.io/name: redis-cluster
spec:serviceName: redis-headlessreplicas: 6selector:matchLabels:app.kubernetes.io/name: redis-clustertemplate:metadata:labels:app.kubernetes.io/name: redis-clusterspec:affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 100podAffinityTerm:labelSelector:matchExpressions:- key: app.kubernetes.io/nameoperator: Invalues:- redis-clustertopologyKey: kubernetes.io/hostnamecontainers:- name: redisimage: redis:8.0.2imagePullPolicy: IfNotPresentcommand:- "redis-server"args:- "/etc/redis/redis.conf"- "--protected-mode"- "no"- "--cluster-announce-ip"- "$(POD_IP)"env:- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIPports:- name: redis-6379containerPort: 6379protocol: TCPvolumeMounts:- name: configmountPath: /etc/redis- name: redis-cluster-datamountPath: /dataresources:requests:cpu: 50mmemory: 500Milimits:cpu: "2"memory: 4Givolumes:- name: configconfigMap:name: redis-cluster-configitems:- key: redis-configpath: redis.confvolumeClaimTemplates:- metadata:name: redis-cluster-dataspec:accessModes:- ReadWriteOncestorageClassName: nfs-clientresources:requests:storage: 5Gi

注意： POD_IP 是重点，如果不配置会导致线上的 POD 重启换 IP 后，集群状态无法自动同步

2.2.2 创建资源

[root@k8s-master01 1]# kubectl apply -f redis-cluster-sts.yaml
service/redis-headless created
statefulset.apps/redis-cluster created

2.2.3 验证资源

执行下面的命令，查看 StatefulSet、Pod、Service 创建结果

[root@k8s-m1 1]# kubectl get pod,svc,sts
NAME                                READY   STATUS    RESTARTS   AGE
pod/redis-cluster-0                 1/1     Running   0          69m
pod/redis-cluster-1                 1/1     Running   0          67m
pod/redis-cluster-2                 1/1     Running   0          48m
pod/redis-cluster-3                 1/1     Running   0          65m
pod/redis-cluster-4                 1/1     Running   0          65m
pod/redis-cluster-5                 1/1     Running   0          65m
pod/redisinsight-5cb887744f-79h5k   1/1     Running   0          42mNAME                             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
service/kubernetes               ClusterIP   10.0.0.1     <none>        443/TCP          8d
service/redis-cluster-external   NodePort    10.5.96.53   <none>        6379:31379/TCP   62m
service/redis-headless           ClusterIP   None         <none>        6379/TCP         69m
service/redisinsight-external    NodePort    10.9.143.0   <none>        5540:31380/TCP   42mNAME                             READY   AGE
statefulset.apps/redis-cluster   6/6     69m

2.3 创建 k8s 集群外部访问服务

2.3.1 编写资源

我们采用 NodePort 方式在 Kubernetes 集群外发布 Redis 服务，指定的端口为 31379。

[root@k8s-m1 1]# cat redis-cluster-svc-external.yaml 
kind: Service
apiVersion: v1
metadata:name: redis-cluster-externallabels:app: redis-cluster-external
spec:ports:- protocol: TCPport: 6379targetPort: 6379nodePort: 31379selector:app.kubernetes.io/name: redis-clustertype: NodePort

2.3.2 创建资源

[root@k8s-master01 1]# kubectl apply -f redis-cluster-svc-external.yaml
service/redis-cluster-external created

2.3.3 验证资源

执行下面的命令，查看 Service 创建结果

[root@k8s-m1 1]# kubectl get endpointslice
NAME                           ADDRESSTYPE   PORTS   ENDPOINTS                                                 AGE
kubernetes                     IPv4          6443    192.168.10.11                                             24d
my-service-noselector-1        IPv4          80      192.168.10.12,192.168.10.13                               18d
redis-cluster-external-qvl7h   IPv4          6379    10.244.215.78,10.244.215.111,10.244.111.244 + 3 more...   65m
redis-headless-j2x96           IPv4          6379    10.244.111.251,10.244.215.111,10.244.215.73 + 3 more...   73m
redisinsight-external-b2vks    IPv4          5540    10.244.111.252                                            45m
[root@k8s-m1 1]# kubectl get svc
NAME                     TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
kubernetes               ClusterIP   10.0.0.1     <none>        443/TCP          8d
redis-cluster-external   NodePort    10.5.96.53   <none>        6379:31379/TCP   65m
redis-headless           ClusterIP   None         <none>        6379/TCP         73m
redisinsight-external    NodePort    10.9.143.0   <none>        5540:31380/TCP   46m

3. 创建 Redis 集群

Redis POD 创建完成后，不会自动创建 Redis 集群，需要手工执行集群初始化的命令，有自动创建和手工创建两种方式，二选一，建议选择自动。

3.1 自动创建 Redis 集群

执行下面的命令，自动创建 3 个 master 和 3 个 slave 的集群，中间需要输入一次 yes。

[root@k8s-m1 1]# kubectl exec -it redis-cluster-0 -- redis-cli -a Chengke2025 --cluster create --cluster-replicas 1 $(kubectl get pods -l app.kubernetes.io/name=redis-cluster -o jsonpath='{range.items[*]} {.status.podIP}:6379 {end}')
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.244.111.244:6379 to 10.244.111.251:6379
Adding replica 10.244.215.78:6379 to 10.244.215.111:6379
Adding replica 10.244.215.73:6379 to 10.244.111.253:6379
M: 80be000a17cc1b2d1d7180e722876d501c90c9a3 10.244.111.251:6379slots:[0-5460] (5461 slots) master
M: 01a21392ef5998e7da5d382a1ed6d0ec35bef845 10.244.215.111:6379slots:[5461-10922] (5462 slots) master
M: 8e009ec3ccc37b0810db6b1ffc5bd6c241c0cd31 10.244.111.253:6379slots:[10923-16383] (5461 slots) master
S: 48978e7c86fe9bf341880eb9cf1fac3aa8e48dc4 10.244.215.73:6379replicates 8e009ec3ccc37b0810db6b1ffc5bd6c241c0cd31
S: 0d8bab081a2125b7a83c67c588c2a9f93482d69a 10.244.111.244:6379replicates 80be000a17cc1b2d1d7180e722876d501c90c9a3
S: d7f1dc037259b2d2c155ac5a5c5c5b4b8f7818c4 10.244.215.78:6379replicates 01a21392ef5998e7da5d382a1ed6d0ec35bef845
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 10.244.111.251:6379)
M: 80be000a17cc1b2d1d7180e722876d501c90c9a3 10.244.111.251:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
S: 48978e7c86fe9bf341880eb9cf1fac3aa8e48dc4 10.244.215.73:6379slots: (0 slots) slavereplicates 8e009ec3ccc37b0810db6b1ffc5bd6c241c0cd31
S: 0d8bab081a2125b7a83c67c588c2a9f93482d69a 10.244.111.244:6379slots: (0 slots) slavereplicates 80be000a17cc1b2d1d7180e722876d501c90c9a3
M: 8e009ec3ccc37b0810db6b1ffc5bd6c241c0cd31 10.244.111.253:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
M: 01a21392ef5998e7da5d382a1ed6d0ec35bef845 10.244.215.111:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
S: d7f1dc037259b2d2c155ac5a5c5c5b4b8f7818c4 10.244.215.78:6379slots: (0 slots) slavereplicates 01a21392ef5998e7da5d382a1ed6d0ec35bef845
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

3.2 验证集群状态

[root@k8s-m1 1]# kubectl exec -it redis-cluster-0 -- redis-cli -p 6379 -a Chengke2025 cluster info
E0802 16:18:19.072387   92106 websocket.go:297] Unknown stream id 1, discarding messageWarning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:6868
cluster_stats_messages_pong_sent:6767
cluster_stats_messages_sent:13635
cluster_stats_messages_ping_received:6762
cluster_stats_messages_pong_received:6867
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:13634
total_cluster_links_buffer_limit_exceeded:0

4. 集群功能测试

4.1 压力测试

使用 Redis 自带的压力测试工具，测试 Redis 集群是否可用，并简单测试性能。

使用 set 和get命令，发送100000次请求，每个请求包含一个键值对，其中键是随机生成的，值的大小是100字节，同时有20个客户端并发执行。

测试 set 场景：

kubectl exec -it redis-cluster-0 -- redis-benchmark -h 192.168.10.11 -p 31379 -a Chengke2025 -t set -n 100000 -c 20 -d 100 --cluster

测试 get 场景：

[root@k8s-master01 1]# kubectl exec -it redis-cluster-0 -- redis-benchmark -h
192.168.10.11 -p 31379 -a Chengke2025 -t get -n 100000 -c 20 -d 100 --cluster