rabbitmq-k8s下双架构镜像+手动sts部署完全文档（上）

作者：朱雷

1、环境准备

组件	架构	版本	备注
Rabbitmq	Linux/amd64	V4.0.9	64位架构
Rabbitmq	Linux/arm64/v8	V4.0.9	64位架构
K8S	Linux	v1.23.6

2、双架构镜像

Rabbitmq Linux/amd64下载官方镜像：

docker pull --platform Linux/amd64 rabbitmq:4.0.9-management
docker tag rabbitmq:4.0.9-management 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management-amd64-x86
docker push 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management -amd64-x86

Rabbitmq Linux/arm64下载官方镜像：

docker pull --platform Linux/arm64/v8 rabbitmq:4.0.9-management
docker tag rabbitmq:4.0.9-management 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management-arm64-v8
docker push 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management-arm64-v8

docker inspect 77d88125053e | grep -A5 Architecture

docker inspect 5e283cfbf5e6 | grep -A5 Architecture

创建manifest：

docker manifest create --insecure 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management-arm64-v8 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management-amd64-x86Created manifest list 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management

添加manifest系统、架构等注释：

docker manifest annotate --arch arm64 --os linux --variant v8 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management-arm64-v8docker manifest annotate 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management-amd64-x86 --arch amd64 --os linux

推送manifest到Harbor仓库：

docker manifest push --insecure 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management

Manifest清单检查：

docker manifest inspect --insecure 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management

不同架构系统测试拉取：

docker pull 192.168.123.240:8011/sdx/rabbitmq:4.0.9-management

x86 系统下：

Arm系统下：

3、sts部署文件

以下所有文件内 “{{}}” 包含的内容根据实际需要替换

3.1. headless-service.yaml

apiVersion: v1
kind: Service
metadata:name: rabbitmq-headlessnamespace: {{rabbitmq-clu-9}}labels:app: rabbitmq
spec:clusterIP: Noneports:- name: amqpport: 5672targetPort: 5672- name: managementport: 15672 targetPort: 15672- name: epmdport: 4369targetPort: 4369- name: distport: 25672targetPort: 25672selector:app: rabbitmqpublishNotReadyAddresses: true

3.2. svc.yaml（可选）

kind: Service
apiVersion: v1
metadata:namespace: {{rabbitmq-clu-9}}name: rabbitmq-service
spec:ports:- name: httpprotocol: TCPport: 15672nodePort: 30015- name: amqpprotocol: TCPport: 5672targetPort: 5672nodePort: 30016selector:app: rabbitmqtype: NodePort

3.3. rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:name: rabbitmqnamespace: {{rabbitmq-clu-9}}
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: endpoint-readernamespace: {{rabbitmq-clu-9}}
rules:
- apiGroups: [""]resources: ["endpoints"]verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: endpoint-readernamespace: {{rabbitmq-clu-9}}
subjects:
- kind: ServiceAccountname: rabbitmq
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: endpoint-reader

3.4. sc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: hostpath-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

3.5. pv.yaml

修改 {{node x}} 为kubectl get nodes 输出第一列NAME 的内容
Path路径和存储大小，根据实际业务自定义，本文存储大小只作为测试

apiVersion: v1
kind: PersistentVolume
metadata:name: rabbitmq-clu-pv-0
spec:capacity:storage: {{2Gi}}accessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: RetainstorageClassName: hostpath-storagehostPath:path: {{/data/rabbitmq-clu-pv-0}}type: DirectoryOrCreatenodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- {{node01}}
---
apiVersion: v1
kind: PersistentVolume
metadata:name: rabbitmq-clu-pv-1
spec:capacity:storage: {{2Gi}}accessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: RetainstorageClassName: hostpath-storagehostPath:path: {{/data/rabbitmq-clu-pv-1}}type: DirectoryOrCreatenodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- {{node02}}
---
apiVersion: v1
kind: PersistentVolume
metadata:name: rabbitmq-clu-pv-2
spec:capacity:storage: {{2Gi}}accessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: RetainstorageClassName: hostpath-storagehostPath:path: {{/data/rabbitmq-clu-pv-2}}type: DirectoryOrCreatenodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- {{node03}}

3.6. secret.yaml

账号密码及cookie请自行定义，本文内容只作为测试

apiVersion: v1
kind: Secret
metadata:name: rabbitmq-secretnamespace: {{rabbitmq-clu-9}}
type: Opaque
stringData:rabbitmq-username: "{{admin}}"rabbitmq-password: "{{admin123}}"erlang-cookie: "{{ERLANG_COOKIE_VALUE}}"

3.7. cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:name: rabbitmq-confignamespace: {{rabbitmq-clu-9}}
data:enabled_plugins: |[rabbitmq_management,rabbitmq_peer_discovery_k8s].rabbitmq.conf: |# 基础配置listeners.tcp.default = 5672# management.listener.port = 15672
# management.listener.ssl = false
# 定义了磁盘空间的绝对最低限制，当剩余空间低于此值时将触发警报并阻止消息生产
disk_free_limit.absolute = {{2GB}}
# 指定在 Kubernetes 环境中使用 RabbitMQ 自带的 K8s 对等发现后端# cluster_formation.peer_discovery_backend = k8s
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
# K8s API 服务器地址
cluster_formation.k8s.host = {{kubernetes.default.svc.cluster.local}}
# 节点地址类型
cluster_formation.k8s.address_type = hostname
# 主机名后缀，格式：svc(headless)名.命令空间名.svc.cluster.local
cluster_formation.k8s.hostname_suffix = {{.rabbitmq-headless.rabbitmq-clu-9.svc.cluster.local}}
# 节点发现重试次数
cluster_formation.discovery_retry_limit = {{10}}
# 发现重试间隔（毫秒）
cluster_formation.discovery_retry_interval = {{3000}}
# 无头服务名称
cluster_formation.k8s.service_name = rabbitmq-headless
# 设定了检查并清理失效集群节点的间隔时间为 30 秒
cluster_formation.node_cleanup.interval = {{30}}
# 当检测到失效集群节点时, 系统会实际执行节点清理操作，从集群中移除失效节点
cluster_formation.node_cleanup.only_log_warning = false
# 该配置用于 ETCD 服务发现的 SSL/TLS 证书验证
# verify_none 表示禁用对 ETCD 服务器证书的验证, 通常用于内部信任的网络中# cluster_formation.etcd.ssl_options.verify = verify_none## Mnesia 数据库加载元数据时的超时时间和重试次数# wait for 60 seconds instead of 30mnesia_table_loading_retry_timeout = {{60000}}# retry 15 times instead of 10mnesia_table_loading_retry_limit = {{15}}
# 内存配置
# 将内存高水位线设置为总可用内存的 70%，超过此限制将触发流控
vm_memory_high_watermark.relative = {{0.7}}
# 高水位线内存分页比率
vm_memory_high_watermark_paging_ratio = {{0.6}}
# 总内存覆盖值，根据实际物理内存大小调整,参考sts的resources的limit内存配置建议小于等于total_memory_available_override_value = {{8GB}}
# 日志配置
# 启用控制台日志
log.console = {{true}}
# 控制台日志级别
log.console.level = info
# 文件日志log.file = rabbit.loglog.file.level = info
log.file.formatter.level_format = uc4
## 文件轮转大小和时间不能同时使用，二选一
# 文件轮转大小: 例: 100MiB, 大小值根据需要调整
# log.file.rotation.size = {{104857600}}
# 保留文件数
# log.file.rotation.count = {{7}}
# 是否压缩
# log.file.rotation.compress = {{true}}
# 文件轮转时间: 例: 下面配置每天0 点执行轮转值根据需要调整
log.file.rotation.date = $D{{0}}
# 保留天数
log.file.rotation.count = {{7}}
# 是否压缩
log.file.rotation.compress = {{true}}# 临时启用调试日志，生产环境禁用# log.connection.level = debug# log.channel.level = debug# log.queue.level = debug

3.8. rabbitmq-sts.yaml

镜像地址、资源请求限制大小、存储模板资源请求大小请自行定义。

apiVersion: apps/v1
kind: StatefulSet
metadata:name: rabbitmqnamespace: {{rabbitmq-clu-9}}labels:app: rabbitmq
spec:serviceName: rabbitmq-headlessreplicas: 3podManagementPolicy: "Parallel"selector:matchLabels:app: rabbitmqtemplate:metadata:labels:app: rabbitmqspec:terminationGracePeriodSeconds: 10affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 100podAffinityTerm:labelSelector:matchExpressions:- key: appoperator: Invalues:- rabbitmqtopologyKey: kubernetes.io/hostnameserviceAccountName: rabbitmq        containers:- name: rabbitmqimage: {{rabbitmq:4.0.9-management}}imagePullPolicy: IfNotPresentports:- containerPort: 5672name: amqp- containerPort: 15672name: httpresources:limits:cpu: "{{2}}"memory: {{8Gi}}requests:cpu: "{{1}}"memory: {{4Gi}}env:- name: RABBITMQ_USE_LONGNAMEvalue: "true"        - name: RABBITMQ_ERLANG_COOKIEvalueFrom:secretKeyRef:name: rabbitmq-secretkey: erlang-cookie- name: RABBITMQ_DEFAULT_USERvalueFrom:secretKeyRef:name: rabbitmq-secretkey: rabbitmq-username- name: RABBITMQ_DEFAULT_PASSvalueFrom:secretKeyRef:name: rabbitmq-secretkey: rabbitmq-password- name: POD_NAMESPACEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namespace- name: POD_NAMEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namevolumeMounts:- name: configmountPath: /etc/rabbitmq- name: datamountPath: /var/lib/rabbitmqsubPathExpr: $(POD_NAMESPACE)/rabbitmqreadOnly: false- name: datamountPath: /var/log/rabbitmqsubPathExpr: $(POD_NAMESPACE)/rabbitmq/logs/$(POD_NAME)readOnly: falsereadinessProbe:tcpSocket:port: 5672initialDelaySeconds: 60periodSeconds: 60timeoutSeconds: 3successThreshold: 1failureThreshold: 3# livenessProbe:#   exec:#     command: ["rabbitmq-diagnostics", "ping"]#   initialDelaySeconds: 60#   periodSeconds: 30volumes:- name: configconfigMap:name: rabbitmq-configitems:- key: rabbitmq.confpath: rabbitmq.conf- key: enabled_pluginspath: enabled_pluginsvolumeClaimTemplates:- metadata:name: datanamespace: {{rabbitmq-clu-9}}spec:accessModes: [ "ReadWriteOnce" ]resources:requests:storage: {{2000M}}storageClassName: hostpath-storage