k8s集群搭建一主多从的jenkins集群

方案

+---------------------+
| Jenkins Master      |
| - 持久化配置         |<---(hostpath 存储)
| - 自动容灾           |
+----------+----------+|| Jenkins JNLP 通信|
+----------v----------+      +-------------------+
| Jenkins Agent       |      | Kubernetes Plugin |
| (动态 Pod)           |<---->| (自动创建 Agent)   |
+---------------------+      +-------------------+^|
+----------+----------+
| 弹性扩容机制         |
| (HPA + Pod 模板)    |
+---------------------+

一、环境准备

k8s 集群ip

kubectl get nodes -o wide # 显示各节点的 InternalIP 和 ExternalIP

检查pod外网是否可通

# 创建临时 Pod 测试公网访问
kubectl run net-test --image=alpine --rm -it --restart=Never -- sh# 在 Pod 内部执行（测试 HTTP 访问）
curl -v -4 --connect-timeout 10 http://example.com# 测试 DNS 解析（确认域名解析是否正常）
nslookup google.com# 测试 TCP 连接（如需要）
nc -zv www.baidu.com 80

jenkins镜像：

docker login kubernetes-register.sswang.com:80 -u sswang

docker pull jenkins/jenkins:2.344  # 官方 2.34 最新版 :cite[2]:cite[5]:cite[7]
docker tag jenkins/jenkins:2.344 kubernetes-register.sswang.com:80/jenkins/jenkins:2.344
docker push kubernetes-register.sswang.com:80/jenkins/jenkins:2.344

docker pull jenkins/jenkins:lts-jdk17
docker tag jenkins/jenkins:lts-jdk17 kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17
docker push kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17

• agent镜像

docker pull jenkins/inbound-agent:jdk11
docker tag jenkins/inbound-agent:jdk11 kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11
docker push kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11

• 创建namespace

kubectl create namespace jenkins

• hostpath存储准备

kubectl label node kubernetes-master jenkins-master=true # node打标签，支持基于标签选择器进行pod调度
mkdir -p /mnt/jenkins-data
chown 1000:1000 /mnt/jenkins-data  # Jenkins 默认 UID 1000 deck1
chmod -R 755 /mnt/jenkins-data

• 版本兼容性说明

二、jenkins master部署

1. jenkins-master.yaml

apiVersion: apps/v1
kind: Deployment
metadata:name: jenkins-masternamespace: jenkins
spec:replicas: 1 # 副本数量selector:matchLabels:app: jenkins-masterstrategy:type: Recreatetemplate:metadata:labels:app: jenkins-masterspec:# 绑定到专用节点（可选）nodeSelector:jenkins-master: "true"securityContext:fsGroup: 1000containers:- name: jenkinsimage: kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17env:- name: JAVA_OPTSvalue: "-Djenkins.install.runSetupWizard=false -Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Shanghai"ports:- containerPort: 8080- containerPort: 50000volumeMounts:- name: jenkins-homemountPath: /var/jenkins_homelivenessProbe:httpGet:path: /loginport: 8080initialDelaySeconds: 120periodSeconds: 10readinessProbe:httpGet:path: /loginport: 8080initialDelaySeconds: 30resources:limits:memory: "2Gi"cpu: "1"requests:memory: "1Gi"cpu: "500m"volumes:- name: jenkins-homehostPath:path: /mnt/jenkins-datatype: DirectoryOrCreate

2. jenkins-service.yml

nodeport 对外暴露jenkis服务，能够访问集群内pod。

apiVersion: v1
kind: Service
metadata:name: jenkins-servicenamespace: jenkins
spec:selector:app: jenkins-mastertype: NodePort # 永久对外暴露服务ports:- name: httpport: 8080targetPort: 8080- name: agentport: 50000targetPort: 50000

kubectl get svc -n jenkins jenkins-service

其中，32251为外部访问所使用的端口，ip为宿主机ip或者虚拟机的ip地址。

Jenkins 服务地址确认

# 查看 Jenkins Service 名称
kubectl get svc -n jenkins# 验证内部 DNS 解析（在任意 Pod 内）
kubectl run test-dns --image=busybox -it --rm --restart=Never -- \nslookup jenkins-service.jenkins.svc.cluster.local# [svc].[namespace].svc.cluster.local

3. jenkins-rbac.yml

创建相应的RBAC权限，以便Jenkins Master能够创建和管理Agent Pod

1. ServiceAccount：在jenkins命名空间中创建一个名为jenkins的服务账户。
2. ClusterRole：定义了一组权限规则，允许对pods、pods/exec、pods/log等资源的操作，以及获取secrets的权限。
3. ClusterRoleBinding：将ClusterRole绑定到ServiceAccount，使得jenkins服务账户具有ClusterRole中定义的权限。

apiVersion: v1
kind: ServiceAccount
metadata:name: jenkins # jenkins serviceaccountnamespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: jenkins-agent-role
rules:
- apiGroups: [""]resources: ["pods", "pods/exec", "pods/log"]verbs: ["create", "delete", "get", "list", "watch", "patch"]
- apiGroups: [""]resources: ["secrets"]verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: jenkins-agent-role-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: jenkins-agent-role
subjects:
- kind: ServiceAccountname: jenkinsnamespace: jenkins

kubectl get serviceaccount -n jenkins

如何让某个访问作为serviceaccount 为jenkins的账户？

apiVersion: v1
kind: Secret
metadata:name: jenkins-tokennamespace: jenkinsannotations:kubernetes.io/service-account.name: jenkins
type: kubernetes.io/service-account-token

至此就能通过secret访问serviceaccount，进而访问apiserver拥有相应的pod操作权限。

在这里插入图片描述

三、动态Agent配置与kubernetes插件

1. 插件初始化

https://mirrors.huaweicloud.com/jenkins/updates/update-center.json

名称: kubernetes
Kubernetes 地址: https://kubernetes.default.svc.cluster.local
命名空间: jenkins
凭据: jenkins-service-account (选择创建的 ServiceAccount)
Jenkins 地址: http://jenkins-service.jenkins.svc.cluster.local:8080

凭据获取，要求k8s集群配置：serviceaccount账户作为 集群操作权限的承接者，secret绑定account，用于apiserver鉴权，rbac用于将serviceaccount绑定role。

TOKEN=$(kubectl get secret jenkins-token -n jenkins -o jsonpath='{.data.token}' | base64 --decode)
echo $TOKEN

服务证书key获取：

CA_CRT=$(kubectl get secret jenkins-token -n jenkins -o jsonpath='{.data.ca\.crt}' | base64 -d)
echo "$CA_CRT"

# 验证 ServiceAccount 权限
kubectl auth can-i create pods \--as system:serviceaccount:jenkins:jenkins \-n jenkins
# 应返回 yes# 验证列表权限
kubectl auth can-i list pods \--as system:serviceaccount:jenkins:jenkins \-n jenkins
# 应返回 yes
# [命名空间][serviceaccount]

2. pod 模板

名称: jnlp-agent
标签: jnlp-agent
容器模板:
- 名称: jnlp镜像: kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11参数: '${computer.jnlpmac} ${computer.name}'资源限制:CPU: 500m内存: 512Mi资源请求:CPU: 200m内存: 256Mi

podTemplate:name: jenkins-agentlabel: jenkins-agentserviceAccount: jenkins
#   imagePullSecrets:
#   - name: docker-registry-credcontainers:- name: jnlpimage: kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11alwaysPullImage: falseresourceLimitCpu: "1"resourceLimitMemory: "2Gi"args: '$(JENKINS_SECRET) $(JENKINS_NAME)'