Spark on k8s部署

一、环境准备

1.安装Jdk1.8

（1）Jdk1.8下载地址：https://www.oracle.com/java/technologies/downloads/archive/

将压缩包解压到/opt/目录

tar zxf jdk-8u212-linux-x64.tar.gz -C /opt/

（2）配置环境变量

编辑配置文件，vi /etc/profile，添加以下内容

#jdk1.8.0_121
export JAVA_HOME=/opt/jdk1.8.0_212
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

使环境变量生效

source /etc/profile

（3）添加jdk安全认证

引入如下三个包

并在java.security文件中添加配置 security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider

2、获取Spark安装包文件

（1）使用wget命令下载Spark v3.2.3安装包文件。

wget https://archive.apache.org/dist/spark/spark-3.2.3/spark-3.2.3-bin-hadoop3.2.tgz

（2）解压并重命名

tar -zxvf spark-3.2.3-bin-hadoop3.2.tgz -C /opt/module
mv spark-3.2.3-bin-hadoop3.2 spark-3.2.3

3、初始化K8s环境

（1）创建metaSphere Namespace

编写metaSphere-namespace.yaml

vi metaSphere-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:name: metaspherelabels:app.kubernetes.io/name: metasphereapp.kubernetes.io/instance: metasphere

提交yaml创建namespace

kubectl apply -f metaSphere-namespace.yaml

查看namespace

kubectl get ns

（2）创建ServiceAccount

编写spark-service-account.yaml

vi spark-service-account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:namespace: metaspherename: spark-service-accountlabels:app.kubernetes.io/name: metasphereapp.kubernetes.io/instance: metasphereapp.kubernetes.io/version: v3.2.3

提交yaml创建ServiceAccount

kubectl apply -f spark-service-account.yaml

查看ServiceAccount

kubectl get sa -n metasphere

（3）创建Role和RoleBinding

编写spark-role.yaml

vi spark-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:labels:app.kubernetes.io/name: metasphereapp.kubernetes.io/instance: metasphereapp.kubernetes.io/version: v3.2.3namespace: metaspherename: spark-role
rules:- apiGroups: [""]resources: ["pods"]verbs: ["get", "watch", "list", "create", "delete"]- apiGroups: ["extensions", "apps"]resources: ["deployments"]verbs: ["get", "watch", "list", "create", "delete"]- apiGroups: [""]resources: ["configmaps"]verbs: ["get", "create", "update", "delete"]- apiGroups: [""]resources: ["secrets"]verbs: ["get"]- apiGroups: [""]resources: ["services"]verbs: ["get", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:labels:app.kubernetes.io/name: metasphereapp.kubernetes.io/instance: metasphereapp.kubernetes.io/version: v3.2.3name: spark-role-bindingnamespace: metasphere
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: spark-role
subjects:- kind: ServiceAccountname: spark-service-accountnamespace: metasphere

提交yaml创建Role和RoleBinding

kubectl apply -f spark-role.yaml

查看Role和RoleBinding

kubectl get role -n metasphere
kubectl get rolebinding -n metasphere

（4）创建ClusterRole和ClusterRoleBinding

编写cluster-role.yaml

vi cluster-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app.kubernetes.io/name: metasphereapp.kubernetes.io/instance: metasphereapp.kubernetes.io/version: v3.2.3name: apache-spark-clusterrole
rules:- apiGroups:- ''resources:- configmaps- endpoints- nodes- pods- secrets- namespacesverbs:- list- watch- get- apiGroups:- ''resources:- servicesverbs:- get- list- watch- apiGroups:- ''resources:- eventsverbs:- create- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app.kubernetes.io/name: metasphereapp.kubernetes.io/instance: metasphereapp.kubernetes.io/version: v3.2.3name: apache-spark-clusterrole-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: apache-spark-clusterrole
subjects:- kind: ServiceAccountname: spark-service-accountnamespace: metasphere

提交yaml创建ClusterRole和ClusterRoleBinding

kubectl apply -f cluster-role.yaml

查看ClusterRole和ClusterRoleBinding

kubectl get ClusterRole | grep sparkkubectl get ClusterRoleBinding | grep spark 

二、Spark On K8s基本测试

1、拉取apache spark镜像

到Docker Hub查找apache spark的镜像，并拉取到本地

docker pull apache/spark:v3.2.3

如果因为网络原因无法下载镜像，则使用以下镜像

docker pull registry.cn-hangzhou.aliyuncs.com/cm_ns01/apache-spark:v3.2.3

2、查看k8s master的url

获取Kubernetes control plane URL

kubectl cluster-info

3、提交Spark程序到K8s上运行

/opt/module/spark-3.2.3/bin/spark-submit \--name SparkPi \--verbose \--master k8s://https://localhost:6443 \--deploy-mode cluster \--conf spark.network.timeout=300 \--conf spark.executor.instances=3 \--conf spark.driver.cores=1 \--conf spark.executor.cores=1 \--conf spark.driver.memory=1024m \--conf spark.executor.memory=1024m \--conf spark.kubernetes.namespace=metasphere \--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \--conf spark.kubernetes.container.image=registry.cn-hangzhou.aliyuncs.com/cm_ns01/apache-spark:v3.2.3 \--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-service-account \--conf spark.kubernetes.authenticate.executor.serviceAccountName=spark-service-account \--conf spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \--conf spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \--class org.apache.spark.examples.SparkPi \local:///opt/spark/examples/jars/spark-examples_2.12-3.2.3.jar \3000

参数说明：

–master为Kubernetes control plane URL

–deploy-mode为cluster，则driver和executor都运行在K8s里

–conf spark.kubernetes.namespace为前面创建的命名空间metasphere

–conf spark.kubernetes.container.image为Spark的镜像地址

–conf spark.kubernetes.authenticate.executor.serviceAccountName为前面创建的spark-service-account

–class为Spark程序的启动类

local:///opt/spark/examples/jars/spark-examples_2.12-3.2.3.jar为Spark程序所在的Jar文件，spark-examples_2.12-3.2.3.jar是Spark镜像自带的，所以使用local schema

3000是传入Spark程序的启动类的参数

4、观察driver pod和executor pod

watch -n 1 kubectl get all -owide -n metasphere

5、查看日志输出

kubectl logs sparkpi-b9de1a887b1163f1-driver -n metasphere

6、清理Driver Pod

kubectl delete pod sparkpi-b9de1a887b1163f1-driver -n metasphere