【spark】 spark on k8s
准备:
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
实测发现,server如果是最新的1.19以上的,会出现兼容问题,spark3.0才支持
构建镜像:
主要参考这篇:https://www.jianshu.com/p/da20133ecfea
步骤如下:
- 下载对应的spark版本,本案例以spark3.0 为例,下载地址:https://archive.apache.org/dist/spark/spark-3.0.0/,选择 spark-3.0.0-bin-hadoop2.7.tgz
- 解压,本案例解压后的文件夹为spark
- 将你编写的spark jar包放入到spark目录下,本案例直接使用已存在spark/examples/jars/spark-examples_2.12-3.0.0.jar
- 执行 ./bin/docker-image-tool.sh -r 【repourl】 -t 3.0.0 build,生成 repourl-spark:3.0.0镜像,python镜像构建:
bin/docker-image-tool.sh -r 【repourl】 -t 3.0.0 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
- docker push repourl-spark:3.0.0
- 执行spark-submit,其中你上传的jar所在位置在镜像中的 /opt/spark文件夹下
执行:
spark-submit \ --master k8s://https://k8sapi:6443 --deploy-mode cluster \ --name spark-pi --class org.apache.spark.examples.SparkPi \ --conf spark.executor.instances=2 \ --conf spark.kubernetes.container.image=repourl/spark:3.0.0 \ --conf spark.kubernetes.submission.waitAppCompletion=false \ local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar
该操作默认使用default用户,在default用户没有足够权限的情况下,该任务最终会失败
授权用户
spark driver需要获取executor的创建、运行、watch等权限,需要配置对应用户权限。本例使用spark用户,执行命令如下:
kubectl create serviceaccount spark kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
最终的执行命令
spark-submit \ --master k8s://https://k8sapiserver:6443 --deploy-mode cluster \ --name spark-pi --class org.apache.spark.examples.SparkPi \ --conf spark.executor.instances=2 \ --conf spark.kubernetes.container.image=repourl/spark:3.0.0 \ --conf spark.kubernetes.submission.waitAppCompletion=false \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar
授信:
spark在2.2之后开始支持k8s,3.0正式支持k8s,所以有很多功能其实阉割了。
最快捷的解决方案写在最下面
在k8s中运行spark最复杂的问题,我认为就是授信问题。关于授信问题,spark2.4的官方文档写的很不好,官方文档地址: http://spark.apache.org/docs/2.4.1/running-on-kubernetes.html
在授信问题上提供了很多参数如: spark.kubernetes.authenticate.submission.caCertFile,spark.kubernetes.authenticate.submission.clientKeyFile,配置这些证书过程很复杂
- 异常:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
这个是因为自签证书不受信任,需要将ca.pem导入keystore。参考https://blog.csdn.net/ljskr/article/details/84570573
- 异常
2020/10/15 11:09:49.147 WARN WatchConnectionManager : Exec Failure: HTTP 403, Status: 403 - pods "spark-pi-1602731387162-driver" is forbidden: User "system:anonymous" cannot watch resource "pods" in API group "" in the namespace "default"
这是客户端不受信任。解决方案是spark-submit添加参数
--conf spark.kubernetes.authenticate.submission.clientKeyFile=/root/admin-key.pem
--conf spark.kubernetes.authenticate.submission.clientCertFile=/root/admin.pem
或者执行:
kubectl create clusterrolebinding test:anonymous --clusterrole=cluster-admin --user=system:anonymous
这里的证书有一个配置错误都会出问题
所以最佳配置方案是:
将.kube文件拷贝到$HOME目录下
原理是:
spark使用的是io.fabric8库,虽然spark提供了一堆的参数,但是该库默认还是会寻找~/.kube/config文件。
代码逻辑:
而这里的授信认证仅仅在spark-submit时生效。当命令提交后,生成driver pod之后,授信文件pem就失去职能了
driver阶段授信:
spark on k8s大致的流程是driver pod去创建和销毁executor pod,所以driver pod的权限需要很大才行。这里需要配置RBAC,参考 http://spark.apache.org/docs/2.4.1/running-on-kubernetes.html#rbac
配置完成之后,需要告知driver以什么身份执行,因此还需要配置参数:
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
最后
k8s client和server版本兼容很烂
k8s server1.15需要搭载的client版本大概是4.6,而spark2.4.7才更新这个包版本
因此测试时格外要注意client和server版本问题
最后发布命令如下:
/Users/zhouwenyang/Desktop/tmp/spark/bin/spark-submit \ --master k8s://https://knode1:6443 --deploy-mode cluster \ --name spark-pi --class org.apache.spark.examples.SparkPi \ --conf spark.executor.instances=2 \ --conf spark.kubernetes.container.image=vm2173:5000/spark:2.4.7 \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.submission.waitAppCompletion=false \ local:///opt/spark/examples/jars/spark-examples_2.11-2.4.7.jar
spark官方配置: https://spark.apache.org/docs/3.0.0/running-on-kubernetes.html#rbac
问题记录
- 将文件广播到每个节点
sparkContext.addFile() 该方法在standalone和yarn集群都是可以将文件进行广播的,该方法的注释:
Add a file to be downloaded with this Spark job on every node. If a file is added during execution, it will not be available until the next TaskSet starts. Params: path – can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs, use SparkFiles.get(fileName) to find its download location. Note: A path can be added only once. Subsequent additions of the same path are ignored.
按照注释解释,应该是可以将文件广播到每个节点,但实际测试并没有效果。我怀疑是一个bug,【未完待续】