Spark on Yarn 单节点集群 on minikube 的 Mac OS部署趟坑指北
一、环境准备
1.1 安装 Docker-ce
一键式下载安装
https://docs.docker.com/docker-for-mac/install/
1.2 安装 MiniKube
官网 https://kubernetes.io/docs/tasks/tools/install-minikube/
以 macOS 为例：
1. 验证是否开启虚拟化，有输入证明已经开启虚拟化
$ sysctl -a | grep -E --color 'machdep.cpu.features|VMX'
2. 安装 kubernetes-cli，kubectx（可选），kube-ps1（可选）
$ brew install kubernetes-cli kubectx kube-ps1
3. 配置 zsh plugin（可选）
$ vim ~/.zshrc
# 加入
export KUBE_PS1_ENABLED="off"
plugins=(
git
brew
kubectl
kube-ps1
)
source "/usr/local/opt/kube-ps1/share/kube-ps1.sh"
PS1='$(kube_ps1)'$PS1
之后就可以使用 kubectl 的命令缩写和自动补齐功能
4. 安装 Hypervisor
可以安装一下三个虚拟驱动作为监管系统，如果不安装默认使用HyperKit
5. 安装 minikube
$ brew install minikube
6. 尝试启动 minikube，可能需要梯子
# <driver_name> = 上面所安装的虚拟驱动，注意是全小写，还要记得加上引号
$ minikube start --vm-driver=<driver_name>
# 比如我启动的命令
$ minikube start --vm-driver='virtualbox' --kubernetes-version='v1.14.8' --extra-config=apiserver.service-node-port-range=1-65535 --alsologtostderr
# 因为 Spark 和 Yarn 都很需要计算资源
# 所以这里必须使用 3个以上CPU和尽量大的内存
# 但是 minikube start 我加入 --memory 和 --cpus 无效
# 所以需要打开 VirtualBox 手动修改一下
7. 验证开启成功
$ minikube status
# 输出如下内容证明启动成功
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
8. 使用 Minikube Docker Deamon，避免镜像反复拉取
$ eval $(minikube docker-env)
9. 配置 minikube 网络
# 这里需要每次手动修改 minikube 的 /etc/hosts 文件
# 其实可以用 minikube mount 把文件挂载持久化
# 但是 minikube 的 mount 有些小 Bug，比如
# 1. 需要一直开启命令行保持挂载
# 2. 挂载之后如果关闭就GG了，再也挂载不上了（官网issue已经解决）
# 获得 minikube 虚拟机的 IP
$ minikube ip
192.168.99.102
# 进入minikube 修改 /etc/hosts
$ minikube ssh
$ su
$ vi /etc/hosts
127.0.0.1 localhost
192.168.99.102 minikube
10. 配置本机网络
$ sudo vim /etc/hosts
192.168.99.102 minikube
原文地址 https:////www.cnblogs.com/clockq/p/12190971.html
二、启动 HDSP 集群
按照下列配置逐个启动
如果不更改任何配置，也可以直接 kubectl -f https://github.com/PharbersDeveloper/bp-hadoop-container/blob/minikube-v1.1.0/hdsp-deploy/mandatory.yaml 直接全部启动
2.1 Namespace 配置
apiVersion: v1
kind: Namespace
metadata:
name: hdsp-ns
labels:
app.kubernetes.io/name: hdsp-ns
app.kubernetes.io/part-of: hdsp-ns
2.2 StorageClass 配置
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-sc
namespace: hdsp-ns
annotations:
storageclass.kubernetes.io/is-default-class: "true"
creationTimestamp: "2019-12-18"
resourceVersion: "v0.1"
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain # Supported policies: Delete, Retain
allowVolumeExpansion: false
volumeBindingMode: WaitForFirstConsumer
2.3 master PV 配置
需要提前创建 PV 的挂载目录
$ minikube ssh
$ su
$ mkdir -p /tmp/hostpath_pv/k8s-master-hdfs-pv
$ mkdir -p /tmp/hostpath_pv/k8s-worker1-hdfs-pv
apiVersion: v1
kind: PersistentVolume
metadata:
name: master-hdfs-pv
namespace: hdsp-ns
annotations:
use: &use master-hdfs
node_name: &node_name minikube
local_path: &local_path /tmp/hostpath_pv/k8s-master-hdfs-pv
capacity_storage: &capacity_storage 2Gi
labels:
use: *use
app: hdsp
spec:
capacity:
storage: *capacity_storage
accessModes:
- ReadWriteOnce
storageClassName: local-sc
local:
path: *local_path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- *node_name
2.4 master PVC 配置
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: master-hdfs-pvc
namespace: hdsp-ns
annotations:
use: &use master-hdfs
node_name: &node_name minikube
requests_storage: &requests_storage 2Gi
labels:
use: &use master-hdfs
app: hdsp
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-sc
resources:
requests:
storage: *requests_storage
selector:
matchLabels:
use: *use
2.5 master deamon set 配置
先配置 node 标签
$ kubectl label nodes minikube hdsprole=master
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: hdsp-master
namespace: hdsp-ns
labels:
app: hdsp
noderole: master
spec:
selector:
matchLabels:
app: hdsp
noderole: master
template:
metadata:
labels:
app: hdsp
noderole: master
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
- matchExpressions:
- key: hdsprole
operator: In
values:
- master
hostNetwork: true
containers:
- name: hdsp-master
image: pharbers/hadoop:minikube
imagePullPolicy: IfNotPresent
args:
- "-d"
- "-master"
securityContext:
privileged: true
volumeMounts:
- mountPath: /usr/soft/hadoop/hdfs
name: hadoop-data
livenessProbe:
httpGet:
path: /cluster
port: 8088
initialDelaySeconds: 100
timeoutSeconds: 20
periodSeconds: 100
imagePullSecrets:
- name: registry-secret
volumes:
- name: hadoop-data
persistentVolumeClaim:
claimName: master-hdfs-pvc
2.6 测试
$ kubectl -n hdsp-ns describe pod hdsp-master-m75n4
输出如下信息表示启动成功
...
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned hdsp-ns/hdsp-master-m75n4 to minikube
Warning Unhealthy 4m6s (x3 over 7m26s) kubelet, minikube Liveness probe failed: Get http://192.168.99.102:8088/cluster: dial tcp 192.168.99.102:8088: connect: connection refused
Normal Killing 4m6s kubelet, minikube Container hdsp-master failed liveness probe, will be restarted
Normal Pulled 3m36s (x2 over 10m) kubelet, minikube Container image "pharbers/hadoop:minikube" already present on machine
Normal Created 3m36s (x2 over 10m) kubelet, minikube Created container hdsp-master
Normal Started 3m36s (x2 over 10m) kubelet, minikube Started container hdsp-master
如果输入结果类似下方，证明 Yarn 集群启动失败，无法查看 Yarn UI 界面，存活探针出错，Pod会自动重启
...
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned hdsp-ns/hdsp-master-m75n4 to minikube
Warning Unhealthy 4m6s (x3 over 7m26s) kubelet, minikube Liveness probe failed: Get http://192.168.99.102:8088/cluster: dial tcp 192.168.99.102:8088: connect: connection refused
2.7 源码
上方教程使用已编好的镜像，如果需要修改配置或查看源码，欢迎查看 https://github.com/PharbersDeveloper/bp-hadoop-container/tree/hdsp-on-minikube
三、minikube 常见问题
3.1 因为环境隔离，镜像不方便运行
- 使用 Minikube Docker Deamon，避免镜像反复拉取
$ eval $(minikube docker-env)
- 拷贝本地镜像到 minikube
$ docker save pharbers/hadoop:using | (eval $(minikube docker-env) && docker load)
3.2 使用 NodePort 范围受限
扩大 NodePort 范围
minikube start --extra-config=apiserver.service-node-port-range=1-65535
3.3 如果 Pod 需要回连宿主机
获得与 Pod 同网段宿主 IP
minikube ssh "route -n | grep ^0.0.0.0 | awk '{ print \$2 }'"
