Kubernetes搭建Hadoop服务
网上使用Kubernetes搭建Hadoop的资料较少,因此自己尝试做了一个,记录下过程和遇到的问题。
一、选择镜像
首先从官方Docker Hub中选择比较热门的镜像。这里选择了bde2020的系列镜像,因为其Githab上的资料比较完善。https://github.com/big-data-europe/docker-hadoop
二、使用docker-compose进行测试
网站上给出的是使用docker-compose运行此hadoop镜像的方法,按照网站上操作即可。
docker-compose是Docker自带的容器编排工具,操作简单,只需要将docker-compose.yml和hadoop.env文件下载到本地,使用docker-compose up命令即可启动。停止服务执行docker-compose down命令。
三、编写各个组件的Kubernetes yaml文件
上面的docker-compose案例虽然简单,但是功能较少,且运行于同一台机器上。我们要做的就是把docker-compose的yaml文件的语法改写为Kubernetes的yaml文件语法。
1.创建configmap
配置文件可以通过configmap录入。参考hadoop.env,编写configmap.yaml如下:
apiVersion: v1 kind: ConfigMap metadata: name: hadoop-config data: CORE_CONF_fs_defaultFS: "hdfs://namenode:8020" CORE_CONF_hadoop_http_staticuser_user: "root" CORE_CONF_hadoop_proxyuser_hue_hosts: "*" CORE_CONF_hadoop_proxyuser_hue_groups: "*" HDFS_CONF_dfs_webhdfs_enabled: "true" HDFS_CONF_dfs_permissions_enabled: "false" YARN_CONF_yarn_log___aggregation___enable: "true" YARN_CONF_yarn_resourcemanager_recovery_enabled: "true" YARN_CONF_yarn_resourcemanager_store_class: "org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore" YARN_CONF_yarn_resourcemanager_fs_state___store_uri: "/rmstate" YARN_CONF_yarn_nodemanager_remote___app___log___dir: "/app-logs" YARN_CONF_yarn_log_server_url: "http://historyserver:8188/applicationhistory/logs/" YARN_CONF_yarn_timeline___service_enabled: "true" YARN_CONF_yarn_timeline___service_generic___application___history_enabled: "true" YARN_CONF_yarn_resourcemanager_system___metrics___publisher_enabled: "true" YARN_CONF_yarn_resourcemanager_hostname: "resourcemanager" YARN_CONF_yarn_timeline___service_hostname: "historyserver" YARN_CONF_yarn_resourcemanager_address: "resourcemanager:8032" YARN_CONF_yarn_resourcemanager_scheduler_address: "resourcemanager:8030" YARN_CONF_yarn_resourcemanager_resource___tracker_address: "resourcemanager:8031"
2.创建namenode
hadoop节点间的通信使用hostname,但是pod在创建时会被系统随机指定一个hostname并写入自己的/etc/hosts文件中,从而造成节点间的通信问题,出现UnresolvedAddressException等错误信息。这里坑了我好久,查了很多资料才发现问题。
解决方法就是在service中将clusterIP指定为None,并在deployment中指定hostname与service名称一致。为了避免混淆,后面的service name、container name、hostname等都设为相同的值。
注意service中clusterIP一定要设定为None,否则使用yarn处理MapReduce任务时会报错!
namenode需要挂载volume,因此先编写pvc.yaml(需要先创建StorageClass,具体可参考我之前的博客https://www.cnblogs.com/00986014w/p/9406962.html):
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: hadoop-namenode-pvc spec: storageClassName: nfs accessModes: - ReadWriteMany resources: requests: storage: 1Gi
编写namenode的service和deployment文件namenode.yaml如下(把所有可能用到的端口都暴露了,其实不需要这么多):
apiVersion: v1 kind: Service metadata: name: namenode labels: name: namenode spec: ports: - port: 50070 name: http - port: 8020 name: hdfs - port: 50075 name: hdfs1 - port: 50010 name: hdfs2 - port: 50020 name: hdfs3 - port: 9000 name: hdfs4 - port: 50090 name: hdfs5 - port: 31010 name: hdfs6 - port: 8030 name: yarn1 - port: 8031 name: yarn2 - port: 8032 name: yarn3 - port: 8033 name: yarn4 - port: 8040 name: yarn5 - port: 8042 name: yarn6 - port: 8088 name: yarn7 - port: 8188 name: historyserver selector: name: namenode clusterIP: None --- apiVersion: apps/v1beta1 kind: Deployment metadata: name: namenode spec: replicas: 1 template: metadata: labels: name: namenode spec: hostname: namenode containers: - name: namenode image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8 imagePullPolicy: IfNotPresent ports: - containerPort: 50070 name: http - containerPort: 8020 name: hdfs - containerPort: 50075 name: hdfs1 - containerPort: 50010 name: hdfs2 - containerPort: 50020 name: hdfs3 - containerPort: 9000 name: hdfs4 - containerPort: 50090 name: hdfs5 - containerPort: 31010 name: hdfs6 - containerPort: 8030 name: yarn1 - containerPort: 8031 name: yarn2 - containerPort: 8032 name: yarn3 - containerPort: 8033 name: yarn4 - containerPort: 8040 name: yarn5 - containerPort: 8042 name: yarn6 - containerPort: 8088 name: yarn7 - containerPort: 8188 name: historyserver env: - name: CLUSTER_NAME value: test envFrom: - configMapRef: name: hadoop-config volumeMounts: - name: hadoop-namenode mountPath: /hadoop/dfs/name volumes: - name: hadoop-namenode persistentVolumeClaim: claimName: hadoop-namenode-pvc
2.datanode
创建3个datanode。以datanode1为例,编写datanode的datanode.yaml如下(pvc与namenode的类似,不贴出来了):
apiVersion: v1 kind: Service metadata: name: datanode1 labels: name: datanode1 spec: ports: - port: 50070 name: http - port: 8020 name: hdfs - port: 50075 name: hdfs1 - port: 50010 name: hdfs2 - port: 50020 name: hdfs3 - port: 9000 name: hdfs4 - port: 50090 name: hdfs5 - port: 31010 name: hdfs6 - port: 8030 name: yarn1 - port: 8031 name: yarn2 - port: 8032 name: yarn3 - port: 8033 name: yarn4 - port: 8040 name: yarn5 - port: 8042 name: yarn6 - port: 8088 name: yarn7 - port: 8188 name: historyserver selector: name: datanode1 clusterIP: None --- apiVersion: apps/v1beta1 kind: Deployment metadata: name: datanode1 spec: replicas: 1 template: metadata: labels: name: datanode1 spec: hostname: datanode1 containers: - name: datanode1 image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8 imagePullPolicy: IfNotPresent ports: - containerPort: 50070 name: http - containerPort: 8020 name: hdfs - containerPort: 50075 name: hdfs1 - containerPort: 50010 name: hdfs2 - containerPort: 50020 name: hdfs3 - containerPort: 9000 name: hdfs4 - containerPort: 50090 name: hdfs5 - containerPort: 31010 name: hdfs6 - containerPort: 8030 name: yarn1 - containerPort: 8031 name: yarn2 - containerPort: 8032 name: yarn3 - containerPort: 8033 name: yarn4 - containerPort: 8040 name: yarn5 - containerPort: 8042 name: yarn6 - containerPort: 8088 name: yarn7 - containerPort: 8188 name: historyserver envFrom: - configMapRef: name: hadoop-config volumeMounts: - name: hadoop-datanode1 mountPath: /hadoop/dfs/data volumes: - name: hadoop-datanode1 persistentVolumeClaim: claimName: hadoop-datanode1-pvc
创建完成后,一定要用kubectl logs查看一下日志,确认没有错误信息后再继续下一步。
3.resourcemanager
编写resourcemanager.yaml文件如下:
apiVersion: v1 kind: Service metadata: name: resourcemanager labels: name: resourcemanager spec: ports: - port: 50070 name: http - port: 8020 name: hdfs - port: 50075 name: hdfs1 - port: 50010 name: hdfs2 - port: 50020 name: hdfs3 - port: 9000 name: hdfs4 - port: 50090 name: hdfs5 - port: 31010 name: hdfs6 - port: 8030 name: yarn1 - port: 8031 name: yarn2 - port: 8032 name: yarn3 - port: 8033 name: yarn4 - port: 8040 name: yarn5 - port: 8042 name: yarn6 - port: 8088 name: yarn7 - port: 8188 name: historyserver selector: name: resourcemanager clusterIP: None --- apiVersion: apps/v1beta1 kind: Deployment metadata: name: resourcemanager spec: replicas: 1 template: metadata: labels: name: resourcemanager spec: hostname: resourcemanager containers: - name: resourcemanager image: bde2020/hadoop-resourcemanager:1.1.0-hadoop2.7.1-java8 imagePullPolicy: IfNotPresent ports: - containerPort: 50070 name: http - containerPort: 8020 name: hdfs - containerPort: 50075 name: hdfs1 - containerPort: 50010 name: hdfs2 - containerPort: 50020 name: hdfs3 - containerPort: 9000 name: hdfs4 - containerPort: 50090 name: hdfs5 - containerPort: 31010 name: hdfs6 - containerPort: 8030 name: yarn1 - containerPort: 8031 name: yarn2 - containerPort: 8032 name: yarn3 - containerPort: 8033 name: yarn4 - containerPort: 8040 name: yarn5 - containerPort: 8042 name: yarn6 - containerPort: 8088 name: yarn7 - containerPort: 8188 name: historyserver envFrom: - configMapRef: name: hadoop-config
4.nodemanager
编写nodemanager.yaml如下:
apiVersion: v1 kind: Service metadata: name: nodemanager1 labels: name: nodemanager1 spec: ports: - port: 50070 name: http - port: 8020 name: hdfs - port: 50075 name: hdfs1 - port: 50010 name: hdfs2 - port: 50020 name: hdfs3 - port: 9000 name: hdfs4 - port: 50090 name: hdfs5 - port: 31010 name: hdfs6 - port: 8030 name: yarn1 - port: 8031 name: yarn2 - port: 8032 name: yarn3 - port: 8033 name: yarn4 - port: 8040 name: yarn5 - port: 8042 name: yarn6 - port: 8088 name: yarn7 - port: 8188 name: historyserver selector: name: nodemanager1 clusterIP: None --- apiVersion: apps/v1beta1 kind: Deployment metadata: name: nodemanager1 spec: replicas: 1 template: metadata: labels: name: nodemanager1 spec: hostname: nodemanager1 containers: - name: nodemanager1 image: bde2020/hadoop-nodemanager:1.1.0-hadoop2.7.1-java8 imagePullPolicy: IfNotPresent ports: - containerPort: 50070 name: http - containerPort: 8020 name: hdfs - containerPort: 50075 name: hdfs1 - containerPort: 50010 name: hdfs2 - containerPort: 50020 name: hdfs3 - containerPort: 9000 name: hdfs4 - containerPort: 50090 name: hdfs5 - containerPort: 31010 name: hdfs6 - containerPort: 8030 name: yarn1 - containerPort: 8031 name: yarn2 - containerPort: 8032 name: yarn3 - containerPort: 8033 name: yarn4 - containerPort: 8040 name: yarn5 - containerPort: 8042 name: yarn6 - containerPort: 8088 name: yarn7 - containerPort: 8188 envFrom: - configMapRef: name: hadoop-config
5.historyserver
pvc与前面类似。编写historyserver.yaml如下:
apiVersion: v1 kind: Service metadata: name: historyserver labels: name: historyserver spec: ports: - port: 50070 name: http - port: 8020 name: hdfs - port: 50075 name: hdfs1 - port: 50010 name: hdfs2 - port: 50020 name: hdfs3 - port: 9000 name: hdfs4 - port: 50090 name: hdfs5 - port: 31010 name: hdfs6 - port: 8030 name: yarn1 - port: 8031 name: yarn2 - port: 8032 name: yarn3 - port: 8033 name: yarn4 - port: 8040 name: yarn5 - port: 8042 name: yarn6 - port: 8088 name: yarn7 - port: 8188 name: historyserver selector: name: historyserver clusterIP: None --- apiVersion: apps/v1beta1 kind: Deployment metadata: name: historyserver spec: replicas: 1 template: metadata: labels: name: historyserver spec: hostname: historyserver containers: - name: historyserver image: bde2020/hadoop-historyserver:1.1.0-hadoop2.7.1-java8 imagePullPolicy: IfNotPresent ports: - containerPort: 50070 name: http - containerPort: 8020 name: hdfs - containerPort: 50075 name: hdfs1 - containerPort: 50010 name: hdfs2 - containerPort: 50020 name: hdfs3 - containerPort: 9000 name: hdfs4 - containerPort: 50090 name: hdfs5 - containerPort: 31010 name: hdfs6 - containerPort: 8030 name: yarn1 - containerPort: 8031 name: yarn2 - containerPort: 8032 name: yarn3 - containerPort: 8033 name: yarn4 - containerPort: 8040 name: yarn5 - containerPort: 8042 name: yarn6 - containerPort: 8088 name: yarn7 - containerPort: 8188 envFrom: - configMapRef: name: hadoop-config volumeMounts: - name: hadoop-historyserver mountPath: /hadoop/yarn/timeline volumes: - name: hadoop-historyserver persistentVolumeClaim: claimName: hadoop-historyserver-pvc
以上几部分都用kubectl create创建后,参考GitHub,按照这5个部件对应的endpoint加上对应的端口,在浏览器上测试(需要在集群内部的某台机器上进行操作),如果能够正确显示Hadoop的页面,说明搭建成功!
6.测试hdfs
简单地测试一下节点间是否能够正常通行。
使用kubectl exec -it namenode /bin/bash进入namenode内部,执行hdfs dfs -put /etc/issue /,看看是否能够正常上传。
7.测试yarn
进入namenode容器内部,按照https://www.cnblogs.com/ccskun/p/7820977.html中的操作进行测试,看看任务能否正常执行,看看resourcemanager的web页面能否看到finish的任务。