Apache DolphinScheduler2.0.5集群部署

DolphinScheduler测试环境说明

角色 主机名 备注
masters hadoop01,hadoop02
workers hadoop01,hadoop02,hadoop03,hadoop04,hadoop05
alertServer hadoop04
apiServers hadoop05 172.30.96.41
pythonGatewayServers hadoop05
数据库MySQL8.0.25 hadoop05 用户名：dolphinscheduler

Apache DolphinScheduler集群部署

部署（官方文档）档：https://dolphinscheduler.apache.org/zh-cn/docs/2.0.5/user_doc/guide/installation/cluster.html
架构设计（官方文档）：https://dolphinscheduler.apache.org/zh-cn/docs/2.0.5/user_doc/architecture/design.html

分布式部署 DolphinScheduler 需要有外部软件的支持（官网）

  • JDK：下载JDK (1.8+)，并将 JAVA_HOME 配置到以及 PATH 变量中。如果你的环境中已存在，可以跳过这步。
  • 二进制包：在下载页面下载 DolphinScheduler 二进制包
  • 数据库：PostgreSQL (8.2.15+) 或者 MySQL (5.7+)，两者任选其一即可，如 MySQL 则需要 JDBC Driver 8.0.16（建议使用MySQL8.0.16+，JDBC Driver 版本与其保持一致）
  • 注册中心：ZooKeeper (3.4.6+)，下载地址
  • 进程树分析
    • macOS安装pstree
    • Fedora/Red/Hat/CentOS/Ubuntu/Debian安装psmisc

注意: DolphinScheduler 本身不依赖 Hadoop、Hive、Spark，但如果你运行的任务需要依赖他们，就需要有对应的环境支持

安装MySQL8.0.25

见博客 https://www.cnblogs.com/happy-king/p/16408631.html

连上数据库，初始化表
source ${dolphinscheduler_deploy_dir}/sql/dolphinscheduler_mysql.sql

配置修改

目录结构和配置文件（官方文档）：https://dolphinscheduler.apache.org/zh-cn/docs/2.0.5/user_doc/architecture/configuration.html
部署程序目录：dolphinscheduler_deploy_dir=/app/downloads/dolphinscheduler-bin

修改安装配置文件 ${dolphinscheduler_deploy_dir}/conf/config/install_config.conf

${dolphinscheduler_deploy_dir}/install.sh/conf/config/install_config.conf
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="hadoop01,hadoop02,hadoop03,hadoop04,hadoop05", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
**ips="hadoop01,hadoop02,hadoop03,hadoop04,hadoop05"**

# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
**sshPort="22"**

# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="hadoop01,hadoop02", Example for IPs: masters="192.168.8.1,192.168.8.2"
**masters="hadoop01,hadoop02"**

# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="hadoop01:default,hadoop02:default,hadoop03:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
**workers="hadoop01:default,hadoop02:default,hadoop03:default,hadoop04:default,hadoop05:default"**

# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="hadoop04", Example for IP: alertServer="192.168.8.3"
**alertServer="hadoop04"**

# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="hadoop05", Example for IP: apiServers="192.168.8.1"
**apiServers="hadoop05"**

# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="hadoop05", Example for IP: pythonGatewayServers="192.168.8.1"
**pythonGatewayServers="hadoop05"**

# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
**installPath="/app/3rd/dolphinscheduler"**

# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
**deployUser="dolphinscheduler"**

# The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.
**dataBasedirPath="/app/data/dolphinscheduler"**

# ---------------------------------------------------------
# DolphinScheduler ENV
# ---------------------------------------------------------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
**javaHome="/usr/lib/jvm/java-openjdk"**

# DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345
**apiServerPort="12345"**

# ---------------------------------------------------------
# Database
# NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ---------------------------------------------------------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.
**DATABASE_TYPE=${DATABASE_TYPE:-"mysql"}**

# Spring datasource url, following <HOST>:<PORT>/<database>?<parameter> format, If you using mysql, you could use jdbc
# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
#SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:h2:mem:dolphinscheduler;MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=true"}
**SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:mysql://172.30.96.24:3306/dolphinscheduler?useUnicode=true&characterEncoding=utf8&useSSL=false"}**

# Spring datasource username
#SPRING_DATASOURCE_USERNAME=${SPRING_DATASOURCE_USERNAME:-"sa"}
**SPRING_DATASOURCE_USERNAME=${SPRING_DATASOURCE_USERNAME:-"dolphinscheduler"}**

# Spring datasource password
#SPRING_DATASOURCE_PASSWORD=${SPRING_DATASOURCE_PASSWORD:-""}
**SPRING_DATASOURCE_PASSWORD=${SPRING_DATASOURCE_PASSWORD:-"password"}**

# ---------------------------------------------------------
# Registry Server
# ---------------------------------------------------------
# Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
**registryPluginName="zookeeper"**

# Registry Server address.
registryServers="hadoop01:2181,hadoop02:2181,hadoop03:2181"

# Registry Namespace
**registryNamespace="dolphinscheduler"**

# ---------------------------------------------------------
# Worker Task Server
# ---------------------------------------------------------
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"

# resource storage type: HDFS, S3, NONE
**resourceStorageType="HDFS"**

# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
**resourceUploadPath="/dolphinscheduler"**

# if resourceStorageType is HDFS，defaultFS write namenode address，HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3，write S3 address，HA，for example ：s3a://dolphinscheduler，
# Note，S3 be sure to create the root directory /dolphinscheduler
**defaultFS="hdfs://172.30.96.33:8020"**

# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"

# resourcemanager port, the default value is 8088 if not specified
**resourceManagerHttpAddressPort="8088"**

# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
**yarnHaIps="172.30.96.33,172.30.96.36,172.30.96.37"**

# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
**singleYarnIp="172.30.96.33"**

# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
**hdfsRootUser="root"**

# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username,watch out the @ sign should followd by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"

# use sudo or not
sudoEnable="true"

# worker tenant auto create
workerTenantAutoCreate="false"

若配置的是HDFS集群，需要拷贝hdfs的配置文件到dolphscheduler配置文件目录下

cp -rp ${hadoop_home_dir}/etc/hadoop/core-site.xml ${dolphinscheduler_deploy_dir}/conf/
cp -rp ${hadoop_home_dir}/etc/hadoop/hdfs-site.xml ${dolphinscheduler_deploy_dir}/conf/
chown -R dolphinscheduler:dolphinscheduler ${dolphinscheduler_deploy_dir}/conf/

修改各角色的启动参数 ${dolphinscheduler_deploy_dir}/bin/dolphinscheduler-daemon.sh

根据自身环境情况，进行修改

export DOLPHINSCHEDULER_OPTS="-server -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:$DOLPHINSCHEDULER_LOG_DIR/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -XshowSettings:vm $DOLPHINSCHEDULER_OPTS"

export DATABASE_TYPE=${DATABASE_TYPE:-"h2"}
export SPRING_PROFILES_ACTIVE=${SPRING_PROFILES_ACTIVE:-"default"}

if [ "$command" = "api-server" ]; then
  LOG_FILE="-Dlogging.config=classpath:logback-api.xml"
  CLASS=org.apache.dolphinscheduler.api.ApiApplicationServer
  HEAP_OPTS="-Xms1g -Xmx1g -Xmn512m"
  export DOLPHINSCHEDULER_OPTS="$HEAP_OPTS $DOLPHINSCHEDULER_OPTS $API_SERVER_OPTS"
  export SPRING_PROFILES_ACTIVE="${SPRING_PROFILES_ACTIVE},api,${DATABASE_TYPE}"
elif [ "$command" = "master-server" ]; then
  LOG_FILE="-Dlogging.config=classpath:logback-master.xml"
  CLASS=org.apache.dolphinscheduler.server.master.MasterServer
  HEAP_OPTS="-Xms4g -Xmx4g -Xmn2g"
  export DOLPHINSCHEDULER_OPTS="$HEAP_OPTS $DOLPHINSCHEDULER_OPTS $MASTER_SERVER_OPTS"
  export SPRING_PROFILES_ACTIVE="${SPRING_PROFILES_ACTIVE},master,${DATABASE_TYPE}"
elif [ "$command" = "worker-server" ]; then
  LOG_FILE="-Dlogging.config=classpath:logback-worker.xml"
  CLASS=org.apache.dolphinscheduler.server.worker.WorkerServer
  HEAP_OPTS="-Xms2g -Xmx2g -Xmn1g"
  export DOLPHINSCHEDULER_OPTS="$HEAP_OPTS $DOLPHINSCHEDULER_OPTS $WORKER_SERVER_OPTS"
  export SPRING_PROFILES_ACTIVE="${SPRING_PROFILES_ACTIVE},worker,${DATABASE_TYPE}"
elif [ "$command" = "alert-server" ]; then
  LOG_FILE="-Dlogback.configurationFile=conf/logback-alert.xml"
  CLASS=org.apache.dolphinscheduler.alert.AlertServer
  HEAP_OPTS="-Xms1g -Xmx1g -Xmn512m"
  export DOLPHINSCHEDULER_OPTS="$HEAP_OPTS $DOLPHINSCHEDULER_OPTS $ALERT_SERVER_OPTS"
  export SPRING_PROFILES_ACTIVE="${SPRING_PROFILES_ACTIVE},alert,${DATABASE_TYPE}"
elif [ "$command" = "logger-server" ]; then
  CLASS=org.apache.dolphinscheduler.server.log.LoggerServer
  HEAP_OPTS="-Xms1g -Xmx1g -Xmn512m"
  export DOLPHINSCHEDULER_OPTS="$HEAP_OPTS $DOLPHINSCHEDULER_OPTS $LOGGER_SERVER_OPTS"
elif [ "$command" = "standalone-server" ]; then
  CLASS=org.apache.dolphinscheduler.server.StandaloneServer
  export SPRING_PROFILES_ACTIVE="${SPRING_PROFILES_ACTIVE},standalone,${DATABASE_TYPE}"
elif [ "$command" = "python-gateway-server" ]; then
  CLASS=org.apache.dolphinscheduler.server.PythonGatewayServer
  export SPRING_PROFILES_ACTIVE="${SPRING_PROFILES_ACTIVE},python-gateway,${DATABASE_TYPE}"
else
  echo "Error: No command named '$command' was found."
  exit 1
fi

默认日志目录（根据自身情况，进行修改）：$DOLPHINSCHEDULER_HOME/logs

配置任务工具的环境变量 ${dolphinscheduler_deploy_dir}/conf/env/dolphinscheduler_env.sh

dolphinscheduler worker节点必须有对应工具客户端

export HADOOP_HOME=/app/3rd/hadoop-3.3.1
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export SPARK_HOME1=/app/3rd/spark-3.2.1-yarn
#export SPARK_HOME2=/opt/soft/spark2
#export PYTHON_HOME=/opt/soft/python
export JAVA_HOME=/usr/lib/jvm/java-openjdk
export HIVE_HOME=/app/3rd/apache-hive-3.1.3-bin
export FLINK_HOME=/app/3rd/flink-1.13.1
export DATAX_HOME=/app/3rd/datax

部署集群并启动

集群中的一个节点（部署机）上配置修改完毕、并做好免密登录后，执行安装脚本就行${dolphinscheduler_deploy_dir}/install.sh

初始化数据
sh ${dolphinscheduler_deploy_dir}/script/create-dolphinscheduler.sh
# 根据实际情况，更改变量后再执行
bash ${dolphinscheduler_deploy_dir}/install.sh

服务生命周期管理

cd ${DOLPHINSCHEDULER_HOME}/

# 一键停止集群所有服务
sh ./bin/stop-all.sh

# 一键开启集群所有服务
sh ./bin/start-all.sh

# 启停 Master
sh ./bin/dolphinscheduler-daemon.sh stop master-server
sh ./bin/dolphinscheduler-daemon.sh start master-server

# 启停 Worker
sh ./bin/dolphinscheduler-daemon.sh start worker-server
sh ./bin/dolphinscheduler-daemon.sh stop worker-server

# 启停 Api
sh ./bin/dolphinscheduler-daemon.sh start api-server
sh ./bin/dolphinscheduler-daemon.sh stop api-server

# 启停 Logger
sh ./bin/dolphinscheduler-daemon.sh start logger-server
sh ./bin/dolphinscheduler-daemon.sh stop logger-server

# 启停 Alert
sh ./bin/dolphinscheduler-daemon.sh start alert-server
sh ./bin/dolphinscheduler-daemon.sh stop alert-server

# 启停 Python Gateway
sh ./bin/dolphinscheduler-daemon.sh start python-gateway-server
sh ./bin/dolphinscheduler-daemon.sh stop python-gateway-server

dolphinscheduler UI控制台

dolphinscheduler UI登录地址： http://172.30.96.41:12345/dolphinscheduler/ui/view/login/index.html
管理用户名：admin
密码：私发

登录页面如下图

image

登录后如下图

image

问题汇总：

dolphinscheduler 使用MySQL5.7报错：

大数据程序也会报错如下
image

解决：使用mysql8.0.x

posted @ 2022-06-24 18:52  思维无界限  阅读(458)  评论(0)  编辑  收藏  举报