SeaTunnel Web 在 Debian 环境的安装

SeaTunnel Web 在 Debian 环境的安装

环境说明

  • SeaTunnel 2.3.3
  • SeaTunnel Web 1.0.0
  • Debian 11.3

注: SeaTunnel Web 1.0.0 暂未适配 SeaTunnel 2.3.4 。

下载软件

下载SeaTunnel二进制文件
下载地址:https://seatunnel.apache.org/download/

  • apache-seatunnel-2.3.3-bin.tar.gz
  • apache-seatunnel-web-1.0.0-bin.tar.gz
    解压文件:
tar -xvf apache-seatunnel-2.3.3-bin.tar.gz
tar -xvf apache-seatunnel-web-1.0.0-bin.tar.gz

得到:

apache-seatunnel-2.3.3
apache-seatunnel-web-1.0.0-bin

安装目录

mv apache-seatunnel-2.3.3 /home/st/opt/apache-seatunnel-2.3.3
mv apache-seatunnel-web-1.0.0-bin /home/st/opt/apache-seatunnel-web-1.0.0-bin

下载连接器

进入 seatunnel 目录

cd apache-seatunnel-2.3.3

连接器配置列表:
文件路径: apache-seatunnel-2.3.3/config/plugin_config

建议初始下载连接器:

--connectors-v2--
connector-fake
connector-console
connector-cdc-mysql
--end--

下载连接器插件:

bash bin/install-plugin.sh 
或:
./bin/install-plugin.sh 

注意: 不可以使用: sh bin/install-plugin.sh 以防解释器是 dash 而导致出错。
下载位置:
/opt/apache-seatunnel-2.3.3/connectors/seatunnel

注1: 经测试,SeaTunnel 2.3.4 与 SeaTunnel 2.3.3 下载连接器路径不同

2.3.3 : connectors/seatunnel
2.3.4 : connectors/

注 2:seatunnel web 1.0.0 将从 ${SEATUNNEL_HOME}/connectors/seatunnel 加载连接器插件.如果没有下载对应插件将导致配置任务时无法选择配置好的数据源。

注 3:install-plugin.sh

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#This script is used to download the connector plug-ins required during the running process. 
#All are downloaded by default. You can also choose what you need. 
#You only need to configure the plug-in name in config/plugin_config.

# get seatunnel home
SEATUNNEL_HOME=$(cd $(dirname $0);cd ../;pwd)

# connector default version is 2.3.3, you can also choose a custom version. eg: 2.1.2:  sh install-plugin.sh 2.1.2
version=2.3.3

if [ -n "$1" ]; then
    version="$1"
fi

echo "Install hadoop shade jar, usage version is ${version}"

${SEATUNNEL_HOME}/mvnw dependency:get -DgroupId=org.apache.seatunnel -Dclassifier=optional -DartifactId=seatunnel-hadoop3-3.1.4-uber -Dversion=${version} -Ddest=${SEATUNNEL_HOME}/lib

echo "Install SeaTunnel connectors plugins, usage version is ${version}"

# create the connectors directory
if [ ! -d ${SEATUNNEL_HOME}/connectors ];
  then
      mkdir ${SEATUNNEL_HOME}/connectors
      echo "create connectors directory"
fi

# create the seatunnel connectors directory (for v2)
if [ ! -d ${SEATUNNEL_HOME}/connectors/seatunnel ];
  then
      mkdir ${SEATUNNEL_HOME}/connectors/seatunnel
      echo "create seatunnel connectors directory"
fi  

while read line; do
    if  [ ${line:0:1} != "-" ] && [ ${line:0:1} != "#" ]
      	then
      		echo "install connector : " $line
      		${SEATUNNEL_HOME}/mvnw dependency:get -DgroupId=org.apache.seatunnel -DartifactId=${line} -Dversion=${version} -Ddest=${SEATUNNEL_HOME}/connectors/seatunnel
    fi

done < ${SEATUNNEL_HOME}/config/plugin_config

测试SeaTunnel任务

运行示例任务:

cd apache-seatunnel-2.3.3
./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local

注: 如果你使用的是 SeaTunnel 2.3.4 ,因为seatunnel源码的问题,在Linux环境执行需要批量替换代码换行符为Unix格式

find apache-seatunnel-2.3.3 -type f -print0 | xargs -0 dos2unix --

下载数据库驱动

下载MySQL驱动并添加到lib目录
如:Mysql 驱动

cp mysql-connector-j-8.2.0.jar apache-seatunnel-2.3.3/lib/

注:从 Web 发起的数据库同步批任务将使用jdbc 方式连接。Seatunnel 如果没有对应驱动,任务执行将会连接失败。

启动集群模式

mkdir logs
./bin/seatunnel-cluster.sh -d

配置web元数据库

cd apache-seatunnel-web-1.0.0-bin
cat script/seatunnel_server_env.sh

配置数据库连接信息。
注:此账号需要有创建数据库权限

export HOSTNAME="127.0.0.1"
export PORT="3306"
export USERNAME="root"
export PASSWORD="123456"

安装MySQL客户端

sudo apt install mycli

注:debian11 下没有 mysql-client

测试数据库连接:

mycli -h 127.0.0.1 -P 3306 -u root 

修改初始化脚本:
script/init_sql.sh

workDir=`dirname $0`
workDir=`cd ${workDir};pwd`

source ${workDir}/seatunnel_server_env.sh

usage="Usage: seatunnel_server_env.sh must contain hostname/port/username/password."

if [[ ! -n "${HOSTNAME}" ]]  || [[ ! -n "${PORT}" ]] || [[ ! -n "${USERNAME}" ]] || [[ ! -n "${PASSWORD}" ]]; then
    echo $usage
    exit 1
fi

mycli -h${HOSTNAME} -P${PORT} -u${USERNAME} -p${PASSWORD} < ${workDir}/seatunnel_server_mysql.sql

设置数据库连接环境变量:

export HOSTNAME="localhost"
export PORT="3306"
export USERNAME="root"
export PASSWORD="123456"

初始化数据库:

sh script/init_sql.sh

配置Web应用

配置文件路径:conf/application.yml

cat conf/application.yml

修改内容:

datasource:
    # driver-class-name: com.mysql.jdbc.Driver
    driver-class-name: com.mysql.cj.jdbc.Driver
    url: jdbc:mysql://localhost:3306/seatunnel?useSSL=false&useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&allowPublicKeyRetrieval=true
    username: root
    password: 123456

注意:其中MySQL驱动类需要和实际的驱动类修改一致。
Mysql 8.0 及以上版本需使用 : driver-class-name: com.mysql.cj.jdbc.Driver
Mysql 5.0 等较低版本需使用:driver-class-name: com.mysql.jdbc.Driver

注:此步骤配置不正确,将导致无法在页面进行用户登录。

添加MySQL驱动

下载MySQL驱动并添加到libs目录

cp mysql-connector-j-8.2.0.jar apache-seatunnel-web-1.0.0-bin/libs/

配置 JAVA_HOME

echo "export JAVA_HOME=/path/to/java" >> ~/.profile
echo "export PATH=$JAVA_HOME/bin:$PATH" >> ~/.profile

配置 SEATUNNEL_HOME

echo "export SEATUNNEL_HOME=/home/st/opt/apache-seatunnel-2.3.3" >> ~/.profile

下载连接器到Web

下载或编写 download_datasource.sh
下载地址: https://seatunnel.apache.org/assets/files/download_datasource-4b79e6fafe80459590a6a0fc2865e5ac.sh

download_datasource.sh

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#This script is used to download the connector plug-ins required during the running process.
#All are downloaded by default. You can also choose what you need.
#You only need to configure the plug-in name in config/plugin_config.

# get seatunnel web home
SEATUNNEL_WEB_HOME=$(cd $(dirname $0);pwd)

DATASOURCE_LIB_DIR=${SEATUNNEL_WEB_HOME}/../libs

# the datasource default version is 1.0.0, you can also choose a custom version. eg: 1.1.2:  sh install-datasource.sh 2.1.2
version=1.0.0

if [ -n "$1" ]; then
    version="$1"
fi

# If you do not want to download a certain data source, you can delete the element below
datasource_list=(
  "datasource-plugins-api"
  "datasource-elasticsearch"
  "datasource-hive"
  "datasource-jdbc-clickhouse"
  "datasource-jdbc-hive"
  "datasource-jdbc-mysql"
  "datasource-jdbc-oracle"
  "datasource-jdbc-postgresql"
  "datasource-jdbc-redshift"
  "datasource-jdbc-sqlserver"
  "datasource-jdbc-starrocks"
  "datasource-jdbc-tidb"
  "datasource-kafka"
  "datasource-mysql-cdc"
  "datasource-s3"
  "datasource-sqlserver-cdc"
  "datasource-starrocks"
)

echo "Downloading SeaTunnel Web Datasource lib, usage version is ${version}"

for datasource in "${datasource_list[@]}"
do
    echo "Downloading datasource: ${datasource}"
  wget  https://repo.maven.apache.org/maven2/org/apache/seatunnel/${datasource}/${version}/${datasource}-${version}.jar -P $DATASOURCE_LIB_DIR
done

文件放置路径:

cp download_datasource.sh  apache-seatunnel-web-1.0.0-bin/bin/

下载连接器:

sh bin/download_datasource.sh 

下载路径:

apache-seatunnel-web-1.0.0-bin/libs/

注:此步骤不执行将导致SeaTunnel Web 无法在页面配置添加数据源

复制连接器映射配置文件

映射文件在seatunnel安装目录下

cp apache-seatunnel-2.3.3/connectors/plugin-mapping.properties apache-seatunnel-web-1.0.0-bin/conf/

注:此文件用于配置 seatunnel 连接器与任务执行配置时 source 或 sink 的对应关系

启动服务

sh bin/seatunnel-backend-daemon.sh start

页面测试

页面访问:

localhost:8801/

默认账号:

user: admin
password: admin

停止服务

sh bin/seatunnel-backend-daemon.sh stop

常见问题

设置应用日志级别

配置文件:

apache-seatunnel-web-1.0.0-bin/conf/logback-spring.xml

修改内容:

    <root level="DEBUG">
        <appender-ref ref="seatunnel-web" />
        <appender-ref ref="console" />
    </root>

datasource-hive-1.0.0.jar 找不到类:

***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory.configureSession(JettyServletWebServerFactory.java:267)

The following method did not exist:

    'void org.eclipse.jetty.server.session.SessionHandler.setMaxInactiveInterval(int)'

The calling method's class, org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory, was loaded from the following location:

    jar:file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/spring-boot-2.6.8.jar!/org/springframework/boot/web/embedded/jetty/JettyServletWebServerFactory.class

The called method's class, org.eclipse.jetty.server.session.SessionHandler, is available from the following locations:

    jar:file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar!/org/eclipse/jetty/server/session/SessionHandler.class
    jar:file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/jetty-server-9.4.46.v20220331.jar!/org/eclipse/jetty/server/session/SessionHandler.class

The called method's class hierarchy was loaded from the following locations:

    org.eclipse.jetty.server.session.SessionHandler: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.server.handler.ScopedHandler: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.server.handler.HandlerWrapper: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.server.handler.AbstractHandlerContainer: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.server.handler.AbstractHandler: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.util.component.ContainerLifeCycle: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/jetty-util-9.4.46.v20220331.jar
    org.eclipse.jetty.util.component.AbstractLifeCycle: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/jetty-util-9.4.46.v20220331.jar

原因:未知
解决方法:删除 datasource-hive-1.0.0.jar

rm /home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
posted @   葵花牌、阳光  阅读(187)  评论(0编辑  收藏  举报
相关博文:
阅读排行:
· CSnakes vs Python.NET:高效嵌入与灵活互通的跨语言方案对比
· 【.NET】调用本地 Deepseek 模型
· Plotly.NET 一个为 .NET 打造的强大开源交互式图表库
· 上周热点回顾(2.17-2.23)
· 如何使用 Uni-app 实现视频聊天(源码,支持安卓、iOS)
点击右上角即可分享
微信分享提示