SeaTunnel Web 在 Debian 环境的安装
SeaTunnel Web 在 Debian 环境的安装
环境说明
- SeaTunnel 2.3.3
- SeaTunnel Web 1.0.0
- Debian 11.3
注: SeaTunnel Web 1.0.0 暂未适配 SeaTunnel 2.3.4 。
下载软件
下载SeaTunnel二进制文件
下载地址:https://seatunnel.apache.org/download/
- apache-seatunnel-2.3.3-bin.tar.gz
- apache-seatunnel-web-1.0.0-bin.tar.gz
解压文件:
tar -xvf apache-seatunnel-2.3.3-bin.tar.gz
tar -xvf apache-seatunnel-web-1.0.0-bin.tar.gz
得到:
apache-seatunnel-2.3.3
apache-seatunnel-web-1.0.0-bin
安装目录
mv apache-seatunnel-2.3.3 /home/st/opt/apache-seatunnel-2.3.3
mv apache-seatunnel-web-1.0.0-bin /home/st/opt/apache-seatunnel-web-1.0.0-bin
下载连接器
进入 seatunnel 目录
cd apache-seatunnel-2.3.3
连接器配置列表:
文件路径: apache-seatunnel-2.3.3/config/plugin_config
建议初始下载连接器:
--connectors-v2--
connector-fake
connector-console
connector-cdc-mysql
--end--
下载连接器插件:
bash bin/install-plugin.sh
或:
./bin/install-plugin.sh
注意: 不可以使用: sh bin/install-plugin.sh 以防解释器是 dash 而导致出错。
下载位置:
/opt/apache-seatunnel-2.3.3/connectors/seatunnel
注1: 经测试,SeaTunnel 2.3.4 与 SeaTunnel 2.3.3 下载连接器路径不同
2.3.3 : connectors/seatunnel
2.3.4 : connectors/
注 2:seatunnel web 1.0.0 将从 ${SEATUNNEL_HOME}/connectors/seatunnel 加载连接器插件.如果没有下载对应插件将导致配置任务时无法选择配置好的数据源。
注 3:install-plugin.sh
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#This script is used to download the connector plug-ins required during the running process.
#All are downloaded by default. You can also choose what you need.
#You only need to configure the plug-in name in config/plugin_config.
# get seatunnel home
SEATUNNEL_HOME=$(cd $(dirname $0);cd ../;pwd)
# connector default version is 2.3.3, you can also choose a custom version. eg: 2.1.2: sh install-plugin.sh 2.1.2
version=2.3.3
if [ -n "$1" ]; then
version="$1"
fi
echo "Install hadoop shade jar, usage version is ${version}"
${SEATUNNEL_HOME}/mvnw dependency:get -DgroupId=org.apache.seatunnel -Dclassifier=optional -DartifactId=seatunnel-hadoop3-3.1.4-uber -Dversion=${version} -Ddest=${SEATUNNEL_HOME}/lib
echo "Install SeaTunnel connectors plugins, usage version is ${version}"
# create the connectors directory
if [ ! -d ${SEATUNNEL_HOME}/connectors ];
then
mkdir ${SEATUNNEL_HOME}/connectors
echo "create connectors directory"
fi
# create the seatunnel connectors directory (for v2)
if [ ! -d ${SEATUNNEL_HOME}/connectors/seatunnel ];
then
mkdir ${SEATUNNEL_HOME}/connectors/seatunnel
echo "create seatunnel connectors directory"
fi
while read line; do
if [ ${line:0:1} != "-" ] && [ ${line:0:1} != "#" ]
then
echo "install connector : " $line
${SEATUNNEL_HOME}/mvnw dependency:get -DgroupId=org.apache.seatunnel -DartifactId=${line} -Dversion=${version} -Ddest=${SEATUNNEL_HOME}/connectors/seatunnel
fi
done < ${SEATUNNEL_HOME}/config/plugin_config
测试SeaTunnel任务
运行示例任务:
cd apache-seatunnel-2.3.3
./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local
注: 如果你使用的是 SeaTunnel 2.3.4 ,因为seatunnel源码的问题,在Linux环境执行需要批量替换代码换行符为Unix格式
find apache-seatunnel-2.3.3 -type f -print0 | xargs -0 dos2unix --
下载数据库驱动
下载MySQL驱动并添加到lib目录
如:Mysql 驱动
cp mysql-connector-j-8.2.0.jar apache-seatunnel-2.3.3/lib/
注:从 Web 发起的数据库同步批任务将使用jdbc 方式连接。Seatunnel 如果没有对应驱动,任务执行将会连接失败。
启动集群模式
mkdir logs
./bin/seatunnel-cluster.sh -d
配置web元数据库
cd apache-seatunnel-web-1.0.0-bin
cat script/seatunnel_server_env.sh
配置数据库连接信息。
注:此账号需要有创建数据库权限
export HOSTNAME="127.0.0.1"
export PORT="3306"
export USERNAME="root"
export PASSWORD="123456"
安装MySQL客户端
sudo apt install mycli
注:debian11 下没有 mysql-client
测试数据库连接:
mycli -h 127.0.0.1 -P 3306 -u root
修改初始化脚本:
script/init_sql.sh
workDir=`dirname $0`
workDir=`cd ${workDir};pwd`
source ${workDir}/seatunnel_server_env.sh
usage="Usage: seatunnel_server_env.sh must contain hostname/port/username/password."
if [[ ! -n "${HOSTNAME}" ]] || [[ ! -n "${PORT}" ]] || [[ ! -n "${USERNAME}" ]] || [[ ! -n "${PASSWORD}" ]]; then
echo $usage
exit 1
fi
mycli -h${HOSTNAME} -P${PORT} -u${USERNAME} -p${PASSWORD} < ${workDir}/seatunnel_server_mysql.sql
设置数据库连接环境变量:
export HOSTNAME="localhost"
export PORT="3306"
export USERNAME="root"
export PASSWORD="123456"
初始化数据库:
sh script/init_sql.sh
配置Web应用
配置文件路径:conf/application.yml
cat conf/application.yml
修改内容:
datasource:
# driver-class-name: com.mysql.jdbc.Driver
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://localhost:3306/seatunnel?useSSL=false&useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&allowPublicKeyRetrieval=true
username: root
password: 123456
注意:其中MySQL驱动类需要和实际的驱动类修改一致。
Mysql 8.0 及以上版本需使用 : driver-class-name: com.mysql.cj.jdbc.Driver
Mysql 5.0 等较低版本需使用:driver-class-name: com.mysql.jdbc.Driver
注:此步骤配置不正确,将导致无法在页面进行用户登录。
添加MySQL驱动
下载MySQL驱动并添加到libs目录
cp mysql-connector-j-8.2.0.jar apache-seatunnel-web-1.0.0-bin/libs/
配置 JAVA_HOME
echo "export JAVA_HOME=/path/to/java" >> ~/.profile
echo "export PATH=$JAVA_HOME/bin:$PATH" >> ~/.profile
配置 SEATUNNEL_HOME
echo "export SEATUNNEL_HOME=/home/st/opt/apache-seatunnel-2.3.3" >> ~/.profile
下载连接器到Web
下载或编写 download_datasource.sh
下载地址: https://seatunnel.apache.org/assets/files/download_datasource-4b79e6fafe80459590a6a0fc2865e5ac.sh
download_datasource.sh
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#This script is used to download the connector plug-ins required during the running process.
#All are downloaded by default. You can also choose what you need.
#You only need to configure the plug-in name in config/plugin_config.
# get seatunnel web home
SEATUNNEL_WEB_HOME=$(cd $(dirname $0);pwd)
DATASOURCE_LIB_DIR=${SEATUNNEL_WEB_HOME}/../libs
# the datasource default version is 1.0.0, you can also choose a custom version. eg: 1.1.2: sh install-datasource.sh 2.1.2
version=1.0.0
if [ -n "$1" ]; then
version="$1"
fi
# If you do not want to download a certain data source, you can delete the element below
datasource_list=(
"datasource-plugins-api"
"datasource-elasticsearch"
"datasource-hive"
"datasource-jdbc-clickhouse"
"datasource-jdbc-hive"
"datasource-jdbc-mysql"
"datasource-jdbc-oracle"
"datasource-jdbc-postgresql"
"datasource-jdbc-redshift"
"datasource-jdbc-sqlserver"
"datasource-jdbc-starrocks"
"datasource-jdbc-tidb"
"datasource-kafka"
"datasource-mysql-cdc"
"datasource-s3"
"datasource-sqlserver-cdc"
"datasource-starrocks"
)
echo "Downloading SeaTunnel Web Datasource lib, usage version is ${version}"
for datasource in "${datasource_list[@]}"
do
echo "Downloading datasource: ${datasource}"
wget https://repo.maven.apache.org/maven2/org/apache/seatunnel/${datasource}/${version}/${datasource}-${version}.jar -P $DATASOURCE_LIB_DIR
done
文件放置路径:
cp download_datasource.sh apache-seatunnel-web-1.0.0-bin/bin/
下载连接器:
sh bin/download_datasource.sh
下载路径:
apache-seatunnel-web-1.0.0-bin/libs/
注:此步骤不执行将导致SeaTunnel Web 无法在页面配置添加数据源
复制连接器映射配置文件
映射文件在seatunnel安装目录下
cp apache-seatunnel-2.3.3/connectors/plugin-mapping.properties apache-seatunnel-web-1.0.0-bin/conf/
注:此文件用于配置 seatunnel 连接器与任务执行配置时 source 或 sink 的对应关系
启动服务
sh bin/seatunnel-backend-daemon.sh start
页面测试
页面访问:
localhost:8801/
默认账号:
user: admin
password: admin
停止服务
sh bin/seatunnel-backend-daemon.sh stop
常见问题
设置应用日志级别
配置文件:
apache-seatunnel-web-1.0.0-bin/conf/logback-spring.xml
修改内容:
<root level="DEBUG">
<appender-ref ref="seatunnel-web" />
<appender-ref ref="console" />
</root>
datasource-hive-1.0.0.jar 找不到类:
***************************
APPLICATION FAILED TO START
***************************
Description:
An attempt was made to call a method that does not exist. The attempt was made from the following location:
org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory.configureSession(JettyServletWebServerFactory.java:267)
The following method did not exist:
'void org.eclipse.jetty.server.session.SessionHandler.setMaxInactiveInterval(int)'
The calling method's class, org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory, was loaded from the following location:
jar:file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/spring-boot-2.6.8.jar!/org/springframework/boot/web/embedded/jetty/JettyServletWebServerFactory.class
The called method's class, org.eclipse.jetty.server.session.SessionHandler, is available from the following locations:
jar:file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar!/org/eclipse/jetty/server/session/SessionHandler.class
jar:file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/jetty-server-9.4.46.v20220331.jar!/org/eclipse/jetty/server/session/SessionHandler.class
The called method's class hierarchy was loaded from the following locations:
org.eclipse.jetty.server.session.SessionHandler: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
org.eclipse.jetty.server.handler.ScopedHandler: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
org.eclipse.jetty.server.handler.HandlerWrapper: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
org.eclipse.jetty.server.handler.AbstractHandlerContainer: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
org.eclipse.jetty.server.handler.AbstractHandler: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
org.eclipse.jetty.util.component.ContainerLifeCycle: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/jetty-util-9.4.46.v20220331.jar
org.eclipse.jetty.util.component.AbstractLifeCycle: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/jetty-util-9.4.46.v20220331.jar
原因:未知
解决方法:删除 datasource-hive-1.0.0.jar
rm /home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· CSnakes vs Python.NET:高效嵌入与灵活互通的跨语言方案对比
· 【.NET】调用本地 Deepseek 模型
· Plotly.NET 一个为 .NET 打造的强大开源交互式图表库
· 上周热点回顾(2.17-2.23)
· 如何使用 Uni-app 实现视频聊天(源码,支持安卓、iOS)