seatunnle安装部署
seatunnle安装部署
环境准备
安装jdk8或jdk11
在线安装
下载seatunnle
export version="2.3.3"
wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz"
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"
安装连接器
sh bin/install-plugin.sh 2.3.3
配置验证任务
env {
execution.parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
result_table_name = "fake"
row.num = 16
schema = {
fields {
name = "string"
age = "int"
}
}
}
}
transform {
FieldMapper {
source_table_name = "fake"
result_table_name = "fake1"
field_mapper = {
age = age
name = new_name
}
}
}
sink {
Console {
source_table_name = "fake1"
}
}
离线安装
下载seatunnle：https://seatunnel.apache.org/download/
解压安装包
下载所需连接器：https://repo.maven.apache.org/maven2/org/apache/seatunnel/
配置验证任务：
参照离线安装
启动seatunnel任务
cd "apache-seatunnel-${version}" ./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local
TroubleShooting
1. NoClassDefFoundError: com/sun/jersey/client/impl/CopyOnWriteHashMap
2023-11-24 14:53:45,169 INFO org.apache.seatunnel.engine.server.master.JobMaster - Init JobMaster for Job SeaTunnel_Job (780325195574411265)
2023-11-24 14:53:45,170 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (780325195574411265) needed jar urls [file:/opt/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar, file:/opt/seatunnel/apache-seatunnel-2.3.3/lib/seatunnel-transforms-v2.jar, file:/opt/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar]
2023-11-24 14:53:45,245 ERROR org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-82233] [5.1] submit job 780325195574411265 error java.lang.NoClassDefFoundError: com/sun/jersey/client/impl/CopyOnWriteHashMap
at org.apache.seatunnel.engine.server.checkpoint.CheckpointPlan$Builder.<init>(CheckpointPlan.java:66)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointPlan$Builder.<init>(CheckpointPlan.java:63)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointPlan.builder(CheckpointPlan.java:35)
at org.apache.seatunnel.engine.server.dag.physical.PhysicalPlanGenerator.lambda$generate$0(PhysicalPlanGenerator.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.apache.seatunnel.engine.server.dag.physical.PhysicalPlanGenerator.generate(PhysicalPlanGenerator.java:214)
at org.apache.seatunnel.engine.server.dag.physical.PlanUtils.fromLogicalDAG(PlanUtils.java:61)
at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:215)
at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$5(CoordinatorService.java:461)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.client.impl.CopyOnWriteHashMap
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
解决：下载jar包 jersey-bundle-1.19.4.jar，复制到${seatunnel_home}/lib
2. NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
2023-11-24 15:22:41,197 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (780332477079093249) needed jar urls [file:/opt/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar, file:/opt/seatunnel/apache-seatunnel-2.3.3/lib/seatunnel-transforms-v2.jar, file:/opt/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar]
2023-11-24 15:22:41,292 ERROR org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-387863] [5.1] submit job 780332477079093249 error java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
at org.apache.seatunnel.engine.checkpoint.storage.hdfs.common.HdfsFileStorageInstance.getOrCreateStorage(HdfsFileStorageInstance.java:53)
at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorageFactory.create(HdfsStorageFactory.java:75)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.<init>(CheckpointManager.java:103)
at org.apache.seatunnel.engine.server.master.JobMaster.initCheckPointManager(JobMaster.java:251)
at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:234)
at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$5(CoordinatorService.java:461)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 11 more
2023-11-24 15:22:41,298 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-387863] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
解决：下载jar包（seatunnel-hadoop3-3.1.4-uber-2.3.3.jar，seatunnel-hadoop3-3.1.4-uber-2.3.3-optional.jar）具体要结合自己的hadoop版本看下载哪个包，复制到${seatunnel_home}/lib
