b0115 flink 1.13.2 使用遇到问题

目录

• 环境
• 问题
  • 1 org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. 20220514

 

---

回到顶部

环境

flink 1.13.2   scala 2.12   jdk8   python 3.8   hadoop 3.3.1

linux centos 7    win7 

linux hc2108 ，flink 单机  , hc2107客户端

linux hc2151-hc2156 flink集群 ,  hc2150客户端

回到顶部

问题

1 org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. 20220514

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.
    at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:530)
    at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407)
    at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
    at org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:288)
    at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:110)
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:174)
    ... 19 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
    at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55)
    at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526)

描述

hc2108 单机  flink 运行 

flink run --jobmanager hc2108:8081  $FLINK_HOME/examples/batch/WordCount.jar -input hdfs://hc2108:9000/data/T/2021/wordcount -output hdfs://hc2108:9000/out/T/2022/flink/t0514_run1

解决

在环境变量中添加：

export HADOOP_CLASSPATH=`hadoop classpath`

 

参考

https://blog.csdn.net/qq_43224174/article/details/122311901