troubleshooting sink到hdfs 的文件压缩问题

　　

 

启动flume  后报错

 

    at java.lang.Thread.run(Thread.java:748)
2020-06-23 23:05:59,992 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.jav
2020-06-23 23:06:00,023 (conf-file-poller-0) [ERROR - org.apache.flume.node.AbstractConfigurationProvider.loadSinks(Abstracue to an error during configuration
java.lang.IllegalArgumentException: Unsupported compression codec lzop.  Please choose from: [None, BZip2Codec, DefaultCode
    at org.apache.flume.sink.hdfs.HDFSEventSink.getCodec(HDFSEventSink.java:334)
    at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:237)
    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:411)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileCo
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2020-06-23 23:06:00,026 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(A to [r1]
2020-06-23 23:06:00,033 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(A to [r2]
2020-06-23 23:06:00,034 (conf-file-poller-0) [INFO - org.apache.flume.node.Appli

 

 

 

差了一番资料 总结出一个比较简单有效的方法，分两步

第一步：就是把用lzo格式保存文件的hadoop集群中 的core-site.xml 文件直接拿过来放到flume/conf下面

主要是用里面

<configuration>

<property>
    <name>io.compression.codecs</name>
    <value>
         com.hadoop.compression.lzo.LzoCodec,
         com.hadoop.compression.lzo.LzopCodec
     </value>
  </property>
  <property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
 </property>

</configuration>

这些属性告诉flume 解压的类在哪里

第二部 上你们集群搜索下

find / -name hadoop-lzo-*

/share/hadoop/common/hadoop-lzo-0.X.X-SNAPSHOT.jar

肯定有类似于这样的jar

sz hadoop-lzo-0.X.X-SNAPSHOT.jar 下来

rz 到你部署flume机器上的 hadoop 的/share/hadoop/common/ 下


————————————————
版权声明：本文为CSDN博主「旋转的锤子」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/cybj123/article/details/75670047