hive 数据归档问题
1、对数据进行归档
alter table tableName archive partition(current='2020-07-02');
报错如下:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/tools/HadoopArchives
查看hive的日志,日志文件在 ls /tmp/${USER}/hive.log
java.lang.NoClassDefFoundError: org/apache/hadoop/tools/HadoopArchives
3、分析
hive在进行归档的时候需要使用到hadoop的archive相关的类,但是在hive的lib目录下面没有。
2、解决办法:
从hadoop的lib目录下拷贝archive相关类到hive的lib下面
① 找到相关类
[hduser@yjt hive]$ find /data1/hadoop/hadoop/ -name *archive*
② 拷贝
cp /data1/hadoop/hadoop/share/hadoop/tools/lib/hadoop-archives-2.9.2.jar /data1/hadoop/hive/lib/
3、开启archive功能
hive> set hive.archive.enabled=true;
4、测试
归档前:
开始归档:
hive> alter table hive_text archive partition(folder='docs');
intermediate.archived is hdfs://yjt:9000/user/test/hive_text/folder=docs_INTERMEDIATE_ARCHIVED
intermediate.original is hdfs://yjt:9000/user/test/hive_text/folder=docs_INTERMEDIATE_ORIGINAL
Creating data.har for hdfs://yjt:9000/user/test/hive_text/folder=docs
in hdfs://yjt:9000/user/test/hive_text/folder=docs/.hive-staging_hive_2020-07-07_14-39-31_376_7443789806325398792-1/-ext-10000/partlevel
Please wait... (this may take a while)
Moving hdfs://yjt:9000/user/test/hive_text/folder=docs/.hive-staging_hive_2020-07-07_14-39-31_376_7443789806325398792-1/-ext-10000/partlevel to hdfs://yjt:9000/user/test/hive_text/folder=docs_INTERMEDIATE_ARCHIVED
Moving hdfs://yjt:9000/user/test/hive_text/folder=docs to hdfs://yjt:9000/user/test/hive_text/folder=docs_INTERMEDIATE_ORIGINAL
Moving hdfs://yjt:9000/user/test/hive_text/folder=docs_INTERMEDIATE_ARCHIVED to hdfs://yjt:9000/user/test/hive_text/folder=docs
OK
归档后:
5、解压归档文件
hive> alter table hive_text unarchive partition(folder='docs');
记录学习和生活的酸甜苦辣.....哈哈哈