修改hive 默认fs为s3 遇到的坑

问题： hive我修改了 默认的f <property> <name>fs.defaultFS</name> <value>hdfs://。。。:8020</value> </property> 修改为<property> <name>fs.defaultFS</name> <value>s3://。。。</value> </property> 修改后执行窗口函数的时候会执行失败,如果不执行窗口函数的没问题。

失败hive命令简化版：select * ,row_number() over (partition by id order by utime desc) as sort_num from mobi_warehouse.table ； 成功命令简化版：select * from mobi_warehouse.table；

错误：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask

查看hive运行日志，发现报错如下：

 INFO [main([])]: session.SessionState (SessionState.java:close(1573)) - Error closing tez session java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:163) ~[hive-exec-2.3.2-amzn-2.jar:2.3.2-amzn-2] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.toString(TezSessionState.java:125) ~[hive-exec-2.3.2-amzn-2.jar:2.3.2-amzn-2] .....

这个错误指明无法找到 tez 执行引擎的 library 档案，这个路径 hdfs:///apps/tez/tez.tar.gz 表示它存储于 HDFS 上。 由于您已修改 fs.defaultFS 至 S3 桶，因此可行的解决方法是将 tez.tar.gz 也复制到您的 S3 桶中。 以下是我的操作步骤。

1. 将 tez.tar.gz 从 HDFS 上复制到 master node 的本地磁盘 $ hdfs dfs -copyToLocal /apps/tez/tez.tar.gz ./

2. 将 tez.tar.gz 复制到 S3 桶中 $ aws s3 cp ./tez.tar.gz s3://S3桶名称/

3. 修改 /etc/tez/conf/tez-site.xml, 找到 tez.lib.uris property, 并修改其值如下

<property>

<name>tez.lib.uris</name>

<value>s3://S3桶名称/tez.tar.gz</value>

</property>

修改后执行，发现仍然报如下错误：

Caused by: java.util.concurrent.ExecutionException: java.io.IOException: The ownership on the staging directory s3://mobimagic-microl-risk/tmp/hive/hadoop/_tez_session _dir/5db8dd84-5154-460d-9a6d-f875adb6fd6b is not as expected. It is owned by . The directory must be owned by the submitter hadoop or by hadoop at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_121] at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_121] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:168) ~[hive-exec-2.1.1-amzn-0.jar:2.1.1-amzn-0] ... 14 more Caused by: java.io.IOException: The ownership on the staging directory s3://mobimagic-microl-risk/tmp/hive/hadoop/_tez_session_dir/5db8dd84-5154-460d-9a6d-f875adb6fd6b is not as expected. It is owned by . The directory must be owned by the submitter hadoop or by hadoop

最终发现，当前hadoop版本是亚马逊的2.7.2，升级到2.8.3就根据如上配置就没问题了，把emr版本提高到5.12以上就可以了

如果帮到你了，请点下赞

转载注明出处