大数据基础之Oozie(4)oozie使用的spark版本升级
oozie默认使用的spark是1.6,一直没有升级,如果想用最新的2.4,需要自己手工升级
首先看当前使用的spark版本的jar
# oozie admin -oozie http://$oozie_server:11000/oozie -shareliblist # oozie admin -oozie http://$oozie_server:11000/oozie -shareliblist spark
升级:
1 手工修改hdfs上的sharelib目录,同时还要拷贝原有的oozie-sharelib-spark.jar
# hdfs dfs -mv /user/oozie/share/lib/lib_20190424195113/spark /user/oozie/share/lib/lib_20190424195113/spark1 # hdfs dfs -mkdir /user/oozie/share/lib/lib_20190424195113/spark # hdfs dfs -copyFromLocal $SPARK_HOME/jars/* /user/oozie/share/lib/lib_20190424195113/spark/ # hdfs dfs -copyFromLocal $SPARK_HOME/python/lib/*.zip /user/oozie/share/lib/lib_20190424195113/spark/
# hdfs dfs -cp /user/oozie/share/lib/lib_20190424195113/spark1/oozie-sharelib-spark.jar /user/oozie/share/lib/lib_20190424195113/spark/
2 调用oozie的sharelibupdate命令
# oozie admin -oozie http://$oozie_server:11000/oozie -sharelibupdate
3 确认spark版本是否更新
# oozie admin -oozie http://$oozie_server:11000/oozie -shareliblist spark
---------------------------------------------------------------- 结束啦,我是大魔王先生的分割线 :) ----------------------------------------------------------------
- 由于大魔王先生能力有限,文中可能存在错误,欢迎指正、补充!
- 感谢您的阅读,如果文章对您有用,那么请为大魔王先生轻轻点个赞,ありがとう