CDH6.2的spark访问oss

CDH6配置oss后:
spark的配置

 

/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars    #只是这个目录就可以
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib
这几个目录可能都要放jar包

 

这个目录也可以要放:

Caused by: java.lang.ClassNotFoundException: org.jdom.input.JDOMParseException

cd /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/

ls |grep ali
rm -f aliyun-sdk-oss-2.8.3.jar
rm -f hadoop-aliyun.jar
ls |grep ali

cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /etc/hive/auxlib/
cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /etc/hive/auxlib/

cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib/

cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib/

cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/

cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/

cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/

cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/

关于阿里云OSS报错-java.lang.ClassNotFoundException: org.jdom.input.JDOMParseException:
https://tech.antfin.com/docs/2/97907

cp /data/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/jdom-1.1.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/