sqoop提供数据库密码的4种方式

hadoop credential create oracle.pwd.core -provider jceks://hdfs/user/ericlin/oracle.password.core

 

sqoop import -Dhadoop.security.credential.provider.path=jceks://hdfs/user/ericlin/oracle.password.core \
 --connect jdbc:oracle:thin:@172.17.101.90:1521/bdpdb \
 --username etluser \
 --password-alias oracle.pwd.core \
 --query "select ID,DICT_CODE,DICT_NAME,DICT_TYPE,VERSION,VALIDATE_STATE from CORE_0111.SYS_DICT where \$CONDITIONS" \
 --hive-import \
 --hive-table SYS_DICT \
 --target-dir /user/SYS_DICT \
 --delete-target-dir \
 --split-by ID  \
 --hive-drop-import-delims \
 --hive-database core \
 --hive-overwrite \
 --hive-partition-key ETL_DATE \
 --hive-partition-value 2018-01-11 \
 --verbose

 

在操作的过程中失败了，出现如下错误信息：

16/12/09 17:18:47 INFO mapreduce.ImportJobBase: Transferred 49 bytes in 19.624 seconds (2.4969 bytes/sec)
16/12/09 17:18:47 INFO mapreduce.ImportJobBase: Retrieved 6 records.
16/12/09 17:18:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sss` AS t LIMIT 1
16/12/09 17:18:47 INFO hive.HiveImport: Loading uploaded data into Hive
16/12/09 17:18:47 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
16/12/09 17:18:47 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
        at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)
        at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)
        at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)
        at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
        at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:524)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:190)
        at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)
        ... 12 more

通过排查，是由于hive的的配置出现问题，解决办法 ：（在环境变量配置文件中HADOOP_CLASSPATH中添加$HIVE_HOME/lib/*）

vim ~/.bash_profile 

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*

source ~/.bash_profile

最后问题得到解决！

参考资料：

https://zhuanlan.zhihu.com/p/28321067

http://www.ericlin.me/2015/06/securely-managing-passwords-in-sqoop/