Sqoop下载安装配置

1 下载Sqoop

下载地址：https://archive.apache.org/dist/sqoop/

 

2 安装Sqoop

将Sqoop包上传，这里使用了Xftp

进入Sqoop安装包目录，安装

tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

为了方便，可以对Sqoop目录重命名

mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop

配置环境变量

vim /etc/profile

export SQOOP_HOME=/opt/sqoop
export PATH=$PATH:$SQOOP_HOME/bin

source /etc/profile

测试，查看Sqoop版本

 sqoop version

 

3 配置Sqoop

拷贝Sqoop配置文件并重命名

cp $SQOOP_HOME/conf/sqoop-env-template.sh $SQOOP_HOME/conf/sqoop-env.sh

编辑配置文件

vim $SQOOP_HOME/conf/sqoop-env.sh

取消注释，并修改如下配置，如果安装了Hbase，zookeeper也可以配置上

export HADOOP_COMMON_HOME=/opt/hadoop
export HADOOP_MAPRED_HOME=/opt/hadoop
export HIVE_HOME=/opt/hive

重新加载配置文件

source $SQOOP_HOME/conf/sqoop-env.sh

 

4 测试Sqoop

一般Sqoop导出的数据都是到MySQL数据库，所以这里使用MySQL数据库测试。
注意：测试前需保证你mysql所在主机和hive所在主机能正常通信，这里我测试的mysql位于本地电脑上，不在虚拟机中，所以我关闭了Windows防火墙。
将MySQL驱动包加入Sqoop依赖目录。
MySQL驱动下载地址：https://downloads.mysql.com/archives/c-j/

用查看MySQL中所有的数据库来测试

sqoop list-databases --connect jdbc:mysql://192.168.204.1:3306/?useSSL=false --username root --password '12345678'

这时遇到了错误

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
	at org.apache.sqoop.manager.MySQLManager.initOptionDefaults(MySQLManager.java:73)
	at org.apache.sqoop.manager.SqlManager.<init>(SqlManager.java:89)
	at com.cloudera.sqoop.manager.SqlManager.<init>(SqlManager.java:33)
	at org.apache.sqoop.manager.GenericJdbcManager.<init>(GenericJdbcManager.java:51)
	at com.cloudera.sqoop.manager.GenericJdbcManager.<init>(GenericJdbcManager.java:30)
	at org.apache.sqoop.manager.CatalogQueryManager.<init>(CatalogQueryManager.java:46)
	at com.cloudera.sqoop.manager.CatalogQueryManager.<init>(CatalogQueryManager.java:31)
	at org.apache.sqoop.manager.InformationSchemaManager.<init>(InformationSchemaManager.java:38)
	at com.cloudera.sqoop.manager.InformationSchemaManager.<init>(InformationSchemaManager.java:31)
	at org.apache.sqoop.manager.MySQLManager.<init>(MySQLManager.java:65)
	at org.apache.sqoop.manager.DefaultManagerFactory.accept(DefaultManagerFactory.java:67)
	at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:184)
	at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:272)
	at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:44)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringUtils
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 20 more

解决：https://blog.csdn.net/weixin_45462732/article/details/112135458
再次测试，成功。

sqoop list-databases --connect jdbc:mysql://192.168.204.1:3306/?useSSL=false --username root --password '12345678'