Sqoop 安装、配置、使用

下载：http://mirrors.hust.edu.cn/apache/

安装：  tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz  

配置：

① 修改 sqoop-env.sh mv sqoop-env-template.sh sqoop-env.sh 

export HADOOP_COMMON_HOME=/home/hadoop/apps/hadoop-2.7.5

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/hadoop/apps/hadoop-2.7.5

#set the path to where bin/hbase is available
export HBASE_HOME=/home/hadoop/apps/hbase-1.2.6

#Set the path to where bin/hive is available
export HIVE_HOME=/home/hadoop/apps/apache-hive-2.3.3-bin

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/home/hadoop/apps/zookeeper-3.4.10/conf

②加入 mysql 驱动包到 sqoop1.4.6/lib 目录下

cp mysql-connector-java-5.1.41-bin.jar /root/sqoop-1.4.7/lib/

③配置系统环境变量 vim /etc/profile 

export SQOOP_HOME=/root/sqoop-1.4.7
export PATH=$PATH:$SQOOP_HOME/bin

④使环境变量生效

⑤验证安装是否成功 sqoop-version 

 

使用：

①列出有哪些库:

sqoop list-databases 
--connect jdbc:mysql://localhost:3306/ 
--username root 
--password root

②列出有哪些表:

sqoop list-tables
--connect jdbc:mysql://localhost:3306/ 
--username root 
--password root

③Mysql导入HDFS:

sqoop import 
--connect jdbc:mysql://localhost:3306/myhive 
--username root 
--password root 
--table depart_name -m 1

④把MySQL数据库中的表数据导入到HBase:

sqoop import 
--connect jdbc:mysql://localhost:3305/myhive 
--username root 
--password root 
--table depart_name 
--hbase-table depart_name 
--column-family person 
--hbase-row-key depart_id

⑤把HDFS导出到Mysql

sqoop export 
--connect jdbc:mysql://localhost:3360/myhive
--username root 
--password root 
--table depart_name   
--export-dir /user/myhive 

⑥Sqoop Job

创建增量导入：

sqoop job 
--create myjob 
--import 
--connect jdbc:mysql://localhost:3306/myhive 
--username root 
--password root 
--table depart_name 
-m 1 
--incremental append 
--check-column id 
--last_value 1 
--verbose

查看：

sqoop job --list
sqoop job --show myjob  #myjob任务名称
sqoop job --exec myjob  #执行myjob

⑦Sqoop定时作业

Linux Crontab调度：

/root/sqoop/bin/sqoop job --exec  myjob >myjob.log 2>&1 &

 

 

Sqoop 错误：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
        at org.apache.sqoop.manager.MySQLManager.initOptionDefaults(MySQLManager.java:73)
        at org.apache.sqoop.manager.SqlManager.<init>(SqlManager.java:89)
        at com.cloudera.sqoop.manager.SqlManager.<init>(SqlManager.java:33)
        at org.apache.sqoop.manager.GenericJdbcManager.<init>(GenericJdbcManager.java:51)
        at com.cloudera.sqoop.manager.GenericJdbcManager.<init>(GenericJdbcManager.java:30)
        at org.apache.sqoop.manager.CatalogQueryManager.<init>(CatalogQueryManager.java:46)
        at com.cloudera.sqoop.manager.CatalogQueryManager.<init>(CatalogQueryManager.java:31)
        at org.apache.sqoop.manager.InformationSchemaManager.<init>(InformationSchemaManager.java:38)
        at com.cloudera.sqoop.manager.InformationSchemaManager.<init>(InformationSchemaManager.java:31)
        at org.apache.sqoop.manager.MySQLManager.<init>(MySQLManager.java:65)
        at org.apache.sqoop.manager.DefaultManagerFactory.accept(DefaultManagerFactory.java:67)
        at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:184)
        at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:272)
        at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:44)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringUtils
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 20 more

解决方法：

在apache 上下载commons-lang-2.6.jar包并上传到lib下即可，下载地址：
http://mirrors.tuna.tsinghua.edu.cn/apache//commons/lang/binaries/commons-lang-2.6-bin.zip