Sqoop入门
1 下载地址
安装包 sqoop-1.4.6-cdh5.7.0.tar.gz 上传到hadoop服务器环境
2 解压软件
[root@hdp1 /mnt/software]#tar -zvxf sqoop-1.4.6-cdh5.7.0.tar.gz
3 重命名配置文件
[root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0]#cd conf/
[root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/conf]#ls
oraoop-site-template.xml sqoop-env-template.sh sqoop-site.xml
sqoop-env-template.cmd sqoop-site-template.xml
[root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/conf]#cp sqoop-env-template.sh sqoop-env.sh
4 修改配置文件
[root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/conf]#vim sqoop-env.sh
export HADOOP_COMMON_HOME=/mnt/software/hadoop-2.6.0-cdh5.7.0
export HADOOP_MAPRED_HOME=/mnt/software/hadoop-2.6.0-cdh5.7.0
export HIVE_HOME=/mnt/software/hive-1.1.0-cdh5.7.0
export ZOOKEEPER_HOME=/mnt/software/zookeeper-3.4.5-cdh5.7.05
export ZOOCFGDIR=/mnt/software/zookeeper-3.4.5-cdh5.7.0
5 拷贝jdbc驱动 到 Sqoop lib目录
[root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/conf]#cp /mnt/software/hive-1.1.0-cdh5.7.0/lib/mysql-connector-java-5.1.45.jar /mnt/software/sqoop-1.4.6-cdh5.7.0/lib/
6 验证Sqoop
配置Sqoop环境变量
[root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/bin]#vim /etc/profile
export SQOOP_HOME=/mnt/software/sqoop-1.4.6-cdh5.7.0
export PATH=.:$PATH:$SQOOP_HOME/bin
[root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/bin]#source /etc/profile
验证Sqoop
7 测试Sqoop是否能够成功连接数据库
[root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0]#sqoop list-databases --connect jdbc:mysql://hdp1:3306/ --username root --password yourpasswd
8 导入数据
(1)确定Mysql服务开启正常
(2)在Mysql中新建一张表并插入一些数据
[root@hdp1 /mnt/software/hive-1.1.0-cdh5.7.0]#mysql -uroot -p yourpasswd
mysql> create database company;
mysql> create table company.staff(id int(4) primary key not null auto_increment, name varchar(255), sex varchar(255));
mysql> insert into company.staff(name, sex) values('ajin', 'Male’);
mysql> insert into company.staff(name, sex) values('awen', 'FeMale');
(3)导入数据
全部导入
sqoop import \
--connect jdbc:mysql://hdp1:3306/company \
--username root \
--password yourpasswd \
--table staff \
--target-dir /mnt/software/tmp/sqoopdir \
--delete-target-dir \
--num-mappers 1 \
--fields-terminated-by “\t"
遇到异常
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject
解决异常
这是因为sqoop缺少java-json.jar包. 下载jar包导入lib目录下 http://www.java2s.com/Code/Jar/j/Downloadjavajsonjar.htm
mv java-json.jar /mnt/software/sqoop-1.4.6-cdh5.7.0/lib
继续导入
再次遇到异常
设置accumulo目录
[root@hdp1 /mnt/software/tmp]#mkdir /mnt/software/tmp/accumulo
导入环境变量
[root@hdp1 /mnt/software/tmp]#export ACCUMULO_HOME=/mnt/software/tmp/accumulo
再次导入 成功