Sqoop 安装配置与使用

Sqoop 只用在一台有 HIVE 的服务器上安装即可
这里我选择的是 slave2

下载安装#

Apacha Sqoop 官方下载地址

http://archive.apache.org/dist/sqoop/1.4.7/

1. 下载上传至 /opt/software
2. 解压至 /opt/servers

tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/servers

cd /opt/servers

mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-1.4.7

设置环境变量#

# Sqoop
export SQOOP_HOME=/opt/servers/sqoop-1.4.7
export PATH=$PATH:$SQOOP_HOME/bin

使其生效

source /etc/profile

创建并修改配置文件#

cd $SQOOP_HOME/conf
cp sqoop-env-template.sh sqoop-env.sh

vim sqoop-env.sh
在文件末尾增加：

export HADOOP_COMMON_HOME=/opt/servers/hadoop-2.10.1
export HADOOP_MAPRED_HOME=/opt/servers/hadoop-2.10.1
export HIVE_HOME=/opt/servers/hive-2.3.9

链接 Jar 包#

JDBC#

注意自己的JDBC版本

ln -s /opt/servers/hive-2.3.9/lib/mysql-connector-java-8.0.26.jar /opt/servers/sqoop-1.4.7/lib

hive-common.jar#

将$HIVE_HOME/lib下的hive-common-2.3.7.jar,拷贝到$SQOOP_HOME/lib目录下。如不拷贝在MySQL往Hive导数据的时候将会出现错误：ClassNotFoundException:org.apache.hadoop.hive.conf.HiveConf

ln -s /opt/servers/hive-2.3.9/lib/hive-common-2.3.9.jar /opt/servers/sqoop-1.4.7/lib

验证安装#

sqoop version

数据导入#

懒了，有需要在我补充上去。。。。

数据导出Mysql#

CREATE TABLE jobdata_origin (
	city VARCHAR ( 100 ) COMMENT '城市',
	salary VARCHAR ( 100 ) COMMENT '薪资',
	company VARCHAR ( 100 ) COMMENT '福利',
`kill` VARCHAR ( 100 ) COMMENT '技能' 
) COMMENT "原始职位数据表";

sqoop export \
--connect jdbc:mysql://slave2:3306/fengrengui \
--username hive \
--password 0000 \
--table jobdata_origin \
-m 1 \
--export-dir /user/hive/warehouse/fengrengui.db/ods_jobdata_origin \n
--input-fields-terminated-by ","

导出成功后到mysql查看是否有数据

SELECT * FROM jobdata_origin;