【Sqoop学习之一】Sqoop简介

环境

　　sqoop-1.4.6

 

Sqoop:将关系数据库（oracle、mysql、postgresql等）数据与hadoop数据进行转换的工具。

两个版本：两个版本完全不兼容，sqoop1使用最多：
sqoop1：1.4.x
sqoop2：1.99.x

同类产品:DataX 阿里顶级数据交换工具

一、架构
Sqoop架构非常简单，是hadoop生态系统的架构最简单的框架。sqoop1由client端直接接入hadoop，任务通过解析生成对应的maprecue执行。

二、安装

 

1、上传解压

[root@node101 src]# tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /usr/local
[root@node101 local]# mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop-1.4.6

 

2、配置sqoop-env.sh
注释掉未安装服务的相关内容，比如HBase、HCatalog、Accumulo

[root@node101 conf]# cd /usr/local/sqoop-1.4.6/conf && mv sqoop-env-template.sh sqoop-env.sh

 

#HADOOP安装目录
export HADOOP_COMMON_HOME=/usr/local/hadoop-2.6.5

#HADOOP安装目录
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/usr/local/hadoop-2.6.5

#HBASE安装目录
#set the path to where bin/hbase is available
#export HBASE_HOME=

#HIVE安装目录
#Set the path to where bin/hive is available
#export HIVE_HOME=

#ZOOKEEPER安装目录
#Set the path for where zookeper config dir is
#export ZOOCFGDIR=

 

3、这里连接关系数据库是mysql 需要依赖mysql jar包

[root@node101 src]# cp /usr/local/src/mysql-connector-java-5.1.26-bin.jar /usr/local/sqoop-1.4.6/lib/

 

4、配置环境变量

export SQOOP_HOME=/usr/local/sqoop-1.4.6

 

5、验证是否安装成功
sqoop-version 或者 sqoop version

[root@node101 conf]# sqoop version
Warning: /usr/local/sqoop-1.4.6/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/local/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/sqoop-1.4.6/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
19/07/01 16:26:50 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015

 

 

参考：

用户手册
Sqoop学习之路