Sqoop环境安装

环境下载

　　首先将下载的 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz放到 /usr/hadoop/目录下（该目录可以自定义，一般为Hadoop集群安装目录），然后对安装包解压、修改文件名和修改用户权限。

[hadoop@centpy /]$ cd /usr/hadoop/
[hadoop@centpy hadoop]$ ls
flume  hadoop-2.6.0
[hadoop@centpy hadoop]$ rz   //上传安装包
 
[hadoop@centpy hadoop]$ ls
flume  hadoop-2.6.0  sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
[hadoop@centpy hadoop]$ tar -zxf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz   //解压
[hadoop@centpy hadoop]$ mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop   //修改安装文件目录
[hadoop@centpy hadoop]$ ls
flume  hadoop-2.6.0  sqoop  sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz 
[hadoop@centpy hadoop]$ rm -f sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz   //删除安装包
[hadoop@centpy hadoop]$ ls
flume  hadoop-2.6.0  sqoop

 

 

环境配置

1.配置hadoop版本号

　　切换到/sqoop/conf 目录下，执行以下命令修改对应Hadoop版本号为自己所安装的集群的版本号。

[hadoop@centpy hadoop]$ cd sqoop/
[hadoop@centpy sqoop]$ ls
bin            COMPILING.txt  ivy      LICENSE.txt  README.txt             sqoop-test-1.4.6.jar
build.xml      conf           ivy.xml  NOTICE.txt   sqoop-1.4.6.jar        src
CHANGELOG.txt  docs           lib      pom-old.xml  sqoop-patch-review.py  testdata
　[hadoop@centpy sqoop]$ vi build.xml

 

修改为：

 

2.配置sqoop-env.sh文件

　　执行以下命令生成sqoop-env.sh配置文件。

[hadoop@centpy sqoop]$ cd conf/
[hadoop@centpy conf]$ ls
oraoop-site-template.xml  sqoop-env-template.sh    sqoop-site.xml
sqoop-env-template.cmd    sqoop-site-template.xml
[hadoop@centpy conf]$ mv sqoop-env-template.sh sqoop-env.sh

 

  　　然后使用 vi sqoop-env.sh 命令，打开文件添加如下内容。

　　如果数据读取不涉及hbase和hive，那么相关hbase和hive的配置可以不加；如果集群有独立的zookeeper集群，那么配置zookeeper，反之，不用配置。

 

3.配置环境变量　

[hadoop@centpy conf]$ su
Password: 
[root@centpy conf]# vim /etc/profile   //追加以下内容
　　#sqoop
　　export SQOOP_HOME=/usr/hadoop/sqoop
　　export PATH=$PATH:$SQOOP_HOME/bin
[root@centpy conf]# source /etc/profile

 

4.上传jar包

　　将相关的驱动 jar 包拷贝到 sqoop/lib 目录下。数据库驱动 jar 包需要导入，这里我们使用的是 mysql 数据库，所以需要导入mysql-connector-java-5.1.32.jar包。

 

 

  5.确定用户组

　　需要确定sqoop所属用户组是否为hadoop

[hadoop@centpy hadoop]$ ll
total 12
drwxrwxr-x.  9 hadoop hadoop 4096 Jun 19 15:30 flume
drwxr-xr-x. 12 hadoop hadoop 4096 Jun 13 21:14 hadoop-2.6.0
drwxr-xr-x.  9 hadoop hadoop 4096 Jun 21 09:56 sqoop

 

　　如果不是请通过以下语句修改：

chown –R hadoop:hadoop sqoop 

 

测试运行

//首先启动hadoop集群
[hadoop@centpy sqoop]$ cd ../hadoop-2.6.0/
[hadoop@centpy hadoop-2.6.0]$ sbin/start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [centpy]
centpy: starting namenode, logging to /usr/hadoop/hadoop-2.6.0/logs/hadoop-hadoop-namenode-centpy.out
centpy: starting datanode, logging to /usr/hadoop/hadoop-2.6.0/logs/hadoop-hadoop-datanode-centpy.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop/hadoop-2.6.0/logs/hadoop-hadoop-secondarynamenode-centpy.out
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop/hadoop-2.6.0/logs/yarn-hadoop-resourcemanager-centpy.out
centpy: starting nodemanager, logging to /usr/hadoop/hadoop-2.6.0/logs/yarn-hadoop-nodemanager-centpy.out
[hadoop@centpy hadoop-2.6.0]$ jps
2624 SecondaryNameNode
3088 Jps
2769 ResourceManager
2340 NameNode
2870 NodeManager
2439 DataNode
//测试sqoop
[hadoop@centpy hadoop-2.6.0]$ cd ../sqoop/
[hadoop@centpy sqoop]$ sqoop list-databases --connect jdbc:mysql://202.193.60.117/dataweb --username root --password 20134997
Warning: /usr/hadoop/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/hadoop/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/hadoop/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/hadoop/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/06/21 10:47:00 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/06/21 10:47:00 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/06/21 10:47:01 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
dataweb
mysql
performance_schema
test

　　sqoop 命令执行成功，代表安装成功。

sqoop list-databases --connect jdbc:mysql://202.193.60.117/dataweb --username root --password 20134997

 

 　　该语句中使用我们上传的mysql-connector-java-5.1.32.jar包链接本地数据库，其中ip地址可以通过命令行中输入ipconfig进行查看。

 

　　执行该命令之前要先启动数据库服务，若出现2003错误时因为mysql服务没有成功启动，解决方案如下：

　　选中后右键开启服务即可。

　　到此，sqoop的安装和配置就结束了！

 

 以上就是博主为大家介绍的这一板块的主要内容，这都是博主自己的学习过程，希望能给大家带来一定的指导作用，有用的还望大家点个支持，如果对你没用也望包涵，有错误烦请指出。如有期待可关注博主以第一时间获取更新哦，谢谢！