Sqoop 安装部署

1. 上传并解压 Sqoop 安装文件

将 sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz 安装包上传到 node-01 的 /root/ 目录下并将其解压

[root@node-01 ~]# tar -zxf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C apps/
[root@node-01 ~]# rm -rf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz 
[root@node-01 ~]# cd /root/apps/
[root@node-01 apps]# mv sqoop-1.4.7.bin__hadoop-2.6.0/ sqoop-1.4.7

2. 修改配置文件

• 进入到 conf 目录，重命名配置文件

[root@node-01 ~]# cd /root/apps/sqoop-1.4.7/conf
[root@node-01 conf]# mv sqoop-env-template.sh sqoop-env.sh

[root@hdp-01 conf]# vim sqoop-env.sh 

#取消23行注释
export HADOOP_COMMON_HOME=/root/apps/hadoop-3.2.1/

#取消26行注释
export HADOOP_MAPRED_HOME=/root/apps/hadoop-3.2.1/

#取消29行注释
export HBASE_HOME=/root/apps/hbase-2.2.7

#取消32行注释
export HIVE_HOME=/root/apps/hive-3.1.2

#取消35行注释
export ZOOCFGDIR=/root/apps/zookeeper-3.4.6

3. 将 mysql 的驱动包放入 Sqoop 的 lib 中

上传 mysql-connector-java-8.0.21.jar 到 node-01 的 /root 目录下

[root@node-01 ~]# cd /root
[root@node-01 ~]# mv /root/mysql-connector-java-8.0.21.jar /root/apps/sqoop-1.4.7/lib/
[root@node-01 ~]# mv /root/commons-lang-2.6.jar /root/apps/sqoop-1.4.7/lib/
[root@node-01 ~]# cp /root/apps/hive-3.1.2/lib/hive-common-3.1.2.jar /root/apps/sqoop-1.4.7/lib/
[root@node-01 ~]# vim /etc/profile
#行尾添加
export SQOOP_HOME=/root/apps/sqoop-1.4.7
export PATH=$PATH:$SQOOP_HOME/bin
[root@node-01 ~]# source /etc/profile

4. 验证 Sqoop 可用性

• 查看 Sqoop 帮助（help 命令）

[root@node-01 conf]# sqoop help
2020-12-03 17:25:57,421 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]  # Sqoop 命令的用法

# 以下就是 Sqoop 工具支持的所有命令
Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS 
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server 
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information 

See 'sqoop help COMMAND' for information on a specific command.

• 查看 Sqoop 安装版本（version 命令）

[root@node-01 ~]# sqoop version 
2020-12-03 01:38:43,879 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7 # 版本号
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017

• 使用 Sqoop 远程连接查看 MySQL 所有数据库列表（list-databases 命令）

[root@node-01 ~]# sqoop list-databases --connect jdbc:mysql://node-01:3306/?serverTimezone=UTC --username root --password root

mysql
information_schema
performance_schema
sys
shopcar
db
sgs
hive_remote

• 使用 Sqoop 远程连接查看 MySQL 某个数据库的所有表（list-tables 命令）

[root@node-01 ~]#  sqoop list-tables --connect jdbc:mysql://node-01:3306/db?serverTimezone=UTC --username root --password root
2020-12-03 17:49:14,368 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2020-12-03 17:49:14,493 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2020-12-03 17:49:14,632 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
t_class
t_emp
t_student
t_subject

至此，Sqoop 安装成功。