数据迁移工具sqoop

有问题。。。。。。。。

数据迁移工具sqoop  

sqoop安装

[root@sqoop data]# wget  wget http://apache.fayea.com/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

[root@sqoop data]# tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz 

拷贝hadoop 目录到 sqoop服务器/data目录上

[root@sqoop data]#  scp -r namenode:/data/hadoop /data/ 

修改sqoop 配置文件，指定hadoop 目录

[root@sqoop conf]# cp /data/sqoop/conf/sqoop-env-template.sh  /data/sqoop/conf/sqoop-env.sh

[root@sqoop conf]# vim /data/sqoop/config/sqoop-env.sh

export HADOOP_COMMON_HOME=/data/hadoop
export HADOOP_MAPRED_HOME=/data/hadoop/share/hadoop/mapreduce

关闭hbase hive zk 等报警，通过sqoop 可以把 关系型数据库存储到 hbase中

[root@sqoop bin]# vim /data/sqoop/bin/configure-sqoop

## Moved to be a runtime check in sqoop.
#if [ ! -d "${HBASE_HOME}" ]; then
#  echo "Warning: $HBASE_HOME does not exist! HBase imports will fail."
#  echo 'Please set $HBASE_HOME to the root of your HBase installation.'
#fi

## Moved to be a runtime check in sqoop.
#if [ ! -d "${HCAT_HOME}" ]; then
#  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
#  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
#fi

#if [ ! -d "${ACCUMULO_HOME}" ]; then
#  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
fi
#if [ ! -d "${ZOOKEEPER_HOME}" ]; then
#  echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.'
#fi

 

  ###也可以设置环境变量

vim /etc/profile
HADOOP_HOME=/data/hadoop/
PATH=$HADOOP_HOME/bin:$PATH:$HOME/bin
HADOOP_MAPRED_HOME=/data/hadoop/share/hadoop/mapreduce
PATH=$HADOOP_MAPRED_HOME/bin:$PATH:$HOME/bin

 

 sqool 使用命名

[root@sqoop bin]# ./sqoop help
16/12/12 14:51:49 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.

 

官方文档：http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html

把192.168.20.60 的mysql 的senyintmain 导入到 hdfs中

wget http://central.maven.org/maven2/mysql/mysql-connector-java/6.0.5/mysql-connector-java-6.0.5.jar

cp mysql-connector-java-6.0.5.jar /data/sqoop/lib

[hadoop@sqoop ~]$ /data/sqoop/bin/sqoop import  --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --table sys_log

 

 

 

hdfs 权限是 hadoop. supergroup,所以需要把 /data  hadoop  sqoop 修改权限

[root@sqoop data]# chown -R hadoop.hadoop  hadoop/ sqoop

[root@sqoop data]# su - hadoop

[hadoop@sqoop data]# [hadoop@sqoop ~]$ /data/sqoop/bin/sqoop import  --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --table sys_log

 

database 没有数据库权限,在mysql 192.168.20.60服务器上添加权限

[root@localhost:mysql>grant all on  *.* to 'fengjian'@'192.168.20.%'  IDENTIFIED BY '123456';

 

在此执行

[hadoop@sqoop data]# [hadoop@sqoop ~]$ /data/sqoop/bin/sqoop import  --connect jdbc:mysql://192.168.20.60:3306/fengjian --username feng --password 123456 --table sys_log --target-dir /sqoop/td2 -m 2 --fields-terminated-by '\t' --columns 'id,account,income' --where 'id>2 and id<=9' 

 

显示192.168.20.60 中的数据库

[hadoop@sqoop ~]$ [hadoop@sqoop ~]$ /data/sqoop/bin/sqoop import  --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --table sys_log

 

连接mysql 需要使用mysql连接的jar包
--table trade_detail 导入到hdfs的mysql 表
--target-dir 导入到hdfs的目录名称
-m 2 使用2个map
--fields-terminated-by '\t' 设置分隔符，默认使用逗号分隔，
--columns "id,account,income" 设置导入的列
--where 'id>2 and id<=9' 加上查询条件

如果有条件 where id > 5 and $CONDITIONS'
./sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --query 'select * from trade_detail where id > 5 and $CONDITIONS' -m 1 --target-dir /sqoop/td3
如果没有条件 where $CONDITIONS'
./sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --query 'select * from trade_detail where $CONDITIONS' -m 1 --target-dir /sqoop/td3

--query 指定sql语句 后面必须加 $CONDITIONS
-m 1 必须使用1个map

如果指定map为多个，必须指定使用--split-by
./sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --query 'select * from trade_detail where id > 5 and $CONDITIONS' -m 2 --target-dir /sqoop/td4 --split-by trade_detail.id
用trade_detail.id 分隔，1个map将所有的数据都读取进来，不用切分数据。

建议使用单引号，如果使用双引号 "\$CONDITIONS"

从hdfs 导入到mysql中
在msql表中创建一个 与 hdfs 相同的表结构
create table feng like templatesql;

./sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --export-dir '/sqoop/td3/' --table feng -m 1 --fields-terminated-by '\t'