sqoop 创建跟mysql相同表结构的hive表报错
执行
sqoop create-hive-table --connect jdbc:mysql://localhost:3306/test --username root --password 123456 --table A --hive-table mysqlToA
错误1
报错如下：
20/07/15 14:02:34 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
20/07/15 14:02:34 ERROR tool.CreateHiveTableTool: Encountered IOException running create table job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)
at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:57)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)
... 11 more
解决办法：
在 ~/.bashrc文件里面添加如下的信息
export HIVE_CONF_DIR=/data1/hadoop/hive/conf
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*
重新source一下
# source ~/.bashrc
接着执行，报如下错误
错误2
20/07/15 14:24:28 INFO conf.HiveConf: Found configuration file file:/data1/hadoop/sqoop/conf/hive-site.xml
20/07/15 14:24:30 ERROR tool.CreateHiveTableTool: Encountered IOException running create table job: java.io.IOException: Exception thrown in Hive
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:358)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:57)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:331)
... 8 more
Caused by: java.lang.NoSuchMethodError: com.lmax.disruptor.dsl.Disruptor.<init>(Lcom/lmax/disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
at org.apache.logging.log4j.core.async.AsyncLoggerDisruptor.start(AsyncLoggerDisruptor.java:97)
at org.apache.logging.log4j.core.async.AsyncLoggerContext.maybeStartHelper(AsyncLoggerContext.java:97)
at org.apache.logging.log4j.core.async.AsyncLoggerContext.start(AsyncLoggerContext.java:86)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:240)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:158)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:131)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:101)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:188)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:173)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:106)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:98)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
... 13 more
这个是由于hive3.x里面使用的disruptor版本是3.3.6，可能是版本过低导致没有对应的方法
解决：
下载高版本的disruptor包，下载地址：
https://www.mvnjar.com/com.lmax/disruptor/3.3.11/detail.html
当前下载的版本是3.3.11
把disruptor对应的jar包放到sqoop的lib目录下
重新执行创建表就可以了
hive查看创建的表
默认创建在default数据库
0: jdbc:hive2://localhost:10000> show create table mysqltoa;
+----------------------------------------------------+
| createtab_stmt |
+----------------------------------------------------+
| CREATE TABLE `mysqltoa`( |
| `id` int, |
| `name` string) |
| COMMENT 'Imported by sqoop on 2020/07/15 14:33:27' |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
| WITH SERDEPROPERTIES ( |
| 'field.delim'='', |
| 'line.delim'='\n', |
| 'serialization.format'='') |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.mapred.TextInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
| LOCATION |
| 'hdfs://yjt:9000/hive/warehouse/mysqltoa' |
| TBLPROPERTIES ( |
| 'bucketing_version'='2', |
| 'transient_lastDdlTime'='1594794817') |
+----------------------------------------------------+
对应的mysql数据表的创建语句
MySQL [test]> show create table A;
+-------+---------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+---------------------------------------------------------------------------------------------------------------------+
| A | CREATE TABLE `A` (
`id` int(11) DEFAULT NULL,
`name` char(25) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
记录学习和生活的酸甜苦辣.....哈哈哈
