Sqoop- sqoop将mysql数据表导入到hive报错
sqoop将mysql数据表导入到hive报错
[root@ip-172-32-1-221 lib]# sqoop import --connect jdbc:mysql://54.223.175.12:3308/gxt3 --username guesttest --password guesttest --table ecomaccessv3 -m 1 --hive-import Warning: /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 18/06/20 16:34:32 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.10.0 18/06/20 16:34:32 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 18/06/20 16:34:32 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 18/06/20 16:34:32 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 18/06/20 16:34:32 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 18/06/20 16:34:32 INFO tool.CodeGenTool: Beginning code generation 18/06/20 16:34:32 ERROR manager.SqlManager: Error executing statement: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:408) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:355) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2490) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:408) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:419) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:904) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786) at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260) at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1858) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1658) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:241) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:258) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:305) ... 33 more 18/06/20 16:34:32 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1664) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
网上说是MySQL驱动包的问题下，需要更换驱动包：“原有的jar包是mysql-connector-java-5.1.17.jar，替换成mysql-connector-java-5.1.32-bin.jar，可以使用了” ------亲测不行
cd /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/sqoop/lib/ ll cp /home/zc.lee/mysql-connector-java-5.1.32-bin.jar ./mysql-connector-java.jar
运行起来还是没有能把mr task跑起来，报错和上次一致。
还有是将localhost换成IP地址，我的机器是通过ssh -i 公钥连接到远端的服务器，通过将运行命令的连接换成IP地址后测试查询库出现报错，换回loucalhost能正常list databases
分析原因：
因为我的jdbc上面用的是localhost+端口/127.0.0.1+端口，然而只有当前的机器做了端口映射，然而sqoop抽数底层的原理是去跑mapreduce，当把任务分配hadoop集群的其他node，在这些worker上面也是调用localhost，但是这些worker上面没有做端口映射所以他们没有能够连接到数据库，导致mapreduce失败，没有办法抽数。
解决方式：
如果在没有复杂的环境下，一般这个命令是完全能执行的，但是环境复杂的情况下需要集群中每个Node（节点）都能于数据库通信才能保证sqoop运行的前提准备做好。
1.所有的worker都做端口映射。
2.如果环境不复杂的话，请用ip的方式去访问数据库，不要用localhost，不要用127.0.01.
当以上的方式都不能解决的情况下只能通过查询结果重定向的方式去完成需求。
mysql -h127.0.0.1 -P3309 -uroot -proot -e "select * from defalut.test where date>='2017-07-01' and date<'2017-08-01' " --skip-column-names |sed -e "s/[\t]/\t/" -e "s/$/\r/">/tmp/test.txt
