用Sqoop实现数据HDFS到mysql到Hive

【Hive】用Sqoop实现数据HDFS到mysql到Hive
大数据协作框架
“大数据协作框架”其实是一个统称，主要是以下四个框架
  • 数据转换工具    Sqoop
  • 文件收集库框架    Flume
  • 任务调度框架    Oozie
  • 大数据WEB工具Hue
 
Sqoop作用
将关系数据库中的某张表数据抽取到Hadoop的HDFS文件系统中，底层运行的还是MapReduce。
利用MapReduce加快数据传输速度。
批处理方式进行数据传输。
也可以将HDFS上的文件数据或者Hive表中的数据导出到关系型数据库当中的某张表中。
 
HDFS →RDBMS
  1. sqoop export \
  2. --connect jdbc:mysql://xxx:3306/xxx \
  3. --username xxx \
  4. --password xxx \
  5. --table xxx \
  6. --export-dir xxx
RDBMS→Hive
  1. sqoop import \
  2. --connect jdbc:mysql://xxx:3306/xxx \
  3. --username xxx \
  4. --password xxx \
  5. --fields-terminated-by "\t" \
  6. --table xxx \
  7. --hive-import \
  8. --hive-table xxx
Hive→RDBMS
  1. sqoop export \
  2. --connect jdbc:mysql://xxx:3306/xxx \
  3. --username xxx \
  4. --password xxx \
  5. --table xxx \
  6. --export-dir xxx \
  7. --input-fields-terminated-by '\t'
RDBMS→HDFS
  1. sqoop import \
  2. --connect jdbc:mysql://xxx:3306/xxx \
  3. --username xxx \
  4. --password xxx \
  5. --table xxx \
  6. --target-dir xxx
规律：
  1. 从RDBMS导入到HDFS或者Hive中的都使用import；从HDFS或者Hive导出到RDBMS中的都使用export；以HDFS和Hive为参考，根据数据流向选择关键字。
  2. connect、username、password、table四个参数为每一种传输都必须的；其中connect参数格式均为--connect jdbc:mysql://主机名:3306/数据库名(使用mysql数据库)；table是指明mysql中的表名。
  3. export-dir参数只有在导出数据到RDBMS中时才会用到，含义为表在hdfs中存放的路径。
区别：
  • HDFS →RDBMS：指明table：mysql中的表，需要自行先创建；指明export-dir：HDFS中数据的存储路径
  • RDBMS→Hive：指明fields-terminated-by：指定分隔符，分隔符指的是存放在Hive中的数据的分隔符，如果目标将存储在Hive中可理解为编码格式，若目标将存储在RDBMS上，则可理解为解码格式；指明table:mysql中的表名；指明hive-import：导入到hive操作;指明hive-table:hive中的表名。注意：table参数不可以与用户家目录下已存在的目录重名，因为sqoop导数据到hive会先将数据导入到HDFS上，然后再将数据load到hive中，最后把这个目录再删除掉。
  • Hive→RDBMS：指明table:mysql中的表名；指明export-dir:hive在hdfs中存储的路径;指明hive-table:hive中的表名。
  • RDBMS→HDFS：指明table:mysql里的表名;指明target-dir:hdfs存储数据的目录;
 
Sqoop安装
配置Sqoop1.x
conf目录【sqoop-env-template.sh】
  • export HADOOP_COMMON_HOME=Hadoop目录
  • export HADOOP_MAPRED_HOME=Hadoop目录
  • export HIVE_HOME=Hive目录
  • export ZOOCFGDIR=Zookeeper目录
将mysqlJDBC驱动包拷到sqoop的lib目录下
测试sqoop
  1. bin/sqoop list-databases \
  2. --connect jdbc:mysql://主机名：3306 \
  3. --username root \
  4. --password 123456 \
查看本地mysql
  2. mysql> show databases;
  3. +--------------------+
  4. | Database |
  5. +--------------------+
  6. | information_schema |
  7. | metastore |
  8. | mysql |
  9. | test |
  10. +--------------------+
  11. 4 rows in set (0.00 sec)
  13. mysql> use test;
  14. Reading table information for completion of table and column names
  15. You can turn off this feature to get a quicker startup with -A
  17. Database changed
  18. mysql> show tables;
  19. +----------------+
  20. | Tables_in_test |
  21. +----------------+
  22. | my_user |
  23. +----------------+
  24. 1 row in set (0.00 sec)
  26. mysql> desc my_user;
  27. +---------+--------------+------+-----+---------+----------------+
  28. | Field | Type | Null | Key | Default | Extra |
  29. +---------+--------------+------+-----+---------+----------------+
  30. | id | tinyint(4) | NO | PRI | NULL | auto_increment |
  31. | account | varchar(255) | YES | | NULL | |
  32. | passwd | varchar(255) | YES | | NULL | |
  33. +---------+--------------+------+-----+---------+----------------+
  34. 3 rows in set (0.00 sec)
  37. mysql> select * from my_user;
  38. +----+----------+----------+
  39. | id | account | passwd |
  40. +----+----------+----------+
  41. | 1 | admin | admin |
  42. | 2 | johnny | 123456 |
  43. | 3 | zhangsan | zhangsan |
  44. | 4 | lisi | lisi |
  45. | 5 | test | test |
  46. | 6 | qiqi | qiqi |
  47. | 7 | hangzhou | hangzhou |
  48. +----+----------+----------+
  49. 7 rows in set (0.00 sec)
hive创建相同结构的空表
  1. hive (test)> create table h_user(
  2. > id int,
  3. > account string,
  4. > passwd string
  5. > )row format delimited fields terminated by '\t';
  6. OK
  7. Time taken: 0.113 seconds
  9. hive (test)> desc h_user;
  10. OK
  11. col_name data_type comment
  12. id int
  13. account string
  14. passwd string
  15. Time taken: 0.228 seconds, Fetched: 3 row(s)
从本地mysql导出数据到Hive里
  1. bin/sqoop import \
  2. --connect jdbc:mysql://cdaisuke:3306/test \
  3. --username root \
  4. --password 123456 \
  5. --table my_user \
  6. --num-mappers 1 \
  7. --delete-target-dir \
  8. --fields-terminated-by "\t" \
  9. --hive-database test \
  10. --hive-import \
  11. --hive-table h_user
  13. hive (test)> select * from h_user;
  14. OK
  15. h_user.id h_user.account h_user.passwd
  16. 1 admin admin
  17. 2 johnny 123456
  18. 3 zhangsan zhangsan
  19. 4 lisi lisi
  20. 5 test test
  21. 6 qiqi qiqi
  22. 7 hangzhou hangzhou
  23. Time taken: 0.061 seconds, Fetched: 7 row(s)
从mysql导入到HDFS里
  1. bin/sqoop import \
  2. --connect jdbc:mysql://cdaisuke:3306/test \
  3. --username root \
  4. --password 123456 \
  5. --table my_user \
  6. --num-mappers 3 \
  7. --target-dir /user/hadoop/ \
  8. --delete-target-dir \
  9. --fields-terminated-by "\t"
  11. ------------------------------------------------------------
  13. [hadoop@cdaisuke sqoop-1.4.5-cdh5.3.6]$ bin/sqoop import \
  14. > --connect jdbc:mysql://cdaisuke:3306/test \
  15. > --username root \
  16. > --password 123456 \
  17. > --table my_user \
  18. > --num-mappers 3 \
  19. > --target-dir /user/hadoop/ \
  20. > --delete-target-dir \
  21. > --fields-terminated-by "\t"
  22. 18/08/14 00:02:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.6
  23. 18/08/14 00:02:11 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
  24. 18/08/14 00:02:12 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
  25. 18/08/14 00:02:12 INFO tool.CodeGenTool: Beginning code generation
  26. 18/08/14 00:02:13 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `my_user` AS t LIMIT 1
  27. 18/08/14 00:02:13 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `my_user` AS t LIMIT 1
  28. 18/08/14 00:02:13 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/modules/hadoop-2.5.0-cdh5.3.6_Hive
  29. Note: /tmp/sqoop-hadoop/compile/7c8bdb7cd3df7b2f4b48700704f46f65/my_user.java uses or overrides a deprecated API.
  30. Note: Recompile with -Xlint:deprecation for details.
  31. 18/08/14 00:02:18 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/7c8bdb7cd3df7b2f4b48700704f46f65/my_user.jar
  32. 18/08/14 00:02:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  33. 18/08/14 00:02:22 INFO tool.ImportTool: Destination directory /user/hadoop is not present, hence not deleting.
  34. 18/08/14 00:02:22 WARN manager.MySQLManager: It looks like you are importing from mysql.
  35. 18/08/14 00:02:22 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
  36. 18/08/14 00:02:22 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
  37. 18/08/14 00:02:22 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
  38. 18/08/14 00:02:22 INFO mapreduce.ImportJobBase: Beginning import of my_user
  39. 18/08/14 00:02:22 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
  40. 18/08/14 00:02:22 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  41. 18/08/14 00:02:23 INFO client.RMProxy: Connecting to ResourceManager at slave01/192.168.79.140:8032
  42. 18/08/14 00:02:28 INFO db.DBInputFormat: Using read commited transaction isolation
  43. 18/08/14 00:02:28 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `my_user`
  44. 18/08/14 00:02:28 INFO mapreduce.JobSubmitter: number of splits:3
  45. 18/08/14 00:02:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533652222364_0078
  46. 18/08/14 00:02:29 INFO impl.YarnClientImpl: Submitted application application_1533652222364_0078
  47. 18/08/14 00:02:29 INFO mapreduce.Job: The url to track the job: http://slave01:8088/proxy/application_1533652222364_0078/
  48. 18/08/14 00:02:29 INFO mapreduce.Job: Running job: job_1533652222364_0078
  49. 18/08/14 00:02:50 INFO mapreduce.Job: Job job_1533652222364_0078 running in uber mode : false
  50. 18/08/14 00:02:50 INFO mapreduce.Job: map 0% reduce 0%
  51. 18/08/14 00:03:00 INFO mapreduce.Job: map 33% reduce 0%
  52. 18/08/14 00:03:01 INFO mapreduce.Job: map 67% reduce 0%
  53. 18/08/14 00:03:02 INFO mapreduce.Job: map 100% reduce 0%
  54. 18/08/14 00:03:02 INFO mapreduce.Job: Job job_1533652222364_0078 completed successfully
  55. 18/08/14 00:03:02 INFO mapreduce.Job: Counters: 30
  56. File System Counters
  57. FILE: Number of bytes read=0
  58. FILE: Number of bytes written=394707
  59. FILE: Number of read operations=0
  60. FILE: Number of large read operations=0
  61. FILE: Number of write operations=0
  62. HDFS: Number of bytes read=295
  63. HDFS: Number of bytes written=106
  64. HDFS: Number of read operations=12
  65. HDFS: Number of large read operations=0
  66. HDFS: Number of write operations=6
  67. Job Counters
  68. Launched map tasks=3
  69. Other local map tasks=3
  70. Total time spent by all maps in occupied slots (ms)=25213
  71. Total time spent by all reduces in occupied slots (ms)=0
  72. Total time spent by all map tasks (ms)=25213
  73. Total vcore-seconds taken by all map tasks=25213
  74. Total megabyte-seconds taken by all map tasks=25818112
  75. Map-Reduce Framework
  76. Map input records=7
  77. Map output records=7
  78. Input split bytes=295
  79. Spilled Records=0
  80. Failed Shuffles=0
  81. Merged Map outputs=0
  82. GC time elapsed (ms)=352
  83. CPU time spent (ms)=3600
  84. Physical memory (bytes) snapshot=316162048
  85. Virtual memory (bytes) snapshot=2523156480
  86. Total committed heap usage (bytes)=77766656
  87. File Input Format Counters
  88. Bytes Read=0
  89. File Output Format Counters
  90. Bytes Written=106
  91. 18/08/14 00:03:02 INFO mapreduce.ImportJobBase: Transferred 106 bytes in 40.004 seconds (2.6497 bytes/sec)
  92. 18/08/14 00:03:02 INFO mapreduce.ImportJobBase: Retrieved 7 records.
设置3个map任务
--num-mappers 3 \
设置HDFS目标存储目录
--target-dir /user/hadoop/ \
如果设置目录存在则删除此目录
--delete-target-dir \
从Hive导出到mysql
在mysql创建新表
  1. create table user_export(
  2. id tinyint(4) not null auto_increment,
  3. account varchar(255) default null,
  4. passwd varchar(255) default null,
  5. primary key(id)
  6. );
用sqoop导出数据
  1. bin/sqoop export \
  2. --connect jdbc:mysql://cdaisuke:3306/test \
  3. --username root \
  4. --password 123456 \
  5. --table user_export \
  6. --num-mappers 1 \
  7. --fields-terminated-by "\t" \
  8. --export-dir /user/hive/warehouse/test.db/h_user
  10. ----------------------------------------------------
  12. [hadoop@cdaisuke sqoop-1.4.5-cdh5.3.6]$ bin/sqoop export \
  13. > --connect jdbc:mysql://cdaisuke:3306/test \
  14. > --username root \
  15. > --password 123456 \
  16. > --table user_export \
  17. > --num-mappers 1 \
  18. > --fields-terminated-by "\t" \
  19. > --export-dir /user/hive/warehouse/test.db/h_user
  20. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
  21. 18/08/14 00:16:32 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.6
  22. 18/08/14 00:16:32 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
  23. 18/08/14 00:16:33 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
  24. 18/08/14 00:16:33 INFO tool.CodeGenTool: Beginning code generation
  25. 18/08/14 00:16:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_export` AS t LIMIT 1
  26. 18/08/14 00:16:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_export` AS t LIMIT 1
  27. 18/08/14 00:16:34 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/modules/hadoop-2.5.0-cdh5.3.6_Hive
  28. Note: /tmp/sqoop-hadoop/compile/6823ffae505b34f7ae8b9881bae4b898/user_export.java uses or overrides a deprecated API.
  29. Note: Recompile with -Xlint:deprecation for details.
  30. 18/08/14 00:16:39 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/6823ffae505b34f7ae8b9881bae4b898/user_export.jar
  31. 18/08/14 00:16:39 INFO mapreduce.ExportJobBase: Beginning export of user_export
  32. 18/08/14 00:16:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  33. 18/08/14 00:16:40 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
  34. 18/08/14 00:16:43 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
  35. 18/08/14 00:16:43 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
  36. 18/08/14 00:16:43 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  37. 18/08/14 00:16:43 INFO client.RMProxy: Connecting to ResourceManager at slave01/192.168.79.140:8032
  38. 18/08/14 00:16:48 INFO input.FileInputFormat: Total input paths to process : 1
  39. 18/08/14 00:16:48 INFO input.FileInputFormat: Total input paths to process : 1
  40. 18/08/14 00:16:48 INFO mapreduce.JobSubmitter: number of splits:1
  41. 18/08/14 00:16:48 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
  42. 18/08/14 00:16:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533652222364_0079
  43. 18/08/14 00:16:50 INFO impl.YarnClientImpl: Submitted application application_1533652222364_0079
  44. 18/08/14 00:16:50 INFO mapreduce.Job: The url to track the job: http://slave01:8088/proxy/application_1533652222364_0079/
  45. 18/08/14 00:16:50 INFO mapreduce.Job: Running job: job_1533652222364_0079
  46. 18/08/14 00:17:11 INFO mapreduce.Job: Job job_1533652222364_0079 running in uber mode : false
  47. 18/08/14 00:17:11 INFO mapreduce.Job: map 0% reduce 0%
  48. 18/08/14 00:17:27 INFO mapreduce.Job: map 100% reduce 0%
  49. 18/08/14 00:17:27 INFO mapreduce.Job: Job job_1533652222364_0079 completed successfully
  50. 18/08/14 00:17:27 INFO mapreduce.Job: Counters: 30
  51. File System Counters
  52. FILE: Number of bytes read=0
  53. FILE: Number of bytes written=131287
  54. FILE: Number of read operations=0
  55. FILE: Number of large read operations=0
  56. FILE: Number of write operations=0
  57. HDFS: Number of bytes read=258
  58. HDFS: Number of bytes written=0
  59. HDFS: Number of read operations=4
  60. HDFS: Number of large read operations=0
  61. HDFS: Number of write operations=0
  62. Job Counters
  63. Launched map tasks=1
  64. Data-local map tasks=1
  65. Total time spent by all maps in occupied slots (ms)=13426
  66. Total time spent by all reduces in occupied slots (ms)=0
  67. Total time spent by all map tasks (ms)=13426
  68. Total vcore-seconds taken by all map tasks=13426
  69. Total megabyte-seconds taken by all map tasks=13748224
  70. Map-Reduce Framework
  71. Map input records=7
  72. Map output records=7
  73. Input split bytes=149
  74. Spilled Records=0
  75. Failed Shuffles=0
  76. Merged Map outputs=0
  77. GC time elapsed (ms)=73
  78. CPU time spent (ms)=1230
  79. Physical memory (bytes) snapshot=113061888
  80. Virtual memory (bytes) snapshot=838946816
  81. Total committed heap usage (bytes)=45613056
  82. File Input Format Counters
  83. Bytes Read=0
  84. File Output Format Counters
  85. Bytes Written=0
  86. 18/08/14 00:17:27 INFO mapreduce.ExportJobBase: Transferred 258 bytes in 44.2695 seconds (5.8279 bytes/sec)
  87. 18/08/14 00:17:27 INFO mapreduce.ExportJobBase: Exported 7 records.
  89. -----------------------------------------------------------------
  91. mysql> select * from user_export;
  92. +----+----------+----------+
  93. | id | account | passwd |
  94. +----+----------+----------+
  95. | 1 | admin | admin |
  96. | 2 | johnny | 123456 |
  97. | 3 | zhangsan | zhangsan |
  98. | 4 | lisi | lisi |
  99. | 5 | test | test |
  100. | 6 | qiqi | qiqi |
  101. | 7 | hangzhou | hangzhou |
  102. +----+----------+----------+
  103. 7 rows in set (0.00 sec)
从HDFS导出到mysql
在mysql创建新表
 
  1. create table my_user2(
  2. id tinyint(4) not null auto_increment,
  3. account varchar(255) default null,
  4. passwd varchar(255) default null,
  5. primary key (id)
  6. );
  8. ---------------------------------------------------------
  10. [hadoop@cdaisuke sqoop-1.4.5-cdh5.3.6]$ bin/sqoop export \
  11. > --connect jdbc:mysql://cdaisuke:3306/test \
  12. > --username root \
  13. > --password 123456 \
  14. > --table my_user2 \
  15. > --num-mappers 1 \
  16. > --fields-terminated-by "\t" \
  17. > --export-dir /user/hadoop
  18. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
  19. 18/08/14 00:39:51 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.6
  20. 18/08/14 00:39:51 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
  21. 18/08/14 00:39:52 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
  22. 18/08/14 00:39:52 INFO tool.CodeGenTool: Beginning code generation
  23. 18/08/14 00:39:53 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `my_user2` AS t LIMIT 1
  24. 18/08/14 00:39:53 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `my_user2` AS t LIMIT 1
  25. 18/08/14 00:39:53 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/modules/hadoop-2.5.0-cdh5.3.6_Hive
  26. Note: /tmp/sqoop-hadoop/compile/7222f42cd6507a21fdcef7600bd14a20/my_user2.java uses or overrides a deprecated API.
  27. Note: Recompile with -Xlint:deprecation for details.
  28. 18/08/14 00:39:59 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/7222f42cd6507a21fdcef7600bd14a20/my_user2.jar
  29. 18/08/14 00:39:59 INFO mapreduce.ExportJobBase: Beginning export of my_user2
  30. 18/08/14 00:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  31. 18/08/14 00:40:00 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
  32. 18/08/14 00:40:04 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
  33. 18/08/14 00:40:04 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
  34. 18/08/14 00:40:04 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  35. 18/08/14 00:40:04 INFO client.RMProxy: Connecting to ResourceManager at slave01/192.168.79.140:8032
  36. 18/08/14 00:40:09 INFO input.FileInputFormat: Total input paths to process : 3
  37. 18/08/14 00:40:09 INFO input.FileInputFormat: Total input paths to process : 3
  38. 18/08/14 00:40:09 INFO mapreduce.JobSubmitter: number of splits:1
  39. 18/08/14 00:40:09 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
  40. 18/08/14 00:40:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533652222364_0084
  41. 18/08/14 00:40:11 INFO impl.YarnClientImpl: Submitted application application_1533652222364_0084
  42. 18/08/14 00:40:11 INFO mapreduce.Job: The url to track the job: http://slave01:8088/proxy/application_1533652222364_0084/
  43. 18/08/14 00:40:11 INFO mapreduce.Job: Running job: job_1533652222364_0084
  44. 18/08/14 00:40:30 INFO mapreduce.Job: Job job_1533652222364_0084 running in uber mode : false
  45. 18/08/14 00:40:30 INFO mapreduce.Job: map 0% reduce 0%
  46. 18/08/14 00:40:46 INFO mapreduce.Job: map 100% reduce 0%
  47. 18/08/14 00:40:46 INFO mapreduce.Job: Job job_1533652222364_0084 completed successfully
  48. 18/08/14 00:40:46 INFO mapreduce.Job: Counters: 30
  49. File System Counters
  50. FILE: Number of bytes read=0
  51. FILE: Number of bytes written=131229
  52. FILE: Number of read operations=0
  53. FILE: Number of large read operations=0
  54. FILE: Number of write operations=0
  55. HDFS: Number of bytes read=365
  56. HDFS: Number of bytes written=0
  57. HDFS: Number of read operations=10
  58. HDFS: Number of large read operations=0
  59. HDFS: Number of write operations=0
  60. Job Counters
  61. Launched map tasks=1
  62. Data-local map tasks=1
  63. Total time spent by all maps in occupied slots (ms)=13670
  64. Total time spent by all reduces in occupied slots (ms)=0
  65. Total time spent by all map tasks (ms)=13670
  66. Total vcore-seconds taken by all map tasks=13670
  67. Total megabyte-seconds taken by all map tasks=13998080
  68. Map-Reduce Framework
  69. Map input records=7
  70. Map output records=7
  71. Input split bytes=250
  72. Spilled Records=0
  73. Failed Shuffles=0
  74. Merged Map outputs=0
  75. GC time elapsed (ms)=89
  76. CPU time spent (ms)=1670
  77. Physical memory (bytes) snapshot=115961856
  78. Virtual memory (bytes) snapshot=838946816
  79. Total committed heap usage (bytes)=45613056
  80. File Input Format Counters
  81. Bytes Read=0
  82. File Output Format Counters
  83. Bytes Written=0
  84. 18/08/14 00:40:46 INFO mapreduce.ExportJobBase: Transferred 365 bytes in 42.3534 seconds (8.618 bytes/sec)
  85. 18/08/14 00:40:46 INFO mapreduce.ExportJobBase: Exported 7 records.
  87. ------------------------------------------------------------------------
  89. mysql> select * from my_user2;
  90. +----+----------+----------+
  91. | id | account | passwd |
  92. +----+----------+----------+
  93. | 1 | admin | admin |
  94. | 2 | johnny | 123456 |
  95. | 3 | zhangsan | zhangsan |
  96. | 4 | lisi | lisi |
  97. | 5 | test | test |
  98. | 6 | qiqi | qiqi |
  99. | 7 | hangzhou | hangzhou |
  100. +----+----------+----------+
  101. 7 rows in set (0.00 sec)
 
posted @ 2021-01-06 00:51  virtual_daemon  阅读(227)  评论(0)  编辑  收藏  举报