hiveQL 本地mapreduce
2013-04-26 15:42 java20130722 阅读(280) 评论(0) 编辑 收藏 举报如果在hive中运行的sql本身数据量很小,那么使用本地mr的效率要比分布式的快很多。。
比如:
- hive> select 1 from dual;
- Total MapReduce jobs = 1
- Launching Job 1 out of 1
- Number of reduce tasks is set to 0 since there's no reduce operator
- Starting Job = job_201208151631_2040444, Tracking URL = http://jt.dc.sh-wgq.sdo.com:50030/jobdetails.jsp?jobid=job_201208151631_2040444
- Kill Command = /home/hdfs/hadoop-current/bin/hadoop job -Dmapred.job.tracker=10.133.10.103:50020 -kill job_201208151631_2040444
- 2012-10-23 10:55:17,646 Stage-1 map = 0%, reduce = 0%
- 2012-10-23 10:55:27,807 Stage-1 map = 100%, reduce = 0%
- Ended Job = job_201208151631_2040444
- OK
- 1
- Time taken: 17.853 seconds
set hive.exec.mode.local.auto=true; //开启本地mr
//设置local mr的最大输入数据量,当输入数据量小于这个值的时候会采用local mr的方式
set hive.exec.mode.local.auto.inputbytes.max=50000000;
//设置local mr的最大输入文件个数,当输入文件个数小于这个值的时候会采用local mr的方式
set hive.exec.mode.local.auto.tasks.max=10;
当这三个参数同时成立时候,才会采用本地mr
- hive> select 1 from dual;
- Total MapReduce jobs = 1
- Launching Job 1 out of 1
- Number of reduce tasks is set to 0 since there's no reduce operator
- Execution log at: /tmp/liuxiaowen/liuxiaowen_20121023105757_31c966be-ee79-4c23-a467-648290b338ac.log
- Job running in-process (local Hadoop)
- 2012-10-23 10:58:03,728 null map = 100%, reduce = 0%
- Ended Job = job_local_0001
- OK
- 1
- Time taken: 4.842 seconds