bigdata_hadoop集群配置_内存分配
haoop集群 做好内存管理跟重要，不然经常会给抛出个 OutMemory ,内存溢出
以horntonworks给出推荐配置为样本，给出一种常见的Hadoop集群上各组件的内存分配方案。配置时通过 ambari对应修改，或者后台同步修改 。
【样本】
he final calculation is to determine the amount of RAM per container:
RAM-per-Container = maximum of (MIN_CONTAINER_SIZE, (Total Available RAM) / Containers))
With these calculations, the YARN and MapReduce configurations can be set:
| Configuration File | Configuration Setting | Value Calculation |
| yarn-site.xml | yarn.nodemanager.resource.memory-mb | = Containers * RAM-per-Container |
| yarn-site.xml | yarn.scheduler.minimum-allocation-mb | = RAM-per-Container |
| yarn-site.xml | yarn.scheduler.maximum-allocation-mb | = containers * RAM-per-Container |
| mapred-site.xml | mapreduce.map.memory.mb | = RAM-per-Container |
| mapred-site.xml | mapreduce.reduce.memory.mb | = 2 * RAM-per-Container |
| mapred-site.xml | mapreduce.map.java.opts | = 0.8 * RAM-per-Container |
| mapred-site.xml | mapreduce.reduce.java.opts | = 0.8 * 2 * RAM-per-Container |
| yarn-site.xml (check) | yarn.app.mapreduce.am.resource.mb | = 2 * RAM-per-Container |
| yarn-site.xml (check) | yarn.app.mapreduce.am.command-opts | = 0.8 * 2 * RAM-per-Container |
Note: After installation, both yarn-site.xml and mapred-site.xml are located in the /etc/hadoop/conf folder.
| Configuration File | Configuration Setting | Value Calculation M |
| yarn-site.xml | yarn.nodemanager.resource.memory-mb | = Containers * RAM-per-Container(54G) |
| yarn-site.xml | yarn.scheduler.minimum-allocation-mb | = 2048 |
| yarn-site.xml | yarn.scheduler.maximum-allocation-mb | = containers * RAM-per-Container (54G) |
| mapred-site.xml | mapreduce.map.memory.mb | = 2048 |
| mapred-site.xml | mapreduce.reduce.memory.mb | = 4096 |
| mapred-site.xml | mapreduce.map.java.opts | = 1638 |
| mapred-site.xml | mapreduce.reduce.java.opts | = 3276 |
| yarn-site.xml (check) | yarn.app.mapreduce.am.resource.mb | = 2048 |
| yarn-site.xml (check) | yarn.app.mapreduce.am.command-opts | = 3276 |
【样例1】
| Configuration File | Configuration Setting | Value Calculation |
| yarn-site.xml | yarn.nodemanager.resource.memory-mb | = Containers * RAM-per-Container |
| yarn-site.xml | yarn.scheduler.minimum-allocation-mb | = RAM-per-Container |
| yarn-site.xml | yarn.scheduler.maximum-allocation-mb | = containers * RAM-per-Container |
| mapred-site.xml | mapreduce.map.memory.mb | = RAM-per-Container |
| mapred-site.xml | mapreduce.reduce.memory.mb | = 2 * RAM-per-Container |
| mapred-site.xml | mapreduce.map.java.opts | = 0.8 * RAM-per-Container |
| mapred-site.xml | mapreduce.reduce.java.opts | = 0.8 * 2 * RAM-per-Container |
| yarn-site.xml (check) | yarn.app.mapreduce.am.resource.mb | = 2 * RAM-per-Container |
| yarn-site.xml (check) | yarn.app.mapreduce.am.command-opts | = 0.8 * 2 * RAM-per-Container |
【样例2】
方案最右侧一栏是一个8G VM的分配方案，方案预留1-2G的内存给操作系统，分配4G给Yarn/MapReduce，当然也包括了HIVE，剩余的2-3G是在需要使用HBase时预留给HBase的。 参考：http://blog.csdn.net/bluishglc/article/details/42436321
【备注】
另外自己通过thrift链接时 ，留意自己加载时 重设配置,造成任务类似失败 <set mapreduce.map.java.opts=-Xmx1024m;> 当hive提交任务
eg:select count(*) from test; 没问题，但是 自己平台提交有问题 。仔细比对配置 例如搜索关键词 ：memory ，opts
参考：http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html
