[hadoop] 集群启动和内存调优
1、启动Hadoop集群
#首先查看下zoo.cfg里面配置的server.id和集群主机的id是否一致 #如果不一致会造成yarn控制不了从节点的启动 cat /home/hadoop/zookeeper/conf/zoo.cfg cat /home/hadoop/zookeeper/data/myid #分别停止和启动各节点的zookeeper #保证每台节点的状态为follow或者leader /home/hadoop/zookeeper/bin/zkServer.sh stop /home/hadoop/zookeeper/bin/zkServer.sh start /home/hadoop/zookeeper/bin/zkServer.sh status #启动hadoop #成功启动后可访问 #http://master:50070/dfshealth.html#tab-overview #http://master:8088/cluster/nodes /home/hadoop/hadoop/sbin/start-all.sh /home/hadoop/hadoop/sbin/stop-all.sh #启动hbase #成功启动后可访问 #http://master:60010/master-status?filter=all#baseStats #Region Servers 会列出hbase的节点列表 /home/hadoop/hbase/bin/start-hbase.sh /home/hadoop/hbase/bin/stop-hbase.sh
2、提示信息
#hadoop [hadoop@master sbin]$ ./start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [master] master: starting namenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-namenode-master.out localhost: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-master.out node1: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-node1.out node2: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-node2.out Starting secondary namenodes [master] master: starting secondarynamenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-master.out starting yarn daemons starting resourcemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-master.out localhost: starting nodemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-master.out node1: starting nodemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-node1.out node2: starting nodemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-node2.out #hbase [hadoop@master sbin]$ start-hbase.sh master: starting zookeeper, logging to /home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-master.out node1: starting zookeeper, logging to /home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-node1.out node2: starting zookeeper, logging to /home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-node2.out starting master, logging to /home/hadoop/hbase/logs/hbase-hadoop-master-master.out node2: starting regionserver, logging to /home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-node2.out node1: starting regionserver, logging to /home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-node1.out
3、守护进程
#主节点 [hadoop@master sbin]$ jps 11178 Jps 10212 ResourceManager（资源管理者） 9691 NameNode（主节点） 10028 SecondaryNameNode（备用主节点） 11032 HMaster（Hbase管理者） 10332 NodeManager（节点管理者） 6923 QuorumPeerMain（zookeeper守护进程） 9809 DataNode（主节点也作为数据节点） #从节点 [hadoop@node2 conf]$ jps 6782 Jps 6640 HRegionServer（Hbase从节点） 6225 NodeManager（节点管理） 6532 HQuorumPeer（zookeeper守护进程） 6103 DataNode（数据节点）
4、心得总结
#启动顺序 (1)先用jps查看是否还有没有停止的hadoop服务，有就先全部停止 (2)先启动 zookeeper：~/zookeeper/bin/zkServer.sh start 不管主从每一个节点的zk都要单独启动 启动后 用jps查看 有没有 QuorumPeerMain 进程 或者用 ~/zookeeper/bin/zkServer.sh status 查看角色 从属：follower 主：leader (3)再启动hadoop：~/hadoop/sbin/start-all.sh 主节点 jps查看是否有Namenode等进程 从节点 jps查看是否有Datanode等进程 (4)最后启动hbase：~/hbase/bin/start-hbase.sh 主节点 jps查看是否有HMaster等进程 从节点 jps查看是否有HRegionserver等进程
#其他
(1)如果从节点HBASE启动失败
1、检查zookeeper/data下myid是否正确
2、实在不行单独启动从节点
hbase-daemon.sh start regionserver
(2)namenode没启动成功先检查是不是端口被占用
netstat -atunp | grep 9000
ps -aux | grep hadoop
(3)如果zk的端口被绑定占用
ps -aux | grep zookeeper
看看是否有其他的用户比如【root】启动过zk
5、内存调优
【hadoop】512 - 4096 /home/hadoop/hadoop/etc/hadoop/hadoop-env.sh export HADOOP_PORTMAP_OPTS="-Xmx4096m $HADOOP_PORTMAP_OPTS" export HADOOP_CLIENT_OPTS="-Xmx4096m $HADOOP_CLIENT_OPTS" 【yarn】2048 - 4096 /home/hadoop/hadoop/etc/hadoop/yarn-env.sh JAVA_HEAP_MAX=-Xmx4096m 【hbase】 #以前的 export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Xmx1000m -Xms1000m -Xmn750m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70" export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Xmx3096m -Xms3096m -Xmn1220m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70" export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Xms100m -Xmx1024m" export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Xms100m -Xmx1024m" #现在修改为 export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Xmx8192m -Xms8192m -Xmn6144m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70" export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Xmx10240m -Xms10240m -Xmn5120m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70" export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Xms1024m -Xmx4096m" export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Xms2048m -Xmx4096m"
