Hadoop启动脚本分析

　　　　　　　　　　　　　　　　　　　　　Hadoop启动脚本分析

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

 

 

　　　　能看到这篇博客的你估计对Hadoop已经有一个系统的了解了，最起码各种搭建方式你应该是会的，不会也没有关系，可以参考我的笔记，里面有各种搭建方式，哈哈哈~

 

[yinzhengjie@s101 ~]$ cat `which xcall.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com


#判断用户是否传参
if [ $# -lt 1 ];then
        echo "请输入参数"
        exit
fi

#获取用户输入的命令
cmd=$@

for (( i=101;i<=104;i++ ))
do
        #使终端变绿色 
        tput setaf 2
        echo ============= s$i $cmd ============
        #使终端变回原来的颜色，即白灰色
        tput setaf 7
        #远程执行命令
        ssh s$i $cmd
        #判断命令是否执行成功
        if [ $? == 0 ];then
                echo "命令执行成功"
        fi
done
[yinzhengjie@s101 ~]$ 

 

 

一.start-all.sh脚本分析

[yinzhengjie@s101 ~]$ cat `which start-all.sh`  | grep -v ^# | grep -v ^$
echo "This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh"
bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin"; pwd`
DEFAULT_LIBEXEC_DIR="$bin"/../libexec
HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}
. $HADOOP_LIBEXEC_DIR/hadoop-config.sh
if [ -f "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh ]; then
  "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh --config $HADOOP_CONF_DIR
fi
if [ -f "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh ]; then
  "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh --config $HADOOP_CONF_DIR
fi
[yinzhengjie@s101 ~]$ 

　　从这个脚本中的第一行我们可以看出来，这个脚本已经过时了，取而代之的是：“This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh”，也就是 “start-dfs.sh”和“start-yarn.sh”。

 

二.start-dfs.sh 脚本分析

[yinzhengjie@s101 ~]$ more `which start-dfs.sh` | grep -v ^# | grep -v ^$
usage="Usage: start-dfs.sh [-upgrade|-rollback] [other options such as -clusterId]"
bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin"; pwd`
DEFAULT_LIBEXEC_DIR="$bin"/../libexec
HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}
. $HADOOP_LIBEXEC_DIR/hdfs-config.sh
if [[ $# -ge 1 ]]; then
  startOpt="$1"
  shift
  case "$startOpt" in
    -upgrade)
      nameStartOpt="$startOpt"
    ;;
    -rollback)
      dataStartOpt="$startOpt"
    ;;
    *)
      echo $usage
      exit 1
    ;;
  esac
fi
nameStartOpt="$nameStartOpt $@"
NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -namenodes)
echo "Starting namenodes on [$NAMENODES]"
"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
  --config "$HADOOP_CONF_DIR" \
  --hostnames "$NAMENODES" \
  --script "$bin/hdfs" start namenode $nameStartOpt
if [ -n "$HADOOP_SECURE_DN_USER" ]; then
  echo \
    "Attempting to start secure cluster, skipping datanodes. " \
    "Run start-secure-dns.sh as root to complete startup."
else
  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
    --config "$HADOOP_CONF_DIR" \
    --script "$bin/hdfs" start datanode $dataStartOpt
fi
SECONDARY_NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -secondarynamenodes 2>/dev/null)
if [ -n "$SECONDARY_NAMENODES" ]; then
  echo "Starting secondary namenodes [$SECONDARY_NAMENODES]"
  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
      --config "$HADOOP_CONF_DIR" \
      --hostnames "$SECONDARY_NAMENODES" \
      --script "$bin/hdfs" start secondarynamenode
fi
SHARED_EDITS_DIR=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.namenode.shared.edits.dir 2>&-)
case "$SHARED_EDITS_DIR" in
qjournal://*)
  JOURNAL_NODES=$(echo "$SHARED_EDITS_DIR" | sed 's,qjournal://\([^/]*\)/.*,\1,g; s/;/ /g; s/:[0-9]*//g')
  echo "Starting journal nodes [$JOURNAL_NODES]"
  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
      --config "$HADOOP_CONF_DIR" \
      --hostnames "$JOURNAL_NODES" \
      --script "$bin/hdfs" start journalnode ;;
esac
AUTOHA_ENABLED=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.ha.automatic-failover.enabled)
if [ "$(echo "$AUTOHA_ENABLED" | tr A-Z a-z)" = "true" ]; then
  echo "Starting ZK Failover Controllers on NN hosts [$NAMENODES]"
  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
    --config "$HADOOP_CONF_DIR" \
    --hostnames "$NAMENODES" \
    --script "$bin/hdfs" start zkfc
fi
[yinzhengjie@s101 ~]$ 

 　　以上的注释已经被我过滤掉了，从这个脚本中大致可以看出这个脚本是用来启动hdfs进程的，即分别是：NameNode，DataNode以及secondaryNameNode。

1>.单独启动NameNode脚本用法如下：

[yinzhengjie@s101 ~]$ hadoop-daemon.sh --hostnames s101 start namenode
starting namenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-namenode-s101.out
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
11531 Jps
11453 NameNode
命令执行成功
============= s102 jps ============
3657 Jps
命令执行成功
============= s103 jps ============
3627 Jps
命令执行成功
============= s104 jps ============
3598 Jps
命令执行成功
[yinzhengjie@s101 ~]$ 

　　以上是单独启动NameNode节点的脚本用法，如果你想要批量启动的话可以使用hadoop-daemons.sh命令，只不过由于我部署的集群环境只有一个NameNode节点，因此看不出来有任何效果。

[yinzhengjie@s101 ~]$ hadoop-daemons.sh --hostnames  ` hdfs getconf -namenodes` start namenode
s101: starting namenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-namenode-s101.out
[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
13395 Jps
13318 NameNode
命令执行成功
============= s102 jps ============
3960 Jps
命令执行成功
============= s103 jps ============
3930 Jps
命令执行成功
============= s104 jps ============
3899 Jps
命令执行成功
[yinzhengjie@s101 ~]$ 

2>.单独启动DataNode脚本如下：

[yinzhengjie@s101 ~]$ hadoop-daemon.sh start datanode
starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s101.out
[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
12119 Jps
12045 DataNode
命令执行成功
============= s102 jps ============
3779 Jps
命令执行成功
============= s103 jps ============
3750 Jps
命令执行成功
============= s104 jps ============
3719 Jps
命令执行成功
[yinzhengjie@s101 ~]$ 

　　以上是单独启动DataNode的脚本用法，想要执行如果你想要批量启动的话可以使用hadoop-daemons.sh命令，由于我有三个节点，看起来效果就很明显了。

[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
14482 Jps
命令执行成功
============= s102 jps ============
4267 Jps
命令执行成功
============= s103 jps ============
4238 Jps
命令执行成功
============= s104 jps ============
4206 Jps
命令执行成功
[yinzhengjie@s101 ~]$ hadoop-daemons.sh start datanode
s102: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s102.out
s104: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s104.out
s103: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s103.out
[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
14552 Jps
命令执行成功
============= s102 jps ============
4386 Jps
4316 DataNode
命令执行成功
============= s103 jps ============
4357 Jps
4287 DataNode
命令执行成功
============= s104 jps ============
4325 Jps
4255 DataNode
命令执行成功
[yinzhengjie@s101 ~]$ 

3>.单独启动secondaryNameNode

[yinzhengjie@s101 ~]$ hadoop-daemon.sh --hostnames s101 start secondarynamenode
starting secondarynamenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-secondarynamenode-s101.out
[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
15127 SecondaryNameNode
15179 Jps
命令执行成功
============= s102 jps ============
4541 Jps
命令执行成功
============= s103 jps ============
4513 Jps
命令执行成功
============= s104 jps ============
4480 Jps
命令执行成功
[yinzhengjie@s101 ~]$ 

　　以上是单独启动secondaryNameNode的脚本用法，想要执行如果你想要批量启动的话可以使用hadoop-daemons.sh命令，由于我有三个节点，看起来效果就很明显了。

[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
17273 Jps
命令执行成功
============= s102 jps ============
4993 Jps
命令执行成功
============= s103 jps ============
4965 Jps
命令执行成功
============= s104 jps ============
4929 Jps
命令执行成功
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ for i in `cat /soft/hadoop/etc/hadoop/slaves | grep -v ^#` ;do  hadoop-daemons.sh --hostnames $i start secondarynamenode ;done
s102: starting secondarynamenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-secondarynamenode-s102.out
s103: starting secondarynamenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-secondarynamenode-s103.out
s104: starting secondarynamenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-secondarynamenode-s104.out
[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
17394 Jps
命令执行成功
============= s102 jps ============
5089 Jps
5042 SecondaryNameNode
命令执行成功
============= s103 jps ============
5061 Jps
5014 SecondaryNameNode
命令执行成功
============= s104 jps ============
5026 Jps
4979 SecondaryNameNode
命令执行成功
[yinzhengjie@s101 ~]$ 

 

三.start-yarn.sh 脚本分析

[yinzhengjie@s101 ~]$ cat /soft/hadoop/sbin/start-yarn.sh | grep -v ^# | grep -v ^$
echo "starting yarn daemons"
bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin"; pwd`
DEFAULT_LIBEXEC_DIR="$bin"/../libexec
HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}
. $HADOOP_LIBEXEC_DIR/yarn-config.sh
"$bin"/yarn-daemon.sh --config $YARN_CONF_DIR  start resourcemanager
"$bin"/yarn-daemons.sh --config $YARN_CONF_DIR  start nodemanager
[yinzhengjie@s101 ~]$ 

　　其实用法跟上面的类似，单独启动进程如下：

[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
18290 Jps
命令执行成功
============= s102 jps ============
5314 Jps
命令执行成功
============= s103 jps ============
5288 Jps
命令执行成功
============= s104 jps ============
5249 Jps
命令执行成功
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ yarn-daemon.sh start  nodemanager
starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-yinzhengjie-nodemanager-s101.out
[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
18344 NodeManager
18474 Jps
命令执行成功
============= s102 jps ============
5337 Jps
命令执行成功
============= s103 jps ============
5311 Jps
命令执行成功
============= s104 jps ============
5273 Jps
命令执行成功
[yinzhengjie@s101 ~]$ 

　　如果想要想要批量启动的，实操如下：

[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
18570 Jps
命令执行成功
============= s102 jps ============
5383 Jps
命令执行成功
============= s103 jps ============
5357 Jps
命令执行成功
============= s104 jps ============
5319 Jps
命令执行成功
[yinzhengjie@s101 ~]$ yarn-daemons.sh start  nodemanager
s102: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-yinzhengjie-nodemanager-s102.out
s104: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-yinzhengjie-nodemanager-s104.out
s103: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-yinzhengjie-nodemanager-s103.out
[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
18645 Jps
命令执行成功
============= s102 jps ============
5562 Jps
5436 NodeManager
命令执行成功
============= s103 jps ============
5536 Jps
5410 NodeManager
命令执行成功
============= s104 jps ============
5498 Jps
5372 NodeManager
命令执行成功
[yinzhengjie@s101 ~]$ 

 

二.stop-all.sh脚本分析

[yinzhengjie@s101 ~]$ cat `which stop-all.sh` | grep -v ^#  | grep -v ^$
echo "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"
bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin"; pwd`
DEFAULT_LIBEXEC_DIR="$bin"/../libexec
HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}
. $HADOOP_LIBEXEC_DIR/hadoop-config.sh
if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh ]; then
  "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh --config $HADOOP_CONF_DIR
fi
if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh ]; then
  "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh --config $HADOOP_CONF_DIR
fi
[yinzhengjie@s101 ~]$ 

　　看到第一行时：echo "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"，估计你已经明白是怎么回事了把，就是把上面的所有start参数换成了stop参数。从这个脚本中的第一行我们可以看出来，这个脚本已经过时了，取而代之的是：“This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh”，也就是 “stop-dfs.sh”和“stop-yarn.sh”。

 

三.小结

　　综上所述，我们可以得到以下四个等式：

1>.start-all.sh = start-dfs.sh + start-yarn.sh

2>.stop-all.sh = stop-dfs.sh + stop-yarn.sh

3>.hadoop-damons.sh = hadoop-damon.sh + slaves

4>.yarn-damons.sh = yarn-damon.sh + slaves