（三）Spark集群搭建-Java&Python版Spark
Spark-Hadoop集群搭建
视频教程：
1、优酷
2、YouTube
配置java
启动ftp
[root@master ~]# /etc/init.d/vsftpd restart
关闭 vsftpd： [失败]
为 vsftpd 启动 vsftpd： [确定]
默认情况下 root不允许使用ftp
[root@master vsftpd]# pwd
/etc/vsftpd
[root@master vsftpd]# ls
ftpusers user_list
将这两个文件中的root注释掉。
然后重启ftp
在两台机器上创建文件夹
[root@master ~]# mkdir installer
[root@master ~]#
上传jdk
E:\开发工具
jdk-7u79-linux-i586.rpm
到installer目录
Master机器
[root@master installer]# rpm -ivh jdk-7u79-linux-i586.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
plugin.jar...
javaws.jar...
Deploy.jar...
Slave机器
[root@slave installer]# rpm -ivh jdk-7u79-linux-i586.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
plugin.jar...
javaws.jar...
deploy.jar...
添加hadoop用户
[root@master ~]# useradd hadoop
[root@master ~]# passwd hadoop
Changing password for user hadoop.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[root@slave ~]# useradd hadoop
[root@slave ~]# passwd hadoop
Changing password for user hadoop.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
并且在slave1和slave2使用hadoop用户创建installer目录
[root@master ~]# su - hadoop
[hadoop@master ~]$ mkdir installer
[root@slave ~]# su - hadoop
[hadoop@slave ~]$ mkdir installer
配置ssh等效性
一路回车键
[hadoop@master ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
32:da:01:64:6a:48:4c:f0:58:4f:60:2e:ea:18:a0:5d hadoop@master
[hadoop@slave ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
80:49:e7:58:18:0b:9c:1f:ee:d8:96:5a:c5:50:a5:bb hadoop@slave
Master
[hadoop@master ~]$ cd .ssh/
[hadoop@master .ssh]$ ls
id_rsa id_rsa.pub
[hadoop@master .ssh]$ cat id_rsa.pub > authorized_keys
[hadoop@master .ssh]$ scp authorized_keys slave:~/.ssh/
The authenticity of host 'slave (192.168.3.101)' can't be established.
RSA key fingerprint is de:5a:2e:67:62:a4:27:2c:d7:de:85:eb:c7:6c:93:c8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave,192.168.3.101' (RSA) to the list of known hosts.
hadoop@slave's password:
authorized_keys
Slave
[hadoop@slave ~]$ cd .ssh/
[hadoop@slave .ssh]$ ls
authorized_keys id_rsa id_rsa.pub
[hadoop@slave .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@slave .ssh]$ scp authorized_keys master:~/.ssh/
The authenticity of host 'master (192.168.3.100)' can't be established.
RSA key fingerprint is de:5a:2e:67:62:a4:27:2c:d7:de:85:eb:c7:6c:93:c8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.3.100' (RSA) to the list of known hosts.
hadoop@master's password:
authorized_keys
最后
[hadoop@master .ssh]$ chmod 600 authorized_keys
[hadoop@slave .ssh]$ chmod 600 authorized_keys
到这里配置完毕，可以直接使用
[hadoop@master ~]$ ssh slave
[hadoop@slave ~]$ ssh master
配置hadoop
解压缩
[hadoop@master installer]$ tar -zxvf hadoop-2.6.0.tar.gz
[hadoop@master installer]$ mv hadoop-2.6.0 hadoop2
[hadoop@master installer]$ ls
hadoop2 hadoop-2.6.0.tar.gz
配置环境变量
[hadoop@master ~]$ vim .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific aliases and functions
export JAVA_HOME=/usr/java/jdk1.7.0_79
export HADOOP_HOME=/home/hadoop/installer/hadoop2
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export CLASSPATH=$CLASSPATH:$HADOOP_HOME/lib:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
让bashrc生效
[hadoop@master ~]$ . .bashrc
[hadoop@master ~]$ scp .bashrc slave:~
.bashrc 100% 341 0.3KB/s 00:00
配置hadoop文件
[hadoop@master hadoop]$ vim hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_79
#export HADOOP_HEAPSIZE=
export HADOOP_HEAPSIZE=100
[hadoop@master hadoop]$ vim yarn-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_79
JAVA_HEAP_MAX=-Xmx300m
YARN_HEAPSIZE=100
[hadoop@master hadoop]$ vim mapred-env.sh
export JAVA_HOME=/usr/java/jdk1.6.0_27
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=100
[hadoop@master hadoop]$ vim slaves
slave
[hadoop@master hadoop]$ vim core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/tmp</value>
</property>
</configuration>
创建tmp目录
[hadoop@master ~]$ mkdir tmp
[hadoop@slave ~]$ mkdir tmp
[hadoop@master hadoop]$ vim hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:50090</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/data/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/data/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
创建目录
[hadoop@master ~]$ mkdir -p data/dfs/data
[hadoop@master ~]$ mkdir -p data/dfs/name
[hadoop@slave ~]$ mkdir -p data/dfs/data
[hadoop@slave ~]$ mkdir -p data/dfs/name
[hadoop@master hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@master hadoop]$ vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>300</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>300</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>100</value>
</property>
</configuration>
[hadoop@master hadoop]$ vim yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<description>The address of the applications manager interface in the RM.</description>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<description>The address of the scheduler interface.</description>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<description>The http address of the RM web application.</description>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<description>The https adddress of the RM web application.</description>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<description>The address of the RM admin interface.</description>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<description>The minimum allocation for every container request at the RM,
in MBs. Memory requests lower than this won't take effect,
and the specified value will get allocated at minimum.</description>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>300</value>
</property>
<property>
<description>The maximum allocation for every container request at the RM,
in MBs. Memory requests higher than this won't take effect,
and will get capped to this value.</description>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>1024</value>
</property>
<property>
<description>Ratio between virtual memory to physical memory when
setting memory limits for containers. Container allocations are
expressed in terms of physical memory, and virtual memory usage
is allowed to exceed this allocation by this ratio.
</description>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>6.1</value>
</property>
</configuration>
[hadoop@master installer]$ cd installer/
[hadoop@master installer]$ scp -r hadoop2 slave:~/installer/
Slave机器上执行
[hadoop@slave ~]$ . .bashrc
格式化
[hadoop@master ~]$ hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
16/11/03 19:42:11 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master/192.168.3.100
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.6.0
STARTUP_MSG: classpath = /home/hadoop/installer/hadoop2/etc/hadoop:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/curator-client-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/htrace-core-3.0.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/hadoop-annotations-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/curator-framework-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/hadoop-auth-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/hadoop-common-2.6.0-tests.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/hadoop-nfs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/hadoop-common-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/hadoop-hdfs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/hadoop-hdfs-2.6.0-tests.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-registry-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-client-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-api-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0.jar:/home/hadoop/installer/hadoop2/contrib/capacity-scheduler/*.jar:/home/hadoop/installer/hadoop2/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by 'jenkins' on 2014-11-13T21:10Z
STARTUP_MSG: java = 1.7.0_79
************************************************************/
16/11/03 19:42:11 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/11/03 19:42:11 INFO namenode.NameNode: createNameNode [-format]
16/11/03 19:42:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/03 19:42:11 WARN common.Util: Path /home/hadoop/data/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
16/11/03 19:42:11 WARN common.Util: Path /home/hadoop/data/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-9c79097a-7f44-48cc-9308-ea9c487a685e
16/11/03 19:42:11 INFO namenode.FSNamesystem: No KeyProvider found.
16/11/03 19:42:11 INFO namenode.FSNamesystem: fsLock is fair:true
16/11/03 19:42:11 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/11/03 19:42:11 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/11/03 19:42:11 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/11/03 19:42:11 INFO blockmanagement.BlockManager: The block deletion will start around 2016 十一月 03 19:42:11
16/11/03 19:42:11 INFO util.GSet: Computing capacity for map BlocksMap
16/11/03 19:42:11 INFO util.GSet: VM type = 32-bit
16/11/03 19:42:11 INFO util.GSet: 2.0% max memory 96.7 MB = 1.9 MB
16/11/03 19:42:11 INFO util.GSet: capacity = 2^19 = 524288 entries
16/11/03 19:42:11 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/11/03 19:42:11 INFO blockmanagement.BlockManager: defaultReplication = 1
16/11/03 19:42:11 INFO blockmanagement.BlockManager: maxReplication = 512
16/11/03 19:42:11 INFO blockmanagement.BlockManager: minReplication = 1
16/11/03 19:42:11 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
16/11/03 19:42:11 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
16/11/03 19:42:11 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/11/03 19:42:11 INFO blockmanagement.BlockManager: encryptDataTransfer = false
16/11/03 19:42:11 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
16/11/03 19:42:12 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
16/11/03 19:42:12 INFO namenode.FSNamesystem: supergroup = supergroup
16/11/03 19:42:12 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/11/03 19:42:12 INFO namenode.FSNamesystem: HA Enabled: false
16/11/03 19:42:12 INFO namenode.FSNamesystem: Append Enabled: true
16/11/03 19:42:12 INFO util.GSet: Computing capacity for map INodeMap
16/11/03 19:42:12 INFO util.GSet: VM type = 32-bit
16/11/03 19:42:12 INFO util.GSet: 1.0% max memory 96.7 MB = 990.1 KB
16/11/03 19:42:12 INFO util.GSet: capacity = 2^18 = 262144 entries
16/11/03 19:42:12 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/11/03 19:42:12 INFO util.GSet: Computing capacity for map cachedBlocks
16/11/03 19:42:12 INFO util.GSet: VM type = 32-bit
16/11/03 19:42:12 INFO util.GSet: 0.25% max memory 96.7 MB = 247.5 KB
16/11/03 19:42:12 INFO util.GSet: capacity = 2^16 = 65536 entries
16/11/03 19:42:12 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/11/03 19:42:12 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/11/03 19:42:12 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
16/11/03 19:42:12 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/11/03 19:42:12 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/11/03 19:42:12 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/11/03 19:42:12 INFO util.GSet: VM type = 32-bit
16/11/03 19:42:12 INFO util.GSet: 0.029999999329447746% max memory 96.7 MB = 29.7 KB
16/11/03 19:42:12 INFO util.GSet: capacity = 2^13 = 8192 entries
16/11/03 19:42:12 INFO namenode.NNConf: ACLs enabled? false
16/11/03 19:42:12 INFO namenode.NNConf: XAttrs enabled? true
16/11/03 19:42:12 INFO namenode.NNConf: Maximum size of an xattr: 16384
16/11/03 19:42:12 INFO namenode.FSImage: Allocated new BlockPoolId: BP-942661971-192.168.3.100-1478173332298
16/11/03 19:42:12 INFO common.Storage: Storage directory /home/hadoop/data/dfs/name has been successfully formatted.
16/11/03 19:42:12 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/11/03 19:42:12 INFO util.ExitUtil: Exiting with status 0
16/11/03 19:42:12 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.3.100
************************************************************/
没有任何错误，格式化成功。
[hadoop@master ~]$ start-dfs.sh
[hadoop@master ~]$ start-yarn.sh
[hadoop@master ~]$ jps
17769 ResourceManager
18020 Jps
17443 NameNode
17631 SecondaryNameNode
[hadoop@slave ~]$ jps
13538 Jps
13297 DataNode
13408 NodeManager
[hadoop@master mapreduce]$ ls
hadoop-mapreduce-client-app-2.6.0.jar hadoop-mapreduce-client-hs-plugins-2.6.0.jar hadoop-mapreduce-examples-2.6.0.jar
hadoop-mapreduce-client-common-2.6.0.jar hadoop-mapreduce-client-jobclient-2.6.0.jar lib
hadoop-mapreduce-client-core-2.6.0.jar hadoop-mapreduce-client-jobclient-2.6.0-tests.jar lib-examples
hadoop-mapreduce-client-hs-2.6.0.jar hadoop-mapreduce-client-shuffle-2.6.0.jar sources
[hadoop@master mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /data/ /output
16/11/03 20:01:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/03 20:01:45 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.3.100:8032
16/11/03 20:01:49 INFO input.FileInputFormat: Total input paths to process : 2
16/11/03 20:01:49 INFO mapreduce.JobSubmitter: number of splits:2
16/11/03 20:01:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1478174305039_0001
16/11/03 20:01:54 INFO impl.YarnClientImpl: Submitted application application_1478174305039_0001
16/11/03 20:01:55 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1478174305039_0001/
16/11/03 20:01:55 INFO mapreduce.Job: Running job: job_1478174305039_0001
16/11/03 20:02:06 INFO mapreduce.Job: Job job_1478174305039_0001 running in uber mode : false
16/11/03 20:02:06 INFO mapreduce.Job: map 0% reduce 0%
16/11/03 20:02:06 INFO mapreduce.Job: Job job_1478174305039_0001 failed with state FAILED due to: Application application_1478174305039_0001 failed 2 times due to AM Container for appattempt_1478174305039_0001_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://master:8088/proxy/application_1478174305039_0001/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=11781,containerID=container_1478174305039_0001_02_000001] is running beyond virtual memory limits. Current usage: 41.4 MB of 100 MB physical memory used; 1.2 GB of 210.0 MB virtual memory used. Killing container.
Dump of the process-tree for container_1478174305039_0001_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 11781 11780 11781 11781 (bash) 0 0 65404928 274 /bin/bash -c /usr/java/jdk1.7.0_79/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001/stdout 2>/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001/stderr
|- 11790 11781 11781 11781 (java) 75 38 1248215040 10317 /usr/java/jdk1.7.0_79/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
16/11/03 20:02:06 INFO mapreduce.Job: Counters: 0
[hadoop@master hadoop2]$ vim ./share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
<property>
<description>Ratio between virtual memory to physical memory when
setting memory limits for containers. Container allocations are
expressed in terms of physical memory, and virtual memory usage
is allowed to exceed this allocation by this ratio.
</description>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value> 这里的2.1是一个系数
</property>
标红错误说 物理内存乘以2.1 得到210M，因为一开始我们设置了物理内存是100M
调整方法要么把2.1调整大，要么调整大物理内存
修改之前最好先停掉yarn
[hadoop@master hadoop]$ vim yarn-site.xml
<property>
<description>The minimum allocation for every container request at the RM,
in MBs. Memory requests lower than this won't take effect,
and the specified value will get allocated at minimum.</description>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>200</value>
</property>
<property>
<description>Ratio between virtual memory to physical memory when
setting memory limits for containers. Container allocations are
expressed in terms of physical memory, and virtual memory usage
is allowed to exceed this allocation by this ratio.
</description>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>6.1</value>
</property>
[hadoop@master hadoop]$ pwd
/home/hadoop/install/hadoop2/etc/hadoop
[hadoop@master hadoop]$ scp yarn-site.xml slave1:/home/hadoop/hadoop2/etc/hadoop
yarn-site.xml 100% 3165 3.1KB/s 00:00
[hadoop@master hadoop]$ scp yarn-site.xml slave2:/home/hadoop/hadoop2/etc/hadoop
yarn-site.xml 100% 3165 3.1KB/s 00:00
可以不用停止hdfs服务
继续测试
[hadoop@master hadoop]$ start-yarn.sh
[hadoop@master mapreduce]$ jps
16511 Jps
14927 NameNode
15112 SecondaryNameNode
[hadoop@master mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /data/ /output
16/11/03 20:10:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/03 20:10:01 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.3.100:8032
16/11/03 20:10:02 INFO input.FileInputFormat: Total input paths to process : 2
16/11/03 20:10:02 INFO mapreduce.JobSubmitter: number of splits:2
16/11/03 20:10:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1478174985535_0001
16/11/03 20:10:03 INFO impl.YarnClientImpl: Submitted application application_1478174985535_0001
16/11/03 20:10:03 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1478174985535_0001/
16/11/03 20:10:03 INFO mapreduce.Job: Running job: job_1478174985535_0001
16/11/03 20:10:13 INFO mapreduce.Job: Job job_1478174985535_0001 running in uber mode : false
16/11/03 20:10:13 INFO mapreduce.Job: map 0% reduce 0%
16/11/03 20:10:54 INFO mapreduce.Job: map 100% reduce 0%
16/11/03 20:11:01 INFO mapreduce.Job: map 100% reduce 100%
16/11/03 20:11:02 INFO mapreduce.Job: Job job_1478174985535_0001 completed successfully
16/11/03 20:11:02 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=62
FILE: Number of bytes written=316503
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=226
HDFS: Number of bytes written=30
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=75720
Total time spent by all reduces in occupied slots (ms)=3903
Total time spent by all map tasks (ms)=75720
Total time spent by all reduce tasks (ms)=3903
Total vcore-seconds taken by all map tasks=75720
Total vcore-seconds taken by all reduce tasks=3903
Total megabyte-seconds taken by all map tasks=22716000
Total megabyte-seconds taken by all reduce tasks=1170900
Map-Reduce Framework
Map input records=4
Map output records=7
Map output bytes=66
Map output materialized bytes=68
Input split bytes=188
Combine input records=7
Combine output records=5
Reduce input groups=4
Reduce shuffle bytes=68
Reduce input records=5
Reduce output records=4
Spilled Records=10
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=1281
CPU time spent (ms)=9240
Physical memory (bytes) snapshot=394346496
Virtual memory (bytes) snapshot=1360240640
Total committed heap usage (bytes)=257433600
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=38
File Output Format Counters
Bytes Written=30
[hadoop@master ~]$ hadoop dfs -cat /output/*
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
16/11/03 20:14:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
123 2
hadoop 2
java 2
spark 1
这样表示hadoop安装成功
