ubuntu14.04下编译安装ambari-2.4.2.0
ubuntu14.04下编译安装ambari-2.4.2.0
编译前的准备工作
准备工作有：
下面是具体配置
系统参数
1、fs.inotify.max_user_watches = 8192 调整到524288
echo fs.inotify.max_user_watches=524288 >> /etc/sysctl.conf && sysctl -p
2、将每个进程可以打开的文件数目加大到65535，缺省为1024
ulimit -SHn 65535
sudo tee /etc/security/limits.d/90-nproc.conf <<-'EOF'
* hard nproc 64000
* soft nproc 64000
root hard nproc 65535
root soft nproc 65535
EOF
ulimit -a
系统依赖（编译环境）
sudo apt-get install build-essential cmake automake autoconf bison libboost-all-dev libtool check libmcrypt-dev libeditline-dev libssl-dev
离线安装包
1. httpserver
sh starthttpd.sh
#/usr/bin
cd /media/gordon/DataDisk/软件包
/usr/bin/python -m SimpleHTTPServer 8000 >/dev/null 2>&1 &
exit 0
2. 布置本地软件源
http://192.168.56.1:8000/ambari-HDP/
-rwxr-x--- 1 gitlab root 23667150 3月 31 16:26 grafana-2.6.0.linux-x64.tar.gz*
-rwxr-x--- 1 gitlab root 210185315 3月 31 17:15 hadoop-2.7.1.2.3.4.0-3347.tar.gz*
-rwxr-x--- 1 gitlab root 101551599 3月 31 16:26 hbase-1.1.2.2.3.4.0-3347.tar.gz*
-rwxr-x--- 1 gitlab root 251655846 3月 31 16:26 phoenix-4.4.0.2.3.4.0-3347.tar.gz*
java环境
vim /etc/profile
#set for java
export JAVA_HOME=/opt/jdk1.8.0_111
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"
maven环境
vim /etc/profile
#set maven environment
M2_HOME=/opt/apache-maven-3.3.9/
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$M2_HOME/bin:$PATH
maven有2个位置可以选择配置M2_HOME/conf/setting.xml和~/.m2/setting.xml
前者是全局性的，后者是针对用户的。
查询资料是说用户配置优先级高于全局配置
设置maven-repo本地源
vim /opt/apache-maven-3.3.9/conf/settings.xml
<localRepository>/opt/maven-repo</localRepository>
gord@gord:/opt$ mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/apache-maven-3.3.9
Java version: 1.8.0_111, vendor: Oracle Corporation
Java home: /opt/jdk1.8.0_111/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-24-generic", arch: "amd64", family: "unix"
Nodejs环境
今天是2017.04.01 https://nodejs.org/en/官网推荐v6.10.1版本 #实验证明这个版本安装ambari存在很大问题，切换到0.10.44版本的nodejs
可在淘宝下载各个版本：https://npm.taobao.org/mirrors/node
tar xfp node-v0.10.44-linux-x64.tar.gz -C /opt
vim /etc/profile
#set for nodejs
export NODE_HOME=/opt/node-v0.10.44-linux-x64
#export NODE_HOME=/opt/node-v4.4.4-linux-x64
#export NODE_HOME=/opt/node-v6.10.1-linux-x64
export PATH=$PATH:$NODE_HOME/bin
source /etc/profile
配置淘宝源
kylin@Ubuntu:~$ npm config set registry https://registry.npm.taobao.org
kylin@Ubuntu:~$ npm info underscor （如果上面配置正确这个命令会有字符串response）
gord@gord:~$ npm --registry https://registry.npm.taobao.org info underscore
修改源地址为官方源
npm config set registry https://registry.npmjs.org/
安装指定版本的brunch
npm root -g
cd ...
npm install -g brunch@1.7.20 #务必要装这个版本
brunch -V
1.7.20
如果存在问题，执行卸载brunch
npm remove -g brunc
git环境
root@gord:~# git --version
git version 1.9.1
ant环境
root@gord:~# apt-get install ant
root@gord:~# ant -version
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Apache Ant(TM) version 1.9.3 compiled on April 8 2014
python环境
apt install python-pip
准备工作总结：
profile配置总结
#set for java
export JAVA_HOME=/opt/jdk1.8.0_111
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"
#set for maven
M2_HOME=/opt/apache-maven-3.3.9/
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$M2_HOME/bin:$PATH
#set for nodejs
export NODE_HOME=/opt/node-v0.10.44-linux-x64
#export NODE_HOME=/opt/node-v4.4.4-linux-x64
#export NODE_HOME=/opt/node-v6.10.1-linux-x64
export PATH=$PATH:$NODE_HOME/bin
至此编译ambari所依赖的环境配置完毕，并可作为通用的编译环境推广使用。
推广到其它编译环境中，注意nodejs的版本可能需要更换！
准备工作至关重要，关系到编译的成败和时间的消耗！
编译安装ambari
主要步骤：
参考
https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.4.2
版本控制
wget http://www.apache.org/dist/ambari/ambari-2.4.2/apache-ambari-2.4.2-src.tar.gz (use the suggested mirror from above)
tar xfvz apache-ambari-2.4.2-src.tar.gz
cd apache-ambari-2.4.2-src
mvn versions:set -DnewVersion=2.4.2.0.0
root@gord:/opt/apache-ambari-2.4.2-src# cd ambari-metrics
/opt/apache-ambari-2.4.2-src/ambari-metrics /opt/apache-ambari-2.4.2-src
mvn versions:set -DnewVersion=2.4.2.0.0
root@gord:/opt/apache-ambari-2.4.2-src/ambari-metrics# cd ..
/opt/apache-ambari-2.4.2-src
编译
-B非交互
mvn -B clean install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6"
-X -e debug
mvn -X -e clean install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6"
如果存在手工下载软件包到指定位置的情况，上述命令去掉clean参数，防止被clean
编译成功显示
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 2.741 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.155 s]
[INFO] Ambari Web ......................................... SUCCESS [01:00 min]
[INFO] Ambari Views ....................................... SUCCESS [ 1.475 s]
[INFO] Ambari Admin View .................................. SUCCESS [ 19.390 s]
[INFO] ambari-metrics ..................................... SUCCESS [ 0.770 s]
[INFO] Ambari Metrics Common .............................. SUCCESS [ 0.732 s]
[INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [ 4.208 s]
[INFO] Ambari Metrics Flume Sink .......................... SUCCESS [ 2.511 s]
[INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [ 1.676 s]
[INFO] Ambari Metrics Storm Sink .......................... SUCCESS [ 4.837 s]
[INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [ 3.420 s]
[INFO] Ambari Metrics Collector ........................... SUCCESS [ 23.296 s]
[INFO] Ambari Metrics Monitor ............................. SUCCESS [ 1.944 s]
[INFO] Ambari Metrics Grafana ............................. SUCCESS [ 2.544 s]
[INFO] Ambari Metrics Assembly ............................ SUCCESS [02:26 min]
[INFO] Ambari Server ...................................... SUCCESS [04:24 min]
[INFO] Ambari Functional Tests ............................ SUCCESS [ 1.385 s]
[INFO] Ambari Agent ....................................... SUCCESS [ 31.964 s]
[INFO] Ambari Client ...................................... SUCCESS [ 0.084 s]
[INFO] Ambari Python Client ............................... SUCCESS [ 1.168 s]
[INFO] Ambari Groovy Client ............................... SUCCESS [ 8.157 s]
[INFO] Ambari Shell ....................................... SUCCESS [ 0.105 s]
[INFO] Ambari Python Shell ................................ SUCCESS [ 1.422 s]
[INFO] Ambari Groovy Shell ................................ SUCCESS [ 3.714 s]
[INFO] ambari-logsearch ................................... SUCCESS [ 0.041 s]
[INFO] Ambari Logsearch Appender .......................... SUCCESS [ 31.627 s]
[INFO] Ambari Logsearch Solr Client ....................... SUCCESS [01:38 min]
[INFO] Ambari Logsearch Portal ............................ SUCCESS [06:10 min]
[INFO] Ambari Logsearch Log Feeder ........................ SUCCESS [06:03 min]
[INFO] Ambari Logsearch Assembly .......................... SUCCESS [ 0.323 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:14 min
[INFO] Finished at: 2017-04-05T18:44:18+08:00
[INFO] Final Memory: 274M/850M
[INFO] ------------------------------------------------------------------------
整理deb包
root@gord:/opt/apache-ambari-2.4.2-src# find ./ -name "*.deb" |awk '{print "cp",$0,"/media/sf_DataDisk/软件包/ambari-repo/Compiled/"}'|sh
./target/ambari_2.4.2.0.0_all.deb
./ambari-metrics/target/ambari-metrics_2.4.2.0.0_all.deb
./ambari-metrics/ambari-metrics-assembly/target/ambari-metrics-assembly_2.4.2.0.0_all.deb
./ambari-agent/target/ambari-agent_2.4.2.0-0.deb
./ambari-shell/target/ambari-shell_2.4.2.0.0_all.deb
./ambari-shell/ambari-python-shell/target/ambari-python-shell_2.4.2.0-0.deb
./ambari-project/target/ambari-project_2.4.2.0.0_all.deb
./ambari-server/target/ambari-server_2.4.2.0-0-dist.deb
./ambari-client/target/ambari-client_2.4.2.0.0_all.deb
./ambari-client/python-client/target/python-client_2.4.2.0-0.deb
./ambari-client/groovy-client/target/groovy-client_2.4.2.0.0_all.deb
安装
服务端安装
##gdebi ambari-server* #This should also pull in postgres packages as well.
dpkg -i ambari-server_2.4.2.0-0-dist.deb
apt-get -f install
设置与启动
ambari-server setup #默认即可
ambari-server start
如果遇到报错：/usr/sbin/ambari-server: line 33: buildNumber: unbound variable
打开ambari-server脚本，把${buildNumber}这行换成 HASH="${VERSION}"
相关的命令有以下：
ambari-server status
ambari-server stop
客户端安装
dpkg -i ambari-agent*
apt-get -f install
编辑ambari-agent配置文件，设置之前安装过Ambari-server 的主机名：
vi /etc/ambari-agent/conf/ambari-agent.ini
[server]
hostname=<your.ambari.server.hostname>
url_port=8440
secured_url_port=8441
ambari-agent start
如果遇到报错：/usr/sbin/ambari-agent: line 23: buildNumber: unbound variable
打开ambari-agent脚本，把${buildNumber}这行换成 HASH="${VERSION}"
相关的命令有以下：
ambari-agent status # 检查agent服务运行状态
ambari-agent stop # 停止agent服务
访问
http://<ambari-server-host>:8080.
Log in with username admin and password admin
问题解决
1、Caused by: org.vafer.jdeb.PackagingException: “/opt/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana/src/main/package/deb/control” is not a valid ‘control’ directory)
/opt/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana# vim pom.xml
<plugin>
<groupId>org.vafer</groupId>
<artifactId>jdeb</artifactId>
<version>1.0.1</version>
<executions>
<execution>
<!--Stub execution on direct plugin call - workaround for ambari deb build process-->
<id>stub-execution</id>
<phase>none</phase>
<goals>
<goal>jdeb</goal>
</goals>
</execution>
</executions>
<configuration>
<skip>true</skip>
<attach>false</attach>
<submodules>false</submodules>
<controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
</configuration>
</plugin>
2、如果中间断掉，建议去掉clean，继续编译。
3、安装Nodejs 6.x.x版本不能编译，建议卸载，安装0.10.44。
[INFO] Ambari Web ......................................... FAILURE [ 13.325 s]
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (compile-brunch) on project ambari-web: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
版本检查：
root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# brunch --version
2.10.9
root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# npm --version
3.10.10
root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# node --version
v6.10.1
改成安装node 0.10.44，brunch1.7.20后正常
4、编译过程中出现版本不一致问题，请参考http://blog.csdn.net/royma_1990/article/details/51749981。
5、下载慢问题处理
修改如下配置文件：
cat ambari-metrics/pom.xml
<!--TODO change to HDP URL-->
<hbase.tar>http://192.168.56.1:8000/ambari-HDP/hbase-1.1.2.2.3.4.0-3347.tar.gz</hbase.tar>
<hbase.folder>hbase-1.1.2.2.3.4.0-3347</hbase.folder>
<hadoop.tar>http://192.168.56.1:8000/ambari-HDP/hadoop-2.7.1.2.3.4.0-3347.tar.gz</hadoop.tar>
<hadoop.folder>hadoop-2.7.1.2.3.4.0-3347</hadoop.folder>
<hbase.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hbase-0.98.4.2.2.4.2-0002-hadoop2.winpkg.zip</hbase.winpkg.zip>
<hbase.winpkg.folder>hbase-0.98.4.2.2.4.2-0002-hadoop2</hbase.winpkg.folder>
<hadoop.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hadoop-2.6.0.2.2.4.2-0002.winpkg.zip</hadoop.winpkg.zip>
<hadoop.winpkg.folder>hadoop-2.6.0.2.2.4.2-0002</hadoop.winpkg.folder>
<grafana.folder>grafana-2.6.0</grafana.folder>
<grafana.tar>http://192.168.56.1:8000/ambari-HDP/grafana-2.6.0.linux-x64.tar.gz</grafana.tar>
<phoenix.tar>http://192.168.56.1:8000/ambari-HDP/phoenix-4.4.0.2.3.4.0-3347.tar.gz</phoenix.tar>
<phoenix.folder>phoenix-4.4.0.2.3.4.0-3347</phoenix.folder>
6、警告：（可忽略）
[INFO] npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
root@gord:/opt/apache-ambari-2.4.2-src/ambari-web# npm ls graceful-fs
Ambari@2.4.0 /opt/apache-ambari-2.4.2-src/ambari-web
├─┬ karma@0.11.14
│ └── graceful-fs@2.0.3
├─┬ karma-coverage@0.2.7
│ └─┬ dateformat@1.0.12
│ └─┬ meow@3.7.0
│ └─┬ read-pkg-up@1.0.1
│ └─┬ read-pkg@1.1.0
│ ├─┬ load-json-file@1.1.0
│ │ └── graceful-fs@4.1.11
│ └─┬ path-type@1.1.0
│ └── graceful-fs@4.1.11
├─┬ mocha-phantomjs@3.1.6
│ └─┬ mocha@1.13.0
│ └─┬ glob@3.2.3
│ └── graceful-fs@2.0.3
└─┬ phantomjs@1.9.20
└─┬ fs-extra@0.26.7
└── graceful-fs@4.1.11
7、找不到org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT
[INFO] Ambari Metrics Storm Sink .......................... FAILURE [ 4.111 s]
[ERROR] Failed to execute goal on project ambari-metrics-storm-sink: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-storm-sink:jar:2.4.2.0.0: Could not find artifact org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT in apache-hadoop (http://repo.hortonworks.com/content/repositories/releases/org/apache/) -> [Help 1]
vim ambari-metrics/ambari-metrics-storm-sink/pom.xml
<properties>
<!--storm.version>1.1.0-SNAPSHOT</storm.version-->
<storm.version>1.1.0</storm.version>
</properties>
8、ambari-funtest
[ERROR] Failed to create debian package /home/kylin/apache-ambari-2.4.2-src/ambari-funtest/target/ambari-funtest_2.4.2.0.0_all.deb
org.vafer.jdeb.PackagingException: “/home/kylin/apache-ambari-2.4.2-src/ambari-funtest/src/main/package/deb/control” is not a valid ‘control’ directory)
解决办法：在上面添加一个元素
/opt/apache-ambari-2.4.2-src/ambari-funtest$ vi pom.xml
<plugins>
...
<plugin>
<groupId>org.vafer</groupId>
<artifactId>jdeb</artifactId>
<version>1.0.1</version>
<executions>
<execution>
<!--Stub execution on direct plugin call - workaround for ambari deb build process-->
<id>stub-execution</id>
<phase>none</phase>
<goals>
<goal>jdeb</goal>
</goals>
</execution>
</executions>
<configuration>
<skip>true</skip>
<attach>false</attach>
<submodules>false</submodules>
<controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
</configuration>
</plugin>
</plugins>
<resources>
<resource>
10、[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /opt/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: java.net.UnknownHostException: www.w3.org: Unknown host www.w3.org -> [Help 1]
解决办法：重新执行
mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true
11、findbugsXml.html
[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /opt/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: The entity name must immediately follow the ‘&’ in the entity reference. -> [Help 1]
解决办法1： 重新执行
mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true
解决办法2: 如果网络情况不好，这一步就很难过去。那只好修改pom.xml文件，将findbugs内容注释掉：
/opt/apache-ambari-2.4.2-src/ambari-server$ vim pom.xml
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>3.0.3</version>
<configuration>
<failOnError>false</failOnError>
<threshold>Low</threshold>
<findbugsXmlOutputDirectory>${project.basedir}/target/findbugs</findbugsXmlOutputDirectory>
</configuration>
<executions>
<!--
<execution>
<phase>verify</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
-->
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>xml-maven-plugin</artifactId>
<version>1.0</version>
<executions>
<!--
<execution>
<phase>verify</phase>
<goals>
<goal>transform</goal>
</goals>
</execution>
-->
</executions>
12、[ERROR] Failed to execute goal org.vafer:jdeb:1.0.1:jdeb (default-cli) on project ambari-logsearch: Failed to create debian package /opt/apache-ambari-2.4.2-src/ambari-logsearch/target/ambari-logsearch_2.4.2.0.0_all.deb: "/opt/apache-ambari-2.4.2-src/ambari-logsearch/src/main/package/deb/control" is not a valid 'control' directory) -> [Help 1]
解决办法：在上面添加一个元素
/opt/apache-ambari-2.4.2-src/ambari-logsearch$ vim pom.xml
<plugin>
<groupId>org.vafer</groupId>
<artifactId>jdeb</artifactId>
<version>1.0.1</version>
<executions>
<execution>
<!--Stub execution on direct plugin call - workaround for ambari deb build process-->
<id>stub-execution</id>
<phase>none</phase>
<goals>
<goal>jdeb</goal>
</goals>
</execution>
</executions>
<configuration>
<skip>true</skip>
<attach>false</attach>
<submodules>false</submodules>
<controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
</configuration>
</plugin>
</plugins>
</build>
参考资源：
@firecontainer
April 6, 2017 4:53 PM
@Gordon_chang
1997年毕业于北京联合大学，先后在中国万网，新媒传信，亚信等公司工作，现在在一家创业型公司担任云计算与大数据运维方面的 PM & Engineer。 专注于以下四个领域：
分布式存储
分布式数据库
云计算
大数据
重点通过技术架构与性能优化（底层）实现基于私有云的大数据平台能力
