虚拟机环境下用docker在centos环境中搭建Ambari HDP大数据平台
利用虚拟机安装docker，在docker环境中搭建HDP大数据平台
前言 : 在一台虚拟机上搭建3个docker容器 作为安装HDP的环境，虚拟机环境centos7.6 docker环境为centos7.6
因为想要对ambari进行二次开发，所以本次选择手动编译ambari。
https://hub.docker.com/_/centos 在此网站查询想要的版本号
一、在虚拟机上准备工作
1.1安装docker
yum -y install docker
1.2下载镜像
docker search centos //搜索镜像
docker pull centos:7.6.1810 //从官网拉取镜像
docker images //查看本地镜像
docker ps -a //查看容器
1.3启动容器
因为是在1台虚拟机搭建3个docker的缘故，所以这次选择的docker 网卡类型为bridge模式 网段为 172.17.0.0/16 网关为172.17.0.1
三台容器的主机名和ip分别为
server 172.17.0.2
agent1 172.17.0.3
agent2 172.17.0.4
※注意点：bridge模式容器ip无法自己指定
开始启动3台容器，如下图
docker run -it -d --name server --network bridge --hostname server docker.io/centos:7.6.1810 /bin/bash
1.4查看容器并进入
docker ps
进入容器
docker exec -it server /bin/bash
二、搭建编译环境
2.1获取Ambari2.7.5源码
下载地址：http://apache.communilink.net/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz
wget http://apache.communilink.net/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz
tar zxvf apache-ambari-2.7.5-src.tar.gz
2.2搭建编译环境
安装包已放到百度云 maven jdk nodejs
链接：https://pan.baidu.com/s/1SVf7n5d2M_byquSpwN5yXg
提取码：i4e6
2.2.1安装网络工具
yum install net-tools.x86_64
2.2.2 安装jdk
rpm -ivh jdk-8u74-linux-x64.rpm
2.2.3 安装maven，nodejs
tar -zxvf apache-maven-3.3.9-bin.tar.gz
tar -zxvf node-v4.5.0-linux-x64.tar.gz
yum install epel-release
2.2.3.1 升级nodejs
npm install n -g
n stable
2.2.4 配置maven，nodejs的环境变量
vim /etc/profile
export MAVEN_HOME=/opt/apache-maven-3.3.9export PATH=$MAVEN_HOME/bin:$PATH
export NODE_HOME=/opt/node-v4.5.0-linux-x64
export PATH=$PATH:$NODE_HOME/bin
export NODE_PATH=$NODE_HOME/lib/node_modules
source /etc/profile
2.2.5 python 2.6或2.7(系统自带)
2.2.6安装rpmbuild
yum install rpm-build
2.2.7安装g++
yum install gcc-c++
2.2.8安装python-devel
yum install python-devel
2.2.9安装bower、gulp
npm install -g bower
npm install -g gulp
2.2.10安装git
yum install git
2.2.11安装yarn
yum install yarn
三、提前下载并设置
下载源码,设置版本信息
wget https://www-eu.apache.org/dist/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz
tar -xfvz apache-ambari-2.7.5-src.tar.gz
cd apache-ambari-2.7.5-src
mvn versions:set -DnewVersion=2.7.5.0.0
pushd ambari-metrics
mvn versions:set -DnewVersion=2.7.5.0.0
popd
有些包比较大，或者编译时下载时间较长，可以提前下载到本地目录，再修改pom.xml文件指定到本地目录(pom文件位置:/opt/apache-ambari-2.7.5-src/ambari-metrics/pom.xml)
3.1 hbase下载
mkdir -p /tmp/hbase
wget -O /tmp/hbase/hbase-2.0.2.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hbase/hbase-2.0.2.3.1.4.0-315-bin.tar.gz
3.2 grafana 包下载
mkdir -p /tmp/grafana
wget -O /tmp/grafana/grafana-6.4.2.linux-amd64.tar.gz https://dl.grafana.com/oss/release/grafana-6.4.2.linux-amd64.tar.gz
3.3 hadoop包下载
mkdir -p /tmp/hadoop
wget -O /tmp/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz
3.4phoenix包下载
mkdir -p /tmp/phoenix
wget -O /tmp/hadoop/phoenix-5.0.0.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/phoenix/phoenix-5.0.0.3.1.4.0-315.tar.gz
3.5 开始编译
mvn -B clean install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -Preplaceurl
最后
编译好后的rpm包在
cd ambari-server/target/rpm/ambari-server/RPMS/x86_64/
ambari-server-2.7.4.0-0.x86_64.rpm
编译遇到的问题
问题1：
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed.
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:362)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:377)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:610)
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:352)
... 22 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] [http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException] [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :ambari-admin
解决方法：修改 /XXXX/ambari-admin/src/main/resources/ui/admin-web/package.json 的bower 版本将1.3.8 改为 1.8.8 问题解决
/ambari-admin/src/main/resources/ui/admin-web/package.json
@@ -3,7 +3,7 @@
"version": "0.0.0",
"dependencies": {},
"devDependencies": {
- "bower": "1.3.8",
+ "bower": "1.8.8",
"gulp": "^3.8.8",
"gulp-add-src": "^0.2.0",
"gulp-autoprefixer": "0.0.7",
问题2：Too many files with unapproved license
解决办法：
由于许可license没有造成的，需要再mvn命令中增加如下内容：
-Drat.skip=true
问题3：Unexpected character 0x0 in identifier编译ambari-utility，报错信息：
说明：
出错的Java文件编码和CheckStyle设置的编码不同。CheckStyle里设置的编码是UTF-8
解决办法：
将“._xxx.java”文件直接删除即可。
(注：其实有很多类似这样的文件报错，将类似这样的文件删除掉即可)
问题4： Download HBase超时
解决办法：
将./ambari-metrics/pom.xml的
<hbase.tar>http://dev.hortonworks.com.s3.amazonaws.com/HDP/centos7/3.x/BUILDS/3.0.0.0-1634/tars/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz</hbase.tar>
替换为本地路径
<hbase.tar>file:///tmp/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz</hbase.tar>
问题5：Unable to build the RPM
Unable to build the RPM: Error while executing process. Cannot run program “rpmbuild”
解决办法：
rpm -qa | grep rpm-build #检测是否已安装rpmbuild
yum install rpm-build #如果没安装则手动安装
问题6. buildNumber : unbound variable
vim /usr/sbin/ambari-server
将${buildNumber}这行换成 HASH="${VERSION}"
该文件源码位置位于：./ambari-server/sbin/ambari-server，修改后，在进行编译，可以一劳永逸。
问题7：
[INFO] Ambari Admin View .................................. FAILURE
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (Bower install) on project ambari-admin: Command execution failed.
单独编译报错：
ECMDERR Failed to execute "git ls-remote --tags --heads https://github.com/twbs/bootstrap.git", exit code of #128
Additional error details:fatal: unable to access 'https://github.com/twbs/bootstrap.git/': Peer reports incompatible or unsupported protocol version.
解决：个人将ambari-admin(包括子目录ambari-admin/src/main/resources/ui/admin-web)里面生成的node、node_modules、package-lock.json删除，保持干净的源码环境，再单独编译ambari-admin
#cd ambari-admin/src/main/resources/ui/admin-web目录下，编辑 .bowerrc ，修改后的内容如下：
{
"directory": "app/bower_components",
"allow_root": true
}
在上述目录下，安装npm依赖包，全局安装gulp、bower
npm install -g bower
npm install -g gulp
安装bower的依赖包(个人在ambari-admin/src/main/resources/ui/admin-web/下执行)
bower install
(上述错误原因在于bower install命令失败，个人将之前成功编译的bower_components目录(ambari-admin/src/main/resources/ui/admin-web/app目录下)拷贝到ambari-release-2.6.1/ambari-admin/src/main/resources/ui/admin-web/app目录下，再次运行成功)
安装 gulp-webserver
npm install gulp-webserver --save-dev
运行命令mvn install -Drat.skip=true -Preplaceurl -X重新单独编译ambari-admin模块(此命令在ambari-admin目录下运行)
