azkaban的安装部署
一、所需环境
1,JDK
2,HADOOP
4,mysql
3,HIVE
二、安装azkaban
1,安装git命令: yum install git
2,下载azkaban源码:git clone https://github.com/azkaban/azkaban.git
3,安装到 azkaban-tar目录下
4,更改配置文件:
vi 安装目录/azkaban-tar/azkaban/git clone https://github.com/azkaban/azkaban.git
将final boolean isExecuteAsUser = this.sysProps.getBoolean(EXECUTE_AS_USER, true);
true修改false
5,在azkaban目录下执行:./gradlew distTar
此步骤会生成5个压缩包,分别放在下图目录中:
azkaban-db-3.37.0-10-gdd67b02.tar.gz
azkaban-exec-server-3.37.0-10-gdd67b02.tar.gz
azkaban-hadoop-security-plugin-3.37.0-10-gdd67b02.tar.gz
azkaban-solo-server-3.37.0-10-gdd67b02.tar.gz
azkaban-web-server-3.37.0-10-gdd67b02.tar.gz
6,在当前azkaban文件夹下,将该五个压缩包copy出来,放到 安装目录/azkaban-tar文件夹中。
cp azkaban-*/build/distributions/*.tar.gz 安装目录/azkaban-tar
7,将这5个压缩包解压,然后重命令,将这5个压缩包放在另外的 安装目录/azkaban下
8,登录mysql mysql –u root –p
9,数据库中执行:create database azkaban;
10,数据库中执行:use azkaban;
11, 将azkaban sql表结构导入mysql,数据库中执行:source 安装目录/azkaban/azkaban-db-3.46.0-2-gbb24a72/create-all-sql-3.46.0-2-gbb24a72.sql
如果执行失败,则将sql语句复制出来执行
12,部署azkaban-web-server
将 安装目录/azkaban/azkaban-solo-server/ 下的conf目录拷贝到 安装目录/azkaban/ azkaban-web-server/conf
命令:cp -r /安装目录/azkaban/azkaban-solo-server/conf /安装目录/azkaban/azkaban-web-server/conf
配置:azkaban.properties
如下(标红的需要更改,其他的可以对比一下):
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/usr/local/azkaban/azkaban-web-server/web
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/usr/local/azkaban/azkaban-web-server/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/usr/local/azkaban/azkaban-web-server/conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=root
mysql.password=root
mysql.numconnections=100
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
azkaban.native.lib=false
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes
13,在conf目录下新增配置文件log4j.properties,内容如下
log4j.rootLogger=INFO,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
14,部署 azkaban-exec-server
将 /安装目录/azkaban/ azkaban-solo-server下的conf文件拷贝到 /安装文件/azkaban/ azkaban-exec-server
命令:cp –r /安装目录/azkaban/azkaban-solo-server/conf /安装目录/azkaban/azkaban-exec-server/conf
配置azkaban.properties:(标红的需要更改,其他的可以对比一下)
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/usr/local/azkaban/azkaban-exec-server/conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=root
mysql.password=root
mysql.numconnections=100
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes
azkaban.native.lib=false
在conf目录下,新建配置log4j.properties文件,内容如下:
log4j.rootLogger=INFO,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
15,在hadoop中的/share/Hadoop/common 路径下找到hadoop-common-2.6.4.jar
在jdk中的/jdk1.8.0_91/db/lib/路径下找到derby.jar
将这俩个jar包拷贝到azkaban-exec-server和azkaban-web-server
安装目录下的extlib目录,没有则创建一个,找不到就下载
16,启动
进入azkaban-exec-server目录,执行start-exec.sh
进入azkaban-web-server目录,执行start-web.sh
查看进程:jps
成功执行
通过http://localhost:8081可访问到web-ui
16,此时的web界面:
需要更改配置文件解决:
将上图中的web目录的路径加到azkaban.properties配置文件中,如下图
再进入web目录可以看到界面变得美观
17, 用户名密码在 azkaban-web-server/conf/azkaban-users.xml 中配置