Azkaban安装-3.72.0

安装
1、下载tar包
https://github.com/azkaban/azkaban/archive/3.72.0.tar.gz
2、解压
tar -xvf azkaban-3.72.0.tar.gz -C /opt/dsfsadf

3、安装依赖
yum install -y gcc-c++ git

4、进入解压家目录，编译执行

-x test 表示跳过测试

./gradlew build installDist -x test

若编译时失败，重新编译执行：

./gradlew clean build installDist -x test

5、配置MySQL
CREATE DATABASE azkaban;
use azkaban;

CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';
GRANT ALL ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

source /opt/azkaban-3.72.0/azkaban-db/build/sql/create-all-sql-0.1.0-SNAPSHOT.sql；

6、copy 到指定目录

解压

[root@localhost azkaban-3.73.1]# cp -a azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz ../azkaban
[root@localhost azkaban-3.73.1]# cp -a azkaban-exec-server/build/distributions/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz ../azkaban
[root@localhost azkaban-3.73.1]# cp -a azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz ../azkaban

重命名

[root@localhost azkaban]# mv azkaban-exec-server-0.1.0-SNAPSHOT/ azkaban-exec-server
[root@localhost azkaban]# mv azkaban-web-server-0.1.0-SNAPSHOT/ azkaban-web-server
[root@localhost azkaban]# mv azkaban-solo-server-0.1.0-SNAPSHOT/ azkaban-solo-server

7、配置web

进入azkaban web 目录下

cd azkaban-web-server

生成keystore 注意开始和结束的时候输入的密码，其他按提示输入即可 我这里都是azkaban

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

修改conf/azkaban.properties

Azkaban Personalization Settings

azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai

Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/root/azkaban/azkaban-web-server/conf/azkaban-users.xml

Loader for projects

executor.global.properties=/root/azkaban/azkaban-web-server/conf/global.properties
azkaban.project.dir=projects

Velocity dev mode

velocity.dev.mode=false

Azkaban Jetty server properties.

jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081

Azkaban Executor settings

mail settings

mail.sender=
mail.host=

User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.

enduser -> myazkabanhost:443 -> proxy -> localhost:8081

when this parameters set then these parameters are used to generate email links.

if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.

azkaban.webserver.external_hostname=myazkabanhost.com

azkaban.webserver.external_ssl_port=443

azkaban.webserver.external_port=8081

job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache

JMX stats

jetty.connector.stats=true
executor.connector.stats=true

Azkaban mysql settings by default. Users should configure their own username and password.

database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

Multiple Executor

azkaban.use.multiple.executors=true

azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus

azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

web端这里：

azkaban.jobtype.plugin.dir=/root/azkaban/azkaban-web-server/plugins/jobtypes

8、配置exec

进入azkaban web 目录下

cd azkaban-exec-server

Azkaban Personalization Settings

azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai

Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/root/azkaban/azkaban-exec-server/conf/azkaban-users.xml

Loader for projects

executor.global.properties=/root/azkaban/azkaban-exec-server/conf/global.properties
azkaban.project.dir=projects

Velocity dev mode

velocity.dev.mode=false

Azkaban Jetty server properties.

jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081

Where the Azkaban web server is located

azkaban.webserver.url=http://localhost:8081

mail settings

mail.sender=
mail.host=

User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.

enduser -> myazkabanhost:443 -> proxy -> localhost:8081

when this parameters set then these parameters are used to generate email links.

if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.

azkaban.webserver.external_hostname=myazkabanhost.com

azkaban.webserver.external_ssl_port=443

azkaban.webserver.external_port=8081

job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache

JMX stats

jetty.connector.stats=true
executor.connector.stats=true

Azkaban plugin settings

azkaban.jobtype.plugin.dir=/root/azkaban/azkaban-exec-server/plugins/jobtypes

Azkaban mysql settings by default. Users should configure their own username and password.

database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

Azkaban Executor settings

executor.maxThreads=50
executor.flow.threads=30

9、启动

查询mysql保持配置一致

select * from executors;

执行exec

./bin/start-exec.sh

执行下面命令激活状态（不可直接修改数据库，jvm没有修改）

curl http://localhost:84654/executor?action=activate
curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo

查看状态

curl http://localhost:59583/executor?action=getStatus

执行web

./bin/start-web.sh

10、查看页面
http://192.168.174.132:8081/index
11、测试
type = command
command = echo 'all flow end'

问题
问题1：
查看log日志发现
azkaban执行错误 Job failed, Cannot request memory (Xms 0 kb, Xmx 0 kb) from system for job
azkaban默认需要3G的内存，剩余内存不足则会报异常。
解决方案：
配置文件，设置（或者增加服务器内存）
azkaban-web-server-2.7.0/plugins/jobtypes/commonprivate.properties
memCheck.enabled=false

问题2：
ERROR [FlowTriggerScheduler] [Azkaban] Unable to get scheduled flow triggers
删掉executor选择器的过滤条件：MinimumFreeMemory

问题3：（遵循启动顺序就好了）
启动顺序不可错，不能直接修改mysql数据库，要通过命令传过去
因为数据库修改了，但是jvm内存的数据没有修改
update executors set active = 1