azkaban集群部署

随着平台上的任务越来越多,且部分任务相互依赖,目前使用的crontab定时调度已经无法满足使用需求.急需一个任务依赖调度系统,笔者以前使用过oozie,但是oozie配置过多,而且维护繁琐,属于重量级系统.所以最终我选择使用azkaban.网上关于oozie和azkaban比较的文章很多,有兴趣可以自行百度.接下来说说azkaban的部署.

一.部署模式

  • solo-server模式  (使用内置h2存储元数据)
  • two-server模式   (1个webServer,1个execServer在同一服务器上,使用mysql存储元数据)
  • multiple-executor模式   (1个webServer,多个execServer分布在不同服务上,使用mysql存储元数据)
    本文选择第三种模式,即一台机器安装webServer服务,多台机器安装execServer.这种模式是使用最广泛的.

二.部署准备

  1. 三台服务器
    192.168.0.10 --azkaban-web-server
192.168.0.11 --azkaban-exec-server
192.168.0.12 --azkaban-exec-server
  2. 默认有一个mysql服务器,首先创建azkaban库,azkaban用户密码,并赋予远程连接.
    1 mysql> CREATE DATABASE azkaban;
2 mysql> CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';
3 mysql> CREATE USER 'azkaban'@'localhost' IDENTIFIED BY 'azkaban';
4 mysql> grant all privileges on azkaban.* to 'azkaban'@'%' identified by 'azkaban';
5 mysql> grant all privileges on azkaban.* to 'azkaban'@'localhost' identified by 'azkaban';
6 mysql> flush privileges;
    View Code
  3. 下载最新azkaban源文件(azkaban-3.47.0.tar.gz),上传到192.168.0.10:/opt/下

三.开始部署

  1. 解压缩文件azkaban-3.47.0.tar.gz 
  2. 进入目录执行下面代码编译源文件
    ./gradlew distTar
  3. 将下面3个文件解压缩,文件如下:
    /opt/azkaban-3.47.0/azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
/opt/azkaban-3.47.0/azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz
/opt/azkaban-3.47.0/azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT.tar.gz
  4. 导入建表语句
    mysql -uazkaban -p
> SOURCE /opt/azkaban-3.47.0/azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql;
  5. 构建Azkaban-web-server
    mv /opt/azkaban-3.47.0/azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT /opt/Azkaban-web-server
cp -r /opt/azkaban-3.47.0/azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT/conf /opt/Azkaban-web-server/conf
  6. 修改 /opt/Azkaban-web-server/conf/azkaban.properties 文件
     1 # Azkaban Personalization Settings
 2 azkaban.name=nyhadoop
 3 azkaban.label=My Local Azkaban
 4 azkaban.color=#FF3601
 5 azkaban.default.servlet.path=/index
 6 web.resource.dir=web/
 7 default.timezone.id=Asia/Shanghai
 8 # Azkaban UserManager class
 9 user.manager.class=azkaban.user.XmlUserManager
10 user.manager.xml.file=conf/azkaban-users.xml
11 # Loader for projects
12 executor.global.properties=conf/global.properties
13 azkaban.project.dir=projects
14 
15 database.type=mysql
16 mysql.port=3306
17 mysql.host=10.46.67.33
18 mysql.database=azkaban
19 mysql.user=azkaban
20 mysql.password=azkaban
21 mysql.numconnections=100
22 
23 # Velocity dev mode
24 velocity.dev.mode=false
25 # Azkaban Jetty server properties.
26 jetty.ssl.port=8443
27 # jetty.use.ssl=false
28 jetty.port=8081
29 jetty.keystore=keystore
30 jetty.password=123456
31 jetty.keypassword=123456
32 jetty.truststore=keystore
33 jetty.trustpassword=123456
34 jetty.maxThreads=25
35 # Azkaban Executor settings
36 executor.port=12321
37 # mail settings
38 mail.sender=
39 mail.host=
40 # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
41 # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
42 # when this parameters set then these parameters are used to generate email links. 
43 # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
44 # azkaban.webserver.external_hostname=myazkabanhost.com
45 # azkaban.webserver.external_ssl_port=443
46 # azkaban.webserver.external_port=8081
47 job.failure.email=
48 job.success.email=
49 lockdown.create.projects=false
50 cache.directory=cache
51 # JMX stats
52 jetty.connector.stats=true
53 executor.connector.stats=true
54 # Azkaban plugin settings
55 azkaban.jobtype.plugin.dir=plugins/jobtypes
56 azkaban.use.multiple.executors=true
57 azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
58 azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
59 azkaban.executorselector.comparator.Memory=1
60 azkaban.executorselector.comparator.LastDispatched=1
61 azkaban.executorselector.comparator.CpuUsage=1
    View Code
  7. 配置jetty ssl
    keytool -keystore keystore -alias jetty -genkey -keyalg RSA
Enter keystore password:  输入密码
Re-enter new password: 再次输入密码
What is your first and last name?
  [Unknown]:  直接回车
What is the name of your organizational unit?
  [Unknown]:  直接回车
What is the name of your organization?
  [Unknown]:  直接回车
What is the name of your City or Locality?
  [Unknown]:  直接回车
What is the name of your State or Province?
  [Unknown]:  直接回车
What is the two-letter country code for this unit?
  [Unknown]:  直接回车
Is CN=YY, OU=YY, O=YY, L=shanghai, ST=shanghai, C=CN correct?
  [no]:  y
    View Code
  8. 将生成的keystore文件 移动到/opt/Azkaban-web-server/目录下

  9. 构建Azkaban-exec-server(192.168.0.11和192.168.0.12都有)
    scp /opt/azkaban-3.47.0/azkaban-exec-server/build/distributions/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz 192.168.0.11:/opt/
tar -zxvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
mv azkaban-exec-server-0.1.0-SNAPSHOT Azkaban-exec-server
  10. /opt/Azkaban-exec-server目录下创建目录和文件conf/azkaban.properties文件
    # Azkaban Personalization Settings
default.timezone.id=Asia/Shanghai
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects

database.type=mysql
mysql.port=3306
mysql.host=10.46.67.33
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes
    View Code
  11. 分别在/opt/Azkaban-web-server/conf和/opt/Azkaban-exec-server/conf目录下创建log4j.properties文件
    log4j.rootLogger=DEBUG,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
    View Code
  12. 分别在/opt/Azkaban-web-server/和/opt/Azkaban-exec-server/目录下创建plugins/jobtypes/commonprivate.properties目录和文件
    azkaban.native.lib=false
execute.as.user=false
    View Code

     

四.运行

  1. 192.168.0.11和192.168.0.12下  /opt/Azkaban-exec-server目录下执行
    1 bin/start-exec.sh
2 curl http://${ip}:12321/executor?action=activate
  2. 192.168.1.10下 /opt/Azkaban-web-server目录下执行
    bin/start-web.sh

     

  3. 浏览器地址 https://192.168.1.10:8443查看服务界面

 

posted @ 2018-05-26 15:55  抠你的鼻屎去  阅读(2485)  评论(1)  编辑  收藏  举报