oozie:hadoop中的工作流引擎
一、maven安装
1) 从网站上下载Linux版本的Maven安装包;
apache-maven-3.0.5-bin.tar.gz下载
具体解压目录在 /home/maven/
========================================
2) 解压apache-maven-3.0.5-bin.tar.gz进行安装;
tar -xzvf apache-maven-3.0.5-bin.tar.gz
3) 设置maven环境变量;
vim /etc/profile或者vim /etc/environment
在末尾处增加如下内容:
#maven envirment export M2_HOME=/home/maven export M2=$M2_HOME/bin export MAVEN_OPTS="-Xms256m -Xmx512m" export PATH=$M2:$PATH
4) 使配置文件生效
source /etc/profile
5)验证maven是否安装成功。
root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin# mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800)
Maven home: /home/maven
Java version: 1.6.0_43, vendor: Sun Microsystems Inc.
Java home: /usr/lib/jvm/jdk1.6.0_43/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-24-server", arch: "amd64", family: "unix"
二、oozie
1、Build
bin/mkdistro.sh -DskipTests
成功build后,会在目录里生成一个distribution.
hadoop安装时创建的hadoop组和hadoop用户对oozie目录是没有权限,所以最好一开始创建oozie组和oozie用户,同时赋予权限,不然会报错.偷懒用了root用户。
PS:后来发现,可以单独拷贝/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2下的oozie-3.3.2-distro.tar.gz包到用户自定义目录进行解压使用,除此之外的所有文件可以全部删除。
2、安装
根据官方文档在distibution的目录下(/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2)创建libext,因为装的hadoop版本是1.0.3,所以拷贝$HADOOP_HOME/lib下的所有jar包到libext下,同时拷贝ext-2.2.zip到该目录下。
还要拷贝$HADOOP_HOME下的所有jar包到libext.
因为oozie默认并不支持1.0.3版,所以对/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin文件夹下的addtowar.sh文件进行修改:
elif [ "${version}" = "0.20.200" ]; then #List is separated by ":" hadoopJars="hadoop-core*.jar:jackson-core-asl-*.jar:jackson-mapper-asl-*.jar:commons-configuration-*.jar"
后边增加如下内容: elif [ "${version}" = "1.0.3" ]; then #List is separated by ":" hadoopJars="hadoop-core*.jar:jackson-core-asl-*.jar:jackson-mapper-asl-*.jar:commons-configuration-*.jar"
运行oozie-setup.sh创建oozie.war
root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie-setup.sh -hadoop 1.0.3 /home/hadoop -extjs /home/oozie/ext-2.2.zip
执行成功后会出现如下界面:
拷贝oozie.war到tomcat部署目录下
root@ubuntu:/home/oozie/webapp/src/main/webapp# cp /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie.war ./
修改配置文件:
<property> <name>oozie.service.JPAService.create.db.schema</name> <value>true</value> <description> Creates Oozie DB. If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP. If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up. </description> </property>
创建Oozie DB
cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin ./ooziedb.sh create -sqlfile oozie.sql -run
结果
DONE Check DB schema does not exist DONE Check OOZIE_SYS table does not exist DONE Create SQL schema DONE DONE Create OOZIE_SYS table DONE Oozie DB has been created for Oozie version '3.3.1'
运行oozie
cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin ./oozie-start.sh
开启服务:
bin/oozie-run.sh
运行结果
root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie-run.sh WARN: Use of this script is deprecated; use 'oozied.sh run' instead Setting OOZIE_HOME: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2 Setting OOZIE_CONFIG: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf Sourcing: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-env.sh setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" Setting OOZIE_CONFIG_FILE: oozie-site.xml Setting OOZIE_DATA: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data Setting OOZIE_LOG: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs Setting OOZIE_LOG4J_FILE: oozie-log4j.properties Setting OOZIE_LOG4J_RELOAD: 10 Setting OOZIE_HTTP_HOSTNAME: dm4 Setting OOZIE_HTTP_PORT: 11000 Setting OOZIE_ADMIN_PORT: 11001 Setting OOZIE_HTTPS_PORT: 11443 Setting OOZIE_BASE_URL: http://dm4:11000/oozie Setting CATALINA_BASE: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server Setting OOZIE_HTTPS_KEYSTORE_FILE: /root/.keystore Setting OOZIE_HTTPS_KEYSTORE_PASS: password Setting CATALINA_OUT: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/catalina.out Setting CATALINA_PID: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid Using CATALINA_OPTS: -Xmx1024m -Dderby.stream.error.file=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/derby.log Adding to CATALINA_OPTS: -Doozie.home.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2 -Doozie.config.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf -Doozie.log.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs -Doozie.data.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=dm4 -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://dm4:11000/oozie -Doozie.https.keystore.file=/root/.keystore -Doozie.https.keystore.pass=password -Djava.library.path= Using CATALINA_BASE: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server Using CATALINA_HOME: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server Using CATALINA_TMPDIR: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp Using JRE_HOME: /usr/lib/jvm/jdk1.6.0_43/jre Using CLASSPATH: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/bin/bootstrap.jar Using CATALINA_PID: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid Jun 2, 2013 3:06:46 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: Jun 2, 2013 3:06:46 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-11000 Jun 2, 2013 3:06:46 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 338 ms Jun 2, 2013 3:06:46 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Jun 2, 2013 3:06:46 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.36 Jun 2, 2013 3:06:46 PM org.apache.catalina.startup.HostConfig deployDescriptor INFO: Deploying configuration descriptor oozie.xml SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib/slf4j-simple-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. log4j:WARN No appenders could be found for logger (org.apache.jasper.compiler.JspRuntimeContext). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Jun 2, 2013 3:06:50 PM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory ROOT Jun 2, 2013 3:06:50 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-11000 Jun 2, 2013 3:06:50 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 3976 ms
界面:
和例子界面类似,没有内容。
案例:
将/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2中的oozie-examples.tar.gz解压,出现examples文件夹
修改
/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/examples/map-reduce/job.properties文件为如下形式:
nameNode=hdfs://211.80.198.148:9000
jobTracker=211.80.198.148:9001
queueName=default
examplesRoot=examples
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
outputDir=map-reduce
将文件夹上传到集群
root@dm4:/home/hadoop# bin/hadoop fs -copyFromLocal /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/examples /user/root
运行命令
root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
运行结果:
job: 0000000-130602150647008-oozie-root-W
不上传会出现如下错误:
root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run Error: E0504 : E0504: App directory [hdfs://211.80.198.148:9000/user/root/examples/apps/map-reduce] does not exist
结果
PS:中间出现的一些错误
ERROR: Oozie could not be started REASON: java.lang.NoClassDefFoundError: org/apache/hadoop/util/ReflectionUtils
将hadoop的jar包(主目录下和lib目录下)拷贝到如下位置
/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib
之后,运行正常。
参考:
1、http://angelosun.iteye.com/blog/1835881
2、http://practicalcloudcomputing.com/post/26337621577/installing-and-running-apache-oozie-3-2-x-and-possibly
3、http://wsysisibeibei.blog.163.com/blog/static/16536806120126233155505/
转载请保留:http://www.cnblogs.com/Deron/archive/2013/06/02/3111178.html