大数据基础之Chronos

官方：https://mesos.github.io/chronos/

 

mesos集群中替换crontab

 

Chronos

A fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedules

简介

Chronos is a replacement for cron. It is a distributed and fault-tolerant scheduler that runs on top of Apache Mesos that can be used for job orchestration. It supports custom Mesos executors as well as the default command executor. Thus by default, Chronos executes sh (on most systems bash) scripts.

Chronos can be used to interact with systems such as Hadoop (incl. EMR), even if the Mesos slaves on which execution happens do not have Hadoop installed. Chronos is also natively able to schedule jobs that run inside Docker containers.

Chronos has a number of advantages over regular cron. It allows you to schedule your jobs using ISO8601 repeating interval notation, which enables more flexibility in job scheduling. Chronos also supports the definition of jobs triggered by the completion of other jobs. It supports arbitrarily long dependency chains.

工作过程

Internally, the Chronos scheduler main loop is quite simple. The pattern is as follows:

1. Chronos reads all job state from the state store (ZooKeeper)
2. Jobs are registered within the scheduler and loaded into the job graph for tracking dependencies.
3. Jobs are separated into a list of those which should be run at the current time (based on the clock of the host machine), and those which should not.
4. Jobs in the list of jobs to run are queued, and will be launched as soon as a sufficient offer becomes available.
5. Chronos will sleep until the next job is scheduled to run, and begin again from step 1.

 

chronos通过zk来维护任务状态，任务会随机分配到一个mesos slave节点运行

 

安装

前提是已经部署好zk和mesos

$ docker run --net=host -e PORT0=8080 -e PORT1=8081 mesosphere/chronos:v3.0.0 --zk_hosts $zk_ip:2181 --master zk://$zk_ip:2181/mesos

其中第一个端口是http端口

使用

访问 http://localhost:8080

添加任务

只需要配置

• 任务名称
• 时间间隔
• 要执行的shell命令/脚本

其中shell命令也可以替换为启动docker容器，如下图：

时间间隔格式示例：

R/2019-06-19T14:40:00.000+08:00/PT5M

以上配置为在北京时间下每5分钟执行

 

任务执行之后可以在mesos上找到任务执行记录

 

任务定义以及执行状态都可以在zk上找到

[zk: localhost:2181(CONNECTED) 7] get /chronos/state/state/J_ $job_name

 

只需要完成一个通用的shell脚本，这个shell脚本根据参数首先将一个hdfs目录（任务目录，包括脚本和配置等）下载到本地，然后进入目录执行参数中的命令，即可实现分布式任务调度；

$ do_job.sh $hdfs_path $cmd

 

参考：https://hub.docker.com/r/mesosphere/chronos/