大数据基础之Mesos+Marathon+Docker部署nginx

一 安装

安装docker：https://www.cnblogs.com/barneywill/p/10343091.html
安装mesos：https://www.cnblogs.com/barneywill/p/10324348.html
安装marathon：https://www.cnblogs.com/barneywill/p/10366688.html

二 部署nginx

使用docker官方nginx image，如果想用私有docker registry，可以参考：https://www.cnblogs.com/barneywill/p/10367341.html

打开marathon页面，使用docker image创建app

 

1 开启mesos对docker的支持

# echo 'docker,mesos' > /ec/mesos-slave/containerizers
# service mesos-slave restart

2 创建app之后nginx正常运行，但是增加端口映射之后（即--network bridge -p 80:80），app一直卡住无法运行，查看mesos master日志如下：

# tail -f /var/log/mesos/mesos-master.log.INFO
I0216 01:01:35.815131 25339 master.cpp:9533] Sending offers [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O171, e11dfbd1-316e-4635-9267-8d5089f5ecfa-O172, e11dfbd1-316e-4635-9267-8d5089f5ecfa-O173, e11dfbd1-316e-4635-9267-8d5089f5ecfa-O174 ] to framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.821087 25338 master.cpp:5886] Processing DECLINE call for offers: [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O171 ] for framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.821200 25338 master.cpp:11531] Removing offer e11dfbd1-316e-4635-9267-8d5089f5ecfa-O171
I0216 01:01:35.821494 25335 master.cpp:5886] Processing DECLINE call for offers: [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O173 ] for framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.821557 25335 master.cpp:11531] Removing offer e11dfbd1-316e-4635-9267-8d5089f5ecfa-O173
I0216 01:01:35.821578 25335 master.cpp:5886] Processing DECLINE call for offers: [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O174 ] for framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.821614 25335 master.cpp:11531] Removing offer e11dfbd1-316e-4635-9267-8d5089f5ecfa-O174
I0216 01:01:35.823659 25334 master.cpp:5886] Processing DECLINE call for offers: [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O172 ] for framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.823747 25334 master.cpp:11531] Removing offer e11dfbd1-316e-4635-9267-8d5089f5ecfa-O172

可见mesos-master一直向marathon提供resource offer，但是一直被拒绝，再仔细查看日志发现

I0216 01:01:12.527853 25338 master.cpp:7503] Re-registered agent 0548c9c0-dd5f-4b3a-8aca-4668f8aacb69-S9 at slave(1)@192.168.0.1:5051 (server1.bj) with cpus:8; mem:14861; disk:35829; ports:[31000-32000]

发现mesos-slave的资源范围有问题，ports:[31000-32000]，nginx是需要80端口的，所以一直被拒绝，关于resources官方文档描述如下：

--resources=VALUE

Total consumable resources per agent. Can be provided in JSON format or as a semicolon-delimited list of key:value pairs, with the role optionally specified.
As a key:value list: name(role):value;name:value...
To use JSON, pass a JSON-formatted string or use --resources=filepath to specify the resources via a file containing a JSON-formatted string. 'filepath' can be of the form file:///path/to/file or /path/to/file.
Example JSON:
[
{
"name": "cpus",
"type": "SCALAR",
"scalar": {
"value": 24
}
},
{
"name": "mem",
"type": "SCALAR",
"scalar": {
"value": 24576
}
}
]

参考：http://mesos.apache.org/documentation/latest/configuration/agent/

 

修改端口范围

# echo 'ports(*):[80-80000]' > /etc/mesos-slave/resources
# service mesos-slave restart

之前的错误没了，但是又出现新的错

Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: E0216 01:37:17.314812 21733 slave.cpp:7311] EXIT with status 1: Failed to perform recovery: Incompatible agent info detected.
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: resources {
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: name: "ports"
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: type: RANGES
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: ranges {
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: range {
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: begin: 31000
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: end: 32000
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: }
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: }
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: }

这个错是因为修改resources之后需要充值agent并重启，详见：https://www.cnblogs.com/barneywill/p/10386239.html

# rm -f /var/lib/mesos/meta/slaves/latest
# service mesos-slave start

3 如果需要制定在哪台机器上执行，可以设置constraints

只允许在一台机器上执行

"constraints": [
[
"hostname",
"CLUSTER",
"server2.bj"
]
],

允许在两台机器上执行，但是每台机器只能启动一个

"constraints": [
[
"hostname",
"UNIQUE"
],
[
"hostname",
"LIKE",
"server[1-2].bj"
]
],

参考：https://mesosphere.github.io/marathon/docs/constraints.html

 

部署完成后

 

还有一点，注意到上边截图中的status是waiting，因为我做了一个restart操作，但是我每个app都指定只部署在一台服务器上，这时其实会一直卡在waiting的，因为marathon在restart时并不会将app先stop再start，而是不断等待mesos-master的resource offer，只有当有可用资源启动新的app之后才会把旧的app停掉，而我这里的app会占用80端口，所以是一个死循环，这时只能手工从docker上去stop container；

 

这样就ok了