Mesos源码分析(4) Mesos Master的启动之三

3. ModuleManager::load(flags.modules.get())如果有参数--modules或者--modules_dir=dirpath，则会将路径中的so文件load进来

 

代码中加载模块的代码如下

 

 

对应的命令行参数如下：

 

 

 

都可以写什么Module呢？

 

首先是Allocator

 

默认是内置的Hierarchical Dominant Resource Fairness allocator

 

要写一个自己的Allocator:

• 通过--modules加载so
• 通过--allocator参数指定

 

当然很少有人会重写这个Allocator，因为这是Mesos的精髓所在，但是可以对默认的Alllocator进行加强的。

 

参考文档http://mesos.apache.org/documentation/latest/allocation-module/

 

Mesos Allocation Modules

The logic that the Mesos master uses to determine which frameworks to make resource offers to is encapsulated in the master's allocator module. The allocator is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, priority, etc., or tune the default hierarchical Dominant Resource Fairness algorithm (see the DRF paper).

To use a custom allocator in Mesos, one must:

• Implement the Allocator interface as defined in mesos/allocator/allocator.hpp,
• Wrap the allocator implementation in a module and load it in the Mesos master.

Writing a custom allocator

Allocator modules are implemented in C++, the same language in which Mesos is written. They must subclass the Allocator interface defined in mesos/allocator/allocator.hpp. However, your implementation can be a C++ proxy, which delegates calls to an actual allocator written in a language of your choice.

The default allocator is HierarchicalDRFAllocatorProcess, which lives in$MESOS_HOME/src/master/allocator/mesos/hierarchical.hpp. Like most Mesos components, it is actor-based, which means all interface methods are non-blocking and return immediately after putting the corresponding action into the actor's queue. If you would like to design your custom allocator in a similar manner, subclass MesosAllocatorProcess from$MESOS_HOME/src/master/allocator/mesos/allocator.hpp and wrap your actor-based allocator inMesosAllocator. This dispatches calls to the underlying actor and controls its lifetime. You can refer to HierarchicalDRFAllocatorProcess as a starting place if you choose to write your own actor-based allocation module.

Additionally, the built-in hierarchical allocator can be extended without the need to reimplement the entirety of the allocation logic. This is possible through the use of the Sorter abstraction. Sorters define the order in which hierarchy layers (e.g. roles or frameworks) should be offered resources by taking "client" objects and some information about those clients and returning an ordered list of clients.

Sorters are implemented in C++ and inherit the Sorter class defined in$MESOS_HOME/src/master/allocator/sorter/sorter.hpp. The default sorter is DRFSorter, which implements fair sharing and can be found in$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp. This sorter is capable of expressing priorities by specifying weights in Sorter::add(). Each client's share is divided by its weight. For example, a role that has a weight of 2 will be offered twice as many resources as a role with weight 1.

Wiring up a custom allocator

Once a custom allocator has been written, the next step is to override the built-in implementation with your own. This process consists of several steps:

• Wrap your allocator in a Mesos allocator module,
• Load this module in Mesos master.

An allocator module is a factory function and a module description, as defined inmesos/module/allocator.hpp. Assuming the allocation logic is implemented by theExternalAllocator class declared in external_allocator.hpp, the following snippet describes the implementation of an allocator module named ExternalAllocatorModule:

#include <mesos/allocator/allocator.hpp>

#include <mesos/module/allocator.hpp>

#include <stout/try.hpp>

 

#include "external_allocator.hpp"

 

using namespace mesos;

using mesos::allocator::Allocator;

using mesos::internal::master::allocator::HierarchicalDRFAllocator;

 

static Allocator* createExternalAllocator(const Parameters& parameters)

{

  Try<Allocator*> allocator = ExternalAllocator::create();

  if (allocator.isError()) {

    return nullptr;

  }

 

  return allocator.get();

}

 

// Declares an ExternalAllocator module named 'ExternalAllocatorModule'.

mesos::modules::Module<Allocator> ExternalAllocatorModule(

    MESOS_MODULE_API_VERSION,

    MESOS_VERSION,

    "Mesos Contributor",

    "engineer@example.com",

    "External Allocator module.",

    nullptr,

    createExternalAllocator);

Refer to the Mesos Modules documentation for instructions on how to compile and load a module in Mesos master.

 

Hook

 

你可以写hook模块，讲代码插在很多关键的步骤，从而改写整个Executor或者Docker或者Task的启动的整个过程。

 

可以干预的hook的地方定义在mesos/hook.hpp中。

 

Class hook定义如下：

 

 

其中比较常用的是slavePrelaunchDockerHook，可以在Docker启动之前做一些事情，比如准备工作。

还有slaveRemoveExecutorHook，这个可以在executor结束的时候，做一些事情，比如清理工作。

 

要加载一个hook需要：

• 通过--modules加载so
• 通过--hooks加载hook

 

./bin/mesos-agent.sh --master=<IP>:<PORT> --modules="file://<path-to-modules-config>.json" --hooks=TestTaskHook

 

Isolator

 

可通过--isolation=VALUE指定，仅仅用于Mesos Containerizer。

在src/slave/containerizer/mesos/container.cpp里面

 

 

 

默认只会有cpu和memory

 

1. if (flags.isolation == "cgroups") {
2.     LOG(WARNING) << "The 'cgroups' isolation flag is deprecated, "
3.                  << "please update your flags to"
4.                  << " '--isolation=cgroups/cpu,cgroups/mem'.";
5.  
6.     flags_.isolation = "cgroups/cpu,cgroups/mem";
7.   }

 

然后是创建Mesos Containerizer的Launcher

 

1. // Create the launcher for the MesosContainerizer.
2.   Try<Launcher*> launcher = [&flags_]() -> Try<Launcher*> {
3. #ifdef __linux__
4.     if (flags_.launcher.isSome()) {
5.       // If the user has specified the launcher, use it.
6.       if (flags_.launcher.get() == "linux") {
7.         return LinuxLauncher::create(flags_);
8.       } else if (flags_.launcher.get() == "posix") {
9.         return PosixLauncher::create(flags_);
10.       } else {
11.         return Error(
12.             "Unknown or unsupported launcher: " + flags_.launcher.get());
13.       }
14.     }
15.  
16.     // Use Linux launcher if it is available, POSIX otherwise.
17.     return LinuxLauncher::available()
18.       ? LinuxLauncher::create(flags_)
19.       : PosixLauncher::create(flags_);
20.  
21. ……
22. return PosixLauncher::create(flags_); //默认是Posix

 

然后就是创建Isolator

 

1.   // Create the isolators.
2.   //
3.   // Currently, the order of the entries in the --isolation flag
4.   // specifies the ordering of the isolators. Specifically, the
5.   // `create` and `prepare` calls for each isolator are run serially
6.   // in the order in which they appear in the --isolation flag, while
7.   // the `cleanup` call is serialized in reverse order.
8.   //
9.   // It is the responsibility of each isolator to check its
10.   // dependency requirements (if any) during its `create`
11.   // execution. This means that if the operator specifies the
12.   // flags in the wrong order, it will produce an error during
13.   // isolator creation.
14.   //
15.   // NOTE: We ignore the placement of the filesystem isolator in
16.   // the --isolation flag and place it at the front of the isolator
17.   // list. This is a temporary hack until isolators are able to
18.   // express and validate their ordering requirements.
19.  
20.   const hashmap<string, lambda::function<Try<Isolator*>(const Flags&)>>
21.     creators = {
22.     // Filesystem isolators.
23. #ifndef __WINDOWS__
24.     {"filesystem/posix", &PosixFilesystemIsolatorProcess::create},
25. #else
26.     {"filesystem/windows", &WindowsFilesystemIsolatorProcess::create},
27. #endif // __WINDOWS__
28. #ifdef __linux__
29.     {"filesystem/linux", &LinuxFilesystemIsolatorProcess::create},
30.  
31.     // TODO(jieyu): Deprecate this in favor of using filesystem/linux.
32.     {"filesystem/shared", &SharedFilesystemIsolatorProcess::create},
33. #endif // __linux__
34.  
35.     // Runtime isolators.
36. #ifndef __WINDOWS__
37.     {"posix/cpu", &PosixCpuIsolatorProcess::create},
38.     {"posix/mem", &PosixMemIsolatorProcess::create},
39.  
40.     // "posix/disk" is deprecated in favor of the name "disk/du".
41.     {"posix/disk", &PosixDiskIsolatorProcess::create},
42.     {"disk/du", &PosixDiskIsolatorProcess::create},
43.  
44. #if ENABLE_XFS_DISK_ISOLATOR
45.     {"disk/xfs", &XfsDiskIsolatorProcess::create},
46. #endif
47. #else
48.     {"windows/cpu", &WindowsCpuIsolatorProcess::create},
49. #endif // __WINDOWS__
50. #ifdef __linux__
51.     {"cgroups/cpu", &CgroupsCpushareIsolatorProcess::create},
52.     {"cgroups/devices", &CgroupsDevicesIsolatorProcess::create},
53.     {"cgroups/mem", &CgroupsMemIsolatorProcess::create},
54.     {"cgroups/net_cls", &CgroupsNetClsIsolatorProcess::create},
55.     {"cgroups/perf_event", &CgroupsPerfEventIsolatorProcess::create},
56.     {"docker/runtime", &DockerRuntimeIsolatorProcess::create},
57.     {"docker/volume", &DockerVolumeIsolatorProcess::create},
58.  
59.     {"gpu/nvidia",
60.       [&nvidia] (const Flags& flags) -> Try<Isolator*> {
61.         if (!nvml::isAvailable()) {
62.           return Error("Cannot create the Nvidia GPU isolator:"
63.                        " NVML is not available");
64.         }
65.  
66.         CHECK_SOME(nvidia)
67.           << "Nvidia components should be set when NVML is available";
68.  
69.         return NvidiaGpuIsolatorProcess::create(flags, nvidia.get());
70.       }},
71.  
72.     {"namespaces/pid", &NamespacesPidIsolatorProcess::create},
73.     {"network/cni", &NetworkCniIsolatorProcess::create},
74. #endif // __linux__
75.     // NOTE: Network isolation is currently not supported on Windows builds.
76. #if !defined(__WINDOWS__) && defined(WITH_NETWORK_ISOLATOR)
77.     {"network/port_mapping", &PortMappingIsolatorProcess::create},
78. #endif
79.   };

 

所有的isolator会放在一个vector里面

 

1. vector<Owned<Isolator>> isolators;
2.  
3. foreach (const string& isolation, isolations) {
4.   Try<Isolator*> isolator = [&]() -> Try<Isolator*> {
5.     if (creators.contains(isolation)) {
6.       return creators.at(isolation)(flags_);
7.     } else if (ModuleManager::contains<Isolator>(isolation)) {
8.       return ModuleManager::create<Isolator>(isolation);
9.     }
10.     return Error("Unknown or unsupported isolator");
11.   }();
12.  
13.   if (isolator.isError()) {
14.     return Error("Failed to create isolator '" + isolation + "': " +
15.                  isolator.error());
16.   }
17.  
18.   // NOTE: The filesystem isolator must be the first isolator used
19.   // so that the runtime isolators can have a consistent view on the
20.   // prepared filesystem (e.g., any volume mounts are performed).
21.   if (strings::contains(isolation, "filesystem/")) {
22.     isolators.insert(isolators.begin(), Owned<Isolator>(isolator.get()));
23.   } else {
24.     isolators.push_back(Owned<Isolator>(isolator.get()));
25.   }
26. }

 

最终创建MesosContainerizer

 

1. return new MesosContainerizer(
2.       flags_,
3.       local,
4.       fetcher,
5.       Owned<ContainerLogger>(logger.get()),
6.       Owned<Launcher>(launcher.get()),
7.       provisioner.get(),
8.       isolators);

 

那isolator是如何起作用的呢？

 

当一个MesosContainerizer启动一个进程的时候，会调用如下的函数。

 

在src/slave/containerizer/mesos/containerizer.cpp中有

 

 

里面最重要的一步是fork一个进程

 

1. Try<pid_t> forked = launcher->fork(
2.     containerId,
3.     path::join(flags.launcher_dir, MESOS_CONTAINERIZER),
4.     argv,
5.     Subprocess::FD(STDIN_FILENO),
6.     (local ? Subprocess::FD(STDOUT_FILENO)
7.            : Subprocess::IO(subprocessInfo.out)),
8.     (local ? Subprocess::FD(STDERR_FILENO)
9.            : Subprocess::IO(subprocessInfo.err)),
10.     launchFlags,
11.     environment,
12.     namespaces); // 'namespaces' will be ignored by PosixLauncher.

 

最后还会调用isolate不断的检验这个进程是否使用适当的资源。

 

1. return isolate(containerId, pid)
2.   .then(defer(self(),
3.               &Self::fetch,
4.               containerId,
5.               executorInfo.command(),
6.               directory,
7.               user,
8.               slaveId))
9.   .then(defer(self(), &Self::exec, containerId, pipes[1]))
10.   .onAny([pipes]() { os::close(pipes[0]); })
11.   .onAny([pipes]() { os::close(pipes[1]); });

 

Isolator定义了以下函数

 

 

 

中会调用

 

Master Contender and Detector

Detector是检测哪个是Mesos-Master的Leader，Contender是自己竞争成为Mesos-Master的leader。默认使用zookeeper。

 

4. Mesos-Master启动的第四步是加载Hook, HookManager::initialize(flags.hooks.get())