Openstack nova-scheduler 源码分析 — Filters/Weighting
2016-08-16 01:09 云物互联 阅读(1512) 评论(0) 编辑 收藏 举报目录
前言
本篇记录了 Openstack 在创建 Instances 时，nova-scheduler 作为调度器的工作原理和代码实现。
Openstack 中会由多个的 Instance 共享同一个 Host，而不是独占。所以就需要使用调度器这种管理规则来协调和管理 Instance 之间的资源分配。
调度器
调度器：调度 Instance 在哪一个 Host 上运行的方式。
目前 Nova 中实现的调度器方式由下列几种：
ChanceScheduler(随机调度器)：从所有正常运行 nova-compute 服务的 Host Node 中随机选取来创建 Instance
FilterScheduler(过滤调度器)：根据指定的过滤条件以及权重来挑选最佳创建 Instance 的 Host Node 。
Caching(缓存调度器)：是 FilterScheduler 中的一种，在其基础上将 Host 资源信息缓存到本地的内存中，然后通过后台的定时任务从数据库中获取最新的 Host 资源信息。
为了便于扩展，Nova 将一个调度器必须要实现的接口提取出来成为 nova.scheduler.driver.Scheduler，只要继承了该类并实现其中的接口，我们就可以自定义调度器。
注意：不同的调度器并不能共存，需要在 /etc/nova/nova.conf 中的选项指定使用哪一个调度器。默认为 FilterScheduler 。
vim /etc/nova/nova.conf
scheduler_driver = nova.scheduler.filter_scheduler.FilterSchedulerFilterScheduler调度器的工作流程
FilterScheduler 首先使用指定的 Filters(过滤器) 过滤符合条件的 Host，EG. 内存使用率小于 2% 。然后对得到的 Host 列表计算 Weighting 权重并排序，获得最佳的 Host 。
Filters 过滤器
Filtering 就是首先根据各个 Host 当前可用的资源情况来过滤掉那些不能满足 Instance 要求的 Host，然后再使用配置文件指定的各种 Filters 去过滤掉不符合过滤条件的 Host。经过 Filters 过滤后，会得到一个 Host 列表。
这样的话 nova-scheduler 就需要从数据库中取得当前各个 Host 最新的资源使用情况，这些资源数据的收集和存储都由 nova-compute 中定义的数据库同步机制来完成。但是 nova-compute 对数据库的更新是周期性的， nova-scheduler 在选择最佳 Host 时需要最新的资源数据。所以在 nova-scheduler 中使用了 nova.scheduler.host_manager:HostState 来维护一份数据。这份数据仅保存在当前进程的内存中，里面包含了从上次数据库更新到现在 Host 资源的变化情况，也就是最新的 Host 资源数据。nova-scheduler 为了保持自己所维护的资源数据是最新的，每创建一个 Instance ，nova-scheduler 都要将这份资源数据更新，并从 Host 可用资源中去掉虚拟机使用的部分。
注意：nova-scheduler 所维护的数据不会同步到数据库，它只会从数据库同步数据到自身，所以 nova-scheduler 并没有写数据库的功能。
Filters 类型
- ALLHostsFilter：不进行任何过滤
- RamFilter：根据内存的可用情况来进行过滤
- ComputeFilter：选取所有处于 Active 的 Host
- TrustedFilter：选取所有可信的 Host
- PciPassthroughFilter：选取提供 PCI SR-IOV 支持的 Host
所有的 Filters 实现都位于nova/scheduler/filters 目录，每个 Filter 都要继承自 nova.scheduler.filters.BaseHostFilter 。如果需要自定义一个 Filter，只需通过继承此类并实现一个函数 host_passes()，返回的结果只有 True or False 。
在配置文件中指定 Filters：
scheduler_available_filters=
scheduler_default_filters=Weighting 权重
Weighting 表示对所有符合过滤条件(通过 Filters)的 Host 计算权重并以此排序从而得到最佳的一个 Host。计算 Host 权重的过程需要调用指定的各种 Weigher Module，得到每个 Host 的权重值。
所有的 Weigher 的实现都位于 nova/scheduler/weights 目录下。
源码实现
关键文件及其意义
/nova/scheduler/driver.py: 文件中最重要的就是 Scheduler 类，是所有调度器实现都要继承的基类，包含了调度器必须要实现的所有接口。
/nova/scheduler/manager.py: 主要实现了 SchedulerManager 类，定义了 Host 的管理操作函数，如：删除 Host 中的 Instance — delete_instance_info
/nova/scheduler/host_manager.py: 有两个类的实现，都是描述了跟调度器相关的 Host 的操作实现，类 HostState 维护了一份最新的 Host 资源数据。类 HostManager 描述了调度器相关的操作函数， EG._choose_host_filters/get_filtered_hosts/get_weighed_hosts
/nova/scheduler/chance.py: 只有 ChanceScheduler 类(随机调度器)，继承自 Scheduler 类，实现随机选取 Host Node 的调度器
/nova/scheduler/client: 客户端调用程序的入口
/nova/scheduler/filter_scheduler.py: 只有 FilterScheduler 类(过滤调度器)，继承自 Scheduler 类，实现了根据指定的过滤条件来选取 Host Node 的调度器
/nova/scheduler/filters 和 /nova/scheduler/weights: 这两个目录下的内容分别对应 过滤器 和 权重 的实现 。
阶段一：nova-scheduler 接收 build_instances RPC 远程调用
nova-conductor ==> RPC scheduler_client.select_destinations() ==> nova-sechduler
#nova.conductor.manager.ComputeTaskManager:build_instances()
def build_instances(self, context, instances, image, filter_properties,
admin_password, injected_files, requested_networks,
security_groups, block_device_mapping=None, legacy_bdm=True):
# TODO(ndipanov): Remove block_device_mapping and legacy_bdm in version
# 2.0 of the RPC API.
# 获取需要创建的 Instance 的参数信息
request_spec = scheduler_utils.build_request_spec(context, image,
instances)
# TODO(danms): Remove this in version 2.0 of the RPC API
if (requested_networks and
not isinstance(requested_networks,
objects.NetworkRequestList)):
# 请求 network 信息
requested_networks = objects.NetworkRequestList(
objects=[objects.NetworkRequest.from_tuple(t)
for t in requested_networks])
# TODO(melwitt): Remove this in version 2.0 of the RPC API
# 获取 flavor 信息
flavor = filter_properties.get('instance_type')
if flavor and not isinstance(flavor, objects.Flavor):
# Code downstream may expect extra_specs to be populated since it
# is receiving an object, so lookup the flavor to ensure this.
flavor = objects.Flavor.get_by_id(context, flavor['id'])
filter_properties = dict(filter_properties, instance_type=flavor)
try:
scheduler_utils.setup_instance_group(context, request_spec,
filter_properties)
# check retry policy. Rather ugly use of instances[0]...
# but if we've exceeded max retries... then we really only
# have a single instance.
scheduler_utils.populate_retry(filter_properties,
instances[0].uuid)
# 获取 Hosts 列表
hosts = self.scheduler_client.select_destinations(context,
request_spec, filter_properties)
except Exception as exc:
updates = {'vm_state': vm_states.ERROR, 'task_state': None}
for instance in instances:
self._set_vm_state_and_notify(
context, instance.uuid, 'build_instances', updates,
exc, request_spec)
return
for (instance, host) in itertools.izip(instances, hosts):
try:
instance.refresh()
except (exception.InstanceNotFound,
exception.InstanceInfoCacheNotFound):
LOG.debug('Instance deleted during build', instance=instance)
continue
local_filter_props = copy.deepcopy(filter_properties)
scheduler_utils.populate_filter_properties(local_filter_props,
host)
# The block_device_mapping passed from the api doesn't contain
# instance specific information
bdms = objects.BlockDeviceMappingList.get_by_instance_uuid(
context, instance.uuid)
self.compute_rpcapi.build_and_run_instance(context,
instance=instance, host=host['host'], image=image,
request_spec=request_spec,
filter_properties=local_filter_props,
admin_password=admin_password,
injected_files=injected_files,
requested_networks=requested_networks,
security_groups=security_groups,
block_device_mapping=bdms, node=host['nodename'],
limits=host['limits'])nova-conductor 在调用 nova-scheduler 来获取能够创建 Instance 的 Host 的同时也获取了：requested_networks/flavor 等信息。
其中获取 Hosts 列表的代码块：
# 获取 Hosts 列表
hosts = self.scheduler_client.select_destinations(context,
request_spec, filter_properties)下面列出了一系列为了获取 Hosts 列表的函数调用跳转：
# nova.scheduler.client.query.SchedulerQueryClient:select_destinations()
from nova.scheduler import rpcapi as scheduler_rpcapi
class SchedulerQueryClient(object):
"""Client class for querying to the scheduler."""
def __init__(self):
self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()
def select_destinations(self, context, request_spec, filter_properties):
"""Returns destinations(s) best suited for this request_spec and
filter_properties.
The result should be a list of dicts with 'host', 'nodename' and
'limits' as keys.
"""
#
return self.scheduler_rpcapi.select_destinations(
context, request_spec, filter_properties)
# nova.scheduler.rpcapi.SchedulerAPI:select_destinations()
def select_destinations(self, ctxt, request_spec, filter_properties):
cctxt = self.client.prepare(version='4.0')
return cctxt.call(ctxt, 'select_destinations',
request_spec=request_spec, filter_properties=filter_properties)阶段二：从 scheduler.rpcapi.SchedulerAPI 到 scheduler.manager.SchedulerManager
rpcapi.py 中的接口函数会在 manager.py 中实现实际操作函数。
所以跳转到 nova.scheduler.manager.SchedulerManager:select_destinations()
# nova.scheduler.manager.SchedulerManager:select_destinations()
class SchedulerManager(manager.Manager):
"""Chooses a host to run instances on."""
target = messaging.Target(version='4.2')
def __init__(self, scheduler_driver=None, *args, **kwargs):
if not scheduler_driver:
scheduler_driver = CONF.scheduler_driver
# 可以看出这里的 driver 是通过配置文件中的选项值指定的类来返回的对象 EG.nova.scheduler.filter_scheduler.FilterScheduler
self.driver = importutils.import_object(scheduler_driver)
super(SchedulerManager, self).__init__(service_name='scheduler',
*args, **kwargs)
def select_destinations(self, context, request_spec, filter_properties):
"""Returns destinations(s) best suited for this request_spec and
filter_properties.
The result should be a list of dicts with 'host', 'nodename' and
'limits' as keys.
"""
dests = self.driver.select_destinations(context, request_spec,
filter_properties)
return jsonutils.to_primitive(dests)阶段三：从 scheduler.manager.SchedulerManager 到调度器 FilterScheduler
vim /etc/nova/nova.conf
scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler从配置文件选项 scheduler_driver 的值可以知道，nova.scheduler.manager.SchedulerManager:driver
是 nova.scheduler.filter_scheduler.FilterScheduler 的实例化对象。
所以跳转到 nova.scheduler.filter_scheduler.FilterScheduler:select_destinations() 。
# nova.scheduler.filter_scheduler.FilterScheduler:select_destinations()
class FilterScheduler(driver.Scheduler):
"""Scheduler that can be used for filtering and weighing."""
def __init__(self, *args, **kwargs):
super(FilterScheduler, self).__init__(*args, **kwargs)
self.options = scheduler_options.SchedulerOptions()
self.notifier = rpc.get_notifier('scheduler')
def select_destinations(self, context, request_spec, filter_properties):
"""Selects a filtered set of hosts and nodes."""
self.notifier.info(context, 'scheduler.select_destinations.start',
dict(request_spec=request_spec))
# 需要创建的 Instances 的数量
num_instances = request_spec['num_instances']
# 获取满足笫一次过滤条件的主机列表 List (详见上述的调度器过滤原理)
# nova.scheduler.filter_scheduler.FilterScheduler:_schedule() ==> return selected_hosts
selected_hosts = self._schedule(context, request_spec,
filter_properties)
# Couldn't fulfill the request_spec
# 当请求的 Instance 数量大于合适的主机数量时，不会创建 Instance 且输出 'There are not enough hosts available.'
if len(selected_hosts) < num_instances:
# NOTE(Rui Chen): If multiple creates failed, set the updated time
# of selected HostState to None so that these HostStates are
# refreshed according to database in next schedule, and release
# the resource consumed by instance in the process of selecting
# host.
for host in selected_hosts:
host.obj.updated = None
# Log the details but don't put those into the reason since
# we don't want to give away too much information about our
# actual environment.
LOG.debug('There are %(hosts)d hosts available but '
'%(num_instances)d instances requested to build.',
{'hosts': len(selected_hosts),
'num_instances': num_instances})
reason = _('There are not enough hosts available.')
raise exception.NoValidHost(reason=reason)
dests = [dict(host=host.obj.host, nodename=host.obj.nodename,
limits=host.obj.limits) for host in selected_hosts]
self.notifier.info(context, 'scheduler.select_destinations.end',
dict(request_spec=request_spec))
return dests
def _schedule(self, context, request_spec, filter_properties):
# 获取所有 Hosts 的状态
hosts = self._get_all_host_states(elevated)
selected_hosts = []
# 获取需要创建的 Instances 数目
num_instances = request_spec.get('num_instances', 1)
# 遍历 num_instances，为每个 Instance 选取合适的主机
for num in range(num_instances):
# Filter local hosts based on requirements ...
# 在 for 循环里，_schedule 的两个关键操作，get_filtered_hosts() 和 get_weighed_hosts()
hosts = self.host_manager.get_filtered_hosts(hosts,
filter_properties, index=num)
if not hosts:
# Can't get any more locally.
break
LOG.debug("Filtered %(hosts)s", {'hosts': hosts})
weighed_hosts = self.host_manager.get_weighed_hosts(hosts,
filter_properties)
LOG.debug("Weighed %(hosts)s", {'hosts': weighed_hosts})
scheduler_host_subset_size = CONF.scheduler_host_subset_size
# 下面两个 if，主要为了防止 random.choice 调用越界
if scheduler_host_subset_size > len(weighed_hosts):
scheduler_host_subset_size = len(weighed_hosts)
if scheduler_host_subset_size < 1:
scheduler_host_subset_size = 1
# 在符合要求的weigh过的host里进行随机选取
chosen_host = random.choice(
weighed_hosts[0:scheduler_host_subset_size])
LOG.debug("Selected host: %(host)s", {'host': chosen_host})
selected_hosts.append(chosen_host)
# Now consume the resources so the filter/weights
# will change for the next instance.
chosen_host.obj.consume_from_instance(instance_properties)
if update_group_hosts is True:
if isinstance(filter_properties['group_hosts'], list):
filter_properties['group_hosts'] = set(
filter_properties['group_hosts'])
filter_properties['group_hosts'].add(chosen_host.obj.host)
# 循环为每一个实例获取合适的主机后，返回选择的主机列表
return selected_hosts上述的函数有三个非常关键的操作函数：
- _get_all_host_states: 获取所有的 Host 状态，并且将初步满足条件的 Hosts 过滤出来。
- get_filtered_hosts：使用 Filters 过滤器将第一个函数返回的 hosts 进行再一次过滤。
- get_weighed_hosts：通过 Weighed 选取最优 Host。
这三个关键函数在后面会继续介绍。
首先看看host_manager.get_filtered_hosts() 中，host_manager 是 nova.scheduler.driver.Scheduler 的成员变量 。如下：
# nova.scheduler.driver.Scheduler:__init__()
# nova.scheduler.filter_scheduler.FilterScheduler 继承了 nova.scheduler.driver.Scheduler
class Scheduler(object):
"""The base class that all Scheduler classes should inherit from."""
def __init__(self):
# 从这里知道 host_manager 会根据配置文件动态导入
self.host_manager = importutils.import_object(
CONF.scheduler_host_manager)
self.servicegroup_api = servicegroup.API()
还需要注意：scheduler.filter_scheduler.FilterScheduler:_schedule() 中获取 Hosts 状态的函数 _get_all_host_states() 实现如下：
# nova.scheduler.host_manager.HostManager:get_all_host_states()
def get_all_host_states(self, context):
service_refs = {service.host: service
for service in objects.ServiceList.get_by_binary(
context, 'nova-compute')}
# 获取 Compute Node 资源
compute_nodes = objects.ComputeNodeList.get_all(context)
# nova.object.__init__()
# ==> nova.object.compute_node.ComputeNodeList:get_all
seen_nodes = set()
for compute in compute_nodes:
service = service_refs.get(compute.host)
if not service:
LOG.warning(_LW(
"No compute service record found for host %(host)s"),
{'host': compute.host})
continue
host = compute.host
node = compute.hypervisor_hostname
state_key = (host, node)
host_state = self.host_state_map.get(state_key)
# 更新主机信息
if host_state:
host_state.update_from_compute_node(compute)
else:
host_state = self.host_state_cls(host, node, compute=compute)
self.host_state_map[state_key] = host_state
# We force to update the aggregates info each time a new request
# comes in, because some changes on the aggregates could have been
# happening after setting this field for the first time
host_state.aggregates = [self.aggs_by_id[agg_id] for agg_id in
self.host_aggregates_map[
host_state.host]]
host_state.update_service(dict(service))
self._add_instance_info(context, compute, host_state)
seen_nodes.add(state_key)
# remove compute nodes from host_state_map if they are not active
# * 移除 not active 的节点
dead_nodes = set(self.host_state_map.keys()) - seen_nodes
for state_key in dead_nodes:
host, node = state_key
LOG.info(_LI("Removing dead compute node %(host)s:%(node)s "
"from scheduler"), {'host': host, 'node': node})
del self.host_state_map[state_key]
return six.itervalues(self.host_state_map)
# get_all_host_states主要用来去除不活跃的节点继续往下看获取 Compute Node 资源信息函数 objects.ComputeNodeList.get_all(context) 的实现。
# nova.object.compute_node:get_all()
@base.remotable_classmethod
def get_all(cls, context):
# 调到了 nova.db.api.compute_node_get_all()
db_computes = db.compute_node_get_all(context)
return base.obj_make_list(context, cls(context), objects.ComputeNode,
db_computes)
#nova.db.api:compute_node_get_all()
def compute_node_get_all(context):
"""Get all computeNodes.
:param context: The security context
:returns: List of dictionaries each containing compute node properties
"""
return IMPL.compute_node_get_all(context)至此，说明 liberty 版本的 nova-scheduler 还是能够访问数据库的。
问题是： nova-scheduler 是怎么更新主机信息的，能够直接数据库进行写操作吗？
答案是：不能，nova-scheduler 不能够对数据库进行写操作，但是却可以从数据库中读取 Host 资源数据并缓存在进程的内存中。如下：
# nova.scheduler.host_manager.HostState:__init__()
class HostState(object):
"""Mutable and immutable information tracked for a host.
This is an attempt to remove the ad-hoc data structures
previously used and lock down access.
"""
def __init__(self, host, node, compute=None):
self.host = host
self.nodename = node
# Mutable available resources.
# These will change as resources are virtually "consumed".
self.total_usable_ram_mb = 0
self.total_usable_disk_gb = 0
self.disk_mb_used = 0
self.free_ram_mb = 0
self.free_disk_mb = 0
self.vcpus_total = 0
self.vcpus_used = 0
self.pci_stats = None
self.numa_topology = None
# Additional host information from the compute node stats:
self.num_instances = 0
self.num_io_ops = 0
# Other information
self.host_ip = None
self.hypervisor_type = None
self.hypervisor_version = None
self.hypervisor_hostname = None
self.cpu_info = None
self.supported_instances = None
nova-scheduler 并没有写数据库的操作函数，但是 nova-scheduler 会将数据库的数据缓存到进程内存中。这样就可以在保证了 nova-scheduler 能使用最新的 Host 资源信息，同时下降低了对数据库的 I/O 请求。
阶段四：从调度器 FilterScheduler 到过滤器 Filters
上面的代码中调用了 Filters 函数：get_filtered_hosts()，实现如下：
# nova.scheduler.host_manager.HostManager:get_filtered_hosts()
def get_filtered_hosts(self, hosts, filter_properties,
filter_class_names=None, index=0):
"""Filter hosts and return only ones passing all filters."""
# 下面定义了若干局部函数，先省略掉
def _strip_ignore_hosts(host_map, hosts_to_ignore):
ignored_hosts = []
for host in hosts_to_ignore:
。。。。
# 返回经过验证的可用的过滤器；
filter_classes = self._choose_host_filters(filter_class_names)
。。。。
# 调用了get_filtered_objects
return self.filter_handler.get_filtered_objects(filters,
hosts, filter_properties, index)
# 继续跳转到 get_filtered_objects()
def get_filtered_objects(self, filters, objs, filter_properties, index=0):
list_objs = list(objs)
LOG.debug("Starting with %d host(s)", len(list_objs))
part_filter_results = []
full_filter_results = []
log_msg = "%(cls_name)s: (start: %(start)s, end: %(end)s)"
for filter_ in filters:
if filter_.run_filter_for_index(index):
cls_name = filter_.__class__.__name__
start_count = len(list_objs)
# 关键的一句话
objs = filter_.filter_all(list_objs, filter_properties)
if objs is None:
LOG.debug("Filter %s says to stop filtering", cls_name)
return
list_objs = list(objs)
end_count = len(list_objs)
part_filter_results.append(log_msg % {"cls_name": cls_name,
"start": start_count, "end": end_count})
if list_objs:
remaining = [(getattr(obj, "host", obj),
getattr(obj, "nodename", ""))
for obj in list_objs]
full_filter_results.append((cls_name, remaining))
return list_objs
# objs 的 return 又调用了 filter_.filter_all(list_objs, filter_properties)
def filter_all(self, filter_obj_list, filter_properties):
for obj in filter_obj_list：
if self._filter_one(obj, filter_properties):
# 符合规则 生产一个obj
yield obj
# 继续调用 _filter_one()
def _filter_one(self, obj, filter_properties):
# 如果符合 Filter 过滤器，就返回 TRUE，否则返回 FALSE
return self.host_passes(obj, filter_properties)经过一连串的调用跳转，Filter 的过滤工作就完成了。
阶段五：Filters 到权重计算与排序
# nova.scheduler.host_manager.HostManager:get_weighed_hosts（）
def get_weighed_hosts(self, hosts, weight_properties):
"""Weigh the hosts."""
return self.weight_handler.get_weighed_objects(self.weighers,
hosts, weight_properties)
# nova.weights.BaseWeightHandler:get_weighed_objects（）
class BaseWeightHandler(loadables.BaseLoader):
object_class = WeighedObject
def get_weighed_objects(self, weighers, obj_list, weighing_properties):
"""Return a sorted (descending), normalized list of WeighedObjects."""
weighed_objs = [self.object_class(obj, 0.0) for obj in obj_list]
if len(weighed_objs) <= 1:
return weighed_objs
for weigher in weighers:
weights = weigher.weigh_objects(weighed_objs, weighing_properties)
# Normalize the weights
weights = normalize(weights,
minval=weigher.minval,
maxval=weigher.maxval)
for i, weight in enumerate(weights):
obj = weighed_objs[i]
obj.weight += weigher.weight_multiplier() * weight
# 进行排序
return sorted(weighed_objs, key=lambda x: x.weight, reverse=True)
转载请注明作者：JmilkFan 范桂飓
