从volume表的host字段开始学习RPC

这里大致包括以下内容：
- devstack安装环境中中cinder mq相关信息
- volume表host字段相关信息
- cinder-volume和cinder-backup部署关系的问题

接着上一篇笔记，”Cinder multiple backend”，来学习一下cinder中的rpc。

为什么要说这个呢？
因为我在开发volume虚机备份功能的时候，参考cinder-backup rpcapi.py，会这样发送rpc request：

# cinder/backup/rpcapi.py
def create_backup(self, ctxt, host, backup_id, volume_id):
    LOG.debug("create_backup in rpcapi backup_id %s", backup_id)
    cctxt = self.client.prepare(server=host)
    cctxt.cast(ctxt, 'create_backup', backup_id=backup_id)

其中的host参数就是cinder数据库中backup表的host字段。而该字段是用如下方法从volume表的host字段中抽取出来的：

# cinder/backup/api.py
volume_host = volume_utils.extract_host(volume['host'], 'host')
if not self._is_backup_service_enabled(volume, volume_host):
    raise exception.ServiceNotFound(service_id='cinder-backup')

Note：后面有关于_is_backup_service_enabled方法的分析。

先来看看volume_utils.extract_host方法，这里说明了cinder中“host”的结构。

“Host”结构

看注释最清楚了：

# cinder/volumes/utils.py

def extract_host(host, level='backend', default_pool_name=False):
    """Extract Host, Backend or Pool information from host string.

    :param host: String for host, which could include host@backend#pool info
    :param level: Indicate which level of information should be extracted
                  from host string. Level can be 'host', 'backend' or 'pool',
                  default value is 'backend'
    :param default_pool_name: this flag specify what to do if level == 'pool'
                              and there is no 'pool' info encoded in host
                              string.  default_pool_name=True will return
                              DEFAULT_POOL_NAME, otherwise we return None.
                              Default value of this parameter is False.
    :return: expected level of information

    For example:
        host = 'HostA@BackendB#PoolC'
        ret = extract_host(host, 'host')
        # ret is 'HostA'
        ret = extract_host(host, 'backend')
        # ret is 'HostA@BackendB'
        ret = extract_host(host, 'pool')
        # ret is 'PoolC'

        host = 'HostX@BackendY'
        ret = extract_host(host, 'pool')
        # ret is None
        ret = extract_host(host, 'pool', True)
        # ret is '_pool0'
    """

host字段结构为：host@backend#pool。分三个层次，具体由来还得以后分析（例如：Spec1）

extract_host默认会抽取到backend层次。
看了Kilo版本代码：

volume相关的方法，都抽取到了默认的backend层次
volume backup相关的方法，都抽取到了host层次（因为backup时，需要读取volume的数据，必须指定特定的host。当然对于Ceph，Sheepdog之类的存储，host不一定是某个物理机。）

分析实验环境

cinder.conf参见“Cinder multiple backend”笔记。

从上面的测试结果来看，数据库中是这样的：

MariaDB [cinder]> select host from volumes where status='available';
+---------------------------------------------------------+
| host                                                    |
+---------------------------------------------------------+
| maqi-openstack.novalocal@lvmdriver-1#lvmdriver-1        |
| maqi-openstack.novalocal@whatever_it_is#new_lvm_backend |
+---------------------------------------------------------+

所以，
host为maqi-openstack.novalocal
backend有两个：lvmdriver-1, whatever_it_is（cinder.conf中的enabled_backends，也就是section name）
pool也有两个：lvmdriver-1, new_lvm_backend （cinder.conf中的volume_backend_name，也是extra_specs中的值）

以下先试图分析创建bacup时，rpc msg中的host以及msg的收发。
创建volume的过程涉及到scheduler，比较复杂，以后再看。

先来看实验环境中一些rabbitmq相关的信息。

Rabbitmq相关的信息

在rabbitmq中，message处理的基本过程是：

publisher构造message，把它发送到指定的exchange上。该message中包含routing_key。
这个exchange收到该message之后，根据自己的类型（fanout，direct，topic等），分析routing_key。routing_key的作用是让exchange找到queue，因为queue在创建的时候，已经把该routing_key注册到exchange上了。所以，exchange根据routing_key把message发送到对应的queue上。
该queue有一个或多个consumer，一直在监听queue上的message。当有新message到达queue时，这个consumer就取出message，并且执行这个message中指定的方法。

先来看看环境中有哪些exchange、queue。

官方文档在说明volume multiple backend时有如下介绍：

Cinder volume backends are spawned as children to cinder-volume, and they are keyed from a unique Queue. They are named cinder-volume.HOST.BACKEND. Example cinder-volume.ubuntu.lvmdriver. The filter scheduler determines where to send the volume based on the volume type thats passed in.

看看有没有这样一个queue：

[root@maqi-openstack admin]# rabbitmqctl list_queues | grep cinder
cinder-scheduler    0
cinder-scheduler.maqi-openstack.novalocal   0
cinder-scheduler_fanout_fc01cfa5062041d9ac0f592ff9d83c90    0
cinder-volume   0
cinder-volume.maqi-openstack.novalocal@lvmdriver-1  0               # 在这里！！！
cinder-volume.maqi-openstack.novalocal@whatever_it_is   0           # 在这里！！！
cinder-volume_fanout_592f4422802444ecb6a51bdf19feb9a0   0
cinder-volume_fanout_e8f7dc451a1d4ece942a9c378ddd9451   0

[root@maqi-openstack admin]# rabbitmqctl list_exchanges | grep cinder
cinder-scheduler_fanout fanout
cinder-volume_fanout    fanout

我的测试主机的hostname如下：

[admin@maqi-openstack ~]$ hostname
maqi-openstack.novalocal

所以形式上跟官方文档说的稍有出入（.变成@）。对于这两个queue来说：
HOST：maqi-openstack.novalocal
BACKEND：whatever_it_is，lvmdriver-1

顺便再看看exchange和cinder service的host：

[admin@maqi-openstack ~]$ sudo rabbitmqctl list_exchanges | grep cinder
cinder-scheduler_fanout fanout
cinder-volume_fanout    fanout

[admin@maqi-openstack ~]$ cinder service-list
+------------------+-----------------------------------------+------+---------+-------+----------------------------+-----------------+
|      Binary      |                   Host                  | Zone |  Status | State |         Updated_at         | Disabled Reason |
+------------------+-----------------------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler |         maqi-openstack.novalocal        | nova | enabled |   up  | 2015-11-01T03:05:28.000000 |        -        |
|  cinder-volume   |   maqi-openstack.novalocal@lvmdriver-1  | nova | enabled |   up  | 2015-11-01T03:05:30.000000 |        -        |
|  cinder-volume   | maqi-openstack.novalocal@whatever_it_is | nova | enabled |   up  | 2015-11-01T03:05:29.000000 |        -        |
+------------------+-----------------------------------------+------+---------+-------+----------------------------+-----------------+

猜测一下，带有不同的host的cinder-volume service会处理不同的queue中的msg。
事实也是如此：

[admin@maqi-openstack ~]$ sudo rabbitmqctl list_bindings source_name  destination_name destination_kind routing_key | grep cinder-volume
    cinder-volume   queue   cinder-volume
    cinder-volume.maqi-openstack.novalocal@lvmdriver-1  queue   cinder-volume.maqi-openstack.novalocal@lvmdriver-1
    cinder-volume.maqi-openstack.novalocal@whatever_it_is   queue   cinder-volume.maqi-openstack.novalocal@whatever_it_is
    cinder-volume_fanout_592f4422802444ecb6a51bdf19feb9a0   queue   cinder-volume_fanout_592f4422802444ecb6a51bdf19feb9a0
    cinder-volume_fanout_e8f7dc451a1d4ece942a9c378ddd9451   queue   cinder-volume_fanout_e8f7dc451a1d4ece942a9c378ddd9451
cinder-volume_fanout    cinder-volume_fanout_592f4422802444ecb6a51bdf19feb9a0   queue   cinder-volume
cinder-volume_fanout    cinder-volume_fanout_e8f7dc451a1d4ece942a9c378ddd9451   queue   cinder-volume
openstack   cinder-volume   queue   cinder-volume
openstack   cinder-volume.maqi-openstack.novalocal@lvmdriver-1  queue   cinder-volume.maqi-openstack.novalocal@lvmdriver-1
openstack   cinder-volume.maqi-openstack.novalocal@whatever_it_is   queue   cinder-volume.maqi-openstack.novalocal@whatever_it_is

[admin@maqi-openstack ~]$ sudo rabbitmqctl list_consumers | grep cinder-volume
cinder-volume   <rabbit@maqi-openstack.3.15042.10>  0   true    0   []
cinder-volume   <rabbit@maqi-openstack.3.15066.10>  0   true    0   []
cinder-volume.maqi-openstack.novalocal@lvmdriver-1  <rabbit@maqi-openstack.3.15066.10>  1   true    0   []
cinder-volume.maqi-openstack.novalocal@whatever_it_is   <rabbit@maqi-openstack.3.15042.10>  1   true    0   []
cinder-volume_fanout_592f4422802444ecb6a51bdf19feb9a0   <rabbit@maqi-openstack.3.15042.10>  2   true    0   []
cinder-volume_fanout_e8f7dc451a1d4ece942a9c378ddd9451   <rabbit@maqi-openstack.3.15066.10>  2   true    0   []

如果只看whatever_it_is相关的：

[admin@maqi-openstack ~]$ sudo rabbitmqctl list_bindings source_name  destination_name destination_kind routing_key | grep whatever_it_is
    cinder-volume.maqi-openstack.novalocal@whatever_it_is   queue   cinder-volume.maqi-openstack.novalocal@whatever_it_is
openstack   cinder-volume.maqi-openstack.novalocal@whatever_it_is   queue   cinder-volume.maqi-openstack.novalocal@whatever_it_is

[admin@maqi-openstack ~]$ sudo rabbitmqctl list_queues | grep whatever_it_is
cinder-volume.maqi-openstack.novalocal@whatever_it_is   0

[admin@maqi-openstack ~]$ sudo rabbitmqctl list_consumers | grep whatever_it_is
cinder-volume.maqi-openstack.novalocal@whatever_it_is   <rabbit@maqi-openstack.3.15042.10>  1   true    0   []

也就是说，如果publisher（client）指定
host=maqi-openstack.novalocal@whatever_it_is，这个就是routing_key，那么：

会根据bindings的routing_key找到对应的queue，也叫cinder-volume.maqi-openstack.novalocal@whatever_it_is；
这个queue和他的consumer同名！所以对应的consumer就是：service为cinder-volume，host为maqi-openstack.novalocal@whatever_it_is

rabbitmq基本概念就差不多了，下面看backup流程。

创建backup中的“host”

上面的实验环境没有安装cinder-backup，来看另一个以Ceph为cinder-volume/cinder-backup为后端的环境：

cinder.conf

admin@maqi-kilo:~|⇒  grep ceph /etc/cinder/cinder.conf
default_volume_type = ceph
enabled_backends = ceph
backup_ceph_stripe_count = 0
backup_ceph_stripe_unit = 0
backup_ceph_user = cinder-bak
backup_ceph_pool = backup
backup_ceph_conf =
backup_driver = cinder.backup.drivers.ceph
[ceph]
rbd_ceph_conf =
volume_backend_name = ceph

admin@maqi-kilo:~|⇒  grep -E "\[ceph\]" -A 8 /etc/cinder/cinder.conf
[ceph]
rbd_max_clone_depth = 5
rbd_flatten_volume_from_snapshot = False
rbd_uuid = f9d1e9d8-978b-4c39-a3cb-c7895eeb023a
rbd_user = cinder
rbd_pool = volumes
rbd_ceph_conf =
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph

cinder services:

admin@maqi-kilo:~|⇒  cinder service-list
+------------------+----------------+------+---------+-------+----------------------------+-----------------+
|      Binary      |      Host      | Zone |  Status | State |         Updated_at         | Disabled Reason |
+------------------+----------------+------+---------+-------+----------------------------+-----------------+
|  cinder-backup   |   maqi-kilo    | nova | enabled |   up  | 2015-11-01T05:02:10.000000 |       None      |
| cinder-scheduler |   maqi-kilo    | nova | enabled |   up  | 2015-11-01T05:02:08.000000 |       None      |
|  cinder-volume   | maqi-kilo@ceph | nova | enabled |  down | 2015-10-29T11:21:19.000000 |       None      |
+------------------+----------------+------+---------+-------+----------------------------+-----------------+

admin@maqi-kilo:~|⇒  hostname
maqi-kilo

根据前面的分析，猜测：

cinder volume表中的host字段为：maqi-kilo@ceph#ceph
事实如此：

mysql> select host from volumes where status='available';
+---------------------+
| host                |
+---------------------+
| maqi-kilo@ceph#ceph |
| maqi-kilo@ceph#ceph |
+---------------------+

cinder backup表中的host字段为：maqi-kilo
也是这样：

mysql> select host from backups where status='available';
+-----------+
| host      |
+-----------+
| maqi-kilo |
| maqi-kilo |
+-----------+

查看exchanges：

admin@maqi-kilo:~|⇒  sudo rabbitmqctl list_exchanges | grep cinder
cinder-backup_fanout    fanout
cinder-scheduler_fanout fanout
cinder-volume_fanout    fanout

为什么都是fanout？（其实也有topic，看后面）

查看cinder-backup bindings：

admin@maqi-kilo:~|⇒  sudo rabbitmqctl list_bindings routing_key source_name source_kind destination_name destination_kind | grep backup
cinder-backup       exchange    cinder-backup   queue
cinder-backup   cinder-backup_fanout    exchange    cinder-backup_fanout_f17aa49b2eaf44bdb1b17b80036143f3   queue
cinder-backup   openstack   exchange    cinder-backup   queue
cinder-backup.maqi-kilo     exchange    cinder-backup.maqi-kilo queue
cinder-backup.maqi-kilo openstack   exchange    cinder-backup.maqi-kilo queue
cinder-backup_fanout_f17aa49b2eaf44bdb1b17b80036143f3       exchange    cinder-backup_fanout_f17aa49b2eaf44bdb1b17b80036143f3   queue

admin@maqi-kilo:~|⇒  sudo rabbitmqctl list_bindings routing_key source_name source_kind destination_name destination_kind | grep volume
cinder-volume       exchange    cinder-volume   queue
cinder-volume   cinder-volume_fanout    exchange    cinder-volume_fanout_b0eeffc83545454f8996b330d9979766   queue
cinder-volume   openstack   exchange    cinder-volume   queue
cinder-volume.maqi-kilo@ceph        exchange    cinder-volume.maqi-kilo@ceph    queue
cinder-volume.maqi-kilo@ceph    openstack   exchange    cinder-volume.maqi-kilo@ceph    queue
cinder-volume_fanout_b0eeffc83545454f8996b330d9979766       exchange    cinder-volume_fanout_b0eeffc83545454f8996b330d9979766   queue

所以，cinder-backup rpcapi作为publisher，指定”server=host”，也就是“maqi-kilo”之后：
1. routing_key = cinder-backup.maqi-kilo
2. 对应的queue=cinder-backup.maqi-kilo

查看queues：

admin@maqi-kilo:~|⇒  sudo rabbitmqctl list_queues name  policy pid  consumers | grep backup
cinder-backup       <'rabbit@maqi-kilo'.2.1267.0>   1
cinder-backup.maqi-kilo     <'rabbit@maqi-kilo'.2.1268.0>   1
cinder-backup_fanout_f17aa49b2eaf44bdb1b17b80036143f3       <'rabbit@maqi-kilo'.2.26083.15> 1

admin@maqi-kilo:~|⇒  sudo rabbitmqctl list_queues name  policy pid  consumers | grep volume
cinder-volume       <'rabbit@maqi-kilo'.2.1292.0>   1
cinder-volume.maqi-kilo@ceph        <'rabbit@maqi-kilo'.2.1293.0>   1
cinder-volume_fanout_b0eeffc83545454f8996b330d9979766       <'rabbit@maqi-kilo'.2.21927.31> 1

每个queue都有1个consumer。

查看consumers（怎么和queues对应起来？？）：

admin@maqi-kilo:~|⇒  sudo rabbitmqctl list_consumers | grep cinder
cinder-backup   <'rabbit@maqi-kilo'.2.26082.15> 1   true    []
cinder-backup.maqi-kilo <'rabbit@maqi-kilo'.2.26082.15> 2   true    []
cinder-backup_fanout_f17aa49b2eaf44bdb1b17b80036143f3   <'rabbit@maqi-kilo'.2.26082.15> 3   true    []
cinder-scheduler    <'rabbit@maqi-kilo'.2.5180.0>   1   true    []
cinder-scheduler.maqi-kilo  <'rabbit@maqi-kilo'.2.5180.0>   2   true    []
cinder-scheduler_fanout_d6753755f99347049ea47562df073621    <'rabbit@maqi-kilo'.2.5180.0>   3   true    []
cinder-volume   <'rabbit@maqi-kilo'.2.21926.31> 1   true    []
cinder-volume.maqi-kilo@ceph    <'rabbit@maqi-kilo'.2.21926.31> 2   true    []
cinder-volume_fanout_b0eeffc83545454f8996b330d9979766   <'rabbit@maqi-kilo'.2.21926.31> 3   true    []

从现象看，现有的信息都对上了（缺了一个queue和consumer之间的关系）。
这里写图片描述
但是为什么是这样的？需要看看cinder-backup/cinder-volume的启动过程。鉴于篇幅限制，另起一篇笔记查看cinder-backup启动过程。

另外，还有一些疑惑，放在To-Do-List中：

To-Do-List:

cinder exchange为何都是fanout？message都是从fanout exchange开始接收吗？【Done】
答案：不是。有topic，叫做”openstack”。这个名字可以通过cinder.conf中的control_exchange配置，oslo_messaging/transport.py把其默认值设为”openstack”。具体分析参考笔记“cinder-backup启动过程跟踪”。

admin@maqi-kilo:~|⇒  sudo rabbitmqctl list_exchanges | grep topic
amq.rabbitmq.log    topic
amq.rabbitmq.trace  topic
amq.topic   topic
neutron topic
nova        topic                # nova、neutron都有自己的topic
openstack   topic                # cinder的topic叫做openstack
admin@maqi-kilo:~|⇒  sudo rabbitmqctl list_bindings | grep openstack
openstack   exchange    cinder-backup   queue   cinder-backup   []
openstack   exchange    cinder-backup.maqi-kilo queue   cinder-backup.maqi-kilo []
openstack   exchange    cinder-scheduler    queue   cinder-scheduler    []
openstack   exchange    cinder-scheduler.maqi-kilo  queue   cinder-scheduler.maqi-kilo  []
openstack   exchange    cinder-volume   queue   cinder-volume   []
openstack   exchange    cinder-volume.maqi-kilo@ceph    queue   cinder-volume.maqi-kilo@ceph    []
openstack   exchange    cinder-volume.new_host_name@ceph    queue   cinder-volume.new_host_name@ceph    []
openstack   exchange    notifications.info  queue   notifications.info  []

在ceph的环境里面试试multiple backend
可以用不同的ceph pool。cinder-scheduler的pool-aware filter也是基于类似思想，以pool为单位进行调度，而不是host，因为就算host的资源够用但pool里的资源不够，创建还是会失败。
看看创建volume的过程
cinder-backup在发送rpc消息时，都是指定host的。看看cinder-scheduler、cinder-volume在创建volume过程中是怎么发送消息的。最好能在某个地方把消息打印出来（rpcapi.py?）
研究_is_backup_service_enabled【还没完成】
看下一节。
怎么查看consumers和queues对应关系？
rabbitmq监控程序能干吗？

Cinder-backup的疑问

cinder-volume和cinder-backup需不需要部署在同一个物理机上？

创建volume之后，他的host为maqi-kilo@ceph#ceph

admin@maqi-kilo:~|⇒  cinder show 97247b32-1327-4bf9-bd1f-a5f16c672c90 | grep host
|         os-vol-host-attr:host         |    maqi-kilo@ceph#ceph

前面说过了，对volume做backup的时候，会这样判断：

# cinder/backup/api.py
volume_host = volume_utils.extract_host(volume['host'], 'host')
if not self._is_backup_service_enabled(volume, volume_host):
    raise exception.ServiceNotFound(service_id='cinder-backup')

_is_backup_service_enabled定义如下：

# cinder/backup/api.py
def _is_backup_service_enabled(self, volume, volume_host):
    """Check if there is a backup service available."""
    topic = CONF.backup_topic
    ctxt = context.get_admin_context()
    services = self.db.service_get_all_by_topic(ctxt,
                                                topic,
                                                disabled=False)
    for srv in services:
        if (srv['availability_zone'] == volume['availability_zone'] and
                srv['host'] == volume_host and
                utils.service_is_up(srv)):
            return True
    return False

根据topic，也就是“cinder-backup”，把services表中的所有记录取出来
检查service和volume的availability_zone是否相同
检查service[‘host’]是否和volume_host相同
检查service是不是up

关键是第3步。

mysql> SELECT host, `binary`, topic, availability_zone FROM services;
+--------------------+------------------+------------------+-------------------+
| host               | binary           | topic            | availability_zone |
+--------------------+------------------+------------------+-------------------+
| maqi-kilo          | cinder-scheduler | cinder-scheduler | nova              |
| maqi-kilo          | cinder-backup    | cinder-backup    | nova              |
| maqi-kilo@ceph     | cinder-volume    | cinder-volume    | nova              |
+--------------------+------------------+------------------+-------------------+

cinder-backup的“host”为“maqi-kilo”。而volume_host也等于“maqi-kilo”，所以在这种情况下是一致的。

因此，最终还是得看cinder-volume和cinder-backup的启动过程，他们是怎么取得“host”值的；并且，cinder-volume在创建volume的时候，是按照什么逻辑填写volume的“host”值的。

Note：

volume的host字段是由cinder-scheduler决定的
cinder-scheduler会以host@backend#pool格式来标识一个host

这篇文章讲host字段还不错。
这里写图片描述

多个cinder-volume可以使用同一个backend
cinder.conf中的每一个backend section可以指定host参数。如果指定了，那么该参数会替代物理机的hostname，出现在cinder service-list以及cinder volume host中。这样的好处是：Case 2中，如果物理机A挂了，还可以通过物理机B上的c-vol访问A创建的volume。
在LVM中，会使用volume_backend_name作为pool name（Ceph中呢？）

Kilo版本cinder.conf reference：

# Name of this node. This can be an opaque identifier. It is
# not necessarily a host name, FQDN, or IP address. (string
# value)
# host=cinder

经实验，在Ceph作为cinder-volume/cinder-backup后端的环境中，在cinder.conf的[ceph]中添加host=new_host_name:

admin@maqi-kilo:~|⇒  cinder service-list
+------------------+--------------------+------+---------+-------+----------------------------+-----------------+
|      Binary      |        Host        | Zone |  Status | State |         Updated_at         | Disabled Reason |
+------------------+--------------------+------+---------+-------+----------------------------+-----------------+
|  cinder-backup   |     maqi-kilo      | nova | enabled |   up  | 2015-11-01T12:15:31.000000 |       None      |
| cinder-scheduler |     maqi-kilo      | nova | enabled |   up  | 2015-11-01T12:15:33.000000 |       None      |
|  cinder-volume   |   maqi-kilo@ceph   | nova | enabled |  down | 2015-11-01T11:50:55.000000 |       None      |
|  cinder-volume   | new_host_name@ceph | nova | enabled |   up  | 2015-11-01T12:15:35.000000 |       None      |
+------------------+--------------------+------+---------+-------+----------------------------+-----------------+

创建volume成功：

admin@maqi-kilo:~|⇒  cinder show 9a5618d3-bcb2-4652-abf4-0964577d71cc | grep host
|         os-vol-host-attr:host         |       new_host_name@ceph#ceph        |

创建backup失败：

2015-11-01 12:18:15.168 30140 INFO cinder.api.openstack.wsgi [req-24d48b30-ef4c-4e2e-8702-94458509e94d - - - - -] HTTP exception thrown: Service cinder-backup could not be found.^M

原因就是volume的host值和cinder-backup的host值不同！
（为什么一定要相同？这是关键问题，为了读取volume数据吗？）

P.S. 可以在cinder.conf的[default]中指定host值
已经试过了，可以。

posted on 2016-01-24 17:49 七里山塘边阅读(321) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

七里山塘边