Nova（三）：Nova组件详解+日志解析：nova-compute
nova-compute；Driver架构 ================================================================== nova-compute nova-compute 在计算节点上运行，负责管理节点上的 instance。 OpenStack 对 instance 的操作，最后都是交给 nova-compute 来完成的。 nova-compute 与 Hypervisor 一起实现 OpenStack 对 instance 生命周期的管理。 ------------------------------------------------------------------------- Driver架构: nova-compute 为这些 Hypervisor 定义了统一的接口，Hypervisor 只需要实现这些接口，就可以 Driver 的形式即插即用到 OpenStack 系统中。 某个特定的计算节点上只会运行一种 Hypervisor，只需在该节点 nova-compute 的配置文件 /etc/nova/nova.conf 中配置所对应的 compute_driver 就可以了。
nova-compute 的功能可以分为两类：
1.定时向 OpenStack 报告计算节点的状态
2.实现 instance 生命周期的管理
========================================================
1.定时向 OpenStack 报告计算节点的状态
定期向 OpenStack 报告计算节点的状态: nova-compute 会定期向 OpenStack 报告。从 nova-compute 的日志（/var/log/nova/nova-compute.log）可以发现： 每隔一段时间，nova-compute 就会报告当前计算节点的资源使用情况和 nova-compute 服务状态。 可以根据代码模块nova.compute.resource_tracker进行搜索，暂时未发现类型教程中的资源汇报日志 nova-compute 可以通过 Hypervisor 的 driver 拿到这些信息。举例来说，在我们的实验环境下 Hypervisor 是 KVM，用的 Driver 是 LibvirtDriver。 LibvirtDriver 可以调用相关的 API 获得资源信息，这些 API 的作用相当于我们在 CLI 里执行 virsh nodeinfo、virsh dominfo 等命令。 ------------------------------------------------------------------------------- 该日志已经从教程的INFO日志改为DEBUG日志了 2021-11-15 23:50:31.394 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-15 23:51:32.189 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-15 23:52:33.203 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-15 23:53:34.313 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-15 23:54:34.325 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-15 23:55:34.195 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-15 23:56:36.199 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-16 07:13:39.534 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-16 07:14:35.219 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-16 07:15:37.223 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-16 07:16:38.224 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-16 07:17:39.208 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-16 07:18:40.321 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-16 07:19:41.213 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-16 07:20:41.412 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061 2021-11-16 07:21:42.222 58619 DEBUG nova.compute.resource_tracker [req-07294362-3636-49be-bd49-81fb893d62b1 - - - - -] Final resource view: name=compute22 phys_ram=9818MB used_ram=1024MB phys_disk=117GB used_disk=0GB total_vcpus=2 used_vcpus=1 pci_stats=[] _report_final_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1061
2.实现 instance 生命周期的管理
OpenStack 对 instance 最主要的操作都是通过 nova-compute 实现的，包括 instance 的 launch、shutdown、reboot、suspend、resume、terminate、resize、migration、snapshot 等。
launch instance操作调用nova各个子服务的流程顺序： 1.nova-api接收到launch instance请求（可能是白屏，也可能是api方式） 2.nova-api做一些必要的处理后，将launch instance请求放入rabbitMQ 3.nova-scheduler从rabbitMQ获取launch instance请求，执行调度算法，选择出节点A，将请求“在节点A launch instance”放入rabbitMQ 4.节点A的nova-compute从rabbitMQ获取请求，通过本节点的Hypervisor Driver创建Instance 5.在launch instance过程中，nova-compute如果需要查询or更新数据库，会通过rabbitMQ向nova-conductor发送请求，由nova-conductor负责数据库的访问
nova-compute创建 instance 的过程；nova-compute日志路径；nova-compute日志分析 ======================================================================================= nova-compute 创建 instance 的过程可以分为 4 步： 1.为 instance 准备资源 2.创建 instance 的镜像文件 3.创建 instance 的 XML 定义文件 4.创建虚拟网络并启动虚拟机 -------------------------------------------------------------------------------------------- nova-compute日志路径 此处以VM launch为例：nova-compute从rabbitMQ中获取到任务。 日志 /opt/stack/logs/n-cpu.log 记录了整个操作过程。 /var/log/nova/nova-compute.log 注意：只有计算节点才部署了nova-compute -------------------------------------------------------------------------------------------- nova-compute日志分析 修改为debug模式，并重启了服务openstack-nova-compute nova-compute 创建 instance 的过程可以分为 4 步： ---------------------------------------------------------------------------------------------- 为了下载镜像的日志，下一步还是还原到初始阶段，然后开启debug，在开始部署 记得直接在answer表内配置为debug模式。。。 1.为 instance 准备资源 nova-compute 首先会根据指定的 flavor 依次为 instance 分配内存、磁盘空间和 vCPU。 日志与教材的不一样了，部分日志由INFO日志变成了DEBUG日志，而且没有找到教材中的分配内存、磁盘空间、vCPU等日志 2021-11-15 23:50:57.075 58421 DEBUG nova.compute.manager [req-587bc111-a94d-4736-832f-d02b60caca7c 7ad71a9654d447419cb6e0cf449a9d8e 45405654b7704f28a28d83815d936dc5 - default default] [instance: 9cf55563-3d6a-4f85-b1c4-25a3c58c4c6a] Starting instance... _do_build_and_run_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:2141 2021-11-15 23:50:57.742 58421 DEBUG oslo_concurrency.lockutils [req-587bc111-a94d-4736-832f-d02b60caca7c 7ad71a9654d447419cb6e0cf449a9d8e 45405654b7704f28a28d83815d936dc5 - default default] Lock "compute_resources" acquired by "nova.compute.resource_tracker.instance_claim" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-15 23:50:57.761 58421 DEBUG nova.virt.hardware [req-587bc111-a94d-4736-832f-d02b60caca7c 7ad71a9654d447419cb6e0cf449a9d8e 45405654b7704f28a28d83815d936dc5 - default default] Require both a host and instance NUMA topology to fit instance on host. numa_fit_instance_to_host /usr/lib/python2.7/site-packages/nova/virt/hardware.py:2016 2021-11-15 23:50:57.761 58421 INFO nova.compute.claims [req-587bc111-a94d-4736-832f-d02b60caca7c 7ad71a9654d447419cb6e0cf449a9d8e 45405654b7704f28a28d83815d936dc5 - default default] [instance: 9cf55563-3d6a-4f85-b1c4-25a3c58c4c6a] Claim successful on node compute23 网络资源也会提前分配。 2021-11-16 23:06:00.700 57736 DEBUG nova.compute.manager [req-ee5271f5-d8bd-45ae-9368-b71f5c51b1e2 7ad71a9654d447419cb6e0cf449a9d8e 45405654b7704f28a28d83815d936dc5 - default default] [instance: 728a057c-33b1-4c10-8315-e0f2bcd78245] Start building networks asynchronously for instance. _build_resources /usr/lib/python2.7/site-packages/nova/compute/manager.py:2596 2021-11-16 23:06:00.764 57736 DEBUG nova.compute.manager [req-ee5271f5-d8bd-45ae-9368-b71f5c51b1e2 7ad71a9654d447419cb6e0cf449a9d8e 45405654b7704f28a28d83815d936dc5 - default default] [instance: 728a057c-33b1-4c10-8315-e0f2bcd78245] Allocating IP information in the background. _allocate_network_async /usr/lib/python2.7/site-packages/nova/compute/manager.py:1687 2021-11-16 23:06:03.164 57736 DEBUG nova.compute.manager [req-ee5271f5-d8bd-45ae-9368-b71f5c51b1e2 7ad71a9654d447419cb6e0cf449a9d8e 45405654b7704f28a28d83815d936dc5 - default default] [instance: 728a057c-33b1-4c10-8315-e0f2bcd78245] Instance network_info: |[{"profile": {}, "ovs_interfaceid": "d89e7ee1-4cb6-4638-8d7d-c4c23bdd938a", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.92"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "45405654b7704f28a28d83815d936dc5", "physical_network": "extnet", "mtu": 1500}, "id": "7761c4cf-fbaf-4ce6-92aa-3ba1857a5bf6", "label": "public"}, "devname": "tapd89e7ee1-4c", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:8b:56:a8", "active": false, "type": "ovs", "id": "d89e7ee1-4cb6-4638-8d7d-c4c23bdd938a", "qbg_params": null}]| _allocate_network_async /usr/lib/python2.7/site-packages/nova/compute/manager.py:1701 2021-11-16 23:06:05.404 57736 DEBUG nova.compute.manager [req-01f4d3f6-267b-4dd9-96bc-73851e6871f6 3088d4458a384b7cac8d9a32d0f1ef71 e1d733a50c984a6290f3ce538c852b6e - default default] [instance: 728a057c-33b1-4c10-8315-e0f2bcd78245] Received event network-changed-d89e7ee1-4cb6-4638-8d7d-c4c23bdd938a external_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:9318 2021-11-16 23:06:05.404 57736 DEBUG nova.compute.manager [req-01f4d3f6-267b-4dd9-96bc-73851e6871f6 3088d4458a384b7cac8d9a32d0f1ef71 e1d733a50c984a6290f3ce538c852b6e - default default] [instance: 728a057c-33b1-4c10-8315-e0f2bcd78245] Refreshing instance network info cache due to event network-changed-d89e7ee1-4cb6-4638-8d7d-c4c23bdd938a. external_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:9322 2.创建 instance 的镜像文件（nova-compute先将 image 下载到计算节点，再将其作为 backing file 创建 instance 的镜像文件。） nova-compute 首先会检查 image 是否已经下载（比如之前已经创建过基于相同 image 的 instance）。如果没有，就从 Glance 下载 image 到本地。由此可知，如果计算节点上要运行多个相同 image 的 instance，只会在启动第一个 instance 的时候从 Glance 下载 image，后面的 instance 启动速度就大大加快了。 日志xxxxxx 1. image（ID为 917d60ef-f663-4e2d-b85b-e4511bb56bc2）是 qcow2 格式，nova-compute 将其下载。Nova 默认会通过 qemu-img 转换成 raw 格式，以提高 IO 性能。 2. image 的存放目录是 /opt/stack/data/nova/instances/_base，这是由 /etc/nova/nova.conf 的下面两个配置选项决定的。 3. 下载的 image 文件被命名为 60bba5916c6c90ed2ef7d3263de8f653111dd35f，这是 image id 的 SHA1 哈希值。 有了 image 之后，instance 的镜像文件直接通过 qemu-img 命令创建，backing file 就是下载的 image。 日志与教程不一样。。。 2021-11-20 09:16:24.937 6217 INFO nova.virt.block_device [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Booting with volume-backed-image fb601b4b-5e21-4574-ad3d-b66ab4075130 at /dev/vda 2021-11-20 09:16:40.989 6217 DEBUG nova.virt.libvirt.driver [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Creating instance directory _create_image /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3857 2021-11-20 09:16:40.990 6217 INFO nova.virt.libvirt.driver [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Creating image 2021-11-20 09:16:40.990 6217 DEBUG nova.virt.libvirt.driver [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Did not create local disks _create_image /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3992 3.创建 instance 的 XML 定义文件 2021-11-20 09:16:40.994 6217 DEBUG nova.virt.libvirt.driver [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Start _get_guest_xml network_info=[{"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": false, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null}] disk_info={'disk_bus': 'virtio', 'cdrom_bus': 'ide', 'mapping': {u'/dev/vda': {'bus': u'virtio', 'boot_index': '1', 'type': u'disk', 'dev': u'vda'}, 'root': {'bus': u'virtio', 'boot_index': '1', 'type': u'disk', 'dev': u'vda'}}} image_meta=ImageMeta(checksum='9067fe7e99229a1a6d908b331c5763df',container_format='bare',created_at=2021-11-20T01:15:22Z,direct_url=<?>,disk_format='qcow2',id=fb601b4b-5e21-4574-ad3d-b66ab4075130,min_disk=0,min_ram=0,name='cirros_1120',owner='f911df6b22d54d34b716efd7a97edf92',properties=ImageMetaProps,protected=<?>,size=16872448,status='active',tags=<?>,updated_at=2021-11-20T01:15:22Z,virtual_size=<?>,visibility=<?>) rescue=None block_device_info={'swap': None, 'root_device_name': u'/dev/vda', 'ephemerals': [], 'block_device_mapping': [{'guest_format': None, 'boot_index': 0, 'mount_device': u'/dev/vda', 'connection_info': {u'driver_volume_type': u'iscsi', 'connector': {'platform': 'x86_64', 'host': 'compute23', 'system uuid': u'A9524D56-E47D-02DE-49ED-687D607FCF9B', 'do_local_attach': False, 'ip': '192.168.1.23', 'os_type': 'linux2', 'multipath': False, 'initiator': u'iqn.1994-05.com.redhat:51f266169dc3'}, 'serial': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'data': {u'access_mode': u'rw', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_lun': 0, u'auth_password': u'***', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}}, 'volume_type': None, 'disk_bus': u'virtio', 'device_type': u'disk', 'attachment_id': None, 'delete_on_termination': True}]} _get_guest_xml /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6154 2021-11-20 09:16:42.450 6217 DEBUG nova.virt.libvirt.driver [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] End _get_guest_xml xml=<domain type="qemu"> <uuid>b78ce7ba-268d-40dd-88c8-4b516e14e4a3</uuid> <name>instance-00000001</name> <memory>524288</memory> <vcpu>1</vcpu> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0"> <nova:package version="20.6.0-1.el7"/> <nova:name>test1</nova:name> <nova:creationTime>2021-11-20 01:16:40</nova:creationTime> <nova:flavor name="m1.tiny"> <nova:memory>512</nova:memory> <nova:disk>1</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="80f373269d7a481b931a9352b42b28b8">admin</nova:user> <nova:project uuid="f911df6b22d54d34b716efd7a97edf92">admin</nova:project> </nova:owner> </nova:instance> </metadata> <sysinfo type="smbios"> <system> <entry name="manufacturer">RDO</entry> <entry name="product">OpenStack Compute</entry> <entry name="version">20.6.0-1.el7</entry> <entry name="serial">b78ce7ba-268d-40dd-88c8-4b516e14e4a3</entry> <entry name="uuid">b78ce7ba-268d-40dd-88c8-4b516e14e4a3</entry> <entry name="family">Virtual Machine</entry> </system> </sysinfo> <os> <type machine="pc">hvm</type> <boot dev="hd"/> <smbios mode="sysinfo"/> </os> <features> <acpi/> <apic/> </features> <cputune> <shares>1024</shares> </cputune> <clock offset="utc"/> <cpu match="exact"> <topology sockets="1" cores="1" threads="1"/> </cpu> <devices> <disk type="block" device="disk"> <driver name="qemu" type="raw" cache="none" io="native"/> <source dev="/dev/sdb"/> <target bus="virtio" dev="vda"/> <serial>712ac45e-68c3-47f9-845e-e5e0b38fe9f5</serial> </disk> <interface type="bridge"> <mac address="fa:16:3e:2b:6e:05"/> <model type="virtio"/> <driver name="qemu"/> <source bridge="br-int"/> <mtu size="1500"/> <target dev="tapffe535ab-d0"/> <virtualport type="openvswitch"> <parameters interfaceid="ffe535ab-d0d0-4deb-bf19-b3814f4ec54e"/> </virtualport> </interface> <serial type="pty"> <log file="/var/lib/nova/instances/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/console.log" append="off"/> </serial> <input type="tablet" bus="usb"/> <graphics type="vnc" autoport="yes" listen="0.0.0.0"/> <video> <model type="cirrus"/> </video> <memballoon model="virtio"> <stats period="10"/> </memballoon> </devices> </domain> _get_guest_xml /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6161 4.创建虚拟网络并启动虚拟机 2021-11-20 09:16:45.330 6217 DEBUG nova.virt.driver [-] Emitting event <LifecycleEvent: 1637371005.33, b78ce7ba-268d-40dd-88c8-4b516e14e4a3 => Started> emit_event /usr/lib/python2.7/site-packages/nova/virt/driver.py:1706 2021-11-20 09:16:45.331 6217 INFO nova.compute.manager [-] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] VM Started (Lifecycle Event) 2021-11-20 09:16:45.345 6217 DEBUG nova.virt.libvirt.driver [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Guest created on hypervisor spawn /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3558 2021-11-20 09:16:45.349 6217 INFO nova.virt.libvirt.driver [-] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance spawned successfully. 2021-11-20 09:16:45.349 6217 INFO nova.compute.manager [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Took 4.36 seconds to spawn the instance on the hypervisor. 2021-11-20 09:16:45.350 6217 DEBUG nova.compute.manager [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 09:16:45.367 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 09:16:45.369 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Synchronizing instance power state after lifecycle event "Started"; current vm_state: building, current task_state: spawning, current DB power_state: 0, VM power_state: 1 handle_lifecycle_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:1250 2021-11-20 09:16:45.435 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] During sync_power_state the instance has a pending task (spawning). Skip. 2021-11-20 09:16:45.436 6217 DEBUG nova.virt.driver [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] Emitting event <LifecycleEvent: 1637371005.33, b78ce7ba-268d-40dd-88c8-4b516e14e4a3 => Paused> emit_event /usr/lib/python2.7/site-packages/nova/virt/driver.py:1706 2021-11-20 09:16:45.436 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] VM Paused (Lifecycle Event) 2021-11-20 09:16:45.482 6217 INFO nova.compute.manager [req-c1003e3c-60a1-4823-89f9-a7aaf8c4a575 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Took 21.02 seconds to build instance. 2021-11-20 09:16:45.485 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 09:16:45.487 6217 DEBUG nova.virt.driver [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] Emitting event <LifecycleEvent: 1637371005.34, b78ce7ba-268d-40dd-88c8-4b516e14e4a3 => Resumed> emit_event /usr/lib/python2.7/site-packages/nova/virt/driver.py:1706 2021-11-20 09:16:45.488 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] VM Resumed (Lifecycle Event) 2021-11-20 09:16:45.523 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Synchronizing instance power state after lifecycle event "Resumed"; current vm_state: active, current task_state: None, current DB power_state: 1, VM power_state: 1 handle_lifecycle_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:1250
shut off instance操作调用nova各个子服务的流程;相关日志 ======================================================================================== shut off instance操作调用nova各个子服务的流程 1.nova-api接收到shut off instance请求 2.nova-api做一些必要的处理后，将请求放入rabbitMQ 3.nova-compute 执行操作 --------------------------------------------------------------------------------------------- shut off instance操作调用nova各个子服务的流程 1.nova-api接收到shut off instance请求；nova-api做一些必要的处理后，将请求放入rabbitMQ #/var/log/nova/nova-api.log 2021-11-20 15:03:56.383 15855 DEBUG nova.api.openstack.wsgi [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Action: 'action', calling method: <bound method ServersController._stop_server of <nova.api.openstack.compute.servers.ServersController object at 0x7fe6428ec710>>, body: {"os-stop": null} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:520 2021-11-20 15:03:56.390 15855 DEBUG oslo_concurrency.lockutils [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 15:03:56.390 15855 DEBUG oslo_concurrency.lockutils [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 15:03:56.440 15855 DEBUG nova.compute.api [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Going to try to stop instance force_stop /usr/lib/python2.7/site-packages/nova/compute/api.py:2442 #注意这条日志 2021-11-20 15:03:56.478 15855 DEBUG nova.objects.instance [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 15:03:56.502 15855 DEBUG nova.objects.instance [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'info_cache' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 15:03:56.557 15855 INFO nova.api.openstack.requestlog [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] 192.168.1.21 "POST /v2.1/f911df6b22d54d34b716efd7a97edf92/servers/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/action" status: 202 len: 0 microversion: 2.1 time: 0.382267 #这里的POST请求应该就是将shut off instance请求放入到rabbitMQ，但这只是我的个人猜测 #其实有另一种方法可以确认nova-api将请求放入到rabbitMQ中： #a.根据“Going to try to stop instance”这条日志，锁定源码位置/usr/lib/python2.7/site-packages/nova/compute/api.py:2442 #b.查看源码，根据self.compute_rpcapi.stop_instance()方法确认了“nova-api将请求放入到rabbitMQ中”。 def force_stop(self, context, instance, do_cast=True, clean_shutdown=True): LOG.debug("Going to try to stop instance", instance=instance) #这里触发dbug日志 instance.task_state = task_states.POWERING_OFF instance.progress = 0 instance.save(expected_task_state=[None]) self._record_action_start(context, instance, instance_actions.STOP) self.compute_rpcapi.stop_instance(context, instance, do_cast=do_cast, clean_shutdown=clean_shutdown) #在OpenStack源码中，以xxx_rpcapi命名的对象，表示的就是 xxx 的消息队列。 xxx_rpcapi.yyy() 方法则表示向 xxx 的消息队列发送 yyy 操作的消息。 2.nova-compute从rabbitMQ获取请求，执行操作 #根据上面的requestID快速过滤日志 #/var/log/nova/nova-compute.log 2021-11-20 15:03:56.627 6217 DEBUG oslo_concurrency.lockutils [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "b78ce7ba-268d-40dd-88c8-4b516e14e4a3" acquired by "nova.compute.manager.do_stop_instance" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 #在对vm操作前，先加锁？？？ 2021-11-20 15:03:56.627 6217 DEBUG nova.compute.manager [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 #看起来是检查状态 2021-11-20 15:03:56.630 6217 DEBUG nova.compute.manager [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Stopping instance; current vm_state: active, current task_state: powering-off, current DB power_state: 1, current VM power_state: 1 do_stop_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:3074 #Stopping instance 2021-11-20 15:03:56.630 6217 DEBUG nova.objects.instance [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 15:03:56.724 6217 DEBUG nova.virt.libvirt.driver [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Shutting down instance from state 1 _clean_shutdown /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3314 #Shutting down instance 2021-11-20 15:04:06.803 6217 DEBUG nova.virt.libvirt.driver [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance in state 1 after 10 seconds - resending shutdown _clean_shutdown /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3340 2021-11-20 15:04:17.883 6217 DEBUG nova.virt.libvirt.driver [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance in state 1 after 21 seconds - resending shutdown _clean_shutdown /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3340 2021-11-20 15:04:28.963 6217 DEBUG nova.virt.libvirt.driver [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance in state 1 after 32 seconds - resending shutdown _clean_shutdown /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3340 2021-11-20 15:04:40.053 6217 DEBUG nova.virt.libvirt.driver [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance in state 1 after 43 seconds - resending shutdown _clean_shutdown /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3340 2021-11-20 15:04:51.140 6217 DEBUG nova.virt.libvirt.driver [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance in state 1 after 54 seconds - resending shutdown _clean_shutdown /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:3340 2021-11-20 15:04:57.184 6217 INFO nova.virt.libvirt.driver [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance failed to shutdown in 60 seconds. #到这里看起来是关机失败了,但是实际却关机成功了？？？？？？？？？？？？ 2021-11-20 15:04:57.387 6217 DEBUG oslo_service.periodic_task [req-1b5628c2-d046-4875-a520-d094d789207d - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:217 2021-11-20 15:04:57.421 6217 INFO nova.virt.libvirt.driver [-] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance destroyed successfully. #Instance destroyed？？？？？ 2021-11-20 15:04:57.422 6217 DEBUG nova.compute.manager [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 15:04:57.638 6217 DEBUG oslo_concurrency.lockutils [req-e7f161a9-1a19-44a1-b185-0035c8aa0409 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "b78ce7ba-268d-40dd-88c8-4b516e14e4a3" released by "nova.compute.manager.do_stop_instance" :: held 61.012s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339
Start instance操作调用nova各个子服务的流程;相关日志 ======================================================================================== Start instance操作调用nova各个子服务的流程 1.nova-api接收到Start instance请求;nova-api做一些必要的处理后，将请求放入rabbitMQ 2.nova-compute从rabbitMQ获取请求，执行操作 ----------------------------------------------------------------------------------------- Start instance操作调用nova各个子服务的流程 1.nova-api接收到Start instance请求;nova-api做一些必要的处理后，将请求放入rabbitMQ #/var/log/nova/nova-api.log 2021-11-20 17:43:00.568 15856 DEBUG nova.api.openstack.wsgi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Action: 'action', calling method: <bound method ServersController._start_server of <nova.api.openstack.compute.servers.ServersController object at 0x7fe64288a110>>, body: {"os-start": null} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:520 2021-11-20 17:43:00.573 15856 DEBUG oslo_concurrency.lockutils [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 17:43:00.574 15856 DEBUG oslo_concurrency.lockutils [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 17:43:00.621 15856 DEBUG nova.compute.api [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Going to try to start instance start /usr/lib/python2.7/site-packages/nova/compute/api.py:2465 #Going to try to start instance；self.compute_rpcapi.start_instance（）方法 2021-11-20 17:43:00.657 15856 DEBUG nova.objects.instance [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 17:43:00.680 15856 DEBUG nova.objects.instance [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'info_cache' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 17:43:00.730 15856 ERROR oslo.messaging._drivers.impl_rabbit [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [a30302b1-71ff-41cf-8c0b-af8190df1dfe] AMQP server on 192.168.1.21:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe 2021-11-20 17:43:01.742 15856 INFO oslo.messaging._drivers.impl_rabbit [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [a30302b1-71ff-41cf-8c0b-af8190df1dfe] Reconnected to AMQP server on 192.168.1.21:5672 via [amqp] client with port 54052. 2021-11-20 17:43:01.744 15856 INFO nova.api.openstack.requestlog [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] 192.168.1.21 "POST /v2.1/f911df6b22d54d34b716efd7a97edf92/servers/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/action" status: 202 len: 0 microversion: 2.1 time: 1.178885 #POST方法 def start(self, context, instance): """Start an instance.""" LOG.debug("Going to try to start instance", instance=instance) instance.task_state = task_states.POWERING_ON instance.save(expected_task_state=[None]) self._record_action_start(context, instance, instance_actions.START) self.compute_rpcapi.start_instance(context, instance) #该方法即向rabbitMQ compute_rpcapi队列发送start_instance方法 2.nova-compute从rabbitMQ获取请求，执行操作 #根据requestID，在/var/log/nova/nova-compute.log中快速锁定日志 2021-11-20 17:43:01.767 6217 DEBUG nova.objects.instance [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 17:43:01.808 6217 DEBUG oslo_concurrency.lockutils [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Acquired lock "refresh_cache-b78ce7ba-268d-40dd-88c8-4b516e14e4a3" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:265 2021-11-20 17:43:01.809 6217 DEBUG nova.network.neutronv2.api [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Building network info cache for instance _get_instance_nw_info /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1758 #看起来是分配网络资源 2021-11-20 17:43:01.809 6217 DEBUG nova.objects.instance [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'info_cache' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 17:43:02.438 6217 DEBUG nova.network.base_api [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Updating instance_info_cache with network_info: [{"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": false, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null}] update_instance_cache_with_nw_info /usr/lib/python2.7/site-packages/nova/network/base_api.py:47 2021-11-20 17:43:02.451 6217 DEBUG oslo_concurrency.lockutils [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Releasing lock "refresh_cache-b78ce7ba-268d-40dd-88c8-4b516e14e4a3" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:281 2021-11-20 17:43:02.474 6217 INFO nova.virt.libvirt.driver [-] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance destroyed successfully. 2021-11-20 17:43:02.474 6217 DEBUG nova.objects.instance [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'resources' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 17:43:02.483 6217 DEBUG nova.virt.libvirt.vif [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2021-11-20T01:16:22Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description='test1',display_name='test1',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),hidden=False,host='compute23',hostname='test1',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2021-11-20T01:16:45Z,launched_on='compute23',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='compute23',numa_topology=<?>,old_flavor=None,os_type=None,pci_devices=<?>,pci_requests=<?>,power_state=4,progress=0,project_id='f911df6b22d54d34b716efd7a97edf92',ramdisk_id='',reservation_id='r-cnbmtisc',resources=None,root_device_name='/dev/vda',root_gb=1,security_groups=<?>,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='admin,reader,member',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='powering-on',terminated_at=None,trusted_certs=<?>,updated_at=2021-11-20T07:04:57Z,user_data=None,user_id='80f373269d7a481b931a9352b42b28b8',uuid=b78ce7ba-268d-40dd-88c8-4b516e14e4a3,vcpu_model=<?>,vcpus=1,vm_mode=None,vm_state='stopped') vif={"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": false, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} unplug /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:868 2021-11-20 17:43:02.484 6217 DEBUG nova.network.os_vif_util [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converting VIF {"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": false, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-11-20 17:43:02.485 6217 DEBUG nova.network.os_vif_util [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converted object VIFOpenVSwitch(active=False,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-11-20 17:43:02.486 6217 DEBUG os_vif [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Unplugging vif VIFOpenVSwitch(active=False,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') unplug /usr/lib/python2.7/site-packages/os_vif/__init__.py:109 2021-11-20 17:43:02.678 6217 INFO os_vif [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Successfully unplugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') 2021-11-20 17:43:02.680 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] calling os-brick to detach iSCSI Volume disconnect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:72 2021-11-20 17:43:02.681 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] ==> disconnect_volume: call u"{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f5c0c59f510>, {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'auth_password': u'***', u'target_lun': 0, u'access_mode': u'rw', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}, None), 'kwargs': {}}" trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:146 2021-11-20 17:43:02.681 6217 DEBUG oslo_concurrency.lockutils [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 17:43:02.711 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: No targets found. iSNS: No targets found. STATIC: Target: iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 Portal: 192.168.1.21:3260,-1 Iface Name: default FIRMWARE: No targets found. stderr= _run_iscsiadm_bare /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1139 2021-11-20 17:43:02.711 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Regex to get portals from discoverydb: ^SENDTARGETS: .*?^DiscoveryAddress: 192.168.1.21,3260.*? (.*?)^(?:DiscoveryAddress|iSNS):.* _get_discoverydb_portals /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:365 2021-11-20 17:43:02.712 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Getting connected devices for (ips,iqns,luns)=[(u'192.168.1.21:3260', u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', 0)] _get_connection_devices /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:824 2021-11-20 17:43:02.727 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('-m', 'session'): stdout=tcp: [1] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) stderr= _run_iscsiadm_bare /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1139 2021-11-20 17:43:02.727 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsi session list stdout=tcp: [1] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) stderr= _run_iscsi_session /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1128 2021-11-20 17:43:02.728 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Resulting device map defaultdict(<function <lambda> at 0x7f5c0e504578>, {(u'192.168.1.21:3260', u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5'): (set([u'sdb']), set([]))}) _get_connection_devices /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:856 2021-11-20 17:43:02.729 6217 DEBUG os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Removing single pathed devices sdb remove_connection /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:301 2021-11-20 17:43:02.761 6217 ERROR os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] multipathd is not running: exit code 1: ProcessExecutionError: Unexpected error while running command. 2021-11-20 17:43:02.761 6217 DEBUG os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Flushing IO for device /dev/sdb flush_device_io /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:359 2021-11-20 17:43:02.773 6217 DEBUG os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete remove_scsi_device /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:76 2021-11-20 17:43:02.802 6217 DEBUG os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Checking to see if SCSI volumes sdb have been removed. wait_for_volumes_removal /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:84 2021-11-20 17:43:02.803 6217 DEBUG os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] SCSI volumes sdb have been removed. wait_for_volumes_removal /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:93 2021-11-20 17:43:02.804 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Disconnecting from: [(u'192.168.1.21:3260', u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5')] _disconnect_connection /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1116 2021-11-20 17:43:02.862 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:02.902 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--logout',): stdout=Logging out of session [sid: 1, target: iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5, portal: 192.168.1.21,3260] Logout of [sid: 1, target: iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5, portal: 192.168.1.21,3260] successful. stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:02.910 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:02.911 6217 DEBUG oslo_concurrency.lockutils [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: held 0.230s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 17:43:02.911 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] <== disconnect_volume: return (229ms) None trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:170 2021-11-20 17:43:02.911 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Disconnected iSCSI Volume disconnect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:78 #上面应该是做了一系列的准备；下面开始启动了 2021-11-20 17:43:02.917 6217 DEBUG nova.virt.libvirt.driver [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Start _get_guest_xml network_info=[{"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": false, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null}] disk_info={'disk_bus': 'virtio', 'cdrom_bus': 'ide', 'mapping': {u'/dev/vda': {'bus': u'virtio', 'boot_index': '1', 'type': u'disk', 'dev': u'vda'}, 'root': {'bus': u'virtio', 'boot_index': '1', 'type': u'disk', 'dev': u'vda'}}} image_meta=ImageMeta(checksum=<?>,container_format='bare',created_at=<?>,direct_url=<?>,disk_format='qcow2',id=<?>,min_disk=1,min_ram=0,name=<?>,owner=<?>,properties=ImageMetaProps,protected=<?>,size=<?>,status=<?>,tags=<?>,updated_at=<?>,virtual_size=<?>,visibility=<?>) rescue=None block_device_info={'swap': None, 'root_device_name': u'/dev/vda', 'ephemerals': [], 'block_device_mapping': [{'guest_format': None, 'boot_index': 0, 'mount_device': u'/dev/vda', 'connection_info': {u'driver_volume_type': u'iscsi', u'connector': {u'initiator': u'iqn.1994-05.com.redhat:51f266169dc3', u'ip': u'192.168.1.23', u'system uuid': u'A9524D56-E47D-02DE-49ED-687D607FCF9B', u'platform': u'x86_64', u'host': u'compute23', u'do_local_attach': False, u'os_type': u'linux2', u'multipath': False}, u'serial': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'data': {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'auth_password': u'***', u'target_lun': 0, u'access_mode': u'rw', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}}, 'volume_type': None, 'disk_bus': u'virtio', 'device_type': u'disk', 'attachment_id': None, 'delete_on_termination': True}]} _get_guest_xml /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6154 2021-11-20 17:43:02.917 6217 DEBUG nova.objects.instance [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'numa_topology' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 17:43:02.933 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Getting desirable topologies for flavor Flavor(created_at=2021-11-19T14:59:54Z,deleted=False,deleted_at=None,description=None,disabled=False,ephemeral_gb=0,extra_specs={},flavorid='1',id=1,is_public=True,memory_mb=512,name='m1.tiny',projects=<?>,root_gb=1,rxtx_factor=1.0,swap=0,updated_at=None,vcpu_weight=0,vcpus=1) and image_meta ImageMeta(checksum=<?>,container_format='bare',created_at=<?>,direct_url=<?>,disk_format='qcow2',id=<?>,min_disk=1,min_ram=0,name=<?>,owner=<?>,properties=ImageMetaProps,protected=<?>,size=<?>,status=<?>,tags=<?>,updated_at=<?>,virtual_size=<?>,visibility=<?>), allow threads: True _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:593 2021-11-20 17:43:02.934 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Flavor limits 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:337 2021-11-20 17:43:02.934 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Image limits 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:341 2021-11-20 17:43:02.935 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Flavor pref 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:377 2021-11-20 17:43:02.935 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Image pref 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:381 2021-11-20 17:43:02.936 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Chose sockets=0, cores=0, threads=0; limits were sockets=65536, cores=65536, threads=65536 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:421 2021-11-20 17:43:02.936 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Topology preferred VirtCPUTopology(cores=0,sockets=0,threads=0), maximum VirtCPUTopology(cores=65536,sockets=65536,threads=65536) _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:597 2021-11-20 17:43:02.937 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Build topologies for 1 vcpu(s) 1:1:1 _get_possible_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:460 2021-11-20 17:43:02.937 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Got 1 possible topologies _get_possible_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:487 2021-11-20 17:43:02.937 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Possible topologies [VirtCPUTopology(cores=1,sockets=1,threads=1)] _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:602 2021-11-20 17:43:02.937 6217 DEBUG nova.virt.hardware [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Sorted desired topologies [VirtCPUTopology(cores=1,sockets=1,threads=1)] _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:627 2021-11-20 17:43:02.938 6217 DEBUG nova.objects.instance [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'vcpu_model' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 17:43:02.948 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Calling os-brick to attach iSCSI Volume connect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:63 2021-11-20 17:43:02.949 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] ==> connect_volume: call u"{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f5c0c59f510>, {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'auth_password': u'***', u'target_lun': 0, u'access_mode': u'rw', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}), 'kwargs': {}}" trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:146 2021-11-20 17:43:02.949 6217 DEBUG oslo_concurrency.lockutils [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.connect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 17:43:02.950 6217 INFO os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Trying to connect to iSCSI portal 192.168.1.21:3260 2021-11-20 17:43:02.958 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm (): stdout= stderr=iscsiadm: No records found _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:02.966 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.1.21,3260,-1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5] added stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:02.975 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:02.983 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', u'CHAP'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:02.993 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', u'HdxMiUcXLrwPMoArwn2z'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:03.000 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', u'***'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:03.009 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. _run_iscsiadm_bare /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1139 2021-11-20 17:43:03.009 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions. _run_iscsi_session /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1128 2021-11-20 17:43:03.009 6217 WARNING os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. 2021-11-20 17:43:13.357 6217 DEBUG oslo_service.periodic_task [req-1b5628c2-d046-4875-a520-d094d789207d - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:217 2021-11-20 17:43:14.357 6217 DEBUG oslo_service.periodic_task [req-1b5628c2-d046-4875-a520-d094d789207d - - - - -] Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:217 2021-11-20 17:43:20.065 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5, portal: 192.168.1.21,3260] (multiple) Login to [iface: default, target: iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5, portal: 192.168.1.21,3260] successful. stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:20.073 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 17:43:20.080 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('-m', 'session'): stdout=tcp: [2] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) stderr= _run_iscsiadm_bare /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1139 2021-11-20 17:43:20.080 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsi session list stdout=tcp: [2] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) stderr= _run_iscsi_session /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1128 2021-11-20 17:43:20.080 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Connected to 192.168.1.21:3260 _connect_vol /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:646 2021-11-20 17:43:20.080 6217 DEBUG os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] HCTL (u'4', '-', '-', 0) found on session 2 with lun 0 get_hctl /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:670 2021-11-20 17:43:20.081 6217 DEBUG os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Scanning host 4 c: -, t: -, l: 0) scan_iscsi /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:698 2021-11-20 17:43:20.089 6217 DEBUG os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Searching for a device in session 2 and hctl [u'4', '*', '*', 0] yield: None device_name_by_hctl /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:690 2021-11-20 17:43:21.090 6217 DEBUG os_brick.initiator.linuxscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Searching for a device in session 2 and hctl [u'4', '*', '*', 0] yield: sdb device_name_by_hctl /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:690 2021-11-20 17:43:21.091 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Connected to sdb using {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'access_mode': u'rw', u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_lun': 0, u'auth_password': u'***', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'} _connect_vol /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:675 2021-11-20 17:43:21.091 6217 DEBUG oslo_concurrency.lockutils [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.connect_volume" :: held 18.142s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 17:43:21.092 6217 DEBUG os_brick.initiator.connectors.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] <== connect_volume: return (18142ms) {'path': u'/dev/sdb', 'scsi_wwn': u'36001405775e9e56a27b49c2b004f84eb', 'type': u'block'} trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:170 2021-11-20 17:43:21.092 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Attached iSCSI volume {'path': u'/dev/sdb', 'scsi_wwn': '36001405775e9e56a27b49c2b004f84eb', 'type': 'block'} connect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:65 2021-11-20 17:43:21.109 6217 DEBUG nova.virt.libvirt.vif [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2021-11-20T01:16:22Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description='test1',display_name='test1',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),hidden=False,host='compute23',hostname='test1',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2021-11-20T01:16:45Z,launched_on='compute23',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='compute23',numa_topology=None,old_flavor=None,os_type=None,pci_devices=<?>,pci_requests=<?>,power_state=4,progress=0,project_id='f911df6b22d54d34b716efd7a97edf92',ramdisk_id='',reservation_id='r-cnbmtisc',resources=None,root_device_name='/dev/vda',root_gb=1,security_groups=<?>,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='admin,reader,member',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='powering-on',terminated_at=None,trusted_certs=<?>,updated_at=2021-11-20T07:04:57Z,user_data=None,user_id='80f373269d7a481b931a9352b42b28b8',uuid=b78ce7ba-268d-40dd-88c8-4b516e14e4a3,vcpu_model=VirtCPUModel,vcpus=1,vm_mode=None,vm_state='stopped') vif={"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": false, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} virt_type=qemu get_config /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:601 2021-11-20 17:43:21.110 6217 DEBUG nova.network.os_vif_util [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converting VIF {"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": false, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-11-20 17:43:21.111 6217 DEBUG nova.network.os_vif_util [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converted object VIFOpenVSwitch(active=False,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-11-20 17:43:21.115 6217 DEBUG nova.objects.instance [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'pci_devices' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 17:43:21.126 6217 DEBUG nova.virt.libvirt.driver [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] End _get_guest_xml xml=<domain type="qemu"> <uuid>b78ce7ba-268d-40dd-88c8-4b516e14e4a3</uuid> <name>instance-00000001</name> <memory>524288</memory> <vcpu>1</vcpu> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0"> <nova:package version="20.6.0-1.el7"/> <nova:name>test1</nova:name> <nova:creationTime>2021-11-20 09:43:02</nova:creationTime> <nova:flavor name="m1.tiny"> <nova:memory>512</nova:memory> <nova:disk>1</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="80f373269d7a481b931a9352b42b28b8">admin</nova:user> <nova:project uuid="f911df6b22d54d34b716efd7a97edf92">admin</nova:project> </nova:owner> </nova:instance> </metadata> <sysinfo type="smbios"> <system> <entry name="manufacturer">RDO</entry> <entry name="product">OpenStack Compute</entry> <entry name="version">20.6.0-1.el7</entry> <entry name="serial">b78ce7ba-268d-40dd-88c8-4b516e14e4a3</entry> <entry name="uuid">b78ce7ba-268d-40dd-88c8-4b516e14e4a3</entry> <entry name="family">Virtual Machine</entry> </system> </sysinfo> <os> <type machine="pc">hvm</type> <boot dev="hd"/> <smbios mode="sysinfo"/> </os> <features> <acpi/> <apic/> </features> <cputune> <shares>1024</shares> </cputune> <clock offset="utc"/> <cpu match="exact"> <topology sockets="1" cores="1" threads="1"/> </cpu> <devices> <disk type="block" device="disk"> <driver name="qemu" type="raw" cache="none" io="native"/> <source dev="/dev/sdb"/> <target bus="virtio" dev="vda"/> <serial>712ac45e-68c3-47f9-845e-e5e0b38fe9f5</serial> </disk> <interface type="bridge"> <mac address="fa:16:3e:2b:6e:05"/> <model type="virtio"/> <driver name="qemu"/> <source bridge="br-int"/> <mtu size="1500"/> <target dev="tapffe535ab-d0"/> <virtualport type="openvswitch"> <parameters interfaceid="ffe535ab-d0d0-4deb-bf19-b3814f4ec54e"/> </virtualport> </interface> <serial type="pty"> <log file="/var/lib/nova/instances/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/console.log" append="off"/> </serial> <input type="tablet" bus="usb"/> <graphics type="vnc" autoport="yes" listen="0.0.0.0"/> <video> <model type="cirrus"/> </video> <memballoon model="virtio"> <stats period="10"/> </memballoon> </devices> </domain> _get_guest_xml /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6161 2021-11-20 17:43:21.127 6217 DEBUG nova.virt.libvirt.driver [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] skipping disk /dev/sdb (vda) as it is a volume _get_instance_disk_info_from_config /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:9586 2021-11-20 17:43:21.128 6217 DEBUG nova.virt.libvirt.vif [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2021-11-20T01:16:22Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description='test1',display_name='test1',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),hidden=False,host='compute23',hostname='test1',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2021-11-20T01:16:45Z,launched_on='compute23',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='compute23',numa_topology=None,old_flavor=None,os_type=None,pci_devices=PciDeviceList,pci_requests=<?>,power_state=4,progress=0,project_id='f911df6b22d54d34b716efd7a97edf92',ramdisk_id='',reservation_id='r-cnbmtisc',resources=None,root_device_name='/dev/vda',root_gb=1,security_groups=<?>,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='admin,reader,member',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='powering-on',terminated_at=None,trusted_certs=<?>,updated_at=2021-11-20T07:04:57Z,user_data=None,user_id='80f373269d7a481b931a9352b42b28b8',uuid=b78ce7ba-268d-40dd-88c8-4b516e14e4a3,vcpu_model=VirtCPUModel,vcpus=1,vm_mode=None,vm_state='stopped') vif={"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": false, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} plug /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:750 2021-11-20 17:43:21.128 6217 DEBUG nova.network.os_vif_util [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converting VIF {"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": false, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-11-20 17:43:21.129 6217 DEBUG nova.network.os_vif_util [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converted object VIFOpenVSwitch(active=False,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-11-20 17:43:21.129 6217 DEBUG os_vif [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Plugging vif VIFOpenVSwitch(active=False,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') plug /usr/lib/python2.7/site-packages/os_vif/__init__.py:76 2021-11-20 17:43:21.170 6217 INFO os_vif [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') 2021-11-20 17:43:22.028 6217 DEBUG nova.virt.driver [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] Emitting event <LifecycleEvent: 1637401402.03, b78ce7ba-268d-40dd-88c8-4b516e14e4a3 => Resumed> emit_event /usr/lib/python2.7/site-packages/nova/virt/driver.py:1706 2021-11-20 17:43:22.028 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] VM Resumed (Lifecycle Event) 2021-11-20 17:43:22.046 6217 INFO nova.virt.libvirt.driver [-] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance rebooted successfully. #instance重启成功 2021-11-20 17:43:22.046 6217 DEBUG nova.compute.manager [req-27a89c6f-774d-412c-b1f6-ad4051913e78 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 17:43:22.066 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 17:43:22.070 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Synchronizing instance power state after lifecycle event "Resumed"; current vm_state: stopped, current task_state: powering-on, current DB power_state: 4, VM power_state: 1 handle_lifecycle_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:1250 2021-11-20 17:43:22.115 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] During sync_power_state the instance has a pending task (powering-on). Skip. 2021-11-20 17:43:22.115 6217 DEBUG nova.virt.driver [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] Emitting event <LifecycleEvent: 1637401402.04, b78ce7ba-268d-40dd-88c8-4b516e14e4a3 => Started> emit_event /usr/lib/python2.7/site-packages/nova/virt/driver.py:1706 2021-11-20 17:43:22.115 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] VM Started (Lifecycle Event) #VM正在运行中
soft reboot 与 hard reboot 的区别在于： 1. soft reboot 只是重启操作系统，整个过程中，instance 依然处于运行状态。相当于在 linux 中执行 reboot 命令 2. hard reboot 是重启 instance，相当于关机之后再开机
soft reboot日志 1.nova-api接收到soft reboot请求;nova-api做一些必要的处理后，将请求放入rabbitMQ（没找到相应的放入rabbitMQ的请求的日志，因为reboot方法中没有输出log） #soft/hard reboot 在 nova-api 的日志里找不到，这是因为 /opt/stack/nova/nova/compute/api.py 的 reboot 方法中没有输出 log。 可以通过关键字 “nova.api.openstack.wsgi” 或者 “reboot” 搜索。 2021-11-20 23:26:32.732 15856 DEBUG nova.api.openstack.wsgi [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Action: 'action', calling method: <bound method ServersController._action_reboot of <nova.api.openstack.compute.servers.ServersController object at 0x7fe64288a110>>, body: {"reboot": {"type": "SOFT"}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:520 #body: {"reboot": {"type": "SOFT"}} 2021-11-20 23:26:32.734 15856 DEBUG nova.compute.api [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2654 2021-11-20 23:26:32.739 15856 DEBUG oslo_concurrency.lockutils [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:26:32.740 15856 DEBUG oslo_concurrency.lockutils [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 23:26:32.740 15856 DEBUG oslo_concurrency.lockutils [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:26:32.740 15856 DEBUG oslo_concurrency.lockutils [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 23:26:32.826 15856 ERROR oslo.messaging._drivers.impl_rabbit [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [91e9e7ca-0be0-4007-9d0e-394e28aa452b] AMQP server on 192.168.1.21:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe 2021-11-20 23:26:33.840 15856 INFO oslo.messaging._drivers.impl_rabbit [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [91e9e7ca-0be0-4007-9d0e-394e28aa452b] Reconnected to AMQP server on 192.168.1.21:5672 via [amqp] client with port 38206. 2021-11-20 23:26:33.850 15856 ERROR oslo.messaging._drivers.impl_rabbit [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [a30302b1-71ff-41cf-8c0b-af8190df1dfe] AMQP server on 192.168.1.21:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe 2021-11-20 23:26:34.860 15856 INFO oslo.messaging._drivers.impl_rabbit [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [a30302b1-71ff-41cf-8c0b-af8190df1dfe] Reconnected to AMQP server on 192.168.1.21:5672 via [amqp] client with port 38208. 2021-11-20 23:26:34.864 15856 INFO nova.api.openstack.requestlog [req-8ca39be8-1c38-48bb-ac4a-f65c821a7b81 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] 192.168.1.21 "POST /v2.1/f911df6b22d54d34b716efd7a97edf92/servers/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/action" status: 202 len: 0 microversion: 2.1 time: 2.333187 #猜测这个就是相应的soft reboot请求的POST 2.nova-compute从rabbitMQ获取请求，执行操作 #看起来我的openstack软重启失败，最终使用了硬重启？？？？？？ #所以并没有看到什么明显的区别。。（在 nova-compute 的日志中可以看到 “soft reboot” 和 “hard reboot” 二者有明显的区别。） [root@compute23 ~]# cat /var/log/nova/nova-compute.log |grep -P "req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c|reboot" 2021-11-20 23:44:14.618 6217 DEBUG oslo_concurrency.lockutils [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "b78ce7ba-268d-40dd-88c8-4b516e14e4a3" acquired by "nova.compute.manager.do_reboot_instance" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:44:14.619 6217 INFO nova.compute.manager [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Rebooting instance #Rebooting instance 2021-11-20 23:44:14.653 6217 DEBUG oslo_concurrency.lockutils [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Acquired lock "refresh_cache-b78ce7ba-268d-40dd-88c8-4b516e14e4a3" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:265 2021-11-20 23:44:14.653 6217 DEBUG nova.network.neutronv2.api [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Building network info cache for instance _get_instance_nw_info /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1758 2021-11-20 23:44:15.327 6217 DEBUG nova.network.base_api [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Updating instance_info_cache with network_info: [{"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null}] update_instance_cache_with_nw_info /usr/lib/python2.7/site-packages/nova/network/base_api.py:47 2021-11-20 23:44:15.347 6217 DEBUG oslo_concurrency.lockutils [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Releasing lock "refresh_cache-b78ce7ba-268d-40dd-88c8-4b516e14e4a3" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:281 2021-11-20 23:44:15.361 6217 DEBUG nova.compute.manager [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 23:44:15.557 6217 DEBUG nova.objects.instance [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'pci_devices' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 23:44:58.964 6217 DEBUG oslo_service.periodic_task [req-1b5628c2-d046-4875-a520-d094d789207d - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:217 2021-11-20 23:46:01.064 6217 DEBUG oslo_service.periodic_task [req-1b5628c2-d046-4875-a520-d094d789207d - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:217 2021-11-20 23:46:16.402 6217 WARNING nova.virt.libvirt.driver [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Failed to soft reboot instance. Trying hard reboot. #Failed to soft reboot instance. Trying hard reboot. #看起来我的openstack软重启失败，最终使用了硬重启？？？？？？ 2021-11-20 23:46:16.642 6217 DEBUG nova.objects.instance [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'resources' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 23:46:16.654 6217 DEBUG nova.virt.libvirt.vif [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2021-11-20T01:16:22Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description='test1',display_name='test1',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),hidden=False,host='compute23',hostname='test1',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2021-11-20T01:16:45Z,launched_on='compute23',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='compute23',numa_topology=None,old_flavor=None,os_type=None,pci_devices=PciDeviceList,pci_requests=<?>,power_state=1,progress=0,project_id='f911df6b22d54d34b716efd7a97edf92',ramdisk_id='',reservation_id='r-cnbmtisc',resources=None,root_device_name='/dev/vda',root_gb=1,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='admin,reader,member',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='reboot_started',terminated_at=None,trusted_certs=<?>,updated_at=2021-11-20T15:44:15Z,user_data=None,user_id='80f373269d7a481b931a9352b42b28b8',uuid=b78ce7ba-268d-40dd-88c8-4b516e14e4a3,vcpu_model=<?>,vcpus=1,vm_mode=None,vm_state='active') vif={"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} unplug /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:868 2021-11-20 23:46:16.654 6217 DEBUG nova.network.os_vif_util [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converting VIF {"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-11-20 23:46:16.655 6217 DEBUG nova.network.os_vif_util [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converted object VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-11-20 23:46:16.655 6217 DEBUG os_vif [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Unplugging vif VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') unplug /usr/lib/python2.7/site-packages/os_vif/__init__.py:109 2021-11-20 23:46:16.678 6217 INFO os_vif [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Successfully unplugged vif VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') 2021-11-20 23:46:16.680 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] calling os-brick to detach iSCSI Volume disconnect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:72 2021-11-20 23:46:16.681 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] ==> disconnect_volume: call u"{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f5c0c59f510>, {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'auth_password': u'***', u'target_lun': 0, u'access_mode': u'rw', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}, None), 'kwargs': {}}" trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:146 2021-11-20 23:46:16.681 6217 DEBUG oslo_concurrency.lockutils [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:46:16.692 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: 2021-11-20 23:46:16.692 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Regex to get portals from discoverydb: ^SENDTARGETS: 2021-11-20 23:46:16.693 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Getting connected devices for (ips,iqns,luns)=[(u'192.168.1.21:3260', u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', 0)] _get_connection_devices /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:824 2021-11-20 23:46:16.713 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('-m', 'session'): stdout=tcp: [3] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) 2021-11-20 23:46:16.714 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsi session list stdout=tcp: [3] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) 2021-11-20 23:46:16.714 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Resulting device map defaultdict(<function <lambda> at 0x7f5c06bdc230>, {(u'192.168.1.21:3260', u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5'): (set([u'sdb']), set([]))}) _get_connection_devices /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:856 2021-11-20 23:46:16.715 6217 DEBUG os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Removing single pathed devices sdb remove_connection /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:301 2021-11-20 23:46:16.723 6217 ERROR os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] multipathd is not running: exit code 1: ProcessExecutionError: Unexpected error while running command. 2021-11-20 23:46:16.724 6217 DEBUG os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Flushing IO for device /dev/sdb flush_device_io /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:359 2021-11-20 23:46:16.732 6217 DEBUG os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete remove_scsi_device /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:76 2021-11-20 23:46:16.774 6217 DEBUG os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Checking to see if SCSI volumes sdb have been removed. wait_for_volumes_removal /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:84 2021-11-20 23:46:16.775 6217 DEBUG os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] SCSI volumes sdb have been removed. wait_for_volumes_removal /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:93 2021-11-20 23:46:16.777 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Disconnecting from: [(u'192.168.1.21:3260', u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5')] _disconnect_connection /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1116 2021-11-20 23:46:16.806 6217 WARNING nova.compute.manager [req-82b58e1c-81b4-4e8c-afc9-ea9de1e7ae5f 83460557e42b4c52a9c628d2d8521980 ce79774931944f0b8ec38d238a03a292 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Received unexpected event network-vif-unplugged-ffe535ab-d0d0-4deb-bf19-b3814f4ec54e for instance with vm_state active and task_state reboot_started. #for instance with vm_state active and task_state reboot_started. 2021-11-20 23:46:16.807 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:46:16.834 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--logout',): stdout=Logging out of session [sid: 3, target: iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5, portal: 192.168.1.21,3260] 2021-11-20 23:46:16.854 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:46:16.854 6217 DEBUG oslo_concurrency.lockutils [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: held 0.173s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 23:46:16.855 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] <== disconnect_volume: return (173ms) None trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:170 2021-11-20 23:46:16.855 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Disconnected iSCSI Volume disconnect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:78 2021-11-20 23:46:16.859 6217 DEBUG nova.virt.libvirt.driver [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Start _get_guest_xml network_info=[{"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null}] disk_info={'disk_bus': 'virtio', 'cdrom_bus': 'ide', 'mapping': {u'/dev/vda': {'bus': u'virtio', 'boot_index': '1', 'type': u'disk', 'dev': u'vda'}, 'root': {'bus': u'virtio', 'boot_index': '1', 'type': u'disk', 'dev': u'vda'}}} image_meta=ImageMeta(checksum=<?>,container_format='bare',created_at=<?>,direct_url=<?>,disk_format='qcow2',id=<?>,min_disk=1,min_ram=0,name=<?>,owner=<?>,properties=ImageMetaProps,protected=<?>,size=<?>,status=<?>,tags=<?>,updated_at=<?>,virtual_size=<?>,visibility=<?>) rescue=None block_device_info={'swap': None, 'root_device_name': u'/dev/vda', 'ephemerals': [], 'block_device_mapping': [{'guest_format': None, 'boot_index': 0, 'mount_device': u'/dev/vda', 'connection_info': {u'driver_volume_type': u'iscsi', u'connector': {u'initiator': u'iqn.1994-05.com.redhat:51f266169dc3', u'ip': u'192.168.1.23', u'system uuid': u'A9524D56-E47D-02DE-49ED-687D607FCF9B', u'platform': u'x86_64', u'host': u'compute23', u'do_local_attach': False, u'os_type': u'linux2', u'multipath': False}, u'serial': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'data': {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'auth_password': u'***', u'target_lun': 0, u'access_mode': u'rw', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}}, 'volume_type': None, 'disk_bus': u'virtio', 'device_type': u'disk', 'attachment_id': None, 'delete_on_termination': True}]} _get_guest_xml /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6154 2021-11-20 23:46:16.861 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Getting desirable topologies for flavor Flavor(created_at=2021-11-19T14:59:54Z,deleted=False,deleted_at=None,description=None,disabled=False,ephemeral_gb=0,extra_specs={},flavorid='1',id=1,is_public=True,memory_mb=512,name='m1.tiny',projects=<?>,root_gb=1,rxtx_factor=1.0,swap=0,updated_at=None,vcpu_weight=0,vcpus=1) and image_meta ImageMeta(checksum=<?>,container_format='bare',created_at=<?>,direct_url=<?>,disk_format='qcow2',id=<?>,min_disk=1,min_ram=0,name=<?>,owner=<?>,properties=ImageMetaProps,protected=<?>,size=<?>,status=<?>,tags=<?>,updated_at=<?>,virtual_size=<?>,visibility=<?>), allow threads: True _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:593 2021-11-20 23:46:16.862 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Flavor limits 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:337 2021-11-20 23:46:16.862 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Image limits 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:341 2021-11-20 23:46:16.863 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Flavor pref 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:377 2021-11-20 23:46:16.863 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Image pref 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:381 2021-11-20 23:46:16.863 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Chose sockets=0, cores=0, threads=0; limits were sockets=65536, cores=65536, threads=65536 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:421 2021-11-20 23:46:16.863 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Topology preferred VirtCPUTopology(cores=0,sockets=0,threads=0), maximum VirtCPUTopology(cores=65536,sockets=65536,threads=65536) _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:597 2021-11-20 23:46:16.864 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Build topologies for 1 vcpu(s) 1:1:1 _get_possible_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:460 2021-11-20 23:46:16.864 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Got 1 possible topologies _get_possible_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:487 2021-11-20 23:46:16.864 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Possible topologies [VirtCPUTopology(cores=1,sockets=1,threads=1)] _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:602 2021-11-20 23:46:16.865 6217 DEBUG nova.virt.hardware [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Sorted desired topologies [VirtCPUTopology(cores=1,sockets=1,threads=1)] _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:627 2021-11-20 23:46:16.865 6217 DEBUG nova.objects.instance [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'vcpu_model' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 23:46:16.874 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Calling os-brick to attach iSCSI Volume connect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:63 2021-11-20 23:46:16.875 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] ==> connect_volume: call u"{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f5c0c59f510>, {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'auth_password': u'***', u'target_lun': 0, u'access_mode': u'rw', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}), 'kwargs': {}}" trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:146 2021-11-20 23:46:16.875 6217 DEBUG oslo_concurrency.lockutils [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.connect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:46:16.876 6217 INFO os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Trying to connect to iSCSI portal 192.168.1.21:3260 2021-11-20 23:46:16.884 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm (): stdout= stderr=iscsiadm: No records found 2021-11-20 23:46:16.893 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.1.21,3260,-1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5] added 2021-11-20 23:46:16.900 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:46:16.907 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', u'CHAP'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:46:16.916 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', u'HdxMiUcXLrwPMoArwn2z'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:46:16.924 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', u'***'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:46:16.930 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. 2021-11-20 23:46:16.931 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions. 2021-11-20 23:46:16.931 6217 WARNING os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. 2021-11-20 23:46:16.983 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5, portal: 192.168.1.21,3260] (multiple) 2021-11-20 23:46:16.990 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:46:16.998 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('-m', 'session'): stdout=tcp: [4] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) 2021-11-20 23:46:16.998 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsi session list stdout=tcp: [4] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) 2021-11-20 23:46:16.998 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Connected to 192.168.1.21:3260 _connect_vol /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:646 2021-11-20 23:46:16.999 6217 DEBUG os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] HCTL (u'6', '-', '-', 0) found on session 4 with lun 0 get_hctl /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:670 2021-11-20 23:46:16.999 6217 DEBUG os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Scanning host 6 c: -, t: -, l: 0) scan_iscsi /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:698 2021-11-20 23:46:17.021 6217 DEBUG os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Searching for a device in session 4 and hctl [u'6', '*', '*', 0] yield: None device_name_by_hctl /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:690 2021-11-20 23:46:18.022 6217 DEBUG os_brick.initiator.linuxscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Searching for a device in session 4 and hctl [u'6', '*', '*', 0] yield: sdb device_name_by_hctl /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:690 2021-11-20 23:46:18.022 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Connected to sdb using {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'access_mode': u'rw', u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_lun': 0, u'auth_password': u'***', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'} _connect_vol /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:675 2021-11-20 23:46:18.023 6217 DEBUG oslo_concurrency.lockutils [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.connect_volume" :: held 1.148s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 23:46:18.023 6217 DEBUG os_brick.initiator.connectors.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] <== connect_volume: return (1148ms) {'path': u'/dev/sdb', 'scsi_wwn': u'36001405775e9e56a27b49c2b004f84eb', 'type': u'block'} trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:170 2021-11-20 23:46:18.024 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Attached iSCSI volume {'path': u'/dev/sdb', 'scsi_wwn': '36001405775e9e56a27b49c2b004f84eb', 'type': 'block'} connect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:65 2021-11-20 23:46:18.040 6217 DEBUG nova.virt.libvirt.vif [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2021-11-20T01:16:22Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description='test1',display_name='test1',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),hidden=False,host='compute23',hostname='test1',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2021-11-20T01:16:45Z,launched_on='compute23',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='compute23',numa_topology=None,old_flavor=None,os_type=None,pci_devices=PciDeviceList,pci_requests=<?>,power_state=1,progress=0,project_id='f911df6b22d54d34b716efd7a97edf92',ramdisk_id='',reservation_id='r-cnbmtisc',resources=None,root_device_name='/dev/vda',root_gb=1,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='admin,reader,member',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='reboot_started',terminated_at=None,trusted_certs=<?>,updated_at=2021-11-20T15:44:15Z,user_data=None,user_id='80f373269d7a481b931a9352b42b28b8',uuid=b78ce7ba-268d-40dd-88c8-4b516e14e4a3,vcpu_model=VirtCPUModel,vcpus=1,vm_mode=None,vm_state='active') vif={"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} virt_type=qemu get_config /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:601 2021-11-20 23:46:18.041 6217 DEBUG nova.network.os_vif_util [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converting VIF {"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-11-20 23:46:18.041 6217 DEBUG nova.network.os_vif_util [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converted object VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-11-20 23:46:18.045 6217 DEBUG nova.virt.libvirt.driver [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] End _get_guest_xml xml=<domain type="qemu"> 2021-11-20 23:46:18.046 6217 DEBUG nova.virt.libvirt.driver [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] skipping disk /dev/sdb (vda) as it is a volume _get_instance_disk_info_from_config /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:9586 2021-11-20 23:46:18.047 6217 DEBUG nova.virt.libvirt.vif [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2021-11-20T01:16:22Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description='test1',display_name='test1',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),hidden=False,host='compute23',hostname='test1',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2021-11-20T01:16:45Z,launched_on='compute23',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='compute23',numa_topology=None,old_flavor=None,os_type=None,pci_devices=PciDeviceList,pci_requests=<?>,power_state=1,progress=0,project_id='f911df6b22d54d34b716efd7a97edf92',ramdisk_id='',reservation_id='r-cnbmtisc',resources=None,root_device_name='/dev/vda',root_gb=1,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='admin,reader,member',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='reboot_started',terminated_at=None,trusted_certs=<?>,updated_at=2021-11-20T15:44:15Z,user_data=None,user_id='80f373269d7a481b931a9352b42b28b8',uuid=b78ce7ba-268d-40dd-88c8-4b516e14e4a3,vcpu_model=VirtCPUModel,vcpus=1,vm_mode=None,vm_state='active') vif={"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} plug /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:750 2021-11-20 23:46:18.047 6217 DEBUG nova.network.os_vif_util [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converting VIF {"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-11-20 23:46:18.048 6217 DEBUG nova.network.os_vif_util [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converted object VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-11-20 23:46:18.048 6217 DEBUG os_vif [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Plugging vif VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') plug /usr/lib/python2.7/site-packages/os_vif/__init__.py:76 2021-11-20 23:46:18.062 6217 INFO os_vif [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Successfully plugged vif VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') 2021-11-20 23:46:18.382 6217 INFO nova.virt.libvirt.driver [-] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance rebooted successfully. #Instance rebooted successfully. 2021-11-20 23:46:18.384 6217 DEBUG nova.compute.manager [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 23:46:18.418 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Synchronizing instance power state after lifecycle event "Resumed"; current vm_state: active, current task_state: reboot_started, current DB power_state: 1, VM power_state: 1 handle_lifecycle_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:1250 2021-11-20 23:46:18.482 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] During sync_power_state the instance has a pending task (reboot_started). Skip. 2021-11-20 23:46:18.537 6217 DEBUG oslo_concurrency.lockutils [req-7d512ebe-1c3d-47ee-86a3-0c5faa29ad9c 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "b78ce7ba-268d-40dd-88c8-4b516e14e4a3" released by "nova.compute.manager.do_reboot_instance" :: held 123.919s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339
hard reboot日志 ================================================================================ 1.nova-api接收到hard reboot请求;nova-api做一些必要的处理后，将请求放入rabbitMQ（没找到相应的放入rabbitMQ的请求的日志，因为reboot方法中没有输出log） 2.nova-compute从rabbitMQ获取请求，执行操作 ------------------------------------------------------------------------------------ hard reboot日志 1.nova-api接收到hard reboot请求;nova-api做一些必要的处理后，将请求放入rabbitMQ（没找到相应的放入rabbitMQ的请求的日志，因为reboot方法中没有输出log） 2021-11-20 23:54:22.697 15854 DEBUG nova.api.openstack.wsgi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Action: 'action', calling method: <bound method ServersController._action_reboot of <nova.api.openstack.compute.servers.ServersController object at 0x7fe6428ec710>>, body: {"reboot": {"type": "HARD"}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:520 #body: {"reboot": {"type": "HARD"}} 2021-11-20 23:54:22.699 15854 DEBUG nova.compute.api [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2654 2021-11-20 23:54:22.704 15854 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:54:22.704 15854 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 23:54:22.705 15854 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:54:22.705 15854 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 23:54:22.826 15854 ERROR oslo.messaging._drivers.impl_rabbit [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [f563d8da-cecf-41a1-8dcb-f581913f143c] AMQP server on 192.168.1.21:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe 2021-11-20 23:54:23.838 15854 INFO oslo.messaging._drivers.impl_rabbit [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [f563d8da-cecf-41a1-8dcb-f581913f143c] Reconnected to AMQP server on 192.168.1.21:5672 via [amqp] client with port 53230. 2021-11-20 23:54:23.840 15854 INFO nova.api.openstack.requestlog [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] 192.168.1.21 "POST /v2.1/f911df6b22d54d34b716efd7a97edf92/servers/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/action" status: 202 len: 0 microversion: 2.1 time: 1.145367 #这个应该是相应的硬重启的POST请求 2.nova-compute从rabbitMQ获取请求，执行操作 [root@compute23 ~]# cat /var/log/nova/nova-compute.log |grep -P "req-8246fa64-52b5-473e-b928-d7904f6a364a|reboot" 2021-11-20 23:54:23.886 6217 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "b78ce7ba-268d-40dd-88c8-4b516e14e4a3" acquired by "nova.compute.manager.do_reboot_instance" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:54:23.887 6217 INFO nova.compute.manager [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Rebooting instance #Rebooting instance 2021-11-20 23:54:23.905 6217 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Acquired lock "refresh_cache-b78ce7ba-268d-40dd-88c8-4b516e14e4a3" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:265 2021-11-20 23:54:23.906 6217 DEBUG nova.network.neutronv2.api [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Building network info cache for instance _get_instance_nw_info /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1758 2021-11-20 23:54:25.025 6217 DEBUG nova.network.base_api [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Updating instance_info_cache with network_info: [{"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null}] update_instance_cache_with_nw_info /usr/lib/python2.7/site-packages/nova/network/base_api.py:47 2021-11-20 23:54:25.052 6217 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Releasing lock "refresh_cache-b78ce7ba-268d-40dd-88c8-4b516e14e4a3" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:281 2021-11-20 23:54:25.055 6217 DEBUG nova.compute.manager [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 23:54:25.504 6217 DEBUG nova.objects.instance [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'resources' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 23:54:25.519 6217 DEBUG nova.virt.libvirt.vif [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2021-11-20T01:16:22Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description='test1',display_name='test1',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),hidden=False,host='compute23',hostname='test1',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2021-11-20T01:16:45Z,launched_on='compute23',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='compute23',numa_topology=None,old_flavor=None,os_type=None,pci_devices=<?>,pci_requests=<?>,power_state=1,progress=0,project_id='f911df6b22d54d34b716efd7a97edf92',ramdisk_id='',reservation_id='r-cnbmtisc',resources=None,root_device_name='/dev/vda',root_gb=1,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='admin,reader,member',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='reboot_started_hard',terminated_at=None,trusted_certs=<?>,updated_at=2021-11-20T15:54:25Z,user_data=None,user_id='80f373269d7a481b931a9352b42b28b8',uuid=b78ce7ba-268d-40dd-88c8-4b516e14e4a3,vcpu_model=<?>,vcpus=1,vm_mode=None,vm_state='active') vif={"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} unplug /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:868 2021-11-20 23:54:25.520 6217 DEBUG nova.network.os_vif_util [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converting VIF {"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-11-20 23:54:25.520 6217 DEBUG nova.network.os_vif_util [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converted object VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-11-20 23:54:25.521 6217 DEBUG os_vif [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Unplugging vif VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') unplug /usr/lib/python2.7/site-packages/os_vif/__init__.py:109 2021-11-20 23:54:25.544 6217 INFO os_vif [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Successfully unplugged vif VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') 2021-11-20 23:54:25.546 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] calling os-brick to detach iSCSI Volume disconnect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:72 2021-11-20 23:54:25.547 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] ==> disconnect_volume: call u"{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f5c0c59f510>, {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'access_mode': u'rw', u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_lun': 0, u'auth_password': u'***', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}, None), 'kwargs': {}}" trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:146 2021-11-20 23:54:25.547 6217 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:54:25.557 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: 2021-11-20 23:54:25.558 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Regex to get portals from discoverydb: ^SENDTARGETS: 2021-11-20 23:54:25.558 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Getting connected devices for (ips,iqns,luns)=[(u'192.168.1.21:3260', u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', 0)] _get_connection_devices /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:824 2021-11-20 23:54:25.576 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('-m', 'session'): stdout=tcp: [4] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) 2021-11-20 23:54:25.577 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsi session list stdout=tcp: [4] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) 2021-11-20 23:54:25.577 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Resulting device map defaultdict(<function <lambda> at 0x7f5c06bdc578>, {(u'192.168.1.21:3260', u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5'): (set([u'sdb']), set([]))}) _get_connection_devices /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:856 2021-11-20 23:54:25.578 6217 DEBUG os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Removing single pathed devices sdb remove_connection /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:301 2021-11-20 23:54:25.586 6217 ERROR os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] multipathd is not running: exit code 1: ProcessExecutionError: Unexpected error while running command. 2021-11-20 23:54:25.587 6217 DEBUG os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Flushing IO for device /dev/sdb flush_device_io /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:359 2021-11-20 23:54:25.593 6217 DEBUG os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete remove_scsi_device /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:76 2021-11-20 23:54:25.653 6217 DEBUG os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Checking to see if SCSI volumes sdb have been removed. wait_for_volumes_removal /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:84 2021-11-20 23:54:25.654 6217 DEBUG os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] SCSI volumes sdb have been removed. wait_for_volumes_removal /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:93 2021-11-20 23:54:25.654 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Disconnecting from: [(u'192.168.1.21:3260', u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5')] _disconnect_connection /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1116 2021-11-20 23:54:25.684 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:54:25.710 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--logout',): stdout=Logging out of session [sid: 4, target: iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5, portal: 192.168.1.21,3260] 2021-11-20 23:54:25.729 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:54:25.730 6217 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: held 0.182s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 23:54:25.730 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] <== disconnect_volume: return (183ms) None trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:170 2021-11-20 23:54:25.730 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Disconnected iSCSI Volume disconnect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:78 2021-11-20 23:54:25.735 6217 DEBUG nova.virt.libvirt.driver [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Start _get_guest_xml network_info=[{"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null}] disk_info={'disk_bus': 'virtio', 'cdrom_bus': 'ide', 'mapping': {u'/dev/vda': {'bus': u'virtio', 'boot_index': '1', 'type': u'disk', 'dev': u'vda'}, 'root': {'bus': u'virtio', 'boot_index': '1', 'type': u'disk', 'dev': u'vda'}}} image_meta=ImageMeta(checksum=<?>,container_format='bare',created_at=<?>,direct_url=<?>,disk_format='qcow2',id=<?>,min_disk=1,min_ram=0,name=<?>,owner=<?>,properties=ImageMetaProps,protected=<?>,size=<?>,status=<?>,tags=<?>,updated_at=<?>,virtual_size=<?>,visibility=<?>) rescue=None block_device_info={'swap': None, 'root_device_name': u'/dev/vda', 'ephemerals': [], 'block_device_mapping': [{'guest_format': None, 'boot_index': 0, 'mount_device': u'/dev/vda', 'connection_info': {u'driver_volume_type': u'iscsi', u'connector': {u'initiator': u'iqn.1994-05.com.redhat:51f266169dc3', u'ip': u'192.168.1.23', u'system uuid': u'A9524D56-E47D-02DE-49ED-687D607FCF9B', u'platform': u'x86_64', u'host': u'compute23', u'do_local_attach': False, u'os_type': u'linux2', u'multipath': False}, u'serial': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'data': {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'access_mode': u'rw', u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_lun': 0, u'auth_password': u'***', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}}, 'volume_type': None, 'disk_bus': u'virtio', 'device_type': u'disk', 'attachment_id': None, 'delete_on_termination': True}]} _get_guest_xml /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6154 2021-11-20 23:54:25.738 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Getting desirable topologies for flavor Flavor(created_at=2021-11-19T14:59:54Z,deleted=False,deleted_at=None,description=None,disabled=False,ephemeral_gb=0,extra_specs={},flavorid='1',id=1,is_public=True,memory_mb=512,name='m1.tiny',projects=<?>,root_gb=1,rxtx_factor=1.0,swap=0,updated_at=None,vcpu_weight=0,vcpus=1) and image_meta ImageMeta(checksum=<?>,container_format='bare',created_at=<?>,direct_url=<?>,disk_format='qcow2',id=<?>,min_disk=1,min_ram=0,name=<?>,owner=<?>,properties=ImageMetaProps,protected=<?>,size=<?>,status=<?>,tags=<?>,updated_at=<?>,virtual_size=<?>,visibility=<?>), allow threads: True _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:593 2021-11-20 23:54:25.739 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Flavor limits 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:337 2021-11-20 23:54:25.740 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Image limits 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:341 2021-11-20 23:54:25.740 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Flavor pref 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:377 2021-11-20 23:54:25.740 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Image pref 0:0:0 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:381 2021-11-20 23:54:25.741 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Chose sockets=0, cores=0, threads=0; limits were sockets=65536, cores=65536, threads=65536 get_cpu_topology_constraints /usr/lib/python2.7/site-packages/nova/virt/hardware.py:421 2021-11-20 23:54:25.741 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Topology preferred VirtCPUTopology(cores=0,sockets=0,threads=0), maximum VirtCPUTopology(cores=65536,sockets=65536,threads=65536) _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:597 2021-11-20 23:54:25.742 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Build topologies for 1 vcpu(s) 1:1:1 _get_possible_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:460 2021-11-20 23:54:25.742 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Got 1 possible topologies _get_possible_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:487 2021-11-20 23:54:25.742 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Possible topologies [VirtCPUTopology(cores=1,sockets=1,threads=1)] _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:602 2021-11-20 23:54:25.743 6217 DEBUG nova.virt.hardware [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Sorted desired topologies [VirtCPUTopology(cores=1,sockets=1,threads=1)] _get_desirable_cpu_topologies /usr/lib/python2.7/site-packages/nova/virt/hardware.py:627 2021-11-20 23:54:25.743 6217 DEBUG nova.objects.instance [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'vcpu_model' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 23:54:25.778 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Calling os-brick to attach iSCSI Volume connect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:63 2021-11-20 23:54:25.779 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] ==> connect_volume: call u"{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f5c0c59f510>, {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'access_mode': u'rw', u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_lun': 0, u'auth_password': u'***', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'}), 'kwargs': {}}" trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:146 2021-11-20 23:54:25.779 6217 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.connect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-20 23:54:25.780 6217 INFO os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Trying to connect to iSCSI portal 192.168.1.21:3260 2021-11-20 23:54:25.793 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm (): stdout= stderr=iscsiadm: No records found 2021-11-20 23:54:25.802 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.1.21,3260,-1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5] added 2021-11-20 23:54:25.812 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:54:25.822 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', u'CHAP'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:54:25.832 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', u'HdxMiUcXLrwPMoArwn2z'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:54:25.843 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', u'***'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:54:25.855 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. 2021-11-20 23:54:25.855 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions. 2021-11-20 23:54:25.855 6217 WARNING os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. 2021-11-20 23:54:25.920 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5, portal: 192.168.1.21,3260] (multiple) 2021-11-20 23:54:25.928 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= _run_iscsiadm /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1012 2021-11-20 23:54:25.936 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsiadm ('-m', 'session'): stdout=tcp: [5] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) 2021-11-20 23:54:25.937 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] iscsi session list stdout=tcp: [5] 192.168.1.21:3260,1 iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5 (non-flash) 2021-11-20 23:54:25.937 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Connected to 192.168.1.21:3260 _connect_vol /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:646 2021-11-20 23:54:25.937 6217 DEBUG os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] HCTL (u'7', '-', '-', 0) found on session 5 with lun 0 get_hctl /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:670 2021-11-20 23:54:25.937 6217 DEBUG os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Scanning host 7 c: -, t: -, l: 0) scan_iscsi /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:698 2021-11-20 23:54:26.021 6217 DEBUG os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Searching for a device in session 5 and hctl [u'7', '*', '*', 0] yield: None device_name_by_hctl /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:690 2021-11-20 23:54:27.021 6217 WARNING nova.compute.manager [req-2ac92028-8f04-4bee-b9d8-ed9f9d8883c0 83460557e42b4c52a9c628d2d8521980 ce79774931944f0b8ec38d238a03a292 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Received unexpected event network-vif-unplugged-ffe535ab-d0d0-4deb-bf19-b3814f4ec54e for instance with vm_state active and task_state reboot_started_hard. #for instance with vm_state active and task_state reboot_started_hard. 2021-11-20 23:54:27.022 6217 DEBUG os_brick.initiator.linuxscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Searching for a device in session 5 and hctl [u'7', '*', '*', 0] yield: sdb device_name_by_hctl /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:690 2021-11-20 23:54:27.022 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Connected to sdb using {u'device_path': u'/dev/sdb', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'target_portal': u'192.168.1.21:3260', u'volume_id': u'712ac45e-68c3-47f9-845e-e5e0b38fe9f5', u'auth_password': u'***', u'target_lun': 0, u'access_mode': u'rw', u'auth_username': u'HdxMiUcXLrwPMoArwn2z', u'auth_method': u'CHAP'} _connect_vol /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:675 2021-11-20 23:54:27.023 6217 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.connect_volume" :: held 1.244s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-20 23:54:27.023 6217 DEBUG os_brick.initiator.connectors.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] <== connect_volume: return (1244ms) {'path': u'/dev/sdb', 'scsi_wwn': u'36001405775e9e56a27b49c2b004f84eb', 'type': u'block'} trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:170 2021-11-20 23:54:27.023 6217 DEBUG nova.virt.libvirt.volume.iscsi [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Attached iSCSI volume {'path': u'/dev/sdb', 'scsi_wwn': '36001405775e9e56a27b49c2b004f84eb', 'type': 'block'} connect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:65 2021-11-20 23:54:27.043 6217 DEBUG nova.virt.libvirt.vif [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2021-11-20T01:16:22Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description='test1',display_name='test1',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),hidden=False,host='compute23',hostname='test1',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2021-11-20T01:16:45Z,launched_on='compute23',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='compute23',numa_topology=None,old_flavor=None,os_type=None,pci_devices=<?>,pci_requests=<?>,power_state=1,progress=0,project_id='f911df6b22d54d34b716efd7a97edf92',ramdisk_id='',reservation_id='r-cnbmtisc',resources=None,root_device_name='/dev/vda',root_gb=1,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='admin,reader,member',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='reboot_started_hard',terminated_at=None,trusted_certs=<?>,updated_at=2021-11-20T15:54:25Z,user_data=None,user_id='80f373269d7a481b931a9352b42b28b8',uuid=b78ce7ba-268d-40dd-88c8-4b516e14e4a3,vcpu_model=VirtCPUModel,vcpus=1,vm_mode=None,vm_state='active') vif={"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} virt_type=qemu get_config /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:601 2021-11-20 23:54:27.043 6217 DEBUG nova.network.os_vif_util [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converting VIF {"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-11-20 23:54:27.044 6217 DEBUG nova.network.os_vif_util [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converted object VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-11-20 23:54:27.048 6217 DEBUG nova.objects.instance [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'pci_devices' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-20 23:54:27.059 6217 DEBUG nova.virt.libvirt.driver [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] End _get_guest_xml xml=<domain type="qemu"> 2021-11-20 23:54:27.059 6217 DEBUG nova.virt.libvirt.driver [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] skipping disk /dev/sdb (vda) as it is a volume _get_instance_disk_info_from_config /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:9586 2021-11-20 23:54:27.060 6217 DEBUG nova.virt.libvirt.vif [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2021-11-20T01:16:22Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description='test1',display_name='test1',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),hidden=False,host='compute23',hostname='test1',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2021-11-20T01:16:45Z,launched_on='compute23',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='compute23',numa_topology=None,old_flavor=None,os_type=None,pci_devices=PciDeviceList,pci_requests=<?>,power_state=1,progress=0,project_id='f911df6b22d54d34b716efd7a97edf92',ramdisk_id='',reservation_id='r-cnbmtisc',resources=None,root_device_name='/dev/vda',root_gb=1,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='admin,reader,member',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='reboot_started_hard',terminated_at=None,trusted_certs=<?>,updated_at=2021-11-20T15:54:25Z,user_data=None,user_id='80f373269d7a481b931a9352b42b28b8',uuid=b78ce7ba-268d-40dd-88c8-4b516e14e4a3,vcpu_model=VirtCPUModel,vcpus=1,vm_mode=None,vm_state='active') vif={"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} plug /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:750 2021-11-20 23:54:27.061 6217 DEBUG nova.network.os_vif_util [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converting VIF {"profile": {}, "ovs_interfaceid": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.24.4.188"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "172.24.4.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.24.4.1"}}], "meta": {"injected": false, "tunneled": false, "tenant_id": "f911df6b22d54d34b716efd7a97edf92", "physical_network": "extnet", "mtu": 1500}, "id": "a1c83cdf-971f-4d3c-abc5-45f87f6746d4", "label": "public"}, "devname": "tapffe535ab-d0", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:2b:6e:05", "active": true, "type": "ovs", "id": "ffe535ab-d0d0-4deb-bf19-b3814f4ec54e", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-11-20 23:54:27.061 6217 DEBUG nova.network.os_vif_util [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Converted object VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-11-20 23:54:27.062 6217 DEBUG os_vif [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Plugging vif VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') plug /usr/lib/python2.7/site-packages/os_vif/__init__.py:76 2021-11-20 23:54:27.078 6217 INFO os_vif [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Successfully plugged vif VIFOpenVSwitch(active=True,address=fa:16:3e:2b:6e:05,bridge_name='br-int',has_traffic_filtering=True,id=ffe535ab-d0d0-4deb-bf19-b3814f4ec54e,network=Network(a1c83cdf-971f-4d3c-abc5-45f87f6746d4),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapffe535ab-d0') 2021-11-20 23:54:27.405 6217 INFO nova.virt.libvirt.driver [-] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Instance rebooted successfully. #Instance rebooted successfully. 2021-11-20 23:54:27.406 6217 DEBUG nova.compute.manager [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-20 23:54:27.421 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Synchronizing instance power state after lifecycle event "Resumed"; current vm_state: active, current task_state: reboot_started_hard, current DB power_state: 1, VM power_state: 1 handle_lifecycle_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:1250 2021-11-20 23:54:27.459 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] During sync_power_state the instance has a pending task (reboot_started_hard). Skip. 2021-11-20 23:54:27.510 6217 DEBUG oslo_concurrency.lockutils [req-8246fa64-52b5-473e-b928-d7904f6a364a 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "b78ce7ba-268d-40dd-88c8-4b516e14e4a3" released by "nova.compute.manager.do_reboot_instance" :: held 3.623s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339
Lock/Unlock日志 对被加锁（Lock）的 instance 执行重启等改变状态的操作会提示操作不允许。执行解锁（Unlock）操作后恢复正常。 Lock/Unlock 操作都是在 nova-api 中进行的。 操作成功后 nova-api 会更新 instance 加锁的状态。 执行其他操作时，nova-api 根据加锁状态来判断是否允许。 Lock/Unlock 不需要 nova-compute 的参与。 1. admin 角色的用户不受 lock 的影响，及无论加锁与否都可以正常执行操作。 2. 根据默认 policy 的配置，任何用户都可以 unlock。也就是说如果发现 instance 被加锁了，可以通过 unlock 解锁，然后在执行操作。 =============================================================================== lock日志： [root@controller ~]# cat /var/log/nova/nova-api.log | grep "req-797a9e40-97ee-4845-9aa9-0e5a18799cd4" 2021-11-21 15:57:13.770 106090 DEBUG nova.api.openstack.wsgi [req-797a9e40-97ee-4845-9aa9-0e5a18799cd4 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Action: 'action', calling method: <bound method LockServerController._lock of <nova.api.openstack.compute.lock_server.LockServerController object at 0x7fe6428a1e10>>, body: {"lock": null} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:520 #wsgi收到了lock请求 2021-11-21 15:57:13.770 106090 DEBUG nova.compute.api [req-797a9e40-97ee-4845-9aa9-0e5a18799cd4 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2654 2021-11-21 15:57:13.775 106090 DEBUG oslo_concurrency.lockutils [req-797a9e40-97ee-4845-9aa9-0e5a18799cd4 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-21 15:57:13.775 106090 DEBUG oslo_concurrency.lockutils [req-797a9e40-97ee-4845-9aa9-0e5a18799cd4 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-21 15:57:13.776 106090 DEBUG oslo_concurrency.lockutils [req-797a9e40-97ee-4845-9aa9-0e5a18799cd4 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-21 15:57:13.776 106090 DEBUG oslo_concurrency.lockutils [req-797a9e40-97ee-4845-9aa9-0e5a18799cd4 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-21 15:57:13.839 106090 DEBUG nova.compute.api [req-797a9e40-97ee-4845-9aa9-0e5a18799cd4 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Locking lock /usr/lib/python2.7/site-packages/nova/compute/api.py:4100 2021-11-21 15:57:13.911 106090 DEBUG nova.objects.instance [req-797a9e40-97ee-4845-9aa9-0e5a18799cd4 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-21 15:57:13.971 106090 INFO nova.api.openstack.requestlog [req-797a9e40-97ee-4845-9aa9-0e5a18799cd4 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] 192.168.1.21 "POST /v2.1/f911df6b22d54d34b716efd7a97edf92/servers/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/action" status: 202 len: 0 microversion: 2.42 time: 0.382271 #这个看来只是nova.api响应的日志。 unlock日志： [root@controller ~]# cat /var/log/nova/nova-api.log | grep "req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3" 2021-11-21 15:57:37.777 106093 DEBUG nova.api.openstack.wsgi [req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Action: 'action', calling method: <bound method LockServerController._unlock of <nova.api.openstack.compute.lock_server.LockServerController object at 0x7fe6428a1e10>>, body: {"unlock": null} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:520 2021-11-21 15:57:37.778 106093 DEBUG nova.compute.api [req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2654 2021-11-21 15:57:37.785 106093 DEBUG oslo_concurrency.lockutils [req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-21 15:57:37.786 106093 DEBUG oslo_concurrency.lockutils [req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-21 15:57:37.786 106093 DEBUG oslo_concurrency.lockutils [req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-21 15:57:37.786 106093 DEBUG oslo_concurrency.lockutils [req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-21 15:57:37.841 106093 DEBUG nova.compute.api [req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Unlocking unlock /usr/lib/python2.7/site-packages/nova/compute/api.py:4129 2021-11-21 15:57:37.896 106093 DEBUG nova.objects.instance [req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-21 15:57:37.959 106093 INFO nova.api.openstack.requestlog [req-32590d15-7426-4d5b-9b47-ae2e5b8b1de3 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] 192.168.1.21 "POST /v2.1/f911df6b22d54d34b716efd7a97edf92/servers/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/action" status: 202 len: 0 microversion: 2.42 time: 0.427677
Nova Pause/Resume；Pause/Resume操作与日志 ======================================================================================================= 通过 Pause 操作将 instance 的状态保存到宿主机的内存中。 当需要恢复的时候，执行 Resume 操作，从内存中读回 instance 的状态，然后继续运行 instance。 ------------------------------------------------------------------------------------------------------------------------ Pause操作与日志 1.nova-api接收到 Pause reboot请求;nova-api做一些必要的处理后，将请求放入rabbitMQ 2.nova-compute从rabbitMQ获取请求，执行操作 ------------------------------------------------------------------------------------------------------------------------ Pause操作与日志 1.nova-api接收到 Pause reboot请求;nova-api做一些必要的处理后，将请求放入rabbitMQ [root@controller ~]# cat /var/log/nova/nova-api.log |grep -i "Pause" [root@controller ~]# cat /var/log/nova/nova-api.log |grep -i "req-63133b0c-44a3-4048-9e51-2103ec8f8e3f" 2021-11-21 19:05:30.841 106093 DEBUG nova.api.openstack.wsgi [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Action: 'action', calling method: <bound method PauseServerController._pause of <nova.api.openstack.compute.pause_server.PauseServerController object at 0x7fe6428a5f90>>, body: {"pause": null} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:520 #接收到pause请求 2021-11-21 19:05:30.841 106093 DEBUG nova.compute.api [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2654 2021-11-21 19:05:30.846 106093 DEBUG oslo_concurrency.lockutils [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-21 19:05:30.847 106093 DEBUG oslo_concurrency.lockutils [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-21 19:05:30.847 106093 DEBUG oslo_concurrency.lockutils [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-21 19:05:30.847 106093 DEBUG oslo_concurrency.lockutils [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-21 19:05:30.910 106093 DEBUG nova.objects.instance [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-21 19:05:30.970 106093 DEBUG oslo_concurrency.lockutils [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Acquired lock "compute-rpcapi-router" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:265 2021-11-21 19:05:30.971 106093 DEBUG oslo_concurrency.lockutils [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Releasing lock "compute-rpcapi-router" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:281 2021-11-21 19:05:30.988 106093 INFO nova.api.openstack.requestlog [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] 192.168.1.21 "POST /v2.1/f911df6b22d54d34b716efd7a97edf92/servers/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/action" status: 202 len: 0 microversion: 2.1 time: 0.149349 #响应记录：关于“响应api POST请求” #/usr/lib/python2.7/site-packages/nova/compute/api.py #pause方法没有输出日志，那么如何通过日志找到pause方法呢？ @check_instance_lock @check_instance_state(vm_state=[vm_states.ACTIVE]) def pause(self, context, instance): """Pause the given instance.""" instance.task_state = task_states.PAUSING instance.save(expected_task_state=[None]) self._record_action_start(context, instance, instance_actions.PAUSE) self.compute_rpcapi.pause_instance(context, instance) 2.nova-compute从rabbitMQ获取请求，执行操作 [root@compute23 ~]# cat /var/log/nova/nova-compute.log |grep -i "req-63133b0c-44a3-4048-9e51-2103ec8f8e3f" -C10 2021-11-21 19:05:31.062 6217 INFO nova.compute.manager [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Pausing 2021-11-21 19:05:31.063 6217 DEBUG nova.objects.instance [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-21 19:05:31.126 6217 DEBUG nova.virt.driver [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] Emitting event <LifecycleEvent: 1637492731.13, b78ce7ba-268d-40dd-88c8-4b516e14e4a3 => Paused> emit_event /usr/lib/python2.7/site-packages/nova/virt/driver.py:1706 2021-11-21 19:05:31.126 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] VM Paused (Lifecycle Event) 2021-11-21 19:05:31.128 6217 DEBUG nova.compute.manager [req-63133b0c-44a3-4048-9e51-2103ec8f8e3f 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-21 19:05:31.161 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-21 19:05:31.164 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Synchronizing instance power state after lifecycle event "Paused"; current vm_state: active, current task_state: pausing, current DB power_state: 1, VM power_state: 3 handle_lifecycle_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:1250 #信息同步的日志：current task_state: pausing, current DB power_state: 1, VM power_state: 3 2021-11-21 19:05:31.200 6217 DEBUG nova.virt.driver [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] Emitting event <LifecycleEvent: 1637492731.13, b78ce7ba-268d-40dd-88c8-4b516e14e4a3 => Paused> emit_event /usr/lib/python2.7/site-packages/nova/virt/driver.py:1706 2021-11-21 19:05:31.200 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] VM Paused (Lifecycle Event) 2021-11-21 19:05:31.232 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-21 19:05:31.234 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Synchronizing instance power state after lifecycle event "Paused"; current vm_state: paused, current task_state: None, current DB power_state: 3, VM power_state: 3 handle_lifecycle_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:1250 #信息同步的日志：current task_state: None, current DB power_state: 3, VM power_state: 3 ======================================================================================================= Resume操作与日志 1.nova-api接收到 Resume reboot请求;nova-api做一些必要的处理后，将请求放入rabbitMQ 2.nova-compute从rabbitMQ获取请求，执行操作 -------------------------------------------------------------------------------------------------------------------- Resume操作与日志 1.nova-api接收到 Resume reboot请求;nova-api做一些必要的处理后，将请求放入rabbitMQ #注意，此处实际搜索"unpause" [root@controller ~]# cat /var/log/nova/nova-api.log |grep -i "unpause" [root@controller ~]# cat /var/log/nova/nova-api.log |grep -i "req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2" 2021-11-21 19:39:00.443 106092 DEBUG nova.api.openstack.wsgi [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Action: 'action', calling method: <bound method PauseServerController._unpause of <nova.api.openstack.compute.pause_server.PauseServerController object at 0x7fe6428a5f90>>, body: {"unpause": null} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:520 #接收到Resume reboot请求 2021-11-21 19:39:00.443 106092 DEBUG nova.compute.api [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2654 2021-11-21 19:39:00.449 106092 DEBUG oslo_concurrency.lockutils [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-21 19:39:00.449 106092 DEBUG oslo_concurrency.lockutils [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-21 19:39:00.450 106092 DEBUG oslo_concurrency.lockutils [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327 2021-11-21 19:39:00.450 106092 DEBUG oslo_concurrency.lockutils [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lock "4809eaf7-095b-41d7-a897-4a885c87aca9" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 2021-11-21 19:39:00.512 106092 DEBUG nova.objects.instance [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-21 19:39:00.567 106092 DEBUG oslo_concurrency.lockutils [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Acquired lock "compute-rpcapi-router" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:265 2021-11-21 19:39:00.568 106092 DEBUG oslo_concurrency.lockutils [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Releasing lock "compute-rpcapi-router" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:281 2021-11-21 19:39:00.587 106092 INFO nova.api.openstack.requestlog [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] 192.168.1.21 "POST /v2.1/f911df6b22d54d34b716efd7a97edf92/servers/b78ce7ba-268d-40dd-88c8-4b516e14e4a3/action" status: 202 len: 0 microversion: 2.1 time: 0.380877 @check_instance_lock @check_instance_state(vm_state=[vm_states.PAUSED]) def unpause(self, context, instance): """Unpause the given instance.""" instance.task_state = task_states.UNPAUSING instance.save(expected_task_state=[None]) self._record_action_start(context, instance, instance_actions.UNPAUSE) self.compute_rpcapi.unpause_instance(context, instance) 2.nova-compute从rabbitMQ获取请求，执行操作 [root@compute23 ~]# cat /var/log/nova/nova-compute.log | grep -i "req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2" 2021-11-21 19:39:00.602 6217 INFO nova.compute.manager [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Unpausing #Unpausing 2021-11-21 19:39:00.603 6217 DEBUG nova.objects.instance [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] Lazy-loading 'flavor' on Instance uuid b78ce7ba-268d-40dd-88c8-4b516e14e4a3 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1091 2021-11-21 19:39:00.773 6217 DEBUG nova.virt.driver [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] Emitting event <LifecycleEvent: 1637494740.77, b78ce7ba-268d-40dd-88c8-4b516e14e4a3 => Resumed> emit_event /usr/lib/python2.7/site-packages/nova/virt/driver.py:1706 2021-11-21 19:39:00.773 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] VM Resumed (Lifecycle Event) #VM Resumed 2021-11-21 19:39:00.803 6217 DEBUG nova.virt.libvirt.guest [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Failed to set time: agent not configured sync_guest_time /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:175 2021-11-21 19:39:00.803 6217 DEBUG nova.compute.manager [req-fa34dc01-feee-4d40-8132-b8c4eb3e1ff2 80f373269d7a481b931a9352b42b28b8 f911df6b22d54d34b716efd7a97edf92 - default default] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-21 19:39:00.852 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Checking state _get_power_state /usr/lib/python2.7/site-packages/nova/compute/manager.py:1505 2021-11-21 19:39:00.854 6217 DEBUG nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] Synchronizing instance power state after lifecycle event "Resumed"; current vm_state: paused, current task_state: unpausing, current DB power_state: 3, VM power_state: 1 handle_lifecycle_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:1250 2021-11-21 19:39:00.924 6217 INFO nova.compute.manager [req-2de1833c-68a6-4432-a436-222e47988697 - - - - -] [instance: b78ce7ba-268d-40dd-88c8-4b516e14e4a3] During sync_power_state the instance has a pending task (unpausing). Skip.
Suspend/Resume（略）；Suspend、Pause 对比 ==================================================================================== Suspend、Pause 对比 相同点： 两者都是暂停 instance 的运行，并保存当前状态，之后可以通过 Resume 操作恢复。 不同点： 1. Suspend 将 instance 的状态保存在磁盘；Pause 是保存在内存中，所以 Resume 被 Pause 的 instance 要比 Suspend 快。 2. instance 被 Suspend 后，状态为 Shut Down；而被 Pause 的 instance 状态是 Paused。 3. 虽然都是通过 Resume 操作恢复，Pause 对应的 Resume 在 OpenStack 内部被叫作 “Unpause”；Suspend 对应的 Resume 才是真正的 “Resume”。这个在日志中能体现出来。
instance 故障恢复的方法 （Nova Suspend/Rescue 操作详解 - 每天5分钟玩转 OpenStack（35）Rescue失败，碰壁了- -）
====================================================================================================
操作系统故障场景（Rescue/Unrescue）：
有时候由于误操作或者突然断电，操作系统重启后却起不来了。 为了最大限度挽救数据，我们通常会使用一张系统盘将系统引导起来，然后在尝试恢复。 问题如果不太严重，完全可以通过这种方式让系统重新正常工作。 比如某个系统文件意外删除， root 密码遗忘等。
Nova 也提供这种故障恢复机制，叫 Rescue。 （简单来说就是nova也提供rescue 救援模式）
Rescue 用指定的 image 作为启动盘引导 instance，将 instance 本身的系统盘作为第二个磁盘挂载到操作系统上。
nova help rescue
Reboots a server into rescue mode, which starts the machine from either the initial image or a specified image, attaching the current boot disk as secondary.
-----------------------------------------------------------------------------------------------------------
Rescue操作与日志
1.nova-api接收到 Rescue 请求;nova-api做一些必要的处理后，将请求放入rabbitMQ
2.nova-compute从rabbitMQ获取请求，执行操作
使用黑屏方式进行操作，记得source admin-openrc.sh
[root@controller ~]# nova rescue b78ce7ba-268d-40dd-88c8-4b516e14e4a3 #使用VM name应该也是可以的
ERROR (BadRequest): Instance b78ce7ba-268d-40dd-88c8-4b516e14e4a3 cannot be rescued: Cannot rescue a volume-backed instance (HTTP 400) (Request-ID: req-53863f47-23e6-4f33-ace8-fc21031811db)
-------------------------------------------------------------------------
以下待完成
033 - Terminate Instance 操作详解
035 - Suspend/Rescue Instance 操作详解
036 - Snapshot Instance 操作详解
037 - Rebuild Instance 操作详解
038 - Shelve Instance 操作详解
039 - Unshelve Instance 操作详解
040 - Migrate Instance 操作详解
041 - Resize Instance 操作详解
042 - Live Migrate 操作
043 - 计算节点宕机了怎么办？
044 - 1 张图秒懂 Nova 16 种操作
【推荐】国内首个AI IDE，深度理解中文开发场景，立即下载体验Trae
【推荐】编程新体验，更懂你的AI，立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包，你的智能百科全书，全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell：AI 加持，快人一步
· SQL Server 2025 AI相关能力初探
· Linux系列：如何用 C#调用 C方法造成内存泄露
· AI与.NET技术实操系列（二）：开始使用ML.NET
· 记一次.NET内存居高不下排查解决与启示
· 探究高空视频全景AR技术的实现原理
· 阿里最新开源QwQ-32B，效果媲美deepseek-r1满血版，部署成本又又又降低了！
· 单线程的Redis速度为什么快？
· SQL Server 2025 AI相关能力初探
· AI编程工具终极对决：字节Trae VS Cursor，谁才是开发者新宠？
· 展开说说关于C#中ORM框架的用法！