注册 Ironic 裸金属节点并部署裸金属实例
2019-05-11 11:53 云物互联 阅读(3591) 评论(0) 编辑 收藏 举报目录
文章目录
- 目录
- 前文列表
- 注册（Enrollment）裸机
- 创建裸金属实例的 Flavor
- 部署裸金属实例
- 日志分析
- 问题：Failed to create neutron ports for any PXE enabled port on node
- 问题：获取 Deploy Image 时 MissingAuthPlugin: An auth plugin is required to determine endpoint URL
- 问题：执行 provide 时长时间卡在 provision_state:clean wait
- 问题：ironic node 无法被调度
- 问题：获取 swift_temp_url 时 MissingAuthPlugin: An auth plugin is required to determine endpoint URL
- 问题：Timeout reached while waiting for callback for node
前文列表
《Ironic 裸金属管理服务》
《Ironic 裸金属管理服务的底层技术支撑》
《Ironic 裸金属实例的部署流程》
《Ironic 裸金属管理服务的网络模型》
《手动集成 Ironic 裸金属管理服务（Rocky）》
注册（Enrollment）裸机
首先我们要针对裸金属服务器的硬件设备来确定所需要使用到的 Ironic Driver 类型，这里我们选择使用比较通用的 IPMI。
[root@controller ~]# openstack baremetal driver show ipmi
+-------------------------------+----------------------------+
| Field | Value |
+-------------------------------+----------------------------+
| default_bios_interface | no-bios |
| default_boot_interface | pxe |
| default_console_interface | ipmitool-socat |
| default_deploy_interface | iscsi |
| default_inspect_interface | inspector |
| default_management_interface | ipmitool |
| default_network_interface | neutron |
| default_power_interface | ipmitool |
| default_raid_interface | agent |
| default_rescue_interface | no-rescue |
| default_storage_interface | noop |
| default_vendor_interface | ipmitool |
| enabled_bios_interfaces | no-bios |
| enabled_boot_interfaces | pxe |
| enabled_console_interfaces | ipmitool-socat, no-console |
| enabled_deploy_interfaces | iscsi, direct |
| enabled_inspect_interfaces | inspector |
| enabled_management_interfaces | ipmitool |
| enabled_network_interfaces | flat, neutron |
| enabled_power_interfaces | ipmitool |
| enabled_raid_interfaces | agent |
| enabled_rescue_interfaces | no-rescue |
| enabled_storage_interfaces | cinder, noop |
| enabled_vendor_interfaces | ipmitool, no-vendor |
| hosts | baremetal |
| name | ipmi |
| type | dynamic |
+-------------------------------+----------------------------+
下列打印出 IPMI 驱动程序的属性清单，这些属性都需要由云管人员提供。
[root@controller ~]# openstack baremetal driver property list ipmi
+------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Description |
+------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| deploy_forces_oob_reboot | Whether Ironic should force a reboot of the Node via the out-of-band channel after deployment is complete. Provides compatibility with older deploy ramdisks. Defaults to False. Optional. |
| deploy_kernel | UUID (from Glance) of the deployment kernel. Required. |
| deploy_ramdisk | UUID (from Glance) of the ramdisk that is mounted at boot time. Required. |
| force_persistent_boot_device | True to enable persistent behavior when the boot device is set during deploy and cleaning operations. Defaults to False. Optional. |
| ipmi_address | IP address or hostname of the node. Required. |
| ipmi_bridging | bridging_type; default is "no". One of "single", "dual", "no". Optional. |
| ipmi_disable_boot_timeout | By default ironic will send a raw IPMI command to disable the 60 second timeout for booting. Setting this option to False will NOT send that command on this node. The [ipmi]disable_boot_timeout will be used if this option is not set. Optional. |
| ipmi_force_boot_device | Whether Ironic should specify the boot device to the BMC each time the server is turned on, eg. because the BMC is not capable of remembering the selected boot device across power cycles; default value is False. Optional. |
| ipmi_local_address | local IPMB address for bridged requests. Used only if ipmi_bridging is set to "single" or "dual". Optional. |
| ipmi_password | password. Optional. |
| ipmi_port | remote IPMI RMCP port. Optional. |
| ipmi_priv_level | privilege level; default is ADMINISTRATOR. One of ADMINISTRATOR, CALLBACK, OPERATOR, USER. Optional. |
| ipmi_protocol_version | the version of the IPMI protocol; default is "2.0". One of "1.5", "2.0". Optional. |
| ipmi_target_address | destination address for bridged request. Required only if ipmi_bridging is set to "single" or "dual". |
| ipmi_target_channel | destination channel for bridged request. Required only if ipmi_bridging is set to "single" or "dual". |
| ipmi_terminal_port | node's UDP port to connect to. Only required for console access. |
| ipmi_transit_address | transit address for bridged request. Required only if ipmi_bridging is set to "dual". |
| ipmi_transit_channel | transit channel for bridged request. Required only if ipmi_bridging is set to "dual". |
| ipmi_username | username; default is NULL user. Optional. |
+------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
因为本文的环境是 Rocky 版本，所以我们可以手动指定更高的 API 版本。e.g.
In the examples below we will use version 1.11 of the Bare metal API. This gives us the following advantages:
- Explicit power credentials validation before leaving the enroll state.
- Running node cleaning before entering the available state.
- Not exposing half-configured nodes to the scheduler.
$ export IRONIC_API_VERSION=1.11
$ export OS_BAREMETAL_API_VERSION=1.11
- 创建一个执行 IPMI 的 ironic node
[root@controller ~]# openstack baremetal node create --help
usage: openstack baremetal node create [-h] [-f {json,shell,table,value,yaml}]
[-c COLUMN] [--max-width <integer>]
[--fit-width] [--print-empty]
[--noindent] [--prefix PREFIX]
[--chassis-uuid <chassis>] --driver
<driver> [--driver-info <key=value>]
[--property <key=value>]
[--extra <key=value>] [--uuid <uuid>]
[--name <name>]
[--bios-interface <bios_interface>]
[--boot-interface <boot_interface>]
[--console-interface <console_interface>]
[--deploy-interface <deploy_interface>]
[--inspect-interface <inspect_interface>]
[--management-interface <management_interface>]
[--network-interface <network_interface>]
[--power-interface <power_interface>]
[--raid-interface <raid_interface>]
[--rescue-interface <rescue_interface>]
[--storage-interface <storage_interface>]
[--vendor-interface <vendor_interface>]
[--resource-class <resource_class>]
[--conductor-group <conductor_group>]
[root@controller ~]# openstack baremetal node create --driver ipmi --name BM01
+------------------------+--------------------------------------+
| Field | Value |
+------------------------+--------------------------------------+
| chassis_uuid | None |
| clean_step | {} |
| console_enabled | False |
| created_at | 2019-05-09T07:51:38+00:00 |
| driver | ipmi |
| driver_info | {} |
| driver_internal_info | {} |
| extra | {} |
| inspection_finished_at | None |
| inspection_started_at | None |
| instance_info | {} |
| instance_uuid | None |
| last_error | None |
| maintenance | False |
| maintenance_reason | None |
| name | BM01 |
| power_state | None |
| properties | {} |
| provision_state | enroll |
| provision_updated_at | None |
| reservation | None |
| target_power_state | None |
| target_provision_state | None |
| updated_at | None |
| uuid | c1729b3f-9ada-4def-8dcb-43f919b9b997 |
+------------------------+--------------------------------------+
当前许多 ironic node info 都是 None，需要后续继续更新。
- 设置部署接口类型，现在可支持 iSCSI、Direct、Ansible 等类型，每种类型都有不同的行为模型，可根据实际情况选择，这里我们选择最简单的、但却在生产环境中并不特别友好（占用 Provisioning Network 的带宽）的 iSCS 类型。
openstack baremetal --os-baremetal-api-version 1.31 node set e322f49a-ad50-468d-a031-29bde068c290 \
--deploy-interface iscsi \
--raid-interface agent
- 设置 driver_info，这里即 IPMI info，主要是提供 IPMI 的登录账户信息。
openstack baremetal node set e322f49a-ad50-468d-a031-29bde068c290 \
--driver-info ipmi_username=admin \
--driver-info ipmi_password=admin \
--driver-info ipmi_address=172.18.22.106 \
--driver-info ipmi_port=623
NOTE：IPMI Driver 官方文档
- 设置 Deploy Images，通过 RAMDisk 的方式启动。
openstack baremetal node set e322f49a-ad50-468d-a031-29bde068c290 \
--driver-info deploy_kernel=e650d33b-8fad-42f7-948c-5c12526bcd07 \
--driver-info deploy_ramdisk=6000a17f-0ab7-418a-990c-2009a59c3392
- 设置 Provisioning/Cleaning Network。
openstack baremetal node set e322f49a-ad50-468d-a031-29bde068c290 \
--driver-info cleaning_network=b90fce07-0f32-4ba5-a1fd-a8e5e00f9c65 \
--driver-info provisioning_network=b90fce07-0f32-4ba5-a1fd-a8e5e00f9c65
- 设置 ironic node 的 PXE 网卡 MAC 地址，在 Provisioning Network 中通过这个 MAC 地址来为其分配 IP 地址。
openstack baremetal port create 2C:60:0C:6E:C2:A8 --node e322f49a-ad50-468d-a031-29bde068c290
NOTE：部署裸金属实例成功之后 PXE 网卡的 MAC 地址会被绑定到对应的 Tenant Network Port。
- 为 ironic node 设置 Placement 筛选候选人的 Resource Class 类型，nova-compute for Ironic 会自动为其创建 Placement Resource Provider。
$ openstack --os-baremetal-api-version 1.21 baremetal node set e322f49a-ad50-468d-a031-29bde068c290 \
--resource-class BAREMETAL_TEST
$ openstack resource provider list
+--------------------------------------+--------------------------------------+------------+--------------------------------------+----------------------+
| uuid | name | generation | root_provider_uuid | parent_provider_uuid |
+--------------------------------------+--------------------------------------+------------+--------------------------------------+----------------------+
| 841d70e5-c3b1-4ded-8bb2-60f4784f7a0d | controller | 23 | 841d70e5-c3b1-4ded-8bb2-60f4784f7a0d | None |
| da6bcc18-34a3-4ad0-9957-7b057fbb1bbc | compute | 34 | da6bcc18-34a3-4ad0-9957-7b057fbb1bbc | None |
| e322f49a-ad50-468d-a031-29bde068c290 | e322f49a-ad50-468d-a031-29bde068c290 | 1 | e322f49a-ad50-468d-a031-29bde068c290 | None |
+--------------------------------------+--------------------------------------+------------+--------------------------------------+----------------------+
$ openstack resource provider inventory list e322f49a-ad50-468d-a031-29bde068c290
+-----------------------+------------------+----------+----------+-----------+----------+-------+
| resource_class | allocation_ratio | max_unit | reserved | step_size | min_unit | total |
+-----------------------+------------------+----------+----------+-----------+----------+-------+
| VCPU | 1.0 | 2 | 2 | 1 | 1 | 2 |
| MEMORY_MB | 1.0 | 8192 | 8192 | 1 | 1 | 8192 |
| DISK_GB | 1.0 | 100 | 100 | 1 | 1 | 100 |
| CUSTOM_BAREMETAL_TEST | 1.0 | 1 | 1 | 1 | 1 | 1 |
+-----------------------+------------------+----------+----------+-----------+----------+-------+
- 为 ironic node 设置 Placement 筛选候选人的 Resource Traits 标签。
export OS_PLACEMENT_API_VERSION=1.17
openstack baremetal node add trait e322f49a-ad50-468d-a031-29bde068c290 \
CUSTOM_TRAIT1 HW_CPU_X86_VMX
[root@controller ~]# openstack resource provider trait list e322f49a-ad50-468d-a031-29bde068c290
+----------------+
| name |
+----------------+
| HW_CPU_X86_VMX |
| CUSTOM_TRAIT1 |
+----------------+
NOTE：这个操作需要较高的 Placement API 版本 <= 1.17
- 设置 ironic node 的基础资源信息，作为 Placement 筛选候选人的参数因子。
openstack baremetal node set e322f49a-ad50-468d-a031-29bde068c290 \
--property cpus=2 \
--property memory_mb=8192 \
--property local_gb=100
- 如果裸机服务器设定的是 UEFI，那么需要设置 ironic node 的 boot mode。
openstack baremetal node set e322f49a-ad50-468d-a031-29bde068c290 --property capabilities='boot_mode:uefi'
- 验证上述录入的 ironic node infos 是否合规。
$ openstack baremetal node validate e322f49a-ad50-468d-a031-29bde068c290
+------------+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Interface | Result | Reason |
+------------+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| bios | False | Driver ipmi does not support bios (disabled or not implemented). |
| boot | False | Cannot validate image information for node e322f49a-ad50-468d-a031-29bde068c290 because one or more parameters are missing from its instance_info and insufficent information is present to boot from a remote volume. Missing are: ['ramdisk', 'kernel', 'image_source'] |
| console | False | Missing 'ipmi_terminal_port' parameter in node's driver_info. |
| deploy | False | Cannot validate image information for node e322f49a-ad50-468d-a031-29bde068c290 because one or more parameters are missing from its instance_info and insufficent information is present to boot from a remote volume. Missing are: ['ramdisk', 'kernel', 'image_source'] |
| inspect | True | |
| management | True | |
| network | True | |
| power | True | |
| raid | True | |
| rescue | False | Driver ipmi does not support rescue (disabled or not implemented). |
| storage | True | |
+------------+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
NOTE：这里出现了 4 个 False，但没有致命影响，其中 bios、console 是因为我们没有提供相应的驱动，属于可选项。而 boot、deploy 在 Nova Driver for Ironic 的环境中是无法通过验证的。
- 验证 ironic node 是否能够被纳管。
# To move a node from enroll to manageable provision state
$ openstack baremetal --os-baremetal-api-version 1.11 node manage e322f49a-ad50-468d-a031-29bde068c290
$ openstack baremetal node show e322f49a-ad50-468d-a031-29bde068c290 | grep provision_state
| provision_state | manageable
# To move a node from manageable to available provision state
$ openstack baremetal --os-baremetal-api-version 1.11 node provide e322f49a-ad50-468d-a031-29bde068c290
$ [root@controller ~]# openstack baremetal node show e322f49a-ad50-468d-a031-29bde068c290 | grep provision_state
| provision_state | available
- 查看当前的 ironic node 状态
[root@controller ~]# openstack baremetal node show e322f49a-ad50-468d-a031-29bde068c290
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| bios_interface | no-bios |
| boot_interface | pxe |
| chassis_uuid | None |
| clean_step | {} |
| conductor_group | |
| console_enabled | False |
| console_interface | ipmitool-socat |
| created_at | 2019-05-09T10:46:33+00:00 |
| deploy_interface | iscsi |
| deploy_step | {} |
| driver | ipmi |
| driver_info | {u'ipmi_port': 623, u'ipmi_username': u'admin', u'deploy_kernel': u'e650d33b-8fad-42f7-948c-5c12526bcd07', u'ipmi_address': u'172.18.22.106', u'deploy_ramdisk': u'6000a17f-0ab7-418a-990c-2009a59c3392', u'ipmi_password': u'******', u'provisioning_network': u'b90fce07-0f32-4ba5-a1fd-a8e5e00f9c65', u'cleaning_network': u'b90fce07-0f32-4ba5-a1fd-a8e5e00f9c65'} |
| driver_internal_info | {u'agent_enable_ata_secure_erase': True, u'agent_erase_devices_iterations': 1, u'agent_erase_devices_zeroize': True, u'agent_continue_if_ata_erase_failed': False} |
| extra | {} |
| fault | None |
| inspect_interface | inspector |
| inspection_finished_at | None |
| inspection_started_at | None |
| instance_info | {} |
| instance_uuid | None |
| last_error | None |
| maintenance | False |
| maintenance_reason | None |
| management_interface | ipmitool |
| name | BM01 |
| network_interface | flat |
| power_interface | ipmitool |
| power_state | power off |
| properties | {u'memory_mb': 8192, u'local_gb': 100, u'cpus': 2, u'capabilities': u'boot_mode:uefi'} |
| provision_state | available |
| provision_updated_at | 2019-05-10T09:56:23+00:00 |
| raid_config | {} |
| raid_interface | agent |
| rescue_interface | no-rescue |
| reservation | None |
| resource_class | BAREMETAL_TEST |
| storage_interface | noop |
| target_power_state | None |
| target_provision_state | None |
| target_raid_config | {} |
| traits | [u'CUSTOM_TRAIT1', u'HW_CPU_X86_VMX'] |
| updated_at | 2019-05-10T10:22:41+00:00 |
| uuid | e322f49a-ad50-468d-a031-29bde068c290 |
| vendor_interface | ipmitool |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
- 验证，自动添加了 Ironic Neutron Agent。
[root@baremetal ~]# openstack network agent list
+--------------------------------------+--------------------+--------------------------------------+-------------------+-------+-------+---------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+--------------------+--------------------------------------+-------------------+-------+-------+---------------------------+
| 02ac17a4-9a27-4dd6-b11f-a6eada895432 | Open vSwitch agent | baremetal | None | :-) | UP | neutron-openvswitch-agent |
| 41925586-9119-4709-bc23-4668433bd413 | Metadata agent | controller | None | :-) | UP | neutron-metadata-agent |
| 43281ac1-7699-4a81-a5b6-d4818f8cf8f9 | Open vSwitch agent | controller | None | :-) | UP | neutron-openvswitch-agent |
| 7f879b42-4f93-4c36-b13e-e6cec004ce07 | Baremetal Node | e322f49a-ad50-468d-a031-29bde068c290 | None | :-) | UP | ironic-neutron-agent |
| b815e569-c85d-4a37-84ea-7bdc5fe5653c | DHCP agent | controller | nova | :-) | UP | neutron-dhcp-agent |
| d1ef7214-d26c-42c8-ba0b-2a1580a44446 | L3 agent | controller | nova | :-) | UP | neutron-l3-agent |
| f55311fc-635c-4985-ae6b-162f3fa8f886 | Open vSwitch agent | compute | None | :-) | UP | neutron-openvswitch-agent |
+--------------------------------------+--------------------+--------------------------------------+-------------------+-------+-------+---------------------------+
创建裸金属实例的 Flavor
官方文档：https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html
在 Queen 版本中，Ironic 项目新增 Trait API，节点的 traits 信息可以注册到计算服务的 Placement API 中，用于创建虚拟机时的调度。添加 Trait API 后，注册到 Ironic 的裸机也可以通过 Trait API 注册到 Placement 资源清单中，最终支持裸机的部署调度。
本文我们实践通过 Placement 来完成裸机的调度，通过 Resource Class 来标识 ironic node 的资源类型，通过 Resource Traits 来标识 ironic node 的特征，还可以通过 resources:VCPU=0、resources:MEMORY_MB=0、resources:DISK_GB=0 来 disable scheduling。
- 创建 Flavor
openstack flavor create --ram 8192 --vcpus 2 --disk 100 my-baremetal-flavor
openstack flavor set --property resources:CUSTOM_BAREMETAL_TEST=1 my-baremetal-flavor
openstack flavor set --property resources:VCPU=0 my-baremetal-flavor
openstack flavor set --property resources:MEMORY_MB=0 my-baremetal-flavor
openstack flavor set --property resources:DISK_GB=0 my-baremetal-flavor
- 验证，获取 Placement 候选人
[root@controller ~]# openstack allocation candidate list --os-placement-api-version 1.17 --resource VCPU=2 --resource DISK_GB=100 --resource MEMORY_MB=8192 --resource CUSTOM_BAREMETAL_TEST=1
+---+-----------------------------------------------------------+--------------------------------------+-------------------------------------------------------------------+------------------------------+
| # | allocation | resource provider | inventory used/capacity | traits |
+---+-----------------------------------------------------------+--------------------------------------+-------------------------------------------------------------------+------------------------------+
| 1 | MEMORY_MB=8192,VCPU=2,DISK_GB=100,CUSTOM_BAREMETAL_TEST=1 | e322f49a-ad50-468d-a031-29bde068c290 | VCPU=0/2,MEMORY_MB=0/8192,DISK_GB=0/100,CUSTOM_BAREMETAL_TEST=0/1 | HW_CPU_X86_VMX,CUSTOM_TRAIT1 |
+---+-----------------------------------------------------------+--------------------------------------+-------------------------------------------------------------------+------------------------------+
部署裸金属实例
NOTE：为了操作方便，这里使用了 DevStack 环境执行部署动作，环境详情详情请浏览前文列表。
- 确保 ironic node 处于可部署的状态
# Maintenance = False
[root@localhost ~]# openstack baremetal node list
+--------------------------------------+--------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------+---------------+-------------+--------------------+-------------+
| 30b7ee8f-7643-40d4-ae45-2bab737e1748 | node-0 | None | power off | available | False |
...
+--------------------------------------+--------+---------------+-------------+--------------------+-------------+
# Not reserved
[root@localhost ~]# openstack resource provider inventory list 30b7ee8f-7643-40d4-ae45-2bab737e1748
+------------------+------------------+----------+----------+-----------+----------+-------+
| resource_class | allocation_ratio | max_unit | reserved | step_size | min_unit | total |
+------------------+------------------+----------+----------+-----------+----------+-------+
| CUSTOM_BAREMETAL | 1.0 | 1 | 0 | 1 | 1 | 1 |
+------------------+------------------+----------+----------+-----------+----------+-------+
# BareMetal Flavor
[root@localhost ~]# openstack flavor show 6d849a74-845f-414b-a1f3-66ae4a3c814d
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| access_project_ids | None |
| disk | 10 |
| id | 6d849a74-845f-414b-a1f3-66ae4a3c814d |
| name | baremetal |
| os-flavor-access:is_public | True |
| properties | cpu_arch='x86_64', resources:CUSTOM_BAREMETAL='1', resources:DISK_GB='0', resources:MEMORY_MB='0', resources:VCPU='0', trait:CUSTOM_GOLD='required' |
| ram | 1280 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
# PXE Ports
[root@localhost ~]# openstack baremetal port show 0191759f-1869-4bc4-89af-bd08d03e146f
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| address | 52:54:00:c0:b1:82 |
| created_at | 2019-05-11T11:34:17+00:00 |
| extra | {} |
| internal_info | {} |
| is_smartnic | False |
| local_link_connection | {} |
| node_uuid | 30b7ee8f-7643-40d4-ae45-2bab737e1748 |
| physical_network | None |
| portgroup_uuid | None |
| pxe_enabled | True |
| updated_at | 2019-05-11T11:37:47+00:00 |
| uuid | 0191759f-1869-4bc4-89af-bd08d03e146f |
+-----------------------+--------------------------------------+
[root@localhost ~]# openstack baremetal port show 4757e6ec-a190-42b1-a098-b05adaa0cbca
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| address | 52:54:00:30:c7:e3 |
| created_at | 2019-05-11T11:34:19+00:00 |
| extra | {} |
| internal_info | {} |
| is_smartnic | False |
| local_link_connection | {} |
| node_uuid | 30b7ee8f-7643-40d4-ae45-2bab737e1748 |
| physical_network | None |
| portgroup_uuid | None |
| pxe_enabled | True |
| updated_at | 2019-05-11T11:37:47+00:00 |
| uuid | 4757e6ec-a190-42b1-a098-b05adaa0cbca |
+-----------------------+--------------------------------------+
- 执行部署
$ net_id=$(openstack network list | egrep "$PRIVATE_NETWORK_NAME"'[^-]' | awk '{ print $2 }')
$ image=$(openstack image show cirros-0.4.0-x86_64-disk -f value -c id)
$ openstack server create --flavor baremetal --nic net-id=$net_id --image $image testing
+-------------------------------------+-----------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | enLVnvyTQHQ3 |
| config_drive | |
| created | 2019-05-11T19:33:54Z |
| flavor | baremetal (6d849a74-845f-414b-a1f3-66ae4a3c814d) |
| hostId | |
| id | a2ccfb9a-0361-4f20-b11a-d1ea39c0f20b |
| image | cirros-0.4.0-x86_64-disk (3f734758-68fa-40ac-ba07-2bf8bd6f1911) |
| key_name | None |
| name | testing |
| progress | 0 |
| project_id | 920c5d4878f948a9879adb77aa5f6023 |
| properties | |
| security_groups | name='default' |
| status | BUILD |
| updated | 2019-05-11T19:33:54Z |
| user_id | 66dbd5cd1af34411860cf304cb4437ee |
| volumes_attached | |
+-------------------------------------+-----------------------------------------------------------------+
- 验证
$ openstack server list
+--------------------------------------+---------+--------+-------------------+--------------------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------+--------+-------------------+--------------------------+-----------+
| a2ccfb9a-0361-4f20-b11a-d1ea39c0f20b | testing | ACTIVE | private=10.0.0.10 | cirros-0.4.0-x86_64-disk | baremetal |
+--------------------------------------+---------+--------+-------------------+--------------------------+-----------+
$ openstack baremetal node list
+--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+
| 30b7ee8f-7643-40d4-ae45-2bab737e1748 | node-0 | a2ccfb9a-0361-4f20-b11a-d1ea39c0f20b | power on | active | False |
| 64d4db44-67b6-4efa-974c-141218060b59 | node-1 | None | power off | available | False |
| bf1b7c3d-85e2-4a1c-a547-1189c4efda69 | node-2 | None | power off | available | False |
+--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+
$ ip netns exec qdhcp-485c431e-b06d-4192-89f6-a4f57b2f921b ping 10.0.0.10
- 查看端口隐射
[root@localhost ~]# openstack baremetal port show 0191759f-1869-4bc4-89af-bd08d03e146f
+-----------------------+------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------+
| address | 52:54:00:c0:b1:82 |
| created_at | 2019-05-11T11:34:17+00:00 |
| extra | {} |
| internal_info | {u'tenant_vif_port_id': u'26dd6399-ddec-496e-9ffe-50dc52429166'} |
| is_smartnic | False |
| local_link_connection | {} |
| node_uuid | 30b7ee8f-7643-40d4-ae45-2bab737e1748 |
| physical_network | None |
| portgroup_uuid | None |
| pxe_enabled | True |
| updated_at | 2019-05-11T19:33:58+00:00 |
| uuid | 0191759f-1869-4bc4-89af-bd08d03e146f |
+-----------------------+------------------------------------------------------------------+
[root@localhost ~]# openstack port show 26dd6399-ddec-496e-9ffe-50dc52429166
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | 30b7ee8f-7643-40d4-ae45-2bab737e1748 |
| binding_profile | |
| binding_vif_details | |
| binding_vif_type | binding_failed |
| binding_vnic_type | baremetal |
| created_at | 2019-05-11T19:33:55Z |
| data_plane_status | None |
| description | |
| device_id | a2ccfb9a-0361-4f20-b11a-d1ea39c0f20b |
| device_owner | compute:nova |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | ip_version='4', opt_name='tag:ipxe6,67', opt_value='http://192.168.1.100:3928/boot.ipxe' |
| | ip_version='4', opt_name='150', opt_value='192.168.1.100' |
| | ip_version='4', opt_name='66', opt_value='192.168.1.100' |
| | ip_version='4', opt_name='tag:ipxe,67', opt_value='http://192.168.1.100:3928/boot.ipxe' |
| | ip_version='4', opt_name='tag:!ipxe,67', opt_value='undionly.kpxe' |
| | ip_version='4', opt_name='tag:!ipxe6,67', opt_value='undionly.kpxe' |
| | ip_version='4', opt_name='server-ip-address', opt_value='192.168.1.100' |
| fixed_ips | ip_address='10.0.0.10', subnet_id='04d5bdb0-f939-4b8f-94bd-0fd1a453ed6d' |
| id | 26dd6399-ddec-496e-9ffe-50dc52429166 |
| location | Munch({'project': Munch({'domain_id': 'default', 'id': u'920c5d4878f948a9879adb77aa5f6023', 'name': 'admin', 'domain_name': None}), 'cloud': '', 'region_name': 'RegionOne', 'zone': None}) |
| mac_address | 52:54:00:c0:b1:82 |
| name | |
| network_id | 485c431e-b06d-4192-89f6-a4f57b2f921b |
| port_security_enabled | True |
| project_id | 920c5d4878f948a9879adb77aa5f6023 |
| propagate_uplink_status | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 15 |
| security_group_ids | 536df95c-ed29-4ee2-8afe-fd317bc070fc |
| status | DOWN |
| tags | |
| trunk_details | None |
| updated_at | 2019-05-11T19:36:56Z |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
日志分析
# 将 Tenant Network 的 Port 挂载到 BM Node
VIF 26dd6399-ddec-496e-9ffe-50dc52429166 successfully attached to node 30b7ee8f-7643-40d4-ae45-2bab737e1748
# BM Node 进入 deploying state
Node 30b7ee8f-7643-40d4-ae45-2bab737e1748 moved to provision state "deploying" from state "available"; target provision state is "active"
# 关闭 BM Node 电源，如果它处于 power on
Not going to change node 30b7ee8f-7643-40d4-ae45-2bab737e1748 power state because current state = requested state = 'power off'.
# unconfigure_tenant_networks，挂上 Provisioning Network
Unbinding instance ports from node 30b7ee8f-7643-40d4-ae45-2bab737e1748
# 设定 BM Node 的 PXE 启动模式
Boot mode is not configured for node 30b7ee8f-7643-40d4-ae45-2bab737e1748 explicitly. The default boot mode is "bios", but, the default will be changed to "uefi" in the future. It is recommended to set the boot option into properties/capabilities/boot_mode for all nodes.
# 执行部署
Executing deploying on node 30b7ee8f-7643-40d4-ae45-2bab737e1748, remaining steps: [{'priority': 100, 'interface': 'deploy', 'step': 'deploy', 'argsinfo': None}]
Executing {'priority': 100, 'interface': 'deploy', 'step': 'deploy', 'argsinfo': None} on node 30b7ee8f-7643-40d4-ae45-2bab737e1748
# 下载 User Images 到 Ironic Conductor 本地
Master cache miss for image 3f734758-68fa-40ac-ba07-2bf8bd6f1911, starting download
# 重启 BM Node
Successfully set node 30b7ee8f-7643-40d4-ae45-2bab737e1748 power state to power on by rebooting.
Deploy step {'priority': 100, 'interface': 'deploy', 'step': 'deploy', 'argsinfo': None} on node 30b7ee8f-7643-40d4-ae45-2bab737e1748 being executed asynchronously, waiting for driver.
# BM Node 进入 wait call-back state，此时为 BM Node 部署 Deploy Image，然后 RAMDisk 里的 IPA 会向 Ironic Conductor 回调
Node 30b7ee8f-7643-40d4-ae45-2bab737e1748 moved to provision state "wait call-back" from state "deploying"; target provision state is "active"
# IPA 向 Ironic Conductor 执行回调，BM Node 再次进入 deploying state，此时正式部署 User Image
Node 30b7ee8f-7643-40d4-ae45-2bab737e1748 moved to provision state "deploying" from state "wait call-back"; target provision state is "active"
# 将 BM Node 的根磁盘挂载到 Ironic Conductor 本地，通过 dd 注入 User Image
iscsiadm -m discovery -t st -p 10.0.0.10:3260
iscsiadm -m node -p 10.0.0.10:3260 -T iqn.2008-10.org.openstack:30b7ee8f-7643-40d4-ae45-2bab737e1748 --login
iscsiadm -m node -S
iscsiadm -m node -T iqn.2008-10.org.openstack:30b7ee8f-7643-40d4-ae45-2bab737e1748 -R
dd if=/var/lib/ironic/images/30b7ee8f-7643-40d4-ae45-2bab737e1748/disk of=/dev/disk/by-path/ip-10.0.0.10:3260-iscsi-iqn.2008-10.org.openstack:30b7ee8f-7643-40d4-ae45-2bab737e1748-lun-1 bs=1M oflag=direct
hexdump -s 440 -n 4 -e "0x%08x" /dev/disk/by-path/ip-10.0.0.10:3260-iscsi-iqn.2008-10.org.openstack:30b7ee8f-7643-40d4-ae45-2bab737e1748-lun-1
iscsiadm -m node -p 10.0.0.10:3260 -T iqn.2008-10.org.openstack:30b7ee8f-7643-40d4-ae45-2bab737e1748 --logout
iscsiadm -m node -p 10.0.0.10:3260 -T iqn.2008-10.org.openstack:30b7ee8f-7643-40d4-ae45-2bab737e1748 -o delete
# remove_provisioning_network，重新挂上 Tenant Network
Unbinding instance ports from node 30b7ee8f-7643-40d4-ae45-2bab737e1748
# 重新设定引导方式（Local Disk）并启动 BM Node，完成部署
Successfully set node 30b7ee8f-7643-40d4-ae45-2bab737e1748 power state to power on by power on.
Node 30b7ee8f-7643-40d4-ae45-2bab737e1748 moved to provision state "active" from state "deploying"; target provision state is "None"
Successfully deployed node 30b7ee8f-7643-40d4-ae45-2bab737e1748 with instance a2ccfb9a-0361-4f20-b11a-d1ea39c0f20b.
问题：Failed to create neutron ports for any PXE enabled port on node
NetworkError: Failed to create neutron ports for any PXE enabled port on node c1729b3f-9ada-4def-8dcb-43f919b9b997.
调试代码定位到触发异常的代码：
def validate_port_info(node, port):
"""Check that port contains enough information for deploy.
Neutron network interface requires that local_link_information field is
filled before we can use this port.
:param node: Ironic node object.
:param port: Ironic port object.
:returns: True if port info is valid, False otherwise.
"""
# Note(moshele): client-id in the port extra field indicates an InfiniBand
# port. In this case we don't require local_link_connection to be
# populated because the network topology is discoverable by the Infiniband
# Subnet Manager.
if port.extra.get('client-id'):
return True
if (node.network_interface == 'neutron'
and not port.local_link_connection):
LOG.warning("The local_link_connection is required for "
"'neutron' network interface and is not present "
"in the nodes %(node)s port %(port)s",
{'node': node.uuid, 'port': port.uuid})
return False
return True
即 Neutron Port 没有 local_link_connection 属性。
解决
原因是因为 Ironic Conductor 使用了 Neutron Interface 来支持多租户网络，但这种实现需要在 Neutron 中插入 networking generic switch ML2 mechanism driver 才得以实现的，主要完成的工作是对裸金属服务器的上联物理交换机进行接口端口配置，例如：VLAN ID 的切换。以此来实现 ironic node 的网络切换（从 Provisioning Network 到 Tenant Network），之中还应用到了 LLDP 链路发现协议。
- 修改配置：
default_network_interface=flat
问题：获取 Deploy Image 时 MissingAuthPlugin: An auth plugin is required to determine endpoint URL
解决
- 是一个 Bug，解决办法：https://bugs.launchpad.net/openstack-ansible/+bug/1793959
- 获取 images 的时候 MissingAuthPlugin 是因为没有配置 [glance]。e.g.
[glance]
url = http://controller:9292
auth_url = http://controller:5000
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = RegionOne
project_name = service
username = glance
password = fanguiju
问题：执行 provide 时长时间卡在 provision_state:clean wait
这是因为 Inspection 阶段从 manageable => available 需要执行 cleaning（抹盘、初始化配置）的操作，需要花费非常长的时间，可以关闭auto-clean。
解决
- 重新接管
[root@controller ~]# openstack baremetal node abort e322f49a-ad50-468d-a031-29bde068c290
[root@controller ~]# openstack baremetal node list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| e322f49a-ad50-468d-a031-29bde068c290 | BM01 | None | power off | clean failed | True |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
[root@controller ~]# openstack baremetal --os-baremetal-api-version 1.11 node manage e322f49a-ad50-468d-a031-29bde068c290
[root@controller ~]# openstack baremetal node list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| e322f49a-ad50-468d-a031-29bde068c290 | BM01 | None | power off | manageable | True |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
- 修改配置文件
# /etc/ironic/ironic.conf
[conductor]
automated_clean = false
clean_callback_timeout = 1800
rescue_callback_timeout = 1800
soft_power_off_timeout = 600
power_state_change_timeout = 30
power_failure_recovery_interval = 300
- 重启服务
systemctl restart openstack-ironic-conductor
- 重新 provide
openstack baremetal --os-baremetal-api-version 1.11 node provide e322f49a-ad50-468d-a031-29bde068c290
问题：ironic node 无法被调度
Nova Scheduler Log：
Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.
Nova Compute for Ironic Log：
Node e322f49a-ad50-468d-a031-29bde068c290 is not ready for a deployment, reporting resources as reserved for it. Node's provision state is available, power state is power off and maintenance is True
BM01 的状态：
[root@controller ~]# openstack baremetal node list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| e322f49a-ad50-468d-a031-29bde068c290 | BM01 | None | power off | available | True |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
BM01 Resource Provider 状态：
[root@controller ~]# openstack resource provider inventory list e322f49a-ad50-468d-a031-29bde068c290
+-----------------------+------------------+----------+----------+-----------+----------+-------+
| resource_class | allocation_ratio | max_unit | reserved | step_size | min_unit | total |
+-----------------------+------------------+----------+----------+-----------+----------+-------+
| VCPU | 1.0 | 2 | 2 | 1 | 1 | 2 |
| MEMORY_MB | 1.0 | 8192 | 8192 | 1 | 1 | 8192 |
| DISK_GB | 1.0 | 100 | 100 | 1 | 1 | 100 |
| CUSTOM_BAREMETAL_TEST | 1.0 | 1 | 1 | 1 | 1 | 1 |
+-----------------------+------------------+----------+----------+-----------+----------+-------+
综上，Ironic node: BM01 被 nova-compute 认定为 not ready，所以通过 resource provider inventory reserved +1 的方式将该节点 “预留” 起来了。
解决
所以，问题是：为什么 BM01 是 not ready 的？原因就在于 ironic node Maintenance: True，节点处于维护状态！
[root@controller ~]# openstack baremetal node maintenance unset e322f49a-ad50-468d-a031-29bde068c290
[root@controller ~]# openstack baremetal node list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| e322f49a-ad50-468d-a031-29bde068c290 | BM01 | None | power off | available | False |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
再次查看 BM01 Resource Provider 状态：
[root@controller ~]# openstack resource provider inventory list e322f49a-ad50-468d-a031-29bde068c290
+-----------------------+------------------+----------+----------+-----------+----------+-------+
| resource_class | allocation_ratio | max_unit | reserved | step_size | min_unit | total |
+-----------------------+------------------+----------+----------+-----------+----------+-------+
| VCPU | 1.0 | 2 | 0 | 1 | 1 | 2 |
| MEMORY_MB | 1.0 | 8192 | 0 | 1 | 1 | 8192 |
| DISK_GB | 1.0 | 100 | 0 | 1 | 1 | 100 |
| CUSTOM_BAREMETAL_TEST | 1.0 | 1 | 0 | 1 | 1 | 1 |
+-----------------------+------------------+----------+----------+-----------+----------+-------+
问题：获取 swift_temp_url 时 MissingAuthPlugin: An auth plugin is required to determine endpoint URL
原因：因为我们选择了 Direct 的部署方式，裸机服务器上的 IPA 会从 Swift Object Storage 将 User Image 拉到本地，在裸机端完成镜像注入。
官方文档：Some drivers of the Baremetal service (in particular, any drivers using Direct deploy or Ansible deploy interfaces, and some virtual media drivers) require target user images to be available over clean HTTP(S) URL with no authentication involved (neither username/password-based, nor token-based).
When using the Baremetal service integrated in OpenStack, this can be achieved by specific configuration of the Image service and Object Storage service as described below.
解决
因为环境中没有 Swift 组件，所我们选择 iSCSI 的部署方式。
openstack baremetal --os-baremetal-api-version 1.31 node set e322f49a-ad50-468d-a031-29bde068c290 \
--deploy-interface iscsi \
--raid-interface agent
问题：Timeout reached while waiting for callback for node
这个 wait call-back 的过程实际上是 Ironic Conductor 在等待裸机上的 RAMDisk IPA 启动并完成回调。如果裸机能够从 PXE 启动并且网络没有问题的话，IPA 应当是可以连接到 Ironic Conductor 的。所以问题大概率出现在网络上。
PXE 启动流程：
- 终端从 PXE 网卡启动，向带内网络中的 DHCP 服务器索取 IP 地址和搜寻引导文件的位置。
- DHCP 服务器返回分给终端 IP 以及 NBP（网络启动程序，会自动加载引导文件并运行操作系统）文件的路径（通常为 TFTP 服务器）。
- 终端从带内网络中的 TFTP 服务器下载 NBP。
- 终端得到了 NBP 后自动从 TFTP 服务器下载引导文件，比如：pxelinux.0（GRUB)、vmlinuz（内核文件），initrd（内存驱动盘）等文件。
- 安装操作系统
从 PXE 启动流程可知裸机会从 Provisioning Network 中的 DHCP 获取 IP 地址，这个 IP 地址是必须要可以访问运行在 Ironic Conductor 上的 TFTP 服务，所以要求 Provisioning Network 与 TFTP 服务器要处于同一个 Flat 网络。
openstack subnet create provisioning-subnet-1 --network provisioning-net-1 \
--subnet-range 172.18.22.0/24 --ip-version 4 --gateway 172.18.22.1 \
--allocation-pool start=172.18.22.237,end=172.18.22.240 --dhcp
实际上在 wait call-back 阶段所发生的事情无法是：裸机重启，引导方式为从 PXE 启动。剩下的就交由 PXE 自动获取 DHCP 地址，自动获取 TFTP 地址、自动获取引导文件、自动部署 Deploy Image 了。
可以在当前环境中调试这个问题的方法可以非常简单粗暴，显示屏接上裸机服务器，然后静候佳音即可。
转载请注明作者：JmilkFan 范桂飓
