openstack PCI透传(GPU)
描述
kolla-ansible部署openstack的GPU透传方法
一、gpu物理服务器配置
在gpu服务器上主启用IOMMU
确认内核⽀支持iommu
$ cat /proc/cmdline | grep iommu
输出以下信息：BOOT_IMAGE=/vmlinuz-3.10.0-1062.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rhgb quiet intel_iommu=on
如果没有输出, 则需要修改kernel启动参数,接着下面操作。
对于Intel cpu：GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
对于AMD cpu：GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt iommu=1"
下面以intel cpu 为修改例子
$ vim /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rhgb quiet intel_iommu=on"
GRUB_DISABLE_RECOVERY="true"
重新生成grub
- 系统采用EFI启动生成方式：
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg - 系统采用非EFI启动生成方式：
grub2-mkconfig -o /boot/grub2/grub.cfg
将下列内容加入到blacklist中以避免被gpu服务器占用
$ vim /etc/modprobe.d/blacklist.conf
blacklist snd_hda_intel
blacklist amd76x_edac
blacklist vga16fb
blacklist nouveau
blacklist rivafb
blacklist nvidiafb
blacklist rivatv
blacklist nvidia
重新生成initramfs
$ mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
$ dracut -v /boot/initramfs-$(uname -r).img $(uname -r)
重启系统
$ reboot
二、配置openstack
确认pci设备驱动信息
$ lspci -vnn | grep -i nvi
0c:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2204] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device [10de:1454]
0c:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aef] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1454]
其中[10de:2204]的10de为NVIDIA pci设备的vendor id, 2204为product id, 0c:00.0为设备地址
增加 PciPassthroughFilter 和 available_filters = nova.scheduler.filters.all_filters
$ vim /etc/kolla/config/nova.conf
*******
[filter_scheduler]
enabled_filters = RetryFilter,AvailabilityZoneFilter,AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,NUMATopologyFilter,PciPassthroughFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateCoreFilter,DifferentHostFilter,SameHostFilter
available_filters = nova.scheduler.filters.all_filters
添加pci设备显卡信息
$ vim /etc/kolla/config/nova.conf
*******
[pci]
alias = { "vendor_id":"10de", "product_id":"2204", "device_type":"type-PCI","name":"Nvidia3090a" }
passthrough_whitelist = { "vendor_id": "10de", "product_id": "2204", "address": "0000:0c:00.0" }
reconfigure nova组件
$ kolla-ansible -i /etc/ansible/hosts -t nova deploy
配置flavor并用这个flavor创建虚拟机测试
$ openstack flavor set GPU-16C32G-e --property pci_passthrough:alias='Nvidia3090a:1'
