OpenStack显卡穿透

OpenStack显卡穿透

• 注意事项

• 服务器主板CPU虚拟化打开

• 如果只有一张显卡物理机不要安装显卡驱动

1. 确认是否支持iommu

cat /proc/cmdline | grep iommu

• 针对Inter CPU（注意自己CPU型号）。如果没有输出，说明不支持iommu，需要手动开启。

vim /etc/default/grub
GRUB_CMDLINE_LINUX="intel_iommu=on crashkernel=auto rhgb quiet"

• 针对AMD CPU（注意自己CPU型号）。如果没有输出，说明不支持iommu，需要手动开启。

vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 amd_iommu=on"

2. 更新grub，并重启设备

• 更新grub（注意自己安装系统用的方式，这种是系统盘）

grub-mkconfig -o /boot/grub/grub.cfg

• 如果是efi安装的系统，那么使用如下更新grub

grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg

• 重启系统

reboot

• 重启以后如下确认是否正常

dmesg | grep -i iommu

• 检查当前显卡设备信息
  • 注意，将得到的输出结果保存下来

lspci -nn | grep NVID
# 输出结果，第一个是显卡器，第二个是显卡音频
83:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050] [10de:1c81] (rev a1)
83:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)

3. vfio

• 配置vfio

vim /etc/modprobe.d/blacklist.conf
blacklist nouveau
options nouveau modeset=0
blacklist xhci_hcd
blacklist nvidia
blacklist snd_hda_intel
blacklist nvidiafb

• 配置系统加载模块

vim /etc/modules-load.d/openstack-gpu.conf
vfio_pci
pci_stub
vfio
vfio_iommu_type1
kvm
kvm_intel

• 配置vfio加载的设备
  • 注意下面的ids设备码对应自己显卡

vim /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1c81,10de:0fb9

• 重启

reboot

4. 挂载VFIO驱动

echo 10de 1c81 > /sys/bus/pci/drivers/vfio-pci/new_id
echo 10de 0fb9 > /sys/bus/pci/drivers/vfio-pci/new_id

• 检查VIFO驱动是否挂载成功

lspci -nnk -d 10de:1c81
lspci -nnk -d 10de:0fb9

5. OpenStack穿透配置

5.1 控制节点配置

• 配置nova文件

vim /etc/nova/nova.conf
[scheduler]
scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_available_filters=nova.scheduler.filters.all_filters
scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter
scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter

• 重启nova-scheduler服务

service nova-scheduler restart

• 配置nova-api
  • 注意显卡编码对应上，有多少个写多少个

vim /etc/nova/nova.conf
[pci]
alias = { "name": "nvidia1050vga", "product_id": "1c81", "vendor_id": "10de", "device_type": "type-PCI" }
alias = { "name": "nvidia1050aud", "product_id": "0fb9", "vendor_id": "10de", "device_type": "type-PCI" }

• 重启相关服务

service nova-api restart

5.2 计算节点

• 配置nova文件
  • 注意显卡编码对应上，有多少个写多少个

vim /etc/nova/nova.conf
[pci]
alias = { "name": "nvidia1050vga", "product_id": "1c81", "vendor_id": "10de", "device_type": "type-PCI" }
alias = { "name": "nvidia1050aud", "product_id": "0fb9", "vendor_id": "10de", "device_type": "type-PCI" }
passthrough_whitelist = [{ "vendor_id": "10de", "product_id": "1c81" },{ "vendor_id": "10de", "product_id": "0fb9" }]

• 重启nova-compute

service nova-compute restart

6. 安装ovmf，支持windows uefi启动

apt install ovmf -y

7. OpenStack调用显卡

• 创建及设置flavor，将显卡透传给虚拟机

openstack flavor create C2-4G-100G --ram 4096 --disk 100 --vcpus 2 --public

openstack flavor set C2-4G-100G --property "pci_passthrough:alias"="nvidia1050vga:1,nvidia1050aud:1"

• 因为NVIDIA显卡的驱动会检测是否跑在虚拟机里，如果在虚拟机里驱动就会出错，所以我们需要对显卡驱动隐藏hypervisor id

openstack image set [IMG-UUID] --property img_hide_hypervisor_id=true

• 之后利用相应的云主机类型创建相关实例即可
• 查看pci设备

root@ubuntu:~# lspci 
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01)
00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:04.0 SCSI storage controller: Red Hat, Inc. Virtio block device
# 这是我们的显卡驱动
00:05.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050] (rev a1)
00:06.0 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)

00:07.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
00:08.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG

• 安装完显卡驱动的效果

root@ubuntu:~# nvidia-smi 
Fri Jan 20 13:07:31 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:00:05.0 Off |                  N/A |
| 21%   18C    P8    N/A /  80W |      6MiB /  2048MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

7.1 uefi启动设置

• 镜像设置元数据

openstack image set af333da9-9529-4ee4-87f4-d21551f58b48  --property img_hide_hypervisor_id=true
openstack image set 6a47b29c-1f9d-4e1c-ac4b-2287fae2a9e1  --property os_type=windows
openstack image set 6a47b29c-1f9d-4e1c-ac4b-2287fae2a9e1  --property hw_cpu_cores=4
openstack image set  6a47b29c-1f9d-4e1c-ac4b-2287fae2a9e1  --property hw_cpu_threads=2
openstack image set 6a47b29c-1f9d-4e1c-ac4b-2287fae2a9e1  --property hw_cpu_sockets=4
openstack image set 6a47b29c-1f9d-4e1c-ac4b-2287fae2a9e1 --property hw_firmware_type=uefi
openstack image set 6a47b29c-1f9d-4e1c-ac4b-2287fae2a9e1  --property hw_machine_type=q35
openstack image set 6a47b29c-1f9d-4e1c-ac4b-2287fae2a9e1  --property os_secure_boot=required

8. 直通鼠标键盘问题

• 在没有图像化的时安装两个工具

  • Windows客户端：Moba
  • 服务器端

  apt install -y virt-manager
  

• 在Moba打开kvm管理页面删除Tablet

• 添加设备，将鼠标键盘都加上即可

9. windows系统优化

• 性能优化

    <hyperv>
.......
      <vpindex state="on"/>
      <runtime state="on"/>
      <synic state="on"/>
      <stimer state="on">
      <direct state="on"/>
      </stimer>
      <reset state="on"/>
      <frequencies state="on"/>
      <reenlightenment state="on"/>

• 给虚拟机分配固定的逻辑处理器

  <vcpu placement="static">8</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="1"/>
    <vcpupin vcpu="2" cpuset="2"/>
    <vcpupin vcpu="3" cpuset="3"/>
    <vcpupin vcpu="4" cpuset="4"/>
    <vcpupin vcpu="5" cpuset="5"/>
    <vcpupin vcpu="6" cpuset="6"/>
    <vcpupin vcpu="7" cpuset="7"/>
    <emulatorpin cpuset="8-9"/>
    <iothreadpin iothread="1" cpuset="10"/>
  </cputune>