08.存储Cinder→5.场景学习→12.Ceph Volume Provider→1.配置

返回总目录

1. 配置ceph(控制节点)这里仅仅是对控制节点的配置文件进行更改，具体安装流程参考见04.搭建实验环境→2.搭建环境(devstack)       (在配置控制节点前记得在计算节点执行unstack.sh，以便关闭计算节点所有服务，使其不影响控制节点)  
1. 在控制节点的local.conf添加ceph plugin： 

   1 2 3 4 5

   stack@controller:~/devstack$ vim local.conf ... #ceph enable_plugin devstack-plugin-ceph https://opendev.org/openstack/devstack-plugin-ceph # use TryStack git mirror

   
   
2. 运行stack.sh文件
3. 在安装中如果出现“umount: /var/lib/ceph/drives/sdb1: mountpoint not found”的问题，解决如下：
1. 出现这个问题往往是因为在安装过程中先出现了Could not find a version that satisfies the requirement...问题 参考见04.搭建实验环境→2.搭建环境(devstack)
1. 添加了新的python源之后重新进行stack，发现问题解决
2. 然后就发现新的问题即“umount: /var/lib/ceph/drives/sdb1: mountpoint not found”
2. 此时应先unstack然后再stack 由于添加的python源解决了第一个问题，因此第一个问题不会出现，第二个问题也没有出现

2. 配置计算节点 
1. 重新安装计算节点devstack环境，并在控制节点发现计算节点：root@controller:~# /opt/stack/devstack/tools/discover_hosts.sh 
2. 安装ceph客户端：root@compute:~# apt-get install ceph-common
3. 授权设置：
1. client.cinder秘钥：root@controller:~# ceph auth get-or-create client.cinder | ssh root@compute tee  /etc/ceph/ceph.client.cinder.keyring
1. tee用法：读取标准输入的数据，并将其内容输出成文件

   1 2 3 4 5 6 7 8

   root@cmp-2:~# tee zhao 36 36 q q root@cmp-2:~# cat zhao 36 q

2. libvirt秘钥：root@controller:~# ceph auth get-key client.cinder | ssh root@compute tee /etc/ceph/client.cinder.key结果一样
1. 配置文件

   1 2 3 4 5 6 7

   root@compute:~# vim /etc/ceph/secret.xml <secret ephemeral='no' private='no'> <uuid>df0d0b60-047a-45f5-b5be-f7d2b4beadee</uuid> <usage type='ceph'> <name>client.cinder secret</name> </usage> </secret>

   其中这个uuid查看控制节点的nova.conf
2. 从xml文件定义或修改secret：
1. root@compute:~# virsh secret-define --file /etc/ceph/secret.xml 
3. 设置secret：
1. root@compute:~# virsh secret-set-value --secret df0d0b60-047a-45f5-b5be-f7d2b4beadee --base64 $(cat /etc/ceph/client.cinder.key) 
4. 查看secret：virsh secret-list所有计算节点的secret相同
5. 删除文件：rm /etc/ceph/client.cinder.key && rm /etc/ceph/secret.xml 
3. client.admin秘钥
1. root@controller:~# scp /etc/ceph/ceph.client.admin.keyring compute:/etc/ceph/
4. 配置文件：
   计算节点：
   

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

   root@compute:~# vim /etc/ceph/ceph.conf [global] rbd default features = 1 osd pool default size = 1 osd journal size = 100 osd crush chooseleaf type = 0 filestore_xattr_use_omap = true auth_client_required = cephx auth_service_required = cephx auth_cluster_required = cephx mon_host = 172.16.1.17 mon_initial_members = controller fsid = eab37548-7aef-466a-861c-3757a12ce9e8 root@compute:~# vim /etc/nova/nova-cpu.conf [libvirt] images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = rbd disk_cachemodes = network=writeback inject_partition = -2 inject_key = false rbd_secret_uuid = df0d0b60-047a-45f5-b5be-f7d2b4beadee rbd_user = cinder live_migration_uri = qemu+ssh://stack@%s/system cpu_mode = none virt_type = kvm

   修改nova-cpu.conf而不是修改nova.conf
5. 重启计算服务
1. root@compute:~# systemctl restart libvirtd.service
2. root@compute:~# systemctl restart devstack@n-cpu.service
6. 验证
1. 创建虚拟机，确定该虚拟机是在计算节点上创建(virsh list)，使用rbd ls vms查看虚机镜像文件

3. 配置对比

不使用ceph plugin搭建的devstack环境

使用ceph plugin搭建的devstack环境

glance-api.conf

1 2	[glance_store] filesystem_store_datadir = /opt/stack/data/glance/images/

1 2 3 4 5 6 7

[glance_store] rbd_store_pool = images rbd_store_user = glance rbd_store_ceph_conf = /etc/ceph/ceph.conf stores = file, http, rbd default_store = rbd filesystem_store_datadir = /opt/stack/data/glance/images/

nova.conf 或 nova-cpu.conf

1 2 3 4

[libvirt] live_migration_uri = qemu+ssh://stack@%s/system cpu_mode = none virt_type = kvm

1 2 3 4 5 6 7 8 9 10 11 12

[libvirt] images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = rbd disk_cachemodes = network=writeback inject_partition = -2 inject_key = false rbd_secret_uuid = df0d0b60-047a-45f5-b5be-f7d2b4beadee rbd_user = cinder live_migration_uri = qemu+ssh://stack@%s/system cpu_mode = none virt_type = kvm

cinder.conf

1 2 3 4 5 6 7 8

[lvmdriver-1] image_volume_cache_enabled = True volume_clear = zero lvm_type = auto iscsi_helper = tgtadm volume_group = stack-volumes-lvmdriver-1 volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver volume_backend_name = lvmdriver-1

cinder.conf

1 2 3 4 5 6 7 8 9 10 11

[ceph] image_volume_cache_enabled = True volume_clear = zero rbd_max_clone_depth = 5 rbd_flatten_volume_from_snapshot = False rbd_secret_uuid = df0d0b60-047a-45f5-b5be-f7d2b4beadee rbd_user = cinder rbd_pool = volumes rbd_ceph_conf = /etc/ceph/ceph.conf volume_driver = cinder.volume.drivers.rbd.RBDDriver volume_backend_name = ceph

ceph.conf
控制节点在配置好后显示的：

1 2 3 4 5 6 7 8 9 10 11 12

[global] rbd default features = 1 osd pool default size = 1 osd journal size = 100 osd crush chooseleaf type = 0 filestore_xattr_use_omap = true auth_client_required = cephx auth_service_required = cephx auth_cluster_required = cephx mon_host = 172.16.1.17 mon_initial_members = controller fsid = eab37548-7aef-466a-861c-3757a12ce9e8

在ceph集群的admin节点(在该节点用ceph-deploy建立ceph存储集群)创建初始化monitor后，会得到多个秘钥文件，要将这些秘钥文件和该节点配置的ceph.conf文件分发到其他所有节点(其他ceph节点，计算节点，控制节点...)
fsid是自动配置的

4. ceph的日志文件

   1 2 3 4 5 6 7 8 9

   root@controller:~# ll /var/log/ceph/ total 2856 drwxrws--T 2 ceph ceph 4096 Jun 25 17:48 ./ drwxrwxr-x 13 root syslog 4096 Jun 25 17:46 ../ -rw------- 1 ceph ceph 35669 Jun 26 16:38 ceph.audit.log -rw------- 1 ceph ceph 4504 Jun 26 15:01 ceph.log -rw-r--r-- 1 ceph ceph 2719445 Jun 26 17:06 ceph-mgr.x.log -rw-r--r-- 1 root ceph 32990 Jun 25 19:31 ceph-mon.controller.log -rw-r--r-- 1 ceph ceph 106920 Jun 26 14:29 ceph-osd.0.log

   
   
1. ceph的日志等级有 ERR、WRN、INFO