cinder对接nfs共享存储

一、cinder使用nfs做后端存储

cinder服务和nova服务很相似。

nova：不提供虚拟化，支持多种虚拟化技术，kvm,xen,qemu,lxc

cinder：不提供存储，支持多种存储技术，lvm，nfs，glusterFS，ceph

后期如果需要对接其他类型后端存储，方法都类似

二、控制节点安装nfs（服务端）

1. 安装包

[root@controller ~]# yum install nfs-utils.x86_64 -y

2. 配置

创建共享目录

[root@controller ~]# mkdir /data
[root@controller ~]# ll -d /data
drwxr-xr-x 2 root root 6 Nov 23 23:04 /data

修改配置文件

[root@controller ~]# vim /etc/exports
[root@controller ~]# cat /etc/exports
/data 10.0.0.0/24(rw,async,no_root_squash,no_all_squash)

3. 启动服务

复制代码
[root@controller ~]# systemctl restart rpcbind nfs
[root@controller ~]# systemctl status rpcbind nfs
● rpcbind.service - RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
   Active: active (running) since Mon 2020-11-23 23:06:03 CST; 11s ago
  Process: 66414 ExecStart=/sbin/rpcbind -w $RPCBIND_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 66415 (rpcbind)
   CGroup: /system.slice/rpcbind.service
           └─66415 /sbin/rpcbind -w

Nov 23 23:06:03 controller systemd[1]: Starting RPC bind service...
Nov 23 23:06:03 controller systemd[1]: Started RPC bind service.

● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
   Active: active (exited) since Mon 2020-11-23 23:06:09 CST; 6s ago
  Process: 66485 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 66476 ExecStartPre=/bin/sh -c /bin/kill -HUP `cat /run/gssproxy.pid` (code=exited, status=0/SUCCESS)
  Process: 66472 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 66485 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

Nov 23 23:06:08 controller systemd[1]: Starting NFS server and services...
Nov 23 23:06:09 controller systemd[1]: Started NFS server and services.
复制代码

三、配置存储节点（客户端）

1.修改/etc/cinder/cinder.conf配置文件

复制代码
[root@computer1 ~]# openstack-config --set /etc/cinder/cinder.conf DEFAULT enabled_backends sata,ssd,nfs
[root@computer1 ~]# openstack-config --set /etc/cinder/cinder.conf nfs volume_driver cinder.volume.drivers.nfs.NfsDriver
[root@computer1 ~]# openstack-config --set /etc/cinder/cinder.conf nfs nfs_shares_config /etc/cinder/nfs_shares
[root@computer1 ~]# openstack-config --set /etc/cinder/cinder.conf nfs volume_backend_name nfs
[root@computer1 ~]# cat /etc/cinder/cinder.conf
[DEFAULT]
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 10.0.0.12
glance_api_servers = http://computer2:9292
enabled_backends = sata,ssd,nfs
[BACKEND]
[BRCD_FABRIC_EXAMPLE]
[CISCO_FABRIC_EXAMPLE]
[COORDINATION]
[FC-ZONE-MANAGER]
[KEYMGR]
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://cinder:123456@controller/cinder
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = 123456
[matchmaker_redis]
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = 123456
[oslo_middleware]
[oslo_policy]
[oslo_reports]
[oslo_versionedobjects]
[ssl]
[ssd]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-ssd
iscsi_protocol = iscsi
iscsi_helper = lioadm
volume_backend_name = ssd
[sata]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-sata
iscsi_protocol = iscsi
iscsi_helper = lioadm
volume_backend_name = sata
[nfs]
volume_driver = cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config = /etc/cinder/nfs_shares
volume_backend_name = nfs
复制代码

2. 修改/etc/cinder/nfs_shares配置文件

[root@computer1 ~]# echo '10.0.0.11:/data' >>/etc/cinder/nfs_shares
[root@computer1 ~]# cat /etc/cinder/nfs_shares
10.0.0.11:/data

3. 重启cinder-volume服务

复制代码
[root@computer1 ~]# systemctl restart openstack-cinder-volume.service 
[root@computer1 ~]# systemctl status openstack-cinder-volume.service 
● openstack-cinder-volume.service - OpenStack Cinder Volume Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-volume.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-11-23 23:14:30 CST; 3min 18s ago
 Main PID: 120582 (cinder-volume)
   CGroup: /system.slice/openstack-cinder-volume.service
           ├─120582 /usr/bin/python2 /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinde...
           ├─120735 /usr/bin/python2 /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinde...
           ├─120764 /usr/bin/python2 /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinde...
           └─120774 /usr/bin/python2 /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinde...

Nov 23 23:16:38 computer1 sudo[121609]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/cinder-rootwrap /etc/cinder/...er-sata
Nov 23 23:16:38 computer1 sudo[121613]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/cinder-rootwrap /etc/cinder/...er-sata
Nov 23 23:17:04 computer1 sudo[121804]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/cinder-rootwrap /etc/cinder/...der-ssd
Nov 23 23:17:05 computer1 sudo[121808]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/cinder-rootwrap /etc/cinder/...der-ssd
Nov 23 23:17:29 computer1 cinder-volume[120582]: 2020-11-23 23:17:29.458 120774 INFO os_brick.remotefs.remotefs [req-8aa11278-9...4f35b3
Nov 23 23:17:29 computer1 sudo[121978]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/cinder-rootwrap /etc/cinder/...a4f35b3
Nov 23 23:17:29 computer1 sudo[121981]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/cinder-rootwrap /etc/cinder/...a4f35b3
Nov 23 23:17:29 computer1 sudo[121984]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/cinder-rootwrap /etc/cinder/...a4f35b3
Nov 23 23:17:38 computer1 sudo[122041]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/cinder-rootwrap /etc/cinder/...er-sata
Nov 23 23:17:38 computer1 sudo[122055]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/cinder-rootwrap /etc/cinder/...er-sata
Hint: Some lines were ellipsized, use -l to show in full.
复制代码

四、检查服务端是否有nfs类型的volume

复制代码
[root@controller ~]# source admin-openrc 
[root@controller ~]# cinder service-list
+------------------+----------------+------+---------+-------+----------------------------+-----------------+
|      Binary      |      Host      | Zone |  Status | State |         Updated_at         | Disabled Reason |
+------------------+----------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler |   controller   | nova | enabled |   up  | 2020-11-23T15:19:47.000000 |        -        |
|  cinder-volume   | computer1@nfs  | nova | enabled |   up  | 2020-11-23T15:19:48.000000 |        -        |
|  cinder-volume   | computer1@sata | nova | enabled |   up  | 2020-11-23T15:19:45.000000 |        -        |
|  cinder-volume   | computer1@ssd  | nova | enabled |   up  | 2020-11-23T15:19:47.000000 |        -        |
+------------------+----------------+------+---------+-------+----------------------------+-----------------+
复制代码

五、创建nfs类型的卷

web页面创建

1. 管理员——>系统——>卷：创建卷类型

2. 项目——>计算——>卷：创建卷

3. 创建成功后，查看卷信息

复制代码
#存储节点上查看
[root@computer1 ~]# df -h
Filesystem       Size  Used Avail Use% Mounted on
/dev/sda2         48G  4.8G   44G  10% /
devtmpfs         983M     0  983M   0% /dev
tmpfs            993M     0  993M   0% /dev/shm
tmpfs            993M  8.8M  984M   1% /run
tmpfs            993M     0  993M   0% /sys/fs/cgroup
/dev/sr0         4.3G  4.3G     0 100% /mnt
tmpfs            199M     0  199M   0% /run/user/0
10.0.0.11:/data   48G  3.0G   46G   7% /var/lib/cinder/mnt/490717a467bd12d34ec324c86a4f35b3
[root@computer1 ~]# ll -h /var/lib/cinder/mnt/490717a467bd12d34ec324c86a4f35b3
total 0
-rw-rw-rw- 1 qemu qemu 2.0G Nov 23 23:32 volume-dea6012b-edbb-4f05-aeb7-328bf5927a52 
[root@computer1 ~]# qemu-img info /var/lib/cinder/mnt/490717a467bd12d34ec324c86a4f35b3/volume-dea6012b-edbb-4f05-aeb7-328bf5927a52 
image: /var/lib/cinder/mnt/490717a467bd12d34ec324c86a4f35b3/volume-dea6012b-edbb-4f05-aeb7-328bf5927a52
file format: raw
virtual size: 2.0G (2147483648 bytes)
disk size: 0
复制代码
复制代码
#服务端共享目录下查看
[root@controller ~]# ll /data
total 0
-rw-rw-rw- 1 107 107 2147483648 Nov 23 23:32 volume-dea6012b-edbb-4f05-aeb7-328bf5927a52
[root@controller ~]# qemu-img info /data/volume-dea6012b-edbb-4f05-aeb7-328bf5927a52 
image: /data/volume-dea6012b-edbb-4f05-aeb7-328bf5927a52
file format: raw
virtual size: 2.0G (2147483648 bytes)
disk size: 0
复制代码

4. 创建卷关联实例

5. 创建实例在卷中

点击启动实例，创建实例

报错提示：Build of instance 2d8aa5f6-3b8e-42b3-8ff8-245cfb7408f5 aborted: Block Device Mapping is Invalid。查看存储日志

复制代码


cat /var/log/cinder/api.log


2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance [req-aad2057d-3ec3-4ef2-9a8b-e27af67c8817 10e57203019442eb85062f7e90e5604b 9a6e662b63de48eeb651a3332c74b092 - - -] Error contacting glance server '10.0.0.11:9292' for 'get', done trying.
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance Traceback (most recent call last):
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance   File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 177, in call
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance     return getattr(controller, method)(*args, **kwargs)
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 132, in get
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance     % urlparse.quote(str(image_id)))
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 272, in head
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance     return self._request('HEAD', url, **kwargs)
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 256, in _request
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance     raise exc.CommunicationError(message=message)
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance CommunicationError: Error finding address for http://10.0.0.11:9292/v1/images/0dc22fbc-be1c-4fe8-9b0f-138e486fd3af: HTTPConnectionPool(host='10.0.0.11', port=9292): Max retries exceeded with url: /v1/images/0dc22fbc-be1c-4fe8-9b0f-138e486fd3af (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7463850>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2020-11-23 23:51:30.587 1795 ERROR cinder.image.glance 


复制代码


解决办法：修改/etc/cinder/cinder.conf配置文件


添加glance api的地址


复制代码


[root@controller ~]# vim /etc/cinder/cinder.conf
[root@controller ~]# cat /etc/cinder/cinder.conf
[DEFAULT]
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 10.0.0.11
glance_api_servers = http://computer2:9292
[BACKEND]
[BRCD_FABRIC_EXAMPLE]
[CISCO_FABRIC_EXAMPLE]
[COORDINATION]
[FC-ZONE-MANAGER]
[KEYMGR]
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://cinder:123456@controller/cinder
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = 123456
[matchmaker_redis]
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = 123456
[oslo_middleware]
[oslo_policy]
[oslo_reports]
[oslo_versionedobjects]
[ssl]


复制代码


重启服务


复制代码


[root@controller ~]# systemctl restart openstack-cinder-api.service openstack-cinder-scheduler.service
[root@controller ~]# systemctl status  openstack-cinder-api.service openstack-cinder-scheduler.service
● openstack-cinder-api.service - OpenStack Cinder API Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-11-24 00:01:36 CST; 6min ago
 Main PID: 82265 (cinder-api)
   CGroup: /system.slice/openstack-cinder-api.service
           ├─82265 /usr/bin/python2 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/ci...
           └─82419 /usr/bin/python2 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/ci...

Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.329 82265 WARNING cinder.api.openstack [req-d866c78a-270c-4f2...source
Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.330 82265 WARNING cinder.api.openstack [req-d866c78a-270c-4f2...source
Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.330 82265 WARNING cinder.api.openstack [req-d866c78a-270c-4f2...source
Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.331 82265 WARNING cinder.api.openstack [req-d866c78a-270c-4f2...source
Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.331 82265 WARNING cinder.api.openstack [req-d866c78a-270c-4f2...source
Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.332 82265 WARNING cinder.api.openstack [req-d866c78a-270c-4f2...source
Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.332 82265 WARNING cinder.api.openstack [req-d866c78a-270c-4f2...source
Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.335 82265 INFO oslo.service.wsgi [req-d866c78a-270c-4f25-a0e9...0:8776
Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.412 82265 INFO oslo_service.service [req-d866c78a-270c-4f25-a...orkers
Nov 24 00:02:05 controller cinder-api[82265]: 2020-11-24 00:02:05.722 82419 INFO eventlet.wsgi.server [-] (82419) wsgi starting...0:8776

● openstack-cinder-scheduler.service - OpenStack Cinder Scheduler Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-11-24 00:01:50 CST; 6min ago
 Main PID: 82334 (cinder-schedule)
   CGroup: /system.slice/openstack-cinder-scheduler.service
           └─82334 /usr/bin/python2 /usr/bin/cinder-scheduler --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cin...

Nov 24 00:01:50 controller systemd[1]: Started OpenStack Cinder Scheduler Server.
Nov 24 00:01:50 controller systemd[1]: Starting OpenStack Cinder Scheduler Server...
Nov 24 00:01:59 controller cinder-scheduler[82334]: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" fr...AULT".
Nov 24 00:01:59 controller cinder-scheduler[82334]: 2020-11-24 00:01:59.343 82334 WARNING oslo_reports.guru_meditation_report [...ports.
Nov 24 00:02:02 controller cinder-scheduler[82334]: 2020-11-24 00:02:02.311 82334 WARNING py.warnings [req-fe826d8d-236a-46b3-a...ported
Nov 24 00:02:02 controller cinder-scheduler[82334]: exception.NotSupportedWarning
Nov 24 00:02:03 controller cinder-scheduler[82334]: 2020-11-24 00:02:03.788 82334 INFO cinder.rpc [req-fe826d8d-236a-46b3-a783-...rsion.
Nov 24 00:02:03 controller cinder-scheduler[82334]: 2020-11-24 00:02:03.795 82334 INFO cinder.rpc [req-fe826d8d-236a-46b3-a783-...rsion.
Nov 24 00:02:03 controller cinder-scheduler[82334]: 2020-11-24 00:02:03.806 82334 INFO cinder.service [-] Starting cinder-sched...8.1.0)
Hint: Some lines were ellipsized, use -l to show in full.


复制代码


重新创建实例










点击启动实例






复制代码


[root@computer1 ~]# cd /var/lib/nova/instances/
[root@computer1 instances]# ll
total 4
drwxr-xr-x 2 nova nova 44 Nov 24 00:11 488d5efc-ee3d-423d-8eaf-b17ea0cece35
drwxr-xr-x 2 nova nova 54 Nov 23 11:02 _base
-rw-r--r-- 1 nova nova 31 Nov 24 00:16 compute_nodes
drwxr-xr-x 2 nova nova 93 Nov 23 11:02 locks
[root@computer1 instances]# ll -h 488d5efc-ee3d-423d-8eaf-b17ea0cece35/
total 28K
-rw------- 1 qemu qemu  24K Nov 24 00:16 console.log
-rw-r--r-- 1 nova nova 2.5K Nov 24 00:11 libvirt.xml


复制代码


复制代码


[root@computer1 ~]# ll -h /var/lib/nova/mnt/490717a467bd12d34ec324c86a4f35b3
total 18M
-rw-rw-rw- 1 root root 2.0G Nov 23 23:32 volume-dea6012b-edbb-4f05-aeb7-328bf5927a52
-rw-rw-rw- 1 qemu qemu 1.0G Nov 24 00:16 volume-e68f499f-8ddc-4bdf-bf48-28e0ebc5a72f
[root@computer1 ~]# qemu-img info /var/lib/nova/mnt/490717a467bd12d34ec324c86a4f35b3/volume-e68f499f-8ddc-4bdf-bf48-28e0ebc5a72f
image: /var/lib/nova/mnt/490717a467bd12d34ec324c86a4f35b3/volume-e68f499f-8ddc-4bdf-bf48-28e0ebc5a72f
file format: raw
virtual size: 1.0G (1073741824 bytes)
disk size: 18M


复制代码


 






		

		
posted @ 
 
出水芙蓉·薇薇 
阅读(1170) 
评论(0) 
编辑 
收藏 
举报


	

	
	








    
    

    
    



    

        
    

        

            
                
                
            
        

    

编辑推荐：



· Linux系列：如何用heaptrack跟踪.NET程序的非托管内存泄露
    

· 开发者必知的日志记录最佳实践
    

· SQL Server 2025 AI相关能力初探
    

· Linux系列：如何用 C#调用 C方法造成内存泄露
    

· AI与.NET技术实操系列（二）：开始使用ML.NET
    



    

    阅读排行：
    

 ·          无需6万激活码！GitHub神秘组织3小时极速复刻Manus，手把手教你使用OpenManus搭建本
        

 ·          C#/.NET/.NET Core优秀项目和框架2025年2月简报
        

 ·          葡萄城 AI 搜索升级：DeepSeek 加持，客户体验更智能
        

 ·          什么是nginx的强缓存和协商缓存
        

 ·          一文读懂知识蒸馏
        



    

    







	

	


	
	

	

	

	





    

    
    








  

  
点击右上角即可分享

  微信分享提示