配置IPSAN存储
本文结构：
一.存储介绍
1.1 常见存储类型的比较
1.2 NAS存储
1.3 SAN存储
1.4 NAS与SAN的主要区别
二.配置IPSAN存储
2.1 存储服务端配置
2.2 存储客户端10配置
2.3 存储客户端11配置
一.存储介绍
1.1 常见存储类型
直连式存储(Direct-Attached Storage，DAS)：消耗服务器本身的IO资源和CPU资源，使用方便，服务器直接识别为一个块设备
非直连式存储：将IO资源和CPU资源消耗转移到后端专门的存储设备上，通常是通过网线或者光纤与需要使用存储的服务器相连接，因此这种存储类型对网络要求很高，常见的两种类型是SAN和NAS
1.2 NAS存储
NAS：网络附属存储基于标准网络协议（Tcp/IP）实现数据传输，为网络中的Windows / Linux / Mac OS 等各种不同操作系统的计算机提供文件共享和数据备份，NAS本身能够支持多种协议，如NFS、CIFS、FTP、HTTP等，由此可以看出NAS对于主机而言是可以直接使用的文件系统，不需要直接面对块设备(就更不用格式化和挂载块设备了)。
1.3 SAN存储
SAN:采用网状通道（Fibre Channel ，简称FC）技术，通过FC交换机连接存储阵列和服务器主机，建立专用于数据存储的区域网络。
SAN的存储类型：
(1). IPSAN: 利用IP网络构建存储网络, 使用TCP/IP协议的iscsi协议封装构建的存储区域网络
(2). FSCAN: 利用光纤线, 通过高速FC交换机组成的存储区域网络，需要使用到光纤交换机和HBA卡，相对而言成本高
1.4 NAS和SAN的主要区别
第一，从网络架构来说，本质区别在于：NAS直接使用TCP/IP传输数据。SAN使用SCSI或iSCSI协议传输数据。
第二，从文件读写实现方法上来说，本质区别在于：NAS采用了NFS和 CIFS技术实现文件共享。说明NAS是基于操作系统的“文件级”读写操作。SAN中计算机和存储间的接口是底层的块协议，它按照协议头的“块地址+偏移地址”来定位。共享的存储和前端的操作系统类型没有关系，任何服务器操作系统，都可以正常识别。
各类存储类型特点总结如下：
|
存储结构/性能对比 |
DAS |
NAS |
FC-SAN |
IP-SAN |
|
成本 |
低 |
较低 |
高 |
较高 |
|
数据传输速度 |
快 |
慢 |
极快 |
较快 |
|
扩展性 |
无扩展性 |
较低 |
易于扩展 |
最易扩展 |
|
服务器访问存储方式 |
直接访问存储数据块 |
以文件方式访问 |
直接访问存储数据块 |
直接访问存储数据块 |
|
服务器系统性能开销 |
低 |
较低 |
低 |
较高 |
|
安全性 |
高 |
低 |
高 |
低 |
|
是否集中管理存储 |
否 |
是 |
是 |
是 |
|
备份效率 |
低 |
较低 |
高 |
较高 |
|
网络传输协议 |
无 |
TCP/IP |
Fibre Channel |
TCP/IP |
二.配置IPSAN存储
2.1 存储服务端配置
(1). 实验环境
|
主机名 |
IP地址 |
身份 |
操作系统 |
|
SAN |
192.168.0.105 |
SAN存储 |
CentOS7 |
|
10 |
10.4.7.10 |
客户端1 |
CentOS7 |
|
11 |
10.4.7.11 |
客户端2 |
CentOS7 |
[root@SAN ~]# iptables -F #清空防火墙规则
[root@SAN ~]# systemctl stop firewalld && systemctl disable firewalld #关闭防火墙
[root@SAN ~]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config #关闭selinux
(2).安装配置工具：
[root@SAN ~]# yum -y install targetcli
(3).配置存储
[root@SAN ~]# targetcli
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.fb49
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> help #">"符号前的"/"表示当前路径为根目录，根Linux shell类似
······ AVAILABLE COMMANDS #可用命令 ================== The following commands are available in the current path: #一下命令在当前路径 - bookmarks action [bookmark] - cd [path] #切换路径 - clearconfig [confirm] #清空配置信息 - exit #退出 - get [group] [parameter...] - help [topic] #帮助 - ls [path] [depth] #列出当前路径的文件和目录 - pwd #查看当前路径 - refresh - restoreconfig [savefile] [clear_existing] - saveconfig [savefile] #保存配置，默认文件路径为/etc/target/saveconfig.json - sessions [action] [sid] #查看当前的客户端与存储服务器的连接情况 - set [group] [parameter=value...] #设置全局和局部属性 - status #状态 - version #版本
/> ls #查看当前根目录下的文件信息
o- / ............................................................................................................. [...] #根目录路径 o- backstores .................................................................................................. [...] #根目录路径下的backstores目录 | o- block ...................................................................................... [Storage Objects: 0] #根目录路径下backstores目录下的目录文件block文件路径，显示当前系统上的磁盘设备 | o- fileio ..................................................................................... [Storage Objects: 0] | o- pscsi ...................................................................................... [Storage Objects: 0] | o- ramdisk .................................................................................... [Storage Objects: 0] o- iscsi ................................................................................................ [Targets: 0] #根目录下的iscsi目录，跟backstores同级别目录 o- loopback ............................................................................................. [Targets: 0] #根目录下的loopbak目录，跟backstores和iscsi目录同级别
/> /backstores/ #/backstores/+空格+TAB键即可显示/backstores/ 可搭配使用的命令参数，其他目录下的搭配命令雷同
bookmarks cd exit get help ls pwd refresh set status #当前目录下可使用的命令
/> /backstores/block #/backstores/block+空格+TAB键即可显示/backstores/block可搭配使用的命令参数，其他目录下的搭配命令雷同，不再例举
bookmarks cd create delete exit get help ls pwd refresh
set status
/> /backstores/block create #/backstores/block create+空格+TAB键可显示当前命令可带的参数
dev= name= readonly= wwn= #"dev="指定当前系统上的磁盘设备，"name="指定前面"dev="的自定义名称，"readonly="指定是否只读模式，“wwn=”指定"dev="的wwn号
/> /backstores/block create dev=/dev/sdb name=sun1 #把当前系统上的/dev/sdb磁盘命名为sun1的块存储设备
Created block storage object sun1 using /dev/sdb. #提示使用/dev/sdb创建块设备对象sun1成功
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...]
| o- block ...................................................................................... [Storage Objects: 1]
| | o- sun1 .............................................................. [/dev/sdb (20.0GiB) write-thru deactivated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 0]
o- loopback ............................................................................................. [Targets: 0
/> /iscsi create iqn.2020-03.cn.192:server01 #配置ISCSITarget命名，命名在同一子网内确保是唯一的，命名格式为：iqn.yyyy-mm.<主机名反写>:自定义名称（自定义名称内不能有下划线）
Created target iqn.2020-03.cn.192:server01. Created TPG 1. Global pref auto_add_default_portal=true Created default portal listening on all IPs (0.0.0.0), port 3260.
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...] #定义后端存储设备，这个是物理磁盘
| o- block ...................................................................................... [Storage Objects: 1]
| | o- sun1 .............................................................. [/dev/sdb (20.0GiB) write-thru deactivated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 1] #定义客户端连接信息
| o- iqn.2020-03.cn.192:server01 ........................................................................... [TPGs: 1]
| o- tpg1 ................................................................................... [no-gen-acls, no-auth]
| o- acls .............................................................................................. [ACLs: 0] #定义允许哪些客户端来连接该块设备
| o- luns .............................................................................................. [LUNs: 0] #定义lun映射信息
| o- portals ........................................................................................ [Portals: 1] #定义端口信息
| o- 0.0.0.0:3260 ......................................................................................... [OK] #默认监听在本机的3260端口，可手动更改
o- loopback ............................................................................................. [Targets: 0]
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/acls create iqn.2020-03.cn.192:clinet #创建ACL允许ISCSI客户机连接
Created Node ACL for iqn.2020-03.cn.192:clinet
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...]
| o- block ...................................................................................... [Storage Objects: 1] #定义后端存储
| | o- sun1 .............................................................. [/dev/sdb (20.0GiB) write-thru deactivated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 1] #定义客户端如何来连接后端存储
| o- iqn.2020-03.cn.192:server01 ........................................................................... [TPGs: 1]
| o- tpg1 ................................................................................... [no-gen-acls, no-auth]
| o- acls .............................................................................................. [ACLs: 1]
| | o- iqn.2020-03.cn.192:clinet ................................................................ [Mapped LUNs: 0]
| o- luns .............................................................................................. [LUNs: 0]
| o- portals ........................................................................................ [Portals: 1]
| o- 0.0.0.0:3260 ......................................................................................... [OK] #默认监听在本机的3260端口，可更改
o- loopback ............................................................................................. [Targets: 0]
配置验证用户名和密码(该项为非必须)
/> cd iscsi/iqn.2020-03.cn.192:server01/tpg1/acls/iqn.2020-03.cn.192:clinet
/iscsi/iqn.20...cn.192:clinet> ls
o- iqn.2020-03.cn.192:clinet .......................................................................... [Mapped LUNs: 1] o- mapped_lun0 ................................................................................ [lun0 block/sun1 (rw)]
/iscsi/iqn.20...cn.192:clinet> set auth userid=admin
Parameter userid is now 'admin'.
/iscsi/iqn.20...cn.192:clinet> set auth password=Admin
Parameter password is now 'Admin'.
/iscsi/iqn.20...cn.192:clinet> cd /
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/luns create /backstores/block/sun1 #创建lun（target块设备的逻辑单元）
Created LUN 0. Created LUN 0->0 mapping in node ACL iqn.2020-03.cn.192:clinet
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...]
| o- block ...................................................................................... [Storage Objects: 1]
| | o- sun1 ................................................................ [/dev/sdb (20.0GiB) write-thru activated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 1]
| o- iqn.2020-03.cn.192:server01 ........................................................................... [TPGs: 1]
| o- tpg1 ................................................................................... [no-gen-acls, no-auth]
| o- acls .............................................................................................. [ACLs: 1]
| | o- iqn.2020-03.cn.192:clinet ................................................................ [Mapped LUNs: 1]
| | o- mapped_lun0 ...................................................................... [lun0 block/sun1 (rw)]
| o- luns .............................................................................................. [LUNs: 1]
| | o- lun0 ........................................................... [block/sun1 (/dev/sdb) (default_tg_pt_gp)]
| o- portals ........................................................................................ [Portals: 1]
| o- 0.0.0.0:3260 ......................................................................................... [OK] #默认监听在3260端口上，可手段更改
o- loopback ............................................................................................. [Targets: 0
#创建IP和端口，可默认不更改
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/portals/ delete 0.0.0.0 3260 #删除默认的3260端口
Deleted network portal 0.0.0.0:3260
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/portals/ create 0.0.0.0 13260 #x修改默认3260端口为13260
Binding to INADDR_ANY (0.0.0.0) Created network portal 0.0.0.0:13260.
/> saveconfig #保存上面所作的配置信息
Last 10 configs saved in /etc/target/backup/. Configuration saved to /etc/target/saveconfig.json #保存配置信息的路径
/> exit #退出
Global pref auto_save_on_exit=true Last 10 configs saved in /etc/target/backup/. Configuration saved to /etc/target/saveconfig.json
[root@SAN ~]# systemctl restart target && systemctl enable target
2.2 存储客户端10配置(即使用后端存储的服务器)
[root@10 ~]# yum install -y iscsi-initiator-utils
[root@10 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-03.cn.192:clinet #iqn.2020-03.cn.192:clinet为在存储服务器定义的允许来连接的名称
[root@10 ~]# vim /etc/iscsi/iscsid.conf #取消下面三行内容的注释，并将username和password修改成我们在存储服务器上设置的用户名和密码，若未在存储服务器上设置用户名和密码则跳过该步骤
57 node.session.auth.authmethod = CHAP 61 node.session.auth.username = admin 62 node.session.auth.password = Admin
[root@10 ~]# lsblk #此时只有主机10本地的磁盘
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 19G 0 part ├─centos-root 253:0 0 15G 0 lvm / └─centos-swap 253:1 0 4G 0 lvm [SWAP] sr0 11:0 1 1024M 0 rom
[root@10 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.0.105:13260 #寻找连接测试
192.168.0.105:13260,1 iqn.2020-03.cn.192:server01 #找到一个目标taget
[root@10 ~]# tree /var/lib/iscsi/ #发现target服务后，目标target的信息会写入/var/lib/iscsi 目录下
/var/lib/iscsi/
├── ifaces
├── isns
├── nodes
│ └── iqn.2020-03.cn.192:server01
│ └── 192.168.0.105,13260,1
│ └── default
├── send_targets
│ └── 192.168.0.105,13260
│ ├── iqn.2020-03.cn.192:server01,192.168.0.105,13260,1,default -> /var/lib/iscsi/nodes/iqn.2020-03.cn.192:server01/192.168.0.105,13260,1
│ └── st_config
├── slp
└── static
[root@10 ~]# iscsiadm -m node -l #登录，登录后就可以看到/dev/sdb磁盘了；登出为iscsiadm -m node --logout，登出后在本机使用lsblk就看不到/dev/sdb了
Logging in to [iface: default, target: iqn.2020-03.cn.192:server01, portal: 192.168.0.105,13260] (multiple) Login to [iface: default, target: iqn.2020-03.cn.192:server01, portal: 192.168.0.105,13260] successful. #提示登录成功
[root@10 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 15G 0 lvm /
└─centos-swap 253:1 0 4G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sr0 11:0 1 1024M 0 rom
[root@10 ~]# systemctl start iscsid && systemctl enbale iscsid
此时我们在存储服务器上查看连接情况：
[root@SAN ~]# targetcli
/> sessions
alias: 10 sid: 1 type: Normal session-state: LOGGED_IN
2.3 存储客户端11配置
#连接第2台服务器，使得10和11共用存储 ，由于我们之前只定义了一个target，因此我们登陆的到存储服务器在定义一个target--iqn.2020-03.cn.192:clinet1
(1).在存储服务器上再添加一个target
[root@SAN ~]# targetcli
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/acls/ create iqn.2020-03.cn.192:clinet1
Created Node ACL for iqn.2020-03.cn.192:clinet1 Created mapped LUN 0.
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...]
| o- block ...................................................................................... [Storage Objects: 1]
| | o- sun1 ................................................................ [/dev/sdb (20.0GiB) write-thru activated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 1]
| o- iqn.2020-03.cn.192:server01 ........................................................................... [TPGs: 1]
| o- tpg1 ................................................................................... [no-gen-acls, no-auth]
| o- acls .............................................................................................. [ACLs: 2]
| | o- iqn.2020-03.cn.192:clinet ................................................................ [Mapped LUNs: 1]
| | | o- mapped_lun0 ...................................................................... [lun0 block/sun1 (rw)]
| | o- iqn.2020-03.cn.192:clinet1 ............................................................... [Mapped LUNs: 1]
| | o- mapped_lun0 ...................................................................... [lun0 block/sun1 (rw)]
| o- luns .............................................................................................. [LUNs: 1]
| | o- lun0 ........................................................... [block/sun1 (/dev/sdb) (default_tg_pt_gp)]
| o- portals ........................................................................................ [Portals: 1]
| o- 0.0.0.0:13260 ........................................................................................ [OK]
o- loopback ............................................................................................. [Targets: 0]
/> saveconfig #保存配置
/> exit
[root@SAN ~]# systemctl restart target #重启服务
(2).在主机11客户端上的配置(跟2.2雷同)：
[root@11 ~]# yum install -y iscsi-initiator-utils
[root@11 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-03.cn.192:clinet1 #名称是新定义的target
[root@11 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.0.105:13260
192.168.0.105:13260,1 iqn.2020-03.cn.192:server01
[root@11 ~]# iscsiadm -m node -l
#如果有多个硬盘，则需要指定登出的磁盘，格式为iscsiadm -m node -T iqn.2020-03.cn.192:server01 -u
Logging in to [iface: default, target: iqn.2020-03.cn.192:server01, portal: 192.168.0.105,13260] (multiple) Login to [iface: default, target: iqn.2020-03.cn.192:server01, portal: 192.168.0.105,13260] successful.
[root@11 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 49G 0 part
├─centos-root 253:0 0 45G 0 lvm /
└─centos-swap 253:1 0 4G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
└─sdb1 8:17 0 20G 0 part #可以发现此处已经做好了分区，在第二台服务器上只需要挂载使用即可
sr0 11:0 1 1024M 0 rom
[root@11 ~]# systemctl start iscsid && systemctl enable iscsid
[root@11 ~]# mkdir /iscsi_data
[root@11 ~]# mount /dev/sdb1 /iscsi_data/
[root@11 ~]# ls /iscsi_data/
hello.txt
此时我们在存储服务器上查看连接情况：
[root@SAN ~]# targetcli
/> sessions
alias: 10 sid: 1 type: Normal session-state: LOGGED_IN alias: 11 sid: 2 type: Normal session-state: LOGGED_IN
此时我们在主机11上新建文件，并查看主机10是否能看到新文件
[root@11 ~]# mkdir /iscsi_data/from_11
[root@10 ~]# ls /data/
hello.txt #并未显示主机11上新建的目录
why??是因为我们使用的XFS 文件系统，XFS文件系统不支持多个客户端同时使用， 使用GFS文件系统就可以同步。
