12网络块存储

iSCSI简介

iSCSI 是⼀种基于 TCP/IP 的协议，⽤于通过基于 IP 的⽹络发送 SCSI 命令。它允许通过⽹络将类似于正常硬盘驱动器或固态驱动器的基于块的存储设备从服务器连接到客⼾端。作为存储区域⽹络(SAN)协议，iSCSI 可帮助您在本地或远程数据中⼼整合您的存储设备。您应该在独⽴于标准 LAN 流量的布线中实施 iSCSI，因为性能可能会由于共享⽹络上的带宽拥塞⽽降级。通常，iSCSI 与专⽤的 10 千兆位以太⽹或更好的⽹络搭配，以最⼤程度提⾼性能。

为最⼤程度地提⾼性能，SAN 流量通常是未加密的。从物理服务器到存储的缆线通常包含在数据中⼼中，因此最好不直接连接到 LAN ⽹络。出于 WAN 安全考虑，iSCSI 管理员可以使⽤ IPsec，这是⽤于保护IP ⽹络流量的协议套件。

iSCSI术语解释

iSCSI 协议以客⼾端/服务器配置的⽅式运⾏。客⼾端系统将启动器软件配置为将 SCSI 命令发送到远程服务器存储⽬标。在客⼾端系统上，iSCSI ⽬标显⽰为本地 SCSI 磁盘，就像使⽤ SCSI 电缆连接或与光纤通道交换结构连接的设备⼀样。

在使⽤ iSCSI 时将听到的⼀些术语包括：

启动器：一个iscsi客户端，通常以软件提供，但是也可以作为iSCSI HBA来实施。必须为启动器授予唯一名称。
目标：一个iSCSI存储资源，针对来自iSCSI服务器的连接而配置。必须为目标授予唯一目标。目标提供一个或多个低啊有编号的快射别，称为逻辑单元。一个iSCSI服务器可以同时提供多个目标。
发现：查询目标服务器以列出配置的目标。
ACL：访问权限控制列表，一种使用节点IQN(通常是启动器名称)来验证启动器的访问权限的访问限制。
IQN：iSCSI限定名称，全球唯一名称，用于以强制命名格式来识别启动器和目标。

IQN格式如下：

iqn.YYYY-MM.com.reversed.domain[:optional_string]
iqn：表示此名称使用域为标识符。
YYYY-MM：表示拥有域名的年月时间。
com.reversed.domain:拥有此iSCSI组织的逆向域名
:optional_string：以冒号为前缀的可选字符串，全球唯一，由域所有者分配，其中可包含冒号为分割符的组织边界

LUN：逻辑单元号，带有编号的块设备，连接到目标且通过目标来使用。可以有一个或多个LUN连接到单个目标，但通常一个目标提供一个LUN。
节点：任何一个iSCSI启动器或iSCSI目标，由其IQN来标识。
TPG：目标门户组，某个特定iSCSI目标要侦听的接口IP地址和TCP端口的集合。可以将目标配置添加到TPG以协调多个LUN的设置。
门户：目标或启动器上用于建立的IP地址和端口。

iSCSI的initiator有三种：

HBA卡，采用内键的SCSI指令及TOE引擎的ASIC芯片的适配卡价格最贵，性能最好，消耗CPU很少
内键的TOE引擎的ASIC芯片适配卡，由于SCSI指令任然以软件方式运行，所以任然会消耗一定的CPU资源，但价格相对便宜
完全由软件驱动的initiator，会消耗大量的CPU资源，性能差

iSCSI配置

服务端

装包

[root@servera ~]# yum -y install targetcli

配置 iSCSI 服务端共享资源

[root@servera ~]# targetcli
/> backstores/block create dev=/dev/vdb name=target1
Created block storage object target1 using /dev/vdb.
/> ls
o- / .......................................................................... [...]
  o- backstores ............................................................... [...]
  | o- block ................................................... [Storage Objects: 1]
  | | o- target1 ......................... [/dev/vdb (5.0GiB) write-thru deactivated]
  | |   o- alua .................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ........................ [ALUA state: Active/optimized]
  | o- fileio .................................................. [Storage Objects: 0]
  | o- pscsi ................................................... [Storage Objects: 0]
  | o- ramdisk ................................................. [Storage Objects: 0]
  o- iscsi ............................................................. [Targets: 0]
  o- loopback .......................................................... [Targets: 0]

创建 iSCSI target 名称及配置共享资源

/> iscsi/ create iqn.2023-06.com.example.lab:server1
Created target iqn.2023-06.com.example.lab:server1.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/> ls
o- / .......................................................................... [...]
  o- backstores ............................................................... [...]
  | o- block ................................................... [Storage Objects: 1]
  | | o- target1 ......................... [/dev/vdb (5.0GiB) write-thru deactivated]
  | |   o- alua .................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ........................ [ALUA state: Active/optimized]
  | o- fileio .................................................. [Storage Objects: 0]
  | o- pscsi ................................................... [Storage Objects: 0]
  | o- ramdisk ................................................. [Storage Objects: 0]
  o- iscsi ............................................................. [Targets: 1]
  | o- iqn.2023-06.com.example.lab:server1 ................................ [TPGs: 1]
  |   o- tpg1 ................................................ [no-gen-acls, no-auth]
  |     o- acls ........................................................... [ACLs: 0]
  |     o- luns ........................................................... [LUNs: 0]
  |     o- portals ..................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ...................................................... [OK]
  o- loopback .......................................................... [Targets: 0]

设置访问控制列表（ACL）

iSCSI 协议是通过客户端名称进行验证的，也就是说，用户在访问存储共享资源时不需要输入密码，只要 iSCSI 客户端的名称与服务端中设置的访问控制列表中某一名称条目一致即可。

因此需要在 iSCSI 服务端的配置文件中写入一串能够验证用户信息的名称。acls 参数目录用于存放能够访问 iSCSI 服务端共享存储资源的客户端名称。

/> iscsi/iqn.2023-06.com.example.lab:server1/tpg1/acls create iqn.2023-06.com.example.lab:serverb
Created Node ACL for iqn.2023-06.com.example.lab:serverb
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- target1 ........................................................................ [/dev/vdb (5.0GiB) write-thru deactivated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2023-06.com.example.lab:server1 ............................................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.2023-06.com.example.lab:serverb .................................................................. [Mapped LUNs: 0]
  |     o- luns .......................................................................................................... [LUNs: 0]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]

创建LUN

/> iscsi/iqn.2023-06.com.example.lab:server1/tpg1/luns create /backstores/block/target1 
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2023-06.com.example.lab:serverb
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- target1 .......................................................................... [/dev/vdb (5.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2023-06.com.example.lab:server1 ............................................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.2023-06.com.example.lab:serverb .................................................................. [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ............................................................................... [lun0 block/target1 (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 .................................................................... [block/target1 (/dev/vdb) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]

/> saveconfig 
Configuration saved to /etc/target/saveconfig.json
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup/.
Configuration saved to /etc/target/saveconfig.json

启动服务及配置防火墙

[root@servera ~]# systemctl enable --now target.service 
Created symlink /etc/systemd/system/multi-user.target.wants/target.service → /usr/lib/systemd/system/target.service.

[root@servera ~]# firewall-cmd --permanent --add-service=iscsi-target
success
[root@servera ~]# firewall-cmd --reload 
success

客户端

装包

[root@serverb ~]# yum search iscsi-init
Last metadata expiration check: 0:25:26 ago on Wed 21 Jun 2023 09:11:55 AM CST.
================================================= Name & Summary Matched: iscsi-init =================================================
python3-iscsi-initiator-utils.x86_64 : Python 3.6 bindings to iscsi-initiator-utils
====================================================== Name Matched: iscsi-init ======================================================
iscsi-initiator-utils.x86_64 : iSCSI daemon and utility programs
iscsi-initiator-utils.i686 : iSCSI daemon and utility programs
iscsi-initiator-utils.x86_64 : iSCSI daemon and utility programs
iscsi-initiator-utils-iscsiuio.x86_64 : Userspace configuration daemon required for some iSCSI hardware
iscsi-initiator-utils-iscsiuio.x86_64 : Userspace configuration daemon required for some iSCSI hardware
[root@serverb ~]# yum -y install iscsi-initiator-utils-iscsiuio.x86_64

配置客户端IQN

[root@serverb ~]# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.2023-06.com.example.lab:serverb

发现目标器

# man帮助里有示例
man iscsiadm
EXAMPLES
Discover targets at a given IP address:
  iscsiadm --mode discoverydb --type sendtargets --portal 192.168.1.10 --discover

Login, must use a node record id found by the discovery:
  iscsiadm --mode node --targetname iqn.2001-05.com.doe:test --portal 192.168.1.1:3260 --login

[root@serverb ~]# iscsiadm --mode discoverydb --type sendtargets --portal servera.lab.example.com --discover
172.25.250.10:3260,1 iqn.2023-06.com.example.lab:server1

登录连接目标器

[root@serverb ~]# iscsiadm --mode node --targetname iqn.2023-06.com.example.lab:server1 --portal 172.25.250.10:3260 --login
Logging in to [iface: default, target: iqn.2023-06.com.example.lab:server1, portal: 172.25.250.10,3260]
Login to [iface: default, target: iqn.2023-06.com.example.lab:server1, portal: 172.25.250.10,3260] successful.

验证

# 此时会发现多了一个块设备
[root@serverb ~]# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda      8:0    0   5G  0 disk 
vda    252:0    0  10G  0 disk 
└─vda1 252:1    0  10G  0 part /
vdb    252:16   0   5G  0 disk 

[root@serverb ~]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root  9 Jun 21 09:44 ip-172.25.250.10:3260-iscsi-iqn.2023-06.com.example.lab:server1-lun-0 -> ../../sda
lrwxrwxrwx. 1 root root  9 Jun 21 09:02 pci-0000:06:00.0 -> ../../vda
lrwxrwxrwx. 1 root root 10 Jun 21 09:02 pci-0000:06:00.0-part1 -> ../../vda1
lrwxrwxrwx. 1 root root  9 Jun 21 09:02 pci-0000:07:00.0 -> ../../vdb
lrwxrwxrwx. 1 root root  9 Jun 21 09:02 virtio-pci-0000:06:00.0 -> ../../vda
lrwxrwxrwx. 1 root root 10 Jun 21 09:02 virtio-pci-0000:06:00.0-part1 -> ../../vda1
lrwxrwxrwx. 1 root root  9 Jun 21 09:02 virtio-pci-0000:07:00.0 -> ../../vdb

分区格式化

[root@serverb ~]# fdisk /dev/sda 

Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x74b75ce3.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): 

Using default response p.
Partition number (1-4, default 1): 
First sector (2048-10485759, default 2048): 
Last sector, +sectors or +size{K,M,G,T,P} (2048-10485759, default 10485759): 

Created a new partition 1 of type 'Linux' and of size 5 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

[root@serverb ~]# mkfs -t xfs /dev/sda1
meta-data=/dev/sda1              isize=512    agcount=4, agsize=327616 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=1310464, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

补充

# 如果想开机自启
[root@serverb ~]# systemctl enable --now iscsi
[root@serverb ~]# systemctl enable --now iscsid
Created symlink /etc/systemd/system/multi-user.target.wants/iscsid.service → /usr/lib/systemd/system/iscsid.service.

挂载

[root@serverb ~]# mkdir /mnt/iscsi
[root@serverb ~]# vim /etc/fstab 
[root@serverb ~]# grep xfs /etc/fstab 
/dev/sda1	/mnt/iscsi	xfs	defaults,_netdev 0 0
[root@serverb ~]# mount -a

[root@serverb ~]# df -H /mnt/iscsi/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       5.4G   72M  5.3G   2% /mnt/iscsi

注意：由于/dev/sdb 是一块网络存储设备，而 iSCSI 协议是基于TCP/IP 网络传输数据的，因此必须在/etc/fstab 配置文件中添加上_netdev 参数，表示当系统联网后再进行挂载操作，以免系统开机时间过长或开机失败

当挂载iSCSI目标时，很重要使用_netdev选项，因为iSCSI initiator需要在文件系统可以挂载之前建立与目标的连接。如果没有此选项，则在网络连接尚未建立时，mount命令可能会失败。

然而，在挂载NFS或Samba共享时，不需要使用_netdev选项，因为NFS和Samba协议已经包含了处理网络连接问题的机制。此外，NFS和Samba共享通常使用automounter进行挂载，它自动处理网络连接并且不需要在fstab文件中使用_netdev选项。

卸载iscsi存储，登出目标器

umoun  /mnt/iscsi/

iscsiadm --mode node --targetname iqn.2023-06.com.example.lab:server1 --portal 172.25.250.10:3260 --logout

rm -rf /var/lib/iscsi/nodes/*

常见错误

在发现目标器之前，没有修改本机IQN，而后再修改去登录连接目标器发现无法正常连接，删除缓存文件即可

[root@serverb ~]# rm -rf /var/lib/iscsi/*

持久挂载iscsi时，因为虚拟机启动太慢，无法正常登录连接，修改客户端连接超时时间即可

[root@serverb ~]# vim /etc/iscsi/iscsid.conf
# To specify the time to wait for login to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.login_timeout = 120

# To specify the time to wait for logout to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.logout_timeout = 120

# Time interval to wait for on connection before sending a ping.
node.conn[0].timeo.noop_out_interval = 30

自动化配置

准备联系环境

[student@workstation ~]$ lab iscsi-automation start

[student@workstation iscsi-automation]$ ls
ansible.cfg  cleanup.yml  inventory  playbook.yml  solution  templates  unmount.yml

编写yaml文件

[student@workstation iscsi-automation]$ cat playbook.yml 
---
- name: Ensure /data is mounted from serverd iSCSI target
  hosts: initiators
  become: true

  tasks:
    - name: the iscsi-initiator-utils package is installed
      yum:
        name: iscsi-initiator-utils
        state: present

    - name: the IQN is set for the initiator
      #FIXME: see "ansible-doc template" for some examples.
      #       Deploy the templates/initiatorname.iscsi.j2 template file
        src: templates/initiatorname.iscsi.j2
        dest: /etc/iscsi/initiatorname.iscsi
      notify: restart iscsid

    # Forces the handler to run so that the iscsid service is restarted
    # and is aware of the new initiator IQN
    - meta: flush_handlers

    - name: the iSCSI target is discovered and available
      #FIXME: see "ansible-doc open_iscsi" for some examples.
      #       The target is iqn.2014-06.com.example:serverd and the portal is
      #       172.25.250.13 (port 3260)
      #       Make sure that the target is automatically connected at startup.
      open_iscsi:
        portal: 172.25.250.10
        port: 3260
        login: yes
        discover: yes
        target: iqn.2023-06.com.example:serverd
      register: target

    - name: display the discovered devices
      debug:
        var: target['devicenodes']

    - name: the new device is formatted and mounted under /data
      include_role:
        name: rhel-system-roles.storage
      vars:
        storage_volumes:
          - name: devdata
            state: present
            type: disk
            disks:
              - "{{ target['devicenodes'][0] }}"
            #FIXME: see /usr/share/doc/rhel-system-roles/storage/README.md
            #       Set the mount point to /data, the file system type to xfs,
            #       and add the _netdev mount option.

  handlers:
    - name: restart iscsid
      service:
        name: iscsid
        state: restarted

[student@workstation iscsi-automation]$ cat templates/initiatorname.iscsi.j2 
InitiatorName=iqn.2023-06.com.example.lab:{{ ansible_facts['hostname'] }}

posted @ 2023-07-19 16:47 风有遗霜阅读(34) 评论(0) 收藏举报

刷新页面返回顶部

风有遗霜