udev配置错误导致的一次故障
问题
有两台ORACLE数据库服务器，同事通过插拔光纤模块测试FC SAN 存储多路径高可用性。后发现在恢复光纤链路后LAN依然无法恢复，重启主机也无济于事，multipath似乎也出现了问题。在此之前系统运行正常，DBA曾安装过oracle数据库和必要的一些软件，在配置安装数据库过程中有过使SSH奔溃的情况，后来进行了修复。
环境
irora01:~ # cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 4
问题现象
恢复存储的FC物理链路，重启系统fdisk -l已经看不外挂的FC存储了，只有本地磁盘。
irora01:~ # fdisk -l
Disk /dev/sdb: 2697.0 GB, 2696999337984 bytes
255 heads, 63 sectors/track, 327891 cylinders, total 5267576832 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sda: 479.0 GB, 478998953984 bytes
255 heads, 63 sectors/track, 58234 cylinders, total 935544832 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000a1f9c
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 10489855 5243904 83 Linux
/dev/sda2 10489856 935544831 462527488 8e Linux LVM
multipath多路径重启失败
irora01:~ # rcmultipathd restart
Shutting down multipathd done
Starting multipathd failed
问题处理
系统日志没有明显错误，重启多路径除了一个failed没有其他有用信息。
lsscsi命令查看scsi设备，能看到本地磁盘，其他的盘显示为横杠“-”。
irora01:~ # lsscsi
[0:2:0:0] disk LSI LSI 3.40 /dev/sda
[0:2:1:0] disk LSI LSI 3.40 /dev/sdb
[7:0:0:0] disk HITACHI OPEN-V 8001 -
[7:0:0:1] disk HITACHI OPEN-V 8001 -
[7:0:0:2] disk HITACHI OPEN-V 8001 -
[7:0:0:3] disk HITACHI OPEN-V 8001 -
[7:0:0:4] disk HITACHI OPEN-V 8001 -
[7:0:0:5] disk HITACHI OPEN-V 8001 -
[7:0:0:6] disk HITACHI OPEN-V 8001 -
[7:0:1:0] disk HITACHI OPEN-V 8001 -
[7:0:1:1] disk HITACHI OPEN-V 8001 -
[7:0:1:2] disk HITACHI OPEN-V 8001 -
[7:0:1:3] disk HITACHI OPEN-V 8001 -
[7:0:1:4] disk HITACHI OPEN-V 8001 -
[7:0:1:5] disk HITACHI OPEN-V 8001 -
[7:0:1:6] disk HITACHI OPEN-V 8001 -
[8:0:0:0] disk HITACHI OPEN-V 8001 -
[8:0:0:1] disk HITACHI OPEN-V 8001 -
[8:0:0:2] disk HITACHI OPEN-V 8001 -
[8:0:0:3] disk HITACHI OPEN-V 8001 -
[8:0:0:4] disk HITACHI OPEN-V 8001 -
[8:0:0:5] disk HITACHI OPEN-V 8001 -
[8:0:0:6] disk HITACHI OPEN-V 8001 -
[8:0:1:0] disk HITACHI OPEN-V 8001 -
[8:0:1:1] disk HITACHI OPEN-V 8001 -
[8:0:1:2] disk HITACHI OPEN-V 8001 -
[8:0:1:3] disk HITACHI OPEN-V 8001 -
[8:0:1:4] disk HITACHI OPEN-V 8001 -
[8:0:1:5] disk HITACHI OPEN-V 8001 -
[8:0:1:6] disk HITACHI OPEN-V 8001 -
irora01:~ # lsscsi -h
Usage: lsscsi [--classic] [--device] [--generic] [--help] [--hosts]
[--kname] [--list] [--long] [--protection] [--size]
[--sysfsroot=PATH] [--transport] [--verbose] [--version]
[<h:c:t:l>]
where:
--classic|-c alternate output similar to 'cat /proc/scsi/scsi'
--device|-d show device node's major + minor numbers
--generic|-g show scsi generic device name
--help|-h this usage information
--hosts|-H lists scsi hosts rather than scsi devices
--kname|-k show kernel name instead of device node name
--list|-L additional information output one
attribute=value per line
--long|-l additional information output
--protection|-p show target and initiator protection information
--protmode|-P show negotiated protection information mode
--size|-s show disk size
--sysfsroot=PATH|-y PATH set sysfs mount point to PATH (def: /sys)
--transport|-t transport information for target or, if '--hosts'
given, for initiator
--verbose|-v output path names where data is found
--version|-V output version string and exit
<h:c:t:l> filter output list (def: '- - - -' (all))
List SCSI devices or hosts, optionally with additional information
lsscsi -sg可以显示SCSI通用设备名称和容量，这么看应该不是HBA卡驱动问题，磁盘在系统下是有识别的。
irora01:~ # lsscsi -sg
[0:2:0:0] disk LSI LSI 3.40 /dev/sda /dev/sg0 478GB
[0:2:1:0] disk LSI LSI 3.40 /dev/sdb /dev/sg1 2.69TB
[7:0:0:0] disk HITACHI OPEN-V 8001 - /dev/sg2 214GB
[7:0:0:1] disk HITACHI OPEN-V 8001 - /dev/sg3 429GB
[7:0:0:2] disk HITACHI OPEN-V 8001 - /dev/sg4 429GB
[7:0:0:3] disk HITACHI OPEN-V 8001 - /dev/sg5 429GB
[7:0:0:4] disk HITACHI OPEN-V 8001 - /dev/sg6 644GB
[7:0:0:5] disk HITACHI OPEN-V 8001 - /dev/sg7 21.4GB
[7:0:0:6] disk HITACHI OPEN-V 8001 - /dev/sg8 10.7GB
[7:0:1:0] disk HITACHI OPEN-V 8001 - /dev/sg9 214GB
[7:0:1:1] disk HITACHI OPEN-V 8001 - /dev/sg10 429GB
[7:0:1:2] disk HITACHI OPEN-V 8001 - /dev/sg11 429GB
[7:0:1:3] disk HITACHI OPEN-V 8001 - /dev/sg12 429GB
[7:0:1:4] disk HITACHI OPEN-V 8001 - /dev/sg13 644GB
[7:0:1:5] disk HITACHI OPEN-V 8001 - /dev/sg14 21.4GB
[7:0:1:6] disk HITACHI OPEN-V 8001 - /dev/sg15 10.7GB
[8:0:0:0] disk HITACHI OPEN-V 8001 - /dev/sg16 214GB
[8:0:0:1] disk HITACHI OPEN-V 8001 - /dev/sg17 429GB
[8:0:0:2] disk HITACHI OPEN-V 8001 - /dev/sg18 429GB
[8:0:0:3] disk HITACHI OPEN-V 8001 - /dev/sg19 429GB
[8:0:0:4] disk HITACHI OPEN-V 8001 - /dev/sg20 644GB
[8:0:0:5] disk HITACHI OPEN-V 8001 - /dev/sg21 21.4GB
[8:0:0:6] disk HITACHI OPEN-V 8001 - /dev/sg22 10.7GB
[8:0:1:0] disk HITACHI OPEN-V 8001 - /dev/sg23 214GB
[8:0:1:1] disk HITACHI OPEN-V 8001 - /dev/sg24 429GB
[8:0:1:2] disk HITACHI OPEN-V 8001 - /dev/sg25 429GB
[8:0:1:3] disk HITACHI OPEN-V 8001 - /dev/sg26 429GB
[8:0:1:4] disk HITACHI OPEN-V 8001 - /dev/sg27 644GB
[8:0:1:5] disk HITACHI OPEN-V 8001 - /dev/sg28 21.4GB
[8:0:1:6] disk HITACHI OPEN-V 8001 - /dev/sg29 10.7GB
lsblk查看块设备，也能看到像sdg、sdh等的大量磁盘，还有容量显示，这应该就是存储授权给主机的LUN，问题是fdisk -l看不到这些盘，是不是DBA动了底层的软件导致的？
irora01:~ # lsblk
NAME MAJ:MIN RM SIZE RO MOUNTPOINT
sdb 8:16 0 2.5T 0
└─data-data (dm-10) 253:10 0 2.5T 0 /data
sdc 8:32 0 200G 0
sde 8:64 0 400G 0
sdd 8:48 0 400G 0
sda 8:0 0 446.1G 0
├─sda1 8:1 0 5G 0 /boot
└─sda2 8:2 0 441.1G 0
├─vg1-home (dm-0) 253:0 0 50G 0 /home
├─vg1-opt (dm-1) 253:1 0 70G 0 /opt
├─vg1-opt_orabak (dm-2) 253:2 0 25G 0 /opt/orabak
├─vg1-opt_oracle (dm-3) 253:3 0 25G 0 /opt/oracle
├─vg1-opt_oracrs (dm-4) 253:4 0 30G 0 /opt/oracrs
├─vg1-root (dm-5) 253:5 0 15G 0 /
├─vg1-swap (dm-6) 253:6 0 50G 0 [SWAP]
├─vg1-tmp (dm-7) 253:7 0 10G 0 /tmp
├─vg1-usr (dm-8) 253:8 0 10G 0 /usr
└─vg1-var (dm-9) 253:9 0 20G 0 /var
sdg 8:96 0 600G 0
sdh 8:112 0 20G 0
sdk 8:160 0 400G 0
sdi 8:128 0 10G 0
sdl 8:176 0 400G 0
sdo 8:224 0 20G 0
sdf 8:80 0 400G 0
sdm 8:192 0 400G 0
sdj 8:144 0 200G 0
sdn 8:208 0 600G 0
sdp 8:240 0 10G 0
sdq 65:0 0 200G 0
sdr 65:16 0 400G 0
sds 65:32 0 400G 0
sdt 65:48 0 400G 0
sdu 65:64 0 600G 0
sdv 65:80 0 20G 0
sdw 65:96 0 10G 0
sdx 65:112 0 200G 0
sdy 65:128 0 400G 0
sdz 65:144 0 400G 0
sdaa 65:160 0 400G 0
sdab 65:176 0 600G 0
sdac 65:192 0 20G 0
sdad 65:208 0 10G 0
loop1 7:1 0 443.5M 1 /mnt
重启multipathd失败，/etc/multipath.conf也没有问题，重新安装multipath问题依旧
irora01:~ # rcmultipathd restart
Shutting down multipathd done
Starting multipathd failed
irora01:~ #
irora01:~ # /sbin/multipathd
irora01:~ #
irora01:~ # ps -ef |grep multipath
root 6320 2535 0 17:10 pts/0 00:00:00 grep multipath
irora01:~ #
irora01:~ # cat /etc/multipath.conf
blacklist {
wwid 36101b5442bcc700027ef1a4b75dd63be
wwid 36101b5442bcc700027ef18be5e2f735e
#devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
#devnode "sda"
#devnode "sdb"
}
multipaths {
multipath {
wwid 360060e8007c79b000030c79b00000313
alias mpatha
}
multipath {
wwid 360060e8007c79b000030c79b00000312
alias mpathb
}
multipath {
wwid 360060e8007c79b000030c79b00000211
alias mpathc
}
multipath {
wwid 360060e8007c79b000030c79b00000013
alias mpathd
}
multipath {
wwid 360060e8007c79b000030c79b00000112
alias mpathe
}
multipath {
wwid 360060e8007c79b000030c79b00000311
alias mpathf
}
multipath {
wwid 360060e8007c79b000030c79b00000212
alias mpathg
}
}
irora01:~ #
irora01:~ # multipath -ll
问题解决
能够排除HBA卡驱动问题导致无法识别外挂磁盘的情况，其实我一直怀疑是底层的某个库或者软件有异常，但始终没有找到问题所在。后来发现在/etc/udev/rules.d/下，DBA配置了UDEV规则96-asmdisk.rules，是为了方便数据库使用磁盘。
irora01:~ # cd /etc/udev/rules.d/
irora01:/etc/udev/rules.d # ls -lrt
total 88
-rw-r--r-- 1 root root 177 Feb 21 2009 81-mptctl.rules
-rw-r--r-- 1 root root 153 Jan 21 2011 99-pcsc_lite.rules
-rw-r--r-- 1 root root 357 Nov 4 2011 40-alsa.rules
-rw-r--r-- 1 root root 93 Feb 3 2012 99-iwlwifi-led.rules
-rw-r--r-- 1 root root 701 Mar 10 2015 81-mount.rules
-rw-r--r-- 1 root root 1420 Mar 10 2015 77-network.rules
-rw-r--r-- 1 root root 154 Apr 7 2015 51-packagekit-firmware.rules
-rw-r--r-- 1 root root 224 May 19 2015 85-usb_elotouch_wakeup.rules
-rw-r--r-- 1 root root 841 May 19 2015 85-usb_autosuspend_devices.rules
-rw-r--r-- 1 root root 2244 Jun 15 2015 70-kpartx.rules
-rw-r--r-- 1 root root 142 Mar 30 2020 90-rdma-umad.rules
-rw-r--r-- 1 root root 1801 Mar 30 2020 75-rdma-description.rules
-rw-r--r-- 1 root root 628 Mar 30 2020 70-persistent-ipoib.rules
-rw-r--r-- 1 root root 853 Mar 30 2020 60-rdma-persistent-naming.rules
-rw-r--r-- 1 root root 206 Mar 30 2020 60-srp_daemon.rules
-rw-r--r-- 1 root root 230 Mar 30 2020 60-rdma-ndd.rules
-rw-r--r-- 1 root root 254 Apr 2 2020 82-net-setup-link.rules
-rw-r--r-- 1 root root 784 Apr 2 2020 90-ib.rules
-rw-r--r-- 1 root root 431 Mar 25 18:01 70-persistent-cd.rules
-rw-r--r-- 1 root root 0 Mar 30 09:29 79-yast2-drivers.rules
-rw-r--r-- 1 root root 1332 Mar 30 09:29 70-persistent-net.rules
-rw-r--r-- 1 root root 1645 Apr 8 11:12 96-asmdisk.rules
-rw-r--r-- 1 root root 1733 Jul 30 14:43 10-knem.rules
irora01:/etc/udev/rules.d #
到这里似乎发现问题所在，DBA配置的udev规则中KERNEL==后面接的是sd，但实际上这里是要操作多路径聚合后的磁盘，聚合后的磁盘通常是指向/dev/dm?的逻辑设备。这里写的sd会使得UDEV规则将指定的sd开头的磁盘命名为/dev/asmdisk/asmdisk-?，这样就跳过了多路径软件对不同路径下相同磁盘的聚合作用，多路径完全成了摆设，磁盘也就没法实现链路冗余。也因为外挂磁盘被udev重命名了，所以fdisk -l无法看到。那我们验证下是不是这问题。
当前96-asmdisk.rules：
irora01:/etc/udev/rules.d # cat 96-asmdisk.rules
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000313", NAME="asmdisk/asmdisk-a", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000312", NAME="asmdisk/asmdisk-b", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000211", NAME="asmdisk/asmdisk-c", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000013", NAME="asmdisk/asmdisk-d", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000112", NAME="asmdisk/asmdisk-e", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000311", NAME="asmdisk/asmdisk-f", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000212", NAME="asmdisk/asmdisk-g", OWNER="grid", GROUP="oinstall", MODE="0660"
irora01:/etc/udev/rules.d #
irora01:/etc/udev/rules.d # ll /dev/asmdisk/
total 0
brw-rw---- 1 grid oinstall 65, 208 Aug 3 14:37 asmdisk-a
brw-rw---- 1 grid oinstall 8, 224 Jul 30 12:58 asmdisk-b
brw-rw---- 1 grid oinstall 65, 112 Aug 3 14:36 asmdisk-c
brw-rw---- 1 grid oinstall 65, 32 Aug 3 14:36 asmdisk-d
brw-rw---- 1 grid oinstall 65, 160 Aug 3 14:36 asmdisk-e
brw-rw---- 1 grid oinstall 65, 128 Aug 3 14:36 asmdisk-f
brw-rw---- 1 grid oinstall 65, 176 Aug 3 14:36 asmdisk-g
修改96-asmdisk.rules将sd替换成dm
irora01:/etc/udev/rules.d # sed -i 's/sd/dm/g' 96-asmdisk.rules
irora01:/etc/udev/rules.d #
irora01:/etc/udev/rules.d # cat 96-asmdisk.rules
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000313", NAME="asmdisk/asmdisk-a", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000312", NAME="asmdisk/asmdisk-b", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000211", NAME="asmdisk/asmdisk-c", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000013", NAME="asmdisk/asmdisk-d", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000112", NAME="asmdisk/asmdisk-e", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000311", NAME="asmdisk/asmdisk-f", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000212", NAME="asmdisk/asmdisk-g", OWNER="grid", GROUP="oinstall", MODE="0660"
irora01:/etc/udev/rules.d #
udevadm触发udev规则
irora01:/etc/udev/rules.d # udevadm trigger --subsystem-match=block --sysname-match=dm-* --verbose
/sys/devices/virtual/block/dm-0
/sys/devices/virtual/block/dm-1
/sys/devices/virtual/block/dm-10
/sys/devices/virtual/block/dm-2
/sys/devices/virtual/block/dm-3
/sys/devices/virtual/block/dm-4
/sys/devices/virtual/block/dm-5
/sys/devices/virtual/block/dm-6
/sys/devices/virtual/block/dm-7
/sys/devices/virtual/block/dm-8
/sys/devices/virtual/block/dm-9
irora01:/etc/udev/rules.d #
fdisk -l查看还是没有FC磁盘
irora01:/etc/udev/rules.d # fdisk -l
Disk /dev/sdb: 2697.0 GB, 2696999337984 bytes
255 heads, 63 sectors/track, 327891 cylinders, total 5267576832 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sda: 479.0 GB, 478998953984 bytes
255 heads, 63 sectors/track, 58234 cylinders, total 935544832 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000a1f9c
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 10489855 5243904 83 Linux
/dev/sda2 10489856 935544831 462527488 8e Linux LVM
重新扫一下盘，还是一样
irora01:/etc/udev/rules.d # echo "1" > /sys/class/fc_host/host7/issue_lip
irora01:/etc/udev/rules.d # echo "1" > /sys/class/fc_host/host8/issue_lip
irora01:/etc/udev/rules.d #
irora01:/etc/udev/rules.d # fdisk -l
Disk /dev/sdb: 2697.0 GB, 2696999337984 bytes
255 heads, 63 sectors/track, 327891 cylinders, total 5267576832 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sda: 479.0 GB, 478998953984 bytes
255 heads, 63 sectors/track, 58234 cylinders, total 935544832 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000a1f9c
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 10489855 5243904 83 Linux
/dev/sda2 10489856 935544831 462527488 8e Linux LVM
irora01:/etc/udev/rules.d #
其实这里需要先触块设备的udev规则，只指定block设备，这次FC磁盘出现了
irora01:/etc/udev/rules.d # udevadm trigger --subsystem-match=block --verbose
/sys/devices/pci0000:00/0000:00:03.2/0000:03:00.0/host0/target0:2:0/0:2:0:0/block/sda
/sys/devices/pci0000:00/0000:00:03.2/0000:03:00.0/host0/target0:2:0/0:2:0:0/block/sda/sda1
/sys/devices/pci0000:00/0000:00:03.2/0000:03:00.0/host0/target0:2:0/0:2:0:0/block/sda/sda2
/sys/devices/pci0000:00/0000:00:03.2/0000:03:00.0/host0/target0:2:1/0:2:1:0/block/sdb
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-2/target7:0:0/7:0:0:0/block/sdc
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-2/target7:0:0/7:0:0:1/block/sdd
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-2/target7:0:0/7:0:0:2/block/sde
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-2/target7:0:0/7:0:0:3/block/sdf
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-2/target7:0:0/7:0:0:4/block/sdg
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-2/target7:0:0/7:0:0:5/block/sdh
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-2/target7:0:0/7:0:0:6/block/sdi
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-3/target7:0:1/7:0:1:0/block/sdj
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-3/target7:0:1/7:0:1:1/block/sdk
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-3/target7:0:1/7:0:1:2/block/sdl
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-3/target7:0:1/7:0:1:3/block/sdm
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-3/target7:0:1/7:0:1:4/block/sdn
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-3/target7:0:1/7:0:1:5/block/sdo
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.0/host7/rport-7:0-3/target7:0:1/7:0:1:6/block/sdp
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-2/target8:0:0/8:0:0:0/block/sdq
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-2/target8:0:0/8:0:0:1/block/sdr
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-2/target8:0:0/8:0:0:2/block/sds
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-2/target8:0:0/8:0:0:3/block/sdt
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-2/target8:0:0/8:0:0:4/block/sdu
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-2/target8:0:0/8:0:0:5/block/sdv
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-2/target8:0:0/8:0:0:6/block/sdw
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-3/target8:0:1/8:0:1:0/block/sdx
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-3/target8:0:1/8:0:1:1/block/sdy
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-3/target8:0:1/8:0:1:2/block/sdz
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-3/target8:0:1/8:0:1:3/block/sdaa
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-3/target8:0:1/8:0:1:4/block/sdab
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-3/target8:0:1/8:0:1:5/block/sdac
/sys/devices/pci0000:40/0000:40:03.2/0000:45:00.1/host8/rport-8:0-3/target8:0:1/8:0:1:6/block/sdad
/sys/devices/virtual/block/loop0
/sys/devices/virtual/block/loop1
/sys/devices/virtual/block/loop2
/sys/devices/virtual/block/loop3
/sys/devices/virtual/block/loop4
/sys/devices/virtual/block/loop5
/sys/devices/virtual/block/loop6
/sys/devices/virtual/block/loop7
/sys/devices/virtual/block/dm-0
/sys/devices/virtual/block/dm-1
/sys/devices/virtual/block/dm-10
/sys/devices/virtual/block/dm-2
/sys/devices/virtual/block/dm-3
/sys/devices/virtual/block/dm-4
/sys/devices/virtual/block/dm-5
/sys/devices/virtual/block/dm-6
/sys/devices/virtual/block/dm-7
/sys/devices/virtual/block/dm-8
/sys/devices/virtual/block/dm-9
触发完成块设备udev规则fdisk -l查看磁盘，外挂磁盘已经可以看到
irora01:/etc/udev/rules.d # fdisk -l
Disk /dev/sdb: 2697.0 GB, 2696999337984 bytes
255 heads, 63 sectors/track, 327891 cylinders, total 5267576832 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders, total 419430400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sde: 429.5 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders, total 838860800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sde doesn't contain a valid partition table
Disk /dev/sdd: 429.5 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders, total 838860800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
略........
lsscsi -sg也可以看到/dev/下的sd？磁盘了
irora01:/etc/udev/rules.d # lsscsi -sg
[0:2:0:0] disk LSI LSI 3.40 /dev/sda /dev/sg0 478GB
[0:2:1:0] disk LSI LSI 3.40 /dev/sdb /dev/sg1 2.69TB
[7:0:0:0] disk HITACHI OPEN-V 8001 /dev/sdc /dev/sg2 214GB
[7:0:0:1] disk HITACHI OPEN-V 8001 /dev/sdd /dev/sg3 429GB
[7:0:0:2] disk HITACHI OPEN-V 8001 /dev/sde /dev/sg4 429GB
[7:0:0:3] disk HITACHI OPEN-V 8001 /dev/sdf /dev/sg5 429GB
[7:0:0:4] disk HITACHI OPEN-V 8001 /dev/sdg /dev/sg6 644GB
[7:0:0:5] disk HITACHI OPEN-V 8001 /dev/sdh /dev/sg7 21.4GB
[7:0:0:6] disk HITACHI OPEN-V 8001 /dev/sdi /dev/sg8 10.7GB
[7:0:1:0] disk HITACHI OPEN-V 8001 /dev/sdj /dev/sg9 214GB
[7:0:1:1] disk HITACHI OPEN-V 8001 /dev/sdk /dev/sg10 429GB
[7:0:1:2] disk HITACHI OPEN-V 8001 /dev/sdl /dev/sg11 429GB
[7:0:1:3] disk HITACHI OPEN-V 8001 /dev/sdm /dev/sg12 429GB
[7:0:1:4] disk HITACHI OPEN-V 8001 /dev/sdn /dev/sg13 644GB
[7:0:1:5] disk HITACHI OPEN-V 8001 /dev/sdo /dev/sg14 21.4GB
[7:0:1:6] disk HITACHI OPEN-V 8001 /dev/sdp /dev/sg15 10.7GB
[8:0:0:0] disk HITACHI OPEN-V 8001 /dev/sdq /dev/sg16 214GB
[8:0:0:1] disk HITACHI OPEN-V 8001 /dev/sdr /dev/sg17 429GB
[8:0:0:2] disk HITACHI OPEN-V 8001 /dev/sds /dev/sg18 429GB
[8:0:0:3] disk HITACHI OPEN-V 8001 /dev/sdt /dev/sg19 429GB
[8:0:0:4] disk HITACHI OPEN-V 8001 /dev/sdu /dev/sg20 644GB
[8:0:0:5] disk HITACHI OPEN-V 8001 /dev/sdv /dev/sg21 21.4GB
[8:0:0:6] disk HITACHI OPEN-V 8001 /dev/sdw /dev/sg22 10.7GB
[8:0:1:0] disk HITACHI OPEN-V 8001 /dev/sdx /dev/sg23 214GB
[8:0:1:1] disk HITACHI OPEN-V 8001 /dev/sdy /dev/sg24 429GB
[8:0:1:2] disk HITACHI OPEN-V 8001 /dev/sdz /dev/sg25 429GB
[8:0:1:3] disk HITACHI OPEN-V 8001 /dev/sdaa /dev/sg26 429GB
[8:0:1:4] disk HITACHI OPEN-V 8001 /dev/sdab /dev/sg27 644GB
[8:0:1:5] disk HITACHI OPEN-V 8001 /dev/sdac /dev/sg28 21.4GB
[8:0:1:6] disk HITACHI OPEN-V 8001 /dev/sdad /dev/sg29 10.7GB
irora01:/etc/udev/rules.d #
multipath -v2初始化多路径已经没有问题
irora01:/etc/udev/rules.d # multipath -ll
irora01:/etc/udev/rules.d #
irora01:/etc/udev/rules.d # multipath -v2
create: mpathc (360060e8007c79b000030c79b00000211) undef HITACHI,OPEN-V
size=200G features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
|- 7:0:0:0 sdc 8:32 undef ready running
|- 7:0:1:0 sdj 8:144 undef ready running
|- 8:0:0:0 sdq 65:0 undef ready running
`- 8:0:1:0 sdx 65:112 undef ready running
create: mpathf (360060e8007c79b000030c79b00000311) undef HITACHI,OPEN-V
size=400G features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
|- 7:0:0:1 sdd 8:48 undef ready running
|- 7:0:1:1 sdk 8:160 undef ready running
|- 8:0:0:1 sdr 65:16 undef ready running
`- 8:0:1:1 sdy 65:128 undef ready running
略........
查看下/dev/mapper/下的设备，注意mpath?的设备
irora01:~ # ll /dev/mapper/
total 0
crw-rw---- 1 root root 10, 236 Aug 4 00:25 control
lrwxrwxrwx 1 root root 8 Aug 3 17:03 data-data -> ../dm-10
lrwxrwxrwx 1 root root 8 Aug 3 17:04 mpatha -> ../dm-17
lrwxrwxrwx 1 root root 8 Aug 3 17:04 mpathb -> ../dm-16
lrwxrwxrwx 1 root root 8 Aug 3 17:04 mpathc -> ../dm-11
lrwxrwxrwx 1 root root 8 Aug 3 17:04 mpathd -> ../dm-13
lrwxrwxrwx 1 root root 8 Aug 3 17:04 mpathe -> ../dm-14
lrwxrwxrwx 1 root root 8 Aug 3 17:04 mpathf -> ../dm-12
lrwxrwxrwx 1 root root 8 Aug 3 17:04 mpathg -> ../dm-15
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-home -> ../dm-0
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-opt -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-opt_orabak -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-opt_oracle -> ../dm-3
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-opt_oracrs -> ../dm-4
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-root -> ../dm-5
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-swap -> ../dm-6
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-tmp -> ../dm-7
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-usr -> ../dm-8
lrwxrwxrwx 1 root root 7 Aug 3 17:03 vg1-var -> ../dm-9
再触发下匹配dm-*的block设备的udev
irora01:/etc/udev/rules.d # udevadm trigger --subsystem-match=block --sysname-match=dm-* --verbose
/sys/devices/virtual/block/dm-0
/sys/devices/virtual/block/dm-1
/sys/devices/virtual/block/dm-10
/sys/devices/virtual/block/dm-11
/sys/devices/virtual/block/dm-12
/sys/devices/virtual/block/dm-13
/sys/devices/virtual/block/dm-14
/sys/devices/virtual/block/dm-15
/sys/devices/virtual/block/dm-16
/sys/devices/virtual/block/dm-17
/sys/devices/virtual/block/dm-2
/sys/devices/virtual/block/dm-3
/sys/devices/virtual/block/dm-4
/sys/devices/virtual/block/dm-5
/sys/devices/virtual/block/dm-6
/sys/devices/virtual/block/dm-7
/sys/devices/virtual/block/dm-8
/sys/devices/virtual/block/dm-9
irora01:/etc/udev/rules.d #
irora01:/etc/udev/rules.d # ll /dev/asmdisk/
total 0
brw-rw---- 1 grid oinstall 253, 17 Aug 3 17:23 asmdisk-a
brw-rw---- 1 grid oinstall 253, 16 Aug 3 17:23 asmdisk-b
brw-rw---- 1 grid oinstall 253, 11 Aug 3 17:23 asmdisk-c
brw-rw---- 1 grid oinstall 253, 13 Aug 3 17:23 asmdisk-d
brw-rw---- 1 grid oinstall 253, 14 Aug 3 17:23 asmdisk-e
brw-rw---- 1 grid oinstall 253, 12 Aug 3 17:23 asmdisk-f
brw-rw---- 1 grid oinstall 253, 15 Aug 3 17:23 asmdisk-g
irora01:/etc/udev/rules.d #
irora01:/etc/udev/rules.d # rcmultipathd restart
Shutting down multipathd done
Starting multipathd done
irora01:/etc/udev/rules.d #
再看下/dev/mapper/下的设备，注意mpath?设备的指向已经发生变化
irora01:~ # ll /dev/mapper/
total 0
lrwxrwxrwx 1 root root 8 Aug 5 11:46 36101b5442bcc700027ef33222bef6d91 -> ../dm-18
crw-rw---- 1 root root 10, 236 Aug 4 00:25 control
lrwxrwxrwx 1 root root 8 Aug 5 11:46 data-data -> ../dm-10
lrwxrwxrwx 1 root root 20 Aug 5 11:46 mpatha -> ../asmdisk/asmdisk-a
lrwxrwxrwx 1 root root 20 Aug 5 11:46 mpathb -> ../asmdisk/asmdisk-b
lrwxrwxrwx 1 root root 20 Aug 5 11:46 mpathc -> ../asmdisk/asmdisk-c
lrwxrwxrwx 1 root root 20 Aug 5 11:46 mpathd -> ../asmdisk/asmdisk-d
lrwxrwxrwx 1 root root 20 Aug 5 11:46 mpathe -> ../asmdisk/asmdisk-e
lrwxrwxrwx 1 root root 20 Aug 5 11:46 mpathf -> ../asmdisk/asmdisk-f
lrwxrwxrwx 1 root root 20 Aug 5 11:46 mpathg -> ../asmdisk/asmdisk-g
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-home -> ../dm-0
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-opt -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-opt_orabak -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-opt_oracle -> ../dm-3
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-opt_oracrs -> ../dm-4
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-root -> ../dm-5
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-swap -> ../dm-6
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-tmp -> ../dm-7
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-usr -> ../dm-8
lrwxrwxrwx 1 root root 7 Aug 5 11:46 vg1-var -> ../dm-9
从上面可以看到经过udev处理后mpath？指向了asmdisk/asmdisk-？，其实这并不是我们想要的，我们更希望asmdisk/asmdisk-？是mpath？的一个软连接，于是需要修改下udev规则，将NAME=替换成SYMLINK+=
irora01:~ # cat /etc/udev/rules.d/96-disk.rules
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000313", SYMLINK+="asmdisk/asmdisk-a", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000312", SYMLINK+="asmdisk/asmdisk-b", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000211", SYMLINK+="asmdisk/asmdisk-c", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000013", SYMLINK+="asmdisk/asmdisk-d", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000112", SYMLINK+="asmdisk/asmdisk-e", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000311", SYMLINK+="asmdisk/asmdisk-f", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="360060e8007c79b000030c79b00000212", SYMLINK+="asmdisk/asmdisk-g", OWNER="grid", GROUP="oinstall", MODE="0660"
重启设备或重新触发udev再查看，mpath和asmdisk-同时指向了/dev/dm-?，这就是我们想要的结果。
irora01:/etc/udev/rules.d # udevadm trigger --subsystem-match=block --sysname-match=dm-* --verbose
irora01:~ # ll /dev/mapper/
total 0
crw-rw---- 1 root root 10, 236 Aug 6 2021 control
lrwxrwxrwx 1 root root 8 Aug 6 2021 data-data -> ../dm-10
lrwxrwxrwx 1 root root 8 Aug 6 13:08 mpatha -> ../dm-17
lrwxrwxrwx 1 root root 8 Aug 6 13:08 mpathb -> ../dm-16
lrwxrwxrwx 1 root root 8 Aug 6 13:19 mpathc -> ../dm-11
lrwxrwxrwx 1 root root 8 Aug 6 13:19 mpathd -> ../dm-13
lrwxrwxrwx 1 root root 8 Aug 6 13:19 mpathe -> ../dm-14
lrwxrwxrwx 1 root root 8 Aug 6 13:19 mpathf -> ../dm-12
lrwxrwxrwx 1 root root 8 Aug 6 13:19 mpathg -> ../dm-15
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-home -> ../dm-0
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-opt -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-opt_orabak -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-opt_oracle -> ../dm-3
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-opt_oracrs -> ../dm-4
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-root -> ../dm-5
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-swap -> ../dm-6
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-tmp -> ../dm-7
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-usr -> ../dm-8
lrwxrwxrwx 1 root root 7 Aug 6 2021 vg1-var -> ../dm-9
irorabj26:~ # ll /dev/asmdisk/
total 0
lrwxrwxrwx 1 root root 8 Aug 6 13:08 asmdisk-a -> ../dm-17
lrwxrwxrwx 1 root root 8 Aug 6 13:08 asmdisk-b -> ../dm-16
lrwxrwxrwx 1 root root 8 Aug 6 13:19 asmdisk-c -> ../dm-11
lrwxrwxrwx 1 root root 8 Aug 6 13:19 asmdisk-d -> ../dm-13
lrwxrwxrwx 1 root root 8 Aug 6 13:19 asmdisk-e -> ../dm-14
lrwxrwxrwx 1 root root 8 Aug 6 13:19 asmdisk-f -> ../dm-12
lrwxrwxrwx 1 root root 8 Aug 6 13:19 asmdisk-g -> ../dm-15
tigs
multipath基本操作命令
# /etc/init.d/multipathd start #开启mulitipath服务
# multipath -F #删除现有路径
# multipath -v2 #格式化路径
# multipath -ll #查看多路径
永远年轻，永远热泪盈眶
