Oracle Rac crs无法启动

OS:
ORACLE LINUX 5.7

DB:
11.2.0.3

RAC：YES

故障：
1、两节点RAC，节点分别为linuxdb1、linuxdb2，其中节点linuxdb2服务器出现故障，无法启动
2、只启动节点linuxdb1即可，然后供测试部门使用
3、crsctl无法启动报错:CRS-1714:Unable to discover any voting files

#crsctl start cluster -n linuxdb1

CRS-2672: Attempting to start 'ora.cssd' on 'linuxdb1'
CRS-2672: Attempting to start 'ora.diskmon' on 'linuxdb1'
CRS-2676: Start of 'ora.diskmon' on 'linuxdb1' succeeded
CRS-2674: Start of 'ora.cssd' on 'linuxdb1' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'linuxdb1'
CRS-2681: Clean of 'ora.cssd' on 'linuxdb1' succeeded
CRS-4000: Command Start failed, or completed with errors.

查看日志：
[cssd(1934)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:03.698
[/u01/app/11.2.0/grid/bin/cssdagent(1920)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:52:6} in /u01/app/11.2.0/grid/log/linuxdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2014-02-12 07:57:03.699
[cssd(1934)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:03.699
[cssd(1934)]CRS-1603:CSSD on node linuxdb1 shutdown by user.
2014-02-12 07:57:05.492
[cssd(1934)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:09.753
[ohasd(6518)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'linuxdb1'.
2014-02-12 07:57:21.551
[cssd(2113)]CRS-1713:CSSD daemon is started in clustered mode
2014-02-12 07:57:21.676
[cssd(2113)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:36.694
[cssd(2113)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:51.713
[cssd(2113)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:58:06.732
[cssd(2113)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log

查看多路径和asmlib状态：

/etc/init.d/oracleasm listdisks一切正常

 

查看磁盘信息：
#fdisk -l

Disk /dev/sdb: 2199.0 GB, 2199023255552 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      267350  2147483647+  ee  EFI GPT

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 2.2 TB (2199023255552 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID 
partition table format (GPT).

Disk /dev/sdc: 2199.0 GB, 2199023255552 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1      267350  2147483647+  ee  EFI GPT

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 2.2 TB (2199023255552 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID 
partition table format (GPT).

Disk /dev/sdd: 2199.0 GB, 2199023255552 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1      267350  2147483647+  ee  EFI GPT

WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 2.2 TB (2199023255552 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID 
partition table format (GPT).

Disk /dev/sde: 2199.0 GB, 2199023255552 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1      267350  2147483647+  ee  EFI GPT

 

查看oracleasm信息：

# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver.  It is generated
# By running /etc/init.d/oracleasm configure.  Please use that method
# to modify this file
#

# ORACLEASM_ENABELED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=asmadmin

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="multipath sd"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sdf sdg sdh sdi"

* fdisk -l 的信息与oracleasm的磁盘信息不符

fdisk -l的磁盘为sdb sdc sdd sde

将ORACLEASM_SCANEXCLUDE="sdf sdg sdh sdi" -> ORACLEASM_SCANEXCLUDE="sdb sdc sdd sde"


重启crs，OK

检查状态：

$ srvctl status asm

$ srvctl status instance -d devdb -i devdb1

$ srvctl status database -d devdb

节点linuxdb1连接数据库一切正常。