ORAC error-节点掉出集群后重新添加
尝试重新注册节点2加入集群
1 [root@rac2 ~]# sh /u01/app/11.2.0/grid/root.sh
2 Performing root user operation for Oracle 11g
3
4 The following environment variables are set as:
5 ORACLE_OWNER= grid
6 ORACLE_HOME= /u01/app/11.2.0/grid
7 Enter the full pathname of the local bin directory: [/usr/local/bin]:
8 The contents of "dbhome" have not changed. No need to overwrite.
9 The contents of "oraenv" have not changed. No need to overwrite.
10 The contents of "coraenv" have not changed. No need to overwrite.
11 Entries will be added to the /etc/oratab file as needed by
12 Database Configuration Assistant when a database is created
13 Finished running generic part of root script.
14 Now product-specific root actions will be performed.
15 Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
16 User ignored Prerequisites during installation
17 Installing Trace File Analyzer
18 Configure Oracle Grid Infrastructure for a Cluster ... succeeded
还是有问题，清理节点2的配置信息，然后重新运行root.sh
1 [root@rac2 trace]$ /u01/app/11.2.0/grid/crs/install/rootcrs.pl -verbose -deconfig -force
2 [root@rac2 ~]# /u01/app/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig -force
3 [root@rac2 bin]# /u01/app/11.2.0/grid/root.sh
4
5 报错：
6 [root@rac2 install]# /u01/app/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig -force
7 Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /u01/app/11.2.0/grid/crs/install) at crsconfig_lib.pm line 703.
8 BEGIN failed--compilation aborted at crsconfig_lib.pm line 703.
9 Compilation failed in require at /u01/app/11.2.0/grid/crs/install/roothas.pl line 166.
10 BEGIN failed--compilation aborted at /u01/app/11.2.0/grid/crs/install/roothas.pl line 166.
11 缺少依赖包 安装命令 yum install perl-Env
12
13 已安装:
14 perl-Env.noarch 0:1.04-2.el7
清理节点2配置信息
1 [root@rac2 install]# /u01/app/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig -force
2 Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
3 CRS-4535: Cannot communicate with Cluster Ready Services
4 CRS-4000: Command Stop failed, or completed with errors.
5 CRS-4535: Cannot communicate with Cluster Ready Services
6 CRS-4000: Command Delete failed, or completed with errors.
7 CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
8 CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
9 CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
10 CRS-2673: Attempting to stop 'ora.crf' on 'rac2'
11 CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded
12 CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
13 CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
14 CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
15 CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
16 CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
17 CRS-4133: Oracle High Availability Services has been stopped.
18 Successfully deconfigured Oracle Restart stack
重新注册到集群中
1 [root@rac2 install]# /u01/app/11.2.0/grid/root.sh
2 Performing root user operation for Oracle 11g
3 The following environment variables are set as:
4 ORACLE_OWNER= grid
5 ORACLE_HOME= /u01/app/11.2.0/grid
6 Enter the full pathname of the local bin directory: [/usr/local/bin]:
7 The contents of "dbhome" have not changed. No need to overwrite.
8 The contents of "oraenv" have not changed. No need to overwrite.
9 The contents of "coraenv" have not changed. No need to overwrite.
10
11 Entries will be added to the /etc/oratab file as needed by
12 Database Configuration Assistant when a database is created
13 Finished running generic part of root script.
14 Now product-specific root actions will be performed.
15 Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
16 User ignored Prerequisites during installation
17 Installing Trace File Analyzer
18 OLR initialization - successful
19 Adding Clusterware entries to inittab
20 CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
21 An active cluster was found during exclusive startup, restarting to join the cluster
22 Start of resource "ora.cssd" failed
23 CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
24 CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
25 CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
26 CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
27 CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
28 CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
29 CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
30 CRS-2674: Start of 'ora.cssd' on 'rac2' failed
31 CRS-2679: Attempting to clean 'ora.cssd' on 'rac2'
32 CRS-2681: Clean of 'ora.cssd' on 'rac2' succeeded
33 CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
34 CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
35 CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac2'
36 CRS-2677: Stop of 'ora.cssdmonitor' on 'rac2' succeeded
37 CRS-5804: Communication error with agent process
38 CRS-4000: Command Start failed, or completed with errors.
39 Failed to start Oracle Grid Infrastructure stack
40 Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1278.
41 /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
42 依然失败
CSSD没有在第二个节点上启动。$grid_home/log/rac2子目录中查找cssd日志文件。查看日志信息。
1 /u01/app/11.2.0/grid/log/rac2/cssd
2 2019-10-12 15:41:19.013: [ CSSD][3199571712]clssgmDiscEndpcl: gipcDestroy 0x8a28
3 2019-10-12 15:41:19.064: [ CSSD][3181754112]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
4 2019-10-12 15:41:19.844: [ CSSD][3186484992]clssnmvDHBValidateNcopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 464729747, wrtcnt, 8055111, LATS 336904, lastSeqNo 8055110, uniqueness 1569234927, timestamp 1570866136/3845241248
5 2019-10-12 15:41:20.064: [ CSSD][3181754112]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
6 2019-10-12 15:41:20.845: [ CSSD][3186484992]clssnmvDHBValidateNcopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 464729747, wrtcnt, 8055112, LATS 337904, lastSeqNo 8055111, uniqueness 1569234927, timestamp 1570866137/3845242248
查看节点2的心跳
1 [grid@rac2 /]$ ping 20.20.20.201 --节点1的priv
2 PING 20.20.20.201 (20.20.20.201) 56(84) bytes of data.
3 From 20.20.20.202 icmp_seq=1 Destination Host Unreachable
4 From 20.20.20.202 icmp_seq=2 Destination Host Unreachable
5 From 20.20.20.202 icmp_seq=3 Destination Host Unreachable
6 From 20.20.20.202 icmp_seq=4 Destination Host Unreachable
心跳不通，节点1的心跳出过好几次问题，估计网卡有问题。
先尝试节点1的网卡重启下，然后把服务重启下，节点1/2服务都正常起来了
