oracel RAC 心跳线对集群的影响
测试，oracle集群心跳线断开测试集群状态，和实例状态
测试环境 Oracle VM VirtualBox + centos7.6 + oracle 11.2.0.4
首先登陆系统 ，切换grid用户 查看集群状态
[grid@ora01 ~]$ [grid@ora01 ~]$ crsctl check cluster -all ************************************************************** ora01: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** ora02: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** [grid@ora01 ~]$ crsctl status resource -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE ora01 ONLINE ONLINE ora02 ora.LISTENER.lsnr ONLINE ONLINE ora01 ONLINE ONLINE ora02 ora.asm ONLINE ONLINE ora01 Started ONLINE ONLINE ora02 Started ora.gsd OFFLINE OFFLINE ora01 OFFLINE OFFLINE ora02 ora.net1.network ONLINE ONLINE ora01 ONLINE ONLINE ora02 ora.ons ONLINE ONLINE ora01 ONLINE ONLINE ora02 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE ora01 ora.cvu 1 ONLINE ONLINE ora01 ora.oc4j 1 ONLINE ONLINE ora01 ora.ora01.vip 1 ONLINE ONLINE ora01 ora.ora02.vip 1 ONLINE ONLINE ora02 ora.racdb.db 1 ONLINE ONLINE ora02 Open 2 ONLINE ONLINE ora01 Open ora.scan1.vip 1 ONLINE ONLINE ora01 [grid@ora01 ~]$
然后执行将心跳网卡进行断开
[root@ora01 ~]# ifconfig -a
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.101 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::bd5b:e413:3edd:c01f prefixlen 64 scopeid 0x20<link>
ether 08:00:27:00:9c:5a txqueuelen 1000 (Ethernet)
RX packets 3572 bytes 400791 (391.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2350 bytes 564088 (550.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s3:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.103 netmask 255.255.255.0 broadcast 192.168.10.255
ether 08:00:27:00:9c:5a txqueuelen 1000 (Ethernet)
enp0s3:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.105 netmask 255.255.255.0 broadcast 192.168.10.255
ether 08:00:27:00:9c:5a txqueuelen 1000 (Ethernet)
enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.11 netmask 255.255.255.0 broadcast 10.10.10.255
ether 08:00:27:11:ea:cc txqueuelen 1000 (Ethernet)
RX packets 79827 bytes 50003883 (47.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 122597 bytes 111817875 (106.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s8:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.3.181 netmask 255.255.0.0 broadcast 169.254.255.255
ether 08:00:27:11:ea:cc txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 41231 bytes 27932232 (26.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 41231 bytes 27932232 (26.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ora01 ~]# ifconfig enp0s8 down
[root@ora01 ~]# ethtool enp0s8
Settings for enp0s8:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown (auto)
Supports Wake-on: umbg
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
[root@ora01 ~]#
可看到enp0s8 卡执行断开处于未连接状态。
节点一上ping 心跳地址
[root@ora01 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.101 ora01 192.168.10.102 ora02 192.168.10.103 ora01-vip 192.168.10.104 ora02-vip 192.168.10.105 rac-scan 10.10.10.11 ora01-priv 10.10.10.12 ora02-priv [root@ora01 ~]# ping 10.10.10.11 PING 10.10.10.11 (10.10.10.11) 56(84) bytes of data. 64 bytes from 10.10.10.11: icmp_seq=1 ttl=64 time=0.018 ms 64 bytes from 10.10.10.11: icmp_seq=2 ttl=64 time=0.021 ms ^C --- 10.10.10.11 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.018/0.019/0.021/0.004 ms [root@ora01 ~]# ping 10.10.10.12 connect: Network is unreachable [root@ora01 ~]# ping 10.10.10.12 connect: Network is unreachable [root@ora01 ~]#
节点二上发起ping
[root@ora02 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.101 ora01 192.168.10.102 ora02 192.168.10.103 ora01-vip 192.168.10.104 ora02-vip 192.168.10.105 rac-scan 10.10.10.11 ora01-priv 10.10.10.12 ora02-priv [root@ora02 ~]# ping 10.10.10.11 PING 10.10.10.11 (10.10.10.11) 56(84) bytes of data. From 10.10.10.12 icmp_seq=1 Destination Host Unreachable From 10.10.10.12 icmp_seq=2 Destination Host Unreachable From 10.10.10.12 icmp_seq=3 Destination Host Unreachable From 10.10.10.12 icmp_seq=4 Destination Host Unreachable From 10.10.10.12 icmp_seq=5 Destination Host Unreachable From 10.10.10.12 icmp_seq=6 Destination Host Unreachable From 10.10.10.12 icmp_seq=7 Destination Host Unreachable ^C --- 10.10.10.11 ping statistics --- 8 packets transmitted, 0 received, +7 errors, 100% packet loss, time 7003ms pipe 4 [root@ora02 ~]# ping 10.10.10.12 PING 10.10.10.12 (10.10.10.12) 56(84) bytes of data. 64 bytes from 10.10.10.12: icmp_seq=1 ttl=64 time=0.012 ms 64 bytes from 10.10.10.12: icmp_seq=2 ttl=64 time=0.047 ms 64 bytes from 10.10.10.12: icmp_seq=3 ttl=64 time=0.038 ms 64 bytes from 10.10.10.12: icmp_seq=4 ttl=64 time=0.024 ms 64 bytes from 10.10.10.12: icmp_seq=5 ttl=64 time=0.025 ms ^C --- 10.10.10.12 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.012/0.029/0.047/0.012 ms [root@ora02 ~]#
查看集群状态 ，（crsctl check cluster -all 此处的命令运行会很慢，）
[grid@ora01 ~]$ crsctl check cluster -all ************************************************************** ora01: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** CRS-4404: The following nodes did not reply within the allotted time: ora02 [grid@ora01 ~]$ crsctl status resource -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE ora01 ora.LISTENER.lsnr ONLINE ONLINE ora01 ora.asm ONLINE ONLINE ora01 Started ora.gsd OFFLINE OFFLINE ora01 ora.net1.network ONLINE ONLINE ora01 ora.ons ONLINE ONLINE ora01 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE ora01 ora.cvu 1 ONLINE ONLINE ora01 ora.oc4j 1 ONLINE ONLINE ora01 ora.ora01.vip 1 ONLINE ONLINE ora01 ora.ora02.vip 1 ONLINE INTERMEDIATE ora01 FAILED OVER ora.racdb.db 1 ONLINE OFFLINE 2 ONLINE ONLINE ora01 Open ora.scan1.vip 1 ONLINE ONLINE ora01 [grid@ora01 ~]$
节点二上执行
[grid@ora02 ~]$ crsctl status resource -t CRS-4535: Cannot communicate with Cluster Ready Services CRS-4000: Command Status failed, or completed with errors. [grid@ora02 ~]$
验证scan ip
[root@ora01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.101 ora01
192.168.10.102 ora02
192.168.10.103 ora01-vip
192.168.10.104 ora02-vip
192.168.10.105 rac-scan
10.10.10.11 ora01-priv
10.10.10.12 ora02-priv
[root@ora01 ~]# ifconfig -a
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.101 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::bd5b:e413:3edd:c01f prefixlen 64 scopeid 0x20<link>
ether 08:00:27:00:9c:5a txqueuelen 1000 (Ethernet)
RX packets 4098 bytes 450704 (440.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2705 bytes 624291 (609.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s3:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.103 netmask 255.255.255.0 broadcast 192.168.10.255
ether 08:00:27:00:9c:5a txqueuelen 1000 (Ethernet)
enp0s3:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.105 netmask 255.255.255.0 broadcast 192.168.10.255
ether 08:00:27:00:9c:5a txqueuelen 1000 (Ethernet)
enp0s3:3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.104 netmask 255.255.255.0 broadcast 192.168.10.255
ether 08:00:27:00:9c:5a txqueuelen 1000 (Ethernet)
enp0s8: flags=4098<BROADCAST,MULTICAST> mtu 1500
inet 10.10.10.11 netmask 255.255.255.0 broadcast 10.10.10.255
ether 08:00:27:11:ea:cc txqueuelen 1000 (Ethernet)
RX packets 80248 bytes 50249772 (47.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 122987 bytes 112043888 (106.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s8:1: flags=4098<BROADCAST,MULTICAST> mtu 1500
inet 169.254.3.181 netmask 255.255.0.0 broadcast 169.254.255.255
ether 08:00:27:11:ea:cc txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 47186 bytes 34097104 (32.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 47186 bytes 34097104 (32.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ora01 ~]#
验证数据库连接是否正常：
[oracle@ora01 ~]$ sqlplus system/oracle@192.168.10.105:1521/racdb
SQL*Plus: Release 11.2.0.4.0 Production on Wed Jun 29 08:09:29 2022
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select username from t_user where rownum<10
2 /
select username from t_user where rownum<10
*
ERROR at line 1:
ORA-00942: table or view does not exist
SQL> c/t_user/dba_users
1* select username from dba_users where rownum<10
SQL> /
USERNAME
------------------------------
SYS
SYSTEM
WENXIAO
OUTLN
MGMT_VIEW
FLOWS_FILES
MDSYS
ORDSYS
EXFSYS
9 rows selected.
SQL> select instance_name ,status from gv$instances
2 /
select instance_name ,status from gvnstances
*
ERROR at line 1:
ORA-00911: invalid character
SQL> select instance_name ,status from gv$instance;
INSTANCE_NAME STATUS
---------------- ------------
racdb1 OPEN
SQL> select open_mode from gv$database;
OPEN_MODE
--------------------
READ WRITE
SQL>
节点一上的心跳网卡down掉，主节点实例不受影响，racdb1正常，racdb2被提出集群，实例受到影响。
将心跳网卡启用后，集群自动恢复，
