19c 重建ocr
某客户ocr盘损坏导致进群起不来需要重建ocr,集群软件版本19.3
远程节点删除ocr配置
/u01/app/grid/crs/install/rootcrs.sh -deconfig -force -verbose [root@ht02 ~]# /u01/app/grid/crs/install/rootcrs.sh -deconfig -force -verbose Using configuration parameter file: /u01/app/grid/crs/install/crsconfig_params The log of current session can be found at: /u01/app/11.2.0/grid/crsdata/ht02/crsconfig/crsdeconfig_ht02_2022-07-30_09-24-58PM.log 2022/07/30 21:25:55 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node
本地节点删除ocr配置
/u01/app/grid/crs/install/rootcrs.sh -deconfig -force -verbose -keepdg -lastnode [root@ht01 ~]# /u01/app/grid/crs/install/rootcrs.sh -deconfig -force -verbose -keepdg -lastnode Using configuration parameter file: /u01/app/grid/crs/install/crsconfig_params The log of current session can be found at: /u01/app/11.2.0/grid/crsdata/ht01/crsconfig/crsdeconfig_ht01_2022-07-30_09-27-14PM.log 2022/07/30 21:27:16 CLSRSC-332: CRS resources for listeners are still configured CRS-2673: Attempting to stop 'ora.crsd' on 'ht01' CRS-2677: Stop of 'ora.crsd' on 'ht01' succeeded ASM de-configuration trace file location: /u01/app/11.2.0/grid/cfgtoollogs/asmca/asmcadc_clean2022-07-30_09-33-05PM.log ASM Clean Configuration START ASM Clean Configuration END ASM instance deleted successfully. Check /u01/app/11.2.0/grid/cfgtoollogs/asmca/asmcadc_clean2022-07-30_09-33-05PM.log for details. 2022/07/30 21:34:38 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node 2022/07/30 21:34:39 CLSRSC-559: Ensure that the GPnP profile data under the 'gpnp' directory in /u01/app/grid is deleted on each node before using the software in the current Grid Infrastructure home for reconfiguration.
删除ocr配置完毕之后需要重新配置ocr
重新配置
[+ASM1]@ht01[/home/grid]$/u01/app/grid/gridSetup.sh -silent -responseFile /home/grid/grid.rsp Launching Oracle Grid Infrastructure Setup Wizard... [FATAL] [INS-45511] Installer has detected that an Oracle Grid Infrastructure Home (/u01/app/grid) is marked incorrectly as a configured home in the central inventory. ACTION: As the Oracle Grid Infrastructure is not configured in the home, the central inventory should be updated to indicate the same. Execute the following command on the nodes where the home exists to update the central inventory, and then continue with the install: /u01/app/grid/oui/bin/runInstaller -updateNodeList -silent ORACLE_HOME=/u01/app/grid -local CRS=false [FATAL] [INS-41101] The specified interface information is incorrect. CAUSE: Either the format of the information entered was incorrect, or the interface name or subnet information was incorrect. ACTION: Enter the correct interface information before proceeding
上面报错需要修改inventory.xml文件
查看inventory.xml没修改前
[+ASM1]@ht01[/u01/app/11.2.0/oraInventory/ContentsXML]$cat inventory.xml <?xml version="1.0" standalone="yes" ?> <!-- Copyright (c) 1999, 2022, Oracle and/or its affiliates. All rights reserved. --> <!-- Do not modify the contents of this file by hand. --> <INVENTORY> <VERSION_INFO> <SAVED_WITH>12.2.0.7.0</SAVED_WITH> <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER> </VERSION_INFO> <HOME_LIST> <HOME NAME="OraGI19Home1" LOC="/u01/app/grid" TYPE="O" IDX="1" CRS="true"/> <HOME NAME="OraDB19Home1" LOC="/u01/app/oracle/product/19c/dbhome_1" TYPE="O" IDX="2"/> </HOME_LIST> <COMPOSITEHOME_LIST> </COMPOSITEHOME_LIST> </INVENTORY>
修改inventory.xml及查看,2个节点都需要执行
/u01/app/grid/oui/bin/runInstaller -updateNodeList -silent ORACLE_HOME=/u01/app/grid -local CRS=false [+ASM1]@ht01[/u01/app/11.2.0/oraInventory/ContentsXML]$cat inventory.xml <?xml version="1.0" standalone="yes" ?> <!-- Copyright (c) 1999, 2022, Oracle and/or its affiliates. All rights reserved. --> <!-- Do not modify the contents of this file by hand. --> <INVENTORY> <VERSION_INFO> <SAVED_WITH>12.2.0.7.0</SAVED_WITH> <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER> </VERSION_INFO> <HOME_LIST> <HOME NAME="OraGI19Home1" LOC="/u01/app/grid" TYPE="O" IDX="1"/> <HOME NAME="OraDB19Home1" LOC="/u01/app/oracle/product/19c/dbhome_1" TYPE="O" IDX="2"/> </HOME_LIST> <COMPOSITEHOME_LIST> </COMPOSITEHOME_LIST> </INVENTORY>
清空磁盘头
[root@ht01 ~]# dd if=/dev/zero of=/dev/asm/asm_vote1 count=10 bs=10M 10+0 records in 10+0 records out 104857600 bytes (105 MB) copied, 0.0799074 s, 1.3 GB/s [root@ht01 ~]# dd if=/dev/zero of=/dev/asm/asm_vote2 count=10 bs=10M 10+0 records in 10+0 records out 104857600 bytes (105 MB) copied, 0.080379 s, 1.3 GB/s [root@ht01 ~]# dd if=/dev/zero of=/dev/asm/asm_vote3 count=10 bs=10M 10+0 records in 10+0 records out 104857600 bytes (105 MB) copied, 0.08005 s, 1.3 GB/s
继续配置ocr
[+ASM1]@ht01[/home/grid]$/u01/app/grid/gridSetup.sh -silent -ignorePrereqFailure -responseFile /home/grid/grid.rsp Launching Oracle Grid Infrastructure Setup Wizard... [WARNING] [INS-30011] The SYS password entered does not conform to the Oracle recommended standards. CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9]. ACTION: Provide a password that conforms to the Oracle recommended standards. [WARNING] [INS-30011] The ASMSNMP password entered does not conform to the Oracle recommended standards. CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9]. ACTION: Provide a password that conforms to the Oracle recommended standards. Enter password for 'root' user: [WARNING] [INS-13013] Target environment does not meet some mandatory requirements. CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/11.2.0/oraInventory/logs/GridSetupActions2022-07-30_11-44-23PM/gridSetupActions2022-07-30_11-44-23PM.log ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/11.2.0/oraInventory/logs/GridSetupActions2022-07-30_11-44-23PM/gridSetupActions2022-07-30_11-44-23PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually. The response file for this session can be found at: /u01/app/grid/install/response/grid_2022-07-30_11-44-23PM.rsp
具体日志报OUI-10166 and OUI-10175错误,需要对oracle_home解锁,参考mos19c upgrade fails with OUI-10166 and OUI-10175 due to incorrect ownership of rootconfig.sh after -dryRunForUpgrade (Doc ID 2581014.1)
[root@ht01 tmp]# /u01/app/grid/crs/install/rootcrs.sh -unlock -crshome /u01/app/grid Using configuration parameter file: /u01/app/grid/crs/install/crsconfig_params The log of current session can be found at: /u01/app/11.2.0/grid/crsdata/ht01/crsconfig/crsunlock_ht01_2022-07-31_00-02-26AM.log 2022/07/31 00:02:27 CLSRSC-347: Successfully unlock /u01/app/grid
继续配置ocr
[+ASM1]@ht01[/home/grid]$/u01/app/grid/gridSetup.sh -silent -ignorePrereqFailure -responseFile /home/grid/grid.rsp Launching Oracle Grid Infrastructure Setup Wizard... [FATAL] [INS-40414] The installer has detected an unused Oracle Cluster Registry (OCR) location pointer file (/etc/oracle/ocr.loc) on the system. CAUSE: OCR location pointer file (/etc/oracle/ocr.loc) is left behind from a previous installation. ACTION: Delete the OCR location pointer file (/etc/oracle/ocr.loc) and restart the installer.
ocr.loc 文件是个空文件需要删除
[root@ht01 tmp]# ls -lrt /etc/oracle/ocr.loc -rw-r--r-- 1 root oinstall 0 Jul 31 00:02 /etc/oracle/ocr.loc [root@ht01 tmp]# rm /etc/oracle/ocr.loc rm: remove regular empty file ‘/etc/oracle/ocr.loc’? y [root@ht01 tmp]# ls -lrt /etc/oracle/ocr.loc ls: cannot access /etc/oracle/ocr.loc: No such file or directory
继续配置ocr
[+ASM1]@ht01[/home/grid]$/u01/app/grid/gridSetup.sh -silent -ignorePrereqFailure -responseFile /home/grid/grid.rsp Launching Oracle Grid Infrastructure Setup Wizard... [WARNING] [INS-30011] The SYS password entered does not conform to the Oracle recommended standards. CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9]. ACTION: Provide a password that conforms to the Oracle recommended standards. [WARNING] [INS-30011] The ASMSNMP password entered does not conform to the Oracle recommended standards. CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9]. ACTION: Provide a password that conforms to the Oracle recommended standards. Enter password for 'root' user: [WARNING] [INS-13013] Target environment does not meet some mandatory requirements. CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/11.2.0/oraInventory/logs/GridSetupActions2022-07-31_00-05-42AM/gridSetupActions2022-07-31_00-05-42AM.log ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/11.2.0/oraInventory/logs/GridSetupActions2022-07-31_00-05-42AM/gridSetupActions2022-07-31_00-05-42AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually. The response file for this session can be found at: /u01/app/grid/install/response/grid_2022-07-31_00-05-42AM.rsp You can find the log of this install session at: /u01/app/11.2.0/oraInventory/logs/UpdateNodeList2022-07-31_00-05-42AM.log Setup failed. [WARNING] [INS-43080] Some of the configuration assistants failed, were cancelled or skipped. ACTION: Refer to the logs or contact Oracle Support Services.
这次ocr配置完成,手动mount磁盘组
SQL> alter diskgroup all mount; alter diskgroup all mount * ERROR at line 1: ORA-15032: not all alterations performed ORA-15017: diskgroup "OCR" cannot be mounted ORA-15013: diskgroup "OCR" is already mounted SQL> exit Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 [+ASM2]@ht02[/home/grid]$asmcmd lsdg State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 512 4096 1048576 8000 7902 0 7902 0 N ARCHDG/ MOUNTED EXTERN N 512 512 4096 8388608 15360 6488 0 6488 0 N DATADG/ MOUNTED NORMAL N 512 512 4096 4194304 15360 14444 5120 4662 0 Y OCR/
把数据库添加到集群中
[ods1]@ht01[/home/oracle]$srvctl add database -d ods -o $ORACLE_HOME [ods1]@ht01[/home/oracle]$srvctl add instance -d ods -i ods1 -n ht01 [ods1]@ht01[/home/oracle]$srvctl add instance -d ods -i ods2 -n ht02 [ods1]@ht01[/home/oracle]$srvctl modify database -d ods -spfile +DATADG/ODS/PARAMETERFILE/spfile.273.1102771089 -pwfile +DATADG/ODS/PASSWORD/pwdods.300.1102770637 [ods1]@ht01[/home/oracle]$srvctl config database -d ods Database unique name: ods Database name: Oracle home: /u01/app/oracle/product/19c/dbhome_1 Oracle user: oracle Spfile: +DATADG/ODS/PARAMETERFILE/spfile.273.1102771089 Password file: +DATADG/ODS/PASSWORD/pwdods.300.1102770637 Domain: Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: Disk Groups: Mount point paths: Services: Type: RAC Start concurrency: Stop concurrency: OSDBA group: dba OSOPER group: oper Database instances: ods1,ods2 Configured nodes: ht01,ht02 CSS critical: no CPU count: 0 Memory target: 0 Maximum memory: 0 Default network number for database services: Database is administrator managed
srvctl 启动数据库并查看进群资源状态
[ods1]@ht01[/home/oracle]$srvctl start database -d ods [+ASM2]@ht02[/home/grid]$crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE ht01 STABLE ONLINE ONLINE ht02 STABLE ora.chad ONLINE ONLINE ht01 STABLE ONLINE ONLINE ht02 STABLE ora.net1.network ONLINE ONLINE ht01 STABLE ONLINE ONLINE ht02 STABLE ora.ons ONLINE ONLINE ht01 STABLE ONLINE ONLINE ht02 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.ARCHDG.dg(ora.asmgroup) 1 ONLINE ONLINE ht01 STABLE 2 ONLINE ONLINE ht02 STABLE 3 OFFLINE OFFLINE STABLE ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup) 1 ONLINE ONLINE ht01 STABLE 2 ONLINE ONLINE ht02 STABLE 3 OFFLINE OFFLINE STABLE ora.DATADG.dg(ora.asmgroup) 1 ONLINE ONLINE ht01 STABLE 2 ONLINE ONLINE ht02 STABLE 3 ONLINE OFFLINE STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE ht02 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE ht01 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE ht01 STABLE ora.OCR.dg(ora.asmgroup) 1 ONLINE ONLINE ht01 STABLE 2 ONLINE ONLINE ht02 STABLE 3 OFFLINE OFFLINE STABLE ora.asm(ora.asmgroup) 1 ONLINE ONLINE ht01 Started,STABLE 2 ONLINE ONLINE ht02 Started,STABLE 3 OFFLINE OFFLINE STABLE ora.asmnet1.asmnetwork(ora.asmgroup) 1 ONLINE ONLINE ht01 STABLE 2 ONLINE ONLINE ht02 STABLE 3 OFFLINE OFFLINE STABLE ora.cvu 1 ONLINE ONLINE ht01 STABLE ora.ht01.vip 1 ONLINE ONLINE ht01 STABLE ora.ht02.vip 1 ONLINE ONLINE ht02 STABLE ora.ods.db 1 ONLINE ONLINE ht01 Open,HOME=/u01/app/o racle/product/19c/db home_1,STABLE 2 ONLINE ONLINE ht02 Open,HOME=/u01/app/o racle/product/19c/db home_1,STABLE ora.qosmserver 1 ONLINE ONLINE ht01 STABLE ora.scan1.vip 1 ONLINE ONLINE ht02 STABLE ora.scan2.vip 1 ONLINE ONLINE ht01 STABLE ora.scan3.vip 1 ONLINE ONLINE ht01 STABLE --------------------------------------------------------------------------------
数据库正常启动,集群资源online,重建ocr完毕。
重建ocr 参考mos How to DeconfigureReconfigure(Rebuild OCR) or Deinstall Grid Infrastructure (Doc ID 1377349.1)