代码改变世界

12C RAC 故障分析 - ORA-17503: ksfdopn:2 Failed to open file +DATA/EIC1/PASSWORD/pwdeic1.256.957086685

2020-11-23 20:46  狂澜与玉昆0950  阅读(2319)  评论(0编辑  收藏  举报

一、故障描述

Oracle 12C PDBS(PDB NAME = WRYPC) 在note1节点上是MOUNTED状态,在note2节点上是READ WRITE状态。note1节点上启动该PDB长时间无反应hang住,并且伴有日志报错。

//note1 
SQL> select DBID,NAME,LOG_MODE,OPEN_MODE from v$database; 

      DBID NAME      LOG_MODE      OPEN_MODE
---------- --------- ------------ --------------------
2140168445 EIC1      ARCHIVELOG   READ WRITE

SQL> select instance_name,status from v$instance;     

INSTANCE_NAME     STATUS
---------------- ------------
eic12         OPEN

SQL> select DBID,NAME,LOG_MODE,OPEN_MODE from v$database; 

      DBID NAME      LOG_MODE      OPEN_MODE
---------- --------- ------------ --------------------
2140168445 EIC1      ARCHIVELOG   READ WRITE

SQL> select con_id,dbid,guid,name,open_mode from v$pdbs;

    CON_ID     DBID GUID                             NAME             OPEN_MODE
---------- ---------- -------------------------------- --------------------------------- ----------
     2 3257667166 5B3C58220E311678E0532CF2640A4165 PDB$SEED         READ ONLY
     3 2473517642 5B3C7CE162C953F6E0532CF2640A910E EIC1PDB            READ WRITE
     4 1620899158 6A1F75EB99FE3131E0532DF2640A95C0 PDB_NAME         READ WRITE
     5  493040432 6A1F75EB9A003131E0532DF2640A95C0 ERWUPU            MOUNTED
     6 2865274021 6A224E09130541B9E0532DF2640AAE5F EWP2                MOUNTED
     7 2198790855 6B3463863B9C1D6BE0532DF2640A61C5 ODS                READ WRITE
     8 2689821097 703D1085A55C3376E0532DF2640A8A3E EWPSHIDIANSHENG    MOUNTED
     9 1167482729 703DEDEFF33FDE66E0532DF2640A8E9F EWPSHIDIANGJ        MOUNTED
    10 1008500739 711B1C42A2F34FA9E0532DF2640AC62B ERWUPUMAP        MOUNTED
    11  148782956 A401C78AC106DC34E0532DF2640A8AA5 WRYPC            READ WRITE
    12 1161538067 75BEA3722224E4DAE0532CF2640AD0F6 ERWUPUMAP_XJ        MOUNTED
    13 1728756282 75BD0918B56DB368E0532CF2640A9170 WRYPCNEI2018W    MOUNTED
    14 1579638091 75BEE5B84C9E23A3E0532CF2640A7E2E ERWUPUMAP_XZ        MOUNTED
    15 2065390671 75BF0ECA028E34A4E0532CF2640AFFC3 ERWUPUMAP_BT        MOUNTED
    16  493305058 75C309E9051CCE0CE0532CF2640ABF80 WRYPCNEI2018Z    MOUNTED
    17  222405979 75CD5BD04A3846F9E0532CF2640AAE77 WRYPCWAI54W        MOUNTED
    18  855842903 75CDC85DE82DB0CCE0532CF2640ABD02 WRYPCWAI54Z        MOUNTED
    19  727403284 75CE11270B33EE13E0532CF2640ADC8A WRYPCWAI65W        MOUNTED
    20 1858334523 75CE387EA16712E1E0532CF2640AE409 WRYPCNEI65Z        MOUNTED
    21 2081645259 75CE6B8F25483504E0532CF2640AC72A WRYPCNEI66Z        MOUNTED
    22 3702096573 78057D27A412F5BEE0532CF2640A3B24 HLWZWFWPT        READ WRITE
    23 4268531149 7805BC8E63A35B78E0532CF2640A3FFA HLWZWFWPTER        READ WRITE
    24  331720291 7841DABFA10E7C55E0532CF2640AC927 EGOVA            READ WRITE
    25  744750214 78CE617BF1D4F540E0532CF2640AF51E HJXXYZT            READ WRITE
    26 1668454202 78421DAF6EF3D301E0532CF2640AB57D STHJXXYZT        READ WRITE
    27 3378782880 796DCEB39C43282FE0532CF2640A26C7 IWHEREORCL        READ WRITE
    28 3623836438 7C2FEDCF7B4291E1E0532CF2640A4072 ODSTEST            READ WRITE
    29 3912941240 7CE2F2FE4E47514DE0532CF2640A5754 ESERVICE         READ WRITE
    30 3016044583 A6874758312BDF72E0532CF2640A2F9F DECWRYPC         READ WRITE

29 rows selected.

SQL> 



//note2
SQL> set linesize 200  
SQL> select DBID,NAME,LOG_MODE,OPEN_MODE from v$database; 

      DBID NAME      LOG_MODE      OPEN_MODE
---------- --------- ------------ --------------------
2140168445 EIC1      ARCHIVELOG   READ WRITE

SQL> select con_id,dbid,guid,name,open_mode from v$pdbs;

    CON_ID     DBID GUID                   NAME                          OPEN_MODE
---------- ---------- -------------------------------- -------------------------------------------------- --------------------
     2 3257667166 5B3C58220E311678E0532CF2640A4165 PDB$SEED                       READ ONLY
     3 2473517642 5B3C7CE162C953F6E0532CF2640A910E EIC1PDB                          READ WRITE
     4 1620899158 6A1F75EB99FE3131E0532DF2640A95C0 PDB_NAME                       READ WRITE
     5  493040432 6A1F75EB9A003131E0532DF2640A95C0 ERWUPU                          MOUNTED
     6 2865274021 6A224E09130541B9E0532DF2640AAE5F EWP2                          MOUNTED
     7 2198790855 6B3463863B9C1D6BE0532DF2640A61C5 ODS                          READ WRITE
     8 2689821097 703D1085A55C3376E0532DF2640A8A3E EWPSHIDIANSHENG                      MOUNTED
     9 1167482729 703DEDEFF33FDE66E0532DF2640A8E9F EWPSHIDIANGJ                      MOUNTED
    10 1008500739 711B1C42A2F34FA9E0532DF2640AC62B ERWUPUMAP                      MOUNTED
    11  148782956 A401C78AC106DC34E0532DF2640A8AA5 WRYPC                          MOUNTED
    12 1161538067 75BEA3722224E4DAE0532CF2640AD0F6 ERWUPUMAP_XJ                      MOUNTED

    CON_ID     DBID GUID                   NAME                          OPEN_MODE
---------- ---------- -------------------------------- -------------------------------------------------- --------------------
    13 1728756282 75BD0918B56DB368E0532CF2640A9170 WRYPCNEI2018W                      MOUNTED
    14 1579638091 75BEE5B84C9E23A3E0532CF2640A7E2E ERWUPUMAP_XZ                      MOUNTED
    15 2065390671 75BF0ECA028E34A4E0532CF2640AFFC3 ERWUPUMAP_BT                      MOUNTED
    16  493305058 75C309E9051CCE0CE0532CF2640ABF80 WRYPCNEI2018Z                      MOUNTED
    17  222405979 75CD5BD04A3846F9E0532CF2640AAE77 WRYPCWAI54W                      MOUNTED
    18  855842903 75CDC85DE82DB0CCE0532CF2640ABD02 WRYPCWAI54Z                      MOUNTED
    19  727403284 75CE11270B33EE13E0532CF2640ADC8A WRYPCWAI65W                      MOUNTED
    20 1858334523 75CE387EA16712E1E0532CF2640AE409 WRYPCNEI65Z                      MOUNTED
    21 2081645259 75CE6B8F25483504E0532CF2640AC72A WRYPCNEI66Z                      MOUNTED
    22 3702096573 78057D27A412F5BEE0532CF2640A3B24 HLWZWFWPT                      READ WRITE
    23 4268531149 7805BC8E63A35B78E0532CF2640A3FFA HLWZWFWPTER                      READ WRITE

    CON_ID     DBID GUID                   NAME                          OPEN_MODE
---------- ---------- -------------------------------- -------------------------------------------------- --------------------
    24  331720291 7841DABFA10E7C55E0532CF2640AC927 EGOVA                          READ WRITE
    25  744750214 78CE617BF1D4F540E0532CF2640AF51E HJXXYZT                          READ WRITE
    26 1668454202 78421DAF6EF3D301E0532CF2640AB57D STHJXXYZT                      READ WRITE
    27 3378782880 796DCEB39C43282FE0532CF2640A26C7 IWHEREORCL                      READ WRITE
    28 3623836438 7C2FEDCF7B4291E1E0532CF2640A4072 ODSTEST                          READ WRITE
    29 3912941240 7CE2F2FE4E47514DE0532CF2640A5754 ESERVICE                       READ WRITE
    30 3016044583 A6874758312BDF72E0532CF2640A2F9F DECWRYPC                       READ WRITE

29 rows selected.

SQL> 

SQL> alter pluggable database WRYPC open;
^C

alter pluggable database WRYPC open
*
ERROR at line 1:
ORA-12152: TNS:unable to send break message

// alert_eic11.log
--日志报错
2020-11-23T12:44:29.311665+08:00
alter pluggable database WRYPC open
2020-11-23T12:44:29.490214+08:00
WRYPC(11):Recovering pdb: pdbid=11
WRYPC(11):JIT: pid 192727 requesting stop
2020-11-23T12:46:37.858454+08:00
DBRM (ospid: 58076) waits for event 'latch: shared pool' for 18 secs.
DBRM (ospid: 58076) waits for latch 'shared pool' for 18 secs.
2020-11-23T12:46:49.379038+08:00
Errors in file /u01/app/oracle/diag/rdbms/eic1/eic11/trace/eic11_lmhb_58243.trc  (incident=2049040) (PDBNAME=C
DB$ROOT):
ORA-29771: process USER (OSID 192727) blocks DBRM (OSID 58076) for more than 70 seconds
Incident details in: /u01/app/oracle/diag/rdbms/eic1/eic11/incident/incdir_2049040/eic11_lmhb_58243_i2049040.t
rc
2020-11-23T12:46:58.068303+08:00
USER (ospid: 192727) is blocking DBRM (ospid: 58076) in a wait
LMHB (ospid: 58243) kills USER (ospid: 192727).
Please check LMHB trace file for more detail.
2020-11-23T12:47:01.204284+08:00

// alert_eic11.log
--多次出现的错误
2020-11-23T05:49:10.563954+08:00
Errors in file /u01/app/oracle/diag/rdbms/eic1/eic11/trace/eic11_ora_186547.trc:
ORA-17503: ksfdopn:2 Failed to open file +DATA/EIC1/PASSWORD/pwdeic1.256.957086685
ORA-27300: OS system dependent operation:open failed with status: 13
ORA-27301: OS failure message: Permission denied
ORA-27302: failure occurred at: sskgmsmr_7
2020-11-23T05:49:10.608316+08:00
Errors in file /u01/app/oracle/diag/rdbms/eic1/eic11/trace/eic11_ora_186547.trc:
ORA-17503: ksfdopn:2 Failed to open file +DATA/EIC1/PASSWORD/pwdeic1.256.957086685
ORA-27300: OS system dependent operation:open failed with status: 13
ORA-27301: OS failure message: Permission denied
ORA-27302: failure occurred at: sskgmsmr_7
故障代码

软件配置

Red Hat Enterprise Linux Server release 6.9 (Santiago)
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
10.100.242.44    eicdb1
10.100.242.45    eicdb2
10.100.242.61    eicdb1-vip
10.100.242.62    eicdb2-vip
10.10.10.44     eicdb1-pri
10.10.10.45     eicdb2-pri
10.100.242.63    eicdb-scan
configuration

二、定位问题

1.关键性报错(错误)

2020-11-23T05:49:10.563954+08:00
Errors in file /u01/app/oracle/diag/rdbms/eic1/eic11/trace/eic11_ora_186547.trc:
ORA-17503: ksfdopn:2 Failed to open file +DATA/EIC1/PASSWORD/pwdeic1.256.957086685
ORA-27300: OS system dependent operation:open failed with status: 13
ORA-27301: OS failure message: Permission denied
ORA-27302: failure occurred at: sskgmsmr_7

2.从trace文件中寻找更有价值的信息

Trace file /u01/app/oracle/diag/rdbms/eic1/eic11/trace/eic11_ora_186547.trc
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Build label:    RDBMS_12.2.0.1.0_LINUX.X64_170125
ORACLE_HOME:    /u01/app/oracle/product/12.2.0/db_1
System name:    Linux
Node name:    eicdb1
Release:    4.1.12-61.1.28.el6uek.x86_64
Version:    #2 SMP Thu Feb 23 20:03:53 PST 2017
Machine:    x86_64
Instance name: eic11
Redo thread mounted by this instance: 1
Oracle process number: 193
Unix process pid: 186547, image: oracle@eicdb1 (TNS V1-V3)

*** 2020-11-23T05:49:10.528101+08:00 (CDB$ROOT(1))
*** SESSION ID:(44.52374) 2020-11-23T05:49:10.528221+08:00
*** CLIENT ID:() 2020-11-23T05:49:10.528244+08:00
*** SERVICE NAME:(SYS$USERS) 2020-11-23T05:49:10.528264+08:00
*** MODULE NAME:(eefprocess@eicdb1 (TNS V1-V3)) 2020-11-23T05:49:10.528285+08:00
*** ACTION NAME:() 2020-11-23T05:49:10.528311+08:00
*** CLIENT DRIVER:() 2020-11-23T05:49:10.528328+08:00
*** CONTAINER ID:(1) 2020-11-23T05:49:10.528348+08:00
 
SUCCESS: retrieved DB password file location:+DATA/EIC1/PASSWORD/pwdeic1.256.957086685
2020-11-23 05:49:10.546 : [    GPNP][    GPNP]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:496] Result: (7) CLSGPNP_IO. (:GPNP01002:)Fatal: Cannot open neither PEER nor PRDR GPnP wallet. No more wallets to verify GPnP configuration (profile). Check GPnP configuration: wallet(s) either missing or do not have access privileges. statres: -8, Wallet home: /u01/12.2.0/grid/gpnp/eicdb1/wallets/ 
2020-11-23 05:49:10.546 : [    GPNP][    GPNP]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:517] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP01003:)Failed to find peer wallet dir in either OCR_CRS_HOME, or ORACLE_HOME locations.
2020-11-23 05:49:10.546 : [    GPNP][    GPNP]clsgpnp_InitCKProviders: [at clsgpnp0.c:5225] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP00254:)Error initializing gpnp security key provider: file wallet (LSKP-FSW).
2020-11-23 05:49:10.551 : [    GPNP][    GPNP]clsgpnp_getCK: [at clsgpnp0.c:2879] (:GPNP00233:)Fatal error: failed to get local gpnp security keys (wallet). Gpnp profiles cannot be verified. ***LOCAL GPNP SETUP INVALID***.>
2020-11-23 05:49:10.551 : [    GPNP][    GPNP]clsgpnp_InitIdSetCtxCK: [at clsgpnp0.c:5612] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP00267:)Cannot get stored certkey for id=1
2020-11-23 05:49:10.551 : [    GPNP][    GPNP]clsgpnp_Init: [at clsgpnp0.c:1063] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP00211:)Error getting certkeys. 
2020-11-23 05:49:10.551 : [    GPNP][    GPNP]clsgpnp_Init init failed. Error: CLSGPNP_ERR (1) .
2020-11-23 05:49:10.553 : [ default][ default](:CLSUG0001:)Failed to initialize clsgpnp context grv CLSGPNP_ERR (1). Return GPNP [12]
2020-11-23 05:49:10.553 : [    GPNP][    GPNP][at clsgpnp_dbms.c:263] (:GPNPD01786:)Failed to get gpnp global profile. Returned: GPNP (12)
ORA-17503: ksfdopn:2 Failed to open file +DATA/EIC1/PASSWORD/pwdeic1.256.957086685
ORA-27300: OS system dependent operation:open failed with status: 13
ORA-27301: OS failure message: Permission denied
ORA-27302: failure occurred at: sskgmsmr_7
SUCCESS: retrieved DB password file location:+DATA/EIC1/PASSWORD/pwdeic1.256.957086685
2020-11-23 05:49:10.595 : [    GPNP][    GPNP]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:496] Result: (7) CLSGPNP_IO. (:GPNP01002:)Fatal: Cannot open neither PEER nor PRDR GPnP wallet. No more wallets to verify GPnP configuration (profile). Check GPnP configuration: wallet(s) either missing or do not have access privileges. statres: -8, Wallet home: /u01/12.2.0/grid/gpnp/eicdb1/wallets/ 
2020-11-23 05:49:10.595 : [    GPNP][    GPNP]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:517] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP01003:)Failed to find peer wallet dir in either OCR_CRS_HOME, or ORACLE_HOME locations.
2020-11-23 05:49:10.595 : [    GPNP][    GPNP]clsgpnp_InitCKProviders: [at clsgpnp0.c:5225] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP00254:)Error initializing gpnp security key provider: file wallet (LSKP-FSW).
2020-11-23 05:49:10.598 : [    GPNP][    GPNP]clsgpnp_getCK: [at clsgpnp0.c:2879] (:GPNP00233:)Fatal error: failed to get local gpnp security keys (wallet). Gpnp profiles cannot be verified. ***LOCAL GPNP SETUP INVALID***.>
2020-11-23 05:49:10.598 : [    GPNP][    GPNP]clsgpnp_InitIdSetCtxCK: [at clsgpnp0.c:5612] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP00267:)Cannot get stored certkey for id=1
2020-11-23 05:49:10.598 : [    GPNP][    GPNP]clsgpnp_Init: [at clsgpnp0.c:1063] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP00211:)Error getting certkeys. 
2020-11-23 05:49:10.598 : [    GPNP][    GPNP]clsgpnp_Init init failed. Error: CLSGPNP_ERR (1) .
2020-11-23 05:49:10.600 : [    GPNP][    GPNP](:CLSUG0001:)Failed to initialize clsgpnp context grv CLSGPNP_ERR (1). Return GPNP [12]
2020-11-23 05:49:10.600 : [    GPNP][    GPNP][at clsgpnp_dbms.c:263] (:GPNPD01786:)Failed to get gpnp global profile. Returned: GPNP (12)
ORA-17503: ksfdopn:2 Failed to open file +DATA/EIC1/PASSWORD/pwdeic1.256.957086685
ORA-27300: OS system dependent operation:open failed with status: 13
ORA-27301: OS failure message: Permission denied
ORA-27302: failure occurred at: sskgmsmr_7
trace file

trace文件【eic11_ora_186547.trc】初步定位问题点:

2020-11-23 05:49:10.546 : [    GPNP][    GPNP]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:496] Result: (7) CLSGPNP_IO. (:GPNP01002:)Fatal: Cannot open neither PEER nor PRDR GPnP wallet. No more wallets to verify GPnP configuration (profile). Check GPnP configuration: wallet(s) either missing or do not have access privileges. statres: -8, Wallet home: /u01/12.2.0/grid/gpnp/eicdb1/wallets/ 

检查Wallet home文件:

//note1
[oracle@eicdb1 ~]$ ls -ltr /u01/12.2.0/grid/gpnp/eicdb1/wallets/**/
ls: cannot open directory /u01/12.2.0/grid/gpnp/eicdb1/wallets/root: Permission denied
/u01/12.2.0/grid/gpnp/eicdb1/wallets/prdr:
total 4
-rwxr-x--- 1 grid oinstall 1725 Oct 11  2017 cwallet.sso
-rwxrwx--- 1 grid oinstall    0 Oct 11  2017 cwallet.sso.lck
ls: cannot open directory /u01/12.2.0/grid/gpnp/eicdb1/wallets/peer: Permission denied
ls: cannot open directory /u01/12.2.0/grid/gpnp/eicdb1/wallets/pa: Permission denied
[oracle@eicdb1 ~]$ exit
logout
[root@eicdb1 ~]# su - grid
[grid@eicdb1 ~]$ ls -ltr /u01/12.2.0/grid/gpnp/eicdb1/wallets/**/
/u01/12.2.0/grid/gpnp/eicdb1/wallets/root:
total 4
-rwx------ 1 grid oinstall 2416 Oct 11  2017 ewallet.p12

/u01/12.2.0/grid/gpnp/eicdb1/wallets/prdr:
total 4
-rwxr-x--- 1 grid oinstall 1725 Oct 11  2017 cwallet.sso
-rwxrwx--- 1 grid oinstall    0 Oct 11  2017 cwallet.sso.lck

/u01/12.2.0/grid/gpnp/eicdb1/wallets/peer:
total 4
-rwx------ 1 grid oinstall 2941 Oct 11  2017 cwallet.sso
-rwx------ 1 grid oinstall    0 Oct 11  2017 cwallet.sso.lck

/u01/12.2.0/grid/gpnp/eicdb1/wallets/pa:
total 4
-rwx------ 1 grid oinstall 2933 Oct 11  2017 cwallet.sso
-rwx------ 1 grid oinstall    0 Oct 11  2017 cwallet.sso.lck
[grid@eicdb1 ~]$ 

//note2
[oracle@eicdb2 dbs]$ ls -ltr /u01/12.2.0/grid/gpnp/eicdb1/wallets/*  */
ls: cannot access /u01/12.2.0/grid/gpnp/eicdb1/wallets/*: No such file or directory  */

3.发现

note1节点oracle用户Wallet home下没有文件夹peer的权限,这一点与上面定位的问题点贴合。note2节点oracle用户Wallet home下没有文件。

三、故障修复

//修改Wallet home文件夹权限
[grid@eicdb1 ~]$ cd /u01/12.2.0/grid/gpnp/eicdb1/wallets
[grid@eicdb1 wallets]$ ls -ltr
total 16
drwx-----T 2 grid oinstall 4096 Nov 23 16:03 root
drwx-----T 2 grid oinstall 4096 Nov 23 16:03 peer
drwx-----T 2 grid oinstall 4096 Nov 23 16:03 pa
drwxr-x--T 2 grid oinstall 4096 Nov 23 16:03 prdr
[grid@eicdb1 wallets]$ cd ..
[grid@eicdb1 eicdb1]$ cp -r wallets/ wallets.bak
[grid@eicdb1 eicdb1]$ chmod -R 775 wallets
[grid@eicdb1 eicdb1]$ ls -ltr
total 12
drwxr-x--- 3 grid oinstall 4096 Oct 11  2017 profiles
drwxrwxr-x 6 grid oinstall 4096 Oct 11  2017 wallets
drwxr-x--T 6 grid oinstall 4096 Nov 23 16:03 wallets.bak
[grid@eicdb1 eicdb1]$ cd wallets
[grid@eicdb1 wallets]$ ls -ltr
total 16
drwxrwxr-x 2 grid oinstall 4096 Oct 11  2017 root
drwxrwxr-x 2 grid oinstall 4096 Oct 11  2017 prdr
drwxrwxr-x 2 grid oinstall 4096 Oct 11  2017 peer
drwxrwxr-x 2 grid oinstall 4096 Oct 11  2017 pa

//启动PDB
SQL> alter pluggable database WRYPC open;

Pluggable database altered.

//检查alert日志
2020-11-23T16:05:16.663872+08:00
alter pluggable database WRYPC open
2020-11-23T16:05:17.752765+08:00
WRYPC(11):Recovering pdb: pdbid=11
WRYPC(11):JIT: pid 168631 requesting stop
2020-11-23T16:06:45.136175+08:00
WRYPC(11):detach called for domid 11 (domuid: 0x8de3f6c, options: 0x10, pid: 168631)
WRYPC(11):Autotune of undo retention is turned on.
WRYPC(11):attach called for domid 11 (domuid: 0x8de3f6c, options: 0x4, pid: 168631)
WRYPC(11):queued attach broadcast request 0x5e4dd47f30
WRYPC(11):Endian type of dictionary set to little
2020-11-23T16:06:46.854466+08:00
WRYPC(11):[168631] Successfully onlined Undo Tablespace 2.
WRYPC(11):Undo initialization finished serial:0 start:3416128516 end:3416129053 diff:537 ms (0.5 seconds)
WRYPC(11):Database Characterset for WRYPC is AL32UTF8
2020-11-23T16:06:50.341552+08:00
WRYPC(11):Opening pdb with no Resource Manager plan active
Pluggable database WRYPC opened read write
Completed: alter pluggable database WRYPC open

note1节点该PDB状态:

[oracle@eicdb1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Mon Nov 23 20:31:20 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> set lines 200
SQL> show pdbs;

    CON_ID CON_NAME              OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
     2 PDB$SEED              READ ONLY  NO
     3 EIC1PDB              READ WRITE NO
     4 PDB_NAME              READ WRITE NO
     5 ERWUPU              MOUNTED
     6 EWP2               MOUNTED
     7 ODS                  READ WRITE NO
     8 EWPSHIDIANSHENG          MOUNTED
     9 EWPSHIDIANGJ           MOUNTED
    10 ERWUPUMAP              MOUNTED
    11 WRYPC              READ WRITE NO
    12 ERWUPUMAP_XJ           MOUNTED

    CON_ID CON_NAME              OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
    13 WRYPCNEI2018W          MOUNTED
    14 ERWUPUMAP_XZ           MOUNTED
    15 ERWUPUMAP_BT           MOUNTED
    16 WRYPCNEI2018Z          MOUNTED
    17 WRYPCWAI54W              MOUNTED
    18 WRYPCWAI54Z              MOUNTED
    19 WRYPCWAI65W              MOUNTED
    20 WRYPCNEI65Z              MOUNTED
    21 WRYPCNEI66Z              MOUNTED
    22 HLWZWFWPT              READ WRITE NO
    23 HLWZWFWPTER              READ WRITE NO

    CON_ID CON_NAME              OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
    24 EGOVA              READ WRITE NO
    25 HJXXYZT              READ WRITE NO
    26 STHJXXYZT              READ WRITE NO
    27 IWHEREORCL              READ WRITE NO
    28 ODSTEST              READ WRITE NO
    29 ESERVICE              READ WRITE NO
    30 DECWRYPC              READ WRITE NO
SQL> 
PDB

故障解决。