【案例】ora600

4节点RAC：版本oracle11.2.0.4

22:20——23：40发生ora600

alert日志：

Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_j000_28639362.trc (incident=11489377):
ORA-00600: internal error code, arguments: [kcbzwfcro_2], [4294967295], [13], [0], [0], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_11489377/orcl2_j000_28639362_i11489377.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

trace信息：

*** 2017-08-10 23:08:49.181
DDE: Problem Key 'ORA 600 [kcbzwfcro_2]' was flood controlled (0x2) (incident: 11489382)
ORA-00600: internal error code, arguments: [kcbzwfcro_2], [4294967295], [13], [0], [0], [], [], [], [], [], [], []
Potentially stale force-CR-override buffer found before OBJD MISMATCH check.
This issue should be investigated by both cache fusion and space layer.
BH (0x7000104df789598) file#: 188 rdba: 0x2f000002 (188/2) class: 13 ba: 0x7000104daeba000
set: 139 pool: 3 bsz: 8192 bsi: 0 sflg: 1 pwc: 0,25
dbwrid: 2 obj: -1 objn: -1 tsn: 13 afn: 188 hint: f
hash: [0x700010ea6859cc0,0x700010ea6859cc0] lru: [0x700010a2f5a7e68,0x7000104cf7bfa48]
ckptq: [NULL] fileq: [NULL] objq: [0x700010867a96070,0x700010d138ebf90] objaq: [0x700010867a96080,0x700010d138ebf80]
st: SCURRENT md: NULL fpin: 'ktfbwh0f: ktfbhcf' tch: 230 le: 0x7000106b7bba180
flags: remote_transfered force_cr_override
LRBA: [0x0.0.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [65535]
GLOBAL CACHE ELEMENT DUMP (address: 0x7000106b7bba180):
id1: 0x2 id2: 0xbc pkey: INVALID block: (188/2)
lock: S rls: 0x0 acq: 0x0 latch: 9
flags: 0x20 fair: 0 recovery: 0 fpin: 'ktfbwh0f: ktfbhcf'
bscn: 0x0.0 bctx: 0x0 write: 0 scan: 0x0 
lcp: 0x0 lnk: [NULL] lch: [0x7000104df7896d0,0x7000104df7896d0]
seq: 134 hist: 227 42 4 37:1 145:0 118 227 160 227 160 227 42 4 37:1
LIST OF BUFFERS LINKED TO THIS GLOBAL CACHE ELEMENT:
flg: 0x08000000 sflg: 0x2000 state: SCURRENT tsn: 13 tsh: 230
addr: 0x7000104df789598 obj: INVALID cls: BITMAP HEAD bscn: 0x0.0
buffer tsn: 13 rdba: 0x2f000002 (188/2)
scn: 0x0000.00000000 seq: 0x01 flg: 0x05 tail: 0x00000001
frmt: 0x02 chkval: 0x2ea4 type: 0x00=unknown
Hex dump of block: st=0, typ_found=0
Dump of memory from 0x07000104DAEBA000 to 0x07000104DAEBC000
7000104DAEBA000 00A20000 2F000002 00000000 00000105 [..../...........]
7000104DAEBA010 2EA40000 00000000 00000000 00000000 [................]
7000104DAEBA020 00000000 00000000 00000000 00000000 [................]
Repeat 508 times
7000104DAEBBFF0 00000000 00000000 00000000 00000001 [................]
Dump of memory from 0x07000104DAEBA014 to 0x07000104DAEBBFFC
7000104DAEBA010 00000000 00000000 00000000 [............]
7000104DAEBA020 00000000 00000000 00000000 00000000 [................]
Repeat 508 times
7000104DAEBBFF0 00000000 00000000 00000000 [............] 
GLOBAL CACHE ELEMENT DUMP (address: 0x7000106b7bba180):
id1: 0x2 id2: 0xbc pkey: INVALID block: (188/2)
lock: S rls: 0x0 acq: 0x0 latch: 9
flags: 0x20 fair: 0 recovery: 0 fpin: 'ktfbwh0f: ktfbhcf'
bscn: 0x0.0 bctx: 0x0 write: 0 scan: 0x0 
lcp: 0x0 lnk: [NULL] lch: [0x7000104df7896d0,0x7000104df7896d0]
seq: 134 hist: 227 42 4 37:1 145:0 118 227 160 227 160 227 42 4 37:1
LIST OF BUFFERS LINKED TO THIS GLOBAL CACHE ELEMENT:
flg: 0x08000000 sflg: 0x2000 state: SCURRENT tsn: 13 tsh: 230
addr: 0x7000104df789598 obj: INVALID cls: BITMAP HEAD bscn: 0x0.0
GCS SHADOW 0x7000106b7bba1f8,8 resp[0x700010e12d50690,0x2.bc] pkey 4294967295.0
grant 1 cvt 0 mdrole 0x1 st 0x100 lst 0x40 GRANTQ rl LOCAL
master 2 owner 2 sid 1 remote[0x0,0] hist 0x261e7cf85101930f
history 0xf.0x26.0x6.0x8.0x5.0x1f.0x1f.0xf.0x26.0x0.
cflag 0x0 sender 0 flags 0x0 replay# 0 abast 0x0.x0.1 dbmap 0x0
disk: 0x0000.00000000 write request: 0x0000.00000000
pi scn: 0x0000.00000000 sq[0x700010d1c713360,0x700010e12d506d0]
msgseq 0x0 updseq 0x0 reqids[17,0,0] infop 0x0 lockseq x2e40
GCS SHADOW END
GCS RESOURCE 0x700010e12d50690 hashq [0x700010e36b3a578,0x700010e1e79ba60] name[0x2.bc] pkey 4294967295.0
grant 0x7000106b7bba1f8 cvt 0x0 send 0x0@1,0 write 0x0,0@65536
flag 0x0 mdrole 0x1 mode 1 scan 0.0 role LOCAL
disk: 0x0000.00000000 write: 0x0000.00000000 cnt 0x0 hist 0x0
xid 0x0000.000.00000000 sid 1 pkwait 0s rmacks 0
refpcnt 0 weak: 0x0000.00000000
pkey 4294967295.0
hv 60 [stat 0x0, 2->2, wm 32768, RMno 0, reminc 8, dom 0]
kjga st 0x4, step 0.0.0, cinc 8, rmno 0, flags 0x0
lb 0, hb 0, myb 81920, drmb 81920, apifrz 0
GCS SHADOW 0x7000106b7bba1f8,8 resp[0x700010e12d50690,0x2.bc] pkey 4294967295.0
grant 1 cvt 0 mdrole 0x1 st 0x100 lst 0x40 GRANTQ rl LOCAL
master 2 owner 2 sid 1 remote[0x0,0] hist 0x261e7cf85101930f
history 0xf.0x26.0x6.0x8.0x5.0x1f.0x1f.0xf.0x26.0x0.
cflag 0x0 sender 0 flags 0x0 replay# 0 abast 0x0.x0.1 dbmap 0x0
disk: 0x0000.00000000 write request: 0x0000.00000000
pi scn: 0x0000.00000000 sq[0x700010d1c713360,0x700010e12d506d0]
msgseq 0x0 updseq 0x0 reqids[17,0,0] infop 0x0 lockseq x2e40
GCS SHADOW END
GCS SHADOW 0x700010d1c713360,1548 resp[0x700010e12d50690,0x2.bc] pkey 4294967295.0
grant 1 cvt 0 mdrole 0x1 st 0x100 lst 0x40 GRANTQ rl LOCAL
master 2 owner 1 sid 1 remote[0x70001071fc1b7f8,979] hist 0x18145cf871810439
history 0x39.0x8.0x4.0xc.0x7.0x1f.0x17.0xa.0x18.0x0.
cflag 0x0 sender 0 flags 0x0 replay# 0 abast 0x0.x0.1 dbmap 0x0
disk: 0x0000.00000000 write request: 0x0000.00000000
pi scn: 0x0000.00000000 sq[0x700010e12d506d0,0x7000106b7bba1f8]
msgseq 0x1 updseq 0x0 reqids[1089,0,0] infop 0x0 lockseq xbbdf
GCS SHADOW END
GCS RESOURCE END
2017-08-10 23:08:49.183968 : kjbmbassert [0x2.bc]

参考资源：http://m.blog.itpub.net/26764973/viewspace-1975149/

Verify if next workarounds can be used for your situation:

• Flush buffer cache OR to restart the instance
  
  This is because the error will continue whilst the stale buffer is in cache. We can try flushing the buffer cache to invalidate this.
  If that doesn't help the instance will need to be restarted to clear the cache.
  
  - OR -
  
  
• Set "_gc_override_force_cr=false" to avoid this particular issue
  
  There should not be any noticeable performance impact.

If the issue still persists even after testing above options, then we possibly may need to file a new bug report.

看到bug Bug 19435745

 

本项目信息：

SQL> SELECT x.ksppinm NAME, y.ksppstvl VALUE, x.ksppdesc describ
FROM SYS.x$ksppi x, SYS.x$ksppcv y
WHERE x.inst_id = USERENV ('Instance')
AND y.inst_id = USERENV ('Instance')
AND x.indx = y.indx
AND x.ksppinm LIKE '%&par%'; 6 
Enter value for par: _gc_override_force_cr
old 6: AND x.ksppinm LIKE '%&par%'
new 6: AND x.ksppinm LIKE '%_gc_override_force_cr%'

NAME                                        VALUE                       DESCRIB
------------------------------          ---------- ----------------------------------------
_gc_override_force_cr               TRUE                      if TRUE, try to override force-cr requests