问题现象：
<CRS_HOME>/cdata目录下存在大量数字命名的文件，导致文件系统爆满
$ls -alrth /opt/oracle/product/CRS/cdata/crs -rw-r--r-- 1 root root 3.9M Oct 19 00:51 25974360 -rw-r--r-- 1 root root 3.9M Oct 19 04:51 39631111 -rw-r--r-- 1 root root 3.9M Oct 19 08:51 20660854 -rw-r--r-- 1 root root 3.9M Oct 19 12:51 35589462 -rw-r--r-- 1 root root 3.9M Oct 19 16:51 12309480 $ls /opt/oracle/product/CRS/cdata/crs |egrep '^[0-9]+' |wc -l 9692
Automatic OCR Backup Filling Up <CRS_HOME>/cdata/<clustername> Directory (Doc ID 1191067.1)
ls -l $GRID_HOME/cdata/$CLUSTERNAME -rwxr-xr-x 1 grid oinstall 6791168 Jul 16 22:45 week.ocr -rwxr-xr-x 1 grid oinstall 6791168 Jul 16 22:45 day.ocr -rwxr-xr-x 1 grid oinstall 6791168 Jul 16 22:45 backup02.ocr -rwxr-xr-x 1 grid oinstall 6791168 Jul 17 02:45 backup01.ocr -rwxr-xr-x 1 grid oinstall 6791168 Jul 17 02:45 day_.ocr -rw------- 1 root root 7114752 Jul 17 11:43 backup00.ocr -rw------- 1 root root 7114752 Jul 17 15:43 35568671.ocr -rw------- 1 root root 7114752 Jul 17 19:43 38296812.ocr.. -rw------- 1 root root 7143424 Aug 30 05:40 37208845.ocr -rw------- 1 root root 7143424 Aug 30 09:40 38455890.ocr
Due to bug 9446443, automatic OCR backups are incorrectly owned which is preventing CRSD from overwriting them.
Applies to:Oracle Database - Enterprise Edition - Version 10.2.0.1 and later
Information in this document applies to any platform.Symptoms
Automatic OCR backup rotation is not working with 11gR2 Grid Infrastructure, instead of overwriting existing backup file, CRSD keeps creating new backup files which taking more and more space:
ls -l $GRID_HOME/cdata/$CLUSTERNAME -rwxr-xr-x 1 grid oinstall 6791168 Jul 16 22:45 week.ocr -rwxr-xr-x 1 grid oinstall 6791168 Jul 16 22:45 day.ocr -rwxr-xr-x 1 grid oinstall 6791168 Jul 16 22:45 backup02.ocr -rwxr-xr-x 1 grid oinstall 6791168 Jul 17 02:45 backup01.ocr -rwxr-xr-x 1 grid oinstall 6791168 Jul 17 02:45 day_.ocr -rw------- 1 root root 7114752 Jul 17 11:43 backup00.ocr -rw------- 1 root root 7114752 Jul 17 15:43 35568671.ocr -rw------- 1 root root 7114752 Jul 17 19:43 38296812.ocr .. -rw------- 1 root root 7143424 Aug 30 05:40 37208845.ocr -rw------- 1 root root 7143424 Aug 30 09:40 38455890.ocrCause
Due to bug 9446443, automatic OCR backups are incorrectly owned which is preventing CRSD from overwriting them.
Expected ownership and permission on Linux - all 7 of them:
-rw------- 1 root root 11640832 Aug 30 08:46 backup00.ocr -rw------- 1 root root 11640832 Aug 30 04:46 backup01.ocr -rw------- 1 root root 11640832 Aug 30 00:46 backup02.ocr -rw------- 1 root root 11640832 Aug 30 00:46 day_.ocr -rw------- 1 root root 11640832 Aug 29 00:46 day.ocr -rw------- 1 root root 11640832 Aug 26 00:45 week_.ocr -rw------- 1 root root 11640832 Aug 19 00:44 week.ocrSolution
bug 9446443 is fixed in 11.2.0.2, 12.1.
It's recommended to apply patch to fix the issue, but if patch is unavailable, workaround is to change ownership and permission of all 7 automatic backup files manually. OCR should be owned by root, but depend on platform, group may or may not be root - you can check any randomly named backup file to identify what ownership and permission it should have; in example below:
-rw------- 1 root root 7143424 Aug 30 09:40 38455890.ocr
With this, please change all 7 automatic backup files to be owned by root:root with permission "-rw-------"References
NOTE:1083982.1 - Incorrect Ownership and Permission after Relinking or Patching 11gR2 Grid Infrastructure
BUG:9446443 - ROOTCRS.PL RESETS THE PERMISSIONS ON /CDATA/
BUG:7313884 - NEWLY GENERATED OCR BACKUP IN CDATA DIRECTORY CAN'T BE RENAMED
BUG:5893629 - CRSD CRASHES AFTER BEING UNABLE TO DELETE BACKUP OCR FILE
