ORACLE11gR2-RAC之OCR无备份情况下损坏恢复

情景介绍：
做OCR备份恢复实验，OCR有4份自动备份。将OCR磁盘从+DATA替换为+OCR2(/dev/raw/raw4) 完成之后使用ocrconfig -manualbackup手动备份OCR，完成之后对/dev/raw/raw4执行dd操作。关闭集群，启动集群，发现集群不能启动。

问题分析(假设不知道问题出在哪里，先分析)：
1、检查集群服务，发现CRS和CSS服务未能正常启动
crsctl check crs
2、检查CRS和CSS日志，发现OCR磁盘异常
3、恢复OCR(其实就是使用root.sh重建OCR的过程，重建之后需要重新注册相关的资源如listener/database等)
清空所有节点的cluster配置信息：root用户执行 $GRID_HOME/crs/install/rootcrs.pl
节点1
[[email protected] install]# ./rootcrs.pl
Using configuration parameter file: ./crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

节点2
[[email protected] install]# ./rootcrs.pl
Using configuration parameter file: ./crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

清除所有节点的cluster信息
节点1
[[email protected] install]# ./rootcrs.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘node1‘
CRS-2673: Attempting to stop ‘ora.mdnsd‘ on ‘node1‘
CRS-2673: Attempting to stop ‘ora.crf‘ on ‘node1‘
CRS-2673: Attempting to stop ‘ora.ctssd‘ on ‘node1‘
CRS-2673: Attempting to stop ‘ora.evmd‘ on ‘node1‘
CRS-2673: Attempting to stop ‘ora.asm‘ on ‘node1‘
CRS-2673: Attempting to stop ‘ora.drivers.acfs‘ on ‘node1‘
CRS-2677: Stop of ‘ora.evmd‘ on ‘node1‘ succeeded
CRS-2677: Stop of ‘ora.crf‘ on ‘node1‘ succeeded
CRS-2677: Stop of ‘ora.mdnsd‘ on ‘node1‘ succeeded
CRS-2677: Stop of ‘ora.ctssd‘ on ‘node1‘ succeeded
CRS-2677: Stop of ‘ora.drivers.acfs‘ on ‘node1‘ succeeded
CRS-2677: Stop of ‘ora.asm‘ on ‘node1‘ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip‘ on ‘node1‘
CRS-2677: Stop of ‘ora.cluster_interconnect.haip‘ on ‘node1‘ succeeded
CRS-2673: Attempting to stop ‘ora.cssd‘ on ‘node1‘
CRS-2677: Stop of ‘ora.cssd‘ on ‘node1‘ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd‘ on ‘node1‘
CRS-2677: Stop of ‘ora.gipcd‘ on ‘node1‘ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd‘ on ‘node1‘
CRS-2677: Stop of ‘ora.gpnpd‘ on ‘node1‘ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘node1‘ has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node

节点2
[[email protected] install]# ./rootcrs.pl -deconfig -force -lastnode
Using configuration parameter file: ./crsconfig_params
CRS-5702: Resource ‘ora.cssd‘ is already running on ‘node2‘
CRS-4000: Command Start failed, or completed with errors.
CSS startup failed with return code 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-2673: Attempting to stop ‘ora.ctssd‘ on ‘node2‘
CRS-2673: Attempting to stop ‘ora.evmd‘ on ‘node2‘
CRS-2673: Attempting to stop ‘ora.asm‘ on ‘node2‘
CRS-2677: Stop of ‘ora.evmd‘ on ‘node2‘ succeeded
CRS-2677: Stop of ‘ora.ctssd‘ on ‘node2‘ succeeded
CRS-2677: Stop of ‘ora.asm‘ on ‘node2‘ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip‘ on ‘node2‘
CRS-2677: Stop of ‘ora.cluster_interconnect.haip‘ on ‘node2‘ succeeded
CRS-2673: Attempting to stop ‘ora.cssd‘ on ‘node2‘
CRS-2677: Stop of ‘ora.cssd‘ on ‘node2‘ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor‘ on ‘node2‘
CRS-2676: Start of ‘ora.cssdmonitor‘ on ‘node2‘ succeeded
CRS-2672: Attempting to start ‘ora.cssd‘ on ‘node2‘
CRS-2672: Attempting to start ‘ora.diskmon‘ on ‘node2‘
CRS-2676: Start of ‘ora.diskmon‘ on ‘node2‘ succeeded
CRS-2676: Start of ‘ora.cssd‘ on ‘node2‘ succeeded
CRS-4611: Successful deletion of voting disk +DATA.
ASM de-configuration trace file location: /tmp/asmcadc_clean2016-10-31_02-02-22-PM.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2016-10-31_02-02-22-PM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘node2‘
CRS-2673: Attempting to stop ‘ora.ctssd‘ on ‘node2‘
CRS-2673: Attempting to stop ‘ora.asm‘ on ‘node2‘
CRS-2673: Attempting to stop ‘ora.mdnsd‘ on ‘node2‘
CRS-2677: Stop of ‘ora.mdnsd‘ on ‘node2‘ succeeded
CRS-2677: Stop of ‘ora.ctssd‘ on ‘node2‘ succeeded
CRS-2677: Stop of ‘ora.asm‘ on ‘node2‘ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip‘ on ‘node2‘
CRS-2677: Stop of ‘ora.cluster_interconnect.haip‘ on ‘node2‘ succeeded
CRS-2673: Attempting to stop ‘ora.cssd‘ on ‘node2‘
CRS-2677: Stop of ‘ora.cssd‘ on ‘node2‘ succeeded
CRS-2673: Attempting to stop ‘ora.crf‘ on ‘node2‘
CRS-2677: Stop of ‘ora.crf‘ on ‘node2‘ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd‘ on ‘node2‘
CRS-2677: Stop of ‘ora.gipcd‘ on ‘node2‘ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd‘ on ‘node2‘
CRS-2677: Stop of ‘ora.gpnpd‘ on ‘node2‘ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘node2‘ has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node

重建OCR和OLR，使用root.sh脚本完成重建，其实就是安装RAC中执行的脚本，默认位置为$GRID_HOME

节点1
[[email protected] grid]# ./root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-2672: Attempting to start ‘ora.mdnsd‘ on ‘node1‘
CRS-2676: Start of ‘ora.mdnsd‘ on ‘node1‘ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd‘ on ‘node1‘
CRS-2676: Start of ‘ora.gpnpd‘ on ‘node1‘ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor‘ on ‘node1‘
CRS-2672: Attempting to start ‘ora.gipcd‘ on ‘node1‘
CRS-2676: Start of ‘ora.cssdmonitor‘ on ‘node1‘ succeeded
CRS-2676: Start of ‘ora.gipcd‘ on ‘node1‘ succeeded
CRS-2672: Attempting to start ‘ora.cssd‘ on ‘node1‘
CRS-2672: Attempting to start ‘ora.diskmon‘ on ‘node1‘
CRS-2676: Start of ‘ora.diskmon‘ on ‘node1‘ succeeded
CRS-2676: Start of ‘ora.cssd‘ on ‘node1‘ succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root‘, privgrp ‘root‘..
Operation successful.
Successful addition of voting disk 4331dad495c14f71bfdb6d4f1a82d2f9.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced

STATE File Universal Id File Name Disk group

  1. ONLINE 4331dad495c14f71bfdb6d4f1a82d2f9 (/dev/raw/raw1) [DATA]
    Located 1 voting disk(s).
    CRS-2672: Attempting to start ‘ora.asm‘ on ‘node1‘
    CRS-2676: Start of ‘ora.asm‘ on ‘node1‘ succeeded
    CRS-2672: Attempting to start ‘ora.DATA.dg‘ on ‘node1‘
    CRS-2676: Start of ‘ora.DATA.dg‘ on ‘node1‘ succeeded
    Preparing packages for installation...
    cvuqdisk-1.0.9-1
    Configure Oracle Grid Infrastructure for a Cluster ... succeeded

节点2
[[email protected] grid]# ./root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

检查资源信息

节点1
[[email protected] grid]# crs_stat -t
Name Type Target State Host

ora.DATA.dg ora....up.type ONLINE ONLINE node1
ora....N1.lsnr ora....er.type ONLINE ONLINE node1
ora.asm ora.asm.type ONLINE ONLINE node1
ora.cvu ora.cvu.type ONLINE ONLINE node1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE node1
ora....SM1.asm application ONLINE ONLINE node1
ora.node1.gsd application OFFLINE OFFLINE
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip ora....t1.type ONLINE ONLINE node1
ora....SM2.asm application ONLINE ONLINE node2
ora.node2.gsd application OFFLINE OFFLINE
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip ora....t1.type ONLINE ONLINE node2
ora.oc4j ora.oc4j.type ONLINE ONLINE node1
ora.ons ora.ons.type ONLINE ONLINE node1
ora....ry.acfs ora....fs.type ONLINE ONLINE node1
ora.scan1.vip ora....ip.type ONLINE ONLINE node1
[[email protected] grid]# crsctl stat res -t

NAME TARGET STATE SERVER STATE_DETAILS

Local Resources

ora.DATA.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.asm
ONLINE ONLINE node1 Started
ONLINE ONLINE node2 Started
ora.gsd
OFFLINE OFFLINE node1
OFFLINE OFFLINE node2
ora.net1.network
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.ons
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.registry.acfs
ONLINE ONLINE node1
ONLINE ONLINE node2

Cluster Resources

ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node1
ora.cvu
1 ONLINE ONLINE node1
ora.node1.vip
1 ONLINE ONLINE node1
ora.node2.vip
1 ONLINE ONLINE node2
ora.oc4j
1 ONLINE ONLINE node1
ora.scan1.vip
1 ONLINE ONLINE node1

节点2
[[email protected] grid]# crs_stat -t
Name Type Target State Host

ora.DATA.dg ora....up.type ONLINE ONLINE node1
ora....N1.lsnr ora....er.type ONLINE ONLINE node1
ora.asm ora.asm.type ONLINE ONLINE node1
ora.cvu ora.cvu.type ONLINE ONLINE node1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE node1
ora....SM1.asm application ONLINE ONLINE node1
ora.node1.gsd application OFFLINE OFFLINE
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip ora....t1.type ONLINE ONLINE node1
ora....SM2.asm application ONLINE ONLINE node2
ora.node2.gsd application OFFLINE OFFLINE
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip ora....t1.type ONLINE ONLINE node2
ora.oc4j ora.oc4j.type ONLINE ONLINE node1
ora.ons ora.ons.type ONLINE ONLINE node1
ora....ry.acfs ora....fs.type ONLINE ONLINE node1
ora.scan1.vip ora....ip.type ONLINE ONLINE node1
[[email protected] grid]# crsctl stat res -t

NAME TARGET STATE SERVER STATE_DETAILS

Local Resources

ora.DATA.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.asm
ONLINE ONLINE node1 Started
ONLINE ONLINE node2 Started
ora.gsd
OFFLINE OFFLINE node1
OFFLINE OFFLINE node2
ora.net1.network
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.ons
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.registry.acfs
ONLINE ONLINE node1
ONLINE ONLINE node2

Cluster Resources

ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node1
ora.cvu
1 ONLINE ONLINE node1
ora.node1.vip
1 ONLINE ONLINE node1
ora.node2.vip
1 ONLINE ONLINE node2
ora.oc4j
1 ONLINE ONLINE node1
ora.scan1.vip
1 ONLINE ONLINE node1

查看磁盘组信息，如果没有挂载则手动挂载：
SQL> select name,state from v$asm_diskgroup;

4、添加资源(监听、数据库、实例等)

添加监听
[[email protected] ~]$ srvctl add listener -l listener
查看监听
[[email protected] ~]$ srvctl config listener

添加db和instance
[[email protected] ~]$ srvctl add database -h
[[email protected] ~]$ srvctl add database -d orcl -o /u01/app/oracle/product/11.2.0/db_1 -c RAC
[[email protected] ~]$ srvctl add instance -h
[[email protected] ~]$ srvctl add instance -d orcl -i orcl1 -n node1
[[email protected] ~]$ srvctl add instance -d orcl -i orcl2 -n node2
[[email protected] ~]$ srvctl config database -d orcl

5、资源添加完毕，重新启动集群
[[email protected] grid]# crsctl stop cluster -all
[[email protected] grid]# crsctl start cluster -all

添加完成后，可能出现数据库不能自动启动的问题。尝试执行以下语句：
[[email protected] ~]$ srvctl enable database -d orcl
[[email protected] ~]$ srvctl enable instance -d orcl -i orcl1
[[email protected] ~]$ srvctl enable instance -d orcl -i orcl2
[[email protected] ~]$ srvctl start database -d orcl

原文地址：http://blog.51cto.com/lyzbg/2090815

时间： 2024-10-11 16:59:00

ORACLE11gR2-RAC之OCR无备份情况下损坏恢复的相关文章

案例:Oracle dul数据挖掘 没有备份情况下非常规恢复drop删除的数据表

通过Oracle dul工具在没有备份情况下进行非常规恢复,找出drop删除的Oracle数据表中的数据进行恢复 dul对被drop对象进行恢复,需要提供两个信息1.被删除表所属表空间(非必须)2.被删除表结构(必须) 1.Oracle数据库中模拟删除表 --创建测试表 SQL> create table t_dul_drop tablespace czum 2 as 3 select * from dba_tables; Table created. --备份被删除表数据,便于比较和提供测试表

迁移/home目录至新硬盘分区总结--无备份情况下

搞了一天,终于成功迁移.由于一开始就没备份过程实在很曲折. 希望本篇对那些没有备份习惯的朋友们有所帮助. 准备工作: sudo vim /etc/fstab 在文件中加入: /dev/sdb8       /home            ext4    user,rw 0       2 这里其实有问题的,后面会提到 一个新的linux分区,这里我的新分区是sdb8,个人不尽相同.我用的是ext4格式,注意要新的,不要有坏块,最好重新格式化下.我就是因为没格式化,吃过亏,logo界面就提示挂载

无备份情况下回复undo表空间

UNDO表空间存储着DML操作数据块的前镜像数据,在数据回滚,一致性读,闪回操作,实例恢复的时候都可能用到UNDO表空间中的数据.如果在生产过程中丢失或破坏了UNDO表空间,可能导致某些事务无法回滚,数据库无法恢复到一致性的状态,Oracle实例可能宕机,之后实例无法正常启动:如果有多个UNDO表空间数据文件,丢失其中一个数据文件数据库实例可能不会导致实例宕机,数据库无法干净的关闭(只能SHUTDOWN ABORT),数据库实例能正常的重启,但所有未回滚的数据块依然无法处理,尝试新建UNDO表空

Oracle备份恢复之无备份情况下恢复undo表空间

UNDO表空间存储着DML操作数据块的前镜像数据,在数据回滚,一致性读,闪回操作,实例恢复的时候都可能用到UNDO表空间中的数据.如果在生产过程中丢失或破坏了UNDO表空间,可能导致某些事务无法回滚,数据库无法恢复到一致性的状态,Oracle实例可能宕机,之后实例无法正常启动:如果有多个UNDO表空间数据文件,丢失其中一个数据文件数据库实例可能不会导致实例宕机,数据库无法干净的关闭(只能SHUTDOWN ABORT),数据库实例能正常的重启,但所有未回滚的数据块依然无法处理,尝试新建UNDO表空

Oracle主库归档丢失，备库日志有gap，在不重建备库的情况下，恢复备库

本文主要描述Oracle备库日志与主库日志之间有gap,切主库这部分gap的归档日志已经删除或丢失,如何在不重建备库的情况下,恢复备库. 欢迎转载,请注明作者.出处. 作者:张正 blog:http://space.itpub.net/26355921 QQ:176036317 如有疑问,欢迎联系. 在dataguard环境中,由于主库archivelog丢失,且尚未同步到standby,问如何在避免 重建standby的情况下来将standby恢复成功的. 下面是我的测试过程,供参考! —-主

无网络情况下rpm方式安装

记录一下最近遇到的坑: 在无网络的情况下,rpm方式安装linux软件,遇到报nokey的错误,下载了好几个RPM-GPG-KEY都没有用,最后使用rpm -ivh *.rpm --force --nodeps命令解决问题. 下次好好研究下rpm命令 顺便记录下搜狐强大的mirror:http://mirrors.sohu.com/

C++对象模型——"无继承"情况下的对象构造(第五章)

5.2 继承体系下的对象构造 当定义一个object如下: T object; 时,实际上会发生什么事情呢?如果T有一个constructor(不论是由user提供或是由编译器合成),它会被调用.这很明显,比较不明显的是,constructor的调用真正伴随了什么? constructor可能内带大量的隐藏码,因为编译器会扩充每一个constructor,扩充程度视 class T的继承体系而定.一般而言,编译器所做的扩充操作大约如下: 1.记录在member initialization li

无归档情况下使用BBED处理ORA-01113错误

在丢失归档情况下,恢复时常会遇到ora-01113错误,以下实验模拟表空间offline,然后在丢失归档文件的情况下使用BBED修改文件头信息,最后恢复数据文件: 数据库版本: SQL> select * from v$version; BANNER -------------------------------------------------------------------------------- Oracle Database 11g Enterprise Edition Rele

无网络情况下 如何安装GCC

在有网络的情况下安装gcc只需一条指令:yum install gcc  那么在没有网络的情况下该如何安装gcc呢?虽然没有网络,但是我想你应该有安装光盘或者ISO镜像了,如果这些也没有的话,那就. 假设你有这些吧,我们只需利用安装光盘或ISO镜像来挂载一个本地yum源,利用这个来安装gcc. 一:挂载yum源,我这里用的是ISO镜像     在终端输入指令:mount -o loop xxxx.iso  /media/Centos/ 说明:xxxx.iso   是你的镜像文件名(注意路径正确)