Oracle RAC 故障处理(二)(+DATA磁盘组故障)

环境：测试

DB:Oracle 11.2.0.4.0

OS:Oracle Linux Server release 6.3 on Oracle VM VirtualBox

node:rac1,rac2

instance:cjcdb1,cjcdb2

问题二：DATA磁盘组损坏，导致数据库无法启动和使用

问题原因：测试RAC环境进行过多次增加节点、删除节点、删除实例等操作，导致DATA磁盘损坏

解决方案：

本次案例为测试环境，直接删除重建DATA磁盘组,重建实例，测试数据将丢失，正式环境勿用！

---1 数据库启动失败，+DATA磁盘组故障

---参数文件，控制文件，数据文件等都存储在+DATA磁盘组

SQL> startup

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file ‘+DATA/cjcdb/spfilecjcdb.ora‘

ORA-17503: ksfdopn:2 Failed to open file +DATA/cjcdb/spfilecjcdb.ora

ORA-15056: additional error message

ORA-17503: ksfdopn:2 Failed to open file +DATA/cjcdb/spfilecjcdb.ora

ORA-15001: diskgroup "DATA" does not exist or is not mounted

ORA-06512: at line 4

[[email protected] bin]# ./srvctl start instance -d cjcdb -i cjcdb1 -o open

PRCR-1013 : Failed to start resource ora.cjcdb.db

PRCR-1064 : Failed to start resource ora.cjcdb.db on node rac1

CRS-5017: The resource action "ora.DATA.dg start" encountered the following error:

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk "3" is missing from group number "1"

. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_grid/oraagent_grid.log".

CRS-2674: Start of ‘ora.DATA.dg‘ on ‘rac1‘ failed

---2 查看当前只有OCR磁盘组，没有DATA磁盘组

[[email protected] rac1]# su - grid

[[email protected] ~]$ asmcmd

ASMCMD> ls

OCR/

[[email protected] bin]# su - grid

[[email protected] ~]$ asmcmd

ASMCMD> ls

OCR/

---3 挂载DATA磁盘组失败

SQL> shutdown abort

ASM instance shutdown

SQL> startup

ORA-00099: warning: no parameter file specified for ASM instance

ASM instance started

Total System Global Area 1135747072 bytes

Fixed Size 2260728 bytes

Variable Size 1108320520 bytes

ASM Cache 25165824 bytes

ASM diskgroups mounted

ASM diskgroups volume enabled

SQL> set line 100

SQL> select name,group_number,state from v$asm_diskgroup;

NAME GROUP_NUMBER STATE

OCR 2 MOUNTED

DATA 0 DISMOUNTED

SQL> set line 150

SQL> col path for a30

SQL> select GROUP_NUMBER,disk_number,state,name,path from v$asm_disk;

GROUP_NUMBER DISK_NUMBER STATE NAME PATH

0 0 NORMAL /dev/asm-diske

0 1 NORMAL /dev/asm-diskg

0 2 NORMAL /dev/asm-diskf

1 1 NORMAL OCR_0001 /dev/asm-diskc

1 0 NORMAL OCR_0000 /dev/asm-diskb

1 2 NORMAL OCR_0002 /dev/asm-diskd

6 rows selected.

SQL> alter diskgroup data mount;

alter diskgroup data mount

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk "3" is missing from group number "1"

SQL> drop diskgroup DATA;
drop diskgroup DATA

ERROR at line 1:

ORA-15039: diskgroup not dropped

ORA-15001: diskgroup "DATA" does not exist or is not mounted

---4 重新初始化DATA磁盘组内磁盘

[[email protected] ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VBde6560cc-5b43c5dc", NAME="asm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB0bc9f076-5bc9501f", NAME="asm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB69c7e48f-359e5a75", NAME="asm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VBf71d68a1-6c9355a0", NAME="asm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB6bae812a-c8d1bb88", NAME="asm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB01fde635-97fae482", NAME="asm-diskg", OWNER="grid", GROUP="asmadmin", MODE="0660"

---asm-diskg 1ATA_VBOX_HARDDISK_VB01fde635-97fae482

---asm-diske 1ATA_VBOX_HARDDISK_VBf71d68a1-6c9355a0

---asm-diskf 1ATA_VBOX_HARDDISK_VB6bae812a-c8d1bb88

[[email protected] ~]# scsi_id -g /dev/sdg

1ATA VBOX HARDDISK VB01fde635-97fae482

[[email protected] ~]# scsi_id -g /dev/sde

1ATA VBOX HARDDISK VBf71d68a1-6c9355a0

[[email protected] ~]# scsi_id -g /dev/sdf

1ATA VBOX HARDDISK VB6bae812a-c8d1bb88

[[email protected] ~]# dd if=/dev/zero of=/dev/sdg bs=1024k count=10

10+0 records in

10+0 records out