现象：
节点2 运行：select sum(bytes) from dba_segments 长期等待： waiting for ‘gc cr request‘
节点1 僵死：select sid,serial# from gv$session where username=‘****‘
分析：
数据库内部等待关系：
blocking blocker event
388 773,397 enq: PV - syncstart
773 397 enq: PR - contention
397 enq: PR - contention os thread startup：ora-445
--1
节点1 运行语句select sid,serial# from gv$session where username=‘WANQIONG‘:
失败的原因：PROCESS 53/57 两个会话运行该语句,PROCESS 20: CKPT block这两个会话
PROCESS 20 由于waiting for ‘enq: PR - contention‘ name|mode=0x50520006
0x50520006 所有者：PROCESS 45: CJQ0
根据：awk -f ass109.awk calldb1_ora_15639.trc 分析：
Resource Holder State
Enqueue PR-00000000-00000000 45: 0: waiting for ‘os thread startup‘
Enqueue PV-00000000-00000000 38: 38: is waiting for 45:
Latch 380030bc0 ??? Blocker
PROCESS 45: CJQ0 进程是罪魁祸首！该进程无法启动。
--2 节点2
节点2运行语句：select sum(bytes) from dba_segments，等待时间0: waiting for ‘gc cr request‘
cssd,crsd没有发现明显节点通讯报错，怀疑由于节点1死机引起
--3 节点1 警告日志分析
Mon Aug 11 09:45:49 2014
Archived Log entry 58557 added for thread 1 sequence 27907 ID 0x4bcd9425 dest 1:
Mon Aug 11 16:08:44 2014
Errors in file /oracle/diag/rdbms/calldb/calldb1/trace/calldb1_cjq0_8914.trc (incident=56362):
ORA-00445: background process "J000" did not start after 120 seconds
Incident details in: /oracle/diag/rdbms/calldb/calldb1/incident/incdir_56362/calldb1_cjq0_8914_i56362.trc
kkjcre1p: unable to spawn jobq slave process
Errors in file /oracle/diag/rdbms/calldb/calldb1/trace/calldb1_cjq0_8914.trc:
Mon Aug 11 16:12:25 2014
Errors in file /oracle/diag/rdbms/calldb/calldb1/trace/calldb1_mmon_8497.trc (incident=56202):
ORA-00445: background process "m000" did not start after 120 seconds
Incident details in: /oracle/diag/rdbms/calldb/calldb1/incident/incdir_56202/calldb1_mmon_8497_i56202.trc
Mon Aug 11 16:14:28 2014
awr分析内存16G
Begin End
Host Mem (MB): 15,744.0 15,744.0
SGA use (MB): 4,608.0 4,608.0
PGA use (MB): 462.5 460.1
% Host Mem used for SGA+PGA: 32.21 32.19
重启后内存12g
[[email protected] /home/oracle]$ prtconf | grep ‘Memory‘
Memory size: 12288 Megabytes
怀疑问题出在内存上：
1379200.1 根据文档可以确定：
ORA-00445 实例挂起