数据库rac启动时报错，日志如下，后来使用
Sat Jun 7 06:02:11 2014
GATHER_STATS_JOB encountered errors. Check the trace file.
Sat Jun 7 06:02:11 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-08103: object no longer exists
Sat Jun 7 06:02:51 2014
Thread 2 advanced to log sequence 13292 (LGWR switch)
Current log# 4 seq# 13292 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Sat Jun 7 06:32:10 2014
Trace dumping is performing id=[cdmp_20140607063209]
Sat Jun 7 07:01:33 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j000_6797.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun 7 07:01:35 2014
Trace dumping is performing id=[cdmp_20140607070135]
Sat Jun 7 07:39:01 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_24893.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun 7 07:39:04 2014
Trace dumping is performing id=[cdmp_20140607073904]
Sat Jun 7 08:00:42 2014
Trace dumping is performing id=[cdmp_20140607080042]
Sat Jun 7 08:30:13 2014
Trace dumping is performing id=[cdmp_20140607083013]
Sat Jun 7 09:00:43 2014
Trace dumping is performing id=[cdmp_20140607090043]
Sat Jun 7 09:36:24 2014
Trace dumping is performing id=[cdmp_20140607093623]
Sat Jun 7 10:00:46 2014
Trace dumping is performing id=[cdmp_20140607100046]
Sat Jun 7 10:38:31 2014
Trace dumping is performing id=[cdmp_20140607103831]
Sat Jun 7 11:00:52 2014
Trace dumping is performing id=[cdmp_20140607110051]
Sat Jun 7 11:37:40 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_18978.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun 7 11:37:43 2014
Trace dumping is performing id=[cdmp_20140607113743]
Sat Jun 7 12:00:51 2014
Trace dumping is performing id=[cdmp_20140607120050]
Sat Jun 7 12:38:58 2014
Trace dumping is performing id=[cdmp_20140607123858]
Sat Jun 7 13:00:56 2014
Trace dumping is performing id=[cdmp_20140607130056]
Sat Jun 7 13:39:25 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_31749.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun 7 13:39:28 2014
Trace dumping is performing id=[cdmp_20140607133928]
Sat Jun 7 13:44:08 2014
Thread 2 advanced to log sequence 13293 (LGWR switch)
Current log# 3 seq# 13293 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Sat Jun 7 14:00:59 2014
Trace dumping is performing id=[cdmp_20140607140058]
Sat Jun 7 14:38:47 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_5659.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun 7 14:38:48 2014
Trace dumping is performing id=[cdmp_20140607143848]
Sat Jun 7 15:00:44 2014
Trace dumping is performing id=[cdmp_20140607150043]
Sat Jun 7 15:34:03 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_12186.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun 7 15:34:05 2014
Trace dumping is performing id=[cdmp_20140607153405]
Sat Jun 7 15:43:52 2014
Thread 2 advanced to log sequence 13294 (LGWR switch)
Current log# 4 seq# 13294 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Sat Jun 7 16:02:14 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun 7 16:02:19 2014
Trace dumping is performing id=[cdmp_20140607160219]
Sat Jun 7 16:03:35 2014
Thread 2 advanced to log sequence 13295 (LGWR switch)
Current log# 3 seq# 13295 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Sat Jun 7 16:40:00 2014
Trace dumping is performing id=[cdmp_20140607164000]
Sat Jun 7 17:01:30 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun 7 17:01:33 2014
Trace dumping is performing id=[cdmp_20140607170133]
Sat Jun 7 17:32:37 2014
Trace dumping is performing id=[cdmp_20140607173236]
Sat Jun 7 18:01:51 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun 7 18:01:53 2014
Trace dumping is performing id=[cdmp_20140607180153]
Sat Jun 7 18:29:15 2014
Thread 2 advanced to log sequence 13296 (LGWR switch)
Current log# 4 seq# 13296 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Sat Jun 7 18:30:52 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_19302.trc:
Sat Jun 7 18:30:54 2014
Trace dumping is performing id=[cdmp_20140607183054]
Sat Jun 7 18:30:58 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_19300.trc:
Sat Jun 7 18:31:05 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_19300.trc:
Sat Jun 7 18:42:14 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_31207.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun 7 18:42:20 2014
Trace dumping is performing id=[cdmp_20140607184220]
Sat Jun 7 19:00:04 2014
Trace dumping is performing id=[cdmp_20140607190003]
Sat Jun 7 19:31:17 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_4965.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun 7 19:31:18 2014
Trace dumping is performing id=[cdmp_20140607193118]
Sat Jun 7 20:00:42 2014
Trace dumping is performing id=[cdmp_20140607200042]
Sat Jun 7 20:31:30 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_11315.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun 7 20:31:32 2014
Trace dumping is performing id=[cdmp_20140607203132]
Sat Jun 7 21:33:02 2014
Trace dumping is performing id=[cdmp_20140607213302]
Sat Jun 7 22:02:16 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j000_6797.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun 7 22:17:00 2014
Thread 2 advanced to log sequence 13297 (LGWR switch)
Current log# 3 seq# 13297 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Sat Jun 7 22:30:38 2014
Trace dumping is performing id=[cdmp_20140607223037]
Sat Jun 7 23:01:21 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun 7 23:01:25 2014
Trace dumping is performing id=[cdmp_20140607230125]
Sat Jun 7 23:30:33 2014
Trace dumping is performing id=[cdmp_20140607233032]
Sun Jun 8 00:00:38 2014
Trace dumping is performing id=[cdmp_20140608000038]
Sun Jun 8 00:39:30 2014
Trace dumping is performing id=[cdmp_20140608003930]
Sun Jun 8 01:00:36 2014
Trace dumping is performing id=[cdmp_20140608010036]
Sun Jun 8 01:45:14 2014
Trace dumping is performing id=[cdmp_20140608014514]
Sun Jun 8 02:00:41 2014
Trace dumping is performing id=[cdmp_20140608020041]
Sun Jun 8 02:36:03 2014
Trace dumping is performing id=[cdmp_20140608023603]
Sun Jun 8 03:31:51 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_24887.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun 8 03:31:54 2014
Trace dumping is performing id=[cdmp_20140608033154]
Sun Jun 8 04:05:37 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=54
System State dumped to trace file /oracle/product/admin/dqb/bdump/dqb2_mmon_1398.trc
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 04:47:20 2014
Trace dumping is performing id=[cdmp_20140608044719]
Sun Jun 8 05:00:22 2014
Trace dumping is performing id=[cdmp_20140608050022]
Sun Jun 8 05:39:23 2014
Trace dumping is performing id=[cdmp_20140608053922]
Sun Jun 8 06:00:28 2014
Trace dumping is performing id=[cdmp_20140608060027]
Sun Jun 8 06:32:45 2014
Trace dumping is performing id=[cdmp_20140608063245]
Sun Jun 8 07:00:30 2014
Trace dumping is performing id=[cdmp_20140608070030]
Sun Jun 8 07:11:17 2014
Thread 2 advanced to log sequence 13298 (LGWR switch)
Current log# 4 seq# 13298 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Sun Jun 8 07:30:12 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_6955.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun 8 07:30:15 2014
Trace dumping is performing id=[cdmp_20140608073015]
Sun Jun 8 08:00:32 2014
Trace dumping is performing id=[cdmp_20140608080031]
Sun Jun 8 08:30:04 2014
Trace dumping is performing id=[cdmp_20140608083004]
Sun Jun 8 09:36:04 2014
Trace dumping is performing id=[cdmp_20140608093604]
Sun Jun 8 10:00:27 2014
Trace dumping is performing id=[cdmp_20140608100026]
Sun Jun 8 10:35:29 2014
Trace dumping is performing id=[cdmp_20140608103528]
Sun Jun 8 11:00:32 2014
Trace dumping is performing id=[cdmp_20140608110031]
Sun Jun 8 11:31:07 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_32192.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun 8 11:31:13 2014
Trace dumping is performing id=[cdmp_20140608113113]
Sun Jun 8 12:39:37 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_5862.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun 8 12:39:38 2014
Trace dumping is performing id=[cdmp_20140608123938]
Sun Jun 8 13:00:20 2014
Trace dumping is performing id=[cdmp_20140608130020]
Sun Jun 8 13:40:14 2014
Trace dumping is performing id=[cdmp_20140608134014]
Sun Jun 8 14:00:21 2014
Trace dumping is performing id=[cdmp_20140608140020]
Sun Jun 8 14:35:35 2014
Trace dumping is performing id=[cdmp_20140608143534]
Sun Jun 8 15:34:51 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=54
System State dumped to trace file /oracle/product/admin/dqb/bdump/dqb2_mmon_1398.trc
Sun Jun 8 15:37:47 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_25433.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun 8 15:37:50 2014
Trace dumping is performing id=[cdmp_20140608153750]
Sun Jun 8 16:13:50 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=121
System State dumped to trace file /oracle/product/admin/dqb/udump/dqb2_ora_11844.trc
Sun Jun 8 16:58:04 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=136
System State dumped to trace file /oracle/product/admin/dqb/udump/dqb2_ora_25750.trc
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun 8 21:36:21 2014
Trace dumping is performing id=[cdmp_20140608213621]
Sun Jun 8 21:36:22 2014
Thread 2 advanced to log sequence 13299 (LGWR switch)
Current log# 3 seq# 13299 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Sun Jun 8 22:00:15 2014
Trace dumping is performing id=[cdmp_20140608220015]
Sun Jun 8 22:40:03 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_22234.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun 8 22:40:06 2014
Trace dumping is performing id=[cdmp_20140608224006]
Sun Jun 8 23:00:18 2014
Trace dumping is performing id=[cdmp_20140608230018]
Sun Jun 8 23:29:52 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_27653.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun 8 23:29:53 2014
Trace dumping is performing id=[cdmp_20140608232953]
Mon Jun 9 00:44:11 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_30862.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Mon Jun 9 00:44:15 2014
Trace dumping is performing id=[cdmp_20140609004415]
Mon Jun 9 01:24:41 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=174
System State dumped to trace file /oracle/product/admin/dqb/udump/dqb2_ora_13953.trc
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun 9 05:56:24 2014
Trace dumping is performing id=[cdmp_20140609055624]
Mon Jun 9 05:57:37 2014
Thread 2 advanced to log sequence 13300 (LGWR switch)
Current log# 4 seq# 13300 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Mon Jun 9 06:07:55 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j006_15568.trc:
ORA-12012: error on auto execute of job 524
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at "ZDDX_NEW.PKG_BB_LKZHFJTJ_W", line 537
ORA-06512: at "ZDDX_NEW.PKG_BB_LKZHFJTJ_W", line 9
ORA-06512: at "ZDDX_NEW.PKG_BB_TOOL_W", line 309
ORA-06512: at line 1
Mon Jun 9 06:41:35 2014
Thread 2 advanced to log sequence 13301 (LGWR switch)
Current log# 3 seq# 13301 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Mon Jun 9 06:57:53 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=54
System State dumped to trace file /oracle/product/admin/dqb/bdump/dqb2_mmon_1398.trc
Mon Jun 9 07:24:19 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=174
System State dumped to trace file /oracle/product/admin/dqb/udump/dqb2_ora_1963.trc
Mon Jun 9 09:45:09 2014
Trace dumping is performing id=[cdmp_20140609094508]
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_asmb_1428.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Mon Jun 9 09:45:36 2014
ASMB: terminating instance due to error 15064
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lmon_1337.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_pmon_1331.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms5_1361.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms1_1345.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms2_1349.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms6_1365.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms0_1341.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms7_1369.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms3_1353.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms4_1357.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lmd0_1339.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun 9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun 9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun 9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun 9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun 9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun 9 09:45:38 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_rbal_1432.trc:
ORA-15064: communication failure with ASM instance
Mon Jun 9 09:45:39 2014
Trace dumping is performing id=[cdmp_20140609094539]
Mon Jun 9 09:45:47 2014
Instance terminated by ASMB, pid = 1428
Mon Jun 9 09:53:01 2014
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 192.168.1.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.177.1.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 300
sessions = 335
__shared_pool_size = 452984832
__large_pool_size = 16777216
__java_pool_size = 16777216
__streams_pool_size = 0
spfile = +DG/dqb/spfiledqb.ora
sga_target = 1610612736
control_files = +DG/dqb/controlfile/current.272.803844991
control_file_record_keep_time= 8
db_block_size = 8192
__db_cache_size = 1107296256
compatible = 10.2.0.3.0
log_archive_dest_1 = LOCATION=+dg/dqb/arch/arch1
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +DG
db_recovery_file_dest_size= 10737418240
thread = 2
instance_number = 2
undo_management = AUTO
undo_tablespace = UNDOTBS4
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=dqbXDB)
remote_listener = LISTENERS_DQB
job_queue_processes = 10
background_dump_dest = /oracle/product/admin/dqb/bdump
user_dump_dest = /oracle/product/admin/dqb/udump
core_dump_dest = /oracle/product/admin/dqb/cdump
audit_file_dest = /oracle/product/admin/dqb/adump
db_name = dqb
open_cursors = 300
pga_aggregate_target = 3363831808
Cluster communication is configured to use the following interface(s) for this instance
192.168.1.2
Mon Jun 9 09:53:01 2014
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=4, OS id=13866
DIAG started with pid=8, OS id=13868
PSP0 started with pid=12, OS id=13870
LMON started with pid=16, OS id=13872
LMD0 started with pid=2, OS id=13874
LMS0 started with pid=6, OS id=13876
LMS1 started with pid=3, OS id=13880
LMS2 started with pid=20, OS id=13884
LMS3 started with pid=5, OS id=13888
LMS4 started with pid=10, OS id=13892
LMS5 started with pid=7, OS id=13896
LMS6 started with pid=24, OS id=13900
LMS7 started with pid=9, OS id=13904
MMAN started with pid=28, OS id=13908
DBW0 started with pid=14, OS id=13910
DBW1 started with pid=11, OS id=13912
DBW2 started with pid=32, OS id=13914
DBW3 started with pid=13, OS id=13916
LGWR started with pid=15, OS id=13918
CKPT started with pid=36, OS id=13920
SMON started with pid=40, OS id=13922
RECO started with pid=44, OS id=13924
CJQ0 started with pid=18, OS id=13926
MMON started with pid=48, OS id=13928
Mon Jun 9 09:53:02 2014
starting up 1 dispatcher(s) for network address ‘(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))‘...
MMNL started with pid=52, OS id=13936
Mon Jun 9 09:53:02 2014
starting up 1 shared server(s) ...
Mon Jun 9 09:53:02 2014
lmon registered with NM - instance id 2 (internal mem no 1)
Mon Jun 9 09:53:02 2014
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon Jun 9 09:53:02 2014
LMS 2: 0 GCS shadows cancelled, 0 closed
Mon Jun 9 09:53:02 2014
LMS 0: 0 GCS shadows cancelled, 0 closed
Mon Jun 9 09:53:02 2014
LMS 3: 0 GCS shadows cancelled, 0 closed
Mon Jun 9 09:53:02 2014
LMS 1: 0 GCS shadows cancelled, 0 closed
Mon Jun 9 09:53:02 2014
LMS 6: 0 GCS shadows cancelled, 0 closed
Mon Jun 9 09:53:02 2014
LMS 7: 0 GCS shadows cancelled, 0 closed
Mon Jun 9 09:53:02 2014
LMS 5: 0 GCS shadows cancelled, 0 closed
Mon Jun 9 09:53:02 2014
LMS 4: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Mon Jun 9 09:53:02 2014
LMS 2: 0 GCS shadows traversed, 0 replayed
Mon Jun 9 09:53:02 2014
LMS 3: 0 GCS shadows traversed, 0 replayed
Mon Jun 9 09:53:02 2014
LMS 7: 0 GCS shadows traversed, 0 replayed
Mon Jun 9 09:53:02 2014
LMS 1: 0 GCS shadows traversed, 0 replayed
Mon Jun 9 09:53:02 2014
LMS 0: 0 GCS shadows traversed, 0 replayed
Mon Jun 9 09:53:02 2014
LMS 5: 0 GCS shadows traversed, 0 replayed
Mon Jun 9 09:53:02 2014
LMS 4: 0 GCS shadows traversed, 0 replayed
Mon Jun 9 09:53:02 2014
LMS 6: 0 GCS shadows traversed, 0 replayed
Mon Jun 9 09:53:02 2014
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=30, OS id=13942
Mon Jun 9 09:53:03 2014
ALTER DATABASE MOUNT
Mon Jun 9 09:53:03 2014
This instance was first to mount
Mon Jun 9 09:53:03 2014
Starting background process ASMB
ASMB started with pid=56, OS id=13953
Starting background process RBAL
RBAL started with pid=60, OS id=13957
Mon Jun 9 09:53:06 2014
SUCCESS: diskgroup DG was mounted
Mon Jun 9 09:53:10 2014
Setting recovery target incarnation to 2
Mon Jun 9 09:53:11 2014
Successful mount of redo thread 2, with mount id 4128760447
Mon Jun 9 09:53:11 2014
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Mon Jun 9 09:53:11 2014
ALTER DATABASE OPEN
This instance was first to open
Mon Jun 9 09:53:12 2014
Beginning crash recovery of 2 threads
parallel recovery started with 16 processes
Mon Jun 9 09:53:14 2014
Started redo scan
Mon Jun 9 09:53:14 2014
Completed redo scan
26 redo blocks read, 8 data blocks need recovery
Mon Jun 9 09:53:14 2014
Started redo application at
Thread 1: logseq 16276, block 199337, scn 13914083540861
Thread 2: logseq 13301, block 55078
Mon Jun 9 09:53:14 2014
Recovery of Online Redo Log: Thread 1 Group 2 Seq 16276 Reading mem 0
Mem# 0: +DG/dqb/onlinelog/group_2.275.803844993
Mon Jun 9 09:53:14 2014
Recovery of Online Redo Log: Thread 2 Group 3 Seq 13301 Reading mem 0
Mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Mon Jun 9 09:53:14 2014
Completed redo application
Mon Jun 9 09:53:14 2014
Completed crash recovery at
Thread 1: logseq 16276, block 199337, scn 13914083560862
Thread 2: logseq 13301, block 55104, scn 13914083661134
8 data blocks read, 8 data blocks written, 26 redo blocks read
Attempt to switch log for thread 1 failed. reason: 3
Picked broadcast on commit scheme to generate SCNs
Mon Jun 9 09:53:14 2014
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=58, OS id=14055
Mon Jun 9 09:53:14 2014
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=35, OS id=14062
ARC1: Becoming the ‘no FAL‘ ARCH
ARC1: Becoming the ‘no SRL‘ ARCH
Mon Jun 9 09:53:14 2014
ARC0: Becoming the heartbeat ARCH
Mon Jun 9 09:53:17 2014
Thread 2 advanced to log sequence 13302 (thread open)
Thread 2 opened at log sequence 13302
Current log# 4 seq# 13302 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Successful open of redo thread 2
Mon Jun 9 09:53:17 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Jun 9 09:53:17 2014
SMON: enabling cache recovery
Mon Jun 9 09:53:18 2014
ARC0: Switch failed
Mon Jun 9 09:53:18 2014
Successfully onlined Undo Tablespace 35.
Mon Jun 9 09:53:18 2014
SMON: enabling tx recovery
Mon Jun 9 09:53:18 2014
Database Characterset is ZHS16GBK
Opening with internal Resource Manager plan
where NUMA PG = 4, CPUs = 20
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=62, OS id=14171
Mon Jun 9 09:53:19 2014
Thread 1 advanced to log sequence 16277 (archiver wakeup)
Mon Jun 9 09:53:22 2014
Completed: ALTER DATABASE OPEN
Mon Jun 9 09:53:26 2014
ALTER SYSTEM SET service_names=‘rac‘ SCOPE=MEMORY SID=‘dqb2‘;
Mon Jun 9 10:01:02 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_14197.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Mon Jun 9 10:01:06 2014
Trace dumping is performing id=[cdmp_20140609100106]
Mon Jun 9 10:03:24 2014
ALTER SYSTEM SET service_names=‘rac‘,‘dqb‘ SCOPE=MEMORY SID=‘dqb2‘;
叫同事手工启动ASM实例，一步步看报错信息
export ORACLE_SID=+ASM1
sqlplus / as sysdba
sql>startup;
提示ora-29701；
客户报oracle数据库出现了ora-29701错误，查看oracle的 error message
- ORA-29701: unable to connect to Cluster Manager
- Cause: Connect to CM failed or timed out.
- Action: Verify that the CM was started. If the CM was not started, start it and then retry the database startup.
- If the CM died or is not responding, check the Oracle and CM trace files for errors.
当时以为肯定是cluster出问题了，但是客户说查询都可以执行，就是执行create table语句就出现这个错误，怪了！ 后来通过查看数 据库的告警日志的过程中发现原来是$ORACLE_HOME的空间满了，通过删除没有的trace文件，数据库一切正常！
汗， 后来让同事尝试使用手工启动ASM实例，提示Ora-29701，汗，原来是磁盘空间满
oracle数据库启动报错，不能启动ASM实例