In this Document

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.1.0.6 and later

Information in this document applies to any platform.

***Checked for relevance on 10-June-2013***

PURPOSE

The purpose of the note is to help customers troubleshoot contention for wait event ‘library cache: mutex X‘.

TROUBLESHOOTING STEPS

What is a ‘library cache: mutex X‘ wait?

Library cache: mutex X is similar to a library cache wait in earlier versions.  ‘Library cache: mutex X‘ may be caused by many issues.  So it is important to find the correct cause; so that the right solution can be implemented.

What causes ‘library cache: mutex X‘ wait?

• Frequent Hard Parses - If the frequency of Hard Parsing is extremely high, then contention can occur on this pin.
• High Version Counts - When Version counts become excessive, a long chain of versions needs to be examined and this can lead to contention on this event
• Invalidations and reloads
• Known Bugs

How to diagnose the cause.

1. Check to see if anything has changed:

a. increased load

b. any change in the application, os, or middle tier

c. any os change

2. Is there a trend to library cache: mutex X:

a. is there a certain time of the day when this wait is seen?

b. does something trigger this wait?

3. During the time of the issue, run AWR and ADDM.  Also obtain the baseline to compare the load, parameter changes, and any other differences.

To gather this it is suggested to run AWR and ADDM for half an hour to an hour interval as follows:

SQL>@$ORACLE_HOME/rdbms/admin/awrrpt.sql

SQL>@$ORACLE_HOME/rdbms/admin/addmrpt.sql

4. Sometimes system state dump is necessary to match known issues. For example, if there is no obvious candidate SQL in AWR, capturing holder or waiter processes in systemstate allows you to focus in on potential problems. Run system state when processes appear
hung on ‘library cache: mutex X‘:

(a) Non-Rac

sqlplus "/ as sysdba"

oradebug setmypid

oradebug unlimit

oradebug dump systemstate 266

wait 90 seconds

oradebug dump systemstate 266

wait 90 seconds

oradebug dump systemstate 266

quit

(b) RAC

$ sqlplus ‘/ as sysdba‘

oradebug setmypid

oradebug unlimit

oradebug setinst all

oradebug -g all hanganalyze 4

oradebug -g all dump systemstate 266

quit

5.  Errorstacks: Another way to obtain process information is with errorstack. Assuming you can identify a blocker, taking errorstacks will provide much the same information as systemstates but with a much reduced disk footprint for trace. Once the ospid of
the blocker has been found, an errorstack can be generated:

$ sqlplus

SQL> oradebug setospid <p.spid from above>

oradebug dump errorstack 3

<< wait 1min>>

oradebug dump errorstack 3

<< wait 1min>>

oradebug dump errorstack 3

exit

In particular, the stack from the resultant trace can be used to match known issues.

The system state and errorstacks are not easily readable; so a Service Request may need to be opened to read the files.

6. Sometimes it is not feasible to run system state dump, as it may be resource intensive.  So the following sql can also be ran in interval:

select s.sid, t.sql_text

from v$session s, v$sql t

where s.event like ‘%mutex%‘

and t.sql_id = s.sql_id

Check to see what sessions are waiting on.

7. In 11g RAC, there is another less resource intensive tool that can be used when compared with taking system state dumps:

Document 459694.1 Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware
Processes

How to Examine the Diagnostics.

1. Normally, the top wait event will be the library cache: mutex X in the problematic AWR:

2. First look for high parsing and high version counts from AWR.

Click on *SQL Statistics under Main Report of AWR:

Then, under SQL Statistics click on ‘SQL ordered by Parse Calls‘ and  ‘SQL ordered by Version Count‘ to view that information:

Check for high parse calls.

Check to see if there is high parse calls to execute.  Ideally, there should be less parse to executions.  Notice there is as many parses as executes, indicating cursors are not used well in the application.  Once the cursor is opened and parsed, it should
be kept open.  Check with the application developer on how to keep the cursor opened to re execute the sqls.

Next, check the version count of the sql:

From this list, investigate the SQLs with the high version count. What are the reasons that these statements are not shared? Can this be addressed?

Check V$SQL_SHARED_CURSOR to see the potential reason for the high version count using:

Document 438755.1Formated V$SQL_SHARED_CURSOR Report by SQLID or Hash Value

Document 296377.1Troubleshooting: High Version Count Issues

Potential Solutions

1. Check for high hard  parsing, as this can cause can reloads in the sql area.  Check the hard parse under the load profile:

This load shows 26.3 hard parses per second, indicating high hard parsing.  Check to see if the application is sharing the sql.  If application is mostly using literals, see if the sqls can be shared by using bind variables. Furthermore, review the ‘Over Parsing‘
section of the following note:

Document 33089.1TROUBLESHOOTING: Possible Causes of Poor SQL Performance

Also check for high reloads in the sql area:

If there is a high number of reloads, then look to see if cursors are being shared efficiently. If they are then check to see if the shared pool or sga_target is large enough. Remember that inefficient sharing means that the library cache will fill up with
non-reuseable cursors which may then cause reuseable ones to be flushed out. These will cause reloads when they are re-executed.

If sharing is efficient and the shared pool is too small, then shareable SQL statements will age out and hard parses will be higher. In most cases however this is not the case and the problem is inefficient sharing.

The following note is helpful in tuning the shared pool:

Document 62143.1 Understanding and Tuning the Shared Pool

2. Check for invalidations under Library Cache Activity.  If the invalidation has high number, then check ddl‘s performed during the time such as truncate, drop, grants, dbms_stats, etc.

3. Check the following note for relevant bugs under ‘Known Bugs‘ and relevant version:

Document 727400.1WAITEVENT: "library cache: mutex X"

4. For 11g, make sure cursor_sharing is not similar, as it has been deprecated.  This may also cause mutex waits:

Document 1169017.1ANNOUNCEMENT: Deprecating the cursor_sharing = ‘SIMILAR’ setting

5. If the database has been migrated from 10g to 11g and mutex performance issue surfaces, please consider the 11.2.0.2.2 psu + fix for Unpublished Bug
12431716. Many mutex fixes are already included in this patch:

Document 1291879.1 Oracle Database Patch Set Update 11.2.0.2.2 Known Issues

REFERENCES

NOTE:438755.1 - High SQL Version Counts - Script to determine reason(s)

NOTE:1169017.1 - ANNOUNCEMENT: Deprecating the cursor_sharing = ‘SIMILAR‘ setting

NOTE:1291879.1 - Oracle Database Patch Set Update 11.2.0.2.2 Known Issues

NOTE:296377.1 - Troubleshooting: High Version Count Issues

NOTE:33089.1 - * TROUBLESHOOTING: Possible Causes of Poor SQL Performance

NOTE:459694.1 - Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware
Processes

NOTE:62143.1 - Troubleshooting: Tuning the Shared Pool and Tuning Library Cache Latch
Contention

NOTE:727400.1 - WAITEVENT: "library cache: mutex X"

• Snap1.bmp(196.68
  KB)
• Snap11.bmp(671.54
  KB)
• Snap2.bmp(181.04
  KB)
• Snap3.bmp(191.65
  KB)
• Snap5.bmp(389.55
  KB)
• Snap8.bmp(452.13
  KB)
• Snap9.bmp(485.99
  KB)

• Oracle
  Database Products > Oracle
  Database > Oracle
  Database > Oracle
  Database - Enterprise Edition > RDBMS > Database
  Level Performance Issues (not SQL Tuning)

CACHE;LIBRARY;MUTEX;TROUBLESHOOT

Troubleshooting 'library cache: mutex X' waits. (文档 ID 1357946.1)