现象：
java开发的web应用无法访问
排查：
1．从resin/log/watchdog-manager.log的日志里可以看出来，jvm的内存满，无法创建新进程
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at com.caucho.util.ThreadPool$ThreadLauncher.startConnection(ThreadPool.java:812)
at com.caucho.util.ThreadPool$ThreadLauncher.run(ThreadPool.java:851)
at java.lang.Thread.run(Thread.java:595)
2．用jstat –gcutil pid可以看出jvm的OLD堆内存区域已经100%，与上述判断符合
S0 S1 E O P YGC YGCT FGC FGCT GCT
0.00 0.00 99.19 100.00 20.72 577 62.517 322 2953.102 3015.619
3．用jstack pid查看jvm的内部的执行线程情况，可以看到存在大量的下述线程，阻塞
Thread 15810: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park() @bci=5, line=118 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=41, line=1767 (Interpreted frame)
- com.ctsi.nmp.flux.ecfchart.ECFluxChartService.writeChart(com.ctsi.nmp.flux.ecfchart.ECFluxChartType, com.ctsi.nmp.flux.ecfchart.E
CFluxType, java.lang.String, java.util.Date, java.util.Date, java.util.List, boolean, java.io.OutputStream) @bci=295, line=189 (Comp
iled frame)
- com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.executeChart() @bci=1051, line=203 (Compiled frame)
- sun.reflect.GeneratedMethodAccessor49.invoke(java.lang.Object, java.lang.Object[]) @bci=36 (Compiled frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)
- com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.execute() @bci=94, line=230 (Interpreted frame)
- _jsp._nos._flux._flux._chart._ECFluxChart__jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletR
esponse) @bci=130, line=39 (Compiled frame)
- com.caucho.jsp.JavaPage.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=9, line=61 (Interpreted frame)
- com.caucho.jsp.Page.pageservice(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=252, line=578
(Compiled frame)
- com.caucho.server.dispatch.PageFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=262, line=1
95 (Compiled frame)
- com.zhongying.util.EncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain
) @bci=39, line=80 (Compiled frame)
- com.caucho.server.webapp.WebAppFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=140, line=1
87 (Compiled frame)
- com.caucho.server.dispatch.ServletInvocation.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=97, line=2
66 (Compiled frame)
- com.caucho.server.hmux.HmuxRequest.handleRequest() @bci=478, line=435 (Compiled frame)
- com.caucho.server.port.TcpConnection.run() @bci=375, line=678 (Compiled frame)
- com.caucho.util.ThreadPool$Item.runTasks() @bci=246, line=721 (Compiled frame)
- com.caucho.util.ThreadPool$Item.run() @bci=106, line=643 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=595 (Interpreted frame)
所以可以判断是应用的com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.executeChart部分出现了问题，导致大量的线程阻塞，需要研发
排查优化这部分代码。