那些年我们踩到过的坑（二）：3.1 版 MultiThreadedHttpConnectionManager 未releaseConnection导致应用服务器宕机

昨天短信服务又宕机了，jstack打出线程信息发现 所有线程池的线程都在wait，栈信息如下：

at java.lang.Object.wait(Native Method)
- waiting on [0x000000070754fb60] (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked [0x000000070754fb60] (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at com.dhf.sms.util.HttpUtil.getHtml(HttpUtil.java:75)
at com.dhf.sms.core.impl.MDSmsserviceImpl.send(MDSmsserviceImpl.java:165)
at com.dhf.sms.core.SmsTask.run(SmsTask.java:156)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at com.kuaidadi.framework.thread.bean.TaxiFutureTask.run(TaxiFutureTask.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

发现所有线程都卡在MultiThreadedHttpConnectionManager 的doGetConnection方法：

   hostPool.waitingThreads.addLast(waitingThread);
                        connectionPool.waitingThreads.addLast(waitingThread);
                        connectionPool.wait(timeToWait);//卡在这

查看代码和google，发现应该是connection连接未释放的问题的。如果不调用getMethod.getResponseBodyAsString()，就不会释放connection。并且我们没有指定maxConnectionsPerHost，默认为2，所以当以下代码在第16行抛出exception后连接不会被释放掉，一旦发生两次，那么仅有的2个connection都被占用，新进来的线程就卡住了。

 1  public static String getHtml(String url,String charSet,Integer channelId) throws IOException, SmsHttpStatusException {
 2         StaticLogInfo staticLogInfo = new StaticLogInfo();
 3         staticLogInfo.setCmd("Channel "+channelId+"("+getHost(url)+")");
 4         GetMethod getMethod = null;
 5         try {
 6             if (StringUtils.isBlank(charSet)) {
 7                 charSet = SmsConstant.DEFAULT_CHARSET;
 8             }
 9
10             getMethod = new GetMethod(url);
11             int statusCode = httpClient.executeMethod(getMethod);
12             if (statusCode != HttpStatus.SC_OK
13                     && statusCode != HttpStatus.SC_MOVED_PERMANENTLY
14                     && statusCode != HttpStatus.SC_MOVED_TEMPORARILY) {
15                 log.error("statusCode:" + statusCode + " url请求失败========>" + url);
16                 throw new SmsHttpStatusException("请求失败");
17             }
18
19             String responseBody = new String(getMethod.getResponseBodyAsString().getBytes(charSet),SmsConstant.DEFAULT_CHARSET);
20             return responseBody;
21         } catch (IOException e) {
22             staticLogInfo.setResultCode("1");
23             throw e;
24         } catch (SmsHttpStatusException e) {
25             staticLogInfo.setResultCode("1");
26             throw e;
27         } finally {
28
29             reqLogger.info(staticLogInfo);
30         }
31     }
32     

解决办法是在finally里释放connection：

finally {
            if(getMethod!=null){
                getMethod.releaseConnection();
            }
            reqLogger.info(staticLogInfo);
        }

并且增加最大连接数：

httpConnectionManagerParams.setMaxTotalConnections(100);
        httpConnectionManagerParams.setDefaultMaxConnectionsPerHost(20);

参考：http://blog.csdn.net/xiaojianpitt/article/details/7818971