tomcat服务突然无响应了，导出内存堆栈和线程堆栈，分析后发现是同步锁使用不合理导致的。
[[email protected] ~]# pgrep java
10472
[[email protected] ~]# jmap -heap 10472
Attaching to process ID 10472, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.111-b14
using thread-local object allocation.
Parallel GC with 2 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 1983905792 (1892.0MB)
NewSize = 41943040 (40.0MB)
MaxNewSize = 661127168 (630.5MB)
OldSize = 83886080 (80.0MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 487063552 (464.5MB)
used = 26314992 (25.095932006835938MB)
free = 460748560 (439.40406799316406MB)
5.402784070362958% used
From Space:
capacity = 72351744 (69.0MB)
used = 71945680 (68.61274719238281MB)
free = 406064 (0.3872528076171875MB)
99.43876404693161% used
To Space:
capacity = 84934656 (81.0MB)
used = 0 (0.0MB)
free = 84934656 (81.0MB)
0.0% used
PS Old Generation
capacity = 254279680 (242.5MB)
used = 136744120 (130.40935516357422MB)
free = 117535560 (112.09064483642578MB)
53.77705367570071% used
36326 interned Strings occupying 4333960 bytes.
[[email protected] ~]# jmap -dump:file=dump_dtb 10472
Dumping heap to /root/dump_dtb ...
Heap dump file created
[[email protected] ~]# jstack 10472 > thread_dtb
使用Eclipse MemoryAnalyzer对内存堆栈的分析，发现线程已经占满了。
通过对线程堆栈文件内容的分析，发现大量线程都处于waiting to lock状态，进一步发现，对应代码使用了synchronized同步锁，一个线程内部访问数据库发生了超时，长时间占用了该锁，导致其它线程都处于等待状态。
...
"http-nio-8002-exec-26" #52 daemon prio=5 os_prio=0 tid=0x00007f951c01b000 nid=0x291e waiting for monitor entry [0x00007f9530dc9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at cn.friendsure.tdtb.services.WeixinPayService.payed(WeixinPayService.java:273)
- waiting to lock <0x000000008a9103b0> (a cn.friendsure.tdtb.services.WeixinPayService)
at sun.reflect.GeneratedMethodAccessor264.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:446)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:434)
...
at cn.friendsure.tdtb.logics.OrderLogic.transferOrder(OrderLogic.java:531)
- locked <0x000000008a9104e0> (a cn.friendsure.tdtb.logics.OrderLogic)
at cn.friendsure.tdtb.services.WeixinPayService.transferOrder(WeixinPayService.java:478)
at cn.friendsure.tdtb.services.WeixinPayService.payed(WeixinPayService.java:399)
- locked <0x000000008a9103b0> (a cn.friendsure.tdtb.services.WeixinPayService)
at sun.reflect.GeneratedMethodAccessor264.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
...
【解决方案】
去掉不必要的同步锁。
【总结】
涉及IO的方法，尽量不要使用synchronized关键字，如果一定要用，要确保程序逻辑中有明确的超时控制机制，并且超时时间不要太长。