早上一台rabbitmq和Java所在的服务器,客户端反馈超级卡,看io和cpu都不高。发现六七万消息挤压,临时性问题解决之后,看/var/log/messages,发现很多TCP: time wait bucket table overflow,如下所示:
Nov 22 10:36:08 iZ237hn51s7Z kernel: TCP: time wait bucket table overflow
Nov 22 10:36:08 iZ237hn51s7Z kernel: TCP: time wait bucket table overflow
Nov 22 10:37:07 iZ237hn51s7Z kernel: __ratelimit: 47 callbacks suppressed
Nov 22 10:37:07 iZ237hn51s7Z kernel: TCP: time wait bucket table overflow
Nov 22 10:37:07 iZ237hn51s7Z kernel: TCP: time wait bucket table overflow
Nov 22 10:37:07 iZ237hn51s7Z kernel: TCP: time wait bucket table overflow
Nov 22 10:37:07 iZ237hn51s7Z kernel: TCP: time wait bucket table overflow
Nov 22 10:37:07 iZ237hn51s7Z kernel: TCP: time wait bucket table overflow
Nov 22 10:37:07 iZ237hn51s7Z kernel: TCP: time wait bucket table overflow
Nov 22 10:37:07 iZ237hn51s7Z kernel: TCP: time wait bucket table overflow
经查,发现net.ipv4.tcp_max_tw_buckets = 5000 ##已经修改
netstat -ano | grep wait | wc -l发现4400多,进一步查询,发现大量本机到mysql的连接状态处于该状态,进一步发现,有个应用的maxStatement被设置成了0。