MessagingTimeout: Timed out waiting for a reply to message ID

l3中出现大量消息超时错误,对网络的操作各种异常。

报错如下:

2016-02-25 05:54:59.886 15110 ERROR neutron.agent.l3.agent [req-db9207e6-9270-4f23-8c19-0d91d20cc6fb ] Failed synchronizing routers due to RPC error
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 593, in fetch_and_sync_all_routers
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent     routers = self.plugin_rpc.get_routers(context)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 93, in get_routers
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent     router_ids=router_ids)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent     retry=self.retry)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent     timeout=timeout, retry=retry)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent     retry=retry)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent     result = self._waiter.wait(msg_id, timeout)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent     message = self.waiters.get(msg_id, timeout=timeout)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent     ‘to message ID %s‘ % msg_id)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent MessagingTimeout: Timed out waiting for a reply to message ID d4baae114cee4f6d831c5eec3c5f0de3
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent

所有超时都指向同步路由的操作。

而且同步失败时,rabbit中的队列q-l3-plugin中有大量未应答消息积压,该队列为同步路由时使用,路由同步时会使用消息队列传送所有路由的属性详情,消息量很大
1)测试是否由于消息太大导致,编写测试代码,尝试连续1000次发送该消息,并未出现丢失消息的情况,
2)尝试减少路由器数量,短时内情况有所改善,但是随时间增加,消息积压依然有更加严重的趋势
3)尝试合入K版本oslo_messaging的最新更新,未有改善

最终跟踪neutron代码,发现消息队列出现Timeout的原因是:
neutron在同步路由信息时,会从neutron-server获取所有router的信息,这个过程会比较长(130s左右,和网络资源的多少有关系),而 在/etc/neutron/neutron.conf中会有一个配置项“rpc_response_timeout”,它用来配置RPC的超时时间,默认为60s,所以导致超时异常.解决方法为设置 rpc_response_timeout = 180

延时是解决各种问题的大招啊。。。

原文地址:https://www.cnblogs.com/wangjq19920210/p/9242850.html

时间: 2024-10-09 00:15:37

MessagingTimeout: Timed out waiting for a reply to message ID的相关文章

OpenStack报错:MessagingTimeout: Timed out waiting for a reply to message ID

L3.agent中出现大量消息超时错误,对网络的操作各种异常. 报错如下: 2018-09-24 05:54:59.886 15110 ERROR neutron.agent.l3.agent [req-db9207e6-9270-4f23-8c19-0d91d20cc6fb ] Failed synchronizing routers due to RPC error 2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent Trac

终端 timed out waiting for input: auto-logout解决!

问题: 在ssh密钥连接情况下,一定时间不输入,则自动退出. $timed out waiting for input: auto-logout超时等待输入:自动注销 一开始感觉和ssh检测机制有关系, 1.ssh超时 设置服务器向SSH客户端连接会话发送频率和时间.#vi /etc/ssh/sshd_config,添加如下两行 # 启用客户端活动检查,每60秒检查一次,3次不活动断开连接 ClientAliveInterval 60ClientAliveCountMax 3 # ClientA

timed out waiting for input: auto-logout

The ssh "timed out waiting for input: auto-logout" messages is generated by ssh upon reaching a auto-logout after an inactivity time specified by the TMOUT environment variable. If this variable is not set your session will not be auto-logged ou

timed out waiting for device to boot,unable to boot the iOS Simulator

平常工作中的关于XCode出现bug解决之后一定要做个记录,要不下次真的再遇到只记得遇到过,怎么解决的是真忘了. COM+R运行xcode之后,模拟器半天没反应,待会会出现这个下面图1这样的弹框,另外timed out waiting for device to boot这句话的一个弹框,不过问题解决了这个图2的弹框没有及时截图,只记得这句话. 出现这种问题大部分都是在升级XCode或者删除XCode再安装新的之后,会遇到的问题,我安装删除循环了N遍,问题还是没解决,网上找了半天也都没有解决,最

DHCP request error:Timed out waiting for dhcpcd to start【转】

本文转载自:http://blog.csdn.net/zvivi521/article/details/9166899 [init.svc.dhcpcd_eth0]: [stopped] I/ServiceManager( 2040): Waiting for service android.tvs.daemon...D/NetUtils( 1412): android_net_utils_runDhcpCommon, result:-1E/EthernetDataTracker( 1412):

Xcode Coule not launch "aaa" press launch failed:timed out waiting for app launch

遇见这个问题 可能是 由于 runapp 的时候设置里面 设置为release了. 解决办法是:见图 build configuration 设置成 debug 状态就OK了. 要是上面的不行就试一下 stackoverflow上面的大神方案: 1.itounes 退出 2.手机 重启 然后 卸了app 3.Xcode 重启  clean 一下 4.重新运行 希望能够帮助 一些人.

(adhoc) process launch failed: timed out waiting for app to launch

I had a similar issue when trying to debug an App with Ad-Hoc provisioning... Check which provisioning you're using, it seems ad-hoc provisioning cannot be used for debugging. EDIT In fact, it seems only development provisioning profiles are Ok for d

kubeadm join 超时报错 error execution phase kubelet-start: error uploading crisocket: timed out waiting for the condition

解决: swapoff -a kubeadm reset systemctl daemon-reload systemctl restart kubelet iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X 原文地址:https://www.cnblogs.com/winstom/p/11840732.html

kubeadm join 超时 uploading crisocket: timed out waiting for the condition

swapoff -a # will turn off the swap kubeadm reset systemctl daemon-reload systemctl restart kubelet iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X # will reset iptables 原文地址:https://www.cnblogs.com/zhaoji