OpenStack报错:MessagingTimeout: Timed out waiting for a reply to message ID

L3.agent中出现大量消息超时错误,对网络的操作各种异常。

报错如下:

2018-09-24 05:54:59.886 15110 ERROR neutron.agent.l3.agent [req-db9207e6-9270-4f23-8c19-0d91d20cc6fb ] Failed synchronizing routers due to RPC error
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 593, in fetch_and_sync_all_routers
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent     routers = self.plugin_rpc.get_routers(context)
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 93, in get_routers
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent     router_ids=router_ids)
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent     retry=self.retry)
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent     timeout=timeout, retry=retry)
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent     retry=retry)
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent     result = self._waiter.wait(msg_id, timeout)
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent     message = self.waiters.get(msg_id, timeout=timeout)
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent     ‘to message ID %s‘ % msg_id)
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent MessagingTimeout: Timed out waiting for a reply to message ID d4baae114cee4f6d831c5eec3c5f0de3
2018-09-24 05:54:59.886 15110 TRACE neutron.agent.l3.agent

所有超时都指向同步路由的操作。 而且同步失败时,rabbit中的队列q-l3-plugin中有大量未应答消息积压,该队列为同步路由时使用,路由同步时会使用消息队列传送所有路由的属性详情,消息量很大。

1)测试是否由于消息太大导致,编写测试代码,尝试连续1000次发送该消息,并未出现丢失消息的情况。

2)尝试减少路由器数量,短时内情况有所改善,但是随时间增加,消息积压依然有更加严重的趋势。

最终跟踪neutron代码,发现消息队列出现Timeout的原因是: neutron在同步路由信息时,会从neutron-server获取所有router的信息,这个过程会比较长(130s左右,和网络资源的多少有关系),而 在/etc/neutron/neutron.conf中会有一个配置项“rpc_response_timeout”,它用来配置RPC的超时时间,默认为60s,所以导致超时异常.

解决方法:

vi /etc/neutron/neutron.conf
rpc_response_timeout=180

问题解决。

原文地址:https://www.cnblogs.com/lijinze-tsinghua/p/9696950.html

时间: 2024-10-09 00:15:33

OpenStack报错:MessagingTimeout: Timed out waiting for a reply to message ID的相关文章

MessagingTimeout: Timed out waiting for a reply to message ID

l3中出现大量消息超时错误,对网络的操作各种异常. 报错如下: 2016-02-25 05:54:59.886 15110 ERROR neutron.agent.l3.agent [req-db9207e6-9270-4f23-8c19-0d91d20cc6fb ] Failed synchronizing routers due to RPC error2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent Traceback (

阿里云安装openstack报错

在CentOS 7下使用DevStack一键安装包安装OpenStack,突然来个报错,信息如下: 报错一. 解决方法:修改hosts文件,同步主机名 vim /etc/hosts 127.0.0.1 openstack 报错二. ERROR : Error appeared during Puppet run: 192.168.1.201_mariadb.ppError: Execution of '/usr/bin/yum -d 0 -e 0 -y install mariadb' retu

【求解】WinPhone模拟器报错:模拟器没法确定来宾虚拟机通信的主机ID地址。某些功能已被禁用【未解决】

先看警告 再看错误信息 计算机管理打不开就==>Win+R ==>compmgmt.msc 发现,dnt在管理员权限组里面,也在Hyper-V权限组里面 打开管理工具 重启 [求解]WinPhone模拟器报错:模拟器没法确定来宾虚拟机通信的主机ID地址.某些功能已被禁用[未解决][求解]WinPhone模拟器报错:模拟器没法确定来宾虚拟机通信的主机ID地址.某些功能已被禁用[未解决][求解]WinPhone模拟器报错:模拟器没法确定来宾虚拟机通信的主机ID地址.某些功能已被禁用[未解决][求解

openstack 报错处理

自己捣鼓openstack遇到的报错 openstack 1,建立rdo-release.repo vim /etc/yum.repos.d/rdo-release.repo [openstack-icehouse] name=OpenStack Icehouse Repository baseurl=https://repos.fedorapeople.org/openstack/EOL/openstack-icehouse/epel-6/ enabled=1 skip_if_unavaila

解决hue报错:timed out (code THRIFTSOCKET): None

报错栈: [15/Jun/2017 19:25:15 +0800] decorators ERROR error running <function execute at 0x7fba2804ecf8> Traceback (most recent call last): File "/home/work/hue-3.10.0/desktop/libs/notebook/src/notebook/decorators.py", line 81, in decorator r

xdebug 一直报错 upstream timed out (110: Connection timed out) while reading response header from upstream

本地主机(Windows环境192.168.66.1)访问虚拟机(192.168.66.139)里面的搭建的php环境(系统centos6.5版本,php版本是5.5.30 ,xdebug 2.4.0),通过命令行pecl install xdebug安装的xdebug, 在php.ini配置xdebug [Xdebug]zend_extension=/usr/local/php/lib/php/extensions/no-debug-non-zts-20121212/xdebug.soxdeb

hue报错StructuredException: timed out (code THRIFTSOCKET): None的处理

通过hue的web界面进行hive的sql查询,无法显示结果并报错timeout 报错如下:[28/Jul/2017 11:23:29 +0800] decorators ERROR error running <function execute at 0x7fa741ddc8c0>Traceback (most recent call last): File "/home/hadoop/.versions/hue-3.10.0/desktop/libs/notebook/src/n

nginx proxy超时报错 upstream timed out (110: Connec...

环境介绍 服务器:centos6.4服务:nginx proxy 问题描述: 然后查找  /opt/usr/nginx/1.4.0/logs  错误 error.log日志提示如下 2015/01/04 15:44:13 [error] 10112#0: *994662 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 117.32.232.172

nginx error_log报错upstream timed out (110: Connection timed out)

最近迁移服务器,在网站搬到新服务器第一天就报504错误,服务器及ip都能ping通,查看错误日志,发现upstream timed out (110: Connection timed out)的错误,上网百度了下,原因应该是nginx配置问题 打开/etc/php5/php-fpm/pool.d/www.conf 将pm.max_children修改为40 request_terminate_time修改为900 再修改php.ini max_execution_time = 30,把它修改得