mongo副本集mongos启动报错

背景 

项目采用mongo副本集的形式存储数据，经常出现无故断电导致某个副本mongos启动不起来的问题。

环境介绍

mongodb副本集：
mongo01：192.168.36.218
mongo02：192.168.36.219
mongo03：192.168.36.220

报错信息

mongo03副本上mongos启动不起来，执行命令报错：

[[email protected] ~]# mongos --configdb 192.168.36.218:20000,192.168.36.219:20000,192.168.36.220:20000 --port 30000 --chunkSize 500 --logpath /home/mongo/logs/mongos.log --logappend --fork
about to fork child process, waiting until server is ready for connections.
forked process: 79748
ERROR: child process failed, exited with error number 5

查看mongos.log，错误信息如下：

2018-06-25T09:13:47.607+0800 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-06-25T09:13:47.612+0800 I CONTROL  [main] ** WARNING: You are running this process as the root user, which is not recommended.
2018-06-25T09:13:47.613+0800 I CONTROL  [main]
2018-06-25T09:13:47.613+0800 I SHARDING [mongosMain] MongoS version 3.2.1 starting: pid=80904 port=30000 64-bit host=MongoDB03 (--help for usage)
2018-06-25T09:13:47.613+0800 I CONTROL  [mongosMain] db version v3.2.1
2018-06-25T09:13:47.613+0800 I CONTROL  [mongosMain] git version: a14d55980c2cdc565d4704a7e3ad37e4e535c1b2
2018-06-25T09:13:47.613+0800 I CONTROL  [mongosMain] allocator: tcmalloc
2018-06-25T09:13:47.613+0800 I CONTROL  [mongosMain] modules: none
2018-06-25T09:13:47.613+0800 I CONTROL  [mongosMain] build environment:
2018-06-25T09:13:47.613+0800 I CONTROL  [mongosMain]     distarch: x86_64
2018-06-25T09:13:47.613+0800 I CONTROL  [mongosMain]     target_arch: x86_64
2018-06-25T09:13:47.613+0800 I CONTROL  [mongosMain] options: { net: { port: 30000 }, processManagement: { fork: true }, sharding: { chunkSize: 500, configDB: "192.168.36.218:20000,192.168.36.219:20000,192.168.36.220:20000" }, systemLog: { destination: "file", logAppend: true, path: "/home/mongo/logs/mongos.log" } }
2018-06-25T09:13:47.613+0800 I SHARDING [mongosMain] Updating config server connection string to: 192.168.36.218:20000,192.168.36.219:20000,192.168.36.220:20000
2018-06-25T09:13:47.625+0800 W SHARDING [mongosMain] config servers 192.168.36.218:20000 and 192.168.36.220:20000 differ
2018-06-25T09:13:47.627+0800 W SHARDING [mongosMain] config servers 192.168.36.218:20000 and 192.168.36.220:20000 differ
2018-06-25T09:13:47.628+0800 W SHARDING [mongosMain] config servers 192.168.36.218:20000 and 192.168.36.220:20000 differ
2018-06-25T09:13:47.630+0800 W SHARDING [mongosMain] config servers 192.168.36.218:20000 and 192.168.36.220:20000 differ
2018-06-25T09:13:47.630+0800 E SHARDING [mongosMain] Error initializing sharding system: ConfigServersInconsistent hash from 192.168.36.218:20000: { chunks: "d41d8cd98f00b204e9800998ecf8427e", databases: "95954cb16c029767f4ad050712a28f49", shards: "68f4b37fec8c2ac97cc985aa01f37717", version: "b25e55c19a8c75c87b4f950dcf5eb088" } vs hash from 192.168.36.220:20000: {}

如上我们发现：

config servers 192.168.36.218:20000 and 192.168.36.220:20000 differ

配置服务器192.168.36.218:20000和损坏的配置服务器192.168.36.220:20000不一致，也就是说mongo01:20000和mongo03:20000上面的配置不一样

修复

那我们该如何修复呢？其实我们可以将mongo01:20000上的config库导入到mongo03:20000的config库中，以解决上面的问题。

执行过程

第一步：备份mongo01:20000的config库

[[email protected] ~]#  mongodump --host 192.168.36.218:20000 -d config -o /home/config
2018-06-25T10:06:06.225+0800    writing config.actionlog to
2018-06-25T10:06:06.226+0800    writing config.locks to
2018-06-25T10:06:06.226+0800    writing config.mongos to
2018-06-25T10:06:06.226+0800    writing config.lockpings to
2018-06-25T10:06:06.227+0800    done dumping config.locks (3 documents)
2018-06-25T10:06:06.228+0800    done dumping config.lockpings (2 documents)
2018-06-25T10:06:06.229+0800    done dumping config.mongos (3 documents)
2018-06-25T10:06:06.229+0800    writing config.shards to
2018-06-25T10:06:06.229+0800    writing config.settings to
2018-06-25T10:06:06.229+0800    writing config.version to
2018-06-25T10:06:06.230+0800    done dumping config.shards (1 document)
2018-06-25T10:06:06.230+0800    writing config.databases to
2018-06-25T10:06:06.230+0800    done dumping config.settings (1 document)
2018-06-25T10:06:06.230+0800    done dumping config.version (1 document)
2018-06-25T10:06:06.230+0800    writing config.changelog to
2018-06-25T10:06:06.230+0800    writing config.chunks to
2018-06-25T10:06:06.231+0800    done dumping config.databases (1 document)
2018-06-25T10:06:06.231+0800    writing config.tags to
2018-06-25T10:06:06.232+0800    done dumping config.chunks (0 documents)
2018-06-25T10:06:06.232+0800    done dumping config.changelog (1 document)
2018-06-25T10:06:06.232+0800    done dumping config.tags (0 documents)
2018-06-25T10:06:06.355+0800    done dumping config.actionlog (8160 documents)
[[email protected] ~]#

第二步：导入备份的config到mongo03:20000中

[[email protected] ~]#  mongorestore --host 192.168.36.220:20000 -d config /home/config/config
2018-06-25T10:08:09.136+0800    building a list of collections to restore from /home/config/config dir
2018-06-25T10:08:09.162+0800    reading metadata for config.actionlog from /home/config/config/actionlog.metadata.json
2018-06-25T10:08:09.163+0800    reading metadata for config.locks from /home/config/config/locks.metadata.json
2018-06-25T10:08:09.163+0800    reading metadata for config.changelog from /home/config/config/changelog.metadata.json
2018-06-25T10:08:09.164+0800    reading metadata for config.mongos from /home/config/config/mongos.metadata.json
2018-06-25T10:08:09.164+0800    restoring config.locks from /home/config/config/locks.bson
2018-06-25T10:08:09.165+0800    restoring config.mongos from /home/config/config/mongos.bson
2018-06-25T10:08:09.175+0800    error: multiple errors in bulk operation:
  - E11000 duplicate key error collection: config.mongos index: _id_ dup key: { : "MongoDB01:30000" }
  - E11000 duplicate key error collection: config.mongos index: _id_ dup key: { : "MongoDB02:30000" }

2018-06-25T10:08:09.175+0800    restoring indexes for collection config.mongos from metadata
2018-06-25T10:08:09.204+0800    finished restoring config.mongos (3 documents)
2018-06-25T10:08:09.204+0800    restoring indexes for collection config.locks from metadata
2018-06-25T10:08:09.204+0800    reading metadata for config.lockpings from /home/config/config/lockpings.metadata.json
2018-06-25T10:08:09.205+0800    restoring config.lockpings from /home/config/config/lockpings.bson
2018-06-25T10:08:09.215+0800    restoring config.actionlog from /home/config/config/actionlog.bson
2018-06-25T10:08:09.269+0800    restoring config.changelog from /home/config/config/changelog.bson
2018-06-25T10:08:09.284+0800    restoring indexes for collection config.changelog from metadata
2018-06-25T10:08:09.284+0800    finished restoring config.locks (3 documents)
2018-06-25T10:08:09.284+0800    reading metadata for config.shards from /home/config/config/shards.metadata.json
2018-06-25T10:08:09.284+0800    restoring config.shards from /home/config/config/shards.bson
2018-06-25T10:08:09.286+0800    finished restoring config.changelog (1 document)
2018-06-25T10:08:09.286+0800    reading metadata for config.version from /home/config/config/version.metadata.json
2018-06-25T10:08:09.286+0800    restoring config.version from /home/config/config/version.bson
2018-06-25T10:08:09.287+0800    error: multiple errors in bulk operation:
  - E11000 duplicate key error collection: config.lockpings index: _id_ dup key: { : "MongoDB01:30000:1523346496:1804289383" }
  - E11000 duplicate key error collection: config.lockpings index: _id_ dup key: { : "MongoDB02:30000:1529566939:1804289383" }

2018-06-25T10:08:09.287+0800    restoring indexes for collection config.lockpings from metadata
2018-06-25T10:08:09.297+0800    restoring indexes for collection config.shards from metadata
2018-06-25T10:08:09.318+0800    finished restoring config.lockpings (2 documents)
2018-06-25T10:08:09.318+0800    reading metadata for config.databases from /home/config/config/databases.metadata.json
2018-06-25T10:08:09.318+0800    restoring config.databases from /home/config/config/databases.bson
2018-06-25T10:08:09.330+0800    restoring indexes for collection config.version from metadata
2018-06-25T10:08:09.370+0800    finished restoring config.shards (1 document)
2018-06-25T10:08:09.370+0800    reading metadata for config.settings from /home/config/config/settings.metadata.json
2018-06-25T10:08:09.370+0800    restoring config.settings from /home/config/config/settings.bson
2018-06-25T10:08:09.372+0800    finished restoring config.version (1 document)
2018-06-25T10:08:09.372+0800    reading metadata for config.tags from /home/config/config/tags.metadata.json
2018-06-25T10:08:09.372+0800    restoring config.tags from /home/config/config/tags.bson
2018-06-25T10:08:09.374+0800    restoring indexes for collection config.tags from metadata
2018-06-25T10:08:09.382+0800    restoring indexes for collection config.databases from metadata
2018-06-25T10:08:09.400+0800    finished restoring config.tags (0 documents)
2018-06-25T10:08:09.400+0800    reading metadata for config.chunks from /home/config/config/chunks.metadata.json
2018-06-25T10:08:09.400+0800    restoring config.chunks from /home/config/config/chunks.bson
2018-06-25T10:08:09.409+0800    restoring indexes for collection config.settings from metadata
2018-06-25T10:08:09.409+0800    finished restoring config.databases (1 document)
2018-06-25T10:08:09.411+0800    finished restoring config.settings (1 document)
2018-06-25T10:08:09.555+0800    restoring indexes for collection config.chunks from metadata
2018-06-25T10:08:09.581+0800    finished restoring config.chunks (0 documents)
2018-06-25T10:08:09.754+0800    restoring indexes for collection config.actionlog from metadata
2018-06-25T10:08:09.754+0800    finished restoring config.actionlog (8160 documents)
2018-06-25T10:08:09.754+0800    done
[[email protected] ~]#

注：我是在生产环境中执行的，当时把mongo01:20000和mongo02:20000上面的config库全部备份了一下，然后按顺序全部导入到了mongo03:20000中(备份和导入命令同上)。按理说应该只导入mongo01:20000就可以了，由于是生产环境，我就不加以测试了，等到下次再出现这种问题再试试吧。同时也欢迎大家测试一下，然后回复到评论区，帮助大家解决一下问题。

尝试启动mongo03上的mongos

[[email protected] ~]#  mongos --configdb 192.168.36.218:20000,192.168.36.219:20000,192.168.36.220:20000 --port 30000 --chunkSize 500 --logpath /home/mongo/logs/mongos.log --logappend --fork
about to fork child process, waiting until server is ready for connections.
forked process: 4286
child process started successfully, parent exiting
[[email protected] ~]# ps -ef | grep mongos
root     44760     1  0 Jun21 ?        00:25:14 mongos --configdb 192.168.36.218:20000,192.168.36.219:20000,192.168.36.220:20000 --port 30000 --chunkSize 500 --logpath /home/mongo/logs/mongos.log --logappend --fork
root     66128 66090  0 11:40 pts/0    00:00:00 grep mongos
[[email protected] ~]#

如上，我们可以看到mongos成功启动并在后台保持运行。

原文地址：http://blog.51cto.com/10074802/2132394

时间： 2024-11-08 05:15:19

mongo副本集mongos启动报错的相关文章

zookeeper集群启动报错：Cannot open channel to * at election address /ip:3888

zookeeper集群启动报错:Cannot open channel to * at election address /ip:3888 2018年04月06日 20:52:55 中单大魔王 阅读数:729 标签: zookeeper 下面几点需要注意的: 1.确认在每个$zookeeper_home/data/myid中有对应数字 2.是否关闭防火墙:systemctl stop firewalld,systemctl disable firewalld 3.zoo.cfg中的server需

mongo副本集原理，搭建

#进入mongo mongo   默认端口27017启动的mongo服务,直接进入 mongo --port=27027   指定端口27027启动mongo服务,需加端口号启动 mongo 103.56.195.5:27017   登录指定主机的mongo服务,可以以此测试对端防火墙是否放通,mongo服务器端口 #mongo副本集启动参数: [[email protected] db]# ps aux |grep mongo root     112542  0.7  0.1 1058417

安装mongodb后启动报错libstdc++

安装mongo后启动报错如下图 显然说是libstdc++.so文件版本的问题,这种一般都是gcc版本太低了 接着查询gcc的版本    strings /usr/lib/libstdc++.so.6 | grep GLIBCXX 接着去找下libstdc++.so新点的版本,一定要根据自己的系统版本去找啊,由于我找错了版本报错系统位数不对 随后下载了一个64位的libstdc++.so.6.0.17的文件,然后软链成libstdc++.so.6 ln -s libstdc++.so.6.0.1

我们的一个已投产项目的高可用数据库实战 - mongo 副本集的搭建详细过程

我们的 mongo 副本集有三台 mongo 服务器:一台主库两台从库.主库进行写操作,两台从库进行读操作(至于某次读操作究竟路由给了哪台,仲裁决定),实现了读写分离.这还不止,如果主库宕掉,还能实现不需要用户干预的情况下,将主库自动切换到另外两台从库中的某一台,真正实现了 db 的高可用. 1. 背景 1.1 环境 CPU核数:4 内存配置:8G 带宽:100MB 磁盘:系统盘 40G,数据盘 180G 操作系统版本:Ubuntu 14.04 64位 1.2 系统部署结构图 2. MongoD

【原创】大叔问题定位分享（29）datanode启动报错：50020端口被占用

集群中有一台datanode一直启动报错如下: java.net.BindException: Problem binding to [$server1:50020] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException 查看端口是否被占用 # netstat -tnlp|grep 50020 发现没有进程在监听50020端

rabbimq集群搭建报错：Error: unable TO perform an operation ON node '[email protected]'. Please see diagnostics information AND suggestions below.

在搭建rabbitmq集群的时候,添加内存节点时,抛出异常:Error: unable TO perform an operation ON node '[email protected]'. Please see diagnostics information AND suggestions below. 首先,假设当前服务器为A,我们在A服务器执行rabbitmqctl join_cluster [email protected] --ram,就是说我们要将A服务器作为内存节点加入到B中,然

tomcat启动报错：java.net.BindException: Permission denied <null>:80

1,启动报错显示 [org.springframework.web.servlet.DispatcherServlet]FrameworkServlet 'springMvc': initialization completed in 382 ms Jun 01, 2015 6:39:06 PM org.apache.coyote.http11.Http11Protocol start SEVERE: Error starting endpoint java.net.BindException:

Tomcat启动报错[org.apache.struts2.dispatcher.Dispatcher]Dispatcher initialization failed

Tomcat启动报错: [org.apache.struts2.dispatcher.Dispatcher]Dispatcher initialization failed Unable to load configuration. - bean - jar:file:/E:/SoftwareDevelopment/Software/Eclipse4.3/eclipse-jee-kepler-SR1-win32-x86_64/Workspace/.metadata/.plugins/org.ec

zabbix启动报错:Connection to database 'xxx' failed解决方法

Zabbix 分布式系统监视系统 zabbix是一个基于WEB界面的提供分布式系统监视以及网络监视功能的企业级的开源解决方案. zabbix能监视各种网络参数,保证服务器系统的安全运营:并提供灵活的通知机制以让系统管理员快速定位/解决存在的各种问题. 本文讲解的是zabbix无法启动报错:Connection to database 'xxx' failed: [1045] Access denied for user 'xxx'@'localhost' (using password: NO)