背景:
最近后端基于mongo的项目越来越多,MySQL基于冷备份+binlog可以恢复至任意时间点,那么mongo是否有同样的功能呢?经过调研发现可以通过dump+oplog可以实现粒度更细致的恢复。
官方文档关于oplog的说明 : <a class="mongo-link" href="https://docs.mongodb.com/manual/core/replica-set-oplog/"></a>
以下是实验步骤,附脚本内容。
1、 制作测试数据并且获取冷备
2、模拟线上两个冷备份之间的数据写入
3、备份oplog
4、冷备+oplog恢复至临时实例,确认没问题后,通过临时实例恢复到主库
1、
dump比较重要的几个参数#mongodump --help Usage: mongodump <options> Export the content of a running server into .bson files. Specify a database with -d and a collection with -c to only dump that database or collection. See http://docs.mongodb.org/manual/reference/program/mongodump/ for more information. connection options: -h, --host=<hostname> mongodb host to connect to (setname/host1,host2 for replica sets) --port=<port> server port (can also use --host hostname:port) authentication options: -u, --username=<username> username for authentication -p, --password=<password> password for authentication --authenticationDatabase=<database-name> database that holds the user‘s credentials --authenticationMechanism=<mechanism> authentication mechanism to use --gzip compress archive our collection output with Gzip --oplog use oplog for taking a point-in-time snapshot
test_1234:PRIMARY> show dbs admin 0.000GB local 2.724GB test_1234:PRIMARY> use test switched to db test test_1234:PRIMARY> for (var i=0;i<=5000;i++)(db.test.insert({"test":i})) WriteResult({ "nInserted" : 1 }) test_1234:PRIMARY> db.test.count() 5001 当前test.test有5001条数据,获取冷备份并查看当前时间戳
#date +%s
1553499626
2、 这个时候模拟下冷备之后的数据写入,删除一些数据并记录oplog
test_1234:PRIMARY> use test switched to db test test_1234:PRIMARY> test_1234:PRIMARY> test_1234:PRIMARY> show tables test test_1234:PRIMARY> for (var i=0;i<=5000;i++)(db.after.insert({"after":i})) WriteResult({ "nInserted" : 1 }) test_1234:PRIMARY> db.test.remove({"test":{$gt:1500}}) WriteResult({ "nRemoved" : 3500 }) test_1234:PRIMARY> db.test.count() 1501 test_1234:PRIMARY> db.after.count() 5001 test_1234:PRIMARY>
#date +%s
1553499686
3、备份oplog
./mongoha_dump_oplog.sh options: -S [Start-BSON-Timestamp] -E [End-BSON-Timestamp] is running like mongoha_dump_oplog.sh -S -E #
./mongoha_dump_oplog.sh -S 1553499626 -E 1553499686 2019-03-25T15:42:07.579+0800 writing local.oplog.rs to 2019-03-25T15:42:10.576+0800 local.oplog.rs 0 2019-03-25T15:42:13.575+0800 local.oplog.rs 0 2019-03-25T15:42:16.575+0800 local.oplog.rs 0 2019-03-25T15:42:19.575+0800 local.oplog.rs 0 2019-03-25T15:42:22.575+0800 local.oplog.rs 0 2019-03-25T15:42:25.576+0800 local.oplog.rs 0 2019-03-25T15:42:26.267+0800 local.oplog.rs 8507 2019-03-25T15:42:26.267+0800 done dumping local.oplog.rs (8507 documents) Mongo oplog backup [2019-03-25 15:40:26 - 2019-03-25 15:41:26] -->Sucess
4、 新建临时实例利用刚才冷备+oplog恢复,确认数据没问题之后,恢复至主库。
1)新建临时端口 mongo_install.sh -T 1 -P 6666 -M 1 2019-03-25 15:32:57 start install mongodb-linux-x86_64-rhel62-3.4.0/README mongodb-linux-x86_64-rhel62-3.4.0/THIRD-PARTY-NOTICES mongodb-linux-x86_64-rhel62-3.4.0/MPL-2 mongodb-linux-x86_64-rhel62-3.4.0/GNU-AGPL-3.0 mongodb-linux-x86_64-rhel62-3.4.0/bin/mongodump mongodb-linux-x86_64-rhel62-3.4.0/bin/mongorestore mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoexport mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoimport mongodb-linux-x86_64-rhel62-3.4.0/bin/mongostat mongodb-linux-x86_64-rhel62-3.4.0/bin/mongotop mongodb-linux-x86_64-rhel62-3.4.0/bin/bsondump mongodb-linux-x86_64-rhel62-3.4.0/bin/mongofiles mongodb-linux-x86_64-rhel62-3.4.0/bin/mongooplog mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoreplay mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoperf mongodb-linux-x86_64-rhel62-3.4.0/bin/mongod mongodb-linux-x86_64-rhel62-3.4.0/bin/mongos mongodb-linux-x86_64-rhel62-3.4.0/bin/mongo /usr/local/mongodb is exists .. mongo data,log dir create sucess 2019-03-25 15:32:57 ----> mongodb6666 start sucess note: noprealloc may hurt performance in many applications about to fork child process, waiting until server is ready for connections. forked process: 8394 child process started successfully, parent exiting 2)恢复冷备+oplog
mongoha_fullrestore.sh
options:
-P specify the port
-p specify the restore db path ,must abspath
is running like mongoha_backup.sh -P 27017 -p /test
mongoha_fullrestore.sh -P 6666 -p /data1/backup/mongodb/2019/03/25/_2019-03-25_15\:15/
at 2019-03-25_15:34 to do restore for mongo
2019-03-25T15:34:48.505+0800 preparing collections to restore from
2019-03-25T15:34:48.506+0800 reading metadata for test.test from /data1/backup/mongodb/2019/03/25/_2019-03-25_15:15/test/test.metadata.json
2019-03-25T15:34:48.544+0800 restoring test.test from /data1/backup/mongodb/2019/03/25/_2019-03-25_15:15/test/test.bson
2019-03-25T15:34:48.573+0800 restoring indexes for collection test.test from metadata
2019-03-25T15:34:48.573+0800 finished restoring test.test (5001 documents)
2019-03-25T15:34:48.573+0800 replaying oplog
2019-03-25T15:34:48.573+0800 done
Mongo restore() -->Sucess
冷备恢复成功
临时节点查看下数据
恢复到冷备之前的数据了,但是之前删除的数据及后插入的after document还未恢复,这时候我们恢复oplog
./mongoha_restore_oplog.sh
options:
-P specify the port
-p specify the restore db path ,must abspath
is running like ./mongoha_restore_oplog.sh -P 临时端口 -p /data1/backup/mongodb/oplog/2019/03/25/local/oplog.rs.bson
再次查看临时库数据
临时库数据以恢复,这时候就可以备份出来恢复至线上了。
附mongoha_fullbackup.sh脚本
#!/bin/bash #auth liding@zlongame.com dat=`date "+%Y-%m-%d_%H:%M"` day=`date "+%Y/%m/%d" -d now` #localip=`ip a | grep "\binet\b" |awk ‘{print $2}‘| egrep -e "^10|172" | awk -F"/" ‘{print $1; exit;}‘` basedir="/data1/backup/mongodb/$day" helpfunc(){ echo echo "options:" echo " -P specify the port" echo " is running like mongoha_backup.sh -P 27017 -D test,test1" } while getopts "P:" Option do case $Option in P) ports=$OPTARG;; *) helpfunc; exit 1; ;; esac done dbname=`echo $dbname | sed ‘s/,/ /g‘` dobackup(){ echo "at $dat to do bakcup for mongo$i" mkdir -p "$basedir" #mongodump -u$bacuser -p$pswd --port $i --oplog -o "$basedir"/mongo"$i" if /usr/local/mongodb/bin/mongodump -h 127.0.0.1 --port $ports --oplog -o "$basedir"/"$i"_$dat;then echo "Mongo Backup($i) -->Sucess" else echo "Mongo Backup($i) -->faild" fi } check(){ person=`whoami` if [ "$person" != "root" ]; then echo "the user is now $person, checkout to ‘root‘ to continue" exit 1 fi if [ -z "$ports" ] ;then helpfunc exit 1 fi } cleanbackup(){ rmdate=`date "+%Y/%m/%d" -d "7 days ago"` if [ -d $1 ] ;then rm -rf $1 echo "rm -$1" elif [ -d $basedir/$rmdate ] ;then rm -rf $basedir/$rmdate echo "rm -rf $basedir/$rmdate" else rm -rf $basedir/mongo$1 echo "rm -rf $basedir/mongo$1" exit 1 fi } main(){ check dobackup } main
附mongoha_fullrestore.sh脚本
mongoha_fullrestore.sh #!/bin/bash dat=`date "+%Y-%m-%d_%H:%M"` day=`date "+%Y/%m/%d" -d now` #localip=`ip a | grep "\binet\b" |awk ‘{print $2}‘| egrep -e "^10|172" | awk -F"/" ‘{print $1; exit;}‘` basedir="/data1/backup/mongodb/$day" helpfunc(){ echo echo "options:" echo " -P specify the port" echo " -p specify the restore db path ,must abspath" echo " is running like mongoha_backup.sh -P 27017 -p $backdir/test" } while getopts "P:p:" Option do case $Option in P) ports=$OPTARG;; p) path=$OPTARG;; *) helpfunc; exit 1; ;; esac done #mongorestore -h 127.0.0.1:27017 -d test test/ dorestore(){ echo "at $dat to do restore for mongo${dbname}" if /usr/local/mongodb/bin/mongorestore --drop --oplogReplay -h 127.0.0.1:${ports} $path;then echo "Mongo restore(${dbname}) -->Sucess" else echo "Mongo restore(${dbname}) -->faild" fi } check(){ person=`whoami` if [ "$person" != "root" ]; then echo "the user is now $person, checkout to ‘root‘ to continue" exit 1 fi if [ -z "$ports" ] ;then helpfunc exit 1 fi } main(){ check dorestore } main
附mongoha_dump_oplog.sh 脚本
cat mongoha_dump_oplog.sh #!/bin/bash dat=`date "+%Y-%m-%d_%H:%M"` day=`date "+%Y/%m/%d" -d now` helpfunc(){ echo echo "options:" echo " -S [Start-BSON-Timestamp]" echo " -E [End-BSON-Timestamp] " echo " is running like mongoha_dump_oplog.sh -S -E " } if [ $# -lt 2 ] ; then helpfunc exit 1 else while getopts "S:E:" Option do case $Option in S) start_t=$OPTARG;; E) stop_t=$OPTARG;; *) helpfunc; exit 1; ;; esac done fi start_time="Timestamp(${start_t}, 0)" stop_time="Timestamp(${stop_t}, 0)" date_now=`date "+%Y%m%d%H_%M" -d now` backdir="/data1/backup/mongodb/oplog" #dump function data_fetch(){ mkdir -p $backdir query=‘{ts:{$gt:‘$start_time‘,$lt: ‘$stop_time‘ }}‘ flags=‘-d local -c oplog.rs‘ if /usr/local/mongodb/bin/mongodump $flags --query "$query" -o $backdir/$date_now;then echo "Mongo oplog backup [`date -d @${start_t} "+%Y-%m-%d %H:%M:%S"` - `date -d @${stop_t} "+%Y-%m-%d %H:%M:%S"`] -->Sucess" else echo "Mongo oplog backup [`date -d @${start_t} "+%Y-%m-%d %H:%M:%S"` - `date -d @${stop_t} "+%Y-%m-%d %H:%M:%S"`] -->Faild" fi } main(){ data_fetch } main
附mongoha_restore_oplog.sh 脚本
cat mongoha_restore_oplog.sh #!/bin/bash dat=`date "+%Y-%m-%d_%H:%M"` day=`date "+%Y/%m/%d" -d now` #localip=`ip a | grep "\binet\b" |awk ‘{print $2}‘| egrep -e "^10|172" | awk -F"/" ‘{print $1; exit;}‘` backdir="/data1/backup/mongodb/oplog/$day" helpfunc(){ echo echo "options:" echo " -P specify the port" echo " -p specify the restore db path ,must abspath" echo " is running like $0 -P 临时端口 -p $backdir/local/oplog.rs.bson" } if [ $# -lt 2 ] ; then helpfunc exit 1 else while getopts "P:p:" Option do case $Option in P) port=$OPTARG;; p) path=$OPTARG;; *) helpfunc; exit 1; ;; esac done fi function data_restore(){ flags=‘-d local -c oplog.rs‘ if /usr/local/mongodb/bin/mongorestore --port $port $flags $path;then echo "Mongo restore[$port] -->Sucess" else echo "Mongo restore[$port] -->Faild" fi } data_restore
原文地址:https://www.cnblogs.com/Jason-Born/p/10594254.html