Mongodb基于oplog恢复至任意时间

背景:

  最近后端基于mongo的项目越来越多,MySQL基于冷备份+binlog可以恢复至任意时间点,那么mongo是否有同样的功能呢?经过调研发现可以通过dump+oplog可以实现粒度更细致的恢复。

官方文档关于oplog的说明 : <a class="mongo-link" href="https://docs.mongodb.com/manual/core/replica-set-oplog/"></a>

  以下是实验步骤,附脚本内容。

1、 制作测试数据并且获取冷备

2、模拟线上两个冷备份之间的数据写入

3、备份oplog

4、冷备+oplog恢复至临时实例,确认没问题后,通过临时实例恢复到主库

1、

dump比较重要的几个参数#mongodump --help
Usage:
  mongodump <options>

Export the content of a running server into .bson files.

Specify a database with -d and a collection with -c to only dump that database or collection.

See http://docs.mongodb.org/manual/reference/program/mongodump/ for more information.

connection options:
  -h, --host=<hostname>                                     mongodb host to connect to (setname/host1,host2 for replica sets)
      --port=<port>                                         server port (can also use --host hostname:port)

authentication options:
  -u, --username=<username>                                 username for authentication
  -p, --password=<password>                                 password for authentication
      --authenticationDatabase=<database-name>              database that holds the user‘s credentials
      --authenticationMechanism=<mechanism>                 authentication mechanism to use

      --gzip                                                compress archive our collection output with Gzip
      --oplog                                               use oplog for taking a point-in-time snapshot

  

test_1234:PRIMARY> show dbs
admin  0.000GB
local  2.724GB
test_1234:PRIMARY> use test
switched to db test
test_1234:PRIMARY> for (var i=0;i<=5000;i++)(db.test.insert({"test":i}))
WriteResult({ "nInserted" : 1 })
test_1234:PRIMARY> db.test.count()
5001
当前test.test有5001条数据,获取冷备份并查看当前时间戳

#date +%s
1553499626


2、 这个时候模拟下冷备之后的数据写入,删除一些数据并记录oplog

test_1234:PRIMARY> use test
switched to db test
test_1234:PRIMARY>
test_1234:PRIMARY>
test_1234:PRIMARY> show tables
test
test_1234:PRIMARY> for (var i=0;i<=5000;i++)(db.after.insert({"after":i}))
WriteResult({ "nInserted" : 1 })
test_1234:PRIMARY> db.test.remove({"test":{$gt:1500}})
WriteResult({ "nRemoved" : 3500 })
test_1234:PRIMARY> db.test.count()
1501
test_1234:PRIMARY> db.after.count()
5001
test_1234:PRIMARY> 

#date +%s
1553499686

  

3、备份oplog

./mongoha_dump_oplog.sh 

options:
 -S      [Start-BSON-Timestamp]
 -E      [End-BSON-Timestamp]
 is running like mongoha_dump_oplog.sh -S  -E 

 #

 

./mongoha_dump_oplog.sh -S 1553499626 -E 1553499686
2019-03-25T15:42:07.579+0800 writing local.oplog.rs to
2019-03-25T15:42:10.576+0800 local.oplog.rs 0
2019-03-25T15:42:13.575+0800 local.oplog.rs 0
2019-03-25T15:42:16.575+0800 local.oplog.rs 0
2019-03-25T15:42:19.575+0800 local.oplog.rs 0
2019-03-25T15:42:22.575+0800 local.oplog.rs 0
2019-03-25T15:42:25.576+0800 local.oplog.rs 0
2019-03-25T15:42:26.267+0800 local.oplog.rs 8507
2019-03-25T15:42:26.267+0800 done dumping local.oplog.rs (8507 documents)
Mongo oplog backup [2019-03-25 15:40:26 - 2019-03-25 15:41:26] -->Sucess

  

4、 新建临时实例利用刚才冷备+oplog恢复,确认数据没问题之后,恢复至主库。

1)新建临时端口
mongo_install.sh -T 1 -P 6666 -M 1
 2019-03-25 15:32:57 start install
mongodb-linux-x86_64-rhel62-3.4.0/README
mongodb-linux-x86_64-rhel62-3.4.0/THIRD-PARTY-NOTICES
mongodb-linux-x86_64-rhel62-3.4.0/MPL-2
mongodb-linux-x86_64-rhel62-3.4.0/GNU-AGPL-3.0
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongodump
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongorestore
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoexport
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoimport
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongostat
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongotop
mongodb-linux-x86_64-rhel62-3.4.0/bin/bsondump
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongofiles
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongooplog
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoreplay
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoperf
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongod
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongos
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongo
/usr/local/mongodb is exists ..
mongo data,log dir create sucess

2019-03-25 15:32:57 ----> mongodb6666 start sucess
note: noprealloc may hurt performance in many applications
about to fork child process, waiting until server is ready for connections.
forked process: 8394
child process started successfully, parent exiting

2)恢复冷备+oplog

mongoha_fullrestore.sh

options:
-P specify the port
-p specify the restore db path ,must abspath
is running like mongoha_backup.sh -P 27017 -p /test

mongoha_fullrestore.sh -P 6666 -p /data1/backup/mongodb/2019/03/25/_2019-03-25_15\:15/
at 2019-03-25_15:34 to do restore for mongo
2019-03-25T15:34:48.505+0800 preparing collections to restore from
2019-03-25T15:34:48.506+0800 reading metadata for test.test from /data1/backup/mongodb/2019/03/25/_2019-03-25_15:15/test/test.metadata.json
2019-03-25T15:34:48.544+0800 restoring test.test from /data1/backup/mongodb/2019/03/25/_2019-03-25_15:15/test/test.bson
2019-03-25T15:34:48.573+0800 restoring indexes for collection test.test from metadata
2019-03-25T15:34:48.573+0800 finished restoring test.test (5001 documents)
2019-03-25T15:34:48.573+0800 replaying oplog
2019-03-25T15:34:48.573+0800 done
Mongo restore() -->Sucess

冷备恢复成功

临时节点查看下数据

恢复到冷备之前的数据了,但是之前删除的数据及后插入的after document还未恢复,这时候我们恢复oplog

./mongoha_restore_oplog.sh

options:
-P specify the port
-p specify the restore db path ,must abspath
is running like ./mongoha_restore_oplog.sh -P 临时端口 -p /data1/backup/mongodb/oplog/2019/03/25/local/oplog.rs.bson

再次查看临时库数据

临时库数据以恢复,这时候就可以备份出来恢复至线上了。

附mongoha_fullbackup.sh脚本

 #!/bin/bash
#auth liding@zlongame.com

dat=`date "+%Y-%m-%d_%H:%M"`
day=`date "+%Y/%m/%d" -d now`
#localip=`ip a | grep "\binet\b" |awk ‘{print $2}‘| egrep -e "^10|172" | awk -F"/" ‘{print $1; exit;}‘`
basedir="/data1/backup/mongodb/$day"

helpfunc(){
        echo
        echo "options:"
        echo " -P      specify the port"
        echo " is running like mongoha_backup.sh -P 27017 -D test,test1"
}

while getopts "P:" Option
do
        case $Option in
                P) ports=$OPTARG;;
                *) helpfunc; exit 1; ;;
        esac
done

dbname=`echo $dbname | sed ‘s/,/ /g‘`
dobackup(){
        echo "at $dat to do bakcup for mongo$i"
        mkdir -p "$basedir"
        #mongodump -u$bacuser -p$pswd --port $i --oplog  -o "$basedir"/mongo"$i"
        if /usr/local/mongodb/bin/mongodump -h 127.0.0.1 --port $ports   --oplog  -o "$basedir"/"$i"_$dat;then
                echo "Mongo Backup($i) -->Sucess"
        else
                echo "Mongo Backup($i) -->faild"
        fi

}

check(){
person=`whoami`
        if [ "$person" != "root" ]; then
                echo "the user is now $person, checkout to ‘root‘ to continue"
                exit 1
        fi

        if [ -z "$ports" ] ;then
                helpfunc
                exit 1
        fi

}

cleanbackup(){
rmdate=`date "+%Y/%m/%d" -d "7 days ago"`
if [ -d $1 ] ;then
        rm -rf $1
        echo "rm -$1"
elif [ -d $basedir/$rmdate ] ;then
        rm -rf $basedir/$rmdate
        echo "rm -rf $basedir/$rmdate"
else
        rm -rf $basedir/mongo$1
        echo "rm -rf $basedir/mongo$1"
        exit 1
fi
}
main(){

check
dobackup
}
main

  

附mongoha_fullrestore.sh脚本

mongoha_fullrestore.sh
#!/bin/bash

dat=`date "+%Y-%m-%d_%H:%M"`
day=`date "+%Y/%m/%d" -d now`
#localip=`ip a | grep "\binet\b" |awk ‘{print $2}‘| egrep -e "^10|172" | awk -F"/" ‘{print $1; exit;}‘`
basedir="/data1/backup/mongodb/$day"

helpfunc(){
        echo
        echo "options:"
        echo " -P      specify the port"
        echo " -p      specify the restore db path ,must abspath"
        echo " is running like mongoha_backup.sh -P 27017  -p $backdir/test"
}

while getopts "P:p:" Option
do
        case $Option in
                P) ports=$OPTARG;;
                p) path=$OPTARG;;
                *) helpfunc; exit 1; ;;
        esac
done

#mongorestore -h 127.0.0.1:27017 -d test test/
dorestore(){
        echo "at $dat to do restore for mongo${dbname}"
        if /usr/local/mongodb/bin/mongorestore --drop --oplogReplay -h 127.0.0.1:${ports} $path;then
                echo "Mongo restore(${dbname}) -->Sucess"
        else
                echo "Mongo restore(${dbname}) -->faild"
        fi

}

check(){
person=`whoami`
        if [ "$person" != "root" ]; then
                echo "the user is now $person, checkout to ‘root‘ to continue"
                exit 1
        fi

        if [ -z "$ports" ] ;then
                helpfunc
                exit 1
        fi

}

main(){

check
dorestore
}
main

附mongoha_dump_oplog.sh 脚本

cat mongoha_dump_oplog.sh
#!/bin/bash

dat=`date "+%Y-%m-%d_%H:%M"`
day=`date "+%Y/%m/%d" -d now`

helpfunc(){
        echo
        echo "options:"
        echo " -S      [Start-BSON-Timestamp]"
        echo " -E      [End-BSON-Timestamp] "
        echo " is running like mongoha_dump_oplog.sh -S  -E "
}
if [ $# -lt 2 ] ;
    then
        helpfunc
         exit 1
else

    while getopts "S:E:" Option
    do
            case $Option in
                    S) start_t=$OPTARG;;
                    E) stop_t=$OPTARG;;
                    *) helpfunc; exit 1; ;;
            esac
    done
fi

start_time="Timestamp(${start_t}, 0)"
stop_time="Timestamp(${stop_t}, 0)"
date_now=`date "+%Y%m%d%H_%M" -d now`
backdir="/data1/backup/mongodb/oplog"
#dump
function data_fetch(){
mkdir -p $backdir
query=‘{ts:{$gt:‘$start_time‘,$lt: ‘$stop_time‘ }}‘
flags=‘-d local -c oplog.rs‘
if /usr/local/mongodb/bin/mongodump $flags --query "$query" -o $backdir/$date_now;then
    echo "Mongo oplog backup [`date -d @${start_t} "+%Y-%m-%d %H:%M:%S"` - `date -d @${stop_t} "+%Y-%m-%d %H:%M:%S"`] -->Sucess"
else
    echo "Mongo oplog backup [`date -d @${start_t} "+%Y-%m-%d %H:%M:%S"` - `date -d @${stop_t} "+%Y-%m-%d %H:%M:%S"`] -->Faild"
fi

}

main(){
data_fetch
}
main

   附mongoha_restore_oplog.sh 脚本

cat mongoha_restore_oplog.sh
#!/bin/bash

dat=`date "+%Y-%m-%d_%H:%M"`
day=`date "+%Y/%m/%d" -d now`
#localip=`ip a | grep "\binet\b" |awk ‘{print $2}‘| egrep -e "^10|172" | awk -F"/" ‘{print $1; exit;}‘`
backdir="/data1/backup/mongodb/oplog/$day"

helpfunc(){
        echo
        echo "options:"
        echo " -P      specify the port"
        echo " -p      specify the restore db path ,must abspath"
        echo " is running like $0 -P 临时端口  -p $backdir/local/oplog.rs.bson"
}

if [ $# -lt 2 ] ;
    then
        helpfunc
         exit 1
else
    while getopts "P:p:" Option
    do
            case $Option in
                    P) port=$OPTARG;;
                    p) path=$OPTARG;;
                    *) helpfunc; exit 1; ;;
            esac
    done
fi

function data_restore(){
flags=‘-d local -c oplog.rs‘
if /usr/local/mongodb/bin/mongorestore --port $port $flags $path;then
    echo "Mongo restore[$port] -->Sucess"
else
    echo "Mongo restore[$port] -->Faild"
fi

}
data_restore

  

原文地址:https://www.cnblogs.com/Jason-Born/p/10594254.html

时间: 2024-10-30 09:51:03

Mongodb基于oplog恢复至任意时间的相关文章

Mongodb备份恢复到任意时间点

描述: 根据技术反馈,上午有人误操作删除了线上mongodb表数据,用户已反馈到客服,现在需要给用户恢复数据. 首先要确定被删除数据的数据库架构是什么模式 1.1.单点:是否有备份,有备份情况下,启动一个实例将,将备份数据导入到新实例查找新实例是否有被删除掉的数据,使用mongodump导出数据,再导入到源被删除数据实例,恢复数据.详细见本章下边内容:2.2备份和 2.5恢复 单节点mongodb没有oplog的概念,如果没有备份,数据就会出现丢失,正式环境尽量使用复制集架构. 1.2.复制集:

基于FPGA的DDS任意波形发生器设计

一.简介 DDS技术最初是作为频率合成技术提出的,由于其易于控制,相位连续,输出频率稳定度高,分辨率高, 频率转换速度快等优点,现在被广泛应用于任意波形发生器(AWG).基于DDS技术的任意波形发生器用高速存储器作为查找表,通过高速D/A转换器来合成出存储在存储器内的波形.所以它不仅能产生正弦.余弦.方波.三角波和锯齿波等常见波形,而且还可以利用各种编辑手段,产生传统函数发生器所不能产生的真正意义上的任意波形. 二.原理 根据傅立叶变换定理可知,任何周期信号都可以分解为一系列正弦或余弦信号之和,

dataguard 归档丢失(主库中无此丢失归档处理),备库基于SCN恢复

dataguard 归档丢失(主库中无此丢失归档处理),备库基于SCN恢复 环境: OS: CentOS 6.5 DB: Oracle 10.2.0.5 1.主备库环境 主库: SQL> select dbid,name,LOG_MODE,open_mode,db_unique_name,DATABASE_ROLE,PROTECTION_MODE from v$database; DBID NAME LOG_MODE OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE

C#- 将秒数转化成任意时间格式

将秒数转化成任意时间格式,可以使用C#的一个函数TimeSpan,看示例: TimeSpan ts = new TimeSpan(0, 0, 3661); richTextBox2.Text = ts.Hours + "小时" + ts.Minutes + "分钟" + ts.Seconds + "秒"; 也可以使用传统的方法,看示例:   int TotleTime=3661;//秒         int hour;         int

MongoDB数据库备份恢复与导入导出

一.mongodump/mongorestore方式 使用场景:数据库导出指定collection,无法手工修改导出文件(二进制)允许条件:数据库原始collection导入操作前可以被删除(处理方式:插入)或者保留(处理方式:删除然后插入)导出数据格式:二进制类型,不可手工修改 1.备份数据库指定collection C:\Users\Administrator>mongodump -d webdb -c users -o e:\webdb_users_dumpconnected to: 12

MongoDB误删表恢复

一.场景描述 公司某工程师执行db.giveget_card.drop(),误将线上表删除. 幸好每天都有做备份,这个时候就体现了备份的重要性了,哈哈哈... 二.模拟故障过程: 备份数据大小: rs_test01:PRIMARY> use ycsb switched to db ycsb rs_test01:PRIMARY> db.giveget_card.count(); 3173391 删除之前,此表有更新. rs_test01:PRIMARY> db.giveget_card.i

MongoDB 备份(mongodump)恢复(mongorerstore) 导出 (Mongoexport) 导入( Mongoimport)

MongoDB 备份(mongodump) 在Mongodb中我们使用mongodump命令来备份MongoDB数据.该命令可以导出所有数据到指定目录中. mongodump命令可以通过参数指定导出的数据量级转存的服务器. 语法 mongodump命令脚本语法如下: mongodump -h dbhost -d dbname -o dbdirectory -h: MongDB所在服务器地址,例如:127.0.0.1,当然也可以指定端口号:127.0.0.1:27017 -d: 需要备份的数据库实

MongoDB基于时间戳的导数操作

上班有个需求,协助提取某数据系统中cashloanDb 涉及MongoDB集合: cardBill(时间段 6.1-8.28) , cardReport (时间段 2.1-8.28)如果不支持根据时间提取,可提取全量数据.使用mongo客户端命令进入数据库:jsfkrs0:PRIMARY> show dbs;2018-09-04T17:35:57.692+0800 E QUERY [thread1] Error: listDatabases failed:{"ok" : 0,&q

MongoDB备份和恢复

前面两篇文章介绍了mongodb的搭建与使用:http://msiyuetian.blog.51cto.com/8637744/1720559 以及mongodb的副本集部署:http://msiyuetian.blog.51cto.com/8637744/1722406 下面我们来介绍mongodb的备份和恢复 一.MongoDB备份 1.准备工作 1)创建库.集合 > use mydb