MySQL备份可能遇到的坑

MySQL备份工具,支持各种参数选项,使用不同的选项极有可能影响备份处理过程。本文使用我们常规认为合理的备份参数,测试/验证是否存在容易忽视的坑

# 常规备份参数
# mysqldump
shell> mysqldump --single-transaction --master-data=2 -B replcrash >dbname_dump_serverid_`date +%Y%m%d`.sql

# mysqlpump
shell> mysqlpump --single-transaction -B replcrash >dbname_pump_serverid_`date +%Y%m%d`.sql

# mydumper
shell> mydumper -B replcrash -o /data/backup/mydumper

# XtraBackup
# backup
shell> innobackupex [--defaults-file=MY.CNF] BACKUP-ROOT-DIR
# apply-log
shell> innobackupex --apply-log [--defaults-file=MY.CNF] BACKUP-DIR
# copy-back
shell> innobackupex --copy-back [--defaults-file=MY.CNF] BACKUP-DIR

常规备份参数

ROLE HOSTNAME BASEDIR DATADIR IP PORT
Master ZST1 /usr/local/mysql /data/mysql/mysql3306/data 192.168.85.132 3306
Slave ZST1 /usr/local/mysql /data/mysql/mysql3308/data 192.168.85.132 3308

官方社区版MySQL 5.7.19 基于Row+Position搭建的一主一从异步复制结构:Master->{Slave}。每次使用备份文件还原数据库后,重新搭建这个复制结构
备份工具版本:mysqldump、mysqlpump是MySQL 5.7.19中自带的;mydumper version 0.9.3、innobackupex version 2.4.8

一、mysqldump

1.1、DML操作对备份的影响

创建两张测试表

# 创建两张测试表(192.168.85.132,3306)
use replcrash;
create table py_user_innodb(
uid int not null auto_increment,
name varchar(32),
add_time datetime default current_timestamp,
server_id varchar(10),
primary key(uid),
key(name)
)engine=innodb;

create table py_user_myisam(
uid int not null auto_increment,
name varchar(32),
add_time datetime default current_timestamp,
server_id varchar(10),
primary key(uid),
key(name)
)engine=myisam;

运行下面的脚本持续往测试表中写入数据

#!/user/bin/python
import string
import random
import MySQLdb
import time

conn = MySQLdb.connect(host=‘192.168.85.132‘,
                      port=3306,
                      user=‘mydba‘,
                      passwd=‘mysql5719‘,
                      db=‘replcrash‘)
"""
create table py_user(
uid int not null auto_increment,
name varchar(32),
add_time datetime default current_timestamp,
server_id varchar(10),
primary key(uid),
key(name)
);
"""
while True:
  r_name = ‘‘.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(random.randint(20,30)))
  print r_name

  cursor = conn.cursor()
  cursor.execute("insert into py_user_myisam(name,add_time,server_id) values(‘%s‘,now(),@@server_id);" % str(r_name))
  cursor.execute("insert into py_user_innodb(name,add_time,server_id) values(‘%s‘,now(),@@server_id);" % str(r_name))
  conn.commit()
  time.sleep(0.001)

Python DML

开启general_log,用来查看mysqldump执行过程

# 开启general_log
[email protected]192.168.85.132,3306 [replcrash]> set global general_log_file=‘/data/mysql/mysql3306/data/mysql-general.log‘;
[email protected]192.168.85.132,3306 [replcrash]> set global general_log=1;
# 清空general_log
[[email protected] logs]# cat /dev/null > /data/mysql/mysql3306/data/mysql-general.log

# 备份replcrash数据库
[[email protected] backup]# mysqldump -h127.0.0.1 -P3306 -uroot -p --single-transaction --master-data=2 replcrash >/data/backup/replcrash_dump_1323306_`date +%Y%m%d`.sql

使用备份文件搭建复制

# 还原实例清空GTID信息
[email protected]192.168.85.132,3308 [replcrash]> reset master;
# 还原数据
[[email protected] backup]# mysql -h127.0.0.1 -P3308 -uroot -p replcrash </data/backup/replcrash_dump_1323306_`date +%Y%m%d`.sql

# 搭建复制
[email protected]192.168.85.132,3308 [replcrash]> change master to
master_host=‘192.168.85.132‘,
master_port=3306,
master_user=‘repl‘,
master_password=‘repl‘,
master_auto_position=1;

# 启动复制,查看复制状态
[email protected]192.168.85.132,3308 [replcrash]> start slave;
[email protected]192.168.85.132,3308 [replcrash]> show slave status\G
*************************** 1. row ***************************
              Master_Log_File: mysql-bin.000183
          Read_Master_Log_Pos: 1541377
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 741
        Relay_Master_Log_File: mysql-bin.000183
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
          Exec_Master_Log_Pos: 1042768
               Last_SQL_Errno: 1062
               Last_SQL_Error: Could not execute Write_rows event on table replcrash.py_user_myisam; Duplicate entry ‘332‘ for key ‘PRIMARY‘, Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event‘‘s master log mysql-bin.000183, end_log_pos 1043062
           Retrieved_Gtid_Set: 8ab82362-9c37-11e7-a858-000c29c1025c:251874-253268
            Executed_Gtid_Set: 8ab82362-9c37-11e7-a858-000c29c1025c:1-251874
                Auto_Position: 1

从上面的结果中可以看到,主键冲突了,在从库查询一下这个表中大于等于冲突key的数据

# 查询从库出错表大于等于冲突key的数据
mydba@192.168.85.132,3308 [replcrash]> select * from replcrash.py_user_myisam where uid>=332;
+-----+--------------------------------+---------------------+-----------+
| uid | name                           | add_time            | server_id |
+-----+--------------------------------+---------------------+-----------+
| 332 | X1LME9HO5V7WXNOKBVZE           | 2018-01-02 09:05:07 | 1323306   |
| 333 | 2PBFQ7KS4BPIJ27G88EYXWEDSX5    | 2018-01-02 09:05:07 | 1323306   |
| 334 | E85Y2SS9UD0FZG4YGCNTRSWA8L     | 2018-01-02 09:05:07 | 1323306   |
| 335 | Y2TQOEVJ58NN7EREL4WRZ          | 2018-01-02 09:05:07 | 1323306   |
| 336 | O0MEATAXYIAE2V2IZG96YVQ56WEUHF | 2018-01-02 09:05:07 | 1323306   |
| 337 | A6QKRWEXHRGUA3V2CH61VXUNBVA3H2 | 2018-01-02 09:05:07 | 1323306   |
| 338 | NYCSI1HS61BN6QAVVYTZSC         | 2018-01-02 09:05:07 | 1323306   |
| 339 | 7CFC1JQPIQGNC97MDTT8ZIMIZL7D   | 2018-01-02 09:05:07 | 1323306   |
| 340 | GA78AR4Z12WQTEAM41JB           | 2018-01-02 09:05:07 | 1323306   |
+-----+--------------------------------+---------------------+-----------+
9 rows in set (0.08 sec)

我们查看mysqldump备份文件获取的binlog pos

[[email protected] backup]# more replcrash_dump_1323306_20180102.sql
-- GTID state at the beginning of the backup
SET @@GLOBAL.GTID_PURGED=‘8ab82362-9c37-11e7-a858-000c29c1025c:1-251873‘;
-- Position to start replication or point-in-time recovery from
-- CHANGE MASTER TO MASTER_LOG_FILE=‘mysql-bin.000183‘, MASTER_LOG_POS=1042441;
[[email protected] backup]# 

这里的pos信息是mysqldump通过SHOW MASTER STATUS获取。查看mysqldump得到的general-log;

[[email protected] data]# vim /data/mysql/mysql3306/data/mysql-general.log
...
2018-01-02T01:05:07.693104Z        10 Query     FLUSH /*!40101 LOCAL */ TABLES
2018-01-02T01:05:07.694738Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘7ATZSNFNIBW5DZNMNZYBMV‘,now(),@@server_id)
2018-01-02T01:05:07.701616Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘7ATZSNFNIBW5DZNMNZYBMV‘,now(),@@server_id)
2018-01-02T01:05:07.702139Z        10 Query     FLUSH TABLES WITH READ LOCK
2018-01-02T01:05:07.702344Z         9 Query     commit
2018-01-02T01:05:07.702411Z        10 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2018-01-02T01:05:07.702597Z        10 Query     START TRANSACTION /*!40100 WITH CONSISTENT SNAPSHOT */
2018-01-02T01:05:07.702721Z        10 Query     SHOW VARIABLES LIKE ‘gtid\_mode‘
2018-01-02T01:05:07.713019Z        10 Query     SELECT @@GLOBAL.GTID_EXECUTED
2018-01-02T01:05:07.713179Z        10 Query     SHOW MASTER STATUS
2018-01-02T01:05:07.725821Z        10 Query     UNLOCK TABLES
2018-01-02T01:05:07.732125Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘X1LME9HO5V7WXNOKBVZE‘,now(),@@server_id)
2018-01-02T01:05:07.733237Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘X1LME9HO5V7WXNOKBVZE‘,now(),@@server_id)
2018-01-02T01:05:07.734240Z         9 Query     commit
2018-01-02T01:05:07.740508Z        10 Query     SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = ‘UNDO LOG‘ AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IN (SELECT DISTINCT LOGFILE_GROUP_NAME FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = ‘DATAFILE‘ AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA IN (‘replcrash‘))) GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME
2018-01-02T01:05:07.741895Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘2PBFQ7KS4BPIJ27G88EYXWEDSX5‘,now(),@@server_id)
2018-01-02T01:05:07.742720Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘2PBFQ7KS4BPIJ27G88EYXWEDSX5‘,now(),@@server_id)
2018-01-02T01:05:07.743257Z         9 Query     commit
2018-01-02T01:05:07.749840Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘E85Y2SS9UD0FZG4YGCNTRSWA8L‘,now(),@@server_id)
2018-01-02T01:05:07.750588Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘E85Y2SS9UD0FZG4YGCNTRSWA8L‘,now(),@@server_id)
2018-01-02T01:05:07.750989Z         9 Query     commit
2018-01-02T01:05:07.754180Z        10 Query     SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = ‘DATAFILE‘ AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA IN (‘replcrash‘)) ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME
2018-01-02T01:05:07.756229Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘Y2TQOEVJ58NN7EREL4WRZ‘,now(),@@server_id)
2018-01-02T01:05:07.757030Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘Y2TQOEVJ58NN7EREL4WRZ‘,now(),@@server_id)
2018-01-02T01:05:07.757598Z         9 Query     commit
2018-01-02T01:05:07.763629Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘O0MEATAXYIAE2V2IZG96YVQ56WEUHF‘,now(),@@server_id)
2018-01-02T01:05:07.764626Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘O0MEATAXYIAE2V2IZG96YVQ56WEUHF‘,now(),@@server_id)
2018-01-02T01:05:07.765654Z         9 Query     commit
2018-01-02T01:05:07.766769Z        10 Query     SHOW VARIABLES LIKE ‘ndbinfo\_version‘
2018-01-02T01:05:07.773997Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘A6QKRWEXHRGUA3V2CH61VXUNBVA3H2‘,now(),@@server_id)
2018-01-02T01:05:07.774757Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘A6QKRWEXHRGUA3V2CH61VXUNBVA3H2‘,now(),@@server_id)
2018-01-02T01:05:07.775198Z         9 Query     commit
2018-01-02T01:05:07.779582Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘NYCSI1HS61BN6QAVVYTZSC‘,now(),@@server_id)
2018-01-02T01:05:07.780174Z        10 Init DB   replcrash
2018-01-02T01:05:07.780249Z        10 Query     SAVEPOINT sp
2018-01-02T01:05:07.780913Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘NYCSI1HS61BN6QAVVYTZSC‘,now(),@@server_id)
2018-01-02T01:05:07.781387Z         9 Query     commit
2018-01-02T01:05:07.781776Z        10 Query     show tables
2018-01-02T01:05:07.782078Z        10 Query     show table status like ‘py\_user‘
2018-01-02T01:05:07.782400Z        10 Query     SET SQL_QUOTE_SHOW_CREATE=1
2018-01-02T01:05:07.782513Z        10 Query     SET SESSION character_set_results = ‘binary‘
2018-01-02T01:05:07.787051Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘7CFC1JQPIQGNC97MDTT8ZIMIZL7D‘,now(),@@server_id)
2018-01-02T01:05:07.787810Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘7CFC1JQPIQGNC97MDTT8ZIMIZL7D‘,now(),@@server_id)
2018-01-02T01:05:07.788502Z         9 Query     commit
2018-01-02T01:05:07.788774Z        10 Query     show create table `py_user`
2018-01-02T01:05:07.789570Z        10 Query     SET SESSION character_set_results = ‘utf8‘
2018-01-02T01:05:07.789725Z        10 Query     show fields from `py_user`
2018-01-02T01:05:07.790423Z        10 Query     show fields from `py_user`
2018-01-02T01:05:07.791163Z        10 Query     SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user`
2018-01-02T01:05:07.791447Z        10 Query     SET SESSION character_set_results = ‘binary‘
2018-01-02T01:05:07.791648Z        10 Query     use `replcrash`
2018-01-02T01:05:07.791778Z        10 Query     select @@collation_database
2018-01-02T01:05:07.791929Z        10 Query     SHOW TRIGGERS LIKE ‘py\_user‘
2018-01-02T01:05:07.792383Z        10 Query     SET SESSION character_set_results = ‘utf8‘
2018-01-02T01:05:07.792492Z        10 Query     ROLLBACK TO SAVEPOINT sp
2018-01-02T01:05:07.792651Z        10 Query     show table status like ‘py\_user\_innodb‘
2018-01-02T01:05:07.792874Z        10 Query     SET SQL_QUOTE_SHOW_CREATE=1
2018-01-02T01:05:07.792948Z        10 Query     SET SESSION character_set_results = ‘binary‘
2018-01-02T01:05:07.793024Z        10 Query     show create table `py_user_innodb`
2018-01-02T01:05:07.793131Z        10 Query     SET SESSION character_set_results = ‘utf8‘
2018-01-02T01:05:07.793220Z        10 Query     show fields from `py_user_innodb`
2018-01-02T01:05:07.793607Z        10 Query     show fields from `py_user_innodb`
2018-01-02T01:05:07.793985Z        10 Query     SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_innodb`
2018-01-02T01:05:07.794435Z         9 Query     insert into py_user_myisam(name,add_time,server_id) values(‘GA78AR4Z12WQTEAM41JB‘,now(),@@server_id)
2018-01-02T01:05:07.795204Z         9 Query     insert into py_user_innodb(name,add_time,server_id) values(‘GA78AR4Z12WQTEAM41JB‘,now(),@@server_id)
2018-01-02T01:05:07.795688Z         9 Query     commit
2018-01-02T01:05:07.798108Z        10 Query     SET SESSION character_set_results = ‘binary‘
2018-01-02T01:05:07.798205Z        10 Query     use `replcrash`
2018-01-02T01:05:07.798303Z        10 Query     select @@collation_database
2018-01-02T01:05:07.798408Z        10 Query     SHOW TRIGGERS LIKE ‘py\_user\_innodb‘
2018-01-02T01:05:07.798884Z        10 Query     SET SESSION character_set_results = ‘utf8‘
2018-01-02T01:05:07.798965Z        10 Query     ROLLBACK TO SAVEPOINT sp
2018-01-02T01:05:07.799049Z        10 Query     show table status like ‘py\_user\_myisam‘
2018-01-02T01:05:07.799271Z        10 Query     SET SQL_QUOTE_SHOW_CREATE=1
2018-01-02T01:05:07.799344Z        10 Query     SET SESSION character_set_results = ‘binary‘
2018-01-02T01:05:07.799420Z        10 Query     show create table `py_user_myisam`
2018-01-02T01:05:07.799554Z        10 Query     SET SESSION character_set_results = ‘utf8‘
2018-01-02T01:05:07.799661Z        10 Query     show fields from `py_user_myisam`
2018-01-02T01:05:07.800098Z        10 Query     show fields from `py_user_myisam`
2018-01-02T01:05:07.800418Z        10 Query     SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_myisam`

mysqldump备份过程(--single-transaction --master-data):
会话先执行FTWRL(实例只读),然后设置RR隔离级别->START TRANSACTION WITH CONSISTENT SNAPSHOT;->SHOW MASTER STATUS;->UNLOCK TABLES;->SELECT /*!40001 SQL_NO_CACHE */ * FROM `tbname`;
在UNLOCK TABLES解锁后其他事务就可以进行写入操作。general-log中我们可看到 UNLOCK TABLES 到 SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_myisam` 之间往py_user_myisam、py_user_innodb各写入9条数据
SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_innodb`;读取的是START TRANSACTION WITH CONSISTENT SNAPSHOT建立时的数据
SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_myisam`;读取的是最新的数据
再来查看上述过程期间binary log记录

[[email protected] logs]# mysqlbinlog -vv --base64-output=decode-rows mysql-bin.000183 |more
...
COMMIT/*!*/;
# at 1042059
#180102  9:05:07 server id 1323306  end_log_pos 1042124 CRC32 0x221cda50        GTID    last_committed=2917     sequence_number=2918    rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= ‘8ab82362-9c37-11e7-a858-000c29c1025c:251873‘/*!*/;
# at 1042124
#180102  9:05:07 server id 1323306  end_log_pos 1042209 CRC32 0x5df266e4        Query   thread_id=9     exec_time=0     error_code=0
SET TIMESTAMP=1514855107/*!*/;
BEGIN
/*!*/;
# at 1042209
#180102  9:05:07 server id 1323306  end_log_pos 1042279 CRC32 0xc1d41c5f        Table_map: `replcrash`.`py_user_myisam` mapped to number 254
# at 1042279
#180102  9:05:07 server id 1323306  end_log_pos 1042355 CRC32 0x27badc02        Write_rows: table id 254 flags: STMT_END_F
### INSERT INTO `replcrash`.`py_user_myisam`
### SET
###   @1=331 /* INT meta=0 nullable=0 is_null=0 */
###   @2=‘7ATZSNFNIBW5DZNMNZYBMV‘ /* VARSTRING(96) meta=96 nullable=1 is_null=0 */
###   @3=‘2018-01-02 09:05:07‘ /* DATETIME(0) meta=0 nullable=1 is_null=0 */
###   @4=‘1323306‘ /* VARSTRING(30) meta=30 nullable=1 is_null=0 */
# at 1042355
#180102  9:05:07 server id 1323306  end_log_pos 1042441 CRC32 0x67285443        Query   thread_id=9     exec_time=0     error_code=0
SET TIMESTAMP=1514855107/*!*/;
COMMIT
/*!*/;
# at 1042441
==================== mysqldump备份文件获取的binlog pos ====================
==================== 备份文件还原后,从库GTID_PURGED位置 ====================

#180102  9:05:07 server id 1323306  end_log_pos 1042506 CRC32 0xf77ede80        GTID    last_committed=2918     sequence_number=2919    rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= ‘8ab82362-9c37-11e7-a858-000c29c1025c:251874‘/*!*/;
# at 1042506
#180102  9:05:07 server id 1323306  end_log_pos 1042591 CRC32 0x506a2875        Query   thread_id=9     exec_time=0     error_code=0
SET TIMESTAMP=1514855107/*!*/;
BEGIN
/*!*/;
# at 1042591
#180102  9:05:07 server id 1323306  end_log_pos 1042661 CRC32 0x90b154e8        Table_map: `replcrash`.`py_user_innodb` mapped to number 255
# at 1042661
#180102  9:05:07 server id 1323306  end_log_pos 1042737 CRC32 0x1d693238        Write_rows: table id 255 flags: STMT_END_F
### INSERT INTO `replcrash`.`py_user_innodb`
### SET
###   @1=331 /* INT meta=0 nullable=0 is_null=0 */
###   @2=‘7ATZSNFNIBW5DZNMNZYBMV‘ /* VARSTRING(96) meta=96 nullable=1 is_null=0 */
###   @3=‘2018-01-02 09:05:07‘ /* DATETIME(0) meta=0 nullable=1 is_null=0 */
###   @4=‘1323306‘ /* VARSTRING(30) meta=30 nullable=1 is_null=0 */
# at 1042737
#180102  9:05:07 server id 1323306  end_log_pos 1042768 CRC32 0x87864022        Xid = 4420
COMMIT/*!*/;
# at 1042768
==================== 启动复制后,py_user_innodb写入uid=331记录,成功 ====================

#180102  9:05:07 server id 1323306  end_log_pos 1042833 CRC32 0xe492578a        GTID    last_committed=2919     sequence_number=2920    rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= ‘8ab82362-9c37-11e7-a858-000c29c1025c:251875‘/*!*/;
# at 1042833
#180102  9:05:07 server id 1323306  end_log_pos 1042918 CRC32 0xf08c4165        Query   thread_id=9     exec_time=0     error_code=0
SET TIMESTAMP=1514855107/*!*/;
BEGIN
/*!*/;
# at 1042918
#180102  9:05:07 server id 1323306  end_log_pos 1042988 CRC32 0xf16731d6        Table_map: `replcrash`.`py_user_myisam` mapped to number 257
# at 1042988
#180102  9:05:07 server id 1323306  end_log_pos 1043062 CRC32 0x128aec5e        Write_rows: table id 257 flags: STMT_END_F
### INSERT INTO `replcrash`.`py_user_myisam`
### SET
###   @1=332 /* INT meta=0 nullable=0 is_null=0 */
###   @2=‘X1LME9HO5V7WXNOKBVZE‘ /* VARSTRING(96) meta=96 nullable=1 is_null=0 */
###   @3=‘2018-01-02 09:05:07‘ /* DATETIME(0) meta=0 nullable=1 is_null=0 */
###   @4=‘1323306‘ /* VARSTRING(30) meta=30 nullable=1 is_null=0 */
# at 1043062
#180102  9:05:07 server id 1323306  end_log_pos 1043148 CRC32 0x5cc8cc30        Query   thread_id=9     exec_time=0     error_code=0
SET TIMESTAMP=1514855107/*!*/;
COMMIT
/*!*/;
# at 1043148
==================== 启动复制后,py_user_myisam写入uid=332记录,失败 ====================
==================== 从库py_user_myisam表已存在332记录,sql_thread停止 ====================

#180102  9:05:07 server id 1323306  end_log_pos 1043213 CRC32 0xceb1ce4d        GTID    last_committed=2920     sequence_number=2921    rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= ‘8ab82362-9c37-11e7-a858-000c29c1025c:251876‘/*!*/;
# at 1043213
#180102  9:05:07 server id 1323306  end_log_pos 1043298 CRC32 0x38591b71        Query   thread_id=9     exec_time=0     error_code=0
SET TIMESTAMP=1514855107/*!*/;
BEGIN
/*!*/;
# at 1043298
#180102  9:05:07 server id 1323306  end_log_pos 1043368 CRC32 0xf80c2ae9        Table_map: `replcrash`.`py_user_innodb` mapped to number 258
# at 1043368
#180102  9:05:07 server id 1323306  end_log_pos 1043442 CRC32 0x0bf4ae26        Write_rows: table id 258 flags: STMT_END_F
### INSERT INTO `replcrash`.`py_user_innodb`
### SET
###   @1=332 /* INT meta=0 nullable=0 is_null=0 */
###   @2=‘X1LME9HO5V7WXNOKBVZE‘ /* VARSTRING(96) meta=96 nullable=1 is_null=0 */
###   @3=‘2018-01-02 09:05:07‘ /* DATETIME(0) meta=0 nullable=1 is_null=0 */
###   @4=‘1323306‘ /* VARSTRING(30) meta=30 nullable=1 is_null=0 */
# at 1043442
#180102  9:05:07 server id 1323306  end_log_pos 1043473 CRC32 0x1b75f9e0        Xid = 4431
COMMIT/*!*/;
# at 1043473
...

二进志日志显示,按照py_user_myisam->py_user_innodb->py_user_myisam这样的顺序往表中写入数据。
使用备份文件搭建的从库,py_user_myisam表已包含UNLOCK TABLES之后的9条数据,但在备份文件中的@@GLOBAL.GTID_PURGED却是UNLOCK TABLES时刻的位置。因此在启动复制后,第一条操作py_user_innodb表成功,第二条操作py_user_myisam表失败,从库报主键冲突错误
那么应该如何修复这个错误呢?

# 删除从库py_user_myisam表大于等于冲突key的记录
mydba@192.168.85.132,3308 [replcrash]> delete from py_user_myisam where uid>=332;
# 重新启动sql_thread
mydba@192.168.85.132,3308 [replcrash]> start slave sql_thread;

总得来说就是只有innodb才会提供一致性备份!!!

1.2、DDL操作对备份的影响

建议先阅读后续章节,理解各备份过程后再返回阅读DDL操作对备份的影响
运行下面的脚本持续DDL操作

#!/user/bin/python
import string
import random
import MySQLdb
import time

conn = MySQLdb.connect(host=‘192.168.85.132‘,
                      port=3306,
                      user=‘mydba‘,
                      passwd=‘mysql5719‘,
                      db=‘replcrash‘)
"""
create table py_user(
uid int not null auto_increment,
name varchar(32),
add_time datetime default current_timestamp,
server_id varchar(10),
primary key(uid),
key(name)
);
"""
counter = 1
while counter<=100:
  addcol = ‘col‘ + str(counter)
  print addcol

  cursor = conn.cursor()
  #DDL
  cursor.execute("alter table py_user_innodb add %s int;" % addcol)
  cursor.execute("alter table py_user_innodb drop column %s;" % addcol)
  conn.commit()
  counter += 1
  time.sleep(0.0001)

Python DDL

逻辑备份数据库

# 清空general_log
[[email protected] logs]# cat /dev/null > /data/mysql/mysql3306/data/mysql-general.log

# mysqldump
[[email protected] backup]# mysqldump -h127.0.0.1 -P3306 -uroot -p --single-transaction --master-data=2 replcrash >/data/backup/replcrash_dump_1323306_`date +%Y%m%d`.sql
Enter password:
Warning: A partial dump from a server that has GTIDs will by default include the GTIDs of all transactions, even those that changed suppressed parts of the database. If you don not want to restore GTIDs, pass --set-gtid-purged=OFF. To make a complete dump, pass --all-databases --triggers --routines --events.
mysqldump: Error 1412: Table definition has changed, please retry transaction when dumping table `py_user_innodb` at row: 0

# mysqlpump
[[email protected] backup]# mysqlpump -h127.0.0.1 -P3306 -uroot -p --single-transaction --add-drop-table --exclude-databases=mysql,sakila,backupdb -A >/data/backup/replcrash_pump_1323306_`date +%Y%m%d`.sql
Enter password:
mysqlpump: [ERROR] (1412) Table definition has changed, please retry transaction
Dump process encountered error and will not continue.
Dump progress: 0/3 tables, 250/2431 rows
[[email protected] backup]# 

# mydumper
[[email protected] mydumper]# mydumper -h 127.0.0.1 -P 3306 -u root -p mysql5719 --trx-consistency-only -v 3 -t 2 -o /data/backup/mydumper
..
** Message: Thread 2 dumping data for `replcrash`.`py_user_innodb`
** Message: Thread 1 dumping data for `replcrash`.`py_user_myisam`
** (mydumper:5179): CRITICAL **: Could not read data from replcrash.py_user_innodb: Table definition has changed, please retry transaction
** Message: Empty table replcrash.py_user_innodb
...

mydumper需要使用-v 3 显示详细信息,本身是不会报错的!!!如果想更容易再现错误,最好是备份的数据表较大、较多,适当降低并行线程数(-t),开启--trx-consistency-only,让其尽早解锁
生成的备份文件中,py_user_innodb表只有结构,没有数据
根据备份逻辑,在UNLOCK TABLES解除FTWRL到SELECT /*!40001 SQL_NO_CACHE */ * FROM `tbname`之间如果有DDL操作,就会造成上述错误
使用了with consistent snapshot子句开启一致性快照事务之后,如果一旦表结构定义发生改变,事务将无法对该表执行查询

使用WITH CONSISTENT SNAPSHOT子句,会话1显式开启一个事务之后先不执行查询,会话B使用DDL语句添加一个字段
会话1 会话2

修改隔离级别为RR
[email protected],3306 [replcrash]> set tx_isolation=‘repeatable-read‘;
Query OK, 0 rows affected (0.00 sec)


修改隔离级别为RR
[email protected],3306 [replcrash]> set tx_isolation=‘repeatable-read‘;
Query OK, 0 rows affected (0.00 sec)


显式开启一个事务,先不执行查询
[email protected],3306 [replcrash]> start transaction with consistent snapshot;
Query OK, 0 rows affected (0.00 sec)

 
 
执行DDL语句添加字段,执行成功
[email protected],3306 [replcrash]> alter table py_user_innodb add col1 int;
Query OK, 0 rows affected (0.13 sec)
Records: 0 Duplicates: 0 Warnings: 0


执行查询,报表定义已经改变的错误
[email protected],3306 [replcrash]> select * from py_user_innodb;
ERROR 1412 (HY000): Table definition has changed, please retry transaction

 

物理备份数据库

[[email protected] backup]# innobackupex --defaults-file=/data/mysql/mysql3306/my.cnf -S /tmp/mysql3306.sock -uroot -pmysql5719 /data/backup/full/
180104 15:29:31 [01] Copying ./sakila/payment.ibd to /data/backup/full/2018-01-04_15-29-21/sakila/payment.ibd
180104 15:29:32 [01]        ...done
InnoDB: Last flushed lsn: 19337348200 load_index lsn 19337354022
InnoDB: An optimized (without redo logging) DDLoperation has been performed. All modified pages may not have been flushed to the disk yet.
PXB will not be able take a consistent backup. Retry the backup operation
180104 15:29:32 [01] Copying ./replcrash/py_user.ibd to /data/backup/full/2018-01-04_15-29-21/replcrash/py_user.ibd
180104 15:29:32 [01]        ...done
180104 15:29:32 [01] Copying ./replcrash/py_user_innodb.ibd to /data/backup/full/2018-01-04_15-29-21/replcrash/py_user_innodb.ibd
180104 15:29:32 [01]        ...done

只要在备份期间(实际是备份InnoDB表期间,因为备份non-InnoDB表期间会加FTWRL只读锁,阻塞DDL、DML操作)执行DDL操作,innobackupex就会报错退出。原因就是DDL操作不会记录到redo log,PXB will not be able take a consistent backup.
因此备份期间要避免执行不记录事务日志的操作(ALTER TABLE, CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE)

二、mysqlpump

2.1、备份过程

借助前面已开启general_log,来查看mysqlpump执行过程
MySQL 5.7.11起解决了--single-transaction和--default-parallelism的互斥问题

# 清空general_log
[[email protected] logs]# cat /dev/null > /data/mysql/mysql3306/data/mysql-general.log

# 备份replcrash数据库,一致性备份(源码限制只有备份整个实例时才能返回GTID信息,因此这里使用-A,然后使用--exclude-databases排除不需要备份的db)
[[email protected] backup]# mysqlpump -h127.0.0.1 -P3306 -uroot -p --single-transaction --add-drop-table --exclude-databases=mysql,sakila,backupdb -A >/data/backup/replcrash_pump_1323306_`date +%Y%m%d`.sql
默认mysqlpump使用一个队列两个线程

[[email protected] data]# vim /data/mysql/mysql3306/data/mysql-general.log
...
   4 Time                 Id Command    Argument
   5 2018-01-03T01:24:06.623704Z        14 Connect   [email protected] on  using TCP/IP
   6 2018-01-03T01:24:06.631266Z        14 Query     FLUSH TABLES WITH READ LOCK
   7 2018-01-03T01:24:06.667093Z        14 Query     SHOW WARNINGS
   8 2018-01-03T01:24:06.667310Z        14 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
   9 2018-01-03T01:24:06.667520Z        14 Query     SHOW WARNINGS
  10 2018-01-03T01:24:06.667647Z        14 Query     START TRANSACTION WITH CONSISTENT SNAPSHOT
  11 2018-01-03T01:24:06.667792Z        14 Query     SHOW WARNINGS
  12 2018-01-03T01:24:06.679491Z        15 Connect   [email protected] on  using TCP/IP
  13 2018-01-03T01:24:06.683019Z        15 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
  14 2018-01-03T01:24:06.684865Z        15 Query     SHOW WARNINGS
  15 2018-01-03T01:24:06.685015Z        15 Query     START TRANSACTION WITH CONSISTENT SNAPSHOT
  16 2018-01-03T01:24:06.685114Z        15 Query     SHOW WARNINGS
  17 2018-01-03T01:24:06.686057Z        16 Connect   [email protected] on  using TCP/IP
  18 2018-01-03T01:24:06.688856Z        16 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
  19 2018-01-03T01:24:06.716231Z        16 Query     SHOW WARNINGS
  20 2018-01-03T01:24:06.716447Z        16 Query     START TRANSACTION WITH CONSISTENT SNAPSHOT
  21 2018-01-03T01:24:06.716558Z        16 Query     SHOW WARNINGS
  22 2018-01-03T01:24:06.716701Z        14 Query     UNLOCK TABLES
  23 2018-01-03T01:24:06.716857Z        14 Query     SHOW WARNINGS
...各种SHOW...
1176 2018-01-03T01:24:07.344380Z        16 Query     SHOW CREATE DATABASE IF NOT EXISTS `replcrash`
1177 2018-01-03T01:24:07.344468Z        16 Query     SHOW WARNINGS
1178 2018-01-03T01:24:07.344565Z        16 Query     SHOW TABLE STATUS FROM `replcrash`
1179 2018-01-03T01:24:07.380209Z        16 Query     SHOW WARNINGS
1180 2018-01-03T01:24:07.380416Z        16 Query     SHOW COLUMNS IN `py_user` FROM `replcrash`
1181 2018-01-03T01:24:07.381223Z        16 Query     SHOW WARNINGS
1182 2018-01-03T01:24:07.381408Z        16 Query     SHOW CREATE TABLE `replcrash`.`py_user`
1183 2018-01-03T01:24:07.381614Z        16 Query     SHOW WARNINGS
1184 2018-01-03T01:24:07.381950Z        16 Query     SHOW TRIGGERS FROM `replcrash` LIKE ‘py_user‘
1185 2018-01-03T01:24:07.382575Z        16 Query     SHOW WARNINGS
1186 2018-01-03T01:24:07.382764Z        16 Query     SHOW COLUMNS IN `py_user_innodb` FROM `replcrash`
1187 2018-01-03T01:24:07.383125Z        14 Query     SET SQL_QUOTE_SHOW_CREATE= 1
1188 2018-01-03T01:24:07.383334Z        14 Query     SHOW WARNINGS
1189 2018-01-03T01:24:07.383617Z        14 Query     SET TIME_ZONE=‘+00:00‘
1190 2018-01-03T01:24:07.384037Z        14 Query     SHOW WARNINGS
1191 2018-01-03T01:24:07.385106Z        15 Query     SELECT `COLUMN_NAME`, `EXTRA` FROM `INFORMATION_SCHEMA`.`COLUMNS`WHERE TABLE_SCHEMA =‘replcrash‘ AND TABLE_NAME =‘py_user‘
1192 2018-01-03T01:24:07.386099Z        15 Query     SHOW WARNINGS
1193 2018-01-03T01:24:07.386347Z        15 Query     SELECT SQL_NO_CACHE `uid`,`name`,`add_time`,`server_id`  FROM `replcrash`.`py_user`
1194 2018-01-03T01:24:07.387102Z        15 Query     SHOW WARNINGS
1195 2018-01-03T01:24:07.387644Z        16 Query     SHOW WARNINGS
1196 2018-01-03T01:24:07.387997Z        16 Query     SHOW CREATE TABLE `replcrash`.`py_user_innodb`
1197 2018-01-03T01:24:07.388216Z        16 Query     SHOW WARNINGS
1198 2018-01-03T01:24:07.388487Z        16 Query     SHOW TRIGGERS FROM `replcrash` LIKE ‘py_user_innodb‘
1199 2018-01-03T01:24:07.389053Z        14 Query     SELECT `COLUMN_NAME`, `EXTRA` FROM `INFORMATION_SCHEMA`.`COLUMNS`WHERE TABLE_SCHEMA =‘replcrash‘ AND TABLE_NAME =‘py_user_innodb‘
1200 2018-01-03T01:24:07.390054Z        14 Query     SHOW WARNINGS
1201 2018-01-03T01:24:07.390293Z        14 Query     SELECT SQL_NO_CACHE `uid`,`name`,`add_time`,`server_id`  FROM `replcrash`.`py_user_innodb`
1202 2018-01-03T01:24:07.391566Z        16 Query     SHOW WARNINGS
1203 2018-01-03T01:24:07.391776Z        16 Query     SHOW COLUMNS IN `py_user_myisam` FROM `replcrash`
1204 2018-01-03T01:24:07.392559Z        16 Query     SHOW WARNINGS
1205 2018-01-03T01:24:07.392747Z        16 Query     SHOW CREATE TABLE `replcrash`.`py_user_myisam`
1206 2018-01-03T01:24:07.393065Z        16 Query     SHOW WARNINGS
1207 2018-01-03T01:24:07.393336Z        16 Query     SHOW TRIGGERS FROM `replcrash` LIKE ‘py_user_myisam‘
1208 2018-01-03T01:24:07.394146Z        16 Query     SHOW WARNINGS
1209 2018-01-03T01:24:07.394371Z        16 Query     SHOW FUNCTION STATUS WHERE db = ‘replcrash‘
1210 2018-01-03T01:24:07.396083Z        15 Query     SELECT `COLUMN_NAME`, `EXTRA` FROM `INFORMATION_SCHEMA`.`COLUMNS`WHERE TABLE_SCHEMA =‘replcrash‘ AND TABLE_NAME =‘py_user_myisam‘
1211 2018-01-03T01:24:07.399053Z        15 Query     SHOW WARNINGS
1212 2018-01-03T01:24:07.399425Z        15 Query     SELECT SQL_NO_CACHE `uid`,`name`,`add_time`,`server_id`  FROM `replcrash`.`py_user_myisam`
1213 2018-01-03T01:24:07.405719Z        16 Query     SHOW WARNINGS
1214 2018-01-03T01:24:07.405915Z        16 Query     SHOW PROCEDURE STATUS WHERE db = ‘replcrash‘
1215 2018-01-03T01:24:07.412340Z        14 Query     SHOW WARNINGS
1216 2018-01-03T01:24:07.414496Z        15 Query     SHOW WARNINGS
1217 2018-01-03T01:24:07.416755Z        16 Query     SHOW WARNINGS
1218 2018-01-03T01:24:07.417261Z        16 Query     SHOW EVENTS FROM `replcrash`
1219 2018-01-03T01:24:07.417884Z        16 Query     SHOW WARNINGS

即使备份一个db,general_log中也会出现很多其他库的内容(⊙_⊙)
mysqlpump备份过程(--single-transaction):
对于建立的第一个连接,执行FLUSH TABLES WITH READ LOCK,加上只读锁;对于其他连接(包含第一个连接),设置RR隔离级别,并开启一致性快照读START TRANSACTION WITH CONSISTENT SNAPSHOT;当所有连接(--default-parallelism设置多少个线程)都建立好后,再执行解锁UNLOCK TABLES;最后通过SELECT colname  FROM `tbname`备份数据
general_log中没有看到SHOW MASTER STATUS,猜测它类似于mydumper,也是在主线程FLUSH TABLES WITH READ LOCK后,就去获取GTID信息。
因此mysqlpump也会遇到在mysqldump中的坑
mysqlpump并行备份的部分参数需要在单线程模式才能应用,实际使用中建议先测试,对比输出结果分辨各参数的作用

2.2、--single-transaction

• --single-transaction
This option sets the transaction isolation mode to REPEATABLE READ and sends a START TRANSACTION SQL statement to the server before dumping data. It is useful only with transactional tables such as InnoDB, because then it dumps the consistent state of the database at the time when START TRANSACTION was issued without blocking any applications.
When using this option, you should keep in mind that only InnoDB tables are dumped in a consistent state. For example, any MyISAM or MEMORY tables dumped while using this option may still change state.
While a --single-transaction dump is in process, to ensure a valid dump file (correct table contents and binary log coordinates), no other connection should use the following statements: ALTER TABLE, CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE. A consistent read is not isolated from those statements, so use of them on a table to be dumped can cause the SELECT that is performed by mysqldump/mysqlpump to retrieve the table contents to obtain incorrect contents or fail.
• START TRANSACTION
The WITH CONSISTENT SNAPSHOT modifier starts a consistent read for storage engines that are capable of it. This applies only to InnoDB. The effect is the same as issuing a START TRANSACTION followed by a SELECT from any InnoDB table. 
The WITH CONSISTENT SNAPSHOT modifier does not change the current transaction isolation level, so it provides a consistent snapshot only if the current isolation level is one that permits a consistent read. The only isolation level that permits a consistent read is REPEATABLE READ.

该参数将事务隔离级别设置成Repeatable Read,并在dump之前发送start transaction语句给服务端。这只对事务表(比如innodb)很有用,因为在发出start transaction时,保证了在不阻塞任何应用下的一致性状态。对myisam和memory等非事务表,还是会改变状态的,当使用此参的时候要确保没有其他连接在使用ALTER TABLE、CREATE TABLE、DROP TABLE、RENAME TABLE、TRUNCATE TABLE等语句,否则会出现不正确的内容或者失败。在mysql5.7.11之前,--default-parallelism大于1的时候和此参数互斥,必须使用--default-parallelism=0。5.7.11之后解决了--single-transaction和--default-parallelism的互斥问题

三、mydumper

3.1、备份过程

mydumper备份过程
1、连接目标数据库
2、通过show processlist来判断是否有长查询,根据参数long-query-guard和kill-long-queries决定退出或杀掉长查询
3、主线程flush tables with read lock;start transaction with consistent snapshot
4、主线程读取当前时间点的二进制日志文件名和日志写入的位置并记录在metadata文件中,以供即时点恢复使用
5、创建dump线程,缺省为4个
6、确定候选表,根据类别分别插入non_innodb_table、innodb_tables以及table_schemas链表
7、将候选表通过g_async_queue_push加入任务队列(队列最后元素是thread shutdown),由dump线程从队列中读取表信息并执行数据导出
8、备份完non_innodb_table后立即unlock tables解锁,以减少锁定时间
9、等待dump InnoDB tables完成

从备份逻辑中可以看出,mydumper需要在备份完非事务表之后才解锁,这就保证了事务表和非事务表的一致性备份。默认情况下,DML操作不会影响mydumper备份的一致性

3.2、--trx-consistency-only

这个参数退化为仅保证事务表的一致性备份,在备份非事务表前如果有数据写入就会导致数据与metadata不一致,有点类似前面的mysqldump、mysqlpump的味道
运行前面的PythonDML脚本,一直往py_user_myisam、py_user_innodb表中写入数据,测试--trx-consistency-only选项的效果

# 清空general_log
[[email protected] logs]# cat /dev/null > /data/mysql/mysql3306/data/mysql-general.log

# 备份replcrash库(仅事务表一致性备份)
[[email protected] mydumper]# mydumper -h 127.0.0.1 -P 3306 -u root -p mysql5719 --trx-consistency-only -v 3 -B replcrash -o /data/backup/mydumper
** (mydumper:2808): WARNING **: Using trx_consistency_only, binlog coordinates will not be accurate if you are writing to non transactional tables.
** Message: Connected to a MySQL server
** Message: Started dump at: 2018-01-04 09:34:53

** Message: Written master status
** Message: Thread 1 connected using MySQL connection ID 50
** Message: Thread 2 connected using MySQL connection ID 51
** Message: Thread 3 connected using MySQL connection ID 52
** Message: Thread 4 connected using MySQL connection ID 53
** Message: Transactions started, unlocking tables
** Message: Thread 1 dumping data for `replcrash`.`py_user`
** Message: Thread 2 dumping data for `replcrash`.`py_user_innodb`
** Message: Thread 2 dumping data for `replcrash`.`py_user_myisam`
** Message: Thread 1 dumping schema for `replcrash`.`py_user`
** Message: Thread 3 dumping schema for `replcrash`.`py_user_innodb`
** Message: Thread 1 dumping schema for `replcrash`.`py_user_myisam`
** Message: Thread 3 shutting down
** Message: Thread 1 shutting down
** Message: Thread 2 shutting down
** Message: Thread 4 shutting down
** Message: Finished dump at: 2018-01-04 09:34:53

# 查看metadata
[[email protected] mydumper]# cat metadata
Started dump at: 2018-01-04 09:34:53
SHOW MASTER STATUS:
        Log: mysql-bin.000185
        Pos: 2452139
        GTID:8ab82362-9c37-11e7-a858-000c29c1025c:1-272303

Finished dump at: 2018-01-04 09:34:53
[[email protected] mydumper]# 

# 还原replcrash库
[[email protected] mydumper]# myloader -h 127.0.0.1 -P 3308 -u root -p mysql5719 -o -B replcrash -d /data/backup/mydumper

myloader不会产生binlog,也不会应用metadata中的GTID
# 设置GTID_PURGED
[email protected]192.168.85.132,3308 [replcrash]> reset master;
[email protected]192.168.85.132,3308 [replcrash]> SET @@GLOBAL.GTID_PURGED=‘8ab82362-9c37-11e7-a858-000c29c1025c:1-272303‘;

# 搭建复制
[email protected]192.168.85.132,3308 [replcrash]> change master to
master_host=‘192.168.85.132‘,
master_port=3306,
master_user=‘repl‘,
master_password=‘repl‘,
master_auto_position=1;

# 启动复制,查看复制状态
[email protected]192.168.85.132,3308 [replcrash]> start slave;
[email protected]192.168.85.132,3308 [replcrash]> show slave status\G
*************************** 1. row ***************************
              Master_Log_File: mysql-bin.000185
          Read_Master_Log_Pos: 2758886
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 414
        Relay_Master_Log_File: mysql-bin.000185
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
          Exec_Master_Log_Pos: 2452139
               Last_SQL_Errno: 1062
               Last_SQL_Error: Could not execute Write_rows event on table replcrash.py_user_myisam; Duplicate entry ‘783‘ for key ‘PRIMARY‘, Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event‘‘s master log mysql-bin.000185, end_log_pos 2452443
           Retrieved_Gtid_Set: 8ab82362-9c37-11e7-a858-000c29c1025c:272304-273161
            Executed_Gtid_Set: 8ab82362-9c37-11e7-a858-000c29c1025c:1-272303
                Auto_Position: 1

# 大于等于冲突key的数据
[email protected]192.168.85.132,3308 [replcrash]> select * from replcrash.py_user_myisam where uid>=783;
+-----+--------------------------------+---------------------+-----------+
| uid | name                           | add_time            | server_id |
+-----+--------------------------------+---------------------+-----------+
| 783 | MLGU22VB26RHNNYAY6IPPUJX9A74EM | 2018-01-04 09:34:53 | 1323306   |
+-----+--------------------------------+---------------------+-----------+
1 row in set (0.06 sec)

# mydumper产生的general-log
[[email protected] data]# vim /data/mysql/mysql3306/data/mysql-general.log
...
2018-01-04T01:34:53.769877Z        49 Query     SET SESSION net_write_timeout = 2147483
2018-01-04T01:34:53.770302Z        49 Query     SHOW PROCESSLIST
2018-01-04T01:34:53.770886Z        49 Query     FLUSH TABLES WITH READ LOCK
2018-01-04T01:34:53.771236Z        49 Query     START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2018-01-04T01:34:53.771430Z        49 Query     /*!40101 SET NAMES binary*/
2018-01-04T01:34:53.772084Z        49 Query     SHOW MASTER STATUS
2018-01-04T01:34:53.772331Z        49 Query     SHOW SLAVE STATUS
2018-01-04T01:34:53.773386Z        48 Query     insert into py_user_myisam(name,add_time,server_id) values(‘MLGU22VB26RHNNYAY6IPPUJX9A74EM‘,now(),@@server_id)
2018-01-04T01:34:53.773981Z        50 Connect   [email protected] on  using TCP/IP
2018-01-04T01:34:53.788273Z        50 Query     SET SESSION wait_timeout = 2147483
2018-01-04T01:34:53.788874Z        50 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2018-01-04T01:34:53.788972Z        50 Query     START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2018-01-04T01:34:53.789092Z        50 Query     /*!40103 SET TIME_ZONE=‘+00:00‘ */
2018-01-04T01:34:53.789201Z        50 Query     /*!40101 SET NAMES binary*/
2018-01-04T01:34:53.790091Z        51 Connect   [email protected] on  using TCP/IP
2018-01-04T01:34:53.790315Z        51 Query     SET SESSION wait_timeout = 2147483
2018-01-04T01:34:53.791912Z        51 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2018-01-04T01:34:53.792095Z        51 Query     START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2018-01-04T01:34:53.792307Z        51 Query     /*!40103 SET TIME_ZONE=‘+00:00‘ */
2018-01-04T01:34:53.792418Z        51 Query     /*!40101 SET NAMES binary*/
2018-01-04T01:34:53.793381Z        52 Connect   [email protected] on  using TCP/IP
2018-01-04T01:34:53.794631Z        52 Query     SET SESSION wait_timeout = 2147483
2018-01-04T01:34:53.796301Z        52 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2018-01-04T01:34:53.796404Z        52 Query     START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2018-01-04T01:34:53.796538Z        52 Query     /*!40103 SET TIME_ZONE=‘+00:00‘ */
2018-01-04T01:34:53.796641Z        52 Query     /*!40101 SET NAMES binary*/
2018-01-04T01:34:53.797414Z        53 Connect   [email protected] on  using TCP/IP
2018-01-04T01:34:53.798420Z        53 Query     SET SESSION wait_timeout = 2147483
2018-01-04T01:34:53.799316Z        53 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2018-01-04T01:34:53.799425Z        53 Query     START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2018-01-04T01:34:53.800345Z        53 Query     /*!40103 SET TIME_ZONE=‘+00:00‘ */
2018-01-04T01:34:53.800449Z        53 Query     /*!40101 SET NAMES binary*/
2018-01-04T01:34:53.800661Z        49 Query     UNLOCK TABLES /* trx-only */
2018-01-04T01:34:53.801868Z        49 Init DB   replcrash
2018-01-04T01:34:53.802332Z        49 Query     SHOW TABLE STATUS
2018-01-04T01:34:53.804759Z        50 Query     SELECT /*!40001 SQL_NO_CACHE */ * FROM `replcrash`.`py_user`
2018-01-04T01:34:53.805137Z        51 Query     SELECT /*!40001 SQL_NO_CACHE */ * FROM `replcrash`.`py_user_innodb`
2018-01-04T01:34:53.807164Z        51 Query     SELECT /*!40001 SQL_NO_CACHE */ * FROM `replcrash`.`py_user_myisam`
2018-01-04T01:34:53.808786Z        48 Query     insert into py_user_innodb(name,add_time,server_id) values(‘MLGU22VB26RHNNYAY6IPPUJX9A74EM‘,now(),@@server_id)
2018-01-04T01:34:53.809449Z        49 Query     SHOW CREATE DATABASE `replcrash`
2018-01-04T01:34:53.810326Z        52 Query     SHOW CREATE TABLE `replcrash`.`py_user_innodb`
2018-01-04T01:34:53.810638Z        50 Query     SHOW CREATE TABLE `replcrash`.`py_user`
2018-01-04T01:34:53.811307Z        52 Quit
2018-01-04T01:34:53.811506Z        50 Query     SHOW CREATE TABLE `replcrash`.`py_user_myisam`
2018-01-04T01:34:53.812272Z        50 Quit
2018-01-04T01:34:53.812512Z        48 Query     commit

48是python持续写入的线程、49是主线程、50~53是dump线程
所有连接建立后,主线程立即解锁UNLOCK TABLES /* trx-only */。主线程SHOW MASTER STATUS的后面紧接着python写入nsert into py_user_myisam,实际这个写入操作应该是在主线程UNLOCK TABLES之后,dump线程备份py_user_myisam数据之前。这就导致备份的数据与metadata不一致,因此不推荐使用--trx-consistency-only选项

3.3、逻辑备份差异

mysqldump:只支持单线程工作,这就使得它无法迅速的备份数据
mysqlpump:并行的最小粒度是单个数据库对象,对于每张表的导出只能是单个线程的
mydumper:支持对单表多个线程备份,参数-r
多线程操作提升空间受限于磁盘的IO能力,在使用前做好磁盘IO的评估

四、XtraBackup

4.1、备份过程

innobackupex全备过程
1、start xtrabackup_log
2、copy .ibd、ibdata1
3、FLUSH TABLES WITH READ LOCK
4、copy .frm、.MYD、.MYI、misc files
5、SHOW MASTER STATUS
6、UNLOCK TABLES
7、stop and copy xtrabackup_log

备份开始时首先会开启一个后台检测进程,从当前checkpoint位置开始拷贝redo log,同时持续检测redo log,一旦发现redo中有新的日志写入,立刻将日志记入后台日志文件xtrabackup_log中。之后拷贝innodb的数据文件和系统表空间文件ibdata1,待拷贝结束后,执行flush tables with read lock操作,拷贝.frm,MYI,MYD,等文件,并且在这一时刻获得binlog的位置,最后会发出unlock tables,把表设置为可读可写状态,最终停止xtrabackup_log。
InnoDB表copy [.ibd、ibdata]+[redo log]借助InnoDB Crash Recovery机制保证一致性;non-InnoDB表使用FTWRL加只读锁后backup non-InnoDB tables and files。DML操作不会影响innobackupex备份的一致性

4.2、总结

  mysqldump mysqlpump mydumper innobackupex
FTWRL被阻塞 存在互斥锁,会被阻塞 存在互斥锁,会被阻塞 存在互斥锁,会被阻塞 存在互斥锁,会被阻塞
DDL操作导致备份异常 报错,表定义变更 报错,表定义变更 不报错,-v 3查看 报错,DDL操作没写redo log
DML操作影响数据一致性 non_InnoDB表不一致 non_InnoDB表不一致 默认一致,--trx-consistency-only会导致不一致 一致
其他       没有指定my.cnf导致备份一个错误的实例

对生产库的DDL操作、大事务、或者长时间锁表的操作,一定要避开备份时间(?ω?)

五、参考文档

mysqldump与innobackupex备份过程你知多少:http://www.cnblogs.com/xiaoboluo768/p/7560105.html
mysqlpump两个疑问点:http://www.fordba.com/mysqlpump_some_questions.html
MySQL 5.7 mysqlpump 备份工具说明:http://www.cnblogs.com/zhoujinyi/p/5684903.html
mysqlpump — A Database Backup Program:https://dev.mysql.com/doc/refman/5.7/en/mysqlpump.html
mydumper:https://github.com/maxbube/mydumper
MySQL备份之【mydumper 学习】:http://www.cnblogs.com/zhoujinyi/p/3423641.html
mydumper备份数据库详解(已详细说明):https://yq.aliyun.com/articles/45741
mydumper备份原理和使用方法:http://www.cnblogs.com/linuxnote/p/3817698.html

原文地址:https://www.cnblogs.com/Uest/p/8179082.html

时间: 2024-10-17 17:25:18

MySQL备份可能遇到的坑的相关文章

选择适用才最好 盘点MySQL备份方式

我们要备份什么? 一般情况下, 我们需要备份的数据分为以下几种 数据 二进制日志, InnoDB事务日志 代码(存储过程.存储函数.触发器.事件调度器) 服务器配置文件 备份工具 这里我们列举出常用的几种备份工具 mysqldump : 逻辑备份工具, 适用于所有的存储引擎, 支持温备.完全备份.部分备份.对于InnoDB存储引擎支持热备 cp, tar 等归档复制工具: 物理备份工具, 适用于所有的存储引擎, 冷备.完全备份.部分备份 lvm2 snapshot: 几乎热备, 借助文件系统管理

(转)MySQL备份原理详解

MySQL备份原理详解 原文:http://www.cnblogs.com/cchust/p/5452557.html 备份是数据安全的最后一道防线,对于任何数据丢失的场景,备份虽然不一定能恢复百分之百的数据(取决于备份周期),但至少能将损失降到最低.衡量备份恢复有两个重要的指标:恢复点目标(RPO)和恢复时间目标(RTO),前者重点关注能恢复到什么程度,而后者则重点关注恢复需要多长时间.这篇文章主要讨论MySQL的备份方案,重点介绍几种备份方式的原理,包括文件系统快照(LVM),逻辑备份工具M

mysql备份恢复详解

前言 为什么需要备份数据? 数据的备份类型 MySQL备份数据的方式 备份需要考虑的问题 设计合适的备份策略 实战演练 使用cp进行备份 使用mysqldump+复制BINARY LOG备份 使用lvm2快照备份数据 使用Xtrabackup备份 总结 前言 我们试着想一想, 在生产环境中什么最重要?如果我们服务器的硬件坏了可以维修或者换新, 软件问题可以修复或重新安装, 但是如果数据没了呢?这可能是最恐怖的事情了吧, 我感觉在生产环境中应该没有什么比数据跟更为重要. 那么我们该如何保证数据不丢

MySQL备份与增量备份

MySQL备份与增量备份 01)MySQL备份常用参数 -B 1.会备份创建和删除库的语句:2.备份多个库: --compact 去注释,适合调试 gzip 压缩:# mysqldump -uroot -pqb123 -S /data/3306/mysql.sock -F -B --master-data=2 mydb|gzip > /server/backup/bak_$(date +%F).sql.gz -A 备份所有库 -F 刷新binlog日志 --master-data=1/2 增加b

Mysql备份系列(4)--lvm-snapshot备份mysql数据(全量+增量)操作记录

Mysql最常用的三种备份工具分别是mysqldump.Xtrabackup(innobackupex工具).lvm-snapshot快照.前面分别介绍了:Mysql备份系列(1)--备份方案总结性梳理Mysql备份系列(2)--mysqldump备份(全量+增量)方案操作记录Mysql备份系列(3)--innobackupex备份mysql大数据(全量+增量)操作记录 废话不多说,下面即是使用lvm-snapshot快照方式备份mysql的操作记录,仅依据本人实验中使用而述. 操作记录:如下环

bat批处理实现mysql备份

一.新建bat文件,实现mysql备份功能.(即把txt文件后缀改为.bat) 代码解释: d:                                                                                    //选择d盘cd "D:\MySQL\MySQL Server 5.5\bin" //进入mysql安装目录下的bin文件set "Ymd=%date:~,4%%date:~5,2%%date:~8,2%"

Mysql备份和恢复

Mysql备份和恢复 备份 热备,是指对数据库运行中直接进行备份,对正在运行的数据库没有任何影响: 冷备,是指在数据库停止(关闭)下进行备份,只需拷贝相关的数据物理文件: 温备,是指对数据库运行中直接进行备份,但对当前数据库操作有所影响,很少用. 按备份出来的文件内容分为: 逻辑备份: 裸文件备份. 逻辑备份后文件内容是可读的,通常是文本文件,内容一般是Sql语句或表内的实际数据,如mysqldump或select * into outfile等方法.适用于数据库的升级.迁移,恢复时间较长. 裸

MySQL备份数据库脚本

Windows @echo offmysqldump --add-drop-database --add-drop-table -u root -p123456 dbname > d:\db_backup\%date:~0,4%-%date:~5,2%-%date:~8,2%_%time:~0,2%-%time:~3,2%-%time:~6,2%_dbname.sql Linux #!/bin/shcd /usr/binDATE=`date +%Y-%m-%d`# 备份dbname数据库/usr

mysql 备份类型

如果数据库已经安装在raid01 或者raid10上面,那我们的数据库是否还需要备份 答:需要的,因为我们的raid只是为了保证硬件损坏而不会中断业务.但是软件上的故障或者人为删除数据库.raid是无法对数据进行还原的.因为你删除了的时候,假如是在raid上面,也会删除掉两份数据.因此数据库备份和raid 是两个不同层次的概念. mysql备份如何进行,以及应该备份哪些数据? 备份类型: 我们的数据库非常繁忙,每一秒都有读写操作.假如你执行一个拷贝命令,那你只能拷贝出来当时那个时间段的数据,后面