手动创建binary log files和手动编辑binary log index file会有什么影响

一、了解Binary Log结构

1.1、High-Level Binary Log Structure and Contents

• Binlog包括binary log files和index file
• 每个binary log文件的前4字节是Magic Number，紧接着是一组描述数据修改的Events
　　• The magic number bytes are 0xfe 0x62 0x69 0x6e = 0xfe ‘b‘‘i‘‘n‘
　　• 每个Event包括header bytes和data bytes
　　　　header bytes：Event类型，什么时候，由哪个server生成等信息
　　　　data bytes：特定的数据修改
　　• 第一个Event是Description Event，描述文件的格式版本
　　• 其他Events按照binlog_version(1、3、4)进行记录
　　• 最后一个Event是Log-rotation Event，指定下一个binary log的文件名称
• index文件里面是当前binary log的一个列表

1.2、Binary Log结构示意图

# Binary Log结构示意图(binlog_version=4)
               |<----------------------------------Events---------------------------------->|
               |<---------Fisrt Event--------->|<------Mid Events----->|<---Final Event---->|
+--------------+-------------------------------+--------+-----+--------+--------------------+
| Magic Number | FDE(Format Description Event) | Event1 | ... | EventN | Log-rotation Event |
+--------------+-------------------------------+--------+-----+--------+--------------------+
               |<-event header->|<-event data->|


上图是binlog_version=4(MySQL 5.0 and up)的示意图

二、index file添加一行不存在的binary log记录

查看当前binary log

# 查看当前binary log
[[email protected] logs]# ll
total 20
-rw-r-----. 1 mysql mysql 201 Jan 12 11:32 mysql-bin.000004
-rw-r-----. 1 mysql mysql 201 Jan 12 11:32 mysql-bin.000005
-rw-r-----. 1 mysql mysql 177 Jan 12 11:34 mysql-bin.000006
-rw-r-----. 1 mysql mysql 177 Jan 12 11:37 mysql-bin.000007
-rw-r-----. 1 mysql mysql 220 Jan 12 11:34 mysql-bin.index
[[email protected] logs]# cat mysql-bin.index
/data/mysql/mysql3308/logs/mysql-bin.000004
/data/mysql/mysql3308/logs/mysql-bin.000005
/data/mysql/mysql3308/logs/mysql-bin.000006
/data/mysql/mysql3308/logs/mysql-bin.000007
[[email protected] logs]#

mysql-bin.index 添加一行已经删除的记录/data/mysql/mysql3308/logs/mysql-bin.000001，重新启动数据库服务

# 查看error log
[[email protected] ~]# more /data/mysql/mysql3308/data/error.log
...
2018-01-12T03:34:33.467474Z 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool
mysqld: File ‘/data/mysql/mysql3308/logs/mysql-bin.000001‘ not found (Errcode: 2 - No such file or directory)
2018-01-12T03:34:33.546276Z 0 [ERROR] Failed to open log (file ‘/data/mysql/mysql3308/logs/mysql-bin.000001‘, errno 2)
2018-01-12T03:34:33.546294Z 0 [ERROR] Could not open log file
2018-01-12T03:34:33.563485Z 0 [Warning] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2018-01-12T03:34:33.563528Z 0 [Note] Server hostname (bind-address): ‘*‘; port: 3308
数据库服务最终启动起来

mysql-bin.index 添加一行未来的记录/data/mysql/mysql3308/logs/mysql-bin.000017，重新启动数据库服务

# 查看error log
[[email protected] ~]# more /data/mysql/mysql3308/data/error.log
...
2018-01-12T03:44:45.921370Z 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool
mysqld: File ‘/data/mysql/mysql3308/logs/mysql-bin.000017‘ not found (Errcode: 2 - No such file or directory)
2018-01-12T03:44:46.017490Z 0 [ERROR] Failed to open log (file ‘/data/mysql/mysql3308/logs/mysql-bin.000017‘, errno 2)
2018-01-12T03:44:46.017509Z 0 [ERROR] Could not open log file
2018-01-12T03:44:46.017532Z 0 [ERROR] Can not init tc log
2018-01-12T03:44:46.017545Z 0 [ERROR] Aborting
数据库服务最终退出

启动过程会读取index file中的第一条和最后一条记录，然后检查对应的文件是否存在。如果第一条不存在，error log中会有报错，但服务正常启动；如果最后一条不存在，error log中会有报错，服务退出。即使你打乱index file中列表顺序，或者写入其他信息，它始终只验证index file的第一条和最后一条~

三、手动创建一个binary log，但不添加到index file

如果touch_seq < max(exists_seq)，正常启动，最新的new_seq=max(exists_seq)+1
如果max(exists_seq) < touch_seq < pow(2,31)-1，正常启动，最新的new_seq=touch_seq+1，当new_seq > pow(2,31)-1001 时会往error log中写入警告
如果touch_seq = pow(2,31)-1，mysqld退出
如果touch_seq > pow(2,31)-1，正常启动，会往error log中写入警告

3.1、MySQL binlog后面的编号最大是多大

最大编号参考文章：MySQL binlog后面的编号最大是多大

原文结论：
1、MySQL binlog的最大sequence是：pow(2,31)-1 = 2147483647
2、当 pow(2,31)-1001 < sequence < pow(2,31)-1 会往error log中写入警告
3、binlog的sequence达到最大值(2147483647)时，不管有没有mysql-bin.000001类似这样的文件，mysqld都是退出
4、在MySQL产生binlog时会读取当前日件文目录下的log-bin的base name获取下一个日志文件的后面的Seq。所以日志目录下文件太多，会影响MySQL的启动及日志切换。这里也有一个大的隐患，运行中给放一个较大的日志文件，在下次日志文件切换时有可能很快就接近于最大值，造成mysqld crash退出
5、一定要监控error log的输出，并足够重视

查看源码

# 源码信息 sql/binlog.cc
#define MAX_LOG_UNIQUE_FN_EXT 0x7FFFFFFF
#define LOG_WARN_UNIQUE_FN_EXT_LEFT 1000

  /* check if reached the maximum possible extension number */
  if (max_found == MAX_LOG_UNIQUE_FN_EXT)
  {
    sql_print_error("Log filename extension number exhausted: %06lu. \
Please fix this by archiving old logs and updating the index files.", max_found);
    error= 1;
    goto end;
  }

  /* print warning if reaching the end of available extensions. */
  if ((next > (MAX_LOG_UNIQUE_FN_EXT - LOG_WARN_UNIQUE_FN_EXT_LEFT)))
    sql_print_warning("Next log extension: %lu. \
Remaining log filename extensions: %lu. Please consider archiving some logs.", next, (MAX_LOG_UNIQUE_FN_EXT - next));

mysql> select conv(‘7FFFFFFF‘,16,10);
+------------------------+
| conv(‘7FFFFFFF‘,16,10) |
+------------------------+
| 2147483647             |
+------------------------+

源码逻辑正是上面的结论1、2、3。如果我们的binlog的sequence疯涨到2147483647，那很不幸mysqld会退出。。。但如果我们在正常情况下，touch一个大于2147483647的binary log的文件会出现什么情况呢？

3.2、MySQL binlog后面的编号最大到底是多大

首先我们从2147483648开始

# reset master
mydb[email protected]192.168.85.132,3308 [replcrash]> reset master;
Query OK, 0 rows affected (0.01 sec)
# 查看binary log
[email protected]192.168.85.132,3308 [replcrash]> show binary logs;
+------------------+-----------+
| Log_name         | File_size |
+------------------+-----------+
| mysql-bin.000001 |       154 |
+------------------+-----------+
row in set (0.00 sec)
# touch 2^31
[[email protected] logs]# touch mysql-bin.2147483648
[[email protected] logs]# chown -R mysql:mysql *
# flush执行5次
[email protected]192.168.85.132,3308 [replcrash]> flush binary logs;
Query OK, 0 rows affected (0.01 sec)
# 查看binary log
[email protected]192.168.85.132,3308 [replcrash]> show binary logs;
+----------------------+-----------+
| Log_name             | File_size |
+----------------------+-----------+
| mysql-bin.000001     |       205 |
| mysql-bin.2147483649 |       205 |
| mysql-bin.2147483650 |       205 |
| mysql-bin.2147483651 |       205 |
| mysql-bin.2147483652 |       205 |
| mysql-bin.2147483653 |       154 |
+----------------------+-----------+
rows in set (0.00 sec)
# 查看error log
[[email protected] data]# cat error.log
2018-01-08T09:09:06.235149Z 3 [Warning] Next log extension: 2147483649. Remaining log filename extensions: 18446744073709551614. Please consider archiving some logs.
2018-01-08T09:09:23.868883Z 3 [Warning] Next log extension: 2147483650. Remaining log filename extensions: 18446744073709551613. Please consider archiving some logs.
2018-01-08T09:10:29.670811Z 3 [Warning] Next log extension: 2147483651. Remaining log filename extensions: 18446744073709551612. Please consider archiving some logs.
2018-01-08T09:10:30.401541Z 3 [Warning] Next log extension: 2147483652. Remaining log filename extensions: 18446744073709551611. Please consider archiving some logs.
2018-01-08T09:10:31.075702Z 3 [Warning] Next log extension: 2147483653. Remaining log filename extensions: 18446744073709551610. Please consider archiving some logs.
[[email protected] logs]# 

# 再次reset master
[email protected]192.168.85.132,3308 [replcrash]> reset master;
Query OK, 0 rows affected (0.02 sec)
# 查看binary log
[email protected]192.168.85.132,3308 [replcrash]> show binary logs;
+----------------------+-----------+
| Log_name             | File_size |
+----------------------+-----------+
| mysql-bin.2147483649 |       154 |
+----------------------+-----------+
row in set (0.00 sec)

可以看到，执行flush binary logs后，sequence一直往后+1(next)。error log中会有警告日志，next > (MAX_LOG_UNIQUE_FN_EXT - LOG_WARN_UNIQUE_FN_EXT_LEFT)恒成立；Remaining log filename extensions后面的数值看着有点怪异~
第二次reset后的sequence从2147483649开始，并不是从000001开始，不知道什么原因。
我们继续加大sequence，直到接近9223372036854775808

# touch 2^63-2
[[email protected] logs]# touch mysql-bin.9223372036854775806
[[email protected] logs]# chown -R mysql:mysql *
# flush执行5次
[email protected]192.168.85.132,3308 [replcrash]> flush binary logs;
Query OK, 0 rows affected (0.01 sec)
# 查看binary log
[email protected]192.168.85.132,3308 [replcrash]> show binary logs;
+-------------------------------+-----------+
| Log_name                      | File_size |
+-------------------------------+-----------+
| mysql-bin.000001              |       214 |
| mysql-bin.9223372036854775807 |       214 |
| mysql-bin.9223372036854775808 |       154 |
| mysql-bin.9223372036854775808 |       154 |
| mysql-bin.9223372036854775808 |       154 |
| mysql-bin.9223372036854775808 |       154 |
+-------------------------------+-----------+
rows in set (0.00 sec)
# 查看error log
[[email protected] data]# cat error.log
2018-01-08T09:35:20.879612Z 3 [Warning] Next log extension: 9223372036854775807. Remaining log filename extensions: 9223372039002259456. Please consider archiving some logs.
2018-01-08T09:35:22.082530Z 3 [Warning] Next log extension: 9223372036854775808. Remaining log filename extensions: 9223372039002259455. Please consider archiving some logs.
2018-01-08T09:35:22.934636Z 3 [Warning] Next log extension: 9223372036854775808. Remaining log filename extensions: 9223372039002259455. Please consider archiving some logs.
2018-01-08T09:35:23.544604Z 3 [Warning] Next log extension: 9223372036854775808. Remaining log filename extensions: 9223372039002259455. Please consider archiving some logs.
2018-01-08T09:35:24.134664Z 3 [Warning] Next log extension: 9223372036854775808. Remaining log filename extensions: 9223372039002259455. Please consider archiving some logs.
[[email protected] logs]#

当sequence达到9223372036854775808后，再flush binary log，看不出产生新binary log~

[[email protected] logs]# mysqlbinlog -v --base64-output=decode-rows mysql-bin.9223372036854775808
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @[email protected]@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#180112 15:57:18 server id 1323308  end_log_pos 123 CRC32 0x798b200f    Start: binlog v 4, server v 5.7.19-log created 180112 15:57:18
# Warning: this binlog is either in use or was not closed properly.
# at 123
#180112 15:57:18 server id 1323308  end_log_pos 154 CRC32 0xf3252c32    Previous-GTIDs
# [empty]
# at 154
#180112 15:57:18 server id 1323308  end_log_pos 214 CRC32 0x6054f256    Rotate to mysql-bin.9223372036854775808  pos: 4
SET @@SESSION.GTID_NEXT= ‘AUTOMATIC‘ /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET [email protected]_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
[[email protected] logs]#

解析binlog可以在末尾看到Rotate to mysql-bin.9223372036854775808  pos: 4
也就是说再怎么切，也只能切到9223372036854775808，这个文件感觉有点不太正常，不受max_binlog_size限制，解析出的日志莫名不完整~
恢复原始：手工删除 mysql-bin.2147483647 这类大文件，然后修改 mysql-bin.index；或者直接清空logs文件夹，再重新启动数据库服务

四、参考文档

MySQL binlog后面的编号最大是多大：http://wubx.net/mysql-binlog-max-sequence/
High-Level Binary Log Structure and Contents：https://dev.mysql.com/doc/internals/en/binary-log-structure-and-contents.html
Binary Log Versions：https://dev.mysql.com/doc/internals/en/binary-log-versions.html
Event Structure：https://dev.mysql.com/doc/internals/en/event-structure.html

原文地址：https://www.cnblogs.com/Uest/p/8244050.html

时间： 2024-11-05 18:51:50

手动创建binary log files和手动编辑binary log index file会有什么影响的相关文章

如何手动创建oracle数据库

下面的实验室是如何不要通过DBCA创建ORACLE 数据库,而是通过ORACLE ONLINE HELP DOCUMENT进行手动的创建数据库的详细步骤: 1,编辑Oracle profile [[email protected] ~]# su - oracle [[email protected] ~]$ vi ./.bash_profile PATH=$PATH:$HOME/bin export PATH export ORACLE_BASE=/u01/app/oracle export O

MySQL log files &nbsp; (1)

MySQL has several log files that can help you find out what activity is taking place. log type      information write to log file error log  Problems encoutered  starting, running or stopping mysqld General query  log      Established client connecti

Oracle log files Troubleshooting Oracle

Oracle log files The following Oracle log files are helpful for resolving issues with Oracle components: Table: Log files for Oracle RAC 10g Release 2 Table: Log files for Oracle RAC 11g Release 2 and later versions Table: Log files for Oracle databa

怎样手动创建oracle数据库

以下的实验室是怎样不要通过DBCA创建ORACLE 数据库,而是通过ORACLE ONLINE HELP DOCUMENT进行手动的创建数据库的具体步骤: 1,编辑Oracle profile [[email protected] ~]# su - oracle [[email protected] ~]$ vi ./.bash_profile PATH=$PATH:$HOME/bin export PATH export ORACLE_BASE=/u01/app/oracle export O

CentOS 7 静默安装oracle 11R2 ，并手动创建数据库 createdatabase

官方文档:http://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm#BHCCADGD 需要的安装包: binutils-2.23.52.0.1-12.el7.x86_64 compat-libcap1-1.10-3.el7.x86_64 compat-libstdc++-33-3.2.3-71.el7.i686 compat-libstdc++-33-3.2.3-71.el7.x86_64 gcc-4.8.2-3.el7.x86

10_Oracle_Admin_手动创建一个比较实用的数据库

上一节中手动创建数据库由于参数太过简单,且没有安装数据字典,是无法实际应用的,本章节是对上节内容的深入,增加了很多细节和参数配置,按照以下步骤,可以成功安装一个能实际使用的数据库. ======清理现有数据库文件====== [[email protected] dbs]$ ll total 240396 -rw-r-----. 1 oracle oinstall  7847936 Aug 28 11:43 cntrlwly.dbf -rw-r-----. 1 oracle oinstall 8

Linux手动创建Swap

Swap是Linux下的交换分区,类似Windows的虚拟内存,当物理内存不足时,系统可把一些内存中不常用到的程序放入Swap,解决物理内存不足的情况. 若系统安装时开辟的Swap空间太小,可通过手动创建Swap文件. 一.创建文件 dd if=/dev/zero of=/root/swapfile bs=1024 count=1024000 SSH执行以上命令,创建一个名为kwxgd的1G 空文件(写0占用磁盘). 二.制作为Swap文件 mkswap kwxgd SSH执行以上命令,将生成的

ubuntu下手动创建应用的快捷方式

ubuntu下手动创建应用的快捷方式 在使用ubuntu开发的时候总是难免要安装一些开发类 的软件, 那么来谈下ubuntu下几种软件的安装方式: 1.★★★★★ 通过命令安装的,如 apt-get,apt-repository,ppa等,这种没有任何问题,最方便最好用,稳定性也最好,只是需要提前知道某个软件的包名,不知道的话就没得玩,没法安装,比较蛋疼.虽然可以通过apt-cache search 命令搜索包名,但是还是要知道包名,而不是包显示的名字,蛋疼.软件中心有个应用可以下载下来,叫做"

docker仓库（手动创建）

实验需求: 创建自己的仓库----镜像仓库 实验环境: 两台centos7虚拟机 实验步骤: 说明: docker.benet.com这是docker registry服务器的主机名称,ip是192.168.1.107;因为https的SSL证书要用到主机名,所以要设置主机名. dockerregistry 服务器作为处理docker镜像的最终上传和下载,用的是官方的镜像registry. nginx 1.6.x 是一个用nginx作为反向代理服务器 注:关闭selinux 挂载centos7系