Mysql千万级大表同步复制的一些问题

 1 [[email protected] ~]#   pt-online-schema-change  -u root -p xxxxxx -h localhost -A utf8 D=jinyuanbao,t=DistributionGoods --new-table-name DistributionGoods_0 --no-drop-trigger --no-swap-table --no-drop-new-table --execute
 2 No slaves found.  See --recursion-method if host iZ23eoaorbaZ has slaves.
 3 Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
 4 Operation, tries, wait:
 5   copy_rows, 10, 0.25
 6   create_triggers, 10, 1
 7   drop_triggers, 10, 1
 8   swap_tables, 10, 1
 9   update_foreign_keys, 10, 1
10 Altering `jinyuanbao`.`DistributionGoods`...
11 Creating new table...
12 Created new table jinyuanbao.DistributionGoods_0 OK.
13 2015-08-18T10:21:47 Creating triggers...
14 2015-08-18T10:21:47 Created triggers OK.
15 2015-08-18T10:21:47 Copying approximately 16259938 rows...
16 Copying `jinyuanbao`.`DistributionGoods`:   4% 11:05 remain
17 Copying `jinyuanbao`.`DistributionGoods`:   8% 11:18 remain
18 Copying `jinyuanbao`.`DistributionGoods`:  11% 12:02 remain
19 Copying `jinyuanbao`.`DistributionGoods`:  14% 12:16 remain
20 Copying `jinyuanbao`.`DistributionGoods`:  16% 12:18 remain
21 Copying `jinyuanbao`.`DistributionGoods`:  19% 12:25 remain
22 Copying `jinyuanbao`.`DistributionGoods`:  22% 12:05 remain
23 Copying `jinyuanbao`.`DistributionGoods`:  25% 11:44 remain
24 Copying `jinyuanbao`.`DistributionGoods`:  28% 11:07 remain
25 Copying `jinyuanbao`.`DistributionGoods`:  32% 10:27 remain
26 Copying `jinyuanbao`.`DistributionGoods`:  35% 09:50 remain
27 Copying `jinyuanbao`.`DistributionGoods`:  39% 09:13 remain
28 Copying `jinyuanbao`.`DistributionGoods`:  42% 08:37 remain
29 Copying `jinyuanbao`.`DistributionGoods`:  46% 08:02 remain
30 Copying `jinyuanbao`.`DistributionGoods`:  50% 07:27 remain
31 Copying `jinyuanbao`.`DistributionGoods`:  53% 06:54 remain
32 Copying `jinyuanbao`.`DistributionGoods`:  57% 06:24 remain
33 Copying `jinyuanbao`.`DistributionGoods`:  60% 05:51 remain
34 Copying `jinyuanbao`.`DistributionGoods`:  64% 05:19 remain
35 Copying `jinyuanbao`.`DistributionGoods`:  67% 04:47 remain
36 Copying `jinyuanbao`.`DistributionGoods`:  70% 04:19 remain
37 Copying `jinyuanbao`.`DistributionGoods`:  74% 03:48 remain
38 Copying `jinyuanbao`.`DistributionGoods`:  78% 03:14 remain
39 Copying `jinyuanbao`.`DistributionGoods`:  81% 02:47 remain
40 Copying `jinyuanbao`.`DistributionGoods`:  84% 02:18 remain
41 Copying `jinyuanbao`.`DistributionGoods`:  87% 01:48 remain
42 Copying `jinyuanbao`.`DistributionGoods`:  91% 01:18 remain
43 Copying `jinyuanbao`.`DistributionGoods`:  94% 00:48 remain
44 Copying `jinyuanbao`.`DistributionGoods`:  97% 00:20 remain
45 2015-08-18T10:38:03 Copied rows OK.
46 Not dropping old table because --no-drop-triggers was specified.
47 Not dropping triggers because --no-drop-triggers was specified.  To drop the triggers, execute:
48 DROP TRIGGER IF EXISTS `jinyuanbao`.`pt_osc_jinyuanbao_DistributionGoods_del`;
49 DROP TRIGGER IF EXISTS `jinyuanbao`.`pt_osc_jinyuanbao_DistributionGoods_upd`;
50 DROP TRIGGER IF EXISTS `jinyuanbao`.`pt_osc_jinyuanbao_DistributionGoods_ins`;
51 Not dropping the new table `jinyuanbao`.`DistributionGoods_0` because --no-drop-new-table was specified.  To drop the new table, execute:
52 DROP TABLE IF EXISTS `jinyuanbao`.`DistributionGoods_0`;
53 Successfully altered `jinyuanbao`.`DistributionGoods`.
54 [[email protected] ~]#     pt-online-schema-change  -u root -p xxxxxxx -h localhost -A utf8 D=jinyuanbao,t=DistributionGoods_0 --new-table-name DistributionGoods_1 --no-drop-trigger --no-swap-table --no-drop-new-table --execute
55 No slaves found.  See --recursion-method if host iZ23eoaorbaZ has slaves.
56 Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
57 Operation, tries, wait:
58   copy_rows, 10, 0.25
59   create_triggers, 10, 1
60   drop_triggers, 10, 1
61   swap_tables, 10, 1
62   update_foreign_keys, 10, 1
63 Altering `jinyuanbao`.`DistributionGoods_0`...
64 Creating new table...
65 Created new table jinyuanbao.DistributionGoods_1 OK.
66 2015-08-18T10:46:06 Creating triggers...
67 2015-08-18T10:46:06 Created triggers OK.
68 2015-08-18T10:46:06 Copying approximately 16113957 rows...
69 Copying `jinyuanbao`.`DistributionGoods_0`:   4% 11:21 remain
70 Copying `jinyuanbao`.`DistributionGoods_0`:   8% 11:08 remain
71 Copying `jinyuanbao`.`DistributionGoods_0`:  12% 10:40 remain
72 Copying `jinyuanbao`.`DistributionGoods_0`:  16% 10:17 remain
73 Copying `jinyuanbao`.`DistributionGoods_0`:  19% 10:02 remain
74 Copying `jinyuanbao`.`DistributionGoods_0`:  23% 09:48 remain
75 Copying `jinyuanbao`.`DistributionGoods_0`:  27% 09:16 remain
76 Copying `jinyuanbao`.`DistributionGoods_0`:  30% 08:54 remain
77 Copying `jinyuanbao`.`DistributionGoods_0`:  34% 08:31 remain
78 Copying `jinyuanbao`.`DistributionGoods_0`:  37% 08:11 remain
79 Copying `jinyuanbao`.`DistributionGoods_0`:  41% 07:51 remain
80 Copying `jinyuanbao`.`DistributionGoods_0`:  44% 07:27 remain
81 Copying `jinyuanbao`.`DistributionGoods_0`:  48% 07:02 remain
82 Copying `jinyuanbao`.`DistributionGoods_0`:  51% 06:40 remain
83 Copying `jinyuanbao`.`DistributionGoods_0`:  54% 06:16 remain
84 Copying `jinyuanbao`.`DistributionGoods_0`:  58% 05:47 remain
85 Not dropping triggers because --no-drop-triggers was specified.  To drop the triggers, execute:
86 DROP TRIGGER IF EXISTS `jinyuanbao`.`pt_osc_jinyuanbao_DistributionGoods_0_del`;
87 DROP TRIGGER IF EXISTS `jinyuanbao`.`pt_osc_jinyuanbao_DistributionGoods_0_upd`;
88 DROP TRIGGER IF EXISTS `jinyuanbao`.`pt_osc_jinyuanbao_DistributionGoods_0_ins`;
89 Not dropping the new table `jinyuanbao`.`DistributionGoods_1` because --no-drop-new-table was specified.  To drop the new table, execute:
90 DROP TABLE IF EXISTS `jinyuanbao`.`DistributionGoods_1`;
91 `jinyuanbao`.`DistributionGoods_0` was not altered.
92 2015-08-18T10:54:31 Error copying rows from `jinyuanbao`.`DistributionGoods_0` to `jinyuanbao`.`DistributionGoods_1`: Threads_running=58 exceeds its critical threshold 50

2015-08-18T10:54:31 Error copying rows from `jinyuanbao`.`DistributionGoods_0` to `jinyuanbao`.`DistributionGoods_1`: Threads_running=58 exceeds its critical threshold 50原始表2GB，复制出DistributionGoods_0,DistributionGoods_1表,依次安装触发器。每次chunk操作前后，会根据show global status统计指定的状态量的变化，默认是统计Thread_running。目的是为了安全，防止原始表上的触发器引起负载过高。这也是为了防止在线DDL对线上的影响。超过设置的阀值，就会终止操作，在线DDL就会中断。提示的异常如上报错信息。

时间： 2024-11-01 09:50:00

Mysql千万级大表同步复制的一些问题的相关文章

Mysql千万级大表优化

Mysql的单张表的最大数据存储量尚没有定论,一般情况下mysql单表记录超过千万以后性能会变得很差.因此,总结一些相关的Mysql千万级大表的优化策略. 1.优化sql以及索引 1.1优化sql 1.有索引但未被用到的情况(不建议) (1)避免like的参数以通配符开头时尽量避免Like的参数以通配符开头,否则数据库引擎会放弃使用索引而进行全表扫描. 以通配符开头的sql语句,例如:select * from t_credit_detail where Flistid like '%0'\G

【优化】MySQL千万级大表优化解决方案

问题概述使用阿里云rds for MySQL数据库(就是MySQL5.6版本),有个用户上网记录表6个月的数据量近2000万,保留最近一年的数据量达到4000万,查询速度极慢,日常卡死.严重影响业务. 问题前提:老系统,当时设计系统的人大概是大学没毕业,表设计和sql语句写的不仅仅是垃圾,简直无法直视.原开发人员都已离职,到我来维护,这就是传说中的维护不了就跑路,然后我就是掉坑的那个!!! 我尝试解决该问题,so,有个这个日志. 方案概述方案一:优化现有mysql数据库.优点:不影响现有业务

MYSQL千万级数据表，创建表及字段扩展的几条建议

MYSQL千万级数据表,创建表及字段扩展的几条建议一:概述当我们设计一个系统时,需要考虑到系统的运行一段时间后,表里数据量大约有多少,如果在初期,就能估算到某几张表数据量非常庞大时(比如聊天消息表),就要把表创建好,这篇文章从创建表,增加数据,以及字段扩展,这几个方面来给出建议. 二:创建表假如现在我们需要创建IM项目中的聊天消息表,这个表数据量大,读操作远超过写操作,我们都知道,mysql常用的数据库引擎主要有innodb,myisam,这两个数据库引擎主要区别是,innodb支持事务,

30个MySQL千万级大数据SQL查询优化技巧详解

本文总结了30个mysql千万级大数据SQL查询优化技巧,特别适合大数据里的MYSQL使用. 1.对查询进行优化,应尽量避免全表扫描,首先应考虑在 where 及 order by 涉及的列上建立索引. 2.应尽量避免在 where 子句中对字段进行 null 值判断,否则将导致引擎放弃使用索引而进行全表扫描,如:select id from t where num is null可以在num上设置默认值0,确保表中num列没有null值,然后这样查询:select id from t wher

Mysql千万级大数据查询优化经验一点课堂(多岸学院)

提高mysql千万级大数据SQL查询优化30条经验(Mysql索引优化注意) 1.对查询进行优化,应尽量避免全表扫描,首先应考虑在 where 及 order by 涉及的列上建立索引. 2.应尽量避免在 where 子句中对字段进行 null 值判断,否则将导致引擎放弃使用索引而进行全表扫描,如:select id from t where num is null可以在num上设置默认值0,确保表中num列没有null值,然后这样查询:select id from t where num=0

MySQL千万级多表关联SQL语句调优

本文不涉及复杂的底层数据结构,通过explain解释SQL,并根据可能出现的情况,来做具体的优化,使千万级表关联查询第一页结果能在2秒内完成(真实业务告警系统优化结果). 需要优化的查询:使用explain 出现了Using temporary: 有分页时出现了Using filesort则表示使用不了索引,需要根据下面的技巧来调整语句 rows过多,或者几乎是全表的记录数: key 是 (NULL): possible_keys 出现过多(待选)索引. 1.使用explain语法,对SQL进行

提高mysql千万级大数据SQL查询优化30条经验

1.对查询进行优化,应尽量避免全表扫描,首先应考虑在 where 及 order by 涉及的列上建立索引. 2.应尽量避免在 where 子句中对字段进行 null 值判断,否则将导致引擎放弃使用索引而进行全表扫描,如:select id from t where num is null可以在num上设置默认值0,确保表中num列没有null值,然后这样查询:select id from t where num=0 3.应尽量避免在 where 子句中使用!=或<>操作符,否则引擎将放弃使用

提高mysql千万级大数据SQL查询优化30条经验（Mysql索引优化注意）

转自http://blog.163.com/zhangjie_0303/blog/static/9908270620146951355834/ 1.对查询进行优化,应尽量避免全表扫描,首先应考虑在 where 及 order by 涉及的列上建立索引. 2.应尽量避免在 where 子句中对字段进行 null 值判断,否则将导致引擎放弃使用索引而进行全表扫描,如: select id from t where num is null 可以在num上设置默认值0,确保表中num列没有null值,然