记一次mysql事故---纪念逝去的一上午

虚拟机关机后第二天mysql起不来,回想一下我关机前和关机后的操作发现:关机前没关闭mysqld服务就直接init 0了,关机后将虚拟机内存由1G降到724M。笔者保证再也做过别的骚操作了。

2017-09-05 09:19:21 1940 [Note] Plugin ‘FEDERATED‘ is disabled.
2017-09-05 09:19:21 1940 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-09-05 09:19:21 1940 [Note] InnoDB: The InnoDB memory heap is disabled
2017-09-05 09:19:21 1940 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-09-05 09:19:21 1940 [Note] InnoDB: Memory barrier is not used
2017-09-05 09:19:21 1940 [Note] InnoDB: Compressed tables use zlib 1.2.3
2017-09-05 09:19:21 1940 [Note] InnoDB: Using CPU crc32 instructions
2017-09-05 09:19:21 1940 [Note] InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
2017-09-05 09:19:21 1940 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2017-09-05 09:19:21 1940 [ERROR] Plugin ‘InnoDB‘ init function returned error.
2017-09-05 09:19:21 1940 [ERROR] Plugin ‘InnoDB‘ registration as a STORAGE ENGINE failed.
2017-09-05 09:19:21 1940 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-09-05 09:19:21 1940 [ERROR] Aborting

2017-09-05 09:19:21 1940 [Note] Binlog end
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘partition‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_DATAFILES‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_TABLESPACES‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN_COLS‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_FIELDS‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_COLUMNS‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_INDEXES‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_TABLESTATS‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_TABLES‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_INDEX_TABLE‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_INDEX_CACHE‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_CONFIG‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_BEING_DELETED‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_DELETED‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_DEFAULT_STOPWORD‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_METRICS‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_BUFFER_POOL_STATS‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE_LRU‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX_RESET‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMPMEM_RESET‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMPMEM‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMP_RESET‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMP‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_LOCK_WAITS‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_LOCKS‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_TRX‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘PERFORMANCE_SCHEMA‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘ARCHIVE‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘BLACKHOLE‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘MyISAM‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘MRG_MYISAM‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘CSV‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘MEMORY‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘sha256_password‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘mysql_old_password‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘mysql_native_password‘
2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘binlog‘
2017-09-05 09:19:21 1940 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete

日志

2017-09-05 09:19:21 1940 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2017-09-05 09:19:21 1940 [ERROR] Plugin ‘InnoDB‘ init function returned error.
2017-09-05 09:19:21 1940 [ERROR] Plugin ‘InnoDB‘ registration as a STORAGE ENGINE failed.
2017-09-05 09:19:21 1940 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-09-05 09:19:21 1940 [ERROR] Aborting

貌似是innodb引擎错误,还有就是分配内存失败,内存分配失败会导致innodb错误吗,还是断电导致的?不得而知!!!

百度答案花样百出,最后试着在/etc/my.cnf中加入下列一行再重启

tmp_table_size=500M

***************************************

PS:下列谷歌出来的方法笔者没试过,可以一试:

主要是innodb日志大小不对造成的
innodb_buffer_pool_size=512M
innodb_log_file_size=128M
把以上几个参数调大点,然后把ibdata1,ib_logfile0,ib_logfile1这三个文件删除后,再重启服务就好了。
如果还是不行,在my.ini文件里还需要加入tmpdir="usr/tmp"这样的路径,因为innodb还需要一个临时的文件缓存区。

没试过的方法

****************************************

事实证明没用,该句只是用来设置内部(内存中)临时表的最大大小

重启后仍然报错,网上解决方法一般是:

问题描述:

非正常关闭mysql,同时更改了my.cnf 导致启动时不支持innodb,出现如下错误:

[ERROR] Plugin ‘InnoDB’ init function returned error.

[ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.

[ERROR] Unknown/unsupported table type: InnoDB

[ERROR] Aborting

解决办法:

删除log文件和data的初始化文件  

重新运行mysql_install_db
再启动

网上某解决方案

笔者并没有删除文件,只是将/data/dbdata下的三个文件移动到/tmp下备份了,万一以后用得着呢?

三个文件:ibdata1、ib_logfile0、ib_logfile1

再用mysql-install-db初始化

但是这一系列操作后mysqld仍然启动不了,直接关机,将虚拟机内存调回1G,没想到居然好了,可以成功启动mysqld服务啦!

##########

但是这还不够,进入数据库后出现了奇怪的一幕:数据库和表都在,却打不开,什么鬼

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| binlog             |
| log                |
| mysql              |
| performance_schema |
| test               |
| zabbix             |
+--------------------+
7 rows in set (0.00 sec)

mysql> use zabbix;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------------------+
| Tables_in_zabbix           |
+----------------------------+
| acknowledges               |
| actions                    |
| alerts                     |
| application_discovery      |
| application_prototype      |
| application_template       |
| applications               |
| auditlog                   |
| auditlog_details           |
| autoreg_host               |
| conditions                 |
| config                     |
| corr_condition             |
| corr_condition_group       |
| corr_condition_tag         |
| corr_condition_tagpair     |
| corr_condition_tagvalue    |
| corr_operation             |
| correlation                |
| dbversion                  |
| dchecks                    |
| dhosts                     |
| drules                     |
| dservices                  |
| escalations                |
| event_recovery             |
| event_tag                  |
| events                     |
| expressions                |
| functions                  |
| globalmacro                |
| globalvars                 |
| graph_discovery            |
| graph_theme                |
| graphs                     |
| graphs_items               |
| group_discovery            |
| group_prototype            |
| groups                     |
| history                    |
| history_log                |
| history_str                |
| history_text               |
| history_uint               |
| host_discovery             |
| host_inventory             |
| hostmacro                  |
| hosts                      |
| hosts_groups               |
| hosts_templates            |
| housekeeper                |
| httpstep                   |
| httpstepitem               |
| httptest                   |
| httptestitem               |
| icon_map                   |
| icon_mapping               |
| ids                        |
| images                     |
| interface                  |
| interface_discovery        |
| item_application_prototype |
| item_condition             |
| item_discovery             |
| items                      |
| items_applications         |
| maintenances               |
| maintenances_groups        |
| maintenances_hosts         |
| maintenances_windows       |
| mappings                   |
| media                      |
| media_type                 |
| opcommand                  |
| opcommand_grp              |
| opcommand_hst              |
| opconditions               |
| operations                 |
| opgroup                    |
| opinventory                |
| opmessage                  |
| opmessage_grp              |
| opmessage_usr              |
| optemplate                 |
| problem                    |
| problem_tag                |
| profiles                   |
| proxy_autoreg_host         |
| proxy_dhistory             |
| proxy_history              |
| regexps                    |
| rights                     |
| screen_user                |
| screen_usrgrp              |
| screens                    |
| screens_items              |
| scripts                    |
| service_alarms             |
| services                   |
| services_links             |
| services_times             |
| sessions                   |
| slides                     |
| slideshow_user             |
| slideshow_usrgrp           |
| slideshows                 |
| sysmap_element_url         |
| sysmap_url                 |
| sysmap_user                |
| sysmap_usrgrp              |
| sysmaps                    |
| sysmaps_elements           |
| sysmaps_link_triggers      |
| sysmaps_links              |
| task                       |
| task_close_problem         |
| timeperiods                |
| trends                     |
| trends_uint                |
| trigger_depends            |
| trigger_discovery          |
| trigger_tag                |
| triggers                   |
| users                      |
| users_groups               |
| usrgrp                     |
| valuemaps                  |
+----------------------------+
127 rows in set (0.00 sec)

mysql> desc valuemaps;
ERROR 1146 (42S02): Table ‘zabbix.valuemaps‘ doesn‘t exist
mysql> desc usrgrp;
ERROR 1146 (42S02): Table ‘zabbix.usrgrp‘ doesn‘t exist
mysql> select * from users;
ERROR 1146 (42S02): Table ‘zabbix.users‘ doesn‘t exist
mysql> select user();
+----------------+
| user()         |
+----------------+
| [email protected] |
+----------------+
1 row in set (0.00 sec)

mysql> 

数据库操作

日志文件中大量报错:

2017-09-05 13:57:26 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2017-09-05 13:57:31 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2017-09-05 13:57:41 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/innodb_index_stats from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/innodb_table_stats from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/slave_master_info from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/slave_relay_log_info from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/slave_worker_info from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2017-09-05 13:57:51 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2017-09-05 13:57:56 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.

cannot open file

各种表都不能访问,包括下列五个表:

innodb_table_stats
innodb_index_stats
slave_master_info
slave_relay_log_info
slave_worker_info 

之前备份过旧的ibdata1文件
虽然重新初始化之后,数据库会自动创建一个ibdata1文件,但是上述系统表也是innodb引擎,所以不能访问了.

关闭mysql服务后,尝试着将之前备份的ibdata1恢复。结果很神奇的解决了这个问题!数据库能成功读写了,但查看日志文件后发现还是有问题:

2017-09-05 12:33:11 3421 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-09-05 12:33:11 3421 [Note] InnoDB: The InnoDB memory heap is disabled
2017-09-05 12:33:11 3421 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-09-05 12:33:11 3421 [Note] InnoDB: Memory barrier is not used
2017-09-05 12:33:11 3421 [Note] InnoDB: Compressed tables use zlib 1.2.3
2017-09-05 12:33:11 3421 [Note] InnoDB: Using CPU crc32 instructions
2017-09-05 12:33:11 3421 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-09-05 12:33:11 3421 [Note] InnoDB: Completed initialization of buffer pool
2017-09-05 12:33:11 3421 [Note] InnoDB: Highest supported file format is Barracuda.
2017-09-05 12:33:11 3421 [Warning] InnoDB: The log sequence number in the ibdata files is higher than the log sequence number in the ib_logfiles! Are you sure you are using the right ib_logfiles to start up the database. Log sequence number in the ib_logfiles is 1600617, logsequence numbers stamped to ibdata file headers are between 3116912313 and 3116912313.
2017-09-05 12:33:11 3421 [Note] InnoDB: The log sequence numbers 3116912313 and 3116912313 in ibdata files do not match the log sequence number 1600617 in the ib_logfiles!
2017-09-05 12:33:11 3421 [Note] InnoDB: Database was not shutdown normally!
2017-09-05 12:33:11 3421 [Note] InnoDB: Starting crash recovery.
2017-09-05 12:33:11 3421 [Note] InnoDB: Reading tablespace information from the .ibd files...
2017-09-05 12:33:11 3421 [Note] InnoDB: Restoring possible half-written data pages
2017-09-05 12:33:11 3421 [Note] InnoDB: from the doublewrite buffer...
2017-09-05 12:33:11 7f30b98c47e0 InnoDB: Error: page 7 log sequence number 3116899374
InnoDB: is in the future! Current system log sequence number 1600617.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2017-09-05 12:33:11 7f30b98c47e0 InnoDB: Error: page 2 log sequence number 3116912313
InnoDB: is in the future! Current system log sequence number 1600617.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2017-09-05 12:33:11 7f30b98c47e0 InnoDB: Error: page 4 log sequence number 3116705336
InnoDB: is in the future! Current system log sequence number 1600617.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
。。。

日志超前  

时间: 2024-08-08 20:17:11

记一次mysql事故---纪念逝去的一上午的相关文章

【夯实Mysql基础】记一次mysql语句的优化过程!

1. [事件起因] 今天在做项目的时候,发现提供给客户端的接口时间很慢,达到了2秒多,我第一时间,抓了接口,看了运行的sql,发现就是 2个sql慢,分别占了1秒多. 一个sql是 链接了5个表同时使用了 2个 order by和 1个limit的分页 sql. 一个sql是上一个sql的count(*),即链接了5个表,当然没有limit了(取总数). 2. [着手优化] 1)[优化思路] 第一条是 做client调用 service层的数据缓存 第二条就是 优化sql本身. 这里着重讲一下

纪念逝去的岁月——C/C++排序二叉树

1.代码 2.运行结果 3.分析 1.代码 #include <stdio.h> #include <stdlib.h> typedef struct _Node { int value; struct _Node * pLeft; struct _Node * pRight; } Node; Node * getNewNode(int iValue) { Node * p = (Node *)malloc(sizeof(Node)); if(NULL != p) { p->

记一次MySql入库后,文本出现乱码的问题

最近采用ADO.NET开发了一个工具,解析了一条如下的日志并入库(MySql) 2014-08-19 00:37:10 [INFO] roleName=♣丶伊诺,orderId=1408190037102121039,price=10.0,point=100 发现入库后的roleName中的♣显示为□乱码了,一开始以为是表的编码问题,确认后是utf-8编码木有问题,后来又排除了文件编码及代码中的字符串都没有乱码的问题,凭经验分析,可能是连接字符串的问题吧,后来把MySql的连接字符串中也加入了u

由一次MySQL事故想到的

去年10月入职一家新公司,因公司内部某种不和谐因素造成上一个PHP程序员匆忙离职,相关工作都交接到我手里. 前面说到内部有不和谐的因素存在,因此相关的项目代码也是敷衍了事质量不高,也就为后续事故埋下了炸弹. 到了年底某天,客户的项目需要参加国家级某项评审,炸弹终于爆炸了. 打开该项目,所有的普通用户皆能正常访问,唯独只有官方评选小组的专家无法访问.更可怕的所有的专家只用同一个账号,且我们对专家账号一无所知. 客户炸了,急得不行了.连夜打电话到老板哪里,老板又半夜电话到了我.当晚,按照正常账号.正

记一次MySQL找回用户数据

事情经过 有天,我们公司外区的一个销售C说他8月3号以前的工作流记录找不到了.问清缘由,原来是更新了微信号(我们公司的工作流是基于企业微信开发的).经过分析,微信号和流程数据并没什么关系,所以初步得出结论:本来只需要更新微信号的,结果我们公司的流程系统管理员把用户先删除,再创建了新的用户. 解决过程 1.首先想到的是直接从定时备份数据里面找回原来的用户ID,结果发现系统只备份了十天的记录,而工作流系统上显示销售C只有8月3号以后的流程记录,距今已经40多天,从自动备份的数据里已经无法恢复. 2.

记一次mysql的preparedStatement使用超限问题

[现象&背景] 本服务是个为数据库的分库分表提供路由规则计算,sql过滤执行的中间服务.即上游将请求发给本服务,本服务根据分库分表规则将相应的sql执行发送给对应的分库.分表去执行,并整理结果返回给上游. 2016-07-14下午,上游流量切换后,mysql大量报出" Can't create more than max_prepared_stmt_count statements (current value: 16382)",mysql不能工作导致本层数据库路由服务不能工作

初码-Azure系列-记一次MySQL数据库向Azure的迁移

初码Azure系列文章目录 还在继续给客户迁移不同的系统到Azure,这一次是一个系统的MySQL数据库要迁移,将迁移过程记录一下 原系统环境 数据库版本:MySQL Community Edition 5.7 服务器环境:自建机房虚拟化环境,E7 4820 分配8核32G,主节点+非热备灾备节点 运维方式:每日夜间停止服务,ETL程序备份全库至灾备环境 数据规模:用户与业务表相关表,记录数据量在500万级别以内:日志表,记录数据量在5000万条以内,总数据存储量在50G Azure环境选择 考

记一次mysql数据恢复

确切的说更像是一次数据迁移. 背景介绍: 操作系统:Windows Server 2008 R2 数据库版本:MySQL 5.5 数据库的安装目录与数据文件目录不在同一个磁盘,数据文件所在的目录磁盘损坏.而后通过数据恢复工具恢复数据文件.前期研发的同事尝试启动恢复数据库,不成功,多轮尝试不成功后找到我. 1.得到同事给的数据文件 ibdata1,Mysql安装目录MySQL\MySQL Server 5.5.调整my.ini文件尝试启动数据库. 2.将mysql base dir 拷贝到英文目录

【夯实Mysql基础】记一次mysql语句的优化过程

1. [事件起因] 今天在做项目的时候,发现提供给客户端的接口时间很慢,达到了2秒多,我第一时间,抓了接口,看了运行的sql,发现就是 2个sql慢,分别占了1秒多. 一个sql是 链接了5个表同时使用了 2个 order by和 1个limit的分页 sql. 一个sql是上一个sql的count(*),即链接了5个表,当然没有limit了(取总数). 2. [着手优化] 1)[优化思路] 第一条是 做client调用 service层的数据缓存 第二条就是 优化sql本身. 这里着重讲一下