实验环境:
vm1-ning:172.16.3.1/16
vm2-hong:172.16.3.2/16
drbd操作步骤:
1、创建mysql的drbd资源磁盘(两节点都需要创建)
[[email protected] ~]# fdisk /dev/sda
n
p
+5G
w
[[email protected] ~]# kpartx -af /dev/sda
[[email protected] ~]# partx -a /dev/sda
2、drbd设置(全局配置文件这里和web高可用中的一样没有改变,不再次说明。)
[[email protected] ~]# cd /etc/drbd.d
[[email protected] drbd.d]# ls
global_common.conf global_common.conf.bak global_common.conf.rpmsave web.res
[[email protected] drbd.d]# cp web.res mydata.res
[[email protected] drbd.d]# vim mydata.res 定义资源名称
[[email protected] drbd.d]# cat mydata.res
resource mydata { 定义资源名称也是唯一的,不能相同
on ning {
device /dev/drbd1; 指定drdb设备名称为drbd1
disk /dev/sda4; 定义的分区磁盘
address 172.16.3.1:7790; 两个资源之间不能使用相同的端口号
meta-disk internal;
}
on hong {
device /dev/drbd1;
disk /dev/sda4;
address 172.16.3.2:7790;
meta-disk internal;
}
}
[[email protected] drbd.d]# scp mydata.res 172.16.3.1:/etc/drbd.d/ 双方节点上都需要配置这样复制过去即可
[email protected]‘s password:
mydata.res
100% 278 0.3KB/s 00:00
3、初始化drbd在两节点上都需要初始化,并启动drbd
[[email protected] drbd.d]# drbdadm create-md mydata
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
[[email protected] ~]# drbdadm create-md mydata
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
[[email protected] drbd.d]# service drbd start
[[email protected] ~]# service drbd start
4、提升一个节点为主节点进行分区数据同步和格式化分区
[[email protected] drbd.d]# drbd-overview (查看当前状态)
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Secondary/Secondary Inconsistent/Inconsistent C r----- (分区还没有进行同步,数据不一致状态)
[[email protected] drbd.d]# drbdadm primary --force mydata (提升hong节点为主节点,进行数据同步,并进行格式化)
[[email protected] drbd.d]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 SyncSource Primary/Secondary UpToDate/Inconsistent C r---n-
[>....................] sync‘ed: 4.3% (4912/5128)M (正在同步块按照位在对齐)
[[email protected] drbd.d]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Primary/Secondary UpToDate/UpToDate C r----- (分区同步完成)
[[email protected] ~]# mke2fs -t ext4 /dev/drbd1 (对分区进行格式化)
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe blocks
328656 inodes, 1313263 blocks
65663 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1346371584
41 block groups
32768 blocks per group, 32768 fragments per group
8016 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 39 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
mysql配置:
1、创建mysql用户(注意两节点的mysql的ID号要一致)
hong节点:
[[email protected] drbd.d]# groupadd -r -g 306 mysql
[[email protected] drbd.d]# useradd -r -g 306 -u 306 mysql
[[email protected] drbd.d]# id mysql
uid=306(mysql) gid=306(mysql) groups=306(mysql)
[[email protected] ~]# mkdir /mydata 创建数据的存储目录
ning节点:
[[email protected] ~]# groupadd -r -g 306 mysql
[[email protected] ~]# useradd -r -g 306 -u 306 mysql
[[email protected] ~]# id mysql
uid=306(mysql) gid=306(mysql) groups=306(mysql)
[[email protected] ~]# mkdir /mydata 创建数据的存储目录(磁盘的挂载点)
2、安装mysql
(1)在主节点hong上挂载/dev/drbd1到数据的存储目录/mydata
[[email protected] ~]# mount /dev/drbd1 /mydata/
(2)在挂载目录/mydata下创建数据存储目录为data,并设置属主组
[[email protected] ~]# cd /mydata/
[[email protected] mydata]# mkdir data
[[email protected] mydata]# chown -R mysql.mysql data
[[email protected] mydata]# ll
total 20
drwxr-xr-x 2 mysql mysql 4096 Aug 16 17:21 data
drwx------ 2 root root 16384 Aug 16 17:16 lost+found
验证下是否两节点都是有data目录的
[[email protected] ~]# umount /mydata
[[email protected] ~]# drbdadm secondary mydata 在hong上降级drbd为从节点
[[email protected] ~]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
[[email protected] ~]# drbdadm primary mydata 在ning节点上提升为主节点
[[email protected] ~]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Primary/Secondary UpToDate/UpToDate C r-----
[[email protected] ~]# mount /dev/drbd1 /mydata/
[[email protected] ~]# cd /mydata/
[[email protected] mydata]# ll
total 20
drwxr-xr-x 2 mysql mysql 4096 Aug 16 17:21 data (查看是否data目录为mysql属主组)
drwx------ 2 root root 16384 Aug 16 17:16 lost+found
(3)安装mysql软件
包: mariadb-5.5.36-linux-x86_64.tar.gz
ning节点上:
[[email protected] ~]# tar xf mariadb-5.5.36-linux-x86_64.tar.gz -C /usr/local
[[email protected] ~]# cd /usr/local
[[email protected] local]# ln -sv mariadb-5.5.36-linux-x86_64/ mysql
`mysql‘ -> `mariadb-5.5.36-linux-x86_64/‘
[[email protected] local]# chown -R root.mysql mysql/*
[[email protected] local]# cd mysql
[[email protected] mysql]# scripts/mysql_install_db --datadir=/mydata/data/ --user=mysql
[[email protected] mysql]# mkdir /etc/mysql
[[email protected] mysql]# cp support-files/my-large.cnf /etc/mysql/my.cnf
[[email protected] mysql]# vim /etc/mysql/my.cnf
[mysqld]
datadir = /mydata/data
[[email protected] mysql]# cp support-files/my-large.cnf /etc/mysql/my.cnf
[[email protected] mysql]# vim /etc/mysql/my.cnf
[[email protected] mysql]# vim /etc/mysql/my.cnf
[[email protected] mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[[email protected] mysql]# chmod +x /etc/rc.d/init.d/mysqld
[[email protected] mysql]# chkconfig --add mysqld
[[email protected] mysql]# chkconfig mysqld off
[[email protected] mysql]# service mysqld start
Starting MySQL.... [ OK ]
[[email protected] mysql]# ls /mydata/data
aria_log.00000001 ibdata1 ib_logfile1 mysql-bin.000001 ning.pid test
aria_log_control ib_logfile0 mysql mysql-bin.index performance_schema
[[email protected] mysql]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.36-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
MariaDB [(none)]> create database mysdb;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> \q
Bye
到这里mariadb数据库已经安装完成
[[email protected] mysql]# service mysqld stop
Shutting down MySQL. [ OK ]
[[email protected] mysql]# cd
[[email protected] ~]# umount /dev/drbd1 卸载挂载点
[[email protected] ~]# drbdadm secondary mydata 降级为从节点
[[email protected]ng ~]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
[[email protected] ~]# scp mariadb-5.5.36-linux-x86_64.tar.gz 172.16.3.2:/root 复制包到ning节点,进行安装
[email protected]‘s password:
mariadb-5.5.36-linux-x86_64.tar.gz 100% 212MB 26.5MB/s 00:08
hong节点:
[[email protected] ~]# drbdadm primary mydata 提升hong节点为主节点
[[email protected] ~]# mount /dev/drbd1 /mydata 挂载分区
[[email protected] ~]# scp -r 172.16.3.1:/etc/mysql /etc/ 并复制ning节点上的mysqld配置文件,(因为配置文件是一样的)
[email protected]‘s password:
my.cnf 100% 4924 4.8KB/s 00:00
[[email protected] ~]# tar xf mariadb-5.5.36-linux-x86_64.tar.gz -C /usr/local
[[email protected] ~]# cd /usr/local
[[email protected] local]# ln -sv mariadb-5.5.36-linux-x86_64/ mysql
`mysql‘ -> `mariadb-5.5.36-linux-x86_64/‘
[[email protected] local]# cd mysql
[[email protected] mysql]# chown -R root.mysql ./*
这里说明下,不需要对mariadb做初始化了,因为数据已经有了,在ning节点上做的。在这里只需要提供服务启动脚本即可
[[email protected] mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[[email protected] mysql]# chmod +x /etc/rc.d/init.d/mysqld
[[email protected] mysql]# chkconfig --add mysqld
[[email protected] mysql]# service mysqld start
Starting MySQL.... [ OK ]
[[email protected] mysql]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.36-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
MariaDB [(none)]> show databases; 查看在ning节点上创建的数据库是存在的
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysdb | 看这里呵呵
| mysql |
| performance_schema |
| test |
+--------------------+
5 rows in set (0.05 sec)
MariaDB [(none)]> grant all on *.* to [email protected]‘172.16.%.%‘ identified by ‘ning‘; 授权远程连接用户
Query OK, 0 rows affected (0.03 sec)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)
测试下是否能够连接上数据库:(随便找个节点上测试的(这节点上需要有mysql客户端))
[[email protected] ~]# mysql -uroot -h 172.16.3.2 -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.5.36-MariaDB-log MariaDB Server
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysdb |
| mysql | 这里可以看到创建的数据库
| performance_schema |
| test |
+--------------------+
5 rows in set (0.00 sec)
mysql> \q
Bye
(4)测试完成后,在两节点上关闭mysqld和drbd开始配置corosync
[[email protected] mysql]# service mysqld stop
Shutting down MySQL. [ OK ]
[[email protected] mysql]# umount /mydata
[[email protected] mysql]# drbdadm secondary mydata
[[email protected] mysql]# service drbd stop
Stopping all DRBD resources: .
[[email protected] ~]# service drbd stop
Stopping all DRBD resources: .
配置corosync集群设置
1、定义drbd
[[email protected] ~]# crm
crm(live)# configure
定义drbd资源
crm(live)configure# primitive mydrbd ocf:linbit:drbd params drbd_resource=mydata op monitor role=Master interval=10s timeout=20s op monitor role=Slave interval=20s timeout=30s op start timeout=240s op stop timeout=100s
注意下定义时间间隔的时候主从不能一样。
crm(live)configure# verify
定义drbd克隆资源
crm(live)configure# master ms_mydrbd mydrbd meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=True
crm(live)configure# verify
定义挂载磁盘资源
crm(live)configure# primitive myfs ocf:heartbeat:Filesystem params device=/dev/drbd1 directory=/mydata fstype=ext4 op monitor interval=20s timeout=60s op start timeout=60s op stop timeout=60s
crm(live)configure# verify
定义mysqlIP资源
crm(live)configure# primitive myip ocf:heartbeat:IPaddr params ip=172.16.3.101 op monitor interval=30s timeout=20s
crm(live)configure# verify
定义mysql服务启动资源
crm(live)configure# primitive myserver lsb:mysqld op monitor interval=30s timeout=20s
crm(live)configure# verify
定义规则
crm(live)configure# group myservice myip myfs myserver 定义组名称为myservice资源名称为myip\myfs\myserver
crm(live)configure# colocation myfs_with_ms_mydrbd_master inf: myfs ms_mydrbd:Master 定义myfs挂载磁盘资源一定和ms_mydrbd克隆资源在一起
crm(live)configure# verify
crm(live)configure# order myfs_after_ms_mydrbd_master mandatory: ms_mydrbd:promote myfs:start 定义启动顺序挂载磁盘my_fs一定在克隆资源(成为主节点)ms_mydrbd之后启动
crm(live)configure# colocation myserver_with_myfs inf: myserver myfs 定义mysql服务器启动一定要和挂载磁盘my_fs在一起
crm(live)configure# verify
crm(live)configure# commit
crm(live)# status (查看状态信息)
Last updated: Fri Sep 19 13:16:37 2014
Last change: Fri Sep 19 12:26:20 2014 via cibadmin on hong
Stack: classic openais (with plugin)
Current DC: ning - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
5 Resources configured
Online: [ hong ning ] (两个节点都在线)
Master/Slave Set: ms_mydrbd [mydrbd]
Masters: [ ning ]
Slaves: [ hong ]
Resource Group: myservice
myip (ocf::heartbeat:IPaddr): Started ning
myfs (ocf::heartbeat:Filesystem): Started ning
myserver (lsb:mysqld): Started ning
测试corosync+mariadb+drbd
1、测试是否ning节点下线,资源将被切换到hong节点上去
[[email protected] ~]# crm node standby 让ning节点下线
[[email protected] ~]# crm status
Last updated: Fri Sep 19 13:24:33 2014
Last change: Fri Sep 19 13:24:25 2014 via crm_attribute on ning
Stack: classic openais (with plugin)
Current DC: ning - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
5 Resources configured
Node ning: standby
Online: [ hong ]
Master/Slave Set: ms_mydrbd [mydrbd]
Masters: [ hong ] drbd将自动切换到hong节点上
Stopped: [ ning ]
Resource Group: myservice
myip (ocf::heartbeat:IPaddr): Started hong
myfs (ocf::heartbeat:Filesystem): Started hong
myserver (lsb:mysqld): Started hong
在hong 节点查看下mysqlIP;
[[email protected] ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:d7:f7:9c brd ff:ff:ff:ff:ff:ff
inet 172.16.3.2/16 brd 172.16.255.255 scope global eth0
inet 172.16.3.101/16 brd 172.16.255.255 scope global secondary eth0 这里是有mysqlIP
inet6 fe80::20c:29ff:fed7:f79c/64 scope link
valid_lft forever preferred_lft forever
在liang节点上尝试连接下mariadb
[[email protected] ~]# mysql -uroot -h 172.16.3.101 -p 这里连接的mysqlIP的地址
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.5.36-MariaDB-log MariaDB Server
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysdb |这里有我们在上面测试用到的mysdb
| mysql |
| performance_schema |
| test |
+--------------------+
5 rows in set (0.10 sec)
2、测试mysqld手动当掉,是否会自动启动?
# service mysqld stop
# ss -tnl 观察3306端口将会看到30s后会自动上线