实验环境：Centos 6.7_64位
服务器：
Master节点：dm1 IP地址：10.0.0.61(eth0) 192.168.3.150(eth1，心跳)
Slave节点：dm2 IP地址：10.0.0.62(eth0) 192.168.3.160(eth1，心跳)
VIP地址：192.168.0.180
一、DRBD环境搭建
1. host映射
# vi /etc/hosts
127.0.0.1 localhostlocalhost.localdomain localhost4 localhost4.localdomain4
::1 localhostlocalhost.localdomain localhost6 localhost6.localdomain6
10.0.0.61 dm1
192.168.3.150 dm1
10.0.0.62 dm2
192.168.3.160 dm2
2. 时间同步
# ntpdate 10.0.0.254
18 May 19:49:39 ntpdate[16332]: adjust time server 10.0.0.254 offset-0.023216 sec
3. 添加附加库
官方网站：http://elrepo.org/tiki/tiki-index.php
以下测试针对Centos6.7 版本：
a）import public key：
#rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
b）安装库：
#rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
4. 安装DRBD
DRBD官方网站：http://drbd.linbit.com/docs/install/
# yum install drbd84 kmod-drbd84 -y
5. 两台服务器上的分区/dev/sdb1作为drbd的网络mirror分区
# fdisk /dev/sdb
//格式化分区
# mkfs.ext4 /dev/sdb1 //只在主机(dm1)上做
6. 开始配置NFS(两个节点都要执行)
# yum -y install rpcbind nfs-utils
# mkdir /data
# vi /etc/exports
/data *(rw,no_root_squash,no_all_squash,sync)
# service rpcbind start
# chkconfig rpcbind on
# chkconfig nfs off //NFS不需要启动，也不需要设置成开机启动，这些都将由后面的Heartbeat来完成。
# netstat -tunlp|grep rpc //查看rpcbind是否启动成功
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 16537/rpcbind
tcp 0 0 :::111 :::* LISTEN 16537/rpcbind
udp 0 0 0.0.0.0:600 0.0.0.0:* 16537/rpcbind
udp 0 0 0.0.0.0:111 0.0.0.0:* 16537/rpcbind
udp 0 0 :::600 :::* 16537/rpcbind
udp 0 0 :::111 :::* 16537/rpcbind
7. 开始配置DRBD
# modprobe drbd //加载drbd模块到内核中(两个节点都要执行)
# lsmod | grep drbd //查看drbd模块是否加载成功(两个节点都要执行)
drbd 372759 3
libcrc32c 1246 1 drbd
说明：显示以上信息，说明drbd模块加载成功
# cat /etc/drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
说明:主备节点两端配置文件完全一致
# cd /etc/drbd.d/
# vi nfs.res
//设定资源名称为: nfs
resource nfs {
protocol C;
net {
cram-hmac-algsha1;
shared-secret"abcd";
}
syncer {rate 30M;}
on dm1 {
device /dev/drbd1;
disk /dev/sdb1;
address 10.0.0.61:7788;
meta-disk internal;
}
on dm2 {
device /dev/drbd1;
disk /dev/sdb1;
address 10.0.0.62:7788;
meta-disk internal;
}
}
8. 启动DRBD
# drbdadm create-md nfs //激活前面配置的DRBD资源nfs(两个节点都要执行)
initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.
Success
# service drbd start //(两个节点都要执行)
# chkconfig drbd on //(两个节点都要执行)
# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build [email protected], 2016-01-12 13:27:11
1: cs:Connectedro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0 dw:0 dr:0 al:8bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:6827344
# drbdsetup /dev/drbd1 primary //初始化一个主机(这几步只在主节点上操作)
# drbdadm primary nfs
# drbdadm -- --overwrite-data-of-peer primary nfs
# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build [email protected], 2016-01-12 13:27:11
1: cs:SyncSourcero:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:1624064 nr:0 dw:0dr:1624728 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:5203280
[===>................] sync‘ed: 23.9% (5080/6664)M
finish: 0:02:03 speed:42,116 (36,088) K/sec
9. DRBD的使用
# mkfs.ext4 /dev/drbd1
# mount /dev/drbd1 /data //现在就可以使用DRBD分区了
注意：secondary节点上不允许对DRBD设备进行任何操作,包括只读。所有的读写操作只能在主节点上进行，只有当主节点挂掉时，secondary节点才能提升为主节点，继续进行读写操作。
二、Heartbeat环境搭建
Heartbeat具体的安装配置:http://linuxzkq.blog.51cto.com/9379412/1771152
以下仅提供配置文件供参考：
# vi authkeys //打开下面两项：一共有三种认证方式供选择，第一种是CRC循环冗余校验，第二种是SHA1哈希算法，第三种是MD3哈希算法，其中他们的密码可以任意设置，但是两边密码必须保持一致。
auth 3
3 md5 Hello!
chmod 600 authkeys //给认证文件授权为600
# cd /etc/ha.d/
# cat ha.cf //dm1
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694
ucast eth1 192.168.3.160
auto_failback off
node dm1
node dm2
ping 10.0.0.254
respawn hacluster /usr/lib64/heartbeat/ipfail
# cat ha.cf //dm2
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694
ucast eth1 192.168.3.150
auto_failback off
node dm1
node dm2
ping 10.0.0.254
respawn hacluster /usr/lib64/heartbeat/ipfail
#tail haresources //两台主、备机器上的配置是一样的
#node1 10.0.0.170 Filesystem::/dev/sda1::/data1::ext2
#
# Regarding the node-names in this file:
#
# They must match the names of the nodes listed in ha.cf, which in turn
# must match the `uname -n` of some node in the cluster. So they aren‘t
# virtual in any sense of the word.
#
#dm1 IPaddr::192.168.0.180/32/eth0drbddisk::nfs Filesystem::/dev/drbd1::/data::ext4 nfs
dm1 IPaddr::192.168.0.180/24/eth0drbd_primary
说明：
drbddisk::data <==启动drbd data资源，相当于执行/etc/ha.d/resource.d/drbddiskdata stop/start操作
Filesystem::/dev/drbd1::/data::ext4 <==drbd分区挂载到/data目录，相当于执行/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext4stop/start <==相当于系统中执行mount /dev/drbd1 /data
nfs <==启动nfs服务脚本，相当于/etc/init.d/nfsstop/start
资源切换脚本：
drbd_primary resource-group用来指定需要Heartbeat托管的服务，也就是这些 服务可以由Heartbeat来启动和关闭。如果要托管这些服务，就必须将服务写成可以通过start/stop来启动和关闭的脚步，然后放到/etc/init.d/或者/etc/ha.d/resource.d/目录下，Heartbeat会根据脚本的名称自动去/etc/init.d或者/etc/ha.d/resource.d/目录下找到相应脚本进行启动或关闭操作。
本脚本复制drbd的切换：
# cd /etc/init.d
# cat drbd_primary //两台机器上的脚本一模一样
#!/bin/sh
case "$1" in
start)
drbdadm primary nfs
mount /dev/drbd1 /data
/etc/init.d/nfs start
;;
stop)
/etc/init.d/nfs stop
umount /dev/drbd1
drbdadm secondary nfs
;;
esac
exit 0
# chmod 755 drbd_primary
启动Heartbeat（先主后从）:
# /etc/init.d/heartbeat start
# chkconfig heartbeat on
# netstat -tunlp|grep hear
udp 0 0 0.0.0.0:694 0.0.0.0:* 2447/heartbeat: wri
udp 0 0 0.0.0.0:58330 0.0.0.0:* 2447/heartbeat: wri
# ip a|grep eth0 //dm1
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
inet 10.0.0.61/24 brd 10.0.0.255 scope global eth0
inet 192.168.0.180/32 scope global eth0
# df-h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 14G 2.9G 10G 23% /
tmpfs 238M 0 238M 0% /dev/shm
/dev/sda1 190M 52M 129M 29% /boot
/dev/drbd1 6.3G 15M 6.0G 1% /data
# df-h //dm2
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 14G 2.5G 11G 19% /
tmpfs 238M 0 238M 0% /dev/shm
/dev/sda1 190M 52M 129M 29% /boot
# ip a|grep eth0
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
inet10.0.0.62/24 brd 10.0.0.255 scope global eth0
# netstat -tunlp|grep hear
udp 0 0 0.0.0.0:694 0.0.0.0:* 1789/heartbeat: wri
udp 0 0 0.0.0.0:18128 0.0.0.0:* 1789/heartbeat: wri
三、测试
1. 测试NFS客户端读写共享目录是否正常
# cd/data
# mkdirtest //dm1上
[[email protected]_stmp]# ls //NFS客户端
[[email protected]_stmp]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 14G 1.6G 12G 12% /
tmpfs 238M 0 238M 0% /dev/shm
/dev/sda1 190M 51M 129M 29% /boot
[[email protected]_stmp]# yum -y install rpcbind nfs-utils
[[email protected]_stmp]# service rpcbind start
[[email protected]_stmp]# chkconfig rpcbind on
[[email protected]_stmp]# ping 192.168.0.180
PING192.168.0.180 (192.168.0.180) 56(84) bytes of data.
64 bytesfrom 192.168.0.180: icmp_seq=52 ttl=64 time=0.242 ms
64 bytesfrom 192.168.0.180: icmp_seq=53 ttl=64 time=0.434 ms
64 bytesfrom 192.168.0.180: icmp_seq=54 ttl=64 time=0.364 ms
64 bytesfrom 192.168.0.180: icmp_seq=55 ttl=64 time=0.310 ms
64 bytesfrom 192.168.0.180: icmp_seq=56 ttl=64 time=0.308 ms
[[email protected]_stmp]# showmount -e 192.168.0.180
Exportlist for 192.168.0.180:
/data(everyone)
[[email protected]_stmp]# mount -t nfs 192.168.0.180:/data /media
[[email protected]_s~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 14G 1.6G 12G 12% /
tmpfs 238M 0 238M 0% /dev/shm
/dev/sda1 190M 51M 129M 29% /boot
192.168.0.180:/data 6.3G 15M 6.0G 1% /media
[[email protected]_s~]# ll /media/
0
drwxrwxrwx2 root root 16384 518 22:21 lost+found
drwxrwxrwx2 root root 4096 518 23:18 test
[[email protected]_s~]# cd /media/
[[email protected]_smedia]# ls
lost+found test
[[email protected]_smedia]# touch 333 //NFS客户端在共享目录上读写数据成功
[[email protected]_smedia]# ls
333 lost+found test
[[email protected]~]# ll /data
0
-rw-r--r--1 root root 0 520 20:35 333
drwxrwxrwx2 root root 16384 518 22:21 lost+found
drwxrwxrwx2 root root 4096 518 23:18 test
2. 测试高可用
[[email protected]~]# /etc/init.d/heartbeat stop
StoppingHigh-Availability services: Done.
[[email protected]~]# ip a|grep eth0
2:eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast stateUP qlen 1000
inet 10.0.0.61/24 brd 10.0.0.255 scope global eth0
[[email protected]~]# df -h //可以看到DRBD资源已释放
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 14G 2.9G 10G 23% /
tmpfs 238M 0 238M 0% /dev/shm
/dev/sda1 190M 52M 129M 29% /boot
[[email protected]~]# cat /proc/drbd //我们看到dm1上的DRBD角色由原来的主,已转换成备
version:8.4.7-1 (api:1/proto:86-101)
GIT-hash:3a6a769340ef93b1ba2792c6461250790795db49 build by [email protected],2016-01-12 13:27:11
1: cs:Connected ro:Secondary/Primaryds:UpToDate/UpToDate C r-----
ns:88 nr:16 dw:104 dr:1366 al:3 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1wo:f oos:0
[[email protected]~]# ip a|grep eth0 //已看到VIP飘移成功
2:eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast stateUP qlen 1000
inet 10.0.0.62/24 brd 10.0.0.255 scopeglobal eth0
inet 192.168.0.180/24 scope global eth0
[[email protected]~]# df -h //DRBD资源切换成功,并挂载.
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 14G 2.5G 11G 19% /
tmpfs 238M 0 238M 0% /dev/shm
/dev/sda1 190M 52M 129M 29% /boot
/dev/drbd1 6.3G 15M 6.0G 1% /data
[[email protected]~]# cat /proc/drbd //我们看到dm2上的DRBD角色由原来的备,已转换成主.
version:8.4.7-1 (api:1/proto:86-101)
GIT-hash:3a6a769340ef93b1ba2792c6461250790795db49 build by [email protected],2016-01-12 13:27:11
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDateC r-----
ns:40 nr:112 dw:152 dr:1366 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1wo:f oos:0
[[email protected]~]# ll /data //我们看到数据也同步成功
0
-rw-r--r--1 root root 0 520 20:35 333
drwxrwxrwx2 root root 16384 518 22:21 lost+found
drwxrwxrwx2 root root 4096 518 23:18 test
DRBD+Heartbeat+NFS的高可用至此结束，对高可用的测试，我们上面只是测试了一种情况，当然还有其它几种情况，留给大家去测试吧！