Tengine+keepalived 安装配置
测试环境
系统:Centos-6.5-x86_64
主IP:192.168.1.220
备IP:192.168.219
VIP:192.168.1.226
一、安装Tengine
1.1创建目录并下载安装包
mkdir -p /data/ops/{app,packages,scripts}
cd /data/ops/packages/
wget http://tengine.taobao.org/download/tengine-2.1.0.tar.gz
1.2 安装tengine
tar zxf tengine-2.0.3.tar.gz && cd tengine-2.0.3/ && ./configure --prefix=/data/ops/app/tengine-2.0.3/ --with-http_stub_status_module --with-pcre --with-http_upstream_check_module && make && make install
/data/ops/app/tengine-2.0.3/sbin/nginx -t
部分模块说明:
--with-http_upstream_check_module
该模块可以为Tengine提供主动式后端服务器健康检查的功能。
该模块在Tengine-1.4.0版本以前没有默认开启,它可以在配置编译选项的时候开启:./configure --with-http_upstream_check_module
--with-pcre
设置PCRE库的源码路径
1.3 将tengine加入rc.local以便开机自启动
echo "/data/ops/app/tengine-2.0.3/sbin/nginx" >> /etc/rc.local
二、keepalived安装
2.1 创建目录并下载安装包
mkdir -p /data/ops/{packages,app} && cd /data/ops/packages
wget http://www.keepalived.org/software/keepalived-1.2.12.tar.gz
2.2 解压并安装keepalived
tar zxf keepalived-1.2.12.tar.gz && cd keepalived-1.2.12 && ./configure --prefix=/data/ops/app/keepalived && make && make install
2.3 配置keepalived配置文件路径,启动路径
拷贝启动文件到/etc/init.d目录下
cp -p /data/ops/app/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
cp -p /data/ops/app/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
创建配置文件存放目录
mkdir /etc/keepalived
cp -p /data/ops/app/keepalived/sbin/keepalived /usr/sbin/
配置开启自启动级别
chkconfig --level 2345 keepalived on
三、keepalived配置
说明:备keepalived的配置文件只需去掉nopreempt,修改优先级(priority)的值改为:100(没有设置nopreempt的keepalived优先级必须低于设置nopreempt的keepalived,不然设置非抢占不生效),修改router_id Nginx_Id_1为router_id Nginx_Id_2
3.1 添加主配置文件
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
# written by :ops
global_defs {#全局配置
router_id Nginx_Id_1#运行 keepalive 机器的标示,注意每个机器需要唯一的标示
}
vrrp_script chk_nginx {
script "/data/ops/scripts/check_http/check_http.sh"#定义检测脚本
interval 5#检测间隔
weight 2
}
vrrp_instance Nginx1 {
state BACKUP#定义为backup
Nopreempt#启用非抢占模式
interface em1#实例绑定的网卡
virtual_router_id 51# 0-255 在同一个 instance 中一致 在整个 vrrp 中唯一
priority 150#优先级,优先级最大的会成为 master
authentication {#认证
auth_type PASS#认证的方式,支持PASS和AH
auth_pass hiveview#认证的密码
}
track_script {
chk_nginx
}
virtual_ipaddress {#指定漂移地址
192.168.0.226
}
}
注意:nopreempt就是在优先级高的主机回复后不抢占VIP,这样做为了减小keepalived主从切换带来的意外风险
3.2 Tengine健康检测脚本
cat /data/ops/scripts/check_http/check_http.sh
#!/bin/bash
DATE=`date +"%Y-%m-%d %H:%M:%S"`
NginxNum=`ss -lnt|grep -w :80|wc -l`
printf "+++$DATE\t`ss -lnt|grep -w :80`+++\n" >> /data/ops/scripts/check_http/check.log
if [ $NginxNum -eq 0 ]
then
printf "+++THE START\t$DATE\t`ss -lnt|grep -w :80`+++\n" >> /data/ops/scripts/check_http/check.log
/data/ops/app/tengine-2.1.0/sbin/nginx
printf "+++THE END\t$DATE\t`ss -lnt|grep -w :80`+++\n" >> /data/ops/scripts/check_http/check.log
sleep 2
if [ `ss -lnt|grep -w :80|wc -l` -eq 0 ]
then
printf "+++WANT TO KILL KEEPALIVED\t$DATE\t`ss -lnt|grep -w :80`+++\n" >> /data/ops/scripts/check_http/check.log
killall keepalived
fi
Fi
四、故障切换与检测
4.1 故障转移
从主(192.148.1.220)的角度来说,Keepalived会每5分钟运行一次check_http.sh脚本,当检测到nginx停掉后,尝试启动nginx,如果启动成功,VIP不发生变化,keepalived正常运行。如果启动失败,则停掉keepalived程序并将VIP切换到备的keealived服务器上。当主恢复正常后,因为配置了nopreempt非抢占模式,所以VIP还在备(192.148.1.219)上。
4.2 故障演示
主(192.148.1.220)上操作:
/etc/init.d/keepalived stop 停掉keepalived
4.1 在备上tailf /var/log/messages查看备机成了master,并获得了VIP(见红色字体部分)
May 3 03:17:41 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) Transition to MASTER STATE
May 3 03:17:42 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) Entering MASTER STATE
May 3 03:17:42 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) setting protocol VIPs.
May 3 03:17:42 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) Sending gratuitous ARPs on em1 for 192.148.1.226
May 3 03:17:42 OTT-AP-005 Keepalived_healthcheckers[17098]: Netlink reflector reports IP 192.148.1.226 added
May 3 03:17:47 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) Sending gratuitous ARPs on em1 for 192.148.1.226
4.2 启动主的nginx在启动主的keepalived(见红色字体部分)
May 3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: Opening file ‘/etc/keepalived/keepalived.conf‘.
May 3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: Configuration is using : 64313 Bytes
May 3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: Using LinkWatch kernel netlink reflector...
May 3 02:49:09 OTT-AP-004 Keepalived_healthcheckers[10199]: Opening file ‘/etc/keepalived/keepalived.conf‘.
May 3 02:49:09 OTT-AP-004 Keepalived_healthcheckers[10199]: Configuration is using : 6313 Bytes
May 3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: VRRP_Instance(Nginx1) Entering BACKUP STATE
May 3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 3 02:49:09 OTT-AP-004 Keepalived_healthcheckers[10199]: Using LinkWatch kernel netlink reflector...
May 3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: VRRP_Script(chk_nginx) succeeded
通过上述可以看到,VIP成功漂移到备机,主机keepalived重新启动后,是备状态也没有抢占成主,说明非抢占也成功生效
4.3检测VIP状态变化
check_vip.sh脚本检测VIP是否在当前主机上记录当前VIP状态,并与上一次状态进行比较,如果两次状态不一样,则报警通知
#主上的脚本
#!/bin/bash
###check VIP
MasterServer="92.168.1.220"
SlaveServer="192.168.1.219"
smtpemailfrom="[email protected]"
emailto="[email protected]"
subject="API VIP changed"
smtpserver=smtp.qiye.163.com
[email protected]
smtppass=jiankong123
if [ "`cat /data/ops/scripts/check_vip/status`" = "yes" ]
then
Current_Date=`date +"%Y-%m-%d %H:%M:%S"`
body="The API VIP drift from the MASTER SERVER:$MasterServer to SLAVE SERVER:$SlaveServer at $Current_Date"
bodytwo="The API VIP drift from the SLAVE SERVER:$SlaveServer to MASTER SERVER:$MasterServer at $Current_Date"
vip=`/sbin/ip addr|grep -o "124.192.140.226/32"`
if [ ! -z "$vip" ]
then
/usr/bin/sendEmail -f $smtpemailfrom -t $emailto -u $subject -m $body -s $smtpserver:25 -xu $smtplogin -xp $smtppass >> /data/ops/scripts/check_vip/check_vip.log 2>&1
echo "no" > /data/ops/scripts/check_vip/status
fi
else
Current_Date=`date +"%Y-%m-%d %H:%M:%S"`
body="The API VIP drift from the MASTER SERVER:$MasterServer to SLAVE SERVER:$SlaveServer at $Current_Date"
bodytwo="The API VIP drift from the SLAVE SERVER:$SlaveServer to MASTER SERVER:$MasterServer at $Current_Date"
vip=`/sbin/ip addr|grep -o "124.192.140.226/32"`
if [ -z "$vip" ]
then
/usr/bin/sendEmail -f $smtpemailfrom -t $emailto -u $subject -m $bodytwo -s $smtpserver:25 -xu $smtplogin -xp $smtppass >> /data/ops/scripts/check_vip/check_vip.log 2>&1
echo "yes" > /data/ops/scripts/check_vip/status
fi
fi
4.4 查看VIP 是否在当前主机上,使用ip add命令
#ip add
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether f0:1f:af:e5:ba:c5 brd ff:ff:ff:ff:ff:ff
inet 124.192.140.220/26 brd 124.192.140.255 scope global em1
inet 124.192.140.226/32 scope global em1
inet6 fe80::f21f:afff:fee5:bac5/64 scope link
valid_lft forever preferred_lft forever
五、无健康检测实例
说明:我们现在是一个VIP多种业务,既是nginx高可用也是redis高可用,但是我们不想因为一个业务有问题所有的都切换
5.1 主配置文件
# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
# written by :ops
global_defs {
router_id Redis_Id_3
}
vrrp_instance Tengine1 {
state BACKUP
nopreempt
interface em1
virtual_router_id 54
priority 150
authentication {
auth_type PASS
auth_pass hiveview
}
virtual_ipaddress {
192.168.1..252
}
}