基于keepalived+nginx部署强健的高可用7层负载均衡方案20151214

高可用是个老生常谈的问题了，开源的高可用软件已经做的相当成熟了，之前也在debian下做过lvs+heartbeat的4层LB，一直很稳定（可惜流量不大啊），现在由于业务的需要，做一个基于keepalived+nginx的高可用7层负载均衡。

拓扑结构也比较简单，就不画拓扑图了：2个节点上分别安装配置keepalived和nginx，配置nginx反向代理后端的real server

比较关键的几个点：

1、为避免同一个局域网中有多个keepalived组中的多播相互响应，采用单播通信

2、状态切换的过程触发邮件通知、短信通知、web通知、log记录，便于通过各种途径了解主备工作状态

3、nginx的检测脚本采用了轻量级的方式："killall -0 nginx"，还可以使用 pidof nginx的方式或者调用其他自定义检测脚本的方式

4、特别要注意优先级的大小及检测到异常时权重的变化

5、了解免费ARP的工作机制

6、了解VRRP协议的适用范围：局域网，第一跳网关冗余

7、单个vrrp实例工作在主备模式，为最大程度的利用2个节点的资源，可以做多个vrrp实例，实现高可用和负载均衡

为便于软件包的管理，采用centos自带的keepalived，nginx1.8.0采用nginx官方源，整体安装也比较简单。

#yum install keepalived nginx -y

设置关键服务的开机启动

#chkconfig nginx on
#chkconfig keepalived on

查看keepalived包安装了那些文件（文档一定好好看）：

#rpm -ql keepalived

/etc/keepalived
/etc/keepalived/keepalived.conf
/etc/rc.d/init.d/keepalived
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/libexec/keepalived
/usr/sbin/keepalived
/usr/share/doc/keepalived-1.2.13
/usr/share/doc/keepalived-1.2.13/AUTHOR

.........

keepalived的主配置文件，2台机器的配置文件略有区别，具体请看配置文件中注释

[[email protected] ~]# more /etc/keepalived/keepalived.conf
####Configuration File for keepalived
####内部API网关 keepalived HA配置
#### laijingli 20151213

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id proxy101  ## proxy101 on master101 , proxy102 on backup102
}

###simple check with killall -0 which is less expensive than pidof to verify that nginx is running
vrrp_script chk_nginx {
        script "killall -0 nginx"
        interval 1
        weight 2
fall 2
rise 1
}

vrrp_instance YN_API_GATEWAY {
    state MASTER   ## MASTER on master101 ,  BACKUP on backup102
    interface em1
    virtual_router_id 101
    priority 200   ## 200 on master101 ,  199 on backup102
    advert_int 1
    ###采用单播通信，避免同一个局域网中多个keepalived组之间的相互影响
    unicast_src_ip 192.168.0.101  ##本机ip
    unicast_peer {
192.168.0.102  ##对端ip
    }
    authentication {
        auth_type PASS
        auth_pass testpass
    }
    virtual_ipaddress {
        192.168.0.105  ## VIP
    }
    ###如果只有一块网卡的话监控网络接口就没有必要了
    #track_interface {
    # em1
    #}
    track_script {
        chk_nginx
    }
    ###状态切换是发送邮件通知，本机记录log，后期会触发短信通知
    notify_master /usr/local/bin/keepalived_notify.sh notify_master
    notify_backup /usr/local/bin/keepalived_notify.sh notify_backup
    notify_fault  /usr/local/bin/keepalived_notify.sh notify_fault
    notify        /usr/local/bin/keepalived_notify.sh notify
    smtp_alert
}

VRRP 实例组节点状态切换触发邮件通知、短信通知、log记录的脚本
# more /usr/local/bin/keepalived_notify.sh
#!/bin/bash
###keepalived notify script for record ha state transtion to log files

###将将状态转换过程记录到log，便于排错
logfile=/var/log/keepalived.notify.log
echo --------------- >> $logfile
echo `date` [`hostname`] keepalived HA role state transition:  $1 $2 $3 $4 $5 $6 >> $logfile

###将状态转换记录到nginx的文件，便于通过web查看ha状态（一定注意不要开放到公网）
echo `date` `hostname` $1 $2 $3 $4 $5 $6 > /usr/share/nginx/html/index.html

nginx的部分配置文件，仅供参考

# more /etc/nginx/nginx.conf

###运维管理用途: 用于区别vip跑在那台服务器上
server {
    listen       80;
    server_name  localhost;

location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

## nginx monitor use only
###add by lai monitor nginx status
                location /server-status {
                        stub_status on;
allow 127.0.0.1;
allow 192.168.0.0/24;
                }
        }

通过启停keepalived和nginx来模拟故障，测试vrrp 实例的状态切换过程（即VIP的漂移）：

/etc/init.d/keepalived start/stop

/etc/init.d/nginx stop start/stop

查看VIP跑在那台服务器上：

# ip addr show|grep 192.168
    inet 192.168.0.101/24 brd 192.168.0.255 scope global em1
    inet 192.168.0.105/32 scope global em1

测试：

[[email protected] ~]$ curl 192.168.0.101
Mon Dec 14 16:27:10 CST 2015 proxy101 INSTANCE YN_API_GATEWAY MASTER 202
[[email protected] ~]$ curl 192.168.0.102
Mon Dec 14 16:34:40 CST 2015 proxy102 INSTANCE YN_API_GATEWAY BACKUP 199
[[email protected] ~]$ curl 192.168.0.105
Mon Dec 14 16:27:10 CST 2015 proxy101 INSTANCE YN_API_GATEWAY MASTER 202

# tail /var/log/keepalived.notify.log
Mon Dec 14 16:25:13 CST 2015 [proxy101] keepalived HA role state transition:
Mon Dec 14 16:25:13 CST 2015 [proxy101] keepalived HA role state transition: INSTANCE YN_API_GATEWAY MASTER 202
---------------
---------------
Mon Dec 14 16:26:34 CST 2015 [proxy101] keepalived HA role state transition:
Mon Dec 14 16:26:34 CST 2015 [proxy101] keepalived HA role state transition: INSTANCE YN_API_GATEWAY BACKUP 200
---------------
---------------
Mon Dec 14 16:27:10 CST 2015 [proxy101] keepalived HA role state transition:
Mon Dec 14 16:27:10 CST 2015 [proxy101] keepalived HA role state transition: INSTANCE YN_API_GATEWAY MASTER 202

# tail /var/log/messages
Dec 14 16:27:08 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) forcing a new MASTER election
Dec 14 16:27:08 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) forcing a new MASTER election
Dec 14 16:27:09 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) Transition to MASTER STATE
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) Entering MASTER STATE
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) setting protocol VIPs.
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) Sending gratuitous ARPs on em1 for 192.168.0.105
Dec 14 16:27:10 localhost Keepalived_healthcheckers[74307]: Netlink reflector reports IP 192.168.0.105 added
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: Remote SMTP server [127.0.0.1]:25 connected.
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: SMTP alert successfully sent.
Dec 14 16:27:15 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) Sending gratuitous ARPs on em1 for 192.168.0.105

主备状态转换通知邮件：

通过抓包查看来了解详细的工作过程：

# tcpdump -ni em1 vrrp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 65535 bytes
17:36:47.098225 IP 192.168.0.101 > 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1s, length 20
17:36:47.388540 IP 192.168.0.22 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1s, length 20
17:36:48.099409 IP 192.168.0.101 > 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1s, length 20
17:36:48.389504 IP 192.168.0.22 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1s, length 20
17:36:49.100544 IP 192.168.0.101 > 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1s, length 20
17:36:49.390487 IP 192.168.0.22 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1s, length 20
17:36:50.101713 IP 192.168.0.101 > 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1s, length 20
17:36:50.391453 IP 192.168.0.22 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1s, length 20

参考：https://www.nginx.com/resources/admin-guide/nginx-ha-keepalived/

时间： 2024-10-19 19:20:13

基于keepalived+nginx部署强健的高可用7层负载均衡方案20151214的相关文章

haproxy实现的web反向代理，动静分离，以及基于keepalived实现的haproxy的高可用

   haproxy于Nginx一样都是做反向代理,但是与其相比,haproxy更专注于web代理.HAProxy是单进程多请求,也支持多进程,HAProxy运行在当前的硬件上,完全可以支持数以万计的并发连接.       haproxy功能的实现全部基于配置文件,所以我们需要了解很多的配置指令,玩转指令,再结合实际情况,我们就玩转了haproxy,其实haproxy的配置也很简单,下面我们一起简单认识和了解一些haproxy的基本功能和相关知识.         CentOS6.5自带的rpm

Keepalived+Nginx实现双主高可用负载均衡

Keepalived+Nginx实现双主高可用负载均衡 一.部署Nginx+keepalived高可用有两种配置方案: 1.Nginx+keepalived 主从配置 前端使用两台机器,使用一个VIP,即其中一台为主负责全部请求,一台为备,只用在主出现故障时,才会替代主处理请求,平常处于闲置状态,此方案不够理想.  2.Nginx+keepalived 双主配置 前端使用2台机器,使用两个VIP,互为主备,不考虑其他情况时均衡处理请求,当其中一台机器出现故障时,另一台负责处理对两台机器的请求,此

基于Keepalived实现LVS双主高可用集群

前言 前面说过基于heartbeat的LVS高可用方案,今天带来另一种解决方案:基于Keepalived实现LVS双主高可用集群.什么是Keepalived呢,keepalived观其名可知,保持存活,在网络里面就是保持在线了, 也就是所谓的高可用或热备,用来防止单点故障的发生.本文将详细讲述Keepalived工作原理及高可用解决方案的实现. 相关介绍 Keepalived简介 Keepalived采用VRRP(virtual router redundancy protocol,虚拟路由冗余

通过keepalived搭建高可用的LVS负载均衡集群

一.keepalived软件简介 keepalived是基于vrrp协议实现高可用功能的一种软件,它可以解决单点故障的问题,通过keepalived搭建一个高可用的LVS负载均衡集群时,keepalived还能检测后台服务器的运行状态. 二.vrrp协议原理简介 vrrp(虚拟路由器冗余协议),是为了解决网络上静态路由出现的单点故障的问题,举个例子,如下图 主机A和B均在同一个局域网内,C和D均是该局域网的网关,即A和B想与外网通信,需指网关到C或D,那究竟指向C好还是指向D好呢?都不好!当指向

构建高可用的LVS负载均衡集群 入门篇

一.LVS简介 LVS是Linux Virtual Server的简称,也就是Linux虚拟服务器, 是一个由章文嵩博士发起的自由软件项目,它的官方站点是www.linuxvirtualserver.org.现在LVS已经是 Linux标准内核的一部分,在Linux2.4内核以前,使用LVS时必须要重新编译内核以支持LVS功能模块,但是从Linux2.4内核以后,已经完全内置了LVS的各个功能模块,无需给内核打任何补丁,可以直接使用LVS提供的各种功能. LVS 集群采用IP负载和基于内容请求分

构建高可用的LVS负载均衡集群 进阶篇

一.lvs组件介绍 lvs的组件由两部分组成:工作在内核空间的ipvs模块和工作在用户空间ipvsadm工具.其中ipvsadm是规则生成工具,而ipvs是一个使规则生效的工具. 二.ipvsadm详解 构建高可用的LVS负载均衡集群 进阶篇,布布扣,bubuko.com

基于keepalived实现双主模型高可用lvs

实验环境,使用的操作系统CentOS6.5: Director: node1:IP 172.16.103.2 安装keepalived VIP:172.16.103.20 node2:IP 172.16.103.3 安装keepalived VIP:172.16.103.30 RS: RS1:IP 172.16.103.1 提供httpd服务 RS2:IP 172.16.103.4 提供httpd服务 实验效果: 前端的两台Director运行keepalived,自动生成各自的一组lvs规则,

基于keepalived实现多种模式的高可用集群网站架构

一. 实现主从服务器高可用技术. 大概网络拓扑图: 前提准备条件: 准备四台机器,一台为keepalived的master,一台为keepalived的backup,一台为rs1,一台为rs2,同时都关闭防火墙和selinux. ①安装包. yum install keepalived ②修改位置文件(keepalived的master). vim /etc/keepalived/keepalived.cf ③修改keepalived的主配置文件(keepalived的backup)  ④给RS

基于keepalived搭建mysql双主高可用

目录 概述 环境准备 keepalived搭建 mysql搭建 mysql双主搭建 mysql双主高可用搭建 概述 传统(不借助中间件)的数据库主从搭建,如果主节点挂掉了,从节点只能读取无法写入,只能把人肉去恢复故障,既不想引用中间件也不想人肉恢复故障,可以折中选择双主方案,本文将介绍通过keepalived搭建mysql双主方案. 本例中vip为:172.16.0.169,  两台mysql实例服务器ip分别为:172.16.0.1和172.16.0.2 转帖请注明来源: https://my