Keepalived+LVS的高可用集群系统
一.拓扑图结构:
二.Keepalived的安装
1.主服务器的安装
[[email protected] ~]# wget http://www.keepalived.org/software/keepalived-1.2.20.tar.gz
[[email protected] ~]# tar -zxvf keepalived-1.2.20.tar.gz
[[email protected] ~]# cd keepalived-1.2.20/
[[email protected] keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived
configure: error: !!! OpenSSL is not properly installed on your system. !!! !!! Can not include OpenSSL headers files. !!! |
[[email protected] keepalived-1.2.20]# yum -y install openssl-devel libnl-devel libnfnetlink-devel ipvsadm
[[email protected] keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived
Keepalived configuration ------------------------ Keepalived version : 1.2.20 Compiler : gcc Compiler flags : -g -O2 -DFALLBACK_LIBNL1 Extra Lib : -lssl -lcrypto -lcrypt -lnl Use IPVS Framework : Yes IPVS sync daemon support : Yes IPVS use libnl : Yes fwmark socket support : Yes Use VRRP Framework : Yes Use VRRP VMAC : Yes Use VRRP authentication : Yes SNMP keepalived support : No SNMP checker support : No SNMP RFCv2 support : No SNMP RFCv3 support : No SHA1 support : No Use Debug flags : No libnl version : 1 Use IPv4 devconf : No Use libiptc : No Use libipset : No |
[[email protected] keepalived-1.2.20]# make && make install
[[email protected] keepalived-1.2.20]# ln -s /usr/local/keepalived/sbin/keepalived /usr/bin/keepalived
[[email protected] keepalived-1.2.20]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived
[[email protected] keepalived-1.2.20]# chmod 755 /etc/init.d/keepalived
[[email protected] keepalived-1.2.20]# chkconfig keepalived on
[[email protected] keepalived-1.2.20]# vim /etc/init.d/keepalived
# Source configuration file (we set KEEPALIVED_OPTIONS there) . /etc/sysconfig/keepalived 改为: # Source configuration file (we set KEEPALIVED_OPTIONS there) . /usr/local/keepalived/etc/sysconfig/keepalived |
[[email protected] keepalived-1.2.20]# mkdir /etc/keepalived
[[email protected] keepalived-1.2.20]# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
[[email protected] keepalived-1.2.20]# service keepalived restart
2.主服务器配置文件
[[email protected] /]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { notification_email { [email protected] } notification_email_from [email protected] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance VI_1 { state MASTER interface eno16777736 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.1.100 } } virtual_server 192.168.1.100 80 { delay_loop 6 lb_algo rr lb_kind DR persistence_timeout 50 protocol TCP real_server 192.168.1.30 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.1.40 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } |
3.配置 IPVS
3.1.开启IP转发功能
[[email protected] /]# echo 1 >/proc/sys/net/ipv4/ip_forward 默认是0,关闭ip转发;这里需要开启,所以设置值为1。
3.2.配置重定向
[[email protected] /]# echo "0" >/proc/sys/net/ipv4/ip_forward
[[email protected] /]# echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects
[[email protected] /]# echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects
[[email protected] /]# echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects
3.3.清除ipvsadm表
[[email protected] /]# ipvsadm –C
3.4.使用ipvsadm安装web服务
[[email protected] /]# ipvsadm -A -t 192.168.1.100:80-s rr
3.5.增加二台web服务器
[[email protected] /]# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.30:80 -m -w 1
[[email protected] /]# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.40:80 -m -w 1
3.6查看结果
[[email protected] ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.1.100:80 rr -> 192.168.1.30:80 Route 1 0 1 -> 192.168.1.40:80 Route 1 0 0 |
[[email protected] keepalived-1.2.20]# service keepalived restart
4.主服务器IP查看
[[email protected] /]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:74:1f:4e brd ff:ff:ff:ff:ff:ff inet 192.168.1.10/24 brd 192.168.1.255 scope global eno16777736 valid_lft forever preferred_lft forever inet 192.168.1.100/32 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe74:1f4e/64 scope link valid_lft forever preferred_lft forever |
5.备服务器的安装
[[email protected] ~]# wget http://www.keepalived.org/software/keepalived-1.2.20.tar.gz
[[email protected] ~]# tar -zxvf keepalived-1.2.20.tar.gz
[[email protected] ~]# cd keepalived-1.2.20/
[[email protected] keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived
configure: error: !!! OpenSSL is not properly installed on your system. !!! !!! Can not include OpenSSL headers files. !!! |
[[email protected] keepalived-1.2.20]# yum -y install openssl-devel libnl-devel libnfnetlink-devel ipvsadm
[[email protected] keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived
Keepalived configuration ------------------------ Keepalived version : 1.2.20 Compiler : gcc Compiler flags : -g -O2 -DFALLBACK_LIBNL1 Extra Lib : -lssl -lcrypto -lcrypt -lnl Use IPVS Framework : Yes IPVS sync daemon support : Yes IPVS use libnl : Yes fwmark socket support : Yes Use VRRP Framework : Yes Use VRRP VMAC : Yes Use VRRP authentication : Yes SNMP keepalived support : No SNMP checker support : No SNMP RFCv2 support : No SNMP RFCv3 support : No SHA1 support : No Use Debug flags : No libnl version : 1 Use IPv4 devconf : No Use libiptc : No Use libipset : No |
[[email protected] keepalived-1.2.20]# make && make install
[[email protected] keepalived-1.2.20]# ln -s /usr/local/keepalived/sbin/keepalived /usr/bin/keepalived
[[email protected] keepalived-1.2.20]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived
[[email protected] keepalived-1.2.20]# chmod 755 /etc/init.d/keepalived
[[email protected] keepalived-1.2.20]# chkconfig keepalived on
[[email protected] keepalived-1.2.20]# vim /etc/init.d/keepalived
[[email protected] keepalived-1.2.20]# mkdir /etc/keepalived
[[email protected] keepalived-1.2.20]# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
[[email protected] keepalived-1.2.20]# service keepalived restart
6.备服务器配置文件
[[email protected] /]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { notification_email { [email protected] } notification_email_from [email protected] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance VI_1 { state BACKUP interface eno16777736 virtual_router_id 51 priority 99 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.1.100 } } virtual_server 192.168.1.100 80 { delay_loop 6 lb_algo rr lb_kind DR persistence_timeout 50 protocol TCP real_server 192.168.1.30 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.1.40 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } |
7.配置 IPVS
7.1.开启IP转发功能
[[email protected] /]# echo 1 >/proc/sys/net/ipv4/ip_forward
7.2.配置重定向
[[email protected] /]# echo "0" >/proc/sys/net/ipv4/ip_forward
[[email protected] /]# echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects
[[email protected] /]# echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects
[[email protected] /]# echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects
7.3.清除ipvsadm表
[[email protected] /]# ipvsadm –C
7.4.使用ipvsadm安装web服务
[[email protected] /]# ipvsadm -A -t 192.168.1.100:80-s rr
7.5.增加二台web服务器
[[email protected] /]# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.30:80 -m -w 1
[[email protected] /]# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.40:80 -m -w 1
7.6查看结果
[[email protected] /]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.1.100:80 rr -> 192.168.1.30:80 Route 1 0 1 -> 192.168.1.40:80 Route 1 0 0 |
[[email protected] /]# service keepalived restart
8.备服务器IP查看
[[email protected] /]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:85:b8:f8 brd ff:ff:ff:ff:ff:ff inet 192.168.1.20/24 brd 192.168.1.255 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe85:b8f8/64 scope link valid_lft forever preferred_lft forever |
备注:如果主备服务器都有VIP存在,请关闭selinux和防火墙
# setenforce 0
# systemctl stop firewalld.service
三.web服务器的配置
1.此处以默认安装好的web服务,如Nginx和Apache等等。只进行节点的配置步骤。简单安装apache服务,方便后面的测试。
[[email protected] /]# yum -y install httpd
[[email protected] /]# vim /var/www/html/index.html
<h1>web1 server</h1>
[[email protected] /]# systemctl start httpd.service
[[email protected] /]# yum -y install httpd
[[email protected] /]# vim /var/www/html/index.html
<h1>web2 server</h1>
[[email protected] /]# systemctl start httpd.service
2.web1服务器LVS配置:
[[email protected] /]# vim /etc/init.d/lvsrs
#!/bin/bash #description:start realserver vip=192.168.1.100 source /etc/rc.d/init.d/functions case $1 in start) echo "Start Realserver" /sbin/ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce ;; stop) echo "Stop Realserver" /sbin/ifconfig lo:0 down echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce ;; *) echo "Usage: $0 (start | stop)" exit 1 esac |
[[email protected] /]# chmod 755 /etc/init.d/lvsrs
[[email protected] /]# service lvsrs start
3.web2服务器LVS配置:
[[email protected] ~]# vim /etc/init.d/lvsrs
#!/bin/bash #description:start realserver vip=192.168.1.100 source /etc/rc.d/init.d/functions case $1 in start) echo "Start Realserver" /sbin/ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce ;; stop) echo "Stop Realserver" /sbin/ifconfig lo:0 down echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce ;; *) echo "Usage: $0 (start | stop)" exit 1 esac |
[[email protected] ~]# chmod 755 /etc/init.d/lvsrs
[[email protected] ~]# service lvsrs start
到此为止,Keepalived+LVS的高可用集群系统搭建完成,下面进行测试。
四.高可用keepalived功能测试
1.先停掉主服务器上的keepalived服务,看看备服务器是否接管vip
[[email protected] ~]# service keepalived stop
[[email protected] ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:74:1f:4e brd ff:ff:ff:ff:ff:ff inet 192.168.1.10/24 brd 192.168.1.255 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe74:1f4e/64 scope link valid_lft forever preferred_lft forever |
2.查看备服务器的ip
[[email protected] ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:85:b8:f8 brd ff:ff:ff:ff:ff:ff inet 192.168.1.20/24 brd 192.168.1.255 scope global eno16777736 valid_lft forever preferred_lft forever inet 192.168.1.100/32 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe85:b8f8/64 scope link valid_lft forever preferred_lft forever |
上面信息可以看到备服务器已经接管vip地址。
3.恢复主服务器keepalived服务。
[[email protected] ~]# service keepalived start
[[email protected] ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:74:1f:4e brd ff:ff:ff:ff:ff:ff inet 192.168.1.10/24 brd 192.168.1.255 scope global eno16777736 valid_lft forever preferred_lft forever inet 192.168.1.100/32 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe74:1f4e/64 scope link valid_lft forever preferred_lft forever |
上面信息可以看到主服务器已经重新接管vip地址。
4.再查看备服务器ip地址
[[email protected] ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:85:b8:f8 brd ff:ff:ff:ff:ff:ff inet 192.168.1.20/24 brd 192.168.1.255 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe85:b8f8/64 scope link valid_lft forever preferred_lft forever |
上面信息可以看到主服务器已经释放了vip地址。
五.负载均衡LVS功能测试
打开浏览器输入http://192.168.1.100.不断刷新页面分别看到web1 server和web2 server表示LVS已经负载均衡了。
六.故障切换测试
1.停掉web1服务器的httpd服务
[[email protected] ~]# systemctl stop httpd.service
2. 查看主服务器状态
[[email protected] ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP wjb10000-master:http rr -> 192.168.1.40:http Route 1 0 0 |
3.查看备服务器状态
[[email protected] ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.1.100:http rr -> 192.168.1.40:http Route 1 0 0 |
以上内容可以看出Keepalived检测web1服务器出现故障,将此服务器从集群中移除。
4.恢复web1服务器
[[email protected] ~]# systemctl start httpd.service
5.再查看主服务器状态
[[email protected] ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP wjb10000-master:http rr -> 192.168.1.30:http Route 1 0 0 -> 192.168.1.40:http Route 1 0 0 |
6.再查看备服务器状态
[[email protected] ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.1.100:http rr -> 192.168.1.30:http Route 1 0 0 -> 192.168.1.40:http Route 1 0 0 |
节点恢复重新加入集群当中,浏览器刷新页面,可以重新看到web1 server和web2 server内容交替显示。
结束语:到此Keepalived+LVS的高可用集群系统实验完全结束。大家哪里不明白的欢迎留言。