SIMPLE LIFE

SIMPLE LIFE, COMPLICATED MIND

Tuesday, January 24, 2012

Multiple default routes gateways with two different ISP ipfw PF setfib load balancing

Multiple default routes gateways with two different ISP ipfw PF setfib load balancing

Thanks to phoenix help I was able to setup multiple default routes, or a default route per network/interface to be precise, in Debian/Linux it is as simple as that:

# cat /etc/network/interfaces

iface eth0 inet static
    address 10.0.0.2
    netmask 255.255.255.252
    gateway 10.0.0.1

iface eth1 inet static
    address 20.0.0.2
    netmask 255.255.255.252
    gateway 20.0.0.1

That would be example topology (but more than 2 interfaces is also possible).

ISP NETWORK 0                   ISP NETWORK 1
         \                          /
          \                        /
           \                      /
          ROUTER 0            ROUTER 1
          10.0.0.1/30         20.0.0.1/30
              \                  /
        +------\----------------/------+
        |       \              /       |
        |       wan0          wan1     |
        |    10.0.0.2     20.0.0.2     |
        |                              |
        |   FREEBSD (Firewall Router)  |
        |                              |
        |             dmz0             |
        |       192.168.0.1/24         |
        |       192.168.0.2/24         |
        +------------------------------+
                       |
                   +--------+
                   | switch +
                   +--------+
                       |
                      /
               ______/
              /
    +--------/-----+
    |       /      |
    |      lan0    |
    |  192.168.0.3 |
    |  192.168.0.4 |
    |              |
    | Web Server 0 |
    +--------------+

Now, You can not use the ‘casual‘ defaultrouter="X" cause it will be only for one network.

We will have to use setfib(1) to create two (or more) separete routing tables per network/interface.

Note: FIB (Forward Information Base, synonym for a routing table here)

Add these lines to /boot/loader.conf file:

# vim /boot/loader.conf
### select one of following firewalls to enable.
#ipfw_load="YES" # Firewall
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log

### set number of routing tables to support.
net.fibs=2

Note: to view a list of options: cat /boot/defaults/loader.conf.

It will unfortunately require kernel recompile, but its not as that hard:

# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2

# vim MYKERNEL8.2
options ROUTETABLES=2 # max 16. 1 is back compatible.

Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.

# cat /usr/src/sys/`uname -m`/conf/DEFAULTS

# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2

# sync ; reboot

Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html

Remove the files generated during recompiling the kernel. 
# rm -rf /usr/obj/usr/src/sys/MYKERNEL8.2

After the kernel has been built and rebooted the different routing tables can be accessed as shown in the setfib(1) man page by issuing command setfib 0 netstat -rn. 0 is the default routing table.

After this one has to create the second routing table by prepending every route add command with setfib 1 route add… e.g:

# setfib 0 /sbin/route add -net default 10.0.0.1
# setfib 1 /sbin/route add -net default 20.0.0.1

Note: with packet filter one can control how the routing table is selected by using rtable option but it should be noted that this selection can only be done on the input of the packets as the routing decision is done at the input not at the output.

# vi /etc/rc.conf
### WAN interfaces. You do not need to set defaultrouter="XXX" here.
ifconfig_wan0="inet 10.0.0.2/32"
ifconfig_wan1="inet 20.0.0.2/32"

### Note: commented out for multiple WAN routes setup. Routes will be added in /etc/rc.local file.
#defaultrouter="10.0.0.1"

### LAN static route
#static_routes="lan"
#route_lan="-net 192.168.50.0/24 192.168.0.11"

### PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""

### Enable as LAN gateway
gateway_enable="YES"

All the rest configuration resides in /etc/rc.local file:

# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 10.0.0.1

### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 20.0.0.1

### assing route tables to interfaces
### Note: uncomment following lines if you are using ipfw. Leave it if you are using PF.
###
#ipfw -f flush
#ipfw add allow ip from any to any via lo0
#ipfw add setfib 0 ip from any to any via wan0
#ipfw add setfib 1 ip from any to any via wan1
#ipfw add allow ip from any to any

These would be handy for restarting:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# /etc/rc.d/local restart
# pfctl -f /etc/pf.conf

View the fib:
# sysctl -a | grep fib
net.my_fibnum: 0
net.add_addr_allfibs: 1
net.fibs: 2

View the default route of each routing table:

# setfib 0 netstat -rn
# setfib 1 netstat -rn

Try to ping:
# setfib 0 ping google.com
# setfib 1 ping google.com

===============================================================
pf.conf

Looking for something like this?

# man 1 setfib
# man 2 setfib

And pf.conf(5) has the route-to and reply-to directives, of course.

# man 5 pf.conf | less +/rtable
# man 5 pf.conf | less +/route-to
# man 5 pf.conf | less +/reply-to

# vim /etc/pf.conf
### [VARIABLES]
wan_if0 = "wan0"
wan_if1 = "wan1"
dmz_if0 = "dmz0"
www0_ip0 = "192.168.0.3"
www0_ip1 = "192.168.0.4"

##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"

### [NAT]
nat on $wan_if0 from $dmz_if0/24 to any -> $wan_if0
nat on $wan_if1 from $dmz_if0/24 to any -> $wan_if1

### [HTTP] RDR Outside to DMZ
rdr on $wan_if0 proto tcp from any to $wan_if0 port 80 -> $www0_ip0
rdr on $wan_if1 proto tcp from any to $wan_if1 port 80 -> $www0_ip1

### [HTTP] outside to router
pass in quick on $wan_if0 proto {tcp} from any to $wan_if0 port 80 rtable 0
pass in quick on $wan_if1 proto {tcp} from any to $wan_if1 port 80 rtable 1

### [HTTP] router to dmz
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip0 port 80 rtable 0
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip1 port 80 rtable 1

Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53

You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp

Or just to/from a certain host:
# tcpdump -ni re0 icmp host 172.16.70.12

Reading a PF (packet filter) log file

The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.

To view the log file:

# tcpdump -n -e -ttt -r /var/log/pf.log

Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:

# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0

Filtering Log Output

Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:

# tcpdump -n -e -ttt -r /var/log/pf.log port 80

set up PF (packet filter) firewall on FreeBSD 8.1

Other Solution:
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-aggregation.html
http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4
http://scratching.psybermonkey.net/2009/06/freebsd-combine-2-or-more-nic-using.html
===============================================================
Just a quick note, if doing this on a router with a single internal interface.

Traffic originating on the local network will go out the default route of FIB 0.

Only traffic coming in on the second public interface will go out the same interface.

IOW, the setup above is really only useful for incoming traffic, to make sure that it goes back out the correct interface.

However, a few more IPFW rules can be added to classify traffic on the internal NIC.

It all depends on what you want to accomplish.

Yes, its mainly for that purpose, to assign proper gateways to networks/interfaces (which is a lot more easier @ Debian in /etc/network/interfaces)

We may even use FIB# 1 and 2 for these networks, and use FIB# 0 with other NIC as a default interface, posibilities are of course big, but I wanted to point posibility of having a gateway per network.

http://www.daemonforums.org/showthread.php?t=4610
===============================================================
setfib selects the routing table for locally originated
outgoing packets. Besides locally originated packets, there
are packets arriving from the network and need to be forwarded.
These packets can be classified in a specific routing table
with the aid of ipfw. That‘s all there is. I can‘t think
of something else that needs to be thought with regard to
multiple routing tables.

HTH, Nikos

http://lists.freebsd.org/pipermail/freebsd-questions/2009-July/202462.html
===============================================================
Something has been bugging me for several years now. In that time I have usually had access to multiple WAN connections, owing to my participation in the telecom industry. However, I‘ve never been able to get SSHD to behave the way I wanted it to. I wanted to be able to connect to the SSH daemon on my (FreBSD) router from whichever WAN connection I wanted. Unfortunately, SSHD is stuborn about always routing its response to the default gateway of the router, which breaks an SSH connection coming in from the secondary WAN connection.

I have finally, at long last, found the solution.

The Problem: When describing this problem to other FreeBSD users, they pretty universally assumed that I was mistaken, and that the SSH daemon would by default route its responses out the interface that the initial request had been received on. Evidently, it is uncommon to run SSHD on more than one WAN interface. At its root, this problem just boils down to the routing table. SSHD doesn‘t route it‘s responses to the interface that they were received on, it uses the routing table to determine where to send it‘s responses. If the request came from a local network, then it responds to the local network. If it originated from a non-local network, then it uses the default gateway. It‘s really that simple. There is no logic for having multiple paths to non-local networks.

Frequently Offered Solution: Since I use pf for my firewall, I‘m frequently told to use pf‘s route-to and reply-to functionality to solve this problem. I have at times used route-to and reply-to extensively in my pf.conf. But route-to and reply-to do not trump the default routing table for traffic the originates or terminates on the router itself. They are useful only for traffic passing through the router. pf can only make routing decisions when a packet passes through an interface. It can try and set the reply-to interface to be the second WAN connection when an inbound SSH connection is made, but neither the SSH daemon nor the routing table on the host know or care about the routing preferences of pf.

The Real Solution: FreeBSD has support for multiple routing tables. It‘s little known, and even less documented, but it does exist. Basically, you need to recompile your kernel with multiple routing table support ("options ROUTETABLES=2"), and then use the setfib program to set which routing table to use when starting another program. The syntax is similar to nice: setfib 1 route add default 192.168.1.1 would add a default route of 192.168.1.1 to the second routing table on the host. If not specified, the default routing table is 0. On FreeBSD, pf also has support for multiple routing tables with the little discussed rtable option.

So here are the steps to solving this problem:

Step 1: Rebuild your kernel with the ROUTETABLES option set to a non-zero integer. This is how many routing tables your host will support.

# cat /usr/src/sys/`uname -m`/conf/MYKERNEL8.2 | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.

Step 2: Disable the SSH daemon in your rc.conf:

# cat /etc/rc.conf | grep ssh
#sshd_enable="YES" # This is now handled by /etc/rc.local

Step 3: Create /etc/rc.local file to start multiple SSH daemons. To do this, copy the /etc/ssh/sshd_config file to several alternates, one per interface you want SSHD to listen to, and set the ListenAddress for each file to only the IP for that interface.

# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable0
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable1

# vim /etc/ssh/sshd_config.rtable0
ListenAddress 20.0.0.2
ListenAddress 192.168.0.1

# vim /etc/ssh/sshd_config.rtable1
ListenAddress 30.0.0.2
ListenAddress 192.168.0.2

Note: Multiple ListenAddress options are permitted.

# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 20.0.0.1

### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 30.0.0.1

### Start SSH daemons for each interface
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable0
/usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable1

Step 4: Add rtable awareness to your pf.conf file:

[[email protected]]~ # cat /etc/pf.conf | grep rtable
pass in log on tun0 inet proto icmp from any to (tun0) icmp-type rtable 0
pass in log on tun1 inet proto icmp from any to (tun1) icmp-type rtable 1
pass in log on tun0 inet proto tcp from any to (tun0) port ssh rtable 0
pass in log on tun1 inet proto tcp from any to (tun1) port ssh rtable 1
pass in log on wan0 inet proto tcp from wan0:network to (wan0) port 22 rtable 0

Conclusion: Because the SSH daemon listening on tun1 is using a routing table that features the tun1 interface as the default gateway, the response will go out tun1. An inbound connection to tun0 will hit the SSH daemon listening on tun0 (which is an entirely separate process from the one listening on tun1) and uses the routing table associated with tun0, which features tun0 as the default gateway.

In my above config it‘s worth pointing out that it doesn‘t actually matter which routing table the SSH daemon listening on the LAN interface uses, because both routing tables see the LAN network as a local one. By default on FreeBSD with multiple routing tables enabled, all local networks will still appear in all the routing tables. There is a sysctl option to disable this behavior.

High Availability HA cluster solution for FreeBSD

HAST + CARP + ZFS

HAST (Highly Available Storage) - allows to transparently store data on two physically separated machines connected over the TCP/IP network.

CARP (Common Address Redundancy Protocol) - allows multiple hosts to share the same IP address. In some configurations, this may be used for availability or load balancing. Hosts may use separate IP addresses as well, as in the example provided here.

# man carp
# man ng_one2many
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.

http://www.freebsd.org/doc/handbook/carp.html
http://forums.freebsd.org/showthread.php?t=17133
http://blather.michaelwlucas.com/archives/241
===============================================================

Reference:
http://gala4th.blogspot.com/2011/10/multiple-default-routes-gateways-with.html
High Availability HA cluster solution for freebsd
http://www.daemonforums.org/showthread.php?t=4610
http://freebsd-forums.liquidneon.com/showthread.php?t=20166
http://www.mmacleod.ca/blog/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/
http://www.melen.org/u/jan/wp/?p=102
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html
Running PF & IPFW Together
Prevent default gateway route and resolv.conf being overwritten by DHCP

Posted by Jun Hsieh at 11:28 AM 

来源： <http://blog.ijun.org/2011/10/multiple-default-routes-gateways-with.html>

来自为知笔记(Wiz)

时间： 2024-08-11 17:37:13

SIMPLE LIFE的相关文章

初译 Support Vector Machines：A Simple Tutorial（一）

从本次开始我将开始尝试着逐章翻译一下 Alexey Nefedov的<Support Vector Machines:A Simple Tutorial>这本教材,这可是我们导师极力推荐的SVM教材,看了好久一直感觉一脸懵逼,索性开坑翻译一下吧,也当是加深理解,毕竟我也是一知半解,如果翻译的有不对的地方还望大佬们斧正,欢迎提意见,欢迎讨论. 嗯,就是这样. (一)Introduction 在本章节中将会介绍一些用于定义支持向量机(SVM)的基础的概念,这些概念对于理解SVM至关重要,假定读者了

ubuntu16.04下安装NS-2.35以及对simple例的理解

本人是在VMWare上安装的ubuntu16.04版本,然后安装NS2.35. 1.下载ns2的安装包,这里我选择的是ns-allinone-2.35.tar.gz压缩格式的all in one安装包,all in one 安装包包含所有的组件,比较方便,另附下载地址: http://www.isi.edu/nsnam/ns/ns-build.html 2.安装前要先进行一些准备工作进行配置环境,输入如下代码: $sudo apt-get install build-essential $sud

[bzoj3489]A simple rmq problem

本题既不是rmq也不会simple(对我这种蒟蒻而言) 一开始只能想到树套树套树TAT然后看了看数据范围果断滚去膜拜题解. 然后才知道预先排序一下可以弄掉一个log.不过得写可持久化线段树套可持久化线段树.. 然后愉悦的开码了...感人的是竟然不用调...更感人的是交上去直接tle了. 然后从网上找了别人的代码(方法一样)发现同样的数据我要跑6s+..标称只要2s+.. 之后各种卡常还是慢了一倍TAT...最后自己写个max函数就和标程一样快了TAT这几天怎么总是出些奇怪的状况QAQ. 本来故事

地理数据可视化：Simple，Not Easy

如果要给2015年的地理信息行业打一个标签,地理大数据一定是其中之一.在信息技术飞速发展的今天,“大数据”作为一种潮流铺天盖地的席卷了各行各业,从央视的春运迁徙图到旅游热点预测,从大数据工程师奇货可居到马云布道“DT”时代,“大数据”被推到了一个前所未有的高度,连国家领导人出访演讲都言必称大数据.地理信息数据天生具有大数据属性,作为整天和地理信息数据打交道的地信人自然不甘落后,地理大数据概念脱颖而出. 地理大数据是什么?大体来说就是把社会经济.自然资源.商业信息等但凡具有一点空间维度的数据一股脑

简单工厂模式( Simple Factory Pattern )

1. 简单工厂模式( Simple Factory Pattern ) 1.1. 模式动机 考虑一个简单的软件应用场景,一个软件系统可以提供多个外观不同的按钮(如圆形按钮.矩形按钮.菱形按钮等), 这些按钮都源自同一个基类,不过在继承基类后不同的子类修改了部分属性从而使得它们可以呈现不同的外观,如果我们希望在使用这些按钮时,不需要知道这些具体按钮类的名字,只需要知道表示该按钮类的一个参数,并提供一个调用方便的方法,把该参数传入方法即可返回一个相应的按钮对象,此时,就可以使用简单工厂模式. 1.2

NYOJ 707 A Simple Problem(结构体排序) 睡前一水~~

链接:click here 题意: A Simple Problem 时间限制:3000 ms  |  内存限制:65535 KB 难度:2 描述 You know, just as the title imply, this is a simple problem. In a contest, given the team-id, solved, penalty of all the teams, tell me the champion.If the numbers of solved pr

hdu 4974 A simple water problem（数学题）

题目链接:http://acm.hdu.edu.cn/showproblem.php?pid=4974 Problem Description Dragon is watching competitions on TV. Every competition is held between two competitors, and surely Dragon's favorite. After each competition he will give a score of either 0 or

hdu 4975 A simple Gaussian elimination problem.（网络流，判断矩阵是否存在）

题目链接:http://acm.hdu.edu.cn/showproblem.php?pid=4975 Problem Description Dragon is studying math. One day, he drew a table with several rows and columns, randomly wrote numbers on each elements of the table. Then he counted the sum of each row and col

E - A Simple Problem with Integers

#include <iostream> #include <stdio.h> #include <string.h> #include <stdlib.h> using namespace std; #define N 100002 struct node { int l,r; long long lz,w; }q[4*N]; void pushup(int rt) { q[rt].w=q[rt*2].w+q[rt*2+1].w; } void pushdo

Simple Automated Backups for MongoDB Replica Sets

There are a bunch of different methods you can use to back up your MongoDB data, but if you want to avoid downtime and/or potential performance degradation, the most common advice seems to be that you should simply do all your backups on a slave. Thi