1.
测试环境
两个虚机:nginx-1和nginx-3
centos7 系统内核3.10.0-514.el7.x86_64
ovs_version: "2.0.0"
2.
注意事项
关闭selinux
关闭NetworkManager(必须关闭,不然出莫名其妙的网络故障)
关闭firewalld
3.
安装openvswitch(可编译安装,这里使用yum安装,简单测试)
yum install openvswitch openvswitch-devel openvswitch-test openvswitch-debuginfor
systemctl enable? openvswitch
systemctl start? openvswitch
4.
创建桥前的网络情况
创建桥前
[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
? ? link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
? ? inet 127.0.0.1/8 scope host lo
? ? ? valid_lft forever preferred_lft forever
? ? inet6 ::1/128 scope host
? ? ? valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
? ? link/ether 00:0c:29:80:20:b2 brd ff:ff:ff:ff:ff:ff
? ? inet 192.168.32.139/24 brd 192.168.32.255 scope global dynamic ens33
? ? ? valid_lft 1721sec preferred_lft 1721sec
? ? inet6 fe80::6d0c:96f7:cbae:c576/64 scope link
? ? ? valid_lft forever preferred_lft forever
[[email protected] ~]#
[[email protected] ~]# ip r
default via 192.168.32.2 dev ens33? proto static? metric 100
192.168.32.0/24 dev ens33? proto kernel? scope link? src 192.168.32.132? metric 100
[[email protected] ~]#
5.
ovs基础知识,参考文档:https://opengers.github.io/openstack/openstack-base-use-openvswitch/
port类型说明
normal类型
操作系统中已有的网卡(物理网卡或者虚拟机中的虚拟网卡)连接到ovs桥上,
ovs会生成一个port处理这块网卡的进出数据包.这个端口类型为normal
挂载到OVS上的网卡设备不支持分配IP地址
internal
internal端口是ovs内部创建的虚拟网卡接口
每创建一个Port,OVS会自动创建一个同名接口(Interface)挂载到新创建的Port上
interface可用来配置ip
patch
多个ovs网桥可用patch port连接,类似于veth pair.
从一个Patch Port收到的数据包会被转发到另一个Patch Port
patch port总是成对出现,分别连接在两个网桥上.使用Patch连接的两个网桥跟一个网桥没什么区别
tunnel
OVS中支持添加隧道(Tunnel)端口,常见隧道技术有两种gre或vxlan.
网络之上构建一层虚拟网络,上层应用只与虚拟网络相关.
Interface
nterface是连接到Port的网络接口设备,是OVS与外部交换数据包的组件,在通常情况下,
Port和Interface是一对一的关系,只有在配置Port为 bond模式后,Port和Interface是一对多的关系.
6.
实际操作
虚机1创建桥
[[email protected] ~]# ovs-vsctl add-br br0
[[email protected] ~]#
[[email protected] ~]# ovs-vsctl show
67ccf09b-d0d9-4ccb-9f2f-1ac7918e5c46
? ? Bridge "br0"
? ? ? ? Port "br0"
? ? ? ? ? ? Interface "br0"
? ? ? ? ? ? ? ? type: internal
? ? ovs_version: "2.0.0"
[[email protected] ~]#
虚机2创建桥
[[email protected] ~]# ovs-vsctl add-br br0
[[email protected] ~]#
[[email protected] ~]# ovs-vsctl show
25cdb171-59ee-4eb4-a8af-cd93dc460926
? ? Bridge "br0"
? ? ? ? Port "br0"
? ? ? ? ? ? Interface "br0"
? ? ? ? ? ? ? ? type: internal
? ? ovs_version: "2.0.0"
桥默认有一个br0端口和br0接口,type类型为internel
internal端口是ovs内部创建的虚拟网卡接口
每创建一个桥自动创建个类型为internal的和桥一样名字的port和interface.
7.
测试思路
虚机1和虚机2的br0各定义同网段ip,测试互通.
虚机1配置br0为192.168.100.2/24
虚机2配置br0为192.168.100.3/24
临时配置命令,见下:
[[email protected] ~]# ifconfig br0 192.168.100.2/24 up
[[email protected] ~]# ifconfig br0 192.168.100.3/24 up
在没有建立隧道前,网络是不通的.
虚机1
[[email protected] ~]# ip a |grep br0
5: br0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
? ? inet 192.168.100.2/24 brd 192.168.100.255 scope global br0
[[email protected] ~]# ip r
default via 192.168.32.2 dev ens33
169.254.0.0/16 dev ens33? scope link? metric 1002
192.168.32.0/24 dev ens33? proto kernel? scope link? src 192.168.32.132
192.168.100.0/24 dev br0? proto kernel? scope link? src 192.168.100.2
[[email protected] ~]# ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=0.167 ms
^C
--- 192.168.100.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1042ms
rtt min/avg/max/mdev = 0.045/0.106/0.167/0.061 ms
[[email protected] ~]# ping 192.168.100.3
PING 192.168.100.3 (192.168.100.3) 56(84) bytes of data.
From 192.168.100.2 icmp_seq=1 Destination Host Unreachable
From 192.168.100.2 icmp_seq=2 Destination Host Unreachable
From 192.168.100.2 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.100.3 ping statistics ---
6 packets transmitted, 0 received, +3 errors, 100% packet loss, time 5138ms
pipe 4
[[email protected] ~]#
虚机2
[[email protected] ~]# ip a|grep br0
5: br0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
? ? inet 192.168.100.3/24 brd 192.168.100.255 scope global br0
[[email protected] ~]# ip r
default via 192.168.32.2 dev ens33
169.254.0.0/16 dev ens33? scope link? metric 1002
169.254.0.0/16 dev br0? scope link? metric 1005
192.168.32.0/24 dev ens33? proto kernel? scope link? src 192.168.32.134
192.168.100.0/24 dev br0? proto kernel? scope link? src 192.168.100.3
[[email protected] ~]# ping 192.168.100.3
PING 192.168.100.3 (192.168.100.3) 56(84) bytes of data.
64 bytes from 192.168.100.3: icmp_seq=1 ttl=64 time=0.066 ms
64 bytes from 192.168.100.3: icmp_seq=2 ttl=64 time=0.095 ms
^C
--- 192.168.100.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1042ms
rtt min/avg/max/mdev = 0.066/0.080/0.095/0.017 ms
[[email protected] ~]# ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
From 192.168.100.3 icmp_seq=1 Destination Host Unreachable
From 192.168.100.3 icmp_seq=2 Destination Host Unreachable
From 192.168.100.3 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.100.2 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4092ms
pipe 4
[[email protected] ~]#
8.
上面只是临时配置br0的地址,重启就消失.
可写入网卡配置文件,永久生效.参考见下:
虚机1
[[email protected] ~]# cd /etc/sysconfig/network-scripts/
[[email protected] network-scripts]# pwd
/etc/sysconfig/network-scripts
[[email protected] network-scripts]# cat ifcfg-br0
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
DEVICE=br0
ONBOOT=yes
IPADDR=192.168.100.2
NETMASK=255.255.255.0
[[email protected] network-scripts]#
虚机2
[[email protected] ~]# cd /etc/sysconfig/network-scripts/
[[email protected] network-scripts]# cat ifcfg-br0
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
DEVICE=br0
ONBOOT=yes
IPADDR=192.168.100.3
NETMASK=255.255.255.0
[[email protected] network-scripts]#
9.
创建gre隧道
虚机1
网桥br0添加gre1端口和gre1接口,设置端口类型为gre,配置远端Ip为192.168.32.140,远端ip为另一方的ip.
[[email protected] ~]# ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre option:remote_ip=192.168.32.140
虚机2
网桥br0添加gre1端口和gre1接口,设置端口类型为gre,配置远端Ip为192.168.32.139,远端ip为另一方的ip.
[[email protected] ~]# ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre option:remote_ip=192.168.32.139
ovs信息
虚机1
[[email protected] ~]# ovs-vsctl show
67ccf09b-d0d9-4ccb-9f2f-1ac7918e5c46
? ? Bridge "br0"
? ? ? ? Port "gre1"
? ? ? ? ? ? Interface "gre1"
? ? ? ? ? ? ? ? type: gre
? ? ? ? ? ? ? ? options: {remote_ip="192.168.32.140"}
? ? ? ? Port "br0"
? ? ? ? ? ? Interface "br0"
? ? ? ? ? ? ? ? type: internal
? ? ovs_version: "2.0.0"
[[email protected] ~]#
虚机2
[[email protected] ~]# ovs-vsctl show
25cdb171-59ee-4eb4-a8af-cd93dc460926
? ? Bridge "br0"
? ? ? ? Port "gre1"
? ? ? ? ? ? Interface "gre1"
? ? ? ? ? ? ? ? type: gre
? ? ? ? ? ? ? ? options: {remote_ip="192.168.32.139"}
? ? ? ? Port "br0"
? ? ? ? ? ? Interface "br0"
? ? ? ? ? ? ? ? type: internal
? ? ovs_version: "2.0.0"
[[email protected] ~]#
隧道建立后,网络实现互通.
[[email protected] ~]# ip a |grep br0
5: br0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
? ? inet 192.168.100.2/24 brd 192.168.100.255 scope global br0
[[email protected] ~]# ip r
default via 192.168.32.2 dev ens33
169.254.0.0/16 dev ens33? scope link? metric 1002
169.254.0.0/16 dev br0? scope link? metric 1005
192.168.32.0/24 dev ens33? proto kernel? scope link? src 192.168.32.139
192.168.100.0/24 dev br0? proto kernel? scope link? src 192.168.100.2
[[email protected] ~]# ping 192.168.100.3
PING 192.168.100.3 (192.168.100.3) 56(84) bytes of data.
64 bytes from 192.168.100.3: icmp_seq=1 ttl=64 time=3.61 ms
64 bytes from 192.168.100.3: icmp_seq=2 ttl=64 time=1.61 ms
64 bytes from 192.168.100.3: icmp_seq=3 ttl=64 time=0.936 ms
^C
--- 192.168.100.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.936/2.056/3.613/1.135 ms
[[email protected] ~]#
[[email protected] ~]# ip a |grep br0
5: br0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
? ? inet 192.168.100.3/24 brd 192.168.100.255 scope global br0
[[email protected] ~]# ip r
default via 192.168.32.2 dev ens33
169.254.0.0/16 dev ens33? scope link? metric 1002
169.254.0.0/16 dev br0? scope link? metric 1005
192.168.32.0/24 dev ens33? proto kernel? scope link? src 192.168.32.140
192.168.100.0/24 dev br0? proto kernel? scope link? src 192.168.100.3
[[email protected] ~]# ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=3.59 ms
64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=3.38 ms
64 bytes from 192.168.100.2: icmp_seq=3 ttl=64 time=0.469 ms
^C
--- 192.168.100.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.469/2.483/3.596/1.427 ms
[[email protected] ~]#
10.
创建gre通道后系统重启,br0无法获取ip.
暂时没有找到彻底解决这个问题的方法.怀疑是gre本身的问题导致.
临时测试,可以通过把执行命令写入rc.local,解决掉这个问题,命令见下:
[[email protected] ~]#
[[email protected] network-scripts]# cat /etc/rc.d/rc.local |grep -v ^#
touch /var/lock/subsys/local
ifdown br0 &&ifup br0? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ##加这条命令
[[email protected] network-scripts]#
注意必须保证rc.local具有x权限,可执行命令chmod +x /etc/rc.d/rc.local
虚机2做一样的操作.
11.
互相ping,互相抓包.
必须ping才能抓到包.注意了.
如果没有tcpdump命令,安装即可,参考命令:
yum -y install tcpdump
虚机1br0端口抓包icmp
[[email protected] ~]# tcpdump -i br0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:38:48.871368 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 10, length 64
14:38:48.871442 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 10, length 64
14:38:49.873007 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 11, length 64
14:38:49.873059 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 11, length 64
14:38:50.874202 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 12, length 64
14:38:50.874286 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 12, length 64
14:38:51.876210 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 13, length 64
14:38:51.876272 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 13, length 64
14:38:52.878047 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 14, length 64
14:38:52.878112 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 14, length 64
14:38:53.880465 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 15, length 64
14:38:53.880496 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 15, length 64
14:38:54.881610 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 16, length 64
14:38:54.881681 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 16, length 64
^C
14 packets captured
14 packets received by filter
0 packets dropped by kernel
虚机2br0端口抓包icmp
[[email protected] ~]# tcpdump -i br0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:40:13.642693 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 1, length 64
14:40:13.642725 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 1, length 64
14:40:14.642769 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 2, length 64
14:40:14.642810 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 2, length 64
14:40:15.643391 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 3, length 64
14:40:15.643427 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 3, length 64
14:40:16.645941 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 4, length 64
14:40:16.645992 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 4, length 64
14:40:17.647337 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 5, length 64
14:40:17.647410 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 5, length 64
14:40:18.650543 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 6, length 64
14:40:18.650606 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 6, length 64
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel
虚机1ens33端口抓gre包
[[email protected] ~]#? tcpdump -i ens33 ‘proto gre‘ -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
14:46:39.388735 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 22, length 64
14:46:39.388786 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 22, length 64
14:46:40.389953 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 23, length 64
14:46:40.390074 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 23, length 64
14:46:41.390733 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 24, length 64
14:46:41.390852 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 24, length 64
14:46:42.393785 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 25, length 64
14:46:42.393947 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 25, length 64
14:46:43.396383 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 26, length 64
14:46:43.396545 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 26, length 64
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
虚机2ens33端口抓gre包
[[email protected] ~]#? tcpdump -i ens33 ‘proto gre‘ -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
14:47:57.785331 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo request, id 2298, seq 6, length 64
14:47:57.785451 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo reply, id 2298, seq 6, length 64
14:47:58.789091 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo request, id 2298, seq 7, length 64
14:47:58.789169 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo reply, id 2298, seq 7, length 64
14:47:59.787219 IP 192.168.32.139 > 192.168.32.140: GREv0, length 46: ARP, Request who-has 192.168.100.3 tell 192.168.100.2, length 28
14:47:59.787618 IP 192.168.32.140 > 192.168.32.139: GREv0, length 46: ARP, Reply 192.168.100.3 is-at 12:8f:8c:8f:a8:4e, length 28
14:47:59.789472 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo request, id 2298, seq 8, length 64
14:47:59.789573 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo reply, id 2298, seq 8, length 64
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel
由以上抓包可见,配置实现了gre隧道的连通.
原文地址:https://blog.51cto.com/goome/2418128