openstack起初的网络部分是和计算核心nova合在一起的，后来被拆分出来，独立成为一个模块, 现在名为Neutron.
本博文是学习记录，记录的是基于GRE tunnel技术的neutron和计算主机之间的连接和通信流程。下面直接上图，结合图做一些简单的介绍（这些介绍的内容，主要来自RDO。）
在介绍上面这个图之前，需要简单的介绍一下，什么是GRE tunnel。
- GRE（Generic Routing Encapsulation）：通用路由封装协议。
- GRE是VPN的第三层隧道协议，采用了隧道（Tunnel）技术。
- Tunnel是虚拟的点对点连接，即点对点连接的虚拟接口。
至于更详细的内容，可以google。
下面就对neutron网络做简单的介绍，了解这个后，对openstack的深入学习，会有很大的帮组。因为我个人觉得，云计算里面，网络的虚拟化和管理，是相对比较难的一块，比起存储和计算的虚拟化要难。
网络连接图的Compute Host中，qbr是Linux bridge，虚拟出来的。这个主要是处理openstack的security group的，虽然openstack的网络核心部分采用的是open vswitch，但是这个和iptables规则不兼容，所以，安全策略就落在了虚拟出来的Linux bridge上了。后续，这个linux bridge可以看作firewall bridge。
在具体介绍之前，我先查看下我的测试环境devstack中的网络设备之间的连接关系：
1 [[email protected]10-121-5-244 ~]# ovs-vsctl show
2 56f3ae0a-87b4-4050-bc79-9a92a76590e3
3 Bridge br-ex
4 Port "qg-5d13ee78-0a"
5 Interface "qg-5d13ee78-0a"
6 type: internal
7 Port br-ex
8 Interface br-ex
9 type: internal
10 Bridge br-tun
11 fail_mode: secure
12 Port patch-int
13 Interface patch-int
14 type: patch
15 options: {peer=patch-tun}
16 Port br-tun
17 Interface br-tun
18 type: internal
19 Bridge br-int
20 fail_mode: secure
21 Port "qr-c36567ef-4d"
22 tag: 1
23 Interface "qr-c36567ef-4d"
24 type: internal
25 Port patch-tun
26 Interface patch-tun
27 type: patch
28 options: {peer=patch-int}
29 Port "qr-2e4c72e5-5f"
30 tag: 1
31 Interface "qr-2e4c72e5-5f"
32 type: internal
33 Port br-int
34 Interface br-int
35 type: internal
36 Port "tapf2d4361c-39"
37 tag: 1
38 Interface "tapf2d4361c-39"
39 type: internal
40 ovs_version: "2.4.0"
Compute host: instance networking (A,B,C)
An outbound packet starts on eth0 of the virtual instance, which is connected to a tap device on the host, tap7c7ae61e-05. This tap device is attached to a Linux bridge device, qbr7c7ae61e-05.
Ideally, the TAP device vnet0 would be connected directly to the integration bridge, br-int. Unfortunately, this isn‘t possible because of how OpenStack security groups are currently implemented. OpenStack uses iptables rules on the TAP devices such as vnet0 to implement security groups, and Open vSwitch is not compatible with iptables rules that are applied directly on TAP devices that are connected to an Open vSwitch port.
Compute Host图中，A， Q都是vnet0，也就是虚拟机的网络接口，B/C 以及 R/S都是Linux bridge的tap设备，防火墙的规则就是设置在linux bridge （qbr）的对应tap上的。
Compute host: integration bridge (D,E)
The integration bridge, br-int, performs VLAN tagging and un-tagging for traffic coming from and to your instances.
这个过程，br-int相当于一个VLAN的处理机，将从没有加tag的来自VM1的数据从入口D接收后，进行加tag，然后发向E口，相反，会将带有tag的vlan上的数据从E口接收后，再去tag，根据VLAN的端口映射规则将数据转发到D(或者是去向其他的VM，例如VM2，则是转发到T口)
而对于T/E的数据处理，和D/E的处理流程类似。区别主要在D和T对应的VLAN ID会不同而已。
Compute host: tunnel bridge (F,G)
The tunnel bridge translates VLAN-tagged traffic from the integration bridge into GRE tunnels. The translation between VLAN IDs and tunnel IDs is performed by OpenFlow rules installed on br-tun.
在这里就体现出Neutron网络中的GRE tunnel机制了。就是G和Network Host的H口之间，将会建立GRE Tunnel。
Network host: tunnel bridge (H,I)
Traffic arrives on the network host via the GRE tunnel attached to br-tun. This bridge has a flow table very similar to br-tun on the compute host。
这个过程，非常类似上一个环节，只是数据处理的方向是反的，是从tunnel数据转向vlan的。
Network host: integration bridge
The integration bridge on the network controller serves to connect instances to network services, such as routers and DHCP servers.
Network host: DHCP server (O,P)
Each network for which DHCP is enabled has a DHCP server running on the network controller. The DHCP server is an instance of dnsmasq running inside a network namespace. A network namespace is a Linux kernel facility that allows groups of processes to have a network stack (interfaces, routing tables, iptables rules) distinct from that of the host.
这个namespace，在openstack环境下，可以通过ip netns查看，我的devstack测试平台上，执行这个命令得到下面的内容：
1 [[email protected]10-121-5-244 ~]# ip netns 2 qrouter-a7bf0605-3474-4f5c-bcbd-98a11bd175d4 3 qdhcp-77e45b4b-3238-467e-b38f-33dd86a5f83b
上面的qdhcp namespace的server主要是服务于private网络的IP地址分配的。而qrouter则主要负责数据对外的路由的。
在dhcp的namespace里面执行查看地址的命令，如下，看看都是什么信息呢？
1 [[email protected]10-121-5-244 ~]# ip netns exec qdhcp-77e45b4b-3238-467e-b38f-33dd86a5f83b ip addr 2 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 3 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 4 inet 127.0.0.1/8 scope host lo 5 valid_lft forever preferred_lft forever 6 inet6 ::1/128 scope host 7 valid_lft forever preferred_lft forever 8 9: tapf2d4361c-39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 9 link/ether fa:16:3e:85:fa:a0 brd ff:ff:ff:ff:ff:ff 10 inet 10.0.0.2/24 brd 10.0.0.255 scope global tapf2d4361c-39 11 valid_lft forever preferred_lft forever 12 inet6 fdf8:e668:72e6:0:f816:3eff:fe85:faa0/64 scope global mngtmpaddr dynamic 13 valid_lft 86354sec preferred_lft 14354sec 14 inet6 fe80::f816:3eff:fe85:faa0/64 scope link 15 valid_lft forever preferred_lft forever
You can find the dnsmasq process associated with this namespace by search the output of ps for the id (the number after qdhcp- in the namespace name)
1 [[email protected]10-121-5-244 ~]# ps -fe | grep 77e45b4b-3238-467e-b38f-33dd86a5f83b 2 nobody 1658 1 0 Jan12 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/77e45b4b-3238-467e-b38f-33dd86a5f83b/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/77e45b4b-3238-467e-b38f-33dd86a5f83b/host --addn-hosts=/opt/stack/data/neutron/dhcp/77e45b4b-3238-467e-b38f-33dd86a5f83b/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/77e45b4b-3238-467e-b38f-33dd86a5f83b/opts --dhcp-leasefile=/opt/stack/data/neutron/dhcp/77e45b4b-3238-467e-b38f-33dd86a5f83b/leases --dhcp-match=set:ipxe,175 --bind-interfaces --interface=tapf2d4361c-39 --dhcp-range=set:tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
Network host: Router (M,N)
A Neutron router is a network namespace with a set of routing tables and iptables rules that performs the routing between subnets.
下面，通过ip netns来看看router namespace下的地址信息（router的namespace，前面已经列举出来了的）：
1 [[email protected]10-121-5-244 ~]# ip netns exec qrouter-a7bf0605-3474-4f5c-bcbd-98a11bd175d4 ip addr 2 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 3 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 4 inet 127.0.0.1/8 scope host lo 5 valid_lft forever preferred_lft forever 6 inet6 ::1/128 scope host 7 valid_lft forever preferred_lft forever 8 10: qr-c36567ef-4d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 9 link/ether fa:16:3e:54:fa:80 brd ff:ff:ff:ff:ff:ff 10 inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-c36567ef-4d 11 valid_lft forever preferred_lft forever 12 inet6 fe80::f816:3eff:fe54:fa80/64 scope link 13 valid_lft forever preferred_lft forever 14 11: qg-5d13ee78-0a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 15 link/ether fa:16:3e:37:7c:95 brd ff:ff:ff:ff:ff:ff 16 inet 172.24.4.2/24 brd 172.24.4.255 scope global qg-5d13ee78-0a 17 valid_lft forever preferred_lft forever 18 inet6 2001:db8::1/64 scope global 19 valid_lft forever preferred_lft forever 20 inet6 fe80::f816:3eff:fe37:7c95/64 scope link 21 valid_lft forever preferred_lft forever 22 12: qr-2e4c72e5-5f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 23 link/ether fa:16:3e:cd:92:ad brd ff:ff:ff:ff:ff:ff 24 inet6 fdf8:e668:72e6::1/64 scope global 25 valid_lft forever preferred_lft forever 26 inet6 fe80::f816:3eff:fecd:92ad/64 scope link 27 valid_lft forever preferred_lft forever
这个当中，可以看看其路由信息：
1 [[email protected]10-121-5-244 ~]# ip netns exec qrouter-a7bf0605-3474-4f5c-bcbd-98a11bd175d4 ip route 2 default via 172.24.4.1 dev qg-5d13ee78-0a 3 10.0.0.0/24 dev qr-c36567ef-4d proto kernel scope link src 10.0.0.1 4 172.24.4.0/24 dev qg-5d13ee78-0a proto kernel scope link src 172.24.4.2
nat表的信息如下：
1 [[email protected]10-121-5-244 ~]# ip netns exec qrouter-a7bf0605-3474-4f5c-bcbd-98a11bd175d4 iptables -t nat -S 2 -P PREROUTING ACCEPT 3 -P INPUT ACCEPT 4 -P OUTPUT ACCEPT 5 -P POSTROUTING ACCEPT 6 -N neutron-l3-agent-OUTPUT 7 -N neutron-l3-agent-POSTROUTING 8 -N neutron-l3-agent-PREROUTING 9 -N neutron-l3-agent-float-snat 10 -N neutron-l3-agent-snat 11 -N neutron-postrouting-bottom 12 -A PREROUTING -j neutron-l3-agent-PREROUTING 13 -A OUTPUT -j neutron-l3-agent-OUTPUT 14 -A POSTROUTING -j neutron-l3-agent-POSTROUTING 15 -A POSTROUTING -j neutron-postrouting-bottom 16 -A neutron-l3-agent-POSTROUTING ! -i qg-5d13ee78-0a ! -o qg-5d13ee78-0a -m conntrack ! --ctstate DNAT -j ACCEPT 17 -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697 18 -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat 19 -A neutron-l3-agent-snat -o qg-5d13ee78-0a -j SNAT --to-source 172.24.4.2 20 -A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 172.24.4.2 21 -A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
Network host: External traffic (K,L)
"External" traffic flows through br-ex via the qg-5d13ee78-0a interface in the router name space, which connects to br-ex。