1、物理环境 172.16.0.55 k8s1 master 172.16.0.57 k8s2 node 172.16.0.47 k8s3 node [[email protected] ~]# docker ps |grep per a394542f2340 alectolytic/netperf:latest "sleep 36000000" 16 hours ago Up 16 hours k8s_netperf.468d5a53_netperf-903608992-kbdqx_default_fe8c2f70-e21a-11e6-9e5d-0025904aabd6_101520d3 c951205fcdb5 gcr.io/google_containers/pause-amd64:3.0 "/pause" 16 hours ago Up 16 hours k8s_POD.d8dbe16c_netperf-903608992-kbdqx_default_fe8c2f70-e21a-11e6-9e5d-0025904aabd6_cdb8ccde 6be27d13d319 networkstatic/iperf3:latest "sleep 3600000" 16 hours ago Up 16 hours k8s_iperf3.77255c12_iperf3-225180425-dkqlp_default_8586bf7c-e217-11e6-9e5d-0025904aabd6_2d2f3f61 0911410b49ff gcr.io/google_containers/pause-amd64:3.0 "/pause" 16 hours ago Up 16 hours k8s_POD.d8dbe16c_iperf3-225180425-dkqlp_default_8586bf7c-e217-11e6-9e5d-0025904aabd6_c86ee5f0 [[email protected] ~]# [[email protected] ~]# docker inspect 6be27d13d319|grep Pid "Pid": 21333, "PidMode": "", "PidsLimit": 0, [[email protected] ~]# [[email protected] ~]# mkdir /var/run/net netreport/ net-snmp/ [[email protected] ~]# mkdir /var/run/netns [[email protected] ~]# [[email protected] ~]# ln -s /proc/21333/ns/net /var/run/netns/6be27d13d319 [[email protected] ~]# ip netns 6be27d13d319 (id: 4) [[email protected] ~]# ip netns exec 6be27d13d319 ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether b2:2f:eb:0f:9d:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.167.84/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::b02f:ebff:fe0f:9d04/64 scope link valid_lft forever preferred_lft forever You have mail in /var/spool/mail/root [[email protected] ~]# [[email protected] ~]# ip netns exec 6be27d13d319 tcpdump -i any icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
[[email protected] ~]# ip netns exec 6be27d13d319 ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether b2:2f:eb:0f:9d:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.167.84/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::b02f:ebff:fe0f:9d04/64 scope link valid_lft forever preferred_lft forever [[email protected] ~]# [[email protected] ~]# ip netns exec 6be27d13d319 ethtool -S eth0 NIC statistics: peer_ifindex: 29 [[email protected] ~]# [[email protected] ~]# ip route default via 172.16.0.254 dev enp1s0f0 proto static metric 100 172.16.0.0/24 dev enp1s0f0 proto kernel scope link src 172.16.0.57 metric 100 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 192.168.23.192/26 via 172.16.0.49 dev enp1s0f0 proto bird 192.168.86.0/26 via 172.16.0.55 dev enp1s0f0 proto bird blackhole 192.168.167.64/26 proto bird 192.168.167.84 dev caliab5c3cbfda4 scope link 192.168.167.85 dev caliec347e5f7b0 scope link 192.168.167.87 dev cali59ee912b1e2 scope link 192.168.167.124 dev calia1345165869 scope link 192.168.167.125 dev calif8ea0c0f011 scope link [[email protected] ~]# [[email protected] ~]# ip addr 2: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:25:90:4a:ad:ea brd ff:ff:ff:ff:ff:ff inet 172.16.0.57/24 brd 172.16.0.255 scope global enp1s0f0 valid_lft forever preferred_lft forever inet6 2001:470:fb3c:0:225:90ff:fe4a:adea/64 scope global mngtmpaddr dynamic valid_lft 2455832sec preferred_lft 468632sec inet6 fe80::225:90ff:fe4a:adea/64 scope link valid_lft forever preferred_lft forever 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:ae:fa:da:33 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 scope global docker0 valid_lft forever preferred_lft forever 5: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether ce:b2:25:23:f9:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::ccb2:25ff:fe23:f987/64 scope link valid_lft forever preferred_lft forever 6: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether de:81:bc:83:0d:b5 brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet6 fe80::dc81:bcff:fe83:db5/64 scope link valid_lft forever preferred_lft forever 29: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether 76:b0:6b:12:ef:4d brd ff:ff:ff:ff:ff:ff link-netnsid 4 inet6 fe80::74b0:6bff:fe12:ef4d/64 scope link valid_lft forever preferred_lft forever
以如下两个容器为例来说明容器跨主机节点的通讯
iperf3-225180425-00psw （容器1） 192.168.23.194 k8s3
iperf3-225180425-dkqlp （容器2） 192.168.167.84 k8s2
对端物理主机
calico架构
Felix，Calico Agent，跑在每台需要运行Workload的节点上，主要负责配置路由及ACLs等信息来确保Endpoint的连通状态；
etcd，分布式键值存储，主要负责网络元数据一致性，确保Calico网络状态的准确性；
BGP Client（BIRD）, 主要负责把Felix写入Kernel的路由信息分发到当前Calico网络，确保Workload间的通信的有效性；
BGP Route Reflector（BIRD），大规模部署时使用，摒弃所有节点互联的 mesh 模式，通过一个或者多个BGP Route Reflector来完成集中式的路由分发。
[[email protected] ~]# calicoctl config get asNumber
64512
[[email protected] ~]# calicoctl config get nodeToNodeMesh
on
[[email protected] ~]# calicoctl node status
时间： 2024-10-11 05:30:51