tcpdump 精准分析vxlan网络

一、相关概念

VXLAN（Virtual eXtensible Local Area Network，虚拟扩展局域网），是由IETF定义的NVO3（Network Virtualization over Layer 3）标准技术之一，是对传统VLAN协议的一种扩展。VXLAN的特点是将L2的以太帧封装到UDP报文（即L2 over L4）中，并在L3网络中传输。VXLAN本质上是一种隧道技术，在源网络设备与目的网络设备之间的IP网络上，建立一条逻辑隧道，将用户侧报文经过特定的封装后通过这条隧道转发。从用户的角度来看，接入网络的服务器就像是连接到了一个虚拟的二层交换机的不同端口上，可以方便地通信。
在这里插入图片描述

在这里插入图片描述

我们知道，在云计算中，大部分overlay网络都是基于vxlan实现的，在一般云网络运维场景下，外层报文的三层头都是物理机（宿主机）的 ip 地址，虚拟机实际通信的源/目的 ip 地址都是被封装到了内层报文中，所以需要弄明白虚机的流量路径，所以抓包分析一下。

二、抓包示例

PS：对于dpdk场景下tcpdump使用不了的话可以尝试比如ovs-tcpdump，用法一样。

2.1 抓包示例1

对于内层是 ICMP 报文的 vxlan 报文可使用如下命令进行过滤抓包：

tcpdump 'udp[39]=1' -nv -i bond1

Vxlan报文格式是在原始报文前封装了Vxlan报文，命令中的“39”是指从OUT UDP header协议报文启始位置0偏移至39字节（偏移量40字节），其中包括：OUT UDP header（8字节）+VXLAN header(8字节）+Inner Ethernet header（14字节）+Inner IP header中Protocol位置（10字节，详见下文“解释IP header报文格式的含义”）=40字节

2.2 抓包示例2

同理，对于内层报文源 ip 地址为 172.16.12.7 的报文可使用如下命令进行过滤抓包,这里需要将 ip 地址转换为四字节十六进制数：

tcpdump 'udp[42:4]=0xAC100C07' -nv -i bond1

2.3 抓包示例3

对于内层报文源或者目的 ip 地址为172.16.12.7 的报文可使用如下命令进行过滤抓包：

tcpdump 'udp[42:4]=0xAC100C07' or 'udp[46:4]=0xAC100C07' -nv -i bond1

udp[42:4] 的含义是从UDP Header 启始位置0偏移42字节，数据长度4字节，同理udp[46:4]

2.4 抓包示例3

对于内层报文中通信两端 ip 地址为 172.16.12.7 和 157.255.219.143 的报文可使用如下命令进行过滤抓包：

sip='0xAC100C07' ; dip='0x9DFFDB8F'
tcpdump \(\("udp[42:4]=${sip}" and "udp[46:4]=${dip}"\) or \("udp[46:4]=${sip}" and "udp[42:4]=${dip}"\)\) -nv -i bond1

三、抓包测试

下面基于tcpdump抓包分析，client 下载 qq.com首页，了解虚机流量经宿主机之后的走向

虚机：172.16.12.7

宿主机：*.224.129.215

网关节点：10.224.145.3

在虚拟机上：

#wget http://qq.com/index.html

宿主机上面tcpdump抓包

#sip='0xAC100C07' ; dip='0x9DFFDB8F' 
#tcpdump \(\("udp[42:4]=${sip}" and "udp[46:4]=${dip}"\) or \("udp[46:4]=${sip}" and "udp[42:4]=${dip}"\)\) -nnnee -i bond1

抓包详情如下：

17:33:26.216996 b8:ce:f6:3b:59:aa > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 124: 10.224.129.215.33122 > 10.224.145.3.4789: VXLAN, flags [I] (0x08), vni 96
fa:16:3e:51:97:cc > fa:16:3f:76:d9:12, ethertype IPv4 (0x0800), length 74: 172.16.12.7.53260 > 157.255.219.143.80: Flags [S], seq 3270421384, win 28200, options [mss 1410,sackOK,TS val 170973571 ecr 0,nop,wscale 7], length 0
17:33:26.253795 00:00:5e:00:01:01 > b8:ce:f6:3b:59:aa, ethertype IPv4 (0x0800), length 116: 10.224.145.3.15853 > 10.224.129.215.4789: VXLAN, flags [I] (0x08), vni 96
fe:16:4f:00:00:00 > fa:16:3f:76:d9:19, ethertype IPv4 (0x0800), length 66: 157.255.219.143.80 > 172.16.12.7.53260: Flags [S.], seq 3770813300, ack 3270421385, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
17:33:26.254299 b8:ce:f6:3b:59:aa > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 104: 10.224.129.215.33122 > 10.224.145.3.4789: VXLAN, flags [I] (0x08), vni 96
fa:16:3e:51:97:cc > fa:16:3f:76:d9:12, ethertype IPv4 (0x0800), length 54: 172.16.12.7.53260 > 157.255.219.143.80: Flags [.], ack 3770813301, win 221, length 0
17:33:26.254338 b8:ce:f6:3b:59:aa > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 218: 10.224.129.215.33122 > 10.224.145.3.4789: VXLAN, flags [I] (0x08), vni 96
fa:16:3e:51:97:cc > fa:16:3f:76:d9:12, ethertype IPv4 (0x0800), length 168: 172.16.12.7.53260 > 157.255.219.143.80: Flags [P.], seq 3270421385:3270421499, ack 3770813301, win 221, length 114: HTTP: GET /index.html HTTP/1.1
17:33:26.291074 00:00:5e:00:01:01 > b8:ce:f6:3b:59:aa, ethertype IPv4 (0x0800), length 104: 10.224.145.3.15853 > 10.224.129.215.4789: VXLAN, flags [I] (0x08), vni 96
fe:16:4f:00:00:00 > fa:16:3f:76:d9:19, ethertype IPv4 (0x0800), length 54: 157.255.219.143.80 > 172.16.12.7.53260: Flags [.], ack 3270421499, win 506, length 0
17:33:26.291197 00:00:5e:00:01:01 > b8:ce:f6:3b:59:aa, ethertype IPv4 (0x0800), length 437: 10.224.145.3.15853 > 10.224.129.215.4789: VXLAN, flags [I] (0x08), vni 96
fe:16:4f:00:00:00 > fa:16:3f:76:d9:19, ethertype IPv4 (0x0800), length 387: 157.255.219.143.80 > 172.16.12.7.53260: Flags [P.], seq 3770813301:3770813634, ack 3270421499, win 506, length 333: HTTP: HTTP/1.1 302 Moved Temporarily
17:33:26.291416 b8:ce:f6:3b:59:aa > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 104: 10.224.129.215.33122 > 10.224.145.3.4789: VXLAN, flags [I] (0x08), vni 96
fa:16:3e:51:97:cc > fa:16:3f:76:d9:12, ethertype IPv4 (0x0800), length 54: 172.16.12.7.53260 > 157.255.219.143.80: Flags [.], ack 3770813634, win 229, length 0
17:33:26.652984 b8:ce:f6:3b:59:aa > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 104: 10.224.129.215.33122 > 10.224.145.3.4789: VXLAN, flags [I] (0x08), vni 96
fa:16:3e:51:97:cc > fa:16:3f:76:d9:12, ethertype IPv4 (0x0800), length 54: 172.16.12.7.53260 > 157.255.219.143.80: Flags [F.], seq 3270421499, ack 3770813634, win 229, length 0
17:33:26.689741 00:00:5e:00:01:01 > b8:ce:f6:3b:59:aa, ethertype IPv4 (0x0800), length 104: 10.224.145.3.15853 > 10.224.129.215.4789: VXLAN, flags [I] (0x08), vni 96
fe:16:4f:00:00:00 > fa:16:3f:76:d9:19, ethertype IPv4 (0x0800), length 54: 157.255.219.143.80 > 172.16.12.7.53260: Flags [F.], seq 3770813634, ack 3270421500, win 506, length 0
17:33:26.690134 b8:ce:f6:3b:59:aa > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 104: 10.224.129.215.33122 > 10.224.145.3.4789: VXLAN, flags [I] (0x08), vni 96
fa:16:3e:51:97:cc > fa:16:3f:76:d9:12, ethertype IPv4 (0x0800), length 54: 172.16.12.7.53260 > 157.255.219.143.80: Flags [.], ack 3770813635, win 229, length 0

备注:

通过网站 https://www.osgeo.cn/app/sc126 ，将ip地址 172.16.12.7 转换为16进制 0xAC100C07

或者通过如下python代码将ip地址转为16进制方式

import socket
from binascii import hexlify
import sysary=sys.argv[1]
packed_ip_addr = socket.inet_aton(ary)
hexStr=hexlify(packed_ip_addr)
print('IP %s : 0x' % ary +hexStr)

参考:

https://thiscute.world/posts/linux-virtual-network-interfaces/

https://cloud.tencent.com/developer/article/2336137

https://zhuanlan.zhihu.com/p/684746396