Neutron MTU Path Discovery Fails to Detect Correct MTU in VXLAN-Production Deployments

Answer

Enabling mtu_path_discovery does not guarantee that the reported MTU matches the smallest MTU along the encapsulated VXLAN path for every kernel/vSwitch combination. The feature succeeds only when:

1. The neutron agent can send ICMP "Fragmentation Needed" (type 3, code 4) packets and receive the replies.
2. The Linux kernel on the compute node supports the IP_PMTU_DISCOVER socket option and correctly handles VXLAN encapsulation offset (kernels ≥ 4.14 are known to work reliably).
3. The virtual switch (OVS or Linux bridge) does not drop or alter those ICMP packets.

If any of these conditions fail, the agent falls back to the configured physical NIC MTU minus the VXLAN overhead (typically 1500 − 50 = 1450 bytes), which may be too low (causing unnecessary fragmentation) or too high (leading to packet drops and TCP stalls).

Likely explanation vs. confirmed facts

Confirmed facts (based on the research brief):

• mtu_path_discovery relies on ICMP fragmentation‑needed messages from neutron agents.
• If those packets are filtered by security groups or firewalls, discovery fails and the agent uses the physical MTU minus overhead.
• Kernels prior to 4.14 may mishandle the VXLAN offset, producing incorrect MTU reports.
• The feature works consistently only when the kernel supports IP_PMTU_DISCOVER and the vSwitch does not alter ICMP.

Likely explanation for failure: In many production clouds, compute‑node security groups or upstream firewalls block ICMP type 3/code 4 in either direction, preventing the agent from learning the true path MTU. Additionally, older kernels or mis‑configured OVS bridges may silently drop the needed packets.

Steps needed for this case

1. Verify that ICMP "Fragmentation Needed" is allowed in both directions between compute nodes and the physical infrastructure (check security groups, firewall rules, and any upstream ACLs).
2. Check the neutron‑agent log for messages like MTU path discovery succeeded or MTU path discovery failed. Example command:

   journalctl -u neutron-openvswitch-agent | grep -i "mtu path discovery"
   

3. Confirm kernel version on the compute node (uname -r) is ≥ 4.14; if older, consider upgrading or applying the relevant kernel patch.
4. Ensure the vSwitch does not filter ICMP: for OVS, verify that no flow drops ICMP type 3/code 4 (ovs-ofctl dump-flows br-int); for Linux bridge, check ebtables/nftables rules.
5. If discovery succeeds, compare the reported MTU (ip link show dev vxlan*) with the value from ping -M do -s 1472 <target> from an instance; adjust until no fragmentation‑needed replies are seen.
6. As a fallback, enable TCP MSS clamping on the instance (e.g., via iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mtu 1450) or statically configure the tenant network MTU to the minimum known path MTU (often 1450 bytes for VXLAN over Ethernet).
7. Monitor for TCP retransmits or stalls (ss -ti or netstat -s) after large‑data transfers; if observed, revisit ICMP allowance or static MTU settings.

Missing diagnostic detail

If you cannot confirm whether ICMP type 3/code 4 is allowed in both directions, you must verify that before relying on mtu_path_discovery; the recommendation changes from "trust the feature" to "enforce static MTU or MSS clamping" until the ICMP path is proven.