Path MTU on Linux when there are multiple routes with different MTUs
Aug 2
On my Linux (actually OpenWrt) device, I have two default routes:
root@<>:~# ip route
default via 10.111.7.76 dev eth1 proto static metric 10 linkdown
default via <redacted> dev wwan0 proto static src <redacted> metric 40
eth1 has an MTU of 1500 and wwan0 has an MTU of 1300.
eth1 is currently down and hence the linkdown in the output above.
I see this:
root@<>:~# ip route get 1.1.1.1
1.1.1.1 via 10.111.7.76 dev eth1 src 10.111.7.50 uid 0
cache
root@<>:~#
This is curious because it seems to be reporting the eth1 (linkdown) interface, but I guess that's OK because it just looks up the cache. When I actually ping 1.1.1.1 or any other internet destination, the packet actually goes out on wwan0 as you'd expect.
The problem happens when I make a TCP connection. The SYN goes out with an MSS value of 1460. I'm guessing this is because of the route cache that determines that the packet would go out on eth1 which has an MTU of 1500. Of course this doesn't work because the packet eventually goes out on wwan0, which needs to have an MSS of 1260.
I fixed this with an iptables rule that adjusts the MSS. But it seems odd that a fairly normal scenario like this requires an iptables workaround. Am I looking at a bug, or some config that will cause the linkdown interface to be removed from the route cache?
linux1 answer
Accepted answer · original discussion
Aug 2
linkdown only has an effect when the ignore_routes_with_linkdown sysctl is enabled (either per-interface or globally), which it isn't by default:
sysctl net.ipv4.conf.all.ignore_routes_with_linkdown=1
sysctl net.ipv6.conf.all.ignore_routes_with_linkdown=1
Route lookups are indeed cached (which is also where the learned PMTUD information is stored), but I don't know how this leads to the inconsistency of ping using wwan0 regardless.
ip route show cache
ip route flush cache
Often, unfortunately, if your device needs to forward packets from other hosts, then iptables/nft-based MSS clamping may be unavoidable. Even if everything is good on your side, some remote hosts may just completely ignore the ICMP PMTUD notices that your ISP sends them, and would still get stuck re-sending the same large segment towards you.
(Individual Linux-based hosts have a net.ipv4.tcp_mtu_probing sysctl – shared with IPv6 despite the name – and some other operating systems might perform that as well; but you can't guarantee that all of your devices will, so it might still be necessary for the gateway to clamp MSS.)
3 question comments
Use comments to ask for clarification. Post a solution as an answer.
Aug 2