Cutting One Round-Trip with Linux TCP Fast Open

The problem: every short connection pays a handshake tax

You run a small HTTP service on Linux. Clients connect, send a single request, and disconnect. Before the first byte of application data can leave the client, TCP completes a three-way handshake: SYN, SYN-ACK, ACK. On a fast local network that delay is small. Across the internet or over a lossy link, it can add tens or hundreds of milliseconds. For short-lived connections, the handshake can dominate the total time.

Linux TCP Fast Open (TFO) removes one round-trip after the first connection. The client sends data inside the initial SYN packet, provided it already holds a cookie from a previous handshake to the same destination.

Check and enable TCP Fast Open

The behavior is controlled by the kernel parameter net.ipv4.tcp_fastopen, exposed at /proc/sys/net/ipv4/tcp_fastopen:

• 0 — disabled
• 1 — client only
• 2 — server only
• 3 — both client and server

Check the current value (any user, read-only):

cat /proc/sys/net/ipv4/tcp_fastopen

Expected output is a single digit, typically 0 on a default host.

To enable both sides temporarily (requires root):

sudo sysctl -w net.ipv4.tcp_fastopen=3

The command prints the new value. This changes kernel networking state immediately and affects new TCP connections on the host. Existing connections are not altered.

To make the change persist across reboots, create /etc/sysctl.d/99-tcp-fastopen.conf:

net.ipv4.tcp_fastopen = 3

Then apply it:

sudo sysctl --system

Worked example: verify with curl and tcpdump

You need a local TCP server. In one terminal, start a minimal listener (run as a normal user):

python3 -m http.server 8080 --bind 127.0.0.1

In another terminal, start a packet capture on the loopback interface (requires root):

sudo tcpdump -i lo -s 0 -n -w tfo-test.pcap port 8080

Now request a page with TFO enabled in curl:

curl --tcp-fastopen -o /dev/null -s -w 'connect=%{time_connect}\n' http://127.0.0.1:8080/

Repeat the same request. Because the first response delivered a TFO cookie, the second SYN can carry data.

Stop the capture and inspect it:

tcpdump -r tfo-test.pcap -nn

Look at the first SYN from the client to the server on the second connection. With TFO working, that packet contains application data after the TCP header. Without TFO, data appears only after the SYN-ACK and ACK. You can also compare timing:

curl --tcp-fastopen -o /dev/null -s -w 'tfo=%{time_connect}\n' http://127.0.0.1:8080/
curl -o /dev/null -s -w 'plain=%{time_connect}\n' http://127.0.0.1:8080/

On loopback the difference is often negligible; the gap appears over higher round-trip times. If --tcp-fastopen is missing, your curl build does not support it.

Trade-offs and limitations

• Middleboxes can break it. Older firewalls and load balancers may strip the TCP Fast Open option or drop the packet entirely. The connection then falls back to a normal handshake after retransmits, which adds latency instead of removing it.
• Security exposure. A small SYN can trigger a larger server response. On internet-facing services, combine TFO with rate limiting and avoid sending large responses to unauthenticated fast-open data.
• Cookie binding. The TFO cookie is tied to the destination IP. NAT, IP changes, or client roaming invalidate it and force a fallback handshake.
• Idempotency matters. Data in a SYN may be retransmitted if the packet is lost. Avoid TFO for non-idempotent writes unless the application can safely handle repetition.
• Limited benefit for long-lived flows. Once a connection carries sustained traffic, one saved round-trip is amortized away.

Rollback

Because sysctl -w changes live kernel state, you can revert immediately:

sudo sysctl -w net.ipv4.tcp_fastopen=0

To persist the rollback, remove or comment the line in /etc/sysctl.d/99-tcp-fastopen.conf and run sudo sysctl --system.

Before you enable it broadly

1. Check /proc/sys/net/ipv4/tcp_fastopen on client and server hosts.
2. Enable on a test host and run the curl/tcpdump verification above.
3. Test through the real network path, including firewalls and load balancers; loopback will not reveal middlebox drops.
4. Compare connection-establishment times with and without TFO under realistic load.
5. Roll back to 0 if you see retransmits, handshake failures, or no timing improvement.

TCP Fast Open is a targeted latency optimization, not a blanket speedup. Verify it on your actual path before making it the default.