pfSense TCP retransmissions flagged as out-of-state: when does sloppy state tracking help?

Short answer

By default, pf's state engine enforces TCP sequence-window checks, so a retransmitted segment that falls outside the currently accepted window is treated as out-of-state: it is dropped and, if logging is enabled on the rule, shows up as a state mismatch (messages such as "state mismatch" or "bad tcp window"). It is not silently associated with the existing state. Enabling the per-rule sloppy state option relaxes exactly that sequence-number validation, and the timeouts that matter for very late retries are the TCP state timeouts under System > Advanced > Firewall & NAT, chiefly TCP.ESTABLISHED.

1. What happens to an out-of-window retransmission

Once a pass rule creates a state entry, later packets in the flow are matched against the state table by 5-tuple, not re-evaluated against rules. With strict state tracking (the default), pf also validates that TCP sequence numbers fall inside the negotiated window. A retransmission that arrives after the window has advanced past it fails that check, is dropped, and is logged as a state mismatch if the rule logs blocked traffic. The state entry itself is not torn down by a single bad packet, but the application sees a stall and may re-issue writes or time out.

2. What sloppy state actually relaxes

The sloppy state option on a pass rule keeps 5-tuple matching and basic flag/header sanity checks, but stops enforcing the strict sequence-number window. A packet that matches an existing connection's 5-tuple is accepted even if its sequence number is outside the current window. That is precisely what you want when latency, asymmetric routing, or a middlebox produces legitimate retries that strict tracking would reject.

The trade-off is real but bounded: sloppy state does not let arbitrary traffic through — the packet must still match an existing state — but it does widen acceptance to replayed or injected segments that would otherwise be rejected by sequence validation. Scope it to the specific rule and direction where you see the problem, ideally on trusted internal paths, rather than enabling it globally.

3. Which timeouts matter for late retries

If a retransmission arrives after the state entry has expired, the packet is evaluated as new traffic and will be dropped unless a pass rule matches it. The relevant knobs under System > Advanced > Firewall & NAT are the TCP timeouts — TCP.ESTABLISHED is the one that governs an open connection, with TCP.OPEN, TCP.CLOSING, TCP.FIN_WAIT, and TCP.TIME_WAIT covering setup and teardown. If your path delay plus retransmission back-off approaches the ESTABLISHED timeout, raising it keeps the state alive long enough for late retries to match. Do this conservatively: very large timeouts leave stale entries consuming state-table memory.

Verifying the behavior

1. Enable logging on the relevant pass rule, then reproduce the delayed-retry condition.
2. Check Status > System Logs > Firewall for "state mismatch" / "bad tcp window" drops with sloppy state off.
3. Enable sloppy state on that rule only, repeat, and confirm the same traffic now passes and the entry persists in Diagnostics > States.
4. If retries still fail, compare the state entry's observed lifetime against the configured TCP.ESTABLISHED timeout and raise it above the measured delay.

Exact log wording and timeout defaults vary between pfSense CE and Plus releases, so confirm against your version's logs before tuning. One detail that would sharpen the recommendation: whether the path is actually asymmetric (different return route) — if so, fixing the routing or using sloppy state on both legs matters more than any timeout change.