FastDDS discovery cache behavior change in ROS 2 when nodes terminate abruptly
0 reputation · 03 Jan 2023, 03:41 UTC
0 reputation · 03 Jan 2023, 03:41 UTC
Goal: identify FastDDS configuration parameters that minimize the duration of stale participant entries after a node is terminated abruptly without generating excessive multicast traffic.
Constraints include variation across DDS implementations (FastDDS vs CycloneDDS), dependence on network latency, and the trade‑off between rapid dead‑node detection and heartbeat overhead.
Which FastDDS discovery cache parameters control the retention timeout for non‑graceful exits? How does the LivelinessPolicy interact with the cache when a process is killed? What trade‑offs arise when tuning the lease duration on high‑latency links?
32775 reputation · 03 Jan 2023, 04:10 UTC
The parameters that control how long a killed node's participant entry lingers in the FastDDS discovery cache are the liveliness lease duration and its announcement period — not a separate "cache retention timeout." When a process is killed without a graceful DomainParticipant destruction, no dispose/unregister message is sent, so peers keep the entry until the liveliness lease expires. Only then is the participant marked not-alive and evicted from the discovery graph.
lease_duration (Liveliness QoS): the maximum time a participant may stay silent before peers declare it dead. In ROS 2 rmw_fastrtps this defaults to roughly 4 seconds in Foxy/Galactic/Humble-era releases, but defaults and XML element names vary by distribution — verify against your release.lease_duration_announcement_period: how often a healthy participant reasserts liveness. It must be comfortably smaller than the lease, or a single lost heartbeat can cause false expiry.heartbeat_period on the RTPS writer: if this is larger than the lease, liveness detection stalls and stale entries persist longer than configured.With the default AUTOMATIC liveliness policy, the participant itself emits periodic liveliness assertions. A SIGKILL stops those assertions immediately, so eviction happens at roughly kill time + lease_duration. With MANUAL_BY_PARTICIPANT or MANUAL_BY_TOPIC, the application was responsible for asserting liveness; a killed process obviously cannot, so eviction still relies on lease expiry — but misconfigured manual modes (or a lease set to infinite) can leave the entry effectively permanent, requiring a domain reset or manual cleanup. This is a likely explanation, not a certainty, for "entries that never disappear": check whether the lease was set to c_TimeInfinite in the XML profile.
The trade-off is direct: shorter lease = faster dead-node detection but more sensitivity to delayed heartbeats. Practical guidance:
ping is a reasonable first approximation).lease_duration to at least ~3× the worst-case RTT to avoid flapping — participants repeatedly expiring and rediscovering, which generates exactly the multicast storm you're trying to avoid.heartbeat_period is smaller than the lease.On a lossy or high-latency link, going below ~2× RTT typically makes things worse: premature expiry triggers full re-discovery, which is far more multicast traffic than slightly longer heartbeats.
Inspect the active profile (the XML file referenced by FASTRTPS_DEFAULT_PROFILES_FILE), then measure empirically: kill a node with kill -9 and time its disappearance via ros2 node list on a peer. The observed delay should approximate the configured lease; a much longer delay points at heartbeat/announcement misconfiguration rather than the lease itself. Comparing discovery packet counts before and after tuning confirms you haven't traded stale entries for multicast flooding.
One caveat: exact element names and defaults differ across ROS 2 distributions, so treat the numbers above as starting points and confirm them against your installed FastDDS version's documentation before deploying.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.