Choosing a Watermark Delay for Spark Structured Streaming State
Pick a watermark delay by measuring event lateness, then verify with state-store metrics and console output.
Loading…
ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Pick a watermark delay by measuring event lateness, then verify with state-store metrics and console output.
When IoT devices send logs late, dashboards can become stale. Spark’s event‑time watermarks let you drop or keep late data automatically. This guide shows how to set up watermarks, a working example, and the trade‑offs you need to monitor.