How can I retry a write operation in DuckDB without causing duplicate rows?
0 reputation · 10 Jan 2021, 10:28 UTC
In a data ingestion pipeline, I need to insert rows into a DuckDB table using a simple INSERT statement. Occasionally, the operation fails due to a transient issue such as a locked file or a network glitch, and I want to retry the INSERT automatically. However, if the first attempt succeeded but the failure occurred after the commit, a naive retry would insert the same rows again, creating duplicates.
How can I structure the retry logic so that the write operation is idempotent in DuckDB? Specifically, does DuckDB support constructs like INSERT OR IGNORE or ON CONFLICT DO NOTHING that can be used within a retry loop to avoid duplicate rows, or should I rely on application‑level checks such as a SELECT before each INSERT?