Limits on retrying FileStore writes without duplicate data

Handling Idempotency in FileStore Writes

There are no built-in flags or parameters in the org.eclipse.core.filesystem.FileStore API or the Eclipse Job API that automatically suppress duplicate data during retries. Because FileStore is a generic abstraction for diverse file systems (local, remote, or custom EFS providers), the platform does not enforce a global idempotency contract. Responsibility for preventing duplicate data rests entirely with the caller.

Likely Explanation for API Design

The FileStore API is designed to remain portable across heterogeneous backends. Since different file system providers offer varying levels of atomicity and support for operations like appending or truncating, implementing a universal "retry-safe" write would require the platform to make assumptions that might not hold true for all providers. By leaving recovery to the developer, the API avoids introducing inconsistent behavior across different environments.

Recommended Patterns for Idempotent Retries

To ensure that retrying a failed write does not result in corrupted or duplicated data, use one of the following patterns:

• The Atomic Move Pattern: Instead of writing directly to the target file, write the data to a temporary file in the same directory. Once the write is complete and verified, use IFileStore.move with the EFS.OVERWRITE option to rename the temporary file to the target destination. If the process fails and is retried, the temporary file is simply overwritten, and the target file only changes once the move succeeds.
• Truncate-and-Rewrite: If you are using openOutputStream, ensure you are not using EFS.APPEND unless you have a mechanism to track exactly how many bytes were written before the CoreException occurred. For most cases, rewriting the entire file from the start is the safest way to ensure idempotency.
• State Verification: Before initiating a retry, check the current size or a content hash of the target file. If the file already matches the expected state, the retry can be skipped.

Verification and Safety

Before deploying a retry loop, verify the behavior of your specific EFS provider with the following steps:

1. Open an output stream and simulate a failure (e.g., by throwing a manual exception) halfway through the write.
2. Execute your retry logic.
3. Inspect the resulting file to ensure no duplicated byte sequences exist.

Note: putInfo is used for modifying file attributes (metadata), not for writing file content. Ensure your retry logic targets openOutputStream or IFile.setContents for data operations.

To provide a more specific recommendation, could you clarify if you are targeting a local file system or a remote provider (e.g., SSH/FTP)?