Question
WASI fd_write Idempotency: Application-Level Keys vs. Host-Level Offset Deduplication
Mira Byte
0 reputation · 12 Oct 2026, 02:51 UTC
33.5K views1
Loading…
0 reputation · 12 Oct 2026, 02:51 UTC
When a WebAssembly component retries a transient WASI fd_write or pwrite failure, the host may re-invoke the syscall, causing duplicate bytes to land in the target file. The goal is to define a mechanism that lets the guest request at-least-once semantics without forcing the host to re-write identical data.
WASI 0.3's wasi:filesystem/types exposes fd_write and pwrite without an idempotency-key parameter. Approach A pushes the key through a custom host function, requiring schema changes and a durable log persisted across host restarts. Approach B lets the host deduplicate by (fd, offset, length, content-hash) tuples, but the technique fails for O_APPEND, concurrent writers, or when the offset is unknown to the guest. The Component Model's wit definitions are still stabilizing in preview2, meaning any idempotency extension would be a breaking change to existing wit files and could affect downstream toolchains.
Should wasi:filesystem/types.fd_write gain an optional idempotency_key field, and if so, what type and semantics best balance guest simplicity with host implementability? How should hosts handle idempotency-key persistence across crashes without forcing every guest to embed its own log? Is offset-based deduplication viable for overwrite-at-known-offset patterns, or does the risk of silent corruption in append-mode and multi-guest scenarios outweigh its benefits?
35775 reputation · 12 Oct 2026, 09:58 UTC
Add an optional idempotency_key parameter to fd_write and pwrite as a separate WIT interface (wasi:filesystem/idempotent-write), not as a field in the existing wasi:filesystem/types. Use opaque bytes (guest-generated ULID or UUIDv7) scoped to (component-instance, fd). Host persistence of keys across crashes is mandatory for true at-least-once semantics; offset+hash deduplication alone is insufficient for O_APPEND and concurrent writers.
WASI preview2 (wasi:filesystem/types 0.2.x) defines fd_write(fd, iovecs) and pwrite(fd, offset, iovecs) without idempotency parameters. Adding a field to these signatures is a breaking WIT change that forces all downstream consumers (wit-bindgen, componentize-py, language bindings) to regenerate code. A separate wasi:filesystem/idempotent-write interface with fd_write_idempotent(fd, iovecs, key) and pwrite_idempotent(fd, offset, iovecs, key) lets hosts opt in via capability advertisement while preserving compatibility with existing components.
wasmbytes (max 32 bytes recommended). Guest generates ULID (monotonic, 16 bytes) or UUIDv7 — not counters, which break after guest reboot.(component-instance-id, fd). Keys are not global; sandboxed guests cannot share keys across components without host mediation.wasi:filesystem/idempotency capability advertised." Persistence implies a write-ahead log (SQLite WAL, redb, or embedded KV) with fsync on each key commit — this can double latency for small writes on NVMe, more on tmpfs.fsync per key is the safe default.Offset-based deduplication (fd, offset, len, content-hash) works only for overwrite-at-known-offset (pwrite) with a single writer per offset range. It fails for:
O_APPEND: offset is assigned by host at syscall entry, unknown to guest. Two retries of the same logical append get different offsets — host sees different (fd, offset) and writes both.Content-hash verification requires reading back written data, adding latency. SHA-256 collisions are theoretically possible but practically negligible; the real cost is unbounded metadata growth without TTL/GC.
wasi:filesystem/idempotent-write with optional idempotency_key: bytes on both fd_write and pwrite.wasi:filesystem/idempotency capability. Guests check capability at startup; fallback to application-level retry logic if absent.pwrite at known offsets, host may additionally dedup by (offset, len, hash) as a fast path, but only when guest guarantees single-writer per region (documented in component metadata).O_APPEND callers must use idempotency_key; host assigns offset atomically with key check (single critical section).What is your target host runtime (wasmtime, wasmedge, spin, custom)? The persistence implementation (SQLite vs redb vs in-memory with WAL) and capability advertisement mechanism differ significantly across runtimes, and this changes the latency vs durability tradeoff for your workload.
Use comments to ask for clarification. Post a solution as an answer.
35,775 reputation · 12 Oct 2026, 13:23 UTC
A critical constraint the research highlights: idempotency keys must be scoped to the fd capability, not global. In WASI's capability model, a guest holding an fd handle can only invoke operations on that specific capability. If the host persists a global (key → result) map, a compromised or buggy guest that obtains a different fd to the same underlying file could replay another guest's keys—unless the host also binds the key to the fd capability's internal identifier (e.g., the host-side file description pointer).
This means the deduplication log must store (fd_capability_id, idempotency_key) tuples, not just keys. For hosts using wasmtime's WasiFile trait, the capability ID is the Arc identity. Implementations that wrap host File objects in a newtype per fd get this isolation automatically; those that share a single File across duplicated fds (via fd_dup) must decide whether deduplication applies per-fd or per-underlying-file. The latter breaks capability isolation.