FoundationDB 7.x minimal architecture note: trust boundaries, operational checks, and redesign triggers

The problem: strict serializable KV with minimal operational surface

Choosing FoundationDB 7.x for a new service means accepting a layered, process‑separated architecture in exchange for serializable transactions. The useful takeaway is to keep the initial topology as small as fault tolerance allows, define a clear trust boundary at the cluster file, and instrument status and trace logs before load grows. Redesign is driven by latency percentiles, conflict rates, and cross‑region RPO, not by feature checklists.

Requirements that shape the design

FoundationDB provides ACID transactions with strict serializable isolation by default. Read‑your‑writes and snapshot reads are opt‑in via transaction options. Conflict detection is key‑granular, so hot keys produce retry storms. The store is a key‑value B‑tree on disk; secondary indexes and SQL are not native, they arrive via Record Layer or Document Layer with KV semantics preserved.

Operational requirements are: cluster membership visibility, fault domain awareness, and verifiable backups. Clients must authenticate to the cluster via a TLS‑verified cluster file. The cluster file is the trust root; it lists coordinator addresses.

Smallest suitable design for production

A minimal fault‑tolerant deployment separates roles but avoids over‑provisioning.

• Coordinators: 3 stateless processes on distinct hosts. They hold cluster membership and elect a leader. Quorum is majority. Do not co‑locate coordinators with storage on the same failure domain.
• Transaction Log: 3 log processes, each with a storage partner for durability. Logs are the write‑ahead path.
• Storage: 3 storage servers with on‑disk Redwood B‑tree. Replication factor 3 gives fault tolerance 1.
• Clients: connect via cluster file, e.g. /etc/foundationdb/fdb.cluster. Clients talk to transaction proxies and resolvers, not directly to storage.

This is the smallest topology that survives a single host loss without data unavailability. For a dev cluster, a single process can run all roles, but that removes fault tolerance.

Key‑space design matters early. Use directory layer prefixes or tenant prefixes to isolate workloads and avoid hot keys. Shard high‑write keys by appending a suffix.

Trust and data boundaries

The trust boundary is the cluster file and the TLS channel to coordinators. Compromise of a storage server does not allow forging transactions; it can only serve existing data or cause unavailability. Proxies enforce serializability and conflict checks.

Data boundary: storage servers own ranges of the key space. Re‑re‑replication after loss copies data from peers. Network partitions between data centers can split replicas. Configure datacenter‑aware replication with preferred AZ tags and, for multi‑region, three_datacenter_fallback.

Cluster file rotation is a security operation. Rotate the file via config management with a coordinated client reload. Split‑brain can occur if clients use old and new files during CA rotation.

Operational checks

Supervision: fdbmonitor watches processes and restarts them. Health is summarized by status JSON.

# run on any host with fdbcli and access to the cluster file
fdbcli --exec status

Required permissions: read access to the cluster file and network reachability to coordinators. Expected checks: healthy: true, fault_tolerance matches replication, data_moving: 0 in steady state, and queue metrics within normal ranges.

Latency analysis: enable JSON trace logs for post‑mortem analysis.

fdbcli --exec "configure new"
# set knobs via fdbcli or fdbcli --exec "set trace_format=json"
# trace_log_group default is typical

Risk: trace logging increases disk I/O. Limit duration and retention. Parse with jq to verify transaction latency percentiles and conflict rates under load.

Failure modes and responses

Coordinator quorum loss: majority of coordinators down halts cluster. Clients see unavailability. Recovery is restart of a majority.

Storage server disk loss: triggers re‑re‑replication from peers. Status shows data moving. Do not add capacity during heavy re‑re‑replication.

Network partition: cross‑region latency spikes cause client retries. Async DR via dr_agent has unbounded lag during outages. Monitor dr_seconds_behind and alert. RPO depends on link capacity.

Conditions that change the design

Write latency p99 consistently above 10 ms: add storage servers or review key distribution. Memory storage engine is experimental in 7.x and not recommended for durability‑critical workloads without chaos testing.

Cross‑region latency above 5 ms for synchronous writes: deploy multi‑region with satellite tlogs or switch to async DR for disaster recovery only.

Tenant isolation needed: separate clusters provide hard isolation. The tenant API in 7.3+ offers logical isolation within one cluster; evaluate query patterns first because KV semantics remain.

Backup/restore: incremental backup to blob store via backup agent. Restore creates a new cluster. Verify with fdbrestore dry‑run and checksum comparison. Point‑in‑time recovery requires retaining transaction logs via tlog retention knob.

Limitations and verification

FoundationDB does not natively support secondary indexes or SQL. Java/Go/Python bindings use different retry policies; unify retry behavior via a shared library to avoid inconsistent backoff.

Practical verification steps for a new deployment:

• Run status check and confirm health and fault tolerance.
• Simulate coordinator loss in a test environment by stopping a majority of coordinator processes; cluster should become unavailable and recover on restart.
• Execute backup/restore cycle to a test bucket and compare key counts via dry run.
• Enable JSON trace logs for a bounded window and parse latency and conflict metrics.

These checks confirm the trust boundary, operational visibility, and failure behavior before production traffic.