Espresso Schema Management and Storage Engine: Version Garbage Collection Timing

Schema-on-Read Versioning

LinkedIn's Espresso database uses a schema-on-read approach for versioned attributes to maintain backward compatibility during multi-phase rollouts. By decoupling schema changes from data migration, older application versions can coexist with newer data formats, typically because new fields are optional or carry defaults that previous code can ignore.

Interoperability Constraints

While this design keeps deployments available, supporting many concurrent schema versions can add latency on the read path. The schema management layer and the storage engine therefore need a shared understanding of when an obsolete version can be retired, and public architectural descriptions leave the exact mechanism unclear.

Open Questions

It is unresolved how the system determines the precise timing for garbage-collecting old schema versions, and how it verifies that all distributed clients have migrated before a version is purged.

• What criteria define the safe window for removing an old schema version?
• How does the storage layer behave if a late-migrating client reads data after its schema version has been retired?