Schema mismatch: expected field X of type Y, found Z after LinkedIn online DDL rollback

Mechanisms to auto-signal consumers after a LinkedIn online DDL rollback

When a rollback is triggered, the Schema Registry typically remains on the new Avro version, so consumers that have not explicitly pinned an earlier version encounter fields that do not exist in their current schema, producing errors such as expected field X of type Y, found Z.

The following mechanisms have been identified as possible approaches:

1. Schema Registry rollback tag – a metadata flag attached to the prior schema version that marks it as active after a rollback event. New deserialization requests resolve to this version while producers may continue publishing the new version as a tombstone.
2. Tombstone‑driven broker signal – publishing a tombstone or marker event on the associated messaging topic when a rollback occurs. Consumers subscribe to this event and automatically switch their active schema version.
3. Consumer‑side version pinning with automated fallback – each consumer group pins a specific schema version in configuration. On detecting a rollback event (via registry API polling or broker event), the consumer downgrades to the pinned version.

Likely explanation (research‑brief context)

The research brief notes that LinkedIn’s online DDL tool applies changes incrementally but does not automatically revert the associated schema version in the Schema Registry. After a rollback, the registry still registers the new schema version, and consumers using the prior version encounter the mismatch error. The brief also flags that introducing a rollback tag may change schema‑resolution semantics and that tombstone events increase registry and broker traffic.

Practical verification (scoped checks)

• Check the Schema Registry API documentation for support of version tagging or tombstone publishing in the current release.
• Inspect producer logs during a staged rollback to verify whether any tombstone events are emitted and whether a rollback flag or identifier is set.
• Validate that consumer services are configured to handle schema‑version change events or to fallback to a pinned version.

Missing diagnostic detail: whether the rollback event from the online DDL tool includes a producer‑side tombstone or a rollback identifier that the registry can associate with the prior schema version. Confirming this will determine if a registry‑level rollback tag is feasible without breaking existing producers.