Question
Apache Kafka and Schema Registry: Rollback Boundaries After Producer Schema Changes
Ari Sage
0 reputation · 01 Feb 2024, 13:57 UTC
78.1K views
Loading…
0 reputation · 01 Feb 2024, 13:57 UTC
Apache Kafka treats message payloads as opaque bytes and does not validate, store, or version schemas, while Confluent Schema Registry manages subjects and compatibility modes such as BACKWARD, FORWARD, FULL, and TRANSITIVE variants. A producer serializer embeds a schema ID in each record, and a consumer deserializer resolves that ID through the registry at consume time.
Kafka transactions and idempotent producers guarantee atomic record writes, not schema changes, so there is no native rollback of a published schema version. After an incompatible schema change, consumers pinned to an older schema can fail to deserialize newer records, and Kafka provides no automatic consumer rollback to a previous schema.
Whether rollback is safe remains an operational decision involving consumer schema pinning, compatibility mode selection, and possible record reprocessing or rejection. Registration-time compatibility checks do not govern runtime deserialization, so a compatible schema can still break older consumers. In current Apache Kafka and Confluent Schema Registry deployments, practical verification involves inspecting subject compatibility settings and consumer deserializer logs, though behavior may vary with other registry implementations.
Given these boundaries, what determines whether a consumer can safely roll back to an older schema after a producer has published records under a newer schema? Does the serializer format—Avro, Protobuf, or JSON Schema—change how schema IDs are resolved during rollback? How should teams confirm deserialization behavior before downgrading a consumer schema?
32775 reputation · 01 Feb 2024, 19:50 UTC
After a producer publishes records with a newer schema, a consumer can safely revert to an older schema only when the newer schema is backward compatible with that older version (or when the subject is configured with a mode that guarantees two‑way compatibility, such as FULL or TRANSITIVE). The serializer format—Avro, Protobuf, or JSON Schema—does not change how the schema ID is stored or looked up; each record still carries the numeric ID that the consumer resolves via the registry. What matters is that the registry’s compatibility rules for that format allow the older schema to read the new data.
apachekafkaGET /config/{subject} and verify the mode is BACKWARD, FULL, or TRANSITIVE.
/compatibility/subject/{subject}/version/{latestVersion} with the older schema definition; a compatible:true response means the registry will allow the older schema to read newer records.
org.apache.kafka.common.serialization), and consume a small batch of recent records. Observe whether deserialization succeeds or throws SerializationException.
To tailor the recommendation precisely, please provide: the current compatibility mode of the subject and the specific schema format (Avro, Protobuf, JSON Schema) in use.
Use comments to ask for clarification. Post a solution as an answer.
32,775 reputation · 01 Feb 2024, 21:38 UTC
In Confluent Schema Registry the schema ID is unique across all subjects. When a consumer receives a record it looks up that exact ID, not the latest version of a subject. Therefore pinning a consumer to an older subject‑version does not automatically mean it will read newer records – the registry will still serve the writer’s schema that produced the record.
schemaId from the 5‑byte prefix.GET /schemas/ids/{schemaId} to retrieve the writer schema.SerializationException at runtime.Disabling or deleting a newer schema version in the registry does not affect consumers that already cached the ID; the data on disk remains unchanged. Safe rollback therefore hinges on runtime deserialization tests, not just registry compatibility settings.