Managing Schema Evolution Safely with Apache Pulsar's Schema Registry

The problem: changing a streaming schema without breaking consumers

When a streaming application evolves, producers may need to add or remove fields. If the change is not compatible with existing consumers, messages can fail to deserialize, causing silent data loss or pipeline crashes. The challenge is to enforce compatibility rules automatically so that unsafe changes are rejected before they reach the broker.

Thesis

Apache Pulsar’s built‑in Schema Registry stores schemas as topic metadata and validates every produce request against a configurable compatibility policy (BACKWARD, FORWARD, FULL, NONE). By enabling the registry and choosing an appropriate policy, teams can catch breaking changes at publish time, keeping consumers safe while still allowing evolution.

How the Schema Registry works in Pulsar

The registry lives inside the Pulsar broker cluster. When a producer sends a message, the client library includes the schema (or a schema hash) in the message metadata. The broker looks up the currently registered schema for the topic, runs the compatibility check, and either accepts the message or returns an error. Each successful registration yields a schema version ID that is embedded in the message, allowing consumers to fetch the exact schema used for decoding.

Key components

• Schema metadata – stored alongside topic metadata in the broker’s ledger.
• Compatibility mode – set per‑namespace or per‑topic; determines what schema changes are allowed.
• Schema version ID – monotonic integer returned by the registry; producers embed it, consumers use it to retrieve the schema.

Worked example: evolving a user event schema under BACKWARD compatibility

We start with a simple Avro schema for a user signup event, then add an optional field (backward‑compatible) and finally try to delete a required field (incompatible) to see the broker reject the change.

1. Prepare the environment

Assume a standalone Pulsar cluster is running with the schema registry enabled (default). You need admin rights to run pulsar-admin commands.

# Set variables for your tenant/namespace/topic
TENANT=public
NAMESPACE=default
TOPIC=user-signup
FULL_NAME="${TENANT}/${NAMESPACE}/${TOPIC}"

2. Register the initial schema

Create a file user_v1.avsc:

{
  "type": "record",
  "name": "UserSignup",
  "namespace": "com.example",
  "fields": [
    {"name": "userId", "type": "string"},
    {"name": "email", "type": "string"}
  ]
}

Register it:

pulsar-admin schema upload \
  --schema-definition-file user_v1.avsc \
  --schema-type AVRO \
  $FULL_NAME

The broker returns a schema version ID, e.g., 1.

3. Add an optional field (backward‑compatible)

Update the Avro schema to user_v2.avsc by adding a middle‑name field with a default:

{
  "type": "record",
  "name": "UserSignup",
  "namespace": "com.example",
  "fields": [
    {"name": "userId", "type": "string"},
    {"name": "email", "type": "string"},
    {"name": "middleName", "type": ["null", "string"], "default": null}
  ]
}

Upload the new version:

pulsar-admin schema upload \
  --schema-definition-file user_v2.avsc \
  --schema-type AVRO \
  $FULL_NAME

Because the namespace is set to BACKWARD compatibility (default), the broker accepts the change: the new schema can still read messages written with the old schema.

4. Attempt an incompatible change (delete a required field)

Create user_v3.avsc that removes the required email field:

{
  "type": "record",
  "name": "UserSignup",
  "namespace": "com.example",
  "fields": [
    {"name": "userId", "type": "string"}
  ]
}

Try to upload:

pulsar-admin schema upload \
  --schema-definition-file user_v3.avsc \
  --schema-type AVRO \
  $FULL_NAME

The broker rejects the request with an error similar to:

Failed to register schema: Incompatible schema: BACKWARD compatibility violated.

This prevents producers from publishing messages that existing consumers cannot decode.

Trade‑offs and limitations

While the Schema Registry adds safety, it introduces a small metadata lookup overhead for each produce request. In ultra‑low‑latency workloads this latency can be measurable. You can tune the client‑side schema cache (schemaCacheSize and schemaCacheExpiryTime) to reduce broker hits. Additionally, strict compatibility modes will reject messages from clients that do not send schema metadata; ensure all producer and consumer libraries are configured to include the schema (most official Pulsar clients do this by default).

Actionable closing

1. Enable the Schema Registry (it is on by default in recent Pulsar releases).
2. Set a compatibility mode that matches your evolution strategy (pulsar-admin namespaces set-schema-compatibility-strategy $TENANT/$NAMESPACE BACKWARD).
3. Register the initial schema with pulsar-admin schema upload.
4. Version‑control your schema files and treat schema updates as code changes—review them in pull requests.
5. Monitor schema version IDs in broker logs or via pulsar-admin schema get $FULL_NAME to detect unexpected rejections.
6. If latency becomes a concern, increase the client schema cache size and measure the impact.

By treating schema evolution as a governed operation backed by Pulsar’s Schema Registry, you gain confidence that your streaming pipelines stay compatible as they grow.