Safe Schema Evolution with Apache Pulsar's Schema Registry

The problem: evolving event schemas without breaking consumers

In an event‑driven architecture, producers and consumers are often versioned independently. When a producer adds a field to an Avro schema, older consumers that do not know the field can fail at deserialization unless the change is carefully managed. Manually tracking schema versions across services is error‑prone and leads to runtime errors or data loss.

Thesis: Pulsar’s Schema Registry automates compatibility checks, letting you evolve schemas safely while keeping producers and consumers decoupled.

How the registry works

The Pulsar Schema Registry lives alongside the broker and stores a schema per topic. Each time a producer publishes, the broker:

1. Looks up the latest schema version for the topic.
2. Runs a compatibility check (BACKWARD, FORWARD, or FULL) against the new schema.
3. If the check passes, the schema is stored with an incremented version number and the message is allowed; otherwise the publish is rejected.

Consumers can fetch the schema they need by version ID, so they never need to bundle schemas with their code.

Worked example: adding a nullable email field

Assume an existing topic users with an Avro User record:

{"type":"record","name":"User","namespace":"com.example","fields":[
  {"name":"id","type":"string"},
  {"name":"name","type":"string"}
]}

We want to evolve it to:

{"type":"record","name":"User","namespace":"com.example","fields":[
  {"name":"id","type":"string"},
  {"name":"name","type":"string"},
  {"name":"email","type":["null","string"],"default":null}
]}

Steps to verify the evolution locally (run as a user with permission to start containers and execute pulsar-admin):

1. Start a local Pulsar cluster with schema registry enabled, e.g. using the official docker-compose.yml from the Pulsar repo.
2. Upload the initial schema:

# Run on host with docker access
pulsar-admin schema upload users \
  --schema-type AVRO \
  --schema-file ./user_v1.avsc \
  --properties "{}"

1. Upload the revised schema, specifying BACKWARD compatibility (the default for many teams):

pulsar-admin schema upload users \
  --schema-type AVRO \
  --schema-file ./user_v2.avsc \
  --compatibility BACKWARD

The broker will accept the upload because the new schema is backward compatible: existing consumers can ignore the email field, and the default null ensures no missing data.

1. Produce a message with the old schema:

# Java producer example (replace  with your auth token if needed)
producer.newMessage()\
    .value(new User("123", "Alice"))\
    .send();

1. Produce a message with the new schema (including email):

producer.newMessage()\
    .value(new User("124", "Bob", "[contact removed]"))\
    .send();

1. Consume with an older consumer (v1) – it will deserialize both messages, ignoring email on the second.
2. Consume with a newer consumer (v2) – it will read email when present and see null for the first message.

To check that the registry rejected an incompatible change, try uploading a schema that removes the id field with BACKWARD compatibility; the command will fail with a compatibility error.

Trade‑offs and limitations

Using the registry adds a small latency overhead (schema lookup and validation) on each publish. In high‑throughput scenarios you can mitigate this by:

• Running the registry in a clustered mode behind a load balancer.
• Enabling schema caching at the broker or client side.

Choosing the wrong compatibility mode can block legitimate evolutions (if too strict) or allow data‑loss changes (if too lax). Teams should agree on a mode per topic and document the rationale.

Actionable closing

Start by enabling the schema registry on your Pulsar cluster, pick a compatibility mode that matches your evolution policy, and use pulsar-admin schema to version your Avro (or JSON/Protobuf) schemas. Verify locally with the steps above, then monitor the broker logs for schema‑version updates and compatibility results. This gives you a concrete, automated guardrail against schema‑related failures while keeping producers and consumers independently deployable.