Pulsar's Schema Registry: Evolving Event Schemas Without Breaking Consumers

Every event-driven system eventually hits the same wall: a producer team adds a field, a consumer team hasn't deployed in three weeks, and suddenly messages fail deserialization in production. Apache Pulsar's built-in schema registry exists to make that failure happen at publish time — loudly, in the producer — instead of silently in a consumer at 2 a.m.

The short version: Pulsar stores a versioned schema per topic and enforces a compatibility policy (BACKWARD, FORWARD, FULL, or NONE) at the broker. If you configure it deliberately, schema evolution becomes a controlled operation instead of an incident.

What the registry actually does

Unlike Kafka, where the schema registry is a separate service (Confluent's, typically), Pulsar's registry lives in the brokers themselves, backed by BookKeeper storage. When a producer connects with a typed schema — Avro, Protobuf, JSON Schema, and others — the broker registers the schema, assigns it a version, and attaches that version to every message. Consumers fetch the schema by version and deserialize automatically.

The important consequence: the broker is the enforcement point. A producer whose schema violates the topic's compatibility policy gets its messages rejected before they're ever stored. That's the outage-prevention property you're paying for.

Compatibility policies in practice

Pulsar supports several compatibility levels, settable per topic or per namespace:

• BACKWARD — new consumers can read data written by the old schema. Adding an optional field is fine; deleting a field is not.
• FORWARD — old consumers can read data written by the new schema. Deleting a field is fine; adding one is not.
• FULL — both directions. Only changes that are safe both ways (typically adding or removing optional fields with defaults) pass.
• NONE — no checks. Fast, and a common source of regret.

For most topics, BACKWARD is the pragmatic default: you upgrade consumers first, then producers, and additive changes flow through safely. FULL is what you want when you can't coordinate deploy order at all.

A worked example: adding a field safely

Say your topic persistent://orders/fulfillment/payments uses this Avro schema (version 1):

{
  "type": "record",
  "name": "Payment",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "amountCents", "type": "long"}
  ]
}

Business now wants a currency code. The BACKWARD-compatible version 2 adds it as optional with a default:

{"name": "currency", "type": ["null", "string"], "default": null}

Deploy the new producer. The broker checks the new schema against the topic's BACKWARD policy, accepts it as version 2, and existing consumers keep working — they simply never see the new field. You can confirm the registry state from any machine with the Pulsar admin CLI configured (requires a service URL and a token with produce/consume permissions on the namespace):

pulsar-admin schemas get persistent://orders/fulfillment/payments --version 2

Expect the returned definition to include the currency field and the compatibility type you configured. Now try the inverse experiment on a scratch topic: publish a schema that removes amountCents. With BACKWARD or FULL in place, the broker rejects the registration — which is exactly the behavior that saves you in production.

Multi-tenant isolation

Schemas are scoped to topics, and topics live under tenant/namespace. That hierarchy gives you isolation for free: tenant orders and tenant billing can each have a topic named events with completely different schemas, and neither can accidentally register over the other. The practical guidance is to set compatibility policies at the namespace level (pulsar-admin namespaces set-schema-compatibility-strategy on newer versions, or per-topic via pulsar-admin topicPolicies) so every new topic inherits a sane default instead of NONE.

Trade-offs and failure modes

Three things bite teams in practice:

1. Strict policies cause producer outages by design. If a developer ships an incompatible change, every publish fails until the schema is fixed or the policy is relaxed. That's the point — but make sure your producer alerts on schema rejection, not just on generic send failures, or you'll debug the wrong layer.
2. Registry bloat is real. Frequent additive changes accumulate versions, and every consumer does a schema lookup. Prune obsolete versions after migrations and avoid auto-generating schemas per deploy.
3. JSON Schema validates less than you think. Avro and Protobuf give you real evolution semantics; schemaless JSON topics give you none. If compatibility matters, pick a typed format.

Where to start

Pick one high-traffic topic, set its compatibility to BACKWARD, and run the two experiments above: add an optional field (should succeed) and remove a required one (should be rejected). Then list versions with pulsar-admin schemas get and confirm the history matches what you expect. Once that behavior is verified in your cluster version, roll the namespace-level policy out broadly — the registry only protects the topics that opt in.