MirrorMaker 2 Offset Sync and Consumer Group Restoration Interoperability
0 reputation · 02 Feb 2024, 09:53 UTC
Restoring a topic replicated by MirrorMaker 2 into a target cluster requires resuming consumer groups at the correct logical position without duplicates or gaps. MirrorMaker 2 periodically writes consumer-group offset mappings to an internal offset-sync topic, translating source log positions into target-cluster offsets. These checkpoints are point-in-time approximations that can drift if replication lag or partition reassignment occurred before the backup boundary, and version-sensitive schema or configuration differences may affect translation fidelity.
Kafka provides no built-in row-count or checksum reconciliation for recovered partitions, so operators must compare end offsets, record timestamps, and payloads manually. Because MirrorMaker 2 does not replicate internal topics, cluster metadata, or access-control lists identically, the restoration boundary itself remains uncertain. The unresolved decision is whether to trust translated checkpointed offsets or reset consumer groups manually, particularly when transaction markers or compacted-topic semantics may not be preserved identically across clusters.
Should operators prefer the translated offsets in the offset-sync topic or manually reset consumer groups to avoid duplicates? How can teams validate that translated positions align with expected partition boundaries without a native reconciliation tool? What verification sequence best confirms recovered data integrity before re-enabling downstream consumers?