HPE Nimble Peer Persistence: Designing Synchronous Metro HA Without Split-Brain

The problem: write ownership during a metro partition

When two storage arrays replicate synchronously across a campus or metro pair, the engineering risk is not moving bytes—it is deciding which array may accept a write when the replication link fails. If both sides continue independently, you get split-brain: two copies of the same volume that diverge and cannot be safely merged. HPE Nimble Peer Persistence addresses this by pairing synchronous replication with a quorum witness that arbitrates ownership. The useful takeaway is that the witness, host multipathing, and NVRAM drain rate are design inputs, not operational afterthoughts.

Requirements before you start

Peer Persistence imposes hard prerequisites. Both arrays must run the same NimbleOS major version; mixed-version partnerships are unsupported and will block creation. The typical deployment assumes a distance under 10 km and round-trip time (RTT) under 2 ms, because synchronous replication must acknowledge the remote write before returning to the host.

Host multipathing must use ALUA (Asymmetric Logical Unit Access), the SCSI standard that lets a host know which paths are optimized and which are standby. Legacy active/passive drivers—such as older EMC PowerPath configurations—can cause path thrashing during failover because they do not understand the asymmetric path model Nimble presents.

Volumes must be placed in volume collections (consistency groups) before Peer Persistence is enabled. Adding volumes to an existing partnership later requires a brief quiesce, so plan the membership up front.

Finally, synchronous replication consumes reserved NVRAM (non-volatile RAM used for write caching) on both arrays. If sustained write throughput exceeds roughly 80% of the NVRAM drain rate, the array throttles host I/O to protect the RPO=0 (recovery point objective of zero data loss) guarantee.

The smallest suitable design

The minimum viable topology is two Nimble arrays connected by a low-latency link, a quorum witness, and hosts with ALUA-capable multipathing. The arrays present a single virtual IP and virtual volume (VVOL) to hosts, so failover is transparent: no host-side reconfiguration is needed beyond standard ALUA support.

Replication efficiency comes from CASL (Cache-Accelerated Sequential Layout), Nimble's log-structured filesystem. CASL performs inline compression and deduplication before replication, so only unique blocks cross the wire. This reduces bandwidth compared with raw block mirroring. Snapshots and clones are replicated consistently as part of the same mechanism.

For production, the witness should be a third physical Nimble array, not a simulated VM. A cloud witness hosted by InfoSight is supported but requires outbound HTTPS connectivity and introduces approximately 50 ms of additional arbitration latency.

Trust and data boundaries

Each array is an independent fault domain with its own NVRAM-backed write cache. Neither array trusts the other to decide ownership unilaterally. The quorum witness holds no customer data—only lock state—and acts as the tiebreaker during a network partition. This boundary is what prevents split-brain: if a majority (array plus witness) cannot be reached, an array pauses I/O rather than risk divergence.

The cloud witness shifts part of the trust boundary outside the data center. You are then dependent on InfoSight reachability for arbitration, which is acceptable for many metro designs but should be documented as an external dependency.

Operational checks

InfoSight continuously monitors replication link latency, replication lag (target under 1 second), witness heartbeat, and array health. Alerts typically fire when RTT exceeds 5 ms, lag exceeds 30 seconds, or the witness becomes unreachable.

On the array CLI, peer show reports partner state, synchronization status, and the last failover timestamp. Run it with administrative privileges on either partner.

To measure replication lag under load, use perfstat replication --interval 5 while a representative workload runs—for example, a 4 KB 70/30 read/write mix. Confirm that lag stays below 1 second for synchronous mode.

Configuration example and expected checks

Run the following on array1 with administrative permissions. Replace the placeholders with your actual partner and witness identifiers.

peer create --partner <array2> --witness <array3|cloud>

After creation, run peer show and verify that the partnership reports a partnered state, an in-sync status, and a connected witness. Do not proceed to production cutover until all three conditions are true.

To validate failover behavior, disconnect the replication ports between the arrays. The virtual IP should fail over within 10–15 seconds under typical conditions. On the host, confirm that I/O resumes without SCSI errors: check dmesg on Linux or esxcli storage core path list on ESXi.

To validate witness arbitration, power off array1 while array2 remains online; array2 should continue serving I/O. Then, with array1 still down, power off array2 as well. Both arrays should pause I/O in a conservative pause. Verify with peer show --detail, looking for a paused state.

Removing a partnership is a state-changing operation. Consult the current NimbleOS CLI reference for the exact removal command and required pre-checks before attempting rollback.

Failure modes

Scenario	Witness reachable?	Behavior	Data risk
Replication link loss	Yes	Surviving array serves I/O transparently	None (RPO=0 maintained)
Replication link loss	No	Both arrays pause I/O (conservative pause)	None, but availability is lost
Replication lag exceeds ~60 s	Yes	Falls back to asynchronous mode with RPO warning	Potential data loss window opens
Both arrays fail	N/A	Service down	Data loss unless third-site async replication exists

The conservative pause is the most important behavior to understand: when the witness is unreachable, Nimble chooses consistency over availability. Hosts will see I/O errors until the partition heals or the witness returns.

Conditions that would change the design

• Distance or latency exceeds limits: If the sites are more than 10 km apart or RTT exceeds 2 ms, synchronous replication is no longer appropriate. Move to asynchronous replication and accept a non-zero RPO.
• Workload shift: A high-write, low-deduplication workload changes bandwidth sizing. CASL's inline deduplication helps, but if unique data volume is high, the replication link may need more capacity than the storage capacity alone suggests.
• Regulatory zero-RPO beyond 50 km: If compliance requires zero RPO at distances greater than typical metro limits, add a third-site synchronous leg or reconsider the topology entirely.
• Witness strategy change: Moving from an array witness to a cloud witness (or back) alters the trust boundary and adds or removes the ~50 ms arbitration latency and the InfoSight HTTPS dependency.

Limitations and how to verify

Thresholds, command syntax, and feature availability vary by NimbleOS release. Because Peer Persistence is a mature but evolving feature, verify the current CLI reference and InfoSight alert defaults against your installed version before production.

A practical verification sequence is: (1) confirm peer show reports partnered, in-sync, and witness-connected; (2) run perfstat replication --interval 5 under a production-like load and confirm lag stays under 1 second; (3) review the InfoSight Peer Persistence health tile and confirm it is green, with historical RTT and lag graphs inside expected bounds.