Diagnosing Cache Miss Spikes in McRouter-Managed Memcached Clusters

The Problem: Unexpected Cache Misses During Scaling

When scaling a distributed cache using McRouter, you may encounter a sudden spike in cache misses (a "cache stampede") despite having sufficient memory across your Memcached nodes. This typically occurs during cluster resizing or node failures, where the routing logic fails to map keys to the correct backend server, forcing the application to hit the primary database.

Diagnostic Matrix: Identifying the Root Cause

Symptom	Likely Cause	Primary Metric to Check
Global miss rate spike after adding nodes	Hashing mismatch / Re-sharding	mrouter.cache_miss_rate
High latency on specific keys	Remote DC replication lag	mrouter.remote_request_latency
CPU spikes on DB during node failure	Thundering Herd (Request Collapse failure)	mrouter.collapsed_requests
Inconsistent data across regions	Replication configuration error	mrouter.replication_queue_depth

Step-by-Step Diagnostic Workflow

1. Verify Consistent Hashing State: Check if the McRouter configuration is using a consistent hashing algorithm. Without this, adding a single node to a pool changes the mapping for nearly all keys, invalidating the entire cache.
2. Analyze Request Collapsing: If the database is overwhelmed during a miss, verify that collapsed_requests is enabled. This feature ensures that multiple concurrent requests for the same missing key result in only one request to the backend.
3. Inspect Routing Pool Health: Check for "zombie" nodes in the configuration. If McRouter attempts to route to a node that is unresponsive but not marked as dead, the request will hang until the timeout, increasing perceived latency.
4. Trace Cross-DC Traffic: For multi-region setups, determine if the proxy is incorrectly routing a local read to a remote data center due to a misconfigured pool priority.

Fixes and Configuration Adjustments

Scenario A: Cache Invalidation during Scaling

If adding nodes causes a mass miss event, ensure your mrouter.conf utilizes a pooled configuration with consistent hashing. This limits the number of keys that need to be remapped to only 1/N of the total keys (where N is the number of nodes).

# Example McRouter configuration snippet
# Ensure consistent hashing is applied to the pool
# Run this on the McRouter proxy instance

# Define the backend servers
# Permissions: Root or McRouter service user
# Risk: Incorrect IP addresses will lead to total cache failure

# pool { 
#   servers = ["10.0.0.1:11211", "10.0.0.2:11211", "10.0.0.3:11211"]
#   hashing = "consistent"
# }

Scenario B: Thundering Herd Protection

To prevent the backend database from crashing when a hot key expires, enable request collapsing. This forces McRouter to hold subsequent requests for the same key until the first request returns from the source.

Verification: Monitor the mrouter.collapsed_requests counter. A rising number indicates the proxy is successfully shielding the database from redundant queries.

Limitations and Verification

McRouter is a proxy layer; it does not manage the memory inside Memcached. It cannot fix memory fragmentation within a Memcached slab. To verify that McRouter is routing correctly, use a network tool like tcpdump on the proxy instance to ensure requests are distributed evenly across the backend IP addresses in the pool.

Rollback Procedure

If a configuration change to the routing pool causes increased errors, revert the mrouter.conf file to the previous version and restart the McRouter service to clear the internal routing table.