Jaeger Elasticsearch Storage Backend Cache Staleness in v1.52

Cache Staleness in Jaeger v1.52

The storage-backend-cache configuration in Jaeger v1.52 does not provide a mechanism to detect or mitigate stale results when Elasticsearch indices are refreshed. The cache operates on a fixed Time-To-Live (TTL) basis and is entirely decoupled from the Elasticsearch refresh interval. It cannot invalidate entries in response to new data ingestion or index updates.

Likely Cause of Inconsistency

Staleness occurs because Jaeger caches the result of a query at the service level. In high-throughput environments, Elasticsearch indices may have a refresh interval (e.g., 1s or 30s) that creates a lag between when a trace is written and when it becomes searchable. If a query is cached during this window, the cache will continue to serve the incomplete result set until the TTL expires, regardless of whether the underlying index has since been refreshed.

Recommended Mitigation Steps

Because there are no real-time invalidation hooks, operators must balance data consistency against Elasticsearch cluster load:

1. Reduce Cache TTL: Lower the storage-backend-cache value to a duration that closely aligns with your acceptable window of staleness. Note that this increases the query load on Elasticsearch.
2. Align Elasticsearch Refresh Intervals: Ensure the Elasticsearch index.refresh_interval is tuned for your ingestion rate to minimize the initial window of invisibility before the cache is populated.
3. Apply Changes via Restart: Since cache configuration is not dynamic, you must restart the Jaeger query service for any TTL changes to take effect.

Verification and Monitoring

To verify the impact of staleness and the effectiveness of your TTL settings, use the following approach:

• Comparative Querying: Execute a query via the Jaeger UI, then immediately execute the same query via the Elasticsearch API. A discrepancy in the result set confirms cache staleness.
• Metric Tracking: Monitor the Jaeger query service logs and metrics for cache hit/miss ratios to ensure that reducing the TTL does not cause an unsustainable spike in Elasticsearch latency.

Diagnostic Detail Required: To provide a more specific TTL recommendation, please provide the current index.refresh_interval configured on your Elasticsearch indices.