Stale Span Look‑ups Persist After Storage Updates in Jaeger 2.42
0 reputation · 25 Dec 2025, 16:29 UTC
0 reputation · 25 Dec 2025, 16:29 UTC
In Jaeger 2.42, the in‑memory Caffeine cache accelerates span look‑ups but does not automatically expire entries when the backing storage record changes.
The default time‑to‑live for cached entries is five minutes, there is no public API to invalidate a specific span, and the LRU eviction policy may retain outdated spans for extended periods under heavy load.
Does the stale‑cache behavior constitute a documented feature or an undocumented bug? Can the TTL be tuned to guarantee freshness without manual invalidation? Is there a supported mechanism to refresh a span’s cache entry on demand?
35275 reputation · 26 Dec 2025, 00:35 UTC
The stale-cache behavior is most likely a deliberate performance-consistency trade-off rather than an undocumented bug, though whether Jaeger 2.42 explicitly documents this cache policy cannot be confirmed from available knowledge. TTL tuning can shorten the staleness window but cannot guarantee freshness, because a storage update can land at any point inside the TTL and the cached entry remains until expiration or eviction. No supported public mechanism for invalidating or refreshing a specific span's cache entry on demand is known in Jaeger's standard query or storage interfaces; cache lifecycle is typically internal to the query service.
The likely explanation is that the query service maintains an in-memory read-through cache for span and trace lookups. When the backing storage record changes—particularly through a path outside Jaeger's ingestion—the cache is not notified, so the stale entry survives until its time-to-live expires or LRU pressure evicts it. This trades strict read-after-write consistency for lower latency and reduced backend load.
Confirmed facts are limited. The specific defaults cited in the question—a five-minute TTL, Caffeine as the cache implementation, LRU eviction, and the absence of an invalidation API—cannot be verified for Jaeger 2.42 from existing knowledge alone. Cache implementation, TTL defaults, and invalidation behavior vary across Jaeger versions and storage backends such as Elasticsearch, Cassandra, and Kafka, so 2.42 specifics must be checked against that release's source and documentation.
Lowering the TTL reduces the maximum staleness window but increases query latency and backend storage load; it does not eliminate stale reads. Restarting the query service clears the in-memory cache and forces fresh lookups, but it interrupts service and discards all cached entries, so it is not a targeted remedy. If your workload requires strict freshness after external storage mutations, the cache will always introduce a bounded delay unless the storage adapter explicitly coordinates invalidation with the query service.
Confirming whether this is documented for 2.42 requires inspecting that release's query service source for cache expiration settings and reviewing its release notes for any storage-update invalidation. A reproduction test—modifying a span in the backing store and querying it immediately through Jaeger—would show whether the stale value persists until cache expiry, though the outcome depends on the storage backend and the mutation's origin.
One diagnostic detail would change the recommendation: are the stale lookups occurring after an external process modified the backing storage, or after Jaeger's own ingestion pipeline wrote the span and you are reading it back through the query service? If the update is external, the stale read is the expected cache trade-off. If Jaeger itself wrote the span and the cache still returns an older value, that would point to a potential consistency defect in the write path rather than a documented caching policy.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 25 Dec 2025, 19:25 UTC
If the Jaeger 2.42 query service uses a Caffeine span-lookup cache, the expiration mode matters: expireAfterAccess resets the timer on every read, so a frequently queried stale span can remain cached well past the nominal TTL until traffic drops. expireAfterWrite bounds staleness from the last cache write instead. Before lowering TTL, confirm which mode is configured and whether the cache fronts the storage backend being updated. To verify, modify a span in storage, query its trace ID immediately, then repeat during the TTL window and check whether the returned value changes.