Theta sketch compact() before storage vs. keeping the updatable form for low-cardinality tenant streams
0 reputation · 05 Jan 2022, 11:22 UTC
We run Apache DataSketches (Java 6.x) Theta sketches for per-tenant hourly unique counts. Most tenants see well under 500 distinct values per window, but a default sketch is constructed with 4096 nominal entries, so each instance holds a mostly empty internal buffer. Across thousands of tenants kept in heap, the unused capacity is becoming a measurable memory cost.
The documented options appear to be: call compact() before serializing or caching each window's result, or leave sketches in the default updatable form and accept the larger footprint. Compaction returns a new sketch instance and copies internal arrays, so doing it after every short batch window adds CPU work for updates that are already cheap. There also seems to be no built-in way to pre-size a sketch to a small expected cardinality without giving up the library's accuracy guarantees if a tenant suddenly spikes.
Given this workload, which trade-off is the right default?
- Is calling
compact()on every hourly window worth the copy cost when update volume is low, or should compaction only happen at serialization boundaries? - Is undersizing
nominalEntries(e.g., 512) a defensible alternative, and how badly do error bounds degrade if actual cardinality exceeds it? - Is there a documented pattern for wrapping the sketch lifecycle so compaction is enforced rather than left to each call site?