Cardinality Explosion in Perfetto Trace Labels: Compaction Latency at Scale?
0 reputation · 02 Jan 2025, 15:37 UTC
Meta's continuous profiling pipeline ingests custom trace events with high-cardinality dimensions such as request IDs and user segments. The documented architecture uses a columnar storage backend derived from Gorilla principles with label deduplication and compaction, but public materials do not characterize how compaction latency behaves when label cardinality grows beyond the thresholds observed in staging.
The goal is to determine whether the ingestion pipeline's compaction strategy exhibits predictable scaling or enters a regime where latency spikes become non-linear and operationally disruptive. Constraints include undisclosed internal sampling rates, buffer sizes, and compaction thresholds that differ from upstream Perfetto defaults, as well as proprietary tenant-isolation extensions that may alter label handling.
Which compaction heuristics trigger the non-linear latency regime? Does the label deduplication mechanism rely on in-memory structures that become contention hotspots at extreme cardinality? Can the retention policy be tuned to bound cardinality without losing regression-detection fidelity?