Question
Tag or Field: InfluxDB Schema Trade-off for High-Churn Identifiers
Kavi Indigo
0 reputation · 09 Jun 2023, 07:33 UTC
90.8K views
Loading…
0 reputation · 09 Jun 2023, 07:33 UTC
Storing a high-churn identifier such as a session ID or UUID in InfluxDB presents a schema trade-off when a dataset that passes local testing will face much higher series cardinality in production. Tags are indexed and become part of the series key, so filtering or grouping by a tag uses the index; fields are not indexed, so queries against them scan the measurement. The unresolved decision is whether to accept the index memory and compaction cost of a tag to preserve query efficiency, or keep the identifier as a field to bound cardinality and redesign the access pattern.
InfluxDB 2.x and 3.x use different storage engines and cardinality models, so the same schema is not portable between versions, and exact limits or defaults must be verified per deployment. Additional uncertainty comes from field type conflicts: changing a value between integer and string causes point rejection unless the field or measurement is versioned, which constrains how the identifier can evolve.
Given these constraints, what is the most reliable way to decide between a tag and a field for a high-churn identifier when the production series count is unknown? How should the decision account for version-specific engine behavior between InfluxDB 2.x and 3.x? What verification steps are appropriate to confirm that the chosen schema will not degrade under production load before it is promoted?
35275 reputation · 09 Jun 2023, 13:59 UTC
Use a field for high-churn identifiers (session IDs, UUIDs) in InfluxDB 2.x unless you can prove cardinality stays below ~100,000 unique values per retention period. In InfluxDB 3.x, use a tag only if the identifier is a primary filter or GROUP BY key in production queries, and pair it with a partitioning strategy (e.g., time + hash partition). If the identifier is only used for exact-match point lookups, keep it as a field in both versions and handle lookups externally.
Tags form the series key. Each unique tag value creates a new series, consuming RAM in the in-memory index (TSI) and increasing compaction write amplification. Unbounded session IDs will cause OOM or write stalls once series cardinality exceeds roughly 1–5 million per node (observed practical limit, varies with hardware). Fields are not indexed; filtering on a field scans all TSM files for the measurement.
Both tags and fields are Arrow columns. There is no separate series index. Cardinality pressure appears as dictionary encoding size and partition pruning efficiency. A high-cardinality tag column used as a partition key degrades pruning; as a non-partition tag it mainly increases catalog memory. Fields are not predicate-pushdown candidates unless annotated as tags.
In both versions, a field name cannot change type (integer ↔ string) without write rejection. Store UUIDs as strings from day one. If numeric session IDs might arrive as strings from any client, either enforce string coercion at ingest or version the field name (e.g., session_id_v1).
influxdbSHOW SERIES CARDINALITY, go_memstats_sys_bytes, influxdb_tsm_compaction_duration_seconds.information_schema.columns.estimated_cardinality for the identifier column, iox_catalog_memory_bytes, query latency for WHERE identifier = ? patterns.influxd inspect buildtsi; 3.x: trigger compactor) and verify no write stalls exceed acceptable duration.What is the expected unique identifier count per retention period in production? If you can bound it (e.g., "<50k sessions/day, 30-day retention = 1.5M max"), the 2.x tag option becomes viable with monitoring. If unbounded or unknown, default to field in 2.x and tag-with-partitioning in 3.x.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.