Designing for High-Ingest Time-Series Workloads with InfluxDB TSM

The Challenge of High-Velocity Time-Series Ingestion

Writing thousands of data points per second to a standard relational database often results in disk I/O bottlenecks because the system must update B-Tree indexes and perform random writes to maintain data integrity. In time-series engineering, the primary goal is to convert these random writes into sequential disk I/O to maximize throughput and minimize storage overhead.

The InfluxDB Time-Structured Merge (TSM) engine solves this by treating time-series data as an append-only stream, deferring the cost of organization and compression until data is flushed from memory to disk.

The Minimal Architecture for High-Write Throughput

To implement a high-ingest pipeline, the system relies on a tiered storage path. Data does not go directly to a permanent file; it moves through three distinct stages to balance durability and performance.

1. The Write-Ahead Log (WAL)

Every incoming point is first written to the WAL. This is a sequential log file that ensures durability. If the system crashes before data is compressed, the WAL is replayed upon restart to recover lost points. This allows the engine to acknowledge writes quickly without waiting for complex compression cycles.

2. The Cache (Memtable)

While the WAL handles durability, the data is simultaneously held in a memory cache. Here, the engine organizes points by series (the combination of measurement and tags). This stage allows the system to group data points together, which is critical for the compression algorithms used in the next step.

3. TSM Files (Immutable Storage)

Once the cache reaches a specific threshold, the engine flushes the data into a TSM file. These files are immutable. To save space, InfluxDB uses specialized compression: Gorilla compression for floating-point values and Delta-encoding for timestamps. Because the files are immutable, the system avoids the overhead of locking rows during updates.

Trust and Data Boundaries: The Cardinality Trap

The most critical boundary in a TSM-based design is the Series Cardinality. A series is defined as the unique combination of a measurement name and its tag set. For example, cpu_usage,host=server01,region=us-west is one series.

The index of these series is stored in memory (TSI - Time Series Index). If you use high-cardinality data—such as User IDs or Request IDs—as tags, the index will grow linearly until it exhausts the available RAM, leading to OOM (Out of Memory) crashes or severe performance degradation.

Cardinality Decision Matrix

Data Type	Storage Choice	Reasoning
Host ID, Region, App Name	Tag	Low cardinality; used for filtering/grouping.
User ID, Transaction ID	Field	High cardinality; not indexed, avoids memory bloat.
Sensor Reading, Temp	Field	The actual value being measured.

Operational Checks and Verification

To verify that the TSM engine is operating as expected, you should monitor the filesystem and memory during a load test. Run these checks with administrative permissions on the host machine.

Filesystem Verification

Navigate to the InfluxDB data directory (typically /var/lib/influxdb/data). During active ingestion, you should see the following:

• .wal files: These should be actively growing and rotating.
• .tsm files: These appear after the cache flushes. If you see no TSM files despite high ingest, your cache may be too large or the system is failing to flush.

Retention and Shard Purging

InfluxDB organizes data into Shards based on time intervals. Instead of deleting individual rows (which is expensive in a TSM engine), the system simply drops an entire shard file from the disk.

To verify this, set a short retention policy (e.g., 24 hours) and monitor the data directory over 48 hours. You should see entire directories (shards) disappear from the filesystem without a spike in CPU usage associated with DELETE queries.

Failure Modes and Design Shifts

The TSM architecture is optimized for append-heavy workloads. Certain conditions will break this model and require a change in design:

• Frequent Updates: If your use case requires updating existing data points (Upserts), TSM will struggle. Each update creates a new version of the point, increasing storage and slowing down reads. If updates are frequent, consider a traditional relational database or a Key-Value store.
• Extreme Cardinality: If you must index millions of unique tags, you must move the TSI (Time Series Index) to disk rather than keeping it in memory, or implement a proxy layer to aggregate data before it hits InfluxDB.
• Misconfigured Shard Duration: If your shard duration is too short (e.g., 1 hour for data kept for 1 year), you will create too many small files, overloading the filesystem's inode limit and slowing down queries.