Scaling Gitter Chat History with MongoDB Sharding and TTL Indexes

Requirements at Scale

Gitter must persist 10 million chat messages daily, retain them for 30 days, and serve reads within 200 ms at 99.99 percent availability. The system handles bursts during evenings and weekends, with roughly 115 messages per second sustained peak.

Throughput and Retention

Ten million messages daily translates to roughly 115 messages per second sustained peak, with bursts higher during evenings and weekends. Each message is a small document, but volume drives storage growth and index pressure. A 30-day retention window means roughly 300 million documents before pruning, demanding automatic expiration to control storage cost.

Latency and Availability

Real-time read/write latency under 200 ms places the persistence layer close to the application tier. Nine-nines availability means any single node failure must not cascade into a partition; the design must tolerate node crashes, network splits, and rolling upgrades without losing write availability.

Minimal Suitable Architecture

The smallest design that meets the requirements is a sharded MongoDB cluster paired with application-level channel mapping.

Sharded Cluster Layout

Three shards, each a replica set with two data-bearing members and one arbiter, provide write scalability and failover coverage. A dedicated collection per channel ensures that chat history for #general, #announce, and direct-message threads are isolated. The shard key combines channel ID with a timestamp bucket, spreading writes across shards while keeping messages of the same channel on the same shard.

TTL Index for Automatic Retention

A single-field TTL index on the message timestamp directs MongoDB's background cleanup thread to remove documents older than 30 days (2592000 seconds). Because the index covers the timestamp field only, no separate cleanup job is required for the basic window, though a periodic job can enforce per-channel caps if stricter limits apply.

Trust and Data Boundaries

All inbound messages flow through application servers that authenticate the sender and write to the primary of the nearest shard. Replicas within the same shard serve read traffic; reads from secondaries are consistent within one second under normal sync lag. Data at rest is encrypted with AES-256 using MongoDB's internal key management, and all client-server communication uses TLS 1.3. Channel collections are logically separated, and authentication scopes restrict write access to the channel's owning team.

Operational Checks

Regular diagnostics keep the layer healthy and within the latency and availability SLA.

Replica Set Sync Lag

Run rs.status() on any mongosh instance and verify that the "syncedAt" timestamp difference between primary and secondary stays under 5 seconds. Alerts fire when lag exceeds this threshold.

Shard Distribution Balance

Use sh.status() to inspect chunk ranges per shard. Uneven chunk sizes indicate a need for a sh.splitAt or rebalancing pass. Target a spread where no single shard handles more than 20 percent of total write volume.

TTL Index Coverage

Confirm the TTL index exists on the timestamp field and that MongoDB's TTL monitor is running. A quick check: db.getCollectionInfos({name: "messages"})[0].indexes should list the {ts: 1} index with expireAfterSeconds set to 2592000.

Backup Integrity

Take weekly snapshots of the config database and each shard's data files. Restore a test collection from the latest snapshot and verify document counts match the expected 30-day window.

Failure Modes

• Shard node crash: Writes to the affected shard stall until a new primary is elected (typically under 12 seconds). Read traffic continues from other secondaries in the same shard.
• TTL misconfiguration: If the expireAfterSeconds value is set too low, messages disappear before the 30-day window; if set too high, storage grows unchecked. A mismatch between the index field and the actual timestamp format can silently disable expiration.
• Network partition: Quorum loss prevents writes; the cluster operates in read-only mode until connectivity restores and a primary is re-elected.
• Background cleanup job failure: Though TTL handles expiration automatically, a disabled TTL monitor or corrupted index can leave orphaned documents, requiring manual cleanup.

Conditions That Would Change the Design

If daily message volume sustainedly exceeds 20 M, the three-shard layout may become a write bottleneck and would require adding shards or revisiting the shard key. If sub-100 ms latency is required for global users, adding a read-cache layer (e.g., Redis) in front of MongoDB can absorb burst reads without affecting the persistence layer. If regulatory requirements tighten retention to 90 days or mandate immutable audit logs, replacing the TTL index with a write-once append-only store and separate expiration policy becomes necessary.

Concrete Example: Creating the Sharded Collection and TTL Index

Run the following commands in mongosh on any node with cluster-wide privileges. You need clusterAdmin or a role that includes shardCollection and createIndex. Replace gitter and messages with your database and collection names if different.

sh.enableSharding("gitter") sh.shardCollection("gitter.messages", { "channel": 1, "ts": 1 }) db.messages.createIndex({ "ts": 1 }, { expireAfterSeconds: 2592000 }) db.getCollectionInfos({ name: "messages" })[0].indexes

Expected check: The output of the last command should include an index with key { ts: 1 } and expireAfterSeconds: 2592000. MongoDB's TTL monitor typically runs every 60 seconds; observe that documents older than 30 days are removed automatically. Risk: Setting expireAfterSeconds too low deletes messages before the 30-day window; setting it too high allows storage to grow unchecked. If the timestamp field uses a non-ISODate type, the TTL index will not trigger expiration.