Choosing a Translation Memory Strategy for Large-Scale Weblate Deployments

The Scaling Problem: Built-in Index vs. External TMX

When managing localization for a few thousand strings, Weblate's built-in translation memory (TM) is seamless. However, as a project scales toward 500,000 or 1 million units, the Whoosh full‑text index—the engine powering built‑in TM—can introduce query latency and database overhead. For organizations managing multiple Weblate instances or using external Computer Assisted Translation (CAT) tools, relying solely on the internal database creates a data silo.

The decision is whether to rely on the Built‑in Whoosh Index for real‑time, dynamic updates, or integrate External TMX (Translation Memory eXchange) files to stabilize performance and share memory across platforms.

Comparison of Memory Strategies

Feature	Built‑in Whoosh Index	External TMX Files
Update Speed	Real‑time (as strings are saved)	Asynchronous (via Celery import)
Performance	Degrades as corpus grows >1M	Constant read speed; no index rebuilds
Portability	Locked to Weblate database	Industry standard (.tmx); CAT tool compatible
Access Control	Respects project‑level ACLs	Global (unless filtered by component)
Write Ability	Read/Write	Read‑only (in Weblate 4.x/5.x)

Trade‑offs and Engineering Constraints

The Built‑in Index Trade‑off: The primary advantage is the "instant feedback loop." When a translator saves a string, it is immediately available as a suggestion for others. The cost is maintenance. Large indices require SSD storage to prevent I/O bottlenecks and periodic index rebuilding to maintain search efficiency.

The External TMX Trade‑off: TMX files provide a stable, read‑only baseline. They are ideal for "Golden Sets" of terminology that should not change frequently. However, because TMX imports are handled by Celery workers, there is a lag between the source file update and the suggestion appearing in the UI. Furthermore, importing large TMX files can cause memory spikes in Celery workers if not capped.

Implementation: Configuring External TMX and Index Maintenance

For high‑volume environments, the recommended architecture is a hybrid: use the built‑in TM for project‑specific agility and a mounted TMX directory for corporate‑wide memory.

1. Mounting External TMX Files

To make external TMX files available across projects, define the path in your settings.py or environment variables. This allows Weblate to treat the directory as supplementary memory.

# In your environment configuration or settings.py
WEBLATE_TMX_PATH = '/opt/weblate/tmx_storage/'

Risk: Ensure the user running the Weblate process has read permissions for this directory. If the directory is a network mount (NFS/SMB), latency in the mount can slow down suggestion lookups.

2. Optimizing the Built‑in Index

If you continue using the built‑in TM for large datasets, you must manage the Whoosh index to avoid corruption and performance decay. Run the following command via the Weblate management shell (as the weblate user) during a maintenance window:

# Rebuild the search index to optimize query performance
weblate rebuild_index

Verification: To check if the index is functioning and measure latency for a specific project, you can use the management command to search the memory directly:

# Run from the application shell
weblate search_memory "Your search string" --project=your_project_slug

Handling Large‑Scale Imports

When importing TMX files with over 100k units, Celery workers may exceed available RAM. To prevent OOM (Out of Memory) kills, limit the memory per child process in your Celery configuration:

# Celery configuration
CELERY_WORKER_MAX_MEMORY_PER_CHILD = 200000 # KB (e.g., 200MB)

Summary of Limitations

• Duplicates: TMX imports do not automatically deduplicate against the built‑in TM. This can result in the same suggestion appearing multiple times in the UI.
• Read‑Only: External TMX files cannot be updated by Weblate; they must be managed by an external process or CAT tool and re‑imported/re‑mounted.
• Similarity Thresholds: Ensure your TM similarity threshold (found in project settings) is set above 60%. Lower values often flood translators with irrelevant matches, reducing the utility of both built‑in and external memory.