Kaldi Chain TDNN-F: Bottleneck Tuning and LF-MMI Verification

The Problem: Denominator Lattices Consume GPU Memory and Time

If you are maintaining an older Kaldi DNN/HMM hybrid recipe, training likely generates denominator lattices for every utterance, every epoch. On a single GPU, those lattices can fill memory and force small batch sizes, stretching a Switchboard-scale run into days. Kaldi's chain architecture with time-delay neural network factorization (TDNN-F) and lattice-free MMI (LF-MMI) removes the denominator lattice step. Instead, a phone-level language model runs on the GPU during training, which cuts GPU memory use and training time roughly 3-5x compared with lattice-based MMI or sMBR.

Worked Example: Running the WSJ Chain Recipe

This example assumes a Linux host, Kaldi checked out at version 5.5 or later, compiled with CUDA support, and about 50 GB of free disk space. Run from the Kaldi WSJ recipe directory:

cd kaldi/egs/wsj/s5c
./run.sh --stage 12

Stage 12 enters the chain TDNN-F pipeline after data preparation and i-vector extraction. The heavy lifting happens in local/chain/run_tdnn_f.sh. Inspect these flags before starting:

• --tdnnf-dim — width of each factorized layer, commonly 1024-1536.
• --tdnnf-bottleneck-dim — factorization rank B, typically 128-256. Smaller B saves memory and parameters; too small hurts accuracy.
• --xent-regularize — fixed weight for frame-level cross-entropy regularization, default 0.1.
• --num-epochs — typically 4-6 for WSJ; model averaging over the final epochs is built into train.py.

TDNN-F replaces a full-rank weight matrix with two low-rank matrices separated by a bottleneck. A layer with 1,536 inputs and 1,536 outputs needs roughly 2.3 million parameters full-rank. With bottleneck 160, the factorized pair needs roughly 492 thousand parameters, about a 78% reduction. The savings are real, but B is a tuning knob, not a universal constant.

Verifying Training and Diagnosing Divergence

After training, confirm the LF-MMI objective is improving. Run this from the experiment directory, pointing at the validation diagnostic archive:

nnet3-chain-compute-prob --apply-exp=true exp/chain/tdnn_f/final.mdl \
  'ark:gunzip -c exp/chain/tdnn_f/egs/valid_diagnostic.egs.gz|'

Each printed line corresponds to an epoch or checkpoint. The log-probability per frame should trend downward across epochs. If it plateaus or rises, the run may be overfitting or the phone LM may disagree with the decoding lexicon. Do not treat a single flat line as failure; look at the trend over several epochs.

To confirm factorization took effect, compare component counts against a non-factorized TDNN:

nnet3-info exp/chain/tdnn_f/final.mdl | grep -c 'component'

A factorized chain model should show substantially fewer components than a standard TDNN recipe such as egs/wsj/s5/local/nnet3/run_tdnn.sh. The exact percentage depends on layer sizes and bottleneck dimension.

Limitations and Practical Checks

• Matched i-vectors: Train the i-vector extractor on the same data as the acoustic model. Reusing an extractor from another language can degrade word error rate noticeably.
• Matching phone LM: The G.fst used for chain training must be compiled from the same lexicon and topology as the decoding graph. Regenerate graph/HCLG.fst after any lexicon change.
• Fixed regularization: --xent-regularize 0.1 is not auto-tuned. For low-resource languages, try a small grid search between 0.01 and 0.5 on a held-out subset.
• Kaldi version: Older checkouts lack nnet3-chain-train and related binaries. Verify the binary exists before launching a multi-day job:

which nnet3-chain-train

Online Decoding With the Same Model

Chain models support streaming inference without a separate model format. The binary online2-wav-nnet3-latgen-faster reads the same final.mdl and an online.conf that defines feature extraction, i-vector period, and chunk size. Example for one WAV file:

online2-wav-nnet3-latgen-faster \
  --config=exp/chain/tdnn_f/conf/online.conf \
  --max-active=7000 --beam=15.0 --lattice-beam=6.0 \
  exp/chain/tdnn_f/final.mdl \
  exp/chain/tdnn_f/graph/HCLG.fst \
  'ark:echo utt1 utt1|' 'scp:echo utt1 test.wav|' ark:/dev/null

This runs on CPU for a single utterance. For production, wrap the binary in a service that keeps i-vector state across audio chunks.

Closing Checklist

1. Confirm Kaldi >= 5.5 with CUDA and nnet3-chain-train present.
2. Train i-vectors on your target data; do not reuse cross-lingual extractors.
3. Start with bottleneck dimension 160 and validate with a short run before full training.
4. Watch nnet3-chain-compute-prob output for a downward LF-MMI trend.
5. Regenerate HCLG.fst whenever the lexicon or topology changes.
6. Run one online decode with the same model before deploying.

Chain TDNN-F is not a drop-in replacement that tunes itself. The engineering trade-off is a small set of knobs—bottleneck dimension, cross-entropy weight, matched i-vectors, and a matching phone LM—in exchange for smaller models, less GPU memory, and faster training. Verify the objective trend before you trust the word error rate.