Choosing LF-MMI Chain Training in Kaldi: What Actually Changes in Your Recipe

The problem: alignments and I/O dominate your training loop

If you have trained a conventional cross-entropy (CE) neural network in Kaldi, you know the rhythm: train a GMM system, generate frame-level alignments, dump them to disk, and train the DNN against those fixed labels. Every architecture experiment repeats the alignment step, and the model is constrained by the quality of alignments it can never revise. Kaldi's "chain" models, trained with the lattice-free MMI (LF-MMI) objective, were designed to break that dependency.

Thesis: chain training replaces per-frame labels with a sequence-level objective

Instead of aligning every frame to a phone state, LF-MMI maximizes the log-probability of the correct word (or phone) sequence relative to all competing sequences. The "lattice-free" part is the key engineering decision: rather than generating a denominator lattice per utterance (as older lattice-based MMI did), chain training compiles a single phone-level denominator FST once, before training starts. That graph encodes all possible phone sequences under a small phone language model, so the denominator computation happens on the GPU/CPU during training with no per-utterance lattice I/O.

What you actually need before training

A chain recipe still needs a baseline GMM system, but for a different reason than CE training. The GMM is used to produce numerator alignments (the reference paths) and to build the denominator graph. In a typical recipe such as wsj/s5, the preparation steps look like this (run from the recipe directory, e.g. egs/wsj/s5, with your usual cmd.sh queue configuration):

# 1. Numerator alignments from an existing triphone system
steps/align_fmllr_lats.sh --nj 20 --cmd "$train_cmd" \
  data/train data/lang exp/tri4b exp/tri4b_lats

# 2. Build the phone-level denominator FST (run once)
steps/nnet3/chain/make_den_fst.sh --cmd "$train_cmd" \
  exp/tri4b_lats exp/chain/den_fst

# 3. Generate training examples (egs)
steps/nnet3/chain/get_egs.sh --cmd "$train_cmd" \
  data/train exp/tri4b_lats exp/chain/den_fst exp/chain/egs

These scripts require read access to your data and lang directories and write access under exp/. The --nj value should match your cluster or local parallelism. Note what is not here: there is no make_den_lats.sh step. That script belongs to the older lattice-based MMI/MPE workflows and is not part of chain training.

Worked example: launching chain training

With egs prepared, training is launched through the nnet3 chain training script. A representative invocation (placeholders in angle brackets; exact flags vary across Kaldi revisions, so check steps/nnet3/chain/train.py --help in your checkout):

steps/nnet3/chain/train.py \
  --stage -10 \
  --cmd "$decode_cmd" \
  --feat.cmvn-opts "--norm-means=false --norm-vars=false" \
  --chain.xent-regularize 0.1 \
  --chain.l2-regularize 0.0 \
  --chain.frame-subsampling-factor 3 \
  --trainer.num-epochs 4 \
  --trainer.optimization.initial-effective-lrate 0.001 \
  --trainer.optimization.final-effective-lrate 0.0001 \
  --egs.dir exp/chain/egs \
  --use-gpu yes \
  --dir exp/chain/tdnn_a

Run this from the recipe directory as a user with write permission to exp/chain/tdnn_a. With --use-gpu yes, the script expects a CUDA-capable GPU; on a CPU-only machine set it to no and expect much slower training. The frame-subsampling-factor 3 is part of why chain models are fast: the network outputs labels at one-third the frame rate, which both speeds training and makes the denominator FST tractable.

Monitoring: what the logs actually tell you

Training progress files live under exp/chain/tdnn_a/log/progress.*.log. The quantity to watch is the average log-probability, reported as objective (split into the LF-MMI component and, if enabled, the cross-entropy regularizer from xent-regularize). It should trend upward (toward zero) across iterations. A sudden collapse or NaN usually points to a bad numerator alignment, a mismatched denominator graph, or an aggressive learning rate — not to "lattice problems," since no per-utterance lattices exist in this pipeline.

Trade-offs and limitations

• You still need a GMM baseline. LF-MMI removes frame-level CE targets, but the numerator alignments and the phone LM for the denominator FST come from a triphone system. The alignment step is cheaper than CE's repeated alignment cycles, but it is not gone.
• Memory pressure comes from the denominator FST and chunking, not lattices. Long utterances expand the denominator composition in memory. The standard mitigations are the frame subsampling factor, shorter training chunks (--chunk-width in the egs/config stage), and a stronger phone LM pruning when building the denominator graph.
• Results vary by recipe. Published Kaldi recipes commonly report chain models outperforming CE baselines, but the size of the gain depends on data, topology, and tuning. Treat any specific percentage as recipe-dependent and verify on your own eval set rather than expecting a fixed improvement.
• Decoding graph differences. Chain models use a 3-fold frame subsampling, so decoding uses steps/nnet3/decode.sh with the chain model's own graph; you cannot drop the model into a CE decoding pipeline unchanged.

How to verify it worked

After training, decode a held-out set and compare against your CE baseline:

steps/nnet3/decode.sh --nj 8 --cmd "$decode_cmd" \
  exp/chain/tdnn_a/graph data/test exp/chain/tdnn_a/decode_test

Then read the WER from exp/chain/tdnn_a/decode_test/scoring_kaldi/best_wer (path names vary slightly by Kaldi version). A healthy run shows the objective improving in the progress logs and a final WER at or below your baseline. If the objective plateaus early, try halving the initial effective learning rate; if it diverges, check that the numerator lattice directory and denominator FST were built from the same GMM system and tree.

Actionable closing

Start from an existing recipe rather than assembling the pipeline by hand: egs/wsj/s5/local/chain/run_tdnn.sh shows the full sequence of alignment, denominator FST construction, egs generation, and training. Run it on a small data subset first, confirm the objective rises in the progress logs, decode, and only then scale to your full corpus. The engineering win of LF-MMI is real — no per-utterance denominator lattices, a sequence-level objective, and a single training pipeline for TDNN or CNN-TDNN topologies — but it depends on a sound GMM baseline and a correctly built denominator graph, so verify those two artifacts before blaming the trainer.