Kaldi Two-Pass Decoding: Lattice Rescoring for Lower Latency

When the best model is too slow to run first

\n\n

Running a large Kaldi acoustic model with a rich language model on every utterance can be too slow for interactive transcription or high-volume batch work. Kaldi decoders naturally emit word lattices instead of committing to a single transcript, which makes a two-pass design practical: a fast first pass prunes the search space, and a heavier second pass rescores only the hypotheses that survive. The useful takeaway is that this can reduce end-to-end latency, but only if the first pass is tuned carefully. If it prunes too aggressively, the correct transcript never enters the lattice, and no second-pass model can recover it.

\n\n

Why word lattices make the split possible

\n\n

A word lattice is a directed acyclic graph where each path is a candidate word sequence and each edge carries acoustic and language-model scores. Because Kaldi decoders produce these graphs by default, you do not have to lock in one hypothesis after the first pass. A compact system—such as a smaller GMM-HMM with a heavily pruned n-gram language model—can decode quickly while still leaving many plausible alternatives in the lattice. The second pass reads that lattice and applies a larger n-gram model or a neural network model to rescore the surviving paths, then typically outputs the best path or a small N-best list.

\n\n

Worked example: compact decode, then rescore

\n\n

Exact binary names and recipe scripts differ across Kaldi versions. Confirm the available tools in your local src/latbin or src/latticebin directory before wiring this into a pipeline. The commands below are illustrative patterns, not version-specific instructions.

\n\n

Run the first pass from your recipe working directory. You need read access to the model and feature directories and write access to the output directory.

\n\n

steps/decode.sh --cmd \"$decode_cmd\" --nj 8 \\\n  exp/compact_model/graph_pruned \\\n  data/test_eval \\\n  exp/compact_model/decode_test_eval

\n\n

The --nj 8 flag splits utterances across parallel jobs. The graph_pruned directory should contain a pruned language-model FST built for decoding speed.

\n\n

Next, rescore the lattices produced by that pass. A typical rescoring utility reads a lattice archive and writes updated scores to a new archive.

\n\n

# Illustrative pattern; confirm the exact binary name in src/latbin or src/latticebin\nlattice-rescore --lm larger.lm.gz \\\n  ark:exp/compact_model/decode_test_eval/lat.*.scp \\\n  ark:exp/rescore/lat.scp

\n\n

Expected checks: the rescored archive should contain the same number of lattice entries as the input. Best-path transcripts should differ from the first-pass output on at least some utterances if the larger model is meaningfully different. If every transcript is identical, the second-pass model may be too similar to the first, or the lattices may be too sparse to offer alternatives. Do not assume the second pass improved accuracy until you compare word error rates on held-out reference transcripts.

\n\n

Diagnose the recall bottleneck with oracle WER

\n\n

The first pass acts as a recall bottleneck. If the beam width or language-model pruning is too aggressive, the correct transcript may never enter the lattice, and no amount of second-pass modeling can recover it. This is the most common cause of disappointing rescored results.

\n\n

Measure the oracle word error rate of the first-pass lattices against reference transcripts. The oracle WER is the error rate of the single best path in the lattice when compared to the reference. If the oracle WER is close to the first-pass WER, the lattice lacks coverage and the bottleneck is the acoustic model or pruned language model. If the oracle WER is substantially lower, the lattice contains better hypotheses and the second pass has room to help.

\n\n

Many recipes include a lattice oracle utility. If yours does not, approximate the check by extracting an N-best list and verifying whether reference words appear. Confirm utility availability in your build before relying on it.

\n\n

# Illustrative only; confirm availability in your build\nlattice-to-nbest --n=50 \\\n  ark:exp/compact_model/decode_test_eval/lat.1.scp \\\n  ark:exp/nbest.ark

\n\n

If the reference transcript is absent from the N-best list, widen the first-pass beam or enrich the pruned language model rather than changing the second-pass model. Beam widths must be tuned empirically on your data; copying values from another recipe is a common mistake.

\n\n

Operational costs and validation

\n\n

Two-pass decoding roughly doubles the number of decoding-related jobs to schedule and monitor, which adds overhead for small datasets. Lattices can be large, so place the intermediate lattice directory on fast local storage rather than a shared network filesystem. Run the rescoring pass in parallel over utterances to keep wall-clock time acceptable.

\n\n

Validate the split on a held-out set: decode with the first pass alone, rescore the same lattices, and compare word error rates. Time the first-pass decode and the rescoring pass separately to verify that combined latency meets your requirement. If rescoring yields no improvement, check oracle coverage before investing in a larger second-pass model.

\n\n\n\n\n\n\n\n\n\n\n

Decision point	What to check	Action if problematic
Rescored WER ≈ first-pass WER	Oracle WER of first-pass lattices	Widen first-pass beam or enrich pruned LM
Oracle WER much lower than first-pass WER	Second-pass model quality	Improve or enlarge rescoring model
Combined latency too high	Separate timing of pass 1 and pass 2	Increase first-pass parallelism or reduce lattice size

\n\n

Start with coverage, then scale the model

\n\n

Begin with a compact first-pass model and moderate pruning. Measure oracle WER to confirm the lattices contain the right answers, and only then add a heavier rescoring model. That order keeps you from spending effort on a second pass that cannot fix a first-pass recall problem.