Kaldi online decoding with nnet3 chain models

The practical answer

If you need low-latency speech recognition from a trained Kaldi nnet3 chain model, do not run the batch decoder on short audio clips. The batch path assumes the full utterance is available and adds buffering that defeats streaming. Instead, convert the model with steps/online/nnet3/prepare_online_decoding.sh and decode with online2-wav-nnet3-latgen-faster, which processes audio in fixed-size chunks while preserving the neural network’s training-time receptive field.

An nnet3 chain model is a Kaldi acoustic model trained with the chain objective function; receptive field means the number of surrounding frames the network inspects for each prediction. HCLG.fst is the combined decoding graph that joins HMM states, context-dependent phones, lexicon, and grammar.

How the online decoder works

The decoder reads audio in chunks, for example 20-millisecond frame groups. Because TDNN or convolutional layers look at neighboring frames, each chunk requires extra left context and sometimes extra right context so the network sees the same window it saw during training. The decoder carries that context across chunk boundaries. Smaller chunks reduce latency but increase per-chunk overhead; larger context improves accuracy but adds delay.

Prepare the model for online decoding

Run this from the experiment directory, typically under exp/chain/. You need read permission on the source model files and write permission to create the output directory.

steps/online/nnet3/prepare_online_decoding.sh \
  [options] <source-model-dir> <output-online-dir>

Replace <source-model-dir> with the directory containing final.mdl, tree, and feature transforms. Replace <output-online-dir> with a fresh directory for the online-ready model and conf/online.conf. The script copies and adapts model files; it does not modify the source experiment. If it fails partway, remove the output directory and rerun; there is no transaction to roll back.

Expected check: confirm the output directory contains final.mdl and conf/online.conf. A missing config usually means the script exited before finishing.

Decode a WAV file in streaming mode

Use the online binary with flags that control chunk size and context. The following is a representative invocation; exact flag names vary by Kaldi build, so check online2-wav-nnet3-latgen-faster --help before running.

online2-wav-nnet3-latgen-faster \
  --frames-per-chunk=20 \
  --extra-left-context=10 \
  --extra-right-context=5 \
  --frame-subsampling-factor=3 \
  --config=conf/online.conf \
  <output-online-dir>/final.mdl \
  <graph-dir>/HCLG.fst \
  <words.txt> \
  <wav-input-specifier> \
  <lattice-output-specifier>

--frames-per-chunk=20 sets the audio chunk size. --extra-left-context and --extra-right-context supply the surrounding frames the network needs. --frame-subsampling-factor=3 matches the typical chain-model subsampling; an incorrect value misaligns decoded words with audio time. <graph-dir>/HCLG.fst must come from the same experiment as final.mdl.

Configuration file

A representative conf/online.conf might contain:

--frames-per-chunk=20
--extra-left-context=10
--extra-right-context=5
--frame-subsampling-factor=3
--max-active=7000
--beam=15.0
--lattice-beam=6.0
--acoustic-scale=1.0

Feature extraction must match training exactly: same MFCC or PLP configuration, same CMVN, same i-vector extractor, and same sample rate. A common mistake is decoding with a mismatched mfcc.conf or forgetting the online CMVN used during training.

Limits and common mistakes

• Graph and model mismatch. Mixing an HCLG graph built for a GMM-HMM system with an nnet3 chain model will fail or produce garbage because transition models and context dependencies differ.
• Sample-rate mismatch. If the WAV sample rate differs from the feature pipeline, the decoder may run silently with poor accuracy.
• Missing endpointing. Kaldi’s online decoders are research-grade. They do not include robust voice activity detection, endpointing, or noise robustness out of the box. Production deployments usually wrap the decoder with custom silence and endpoint logic.
• Maintenance. Kaldi is no longer under active upstream development; treat recipes as community-maintained and verify compatibility with your compiler and CUDA version.

How to verify the result

1. Run prepare_online_decoding.sh and confirm it creates final.mdl and conf/online.conf in the output directory.
2. Decode a known test utterance and compare the transcript with the batch decoder output. They should match within a small edit distance if the model and graph are consistent.
3. Run the decoder with --verbose=3 and inspect the log for the configured chunk size, extra contexts, and feature dimensions.
4. Confirm final.mdl and HCLG.fst come from the same experiment directory and share the same tree and transition model.

If the transcript diverges sharply from batch decoding, the most likely causes are a mismatched feature config, wrong frame-subsampling factor, or a graph from a different experiment.