Kaldi online2 Streaming Decoder and nnet3 Models: Adapt Features at Runtime or Fail on Pipeline Mismatch?

Question

Kaldi's online2 decoder performs streaming recognition by extracting features on the fly, and it relies on the acoustic model having been trained with a matching feature configuration — the same feature type, dimensionality, and normalization treatment. When an nnet3 model is loaded, the feature options stored alongside it act as the contract the live pipeline must satisfy, and a mismatch aborts decoding rather than being silently tolerated.

The open integration question sits at this boundary. nnet3 recipes keep feature options in the model directory, but there is no runtime mechanism that reshapes or renormalizes incoming streaming features to satisfy a model whose pipeline differs — for example, when decoding uses binaries built from a different Kaldi revision, or normalization options the decoder build predates. A runtime adapter has been proposed as a graceful-degradation path, but whether it belongs in the decoder, and how it should behave, remains unresolved.

Specific questions:

Should online2 adapt streaming features at runtime to match a model's stored configuration, or should pipeline mismatches remain hard failures?
Where is the supported compatibility boundary between online2 binaries and nnet3 models trained under different Kaldi revisions?
If adaptation is added, how should the decoder signal that recognition ran on adapted rather than native features?

Tasadduq Burney · Answer

Answer
The online2 streaming decoder does **not** adapt incoming features at runtime to match an nnet3 model’s stored configuration. A mismatch between the live feature extraction options and the model’s feature options causes the decoder to abort with a hard failure rather than silently degrading.
Likely Explanation
The decoder reads the feature options (type, dimension, splice left/right, CMVN type) from the model directory (e.g., online.conf or final.feature_proto) and compares them to the options used by the feature extraction binary in the decoding pipeline. If any field differs, the decoder treats this as a contract violation and exits. No runtime feature‑adaptation layer exists in the current codebase, so the decoder assumes the pipeline is fixed at compile time.
Steps to Avoid Mismatch

Use the exact same feature extraction command (e.g., compute-mfcc-feats or compute-plp-feats) with the identical option string that was used when the nnet3 model was trained.
Verify that the model’s online.conf (or final.feature_proto) matches the options you pass to the feature extraction binary in the online2 decoder.
Run the decoder with verbose logging (--verbose=2) to confirm that no “feature config mismatch” error appears before decoding starts.

Missing Diagnostic Detail
To determine whether a lightweight runtime adjustment could safely bridge a gap, we need to know **which specific feature option differs** (e.g., CMVN mean/variance, splice context, or feature type). If only the CMVN statistics differ, a simple mean/variance normalization could be added; any difference in feature type, dimension, or splice would require retraining or rebuilding the decoder.

Kaldi online2 Streaming Decoder and nnet3 Models: Adapt Features at Runtime or Fail on Pipeline Mismatch?

1 answer

Answer

Likely Explanation

Steps to Avoid Mismatch

Missing Diagnostic Detail

Your perspective belongs here.

0 question comments