nnet3 decode fails in production after moving a working Kaldi chain recipe: i-vector input mismatch

The most common cause of this failure is an i-vector extractor configuration mismatch. When an nnet3 model is moved to production, the binary expects a concatenated input vector consisting of the acoustic features and the i-vector. If the production pipeline uses a different ivector_extractor/ directory, different splice options, or a modified MFCC configuration, the resulting input dimension will not match the input layer of the final.mdl, leading to a low-level abort.

Likely Explanation vs. Confirmed Facts

While a graph/model provenance mismatch usually triggers explicit symbol-table or topology errors, i-vector dimension complaints are almost always tied to the feature pipeline. The final.ie file is the core of the extractor, but it relies on the splice_opts and global_cmvn.stats to produce a vector of the correct size.

Confirmed Technical Requirements:

• Directory Integrity: You must replicate the entire ivector_extractor/ directory from the recipe, not just the final.ie file. This includes splice_opts, global_cmvn.stats, and online_cmvn.conf.
• Frontend Parity: The MFCC configuration (mfcc.conf) and the order of CMVN application (utterance-level followed by global) must be byte-for-byte identical to the training recipe.

Switching to Online i-vector Extraction

Switching from offline to online extraction does not require changes to the acoustic model (final.mdl). It only requires a change in the decoding pipeline configuration.

To implement this, use the online2 binaries (e.g., online2-wav-nnet3-latgen-faster) and provide an online_ivector_extraction_config that points to the original ivector_extractor/ directory. Note that this assumes the extractor was trained with the default chain recipe settings (--use-most-recent-ivector=false).

Verification Steps

To verify configuration equivalence before full deployment, run these tests on a single short utterance in both the recipe environment and the production environment:

1. Compare i-vector Values: Extract i-vectors for a test file and compare the first few frames numerically.

   ivector-extract-online2 --config=ivector_extractor/online_ivector_extractor.conf ark:test.wav ark:- | head

2. Verify Input Dimensions: Use nnet3-compute to ensure the model accepts the production features without crashing.

   nnet3-compute --use-gpu=no --print-args=false final.mdl 'ark:feat-to-len scp:test.scp ark:- |' ark,t:- | head -5

3. Hash Check: Verify that final.mdl and HCLG.fst SHA256 hashes are identical across both environments.

Missing Diagnostic: Does your production pipeline apply CMVN using a different binary or library than the original Kaldi recipe? If so, the normalization constants may differ slightly, causing a mismatch in the expected input distribution.