Online i‑vector Adaptation in Kaldi nnet3‑chain Decoding: Setup, Workflow, and Pitfalls

Useful answer

When you need low‑latency, speaker‑aware transcription with a pre‑trained Kaldi nnet3‑chain model, enable online i‑vector adaptation. It injects a short‑term speaker vector into the neural‑net input, compensating for channel and speaker mismatch without retraining the acoustic model.

Mechanism

During decoding, Kaldi processes the audio in chunks:

1. Extract MFCC features.
2. Apply online cepstral mean and variance normalization (CMVN) using a sliding window.
3. Update an i‑vector extractor with the accumulated features to produce a low‑dimensional speaker vector.
4. Project the i‑vector through an affine transform and concatenate it to the MFCC frame before the nnet3 forward pass.
5. Generate a lattice with the HCLG.fst graph that combines HMM topology, context‑dependency, lexicon, and grammar.

The i‑vector extractor must be trained on the same data used to build the nnet3‑chain model; otherwise the affine transform will mis‑align the speaker space.

Worked configuration

Below is a minimal set of files for online decoding with i‑vector adaptation. Adjust paths to match your model directory.

1. Feature configuration (conf/mfcc.conf)

--sample-frequency=16000
--use-energy=false
--window-type=hamming
--frame-length=25.0
--frame-shift=10.0
--num-mel-bins=23
--num-ceps=13

2. Online CMVN configuration (conf/online_cmvn.conf)

--cmvn-window=300
--center=true

3. i‑vector extractor configuration (conf/ivector_extractor.conf)

--ivector-dim=100
--num-gselect=5
--min-post=0.025
--posterior-scale=0.1

4. Decoding wrapper (conf/online_decoding.conf)

--frames-per-chunk=20
--extra-left-context=0
--extra-right-context=0
--extra-left-context-initial=0
--extra-right-context-final=0
--minimize=false

5. Example command (run in the directory containing final.mdl, HCLG.fst, tree)

online2-wav-nnet3-latgen-faster \
  --online=true \
  --do-endpointing=false \
  --config=conf/online_decoding.conf \
  --mfcc-config=conf/mfcc.conf \
  --ivector-extractor-config=conf/ivector_extractor.conf \
  --online-cmvn-config=conf/online_cmvn.conf \
  --max-active=7000 \
  --beam=15.0 \
  --lattice-beam=6.0 \
  --acoustic-scale=1.0 \
  --word-symbol-table=words.txt \
  final.mdl HCLG.fst \
  "ark:echo utterance-id1 utterance-id2|" \
  "scp:echo utterance-id1 /path/to/audio1.wav|" \
  "ark:|lattice-best-path --word-symbol-table=words.txt ark:- ark,t:- 2>/dev/null"

Replace final.mdl, HCLG.fst, words.txt, and the audio paths with your own files. The command streams one utterance at a time; for batch processing replace the ark: and scp: strings with proper archives.

Limits and common mistakes

Short utterances

Online i‑vectors need a few seconds of speech to converge. On utterances shorter than ~1.5 s the estimate is noisy and can actually decrease accuracy. If your application frequently processes very short commands, consider disabling i‑vector adaptation or using a fallback offline decode.

Feature‑configuration mismatch

The MFCC sample‑rate, window length, and number of cepstral coefficients must match those used when training the i‑vector extractor and the nnet3‑chain model. A mismatch (e.g., decoding at 8 kHz while the model was trained at 16 kHz) will produce silent failures or a drastic rise in word error rate, often without any error message.

Online CMVN window

The CMVN window size (--cmvn-window) should be large enough to capture speaker characteristics but not so large that it introduces excessive latency. A typical value is 300 frames (≈3 s at 10 ms shift). Using a window of 0 (global CMVN) defeats the purpose of online adaptation and can cause a mismatch with the training condition.

Version sensitivity

Kaldi’s nnet3 online decoding code changes frequently between commits. Record the exact Kaldi commit or release tag used for both training and deployment. Switching to a newer binary without rebuilding the graph may lead to silent differences in lattice scores.

Verification

To check that your setup works:

1. Decode a known test utterance offline with latgen-faster-mapped (no i‑vector, no online CMVN). Record the transcription.
2. Decode the same utterance online with the command above.
3. Compare the two transcriptions; large differences usually indicate a CMVN or i‑vector mis‑configuration.
4. Optionally, inspect the generated i‑vector (ark,t: output from the extractor) to ensure its values change smoothly over time and are not all zeros.

If the online and offline results match within a small word‑error margin, the adaptation is functioning as intended.