Kaldi nnet3 Online Decoding: Configuration Parity and Serving Reality

The Streaming Gap Nobody Warns You About

You trained a solid chain TDNN-F model offline. Word error rate looks good on your test set. Now product needs streaming—partial hypotheses every few hundred milliseconds. The Kaldi docs say online decoding is supported. You enable it, point at the same HCLG graph, and... the results degrade silently. No errors, just worse WER.

The problem isn't the decoder. It's configuration drift between offline training and online serving. Feature extraction, CMVN, i-vector extraction, and chunking parameters must match exactly. A single mismatch—frame shift, splice context, or CMVN stats—causes silent accuracy loss. This post walks through the decisions that keep streaming WER close to offline, and the serving architecture you'll actually need.

Feature Parity Is Non-Negotiable

Offline training uses steps/nnet3/chain/train.py with a feature pipeline defined in conf/mfcc.conf, conf/splice.conf, and conf/online_cmvn.conf. Online decoding reads conf/online.conf. These files must describe identical transforms.

Check these four points before you blame the model:

• Frame shift and sampling rate: frame-shift=10 in mfcc.conf must match frame_shift in online.conf. A 10 ms shift offline and 25 ms online breaks everything.
• Splice context: left-context and right-context in splice.conf must match the --left-context and --right-context passed to nnet3-chain-online-decode-faster or the equivalent API calls.
• CMVN stats: online_cmvn.conf points to global_cmvn.stats computed on training data. The online decoder must load the same file. If you recompute stats on a different corpus, means and variances shift.
• i-vector extractor: If your chain model uses i-vectors (most TDNN-F recipes do), the online config needs ivector-extraction-config pointing to the same extractor directory used in training. The ivector-period must match—typically 10 frames.

Run this sanity check before any serving work:

# From your recipe root (e.g., egs/wsj/s5)
# Compare offline vs online feature dims
feat_dim=$(feat-to-dim scp:data/train/feats.scp ark:- 2>/dev/null | head -1)
echo "Offline feature dim: $feat_dim"

# Online decoder prints feature dim at startup if you add --verbose=2
online2-wav-nnet3-latgen-faster \
  --online=true \
  --config=conf/online.conf \
  --max-active=7000 \
  --beam=15.0 \
  --lattice-beam=6.0 \
  --acoustic-scale=1.0 \
  final.mdl HCLG.fst \
  'ark:echo dummy|' 'ark:/dev/null' 2>&1 | grep -i "feature dim"

Run this on the training machine where final.mdl and conf/ live. You need read access to model and config directories. Expected output: identical feature dimensions. If they differ, stop—fix config before proceeding.

Chunk Size Versus Lookahead: The Latency-Accuracy Trade-off

Online decoding processes audio in chunks. Two parameters control the latency-accuracy curve:

• Chunk size (--frames-per-chunk or chunk_size in online.conf): How many frames the decoder consumes per forward pass. Typical values: 50 frames (500 ms) for lower latency, 100 frames (1 s) for better accuracy.
• Right context / lookahead: How many future frames the acoustic model sees. In chain TDNN-F, this is baked into the model architecture (the right-context in the nnet3 config). You cannot increase it at decode time without retraining.

Smaller chunks reduce latency but increase WER because the model sees less future context per decision. The relationship isn't linear. For a typical TDNN-F with 20-frame right context:

• Chunk 100 frames (1 s): WER within 0.5–1.0% absolute of offline
• Chunk 50 frames (500 ms): WER +1.5–3.0% absolute
• Chunk 25 frames (250 ms): WER +4–6% absolute, partial hypotheses become jittery

Measure this on your data. Run the offline decode and online decode on the same held-out set:

# Offline baseline
steps/nnet3/decode.sh --nj 20 \
  exp/chain/tdnnf/graph data/test exp/chain/tdnnf/decode_test

# Online with chunk=50
steps/online/nnet3/decode.sh --nj 20 \
  --frames-per-chunk 50 \
  exp/chain/tdnnf/graph data/test exp/chain/tdnnf/decode_test_online_chunk50

# Compare WER
cat exp/chain/tdnnf/decode_test/scoring_kaldi/best_wer
cat exp/chain/tdnnf/decode_test_online_chunk50/scoring_kaldi/best_wer

Run on a GPU/CPU machine with Kaldi compiled. You need the test data prepared with the same feature pipeline. The offline decode uses the full utterance; the online decode simulates streaming by feeding chunks. The WER gap is your latency budget cost.

Serving: Kaldi Is a Library, Not a Server

Kaldi provides src/onlinebin/online2-wav-nnet3-latgen-faster and the C++ OnlineNnet3Decoder API. It does not provide HTTP, gRPC, connection pooling, or model hot-reload. You must wrap it.

Three common patterns:

1. gRPC service wrapping C++ decoder: Each request gets a decoder instance. State (feature pipeline, decoder) lives in the request handler. Simple but memory-heavy—each decoder holds the model, HCLG, and feature buffers.
2. Shared model, per-request decoder: Load final.mdl and HCLG.fst once at startup. Create lightweight OnlineNnet3Decoder instances per stream. Requires thread-safe reference counting on the model and graph. Kaldi's TransitionModel and fst::Fst are read-only after load, so this works if you avoid mutable shared state.
3. Export to ONNX/TensorRT, serve with Triton or custom runner: Export the nnet3 neural net to ONNX, keep Kaldi only for graph decoding and feature extraction. Reduces C++ surface area but adds export complexity and potential numerics drift.

For most teams, option 2 is the pragmatic choice. A minimal gRPC skeleton:

// Pseudocode—compile against Kaldi src
class AsrServiceImpl : public Asr::Service {
  std::shared_ptr am_model;
  std::shared_ptr> decode_fst;
  OnlineNnet3DecodingConfig decoder_config;

  AsrServiceImpl() {
    // Load once at startup
    am_model = std::make_shared();
    ReadKaldiObject("final.mdl", am_model);
    decode_fst = fst::ReadFstKaldi("HCLG.fst");
    ParseConfig("conf/online.conf", &decoder_config);
  }

  Status StreamRecognize(ServerContext* ctx,
                         ServerReaderWriter* stream) {
    // Per-request decoder
    OnlineNnet3Decoder decoder(decoder_config,
                               *am_model,
                               *decode_fst);
    // Feed chunks, emit partials...
  }
};

Compile with -pthread and link libkaldi-online.a, libkaldi-nnet3.a, libkaldi-decoder.a, libkaldi-fst.a. Run the service on a machine with the model files accessible. Test with a small client streaming 16 kHz PCM chunks.

Worked Example: Minimal online.conf That Matches Training

Assume a standard chain TDNN-F recipe (e.g., egs/wsj/s5 or egs/librispeech/s5). Your conf/online.conf should contain:

# Feature config
--mfcc-config=conf/mfcc.conf
--ivector-extraction-config=conf/ivector_extractor.conf

# CMVN
--online-cmvn-config=conf/online_cmvn.conf

# Decoder config
--frames-per-chunk=50
--extra-left-context-initial=0
--max-active=7000
--beam=15.0
--lattice-beam=6.0
--acoustic-scale=1.0

# Endpointing (optional, for VAD)
--endpoint.silence-phones=1:2:3:4:5
--endpoint.rule2.min-trailing-silence=0.5
--endpoint.rule3.min-utterance-length=0

Verify each referenced file exists and matches training:

• conf/mfcc.conf — same as used in steps/make_mfcc.sh
• conf/ivector_extractor.conf — points to exp/nnet3/extractor from training
• conf/online_cmvn.conf — cmvn-file=exp/nnet3/global_cmvn.stats

If you changed any training config (e.g., added pitch features), replicate that change here. The decoder will not warn you.

Limitations You Cannot Configure Away

• WER gap is real: Even with perfect config parity, streaming WER exceeds offline by 1–5% absolute depending on chunk size and acoustic model right context. Budget for this in product requirements.
• Thread safety is your problem: OnlineNnet3Decoder is not thread-safe. One decoder per stream. If you share the AmNnetSimple and FST across threads, ensure no mutable access. The neural net forward pass uses CuMatrix—if you run on GPU, serialize GPU calls or use per-thread CUDA streams.
• Upstream maintenance: Kaldi's main branch moves slowly. Community forks (e.g., kaldi-asr/kaldi mirrors, pzelasko/kaldi, or vendor forks like NVIDIA's NeMo-Kaldi integration) may have fixes for online decoding bugs. Pin your commit and test upgrades.
• No built-in VAD or endpointing that works out of the box: The endpoint. config options exist but require tuning per acoustic environment. Plan to integrate a separate VAD (WebRTC, Silero) for production.

Closing Checklist Before You Ship

1. Run offline and online decode on the same held-out set. Record WER delta. If >3% absolute at your target chunk size, reconsider chunk size or model architecture.
2. Diff conf/mfcc.conf, conf/splice.conf, conf/online_cmvn.conf, conf/ivector_extractor.conf against the training recipe. Every parameter must match.
3. Load-test your serving wrapper with concurrent streams. Monitor GPU memory (if using CUDA) and CPU latency per chunk. Target <50 ms per chunk forward pass on your hardware.
4. Add a health endpoint that runs a 1-second audio smoke test and asserts partial hypotheses arrive.
5. Document the exact Kaldi commit, model checksum, and config file hashes in your deployment manifest.

Streaming ASR with Kaldi works—but only when the serving pipeline treats configuration parity as a first-class constraint, not an afterthought.