Designing a Minimal Kaldi Online Decoder: HCLG Graphs, Streaming Features, and Lattice Output

If you need real-time speech recognition with Kaldi, the tempting move is to copy one of the full online decoding binaries and wire it into your service. That works until you need to reason about latency spikes, corrupted input, or a model upgrade — at which point a black-box binary becomes a liability. This note describes the smallest design that still gives you streaming decode with lattice output: a single-threaded loop around Kaldi's DecodableInterface and an HCLG graph, with explicit trust boundaries and operational checks. It assumes Kaldi 5.5 or later, a 16 kHz acoustic model, and CPU-only inference.

Requirements

Three things must exist before you write any decoding code:

• A pre-built HCLG graph. This is the composed finite-state transducer that maps HMM states (H) through context dependency (C) and the lexicon (L) to the grammar or language model (G). Building it is an offline step; the decoder treats the resulting HCLG.fst as read-only.
• A feature pipeline. Typically MFCC or log-mel filterbank extraction with cepstral mean and variance normalization (CMVN) — a per-utterance or sliding-window affine transform that recenters features so channel and speaker variation hurts less.
• A decoder object you own. SingleUtteranceNnet3Decoder (for chain/TDNN-F models) or the older GMM decoder, driven through DecodableInterface, which is the abstraction the decoder uses to pull acoustic scores frame by frame.

The smallest suitable design

One thread, one loop, no lattice rescoring, no GPU:

1. Read a chunk of audio (e.g., 200–500 ms) from your source.
2. Compute features for the chunk and apply CMVN using accumulated statistics.
3. Validate the feature matrix (dimension, NaN check, energy sanity).
4. Wrap the features in a DecodableInterface implementation and call AdvanceDecoding().
5. Repeat until end-of-utterance, then call FinalizeDecoding() and extract the best path or a lattice via GetLattice().

A sketch of the core loop (C++, inside your service process):

// Runs in the decoding thread you own; no special permissions needed.
// 'feature_matrix' must have NumCols() == model feature dim (e.g., 40 for MFCC).
if (!ValidateFeatures(feature_matrix)) {   // NaN / dim / energy checks
  KALDI_WARN << "Skipping corrupt chunk";
  continue;
}
DecodableMatrixMapped decodable(am_nnet, trans_model,
                                feature_matrix, ivector);
decoder.AdvanceDecoding(&decodable, num_frames);
// After end-of-utterance:
decoder.FinalizeDecoding();
Lattice lat;
decoder.GetLattice(&lat, /*end_of_utterance=*/true);

This design deliberately omits n-best rescoring, endpointing beyond simple energy/silence rules, and multi-pass decoding. Each of those adds latency and failure surface; add them only when a measured requirement forces it.

Trust and data boundaries

Draw the boundaries explicitly, because they determine where validation goes:

• Trusted, immutable: the HCLG graph, the acoustic model, and the decoding configuration. Load once at startup, memory-map or pin them, and never mutate per request. Treat a graph/model mismatch (e.g., wrong final.mdl paired with an HCLG built from a different tree) as a startup failure, not a runtime warning.
• Untrusted: incoming audio and everything derived from it. Feature vectors are the boundary object — validate them before they reach the decoder. NaNs or extreme magnitudes in features can corrupt decoder state or crash the process; a cheap check (finite values, plausible energy range, correct dimension) per chunk is sufficient.
• Trusted-but-derived: CMVN statistics. If you compute them from a fixed enrollment utterance set, treat the resulting transform as configuration. If you compute them online from the stream itself, they inherit the stream's untrusted status and should be clamped.

Operational checks

Instrument three things from day one:

• Feature sanity per chunk: dimension match against the model's expected input, finite values, energy above a noise floor. Log and skip bad chunks rather than aborting the utterance.
• Token count: the decoder's active-token count is your early-warning signal for search-space explosion. Set a configurable limit (max-active in decoder_opts, commonly 7000–10000 for chain models) and alert when it is hit repeatedly — that means your beam or grammar is wrong for the model.
• Real-time factor: wall-clock decode time divided by audio duration, logged per utterance. Sustained RTF above ~0.5 on one core leaves no headroom for load spikes.

Failure modes

Failure	Symptom	Handling
Token explosion (beam too wide, grammar too permissive)	RTF climbs, max-active limit hit	Tighten beam; for chain models use beam ≈ 15, lattice-beam ≈ 6–8 as a starting point, then tune
Corrupt features (NaN, wrong sample rate)	Garbage output or crash	Input validation; resample to 16 kHz before feature extraction
Graph/model mismatch	Nonsense transcripts from first frame	Fail fast at startup by checking model and graph provenance
Decoder shared across threads	Intermittent corruption, hard to reproduce	One decoder per thread; the decoder object is not thread-safe

The sample-rate failure deserves emphasis: Kaldi's online pipeline assumes the rate the model was trained at (usually 16 kHz). Feeding 8 kHz or 48 kHz audio without resampling does not error out — it silently produces misaligned features and degraded output. Verify the rate at the ingestion boundary.

Conditions that would change this design

• Switching acoustic model type. Moving from a GMM-HMM model to a TDNN-F chain model requires rebuilding HCLG (different topology and denominator FST for lattice rescoring) and retuning beam values. Beams tuned for GMM models cause latency blowups or search errors on chain models.
• Concurrency requirements. If one utterance per process is not enough, scale by process or by decoder-per-thread with external locking — never by sharing a decoder instance.
• Word-level confidence or alternatives. If consumers need more than the best path, keep lattice generation (already in this design) but add deterministic pruning (lattice-beam) so lattice size stays bounded.
• Latency budget below ~200 ms. You would need smaller chunk sizes, endpointing, and possibly GPU batching — at which point the single-threaded loop is no longer the right shape.

Verifying the design

Before trusting your own loop, reproduce the behavior with Kaldi's reference tooling. Clone the Kaldi repository (release 5.5 or later), build with ./configure && make in src/ (requires a C++ toolchain and OpenFst; run as a normal user, not root), then run the online decoding example at egs/wsj/s5/online2/online_decode.sh. Inspect the decoder log for token counts and RTF. Then, in your own build, vary --beam and confirm that token counts stay under your configured max-active, and inject a NaN-filled chunk to confirm your validation path logs and skips rather than crashing. Do not treat the reference script's defaults as correct for your model — they are tuned for the WSJ recipe's specific acoustic model.

Limitations

This note covers CPU decoding of a single utterance stream with chain or GMM models. It does not address GPU batched decoding (batched-wav-nnet3-online paths), neural rescoring of lattices, or i-vector adaptation for speaker robustness, all of which change the latency and complexity picture. Beam values quoted are starting points, not guarantees — measure against your own model and audio.