Kaldi HMM-GMM Speech Recognition: Architecture, Checks, and Limits

Problem: The HMM-GMM Stack Looks Like a Wall of Scripts

You have transcribed speech and need a working decoder in Kaldi, but the training recipes look like an opaque sequence of shell scripts. The useful takeaway is that Kaldi's HMM-GMM design is a composition of separate models—acoustic, lexical, grammatical, and linguistic—into a single Finite State Transducer (FST) graph. Most operational failures trace back to a mismatch at one of those boundaries rather than to the Viterbi decoder itself.

Requirements and Version Assumptions

This note assumes a recent Kaldi source distribution with the standard egs/ example recipes. Script paths and flags vary between recipes, so verify the usage message in your checkout before running anything. You need:

• A compiled Kaldi toolchain with write access to an experiment directory such as exp/.
• A data preparation directory containing waveform files and transcripts with consistent sampling rate and channel characteristics.
• A pronunciation lexicon and language-model text for the target domain.
• Enough disk space for feature archives and the composed decoding graph.

HMM-GMM acoustic models consume Mel-frequency cepstral coefficients (MFCCs)—compact spectral summaries of short audio frames—rather than raw waveforms. If your evaluation audio differs in sample rate or channel from training audio, the MFCC pipeline will produce mismatched features even if the model is sound.

Smallest Suitable HMM-GMM Design

Do not build a pipeline from scratch. Start from an existing recipe, strip it to the essential stages, and confirm each artifact before adding complexity. The minimal viable design has five stages:

1. Data and language preparation: organize audio and transcripts into Kaldi data directories; build the lexicon and language model.
2. Feature extraction: compute MFCCs and store them in feature archives.
3. Acoustic model training: train a monophone GMM-HMM first, then a triphone model with delta features.
4. Graph compilation: compose the acoustic model, lexicon, and language model into HCLG.fst.
5. Decoding and scoring: run the decoder on a held-out test set and compute Word Error Rate (WER).

Stage	Primary artifact	What it proves
Feature extraction	MFCC archives	Audio is readable and frame-consistent
Monophone training	Aligned monophone model	Data and lexicon are compatible
Triphone training	Context-dependent model	Alignments are stable across contexts
Graph compilation	HCLG.fst	Models compose without vocabulary gaps
Decoding	Scored hypotheses	End-to-end accuracy is measurable

Trust and Data Boundaries

Treat each stage output as a separate trust boundary. Raw audio is trusted only if its recording conditions match the training corpus. MFCC features are a deterministic transform, but they inherit any sample-rate or channel mismatch from the source. Model parameters are trustworthy only if the training transcripts are clean and the lexicon covers the spoken words. The HCLG.fst graph is trustworthy only if the language model domain matches the test domain. A decoder cannot recover from a lexicon that lacks a pronunciation for a common test word.

In practice, this means you should validate data boundaries before blaming the model. Check that test waveforms share the training sampling rate, that transcripts use the same text normalization, and that the lexicon contains every word in the test references.

Operational Checks

Training and Graph Compilation

Run the training script from a recipe directory such as egs/<corpus>/s5. You need write permission for the experiment directory and read permission for the data and language directories.

# Representative invocation; exact flags vary by recipe
# Run from egs/<corpus>/s5
steps/train_hmm_gmm.sh --cmd "$train_cmd" <data_dir> <lang_dir> <exp_dir>

After training, confirm that the graph directory contains a non-empty HCLG.fst. The filename encodes the composed models: H for the HMM acoustic model, C for context-dependency, L for the lexicon, and G for the grammar/language model. A missing or tiny graph usually indicates a lexicon or language-model preparation failure, not a decoder bug.

Decoding and WER

Decode a held-out set and score against reference transcripts.

# Representative invocation; exact flags vary by recipe
steps/decode.sh --cmd "$decode_cmd" <graph_dir> <test_data_dir> <decode_dir>

Expected checks: the decoder should produce hypothesis files under the decode directory, and the scoring step should report a WER percentage. If the script exits early, inspect the log for out-of-memory errors or missing feature files before re-running.

Failure Modes and Diagnostics

HMM-GMM decoding fails in predictable ways:

• Deletion errors: the model skips spoken words. Often caused by weak acoustic likelihoods, low-speaking-rate mismatch, or missing pronunciations in the lexicon.
• Insertion errors: noise or silence is decoded as speech. Frequently a language-model or silence-modeling issue, especially when the test acoustic conditions differ from training.
• Out-of-memory during graph loading: memory consumption scales with the size of HCLG.fst. Large vocabularies or unpruned language models can exhaust RAM before decoding starts.
• Feature mismatch: inconsistent sampling rate or channel characteristics between training and test audio degrade alignment without producing obvious script errors.

A practical diagnostic decision is to compare the test WER against a small, well-matched development set. If WER is acceptable on matched data but poor on the target set, suspect a data-boundary problem. If WER is poor on both, inspect alignments and lexicon coverage before changing models.

Conditions That Would Change the Design

HMM-GMM is a reasonable baseline, but it is generally less accurate than modern deep-learning acoustic models in complex acoustic environments. Change the design when:

• Vocabulary is large: the composed FST grows until it no longer fits in memory. Options include a pruned language model, a smaller lexicon, or on-the-fly graph composition.
• Acoustic conditions are noisy or far-field: consider a DNN-HMM hybrid or explicit feature enhancement rather than retraining the GMM with more data.
• Latency matters: batch HCLG decoding loads the full graph; a streaming design may require a different decoder configuration.

Limitations and Review Notes

This architecture note is based on established Kaldi behavior and requires current verification against your specific recipe and Kaldi version. Script names, argument order, and scoring output format can differ between example distributions. Before relying on any command, run it with a --help flag or read the script header in your source tree. Claims about accuracy relative to neural models are directional, not measured benchmarks; validate them on your own data.

A practical way to check the final result is to decode a held-out test set, compute WER against gold-standard transcripts, and inspect the most frequent substitution and deletion pairs. If the error patterns cluster around specific words or acoustic conditions, the next fix is usually in the data or language-model boundary, not in the Viterbi search itself.