HMM Graph Compatibility Transition to End-to-End Neural Architectures

Configuration changes for the graph‑processing pipeline

Confirmed facts

HMM‑based decoding in Kaldi relies on a static topology defined in the .fst files and a feature‑space configuration (scf) that matches the dimensionality of the MFCC/fbank vectors used to compute emission probabilities.

Likely explanation

End‑to‑end neural architectures replace the hand‑crafted HMM parameters with a learned encoder (e.g., BiLSTM, Transformer) that produces per‑frame scores directly compatible with a decoding graph built from those scores (CTC, attention, or a neural‑graph hybrid). The graph must therefore reflect the encoder’s output dimension and the new scoring scheme.

1. Update the feature configuration:
   Edit the scf file to set feature_dim to the dimensionality of the neural encoder’s output (e.g., 256 for a BiLSTM hidden size) and remove any HMM‑specific cepstral mean/variance normalization if the network expects raw log‑mel filterbanks.
2. Regenerate the decoding graph:
   • If using CTC: build a HLG.fst from the label set and the encoder’s blank symbol with ctc_build_decoding_graph.sh (or equivalent).
   • If using an attention‑based encoder: construct a graph that encodes the language model (G.fst) and the encoder’s output scores (H.fst) via compose H G L where H now contains neural scores rather than HMM state likelihoods.
   • Ensure the topology (T) reflects the encoder’s frame‑rate (usually 1:1 with input frames) and does not contain HMM transition penalties.
3. Adjust the decoder invocation:

   # Example for CTC
   latgen-faster-mapped --max-active=7000 --beam=13.0 --lattice-beam=6.0 \
       model/HLG.fst model/final.mdl scp:test.scp ark:| lattice-best-path ark:- ark,t:trans.txt
   

   Replace model/HLG.fst with the newly built graph and point model/final.mdl to the neural‑network checkpoint (or a dummy model that simply forwards the encoder scores).
4. Align the acoustic model scale:

   Because neural scores are log‑likelihoods, set the acoustic scale (--acoustic-scale) to 1.0 (or tune on a dev set) to match the scale used during HMM training.

Verification methods for feature‑vector alignment preservation

• Frame‑level log‑likelihood comparison: Run forced alignment on a held‑out set with both the HMM system and the neural system; compute the average per‑frame log‑likelihood difference. Small systematic shifts (<0.1 nats) indicate alignment is preserved.
• Phone‑level posterior correlation: Extract phone posteriors from the HMM system (post) and from the neural system’s softmax/CTC output; compute Pearson correlation across frames. Values >0.9 suggest the neural encoder is reproducing the same distributional information.
• Word‑error‑rate (WER) sanity check: Decode a development set with both pipelines using the same language model; ensure WER difference is within the expected variance (<5% relative). Large deviations point to mis‑alignment.
• Visual inspection of alignments: Use show-alignments or align-text to plot a few utterances; verify that phone boundaries line up within one frame.

Missing diagnostic detail that could change the recommendation

The recommendation assumes the underlying compatibility graph is moderately dense. If the graph is extremely sparse (very low average node degree), HMM priors may still provide useful regularisation, and a pure end‑to‑end model could degrade performance. Please provide the average node degree or sparsity measure of your compatibility graph so we can assess whether a hybrid HMM‑neural approach is warranted.