Choosing fMLLR for Speaker Adaptation in Kaldi Pipelines

Problem: speaker variability hurts recognition

When a speech recognizer is trained on one set of speakers but deployed on others, word error rates (WER) often rise because acoustic features shift with each speaker’s vocal tract characteristics.

Thesis: fMLLR can reduce this mismatch with modest data, but you must weigh its computational cost and the risk of over‑adaptation.

What fMLLR does

Feature Space Maximum Likelihood Linear Regression (fMLLR) estimates a linear transform that maps raw features into a space where the acoustic model likelihood is maximized for a given speaker. The transform is a matrix **A** and offset **b** applied as y = A x + b. In Kaldi the matrix is stored per speaker and applied during feature extraction or nnet3 input.

When to apply it

• You have a small enrollment set (typically 10‑30 seconds) per speaker.
• Your baseline system shows a noticeable WER gap between training and evaluation speakers.
• You can afford the extra CPU time for computing and applying the transform.

Worked example: loading and applying an fMLLR matrix

The following script assumes Kaldi 5.5+, that you have already computed an fMLLR matrix spk001.mat for speaker spk001, and you want to test the transform on a single utterance feature vector.

#!/usr/bin/env bash
# Paths – adjust to your experiment
feat_dir=data/test/feats
mat_dir=exp/fmllr/spk001
out_dir=tmp/fmllr_test
mkdir -p $out_dir
# Read one utterance (replace utt001 with an actual key)
feat_ark=$feat_dir/feats.ark
# Extract the vector for utt001 using copy-feats (Kaldi utility)
copy-feats "ark:$feat_ark" ark,t:- | grep '^utt001' | cut -d' ' -f2- > $out_dir/utt001.vec
# Load the fMLLR matrix (binary format)
# Apply transform: y = A x + b
# Kaldi provides apply-transform
apply-transform --binary=false $mat_dir/spk001.mat ark:$out_dir/utt001.vec ark,t:$out_dir/utt001_fmllr.vec
# Check dimensions
feat_dim=$(feat-to-dim "scp:$feat_dir/feats.scp" | head -1)
echo "Original feature dimension: $feat_dim"
# The transformed vector should have the same dimension
trans_dim=$(feat-to-dim "scp:$out_dir/utt001_fmllr.scp" | head -1)
echo "Transformed feature dimension: $trans_dim"
if [ "$feat_dim" -eq "$trans_dim" ]; then
  echo "Dimension check passed."
else
  echo "Dimension mismatch – verify matrix size."
fi

Explanation: copy-feats pulls the raw feature vector, apply-transform multiplies it by the matrix and adds the offset, and feat-to-dim confirms that the output dimensionality matches the input (the transform is square). If the matrix is rectangular, you would see a dimension change, indicating a mismatch.

Trade‑offs and limitations

Practical verification

1. Confirm the fMLLR matrix dimensions: matrix-dim --binary=false exp/fmllr/spk001/spk001.mat should output D D where D is the feature dimension (e.g., 40 for MFCC).
2. Run a small speaker‑adaptation test: decode a held‑out subset with and without fMLLR, log WERs, and ensure the fMLLR version is not worse.
3. Check saturation ranking in your decode script: the fMLLR transform should appear before speaker‑adaptive training (SAT) or i‑vector extraction, depending on your recipe.

Actionable closing

If you observe a persistent speaker‑related WER gap, try the following steps:

1. Extract a short enrollment utterance per target speaker.
2. Run steps/align_fmllr.sh (or the equivalent in your recipe) to estimate the transform.
3. Insert the resulting .mat files into your feature pipeline using apply-transform or the --fmllr option of compute‑mfcc‑feats.
4. Verify dimension equality and run a quick WER check on a validation set.
5. If WER improves and CPU load is acceptable, keep the transform; otherwise, revisit enrollment quality or consider alternatives like i‑vectors.