Choosing a Learning‑Rate Schedule for Kaldi nnet3 Discriminative Training

Problem: unstable or slow convergence in discriminative nnet3 training

When you run steps/train_nnet3.sh for a sequence‑discriminative criterion such as MMI or sMBR, you may notice that the training loss either oscillates wildly or improves only marginally after many epochs. This often stems from using a constant learning rate that is either too high for the later stages of training or too low to make meaningful progress early on.

Thesis: a simple piecewise‑constant decay, configured via learn_rate_factor and learn_rate_drop_epochs, gives a predictable stepwise reduction that balances early‑stage speed with later‑stage stability.

Understanding the nnet3 learning‑rate schedule

Kaldi’s nnet3 trainer does not apply a built‑in warm‑up; the learning rate stays constant unless you explicitly request drops. The schedule is defined as:

• --learn-rate <float> – the initial learning rate (e.g., 0.01).
• --learn-rate-factor <float> – multiplicative factor applied at each drop (default 10 in Kaldi 5.5, previously 5).
• --learn-rate-drop-epochs <list> – space‑separated epochs after which the rate is multiplied by the factor (e.g., "4 8").

At each listed epoch the trainer multiplies the current rate by learn_rate_factor (i.e., divides by the factor if you think of it as a decay). The resulting rate is logged to train/log/compute_prob.<job>.log and appears in the progress output as "lr:".

Worked example: setting up a decay for Switchboard MMI training

Assume you have already built the denominator lattices and the alignment directory exp/tri4_ali. The following steps illustrate how to apply a decay that drops the rate by 10× at epochs 4 and 8.

1. Create a training configuration file (or edit the existing one) conf/train_nnet3_mmi.conf with:

# conf/train_nnet3_mmi.conf
learning-rate=0.01
learn-rate-factor=10
learn-rate-drop-epochs="4 8"
# other nnet3 options …

1. Run the training script from the egs/switchboard/s5 directory (or your recipe’s root). You need read access to the data and write access to the experiment directory.

steps/train_nnet3.sh \
  --nj 8 \
  --cmd "run.pl" \
  --learn-rate 0.01 \
  --learn-rate-factor 10 \
  --learn-rate-drop-epochs "4 8" \
  data/train data/lang exp/tri4_ali exp/nnet3_mmi

During execution you should see lines such as:

... lr:0.010000 ...
... lr:0.001000 ...   # after epoch 4
... lr:0.000100 ...   # after epoch 8

These values confirm the schedule is active. After training finishes, inspect the final WER:

steps/decode_nnet3.sh --nj 4 exp/tri4/graph data/test exp/nnet3_mmi/decode_test

Compare this WER to a baseline run where you omitted --learn-rate-factor and --learn-rate-drop-epochs (i.e., a constant 0.01 rate). The research note indicates a ~3 % relative WER improvement when the decay is applied at epochs 4 and 8 on Switchboard.

Trade‑offs and limitations

• Overtraining risk: If the decay is too aggressive (large factor or early drops), the optimizer may stall before the model has fully discriminated the training data, leading to higher validation WER. Monitor exp/nnet3_mmi/log/compute_prob.*.log after each drop; a rising validation WER signals trouble.
• Version dependence: The default drop factor changed from 5 to 10 in Kaldi 5.5. If you are using an older release, you must explicitly set --learn-rate-factor 5 to reproduce the historic behavior.
• No warm‑up: The schedule starts at the full rate immediately. For very noisy gradients, some practitioners prepend a manual warm‑up by scaling the initial rate down for the first epoch via a custom script; this is not built‑in.

Actionable checklist

1. Pick an initial learning rate that yields a steady loss decrease in the first epoch (typical range 0.005–0.02).
2. Choose drop epochs where the training loss curve begins to plateau (often after 3–5 epochs for Switchboard‑size data).
3. Set --learn-rate-factor to 10 for Kaldi 5.5+ (or 5 for earlier versions).
4. Launch training with the flags above and watch the learning‑rate log for the expected stepwise drops.
5. After each drop, check the validation WER (or objective) in the log; stop early if it worsens.
6. Finally, decode with the best model (selected by lowest validation WER) and compare to a constant‑rate baseline.

By following these steps you turn an opaque training process into a controllable experiment where the learning‑rate schedule is a visible, tunable lever rather than a hidden source of instability.