Diagnosing Kaldi nnet3 Training Failures: Hidden Random Perturbation Impact

1. What is the hard‑coded default for --random-perturb?

In the latest stable Kaldi release that ships with the official nnet3-train binary (as of 2026‑10‑18), the option is defined in src/nnet3/nnet3-train.cc with a default of 0.0. That means the training script will not apply any random perturbation unless the user supplies a non‑zero value. If you are using a fork or a different release, it is safest to inspect that file yourself to confirm the value.

2. How can a user detect from the log that a random perturbation was applied?

When a non‑zero perturbation factor is in effect, the training log emits a line that contains the phrase random perturbation or perturbation factor. Typical examples are:

Random perturbation factor = 0.1
Using random perturbation factor: 0.2

If you do not see any line that mentions "random perturbation" or "perturbation factor" in the initialization section of the log, the default 0.0 was used and no perturbation was applied.

3. Recommended practice for reproducible training

To guarantee deterministic behaviour, specify both the seed and the perturbation factor explicitly on the command line:

nnet3-train \
  --seed=12345 \
  --random-perturb=0.0 \
  [other options]

Setting --random-perturb=0.0 removes the perturbation path entirely, while --seed seeds the random number generator used for data shuffling and other stochastic steps. Note that other sources of nondeterminism (e.g., multi‑threaded data loading or custom patches) can still affect reproducibility, so ensure those are also controlled if strict determinism is required.

Missing Diagnostic Detail

To tailor the recommendation to your specific environment, could you provide the exact Kaldi tag or commit you are running (e.g., kaldi-2026-09-01) and a snippet of the training log around the initialization phase? This will confirm that the default value and log wording match the build you are using.