Diagnosing Kaldi nnet3 Training Failures: Hidden Random Perturbation Impact

Goal

Determine whether an undocumented default random perturbation caused a training failure in a Kaldi nnet3 run.

Constraints

Training logs do not state whether the --random-perturb option was applied. The default value may differ between Kaldi releases, and the seed handling logic is not fully documented. These factors make it hard to isolate reproducibility issues.

Diagnosis Path

Identify the actual perturbation factor used, confirm if it was applied during the run, and establish a method to enforce reproducibility by explicitly setting both seed and perturbation.

Questions

• What is the hard‑coded default value for --random-perturb when the option is omitted in the current Kaldi release?
• How can a user detect from the log files that a random perturbation was applied during a particular training session?
• Is there a recommended practice for forcing reproducibility by explicitly specifying both the seed and the perturbation factor?