Kaldi nnet3 Upgrade Rollback: Restoring Original Training Scripts After a Failed Build

Likely explanation

The upgrade modified files in src/ and the example directories under egs/ (especially steps/ and local/) while leaving data/ and exp/ untouched. The build failed because the new source references compiler flags that the old binaries cannot use.

Confirmed facts

• If the original training scripts were under version control (git) or were copied to a backup before the upgrade, they can be restored without touching data/ or exp/.
• Reverting only the script directories (steps/ and local/) is sufficient to recover the 4.2 build system; the src/ tree can be cleaned and rebuilt.
• After restoring the scripts, a clean make in src/ followed by the recipe’s run.sh will verify that the old binaries work.

Rollback steps

1. Determine whether you have a git repository covering the Kaldi tree (including egs/) or a manual backup of the pre‑upgrade steps/ and local/ directories.
2. If using git:

   cd $KALDI_ROOT
   git fetch --all
   git checkout <commit‑hash‑of‑last‑working‑build> -- egs/your_recipe/steps egs/your_recipe/local
   

3. If using a backup:

   cd $KALDI_ROOT
   cp -r /path/to/backup/egs/your_recipe/steps egs/your_recipe/
   cp -r /path/to/backup/egs/your_recipe/local egs/your_recipe/
   

4. Remove any partially compiled objects that may have been built with the mixed source:

   make clean -C src
   

5. Re‑run the configuration step to ensure the build system matches the reverted scripts:

   ./configure --use-cuda=no   # adjust flags as needed for your setup
   

6. Compile the source:

   make -j $(nproc) src
   

7. Run the training recipe to confirm everything works:

   cd egs/your_recipe
   ./run.sh
   

Verification

• Check that the compilation finishes without the earlier error messages about incompatible flags.
• Observe the first few lines of run.sh output; they should show the nnet3 chain being built and the model directory being created under exp/.
• Optionally, run a simple decoding command (e.g., steps/nnet3/decode.sh) on a small test set to ensure the binaries are executable.

Missing diagnostic detail

If you are unsure whether the original scripts are still available in git or as a backup, please confirm the location of the last known good commit or backup path; this determines whether to use the git checkout or the manual copy step.