Diagnosing and Fixing Common GPU‑Accelerated Kaldi nnet3 Training Issues

Problem Statement

When training Kaldi’s nnet3 models on a GPU, many teams hit stalls, out‑of‑memory (OOM) errors, high loss, crashes, or inconsistent results. A systematic diagnostic flow can save hours of trial‑and‑error.

Recognizable Symptoms & Quick Cause Table

Ordered Diagnostic Checklist

1. Validate Feature Dimensionality

   Run a quick check on the feature matrix files used by the training script:

   #!/usr/bin/env bash
   set -e
   feat_dir=<your-feat-dir>
   # Show first 5 feature lines
   head -n 5 $feat_dir/feats.scp | awk '{print $2}' | while read f; do
     # Extract dimension using Kaldi's compute-statistics
     dim=$(feat-to-dim $f)
     echo "$f has dim $dim"
   done
   

   Replace <your-feat-dir> with the path to the features. If any line shows a different dimension than expected (e.g., 40 vs. 80), adjust the feature extraction pipeline or remove corrupted files.

2. Check GPU Memory Fit

   Before launching training, estimate memory usage:

   # example: estimate memory for 128 frames per utterance, 40-dim, batch size 256
   frames=128
   dim=40
   batch=256
   # Rough per‑GPU memory (bytes) = frames * dim * 4 bytes * batch
   mem=$(echo "scale=0; $frames*$dim*4*$batch/1024/1024/1024" | bc)
   echo "Estimated per‑GPU RAM: ${mem}GB"
   

   If the estimate exceeds your GPU’s free memory (check with nvidia-smi --query-gpu=memory.free --format=csv,noheader), reduce --batch-size or --frames-per-utterance in the config.

3. Verify Learning‑Rate and Optimizer Settings

   Open your train.conf and confirm learning-rate and momentum are within sane ranges (e.g., 0.01–0.1). Use a smaller rate if loss plateaus:

   learning-rate=0.01
   # or try
   learning-rate=0.001
   

4. Confirm Data‑Augmentation Consistency

   Data‑augmentation scripts (e.g., speed perturbation) can change feature dimensionality. Ensure all stages of the pipeline use the same --feature-transform path. Running steps/check_augmentation.sh (if available) will flag mismatches.

5. Match CUDA Driver and Kaldi Build

   Check the driver:

   nvidia-smi | head -n 1
   # Should show a version that matches the CUDA toolkit used to compile Kaldi

   If mismatched, rebuild Kaldi:

   ./configure --cuda-path=/usr/local/cuda-12.0 --cudnn-dir=/usr/local/cudnn-8.0
   make -j$(nproc)
   

6. Seed Initialization for Determinism

   Add a fixed seed to your training script:

   export KALDI_TRAINING_SEED=12345
   nnet3-train --config=train.conf ...

   Re‑run to confirm identical loss curves.

Corrective Actions & Fixes

• Dimension Mismatch – Re‑run feature extraction with the correct dimensionality flag (e.g., --spk2utt or --apply-cmvn) and re‑index the scp files.
• GPU OOM – Reduce minibatch, increase --gpu-memory-limit, or switch to a larger GPU. If using multiple GPUs, enable --num-gpus=2 and balance the load.
• High Loss – Lower learning rate, double‑check alignment files, or disable aggressive data augmentation temporarily to isolate the issue.
• Crashes – Rebuild Kaldi with the exact CUDA/driver headers; verify binary integrity with md5sum checksums if available.
• Inconsistent Runs – Enable deterministic CUDA ops by setting export CUDA_LAUNCH_BLOCKING=1 and export KALDI_TRAINING_SEED=42.

Escalation Criteria

Escalate to the system admin or GPU vendor support if:

1. GPU OOM persists after batch size reduction and memory limit adjustments.
2. Segmentation faults occur even after re‑building Kaldi with matching CUDA headers.
3. Loss remains high for >10 epochs despite learning‑rate tuning and data‑augmentation rollback.

Provide logs, nvidia-smi output, and the exact train.conf when escalating.

Practical Verification Steps

After applying a fix, run a minimal training job on a subset of data:

nnet3-train --config=mini_train.conf --minibatch-size=32 
   --feature-transform=transform.mat 
   --train-opts="--num-threads=1 --num-epochs=2"

Check that the script completes, loss decreases, and GPU memory stays below the set limit. If the job finishes successfully, the issue is likely resolved.