Kaldi online2 decoding pipeline & nnet3 model pruning – optimal configuration for low‑traffic workloads?

Optimal Configuration for Low-Traffic Deployments

For low-traffic workloads on resource-constrained hardware (such as a t3.medium instance), the optimal configuration is to maintain the 30ms stride and prioritize weight-based pruning over stride reduction. Reducing the stride to 10ms typically increases CPU load by 2-3x without providing a proportional improvement in Word Error Rate (WER) for general speech tasks, which risks exhausting burstable CPU credits on t3 instances.

Likely Performance Trade-offs

While specific impacts vary by acoustic model, the following behaviors are generally observed in nnet3-based online2 pipelines:

• Stride (30ms vs 10ms): A 10ms stride increases the frequency of network evaluations. In low-traffic scenarios, this often leads to higher latency and CPU spikes without significant accuracy gains, as the underlying HMM/Chain state transitions are not typically sensitive enough to justify the 3x computational cost.
• Pruning Aggression: Using nnet3-copy --transform to remove low-magnitude weights can significantly reduce the memory footprint. However, aggressive pruning (removing >30% of weights) often leads to a non-linear spike in WER. For low-traffic use, moderate pruning combined with quantization is the safer path to reducing the memory footprint.
• Hardware Acceleration: On a t3.medium, CPU optimization is more cost-effective than GPU acceleration. The overhead of data transfer to a GPU often outweighs the inference speedup for sporadic, low-volume requests.

Implementation Steps for Resource Optimization

1. Baseline Benchmarking: Run the online2-wav-nnet3-latency.sh script on a small validation set using 30ms and 60ms strides to determine the minimum acceptable accuracy threshold for your specific audio.
2. Apply Model Pruning: Use the nnet3-copy utility to create a pruned version of the model. Start with conservative weight removal and verify the WER before increasing aggression.
3. Resource Monitoring: Deploy the pruned model with a 30ms stride and monitor memory residency and CPU credits using htop during simulated traffic to ensure the process stays within the 4GB RAM limit of a t3.medium.

Verification Command

To compare the memory footprint and inference speed of your original versus pruned model, use the following scoped approach:

# Compare model sizes
ls -lh nnet3.mdl nnet3_pruned.mdl

# Test inference latency on a sample wav
./online2-wav-nnet3-latency.sh --stride 30 --model nnet3_pruned.mdl sample.wav

Missing Diagnostic: To refine this recommendation, please provide the average utterance length of your workload. Very short utterances (under 2 seconds) may tolerate higher strides, whereas long-form audio requires tighter control to prevent decoding drift.