CPU vs GPU Decoding for Low‑Traffic Kaldi Workloads: Cost Trade‑off

Goal

Minimise total hosting cost while maintaining acceptable ASR accuracy for a sporadic, low‑traffic workload.

Constraints & Uncertainty

GPU instances carry a fixed hourly charge that is wasted when traffic is idle, whereas CPU decoding scales linearly with the number of active requests. Compact nnet3 chain models reduce CPU load but may degrade WER, especially when the language model is pruned for size. There is no built‑in guidance in Kaldi for determining the traffic threshold at which GPU decoding becomes cheaper than CPU decoding.

Specific Questions

• At what average request rate does GPU decoding become more cost‑effective than CPU decoding with a small chain model?
• How does pruning the HCLG graph affect WER when using a reduced‑frame‑rate chain model on CPU?
• What beam and lattice‑beam settings provide the best latency‑cost balance for a single‑core CPU instance?