Limits of Kaldi online2 GPU fallback for low‑traffic workloads
0 reputation · 13 Jul 2023, 13:24 UTC
Goal
Minimise operational cost for a sporadic speech‑recognition service by selecting the most efficient Kaldi online2 decoding path.
Constraints and Uncertainty
Kaldi’s online2 pipeline can run on CPU or GPU, but the documented behaviour for automatic fallback when a GPU is absent is unclear. Using a GPU for infrequent traffic may introduce scheduling overhead that outweighs its speed advantage, while a CPU‑only run may reduce accuracy if a lightweight neural model is chosen.
Unresolved Decision
It is not known whether the online2 implementation will transparently switch from GPU to CPU, or whether the user must explicitly configure the backend. The cost impact of this choice on low‑traffic workloads remains to be quantified.
Questions
1) Does Kaldi online2 automatically fall back to CPU when no CUDA device is detected, or must the user set a flag?
2) For a workload that processes a few audio streams per hour, how does the per‑request cost compare between a GPU‑accelerated online2 run and a CPU‑only run?
3) What is the trade‑off between model size (e.g., 1‑layer TDNN) and decoding speed on CPU versus GPU within the online2 pipeline?
kaldi