CPU vs GPU Decoding for Low‑Traffic Kaldi Workloads: Cost Trade‑off

Cost-Effectiveness Threshold

For sporadic, low-traffic workloads, CPU decoding is almost always more cost-effective. Because GPU instances typically incur a high fixed hourly cost regardless of utilization, and Kaldi's GPU decoding benefits primarily from high-concurrency batching, the "break-even" point occurs only when your request rate is high enough to keep the GPU saturated.

To calculate your specific break-even point, use this formula:

Utilization Threshold = (GPU Hourly Cost) / (CPU Hourly Cost × Concurrent Streams per Core)

If your average utilization is below this threshold, the idle cost of the GPU outweighs the efficiency gains in decoding speed.

Performance and Accuracy Trade-offs

HCLG Pruning and WER

Pruning the HCLG graph (via a smaller language model or tighter pruning during graph construction) reduces the memory footprint and CPU cycles required for decoding. However, this often increases the Word Error Rate (WER), particularly for out-of-domain speech or rare vocabulary. When using reduced-frame-rate chain models, the acoustic model is already more efficient; further pruning the HCLG graph can lead to a "double hit" where the decoder lacks the linguistic constraints to recover from acoustic ambiguities.

Single-Core CPU Optimization

To balance latency and cost on a single-core instance, start with the following beam settings for chain models and tune based on your specific domain:

• Beam: 13–15
• Lattice-beam: 6–8

Lowering these values reduces CPU load and latency but increases the risk of pruning the correct hypothesis, thereby increasing WER.

Verification Steps

Since performance varies by model size and audio characteristics, follow these steps to validate your configuration:

1. Benchmark RTF: Measure the Real-Time Factor (RTF) of your compact chain model on a single CPU core using a representative sample of your audio.
2. Grid Search: Run a small decode grid over the beam (13–16) and lattice-beam (6–8) settings on a labeled held-out set to find the lowest cost settings that meet your WER target.
3. Pruning Comparison: Build HCLG graphs at three different pruning levels and compare the resulting WER and memory usage to identify the point of diminishing returns.

Missing Diagnostic: To provide a precise cost recommendation, what is the average duration of your audio requests and the specific cloud instance pricing you are comparing?