Which Kaldi deployment boundary fits zero-downtime model migration for a small ASR service?

Recommended boundary

Use separate old and new decoder pools with traffic shifting. Do not hot-swap model artifacts inside a single running Kaldi process. Kaldi has no built-in zero-downtime migration protocol, and replacing final.mdl, HCLG.fst, or feature configuration while decoders are active can race with in-flight decodes, crash the process, or emit silent wrong transcripts.

Confirmed facts

Kaldi provides no built-in zero-downtime migration protocol. Online2 endpointing state is per-decoder and does not migrate between processes. In-place artifact replacement inside a running process is not officially supported and can race with active decoding threads.

Likely explanation

For a small service, separate pools are preferable because the roughly doubled model memory during cutover is usually acceptable, while isolation simplifies rollback and avoids coupling old and new versions in one address space. Hot-swap saves memory but couples versions in the same address space and risks corrupting in-flight decodes.

Artifacts to version together

• Feature configuration: MFCC or fbank settings plus CMVN
• Acoustic model: final.mdl
• Decoding graph: HCLG.fst and its symbol table
• Decoding and endpointing options, including online2 configuration when used

Changing any one piece alone can cause frame-rate or phone-set mismatches that produce plausible transcripts with regressed accuracy. Treat the bundle as one immutable migration unit tied to the pinned Kaldi revision.

Steps for this case

1. Build the new pool from the same pinned Kaldi revision and load the complete new artifact bundle.
2. Run a smoke decode of known utterances through the new pool and compare word error rate against the old pool before shifting production traffic.
3. Shift a small canary percentage of traffic to the new pool while the old pool handles the rest.
4. Monitor decode success rate, latency, and endpointing errors on the new pool.
5. Increase new-pool traffic only after it passes a success-rate threshold for several minutes.
6. Drain the old pool: stop sending new requests, let in-flight utterances reach endpointing or finish, then shut it down. Keep it warm until the new pool is stable.

Readiness and draining

Readiness is decided by the smoke decode and canary success rate, not by file timestamps. Draining is decided by in-flight utterance completion: Kaldi cannot transfer a partially decoded online2 session to a new process, so connections must complete or time out on the old pool.

Missing diagnostic detail

One missing detail that can change the recommendation: the host's available RAM during cutover relative to peak concurrent sessions. Two pools can roughly double model memory; if that does not fit, resize the host or cut over during a low-traffic window instead of choosing in-process hot-swap.

Version assumption: this guidance assumes a pinned Kaldi revision whose decoder thread-safety and online2 endpointing semantics are fixed. Verify those properties in that revision's source or build configuration before production use.