Which PyTorch quantization strategy minimizes memory overhead for low-traffic CPU inference?

Reducing operational costs for low-traffic workloads often involves migrating from GPU instances to CPU-based environments. To minimize the memory footprint and hardware requirements, PyTorch provides several model compression techniques, including pruning and quantization.

When targeting CPU inference, there is a trade-off between the simplicity of Dynamic Quantization—where weights are quantized offline and activations are quantized during runtime—and the potential performance gains of Static Quantization, which requires a calibration dataset to determine activation ranges.

For a deployment where latency requirements are flexible but memory costs must be minimized, it is unclear which approach provides the most efficient balance of memory reduction and accuracy preservation without extensive fine-tuning.

• Does Dynamic Quantization provide sufficient memory savings compared to Static Quantization for low-traffic CPU workloads?
• What is the expected impact on model accuracy when switching from FP32 to INT8 quantization in a CPU-only environment?