Configuring concurrent packfile generation in Git smart HTTP server

Answer to the core questions

• Configuration name and default: Git itself has no dedicated key for concurrent packfile generation. The practical knob is the number of git-http-backend worker processes exposed by the HTTP server (e.g., worker_processes in Nginx, MaxRequestWorkers in Apache, or a CGI queue size). The default is usually “unlimited” – every request spawns a new process.
• Behavior when the limit is exceeded: The recommended approach is to let the HTTP server queue excess requests. Rejecting with a 503 is rarely helpful because Git clients retry automatically and a 503 can cause confusing timeouts. Queueing keeps latency predictable and preserves fairness.
• Metrics to document for tuning:
  • CPU cores and available RAM per worker – pack.threads runs one thread per worker, so memory scales with that.
  • Average RSS of a single git-http-backend process during a packfile generation.
  • System load average and I/O wait when the worker count is increased.
  • Repository size and depth – large repos can hit the 4‑thread default quickly.

Confirmed facts from current Git releases

• git-http-backend spawns a separate receive-pack or upload-pack process per HTTP request.
• The only intra‑pack parallelism Git offers is pack.threads (default = number of online CPUs). It does not limit the number of concurrent packfile operations.
• Git itself provides no queueing or rate‑limiting mechanism for smart‑HTTP requests.

Recommended steps for your environment

1. Choose an HTTP server that supports worker limits. For Nginx, use the worker_processes and worker_connections directives. For Apache, set MaxRequestWorkers.
2. Configure a fixed worker pool for git-http-backend.

   # Nginx example
   worker_processes 4;
   events { worker_connections 1024; }
   
   http {
     upstream git_backend {
       server unix:/var/run/git-http-backend.sock;
       keepalive 16;
     }
     server {
       listen 80;
       location /git/ {
         proxy_pass http://git_backend;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
       }
     }
   }
   

3. Set pack.threads to a conservative value.

   git config --global pack.threads 2
   

   This limits per‑worker memory usage while still allowing some parallel delta compression.
4. Monitor and iterate. Use top, htop, or system‑dstat to watch RSS, CPU, and I/O. Adjust the worker count until latency remains stable under expected load.

Missing diagnostic detail

To refine the worker count recommendation, could you tell me which HTTP server or CGI wrapper you are using to host git-http-backend?