Choosing between Bash and Docker uploaders to mitigate connection pool exhaustion in concurrent CI

Answer to the Core Question

Bash‑curl shares the host’s curl connection pool. With more than ~10 concurrent uploads the default per‑host limit is hit, leading to connection failures and retries. Docker‑based uploaders run each upload in its own container, giving each instance an isolated curl process and its own connection pool. This scales linearly with the number of containers and prevents shared exhaustion.

Scalability for >50 Jobs

• Docker uploader is the safer choice to avoid exhausting file descriptors on the host.
• Bash can still be used if the CI environment limits concurrent uploads to ≤10 or if the host’s ulimit -n is set very high and you accept the risk of retries.

Docker Startup Latency

Typical lightweight images add 1–2 s of overhead per job. For uploads that run for several minutes this is negligible. If your CI job finishes in under 5 seconds, the overhead may become a noticeable fraction of total runtime.

Adopting a Workaround Now vs. Waiting

If you regularly see connection errors or the CI pipeline already saturates the host’s file‑descriptor limit, switch to the Docker uploader immediately. If your current pipeline runs fewer than 10 parallel uploads and you haven’t hit the connection‑pool ceiling, you can monitor the situation and wait for a configurable pool‑size feature.

What You Should Verify

• Check the host’s ulimit -n value to understand how many descriptors are available per process.
• Measure the average duration of your upload jobs to decide if the 1–2 s Docker start‑up is acceptable.

Once you have these numbers, you can make a definitive choice about whether the Docker approach is worth the added overhead for your particular workload.