Start-first update order limits in Docker Swarm rolling updates
0 reputation · 24 Jul 2023, 16:56 UTC
0 reputation · 24 Jul 2023, 16:56 UTC
Migrating a small application in Docker Swarm without downtime relies on the documented rolling update capability of docker service update. The default stop-first order terminates the old task before starting its replacement, which causes a brief outage. Switching to --update-order start-first is intended to avoid that gap by starting the new task before the old one stops.
However, the overlap window introduces operational uncertainty. While both tasks are running, the routing mesh may send traffic to the old and new versions concurrently, raising unresolved questions about published-port conflicts and mixed-version responses. The start-first strategy also assumes sufficient free capacity on the target nodes; if resources are exhausted, the replacement task cannot start and the update stalls. In addition, Docker does not drain in-flight connections at the proxy layer, so graceful shutdown depends on the application handling SIGTERM within the --stop-grace-period. Flag names and defaults are version-sensitive and should be confirmed against the installed Docker Engine release.
What practical limits should be considered when relying on start-first for a zero-downtime migration? How does the routing mesh behave when old and new tasks share a published port during the overlap? What resource and health-check constraints must be satisfied before attempting this update order in production?
34275 reputation · 24 Jul 2023, 20:01 UTC
Switching to --update-order start-first in docker service update replaces the default stop-first sequence by launching the replacement task before the existing task stops. This overlap window is intended to eliminate the brief service outage, but it introduces several operational conditions that must be evaluated before relying on it for zero-downtime migration.
During the overlap, both the old and new tasks are registered with the Swarm routing mesh on the same published port. The mesh distributes incoming requests across all healthy tasks, which means traffic can be split between the two versions concurrently. If the service is stateless and the two versions are fully compatible, this split occurs transparently. If the service maintains per-request state, depends on a shared backend, or returns version-specific responses, clients may receive mixed results. Docker does not automatically drain in-flight connections at the proxy layer; graceful shutdown still depends on the application responding to SIGTERM within the --stop-grace-period window.
The start-first order assumes each node running the service has sufficient free CPU and memory to host both the existing task and the replacement task simultaneously. If a node's resource headroom is limited, the new task may fail to start, causing the update to stall or roll back. Before initiating, verify that each target node has spare capacity roughly equal to the service's resource allocation, or use node placement constraints and --max-same-snapshot to distribute the workload across the cluster.
The update proceeds past the overlap only when the new task reports HealthStatus: PASS. If health checks are slow, flaky, or misconfigured, the old task may be drained prematurely, breaking the zero-downtime guarantee. Additionally, the --stop-grace-period value must give the application enough time to close existing connections. If the process exits immediately on SIGTERM, in-flight requests are dropped regardless of the start-first ordering.
Flag compatibility is engine-version dependent; releases before 18.09 do not support the --update-order flag and always apply stop-first ordering. Confirm the installed engine version before planning the update strategy.
docker service ps <service> and confirm that two tasks show RUNNING state during the overlap.docker service inspect <service> to verify the new task's HealthStatus is PASS before the old task is removed.docker node ls and docker stats before and during the update to ensure no node is resource-starved.To tailor the recommendation for your environment, what is the current Docker Engine version and the average CPU/memory headroom per node? This detail determines whether start-first can be safely applied or whether a blue-green or staged rollout is preferable.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.