Implementing Graceful Shutdown for Go HTTP Servers

The Problem: Abrupt Process Termination

When a Go HTTP server is terminated abruptly (e.g., during a Kubernetes pod rotation or a manual Ctrl+C), any requests currently being processed are severed immediately. This results in 500-series errors for the client and potentially corrupted data if the server was mid-write to a database. The goal is to stop accepting new connections while allowing active requests a grace period to complete.

Decision Matrix: Shutdown Strategies

The choice of shutdown mechanism depends on your Go version and the complexity of your request lifecycle.

Method	Mechanism	Best Use Case
http.Server.Shutdown()	Closes listeners; waits for active connections to close or context to expire.	Standard Go 1.13+ applications; handles HTTP/2 draining automatically.
Manual Listener Close	Calls listener.Close() and uses a sync.WaitGroup to track requests.	Legacy Go versions (< 1.13) or non-standard network listeners.
errgroup Coordination	Wraps the server and workers in a group that cancels on the first error/signal.	Complex services where the server is one of many concurrent dependencies.

Engineering Trade-offs

• Context Timeout: Using Server.Shutdown(ctx) requires a context with a timeout. If the timeout is too short, you kill active requests; if too long, your deployment pipeline may time out and force-kill the process (SIGKILL).
• Goroutine Leakage: Server.Shutdown only waits for the HTTP handlers to return. If your handler starts a background goroutine that doesn't monitor r.Context().Done(), that goroutine will persist and may prevent the process from exiting cleanly.
• Connection Draining: Server.Shutdown handles the nuances of HTTP/2 (sending GOAWAY frames), which is significantly harder to implement manually.

Concrete Implementation

This implementation uses os/signal to capture termination requests and http.Server.Shutdown to drain connections. Run this in a file named main.go.

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/work", func(w http.ResponseWriter, r *http.Request) {
		// Simulate a long-running task that respects cancellation
		select {
		case <-time.After(10 * time.Second):
			w.Write([]byte("Work completed\n"))
		case <-r.Context().Done():
			log.Println("Request cancelled by client or server shutdown")
			return
		}
	})

	srv := &http.Server{
		Addr:    ":8080",
		Handler: mux,
	}

	// Create channel to listen for OS signals
	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)

	go func() {
		<-sigChan
		log.Println("Shutdown signal received...")

		// Give the server 30 seconds to finish active requests
		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
		defer cancel()

		if err := srv.Shutdown(ctx); err != nil {
			log.Printf("Shutdown error: %v", err)
			_ = srv.Close() // Force close if graceful shutdown fails
		}
	}()

	log.Println("Server starting on :8080")
	if err := srv.ListenAndServe(); err != http.ErrServerClosed {
		log.Fatalf("ListenAndServe error: %v", err)
	}
	log.Println("Server exited cleanly")
}

Validation Steps

1. Run the server: Execute go run main.go.
2. Initiate a request: Run curl http://localhost:8080/work. This request takes 10 seconds.
3. Trigger shutdown: While the curl is pending, press Ctrl+C in the server terminal.
4. Verify result: The curl request should still complete successfully (returning "Work completed"), while the server logs "Server exited cleanly".
5. Verify rejection: Attempt a second curl immediately after the signal; it should be refused as the listener is closed.

Limitations and Verification

• Zombie Goroutines: If you use go func() { ... }() inside a handler without passing the request context, Server.Shutdown will not wait for them.
• TCP Keep-Alive: Some idle keep-alive connections may linger until the timeout expires.
• Diagnostic Check: To verify connection draining in HTTP/2, run the server with GODEBUG=http2debug=1. You will see the server sending GOAWAY frames to clients during the shutdown sequence.

Rollback

Since this operation changes the process lifecycle management, to revert to immediate termination, remove the signal.Notify block and the srv.Shutdown call, allowing the program to exit immediately upon receiving a signal.