Architecture Note: ASP.NET Core Health Checks for Liveness and Readiness

The Problem: Signal Conflation

Container orchestrators like Kubernetes or AWS ECS require two distinct signals to manage service lifecycles: Liveness (Is the process deadlocked or crashed?) and Readiness (Is the service capable of handling requests?).

A common engineering failure is conflating these two. If a readiness check (which verifies a database connection) is used as a liveness probe, a transient database outage will cause the orchestrator to restart every healthy application pod. This creates a "restart storm" that puts more pressure on the recovering database and increases downtime. The goal is a minimal design that separates these signals without introducing heavy custom middleware.

Smallest Suitable Design

The Microsoft.Extensions.Diagnostics.HealthChecks library provides the minimal framework for this. The design relies on registering IHealthCheck implementations and using tags to route them to different endpoints.

Implementation Pattern

In .NET 6+ minimal hosting, register your checks and map them to specific paths using a predicate filter based on tags.

// Program.cs
builder.Services.AddHealthChecks()
    // Liveness: No I/O, just confirms the process is running
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: new[] { "live" })
    // Readiness: Verifies critical dependencies
    .AddSqlServer(connectionString, name: "sql", tags: new[] { "ready" }, timeout: TimeSpan.FromSeconds(3));

// Liveness endpoint: Returns simple status for orchestrator
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("live"),
    ResponseWriter = async (ctx, report) => 
    {
        ctx.Response.ContentType = "application/json";
        await ctx.Response.WriteAsync($"{{\"status\":\"{report.Status}\"}} ");
    }
});

// Readiness endpoint: Returns detailed status for internal diagnostics
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready")
    // Default ResponseWriter provides detailed output
});

Design Logic

• Liveness (/health/live): Should never perform network I/O. If the process is running and the HTTP stack is responsive, it is "live."
• Readiness (/health/ready): Performs a lightweight check (e.g., SELECT 1). If this fails, the pod is removed from the load balancer but not restarted.

Trust and Data Boundaries

Health endpoints are a trust boundary. A default detailed response can leak sensitive architectural data, such as internal server names, database versions, or dependency topology, which aids reconnaissance during an attack.

Boundary Strategy:

• External/Orchestrator: Use a custom ResponseWriter to return only the status (Healthy/Unhealthy).
• Internal/Admin: Allow detailed JSON responses only for requests originating from a trusted monitoring namespace or requiring a specific internal header.

Operational Checks and Failure Modes

To verify the implementation, perform these four checks in a staging environment:

1. Liveness Isolation: Stop the database. /health/live must still return 200 OK.
2. Readiness Sensitivity: Stop the database. /health/ready must return 503 Service Unavailable.
3. Timeout Enforcement: Simulate a network hang (e.g., using a firewall drop). The readiness probe must return a failure within the 3-second timeout defined in AddSqlServer, rather than hanging until the orchestrator kills the request.
4. Leak Test: Curl the liveness endpoint and ensure no connection strings or server IPs are present in the body.

Failure Mode Analysis

Failure Mode	Cause	Mitigation
Cascading Restarts	Readiness check used as Liveness probe.	Strictly separate /live and /ready endpoints.
Probe-Induced Load	High-frequency readiness probes hitting DB.	Implement a short cache (e.g., 5-10s) within the IHealthCheck.
Zombie Pods	Liveness check is too shallow (doesn't detect deadlocks).	Include a check for a critical background worker's last-heartbeat timestamp.

Conditions That Change the Design

This pull-based, per-service design is the smallest suitable starting point, but it should be evolved when:

• Aggregated Visibility is Required: If you need a "single pane of glass" for 50+ services, do not build a custom aggregator. Move to a metrics-based approach using OpenTelemetry and Prometheus.
• Push-Based Alerting: Health checks are for orchestrator automation, not human alerting. For paging, implement push-based metrics that trigger on error-rate thresholds rather than binary status.
• Complex Dependency Graphs: If readiness depends on a sequence of services, implement a "deep health check" that is separate from the orchestrator's readiness probe to avoid circular dependency failures.