Apache 2.4 Reverse Proxy Architecture: Minimal Viable Design for TLS Termination and Backend Load Balancing

The Problem

You need a single Apache 2.4 instance at the network edge that terminates TLS, distributes traffic across two or more backend application servers, preserves the original client IP, enforces request size limits, and allows graceful backend draining during deployments — without adding a separate cache layer, WAF, or L4 load balancer.

Requirements Driving the Design

• Terminate TLS at the edge using certificates managed by the operations team.
• Load-balance across ≥2 backend HTTP servers in a private subnet.
• Forward the real client IP via X-Forwarded-For so backends can log and rate-limit correctly.
• Reject oversized requests before they reach the application (e.g., 10 MB body limit).
• Support draining: stop sending new requests to a backend while in-flight requests complete.

Smallest Suitable Design

A single Apache 2.4 instance running the event MPM with these modules loaded:

• mod_proxy, mod_proxy_http, mod_proxy_balancer
• mod_ssl for TLS termination
• mod_headers for header manipulation
• mod_slotmem_shm — required for shared balancer state across worker processes
• mod_status (with ExtendedStatus On) for observability

One <VirtualHost *:443> handles all inbound HTTPS. Backends speak plain HTTP, so SSLProxyEngine stays Off. The balancer uses lbmethod=byrequests (simple round-robin with request counting) and ProxyPreserveHost On so the original Host header reaches the application.

Minimal Working Configuration

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule headers_module modules/mod_headers.so
LoadModule status_module modules/mod_status.so

Listen 443

<VirtualHost *:443>
    ServerName edge.example.com
    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/edge.example.com.crt
    SSLCertificateKeyFile /etc/pki/tls/private/edge.example.com.key
    SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
    SSLCipherSuite HIGH:!aNULL:!MD5

    # Request size limit (10 MB)
    LimitRequestBody 10485760

    # Preserve original Host header for backend routing
    ProxyPreserveHost On

    # Inject real client IP
    RequestHeader set X-Forwarded-For "%{REMOTE_ADDR}s"

    # Balancer definition
    <Proxy balancer://app>
        BalancerMember http://10.0.1.10:8080 route=app1 hcinterval=10 hcfails=2
        BalancerMember http://10.0.1.11:8080 route=app2 hcinterval=10 hcfails=2
        ProxySet lbmethod=byrequests
    </Proxy>

    # Health check endpoint on backends
    ProxyHCExpr ok200 {%{REQUEST_STATUS} == 200}
    ProxyHCTemplate tcp5 hcinterval=10 hcfails=2 hcexpr=ok200
    ProxyHCTPsize 16

    ProxyPass / balancer://app/ retry=60 timeout=30
    ProxyPassReverse / balancer://app/

    # Runtime balancer management (loopback only)
    <Location /balancer-manager>
        SetHandler balancer-manager
        Require ip 127.0.0.1
    </Location>

    # Status for Prometheus scrape (loopback only)
    <Location /server-status>
        SetHandler server-status
        Require ip 127.0.0.1
        ExtendedStatus On
    </Location>

    ErrorDocument 503 /maintenance.html
    ErrorLog "|/usr/sbin/rotatelogs /var/log/httpd/error_log.%Y%m%d 86400"
    CustomLog "|/usr/sbin/rotatelogs /var/log/httpd/access_log.%Y%m%d 86400" \
        "%{%s}t %a \"%r\" %>s %O \"%{X-Forwarded-For}i\""
</VirtualHost>

Trust and Data Boundaries

Boundary	Control	Rationale
TLS private key	File mode 640, owned by root:www-data; readable only by Apache master process	Prevents unprivileged compromise from leaking the key
Backend network	Security group / iptables allows TCP 8080 only from Apache's DMZ IP	Backends never accept direct internet traffic
X-Forwarded-For header	Set by Apache via RequestHeader; backends configure known-proxies or equivalent to trust only Apache's IP	Prevents header spoofing from clients or compromised upstream
Client certificates	Not forwarded (SSLVerifyClient none)	Design assumes TLS termination only; mTLS would require separate design

Operational Checks

Runtime Balancer State

Access http://127.0.0.1/balancer-manager from the Apache host to see each member's status (Ok, Err, Drn for draining), elected load factor, and last health-check timestamp. Click Disable next to a member to initiate graceful drain — new requests stop routing there, but existing connections finish.

Prometheus Scrape Target

http://127.0.0.1/server-status?auto exposes BusyWorkers, IdleWorkers, TotalAccesses, TotalKBytes, and per-balancer counters. Scrape interval of 15–30 seconds balances visibility with overhead; at >50k RPS consider disabling ExtendedStatus or increasing the interval.

Correlation Logging

The CustomLog format includes epoch timestamp (%{%s}t), client IP (%a), request line, response status, bytes sent, and the injected X-Forwarded-For value. This lets you join Apache logs with backend access logs on request ID or timestamp/IP tuple.

Failure Modes and Mitigations

Backend Health Checks

Each BalancerMember declares hcinterval=10 hcfails=2 with a ProxyHCExpr that expects HTTP 200 on /health. After two consecutive failures (20 seconds), the member transitions to Err state and is excluded from routing. The ProxyHCTPsize 16 directive allocates a thread pool for health checks so they don't block request workers.

Connection and Read Timeouts

ProxyPass / balancer://app/ retry=60 timeout=30 sets a 30-second timeout for both connect and read phases, and a 60-second cooldown before retrying a failed member. Long-running backend endpoints (e.g., report generation) may need a per-location override:

<Location /reports>
    ProxyPass balancer://app/ timeout=300
</Location>

Full Backend Outage

When all members are in Err state, the balancer returns 503. The ErrorDocument 503 /maintenance.html serves a static page from the Apache docroot. Verify this path exists and is readable by the www-data user.

TLS Certificate Expiry

Monitor with a daily cron:

0 6 * * * root openssl x509 -checkend 2592000 -noout -in /etc/pki/tls/certs/edge.example.com.crt || \
    /usr/local/bin/alert-cert-expiry.sh edge.example.com

Replace alert-cert-expiry.sh with your paging/notification mechanism.

Conditions That Change the Design

Trigger	Design Change	Why the Minimal Design Breaks
Sustained >10k RPS	Add second Apache instance behind an L4 load balancer (NLB/HAProxy); enable stickysession=ROUTEID in balancer config	Single Apache CPU saturates at TLS handshake + proxy overhead; sticky sessions preserve affinity for stateful apps
WAF requirement	Insert mod_security before mod_proxy in module load order	Inspection must happen before request forwarding; rule tuning adds operational complexity
Mutual TLS mandated	Enable SSLVerifyClient require, forward SSL_CLIENT_CERT header to backends	Requires client CA bundle, revocation checking, and backend header parsing
Backend gRPC services	Switch to mod_proxy_h2 / mod_proxy_h2c with ProxyPass / h2c://backend/	HTTP/1.1 proxying breaks gRPC framing and trailers
FIPS 140-2 compliance	Rebuild Apache with --enable-fips against OpenSSL 3 FIPS provider	Standard builds use non-validated crypto modules

Critical Cautions

• Missing mod_slotmem_shm causes silent fallback to per-process balancing in prefork MPM — members appear balanced but each worker maintains its own counter, defeating distribution. Verify with httpd -M | grep slotmem.
• ProxyPreserveHost On sends the original Host header. Backends must either trust it (configure ServerName accordingly) or rely on X-Forwarded-Host instead.
• Default ProxyTimeout applies to both connect and read. Long-running requests need per-location overrides as shown above.
• Unequal backend capacity requires explicit loadfactor on BalancerMember (default is 1). A 2:1 capacity ratio means loadfactor=2 on the larger host.
• Apache 2.4.57+ changed default MergeTrailers behavior. If backends emit chunked responses with trailers, test header merging or set MergeTrailers Off explicitly.

Verification Checklist

1. Confirm modules loaded:

   httpd -M | grep -E 'proxy|ssl|status|slotmem'

   Expected: proxy_module, proxy_http_module, proxy_balancer_module, slotmem_shm_module, ssl_module, headers_module, status_module.
2. Test health endpoint from Apache host:

   curl -v http://10.0.1.10:8080/health

   Should return 200 with minimal body.
3. Verify header injection:

   curl -v https://edge.example.com/ 2>&1 | grep -i x-forwarded-for

   Then check backend access log for the same IP.
4. Simulate backend failure: stop one backend service, wait hcinterval * hcfails seconds (20s), then confirm /balancer-manager shows Err and requests route to the remaining member.
5. Validate TLS configuration:

   openssl s_client -connect edge.example.com:443 -servername edge.example.com </dev/null

   Verify certificate chain, TLS 1.2/1.3 only, no weak ciphers.
6. Load test (adjust threads/connections to your scale):

   wrk -t4 -c200 -d60s https://edge.example.com/

   Monitor /server-status?auto for BusyWorkers approaching MaxRequestWorkers; tune ServerLimit/ThreadsPerChild if saturated.
7. Full outage test: stop all backends, request https://edge.example.com/, confirm 503 with custom maintenance page, and check error log for proxy: BALANCER: (balancer://app). All workers are in error state.

Limitations

• This design assumes HTTP backends. HTTPS backends require SSLProxyEngine On and certificate validation configuration.
• No request/response body inspection — cannot block malicious payloads or PII leakage without mod_security.
• Single point of failure at the Apache instance. High availability requires the L4 load balancer tier described in the design-change table.
• Health checks only verify HTTP 200 on /health. Application-level readiness (DB connectivity, queue depth) must be implemented in that endpoint.

Diagram Labels

• Client → Apache (TLS 1.2/1.3)
• Apache → Backend (HTTP, X-Forwarded-For)
• Balancer state (mod_slotmem_shm)
• Health check (/health, 10s interval)