Architecting Resilient JSON REST Endpoints with Ballerina

The Problem: Transient Failures in Microservices

When exposing a REST endpoint that depends on downstream APIs or databases, transient network glitches or temporary service unavailability often cause request failures. Handling these failures manually with nested try-catch loops and sleep timers leads to verbose, unmaintainable code. The goal is to implement a resilient JSON endpoint that handles transient errors automatically while providing operational visibility for orchestration tools.

Minimum Viable Design

The smallest suitable design utilizes Ballerina's native service keyword to define an HTTP listener and the @retry annotation to handle failure recovery. This approach removes the need for external circuit-breaker libraries for basic resilience.

import ballerina/http;
import ballerina/log;

service /api on new http:Listener(9090) {
    // Health check for orchestration (e.g., Kubernetes)
    resource function get health() returns json {
        return {status: "ok"};
    }

    // Resilient data endpoint
    @http:ServiceConfig {
        // Configuration for the service instance
    }
    @retry(maxRetries = 3, backoff = 1000)
    resource function get data() returns json|error {
        log:print("Fetching data from downstream...");
        // Simulate a call to a downstream system
        return this.fetchDownstreamData();
    }

    function fetchDownstreamData() returns json|error {
        // Logic that may throw a transient error
        return {id: 1, value: "Sample Data"};
    }
}

Trust and Data Boundaries

In this architecture, the Trust Boundary is the http:Listener. All incoming data is untrusted. Ballerina enforces this boundary through compile-time type checking. By defining the return type as json, the runtime ensures that the payload is valid JSON before it leaves the service.

• Input Boundary: The HTTP request headers and body are validated against the resource function signature.
• Output Boundary: The json return type prevents the accidental leakage of internal object structures or raw error stacks to the client.

Operational Checks and Verification

To verify the resilience and availability of the service, use the following diagnostic steps. These commands assume the service is running on localhost:9090.

Check	Command (Run from Terminal)	Expected Result
Availability	curl http://localhost:9090/health	{"status":"ok"}
Data Retrieval	curl http://localhost:9090/api/data	Valid JSON payload
Retry Logic	Simulate downstream error in fetchDownstreamData	Console logs showing 3 retry attempts before failure

Failure Modes

Despite the retry logic, several failure modes remain that require specific handling:

• Persistent Downstream Failure: If the downstream service is down longer than the total backoff period (3 retries $\times$ 1000ms), the resource will eventually return an error. This should be mapped to an HTTP 503 (Service Unavailable).
• Resource Exhaustion: High volumes of retries during a system-wide outage can lead to "retry storms," where the service overwhelms the downstream system as it tries to recover.
• Port Conflicts: If port 9090 is occupied, the service will fail to start. Check for existing processes using netstat -ano | findstr :9090 (Windows) or lsof -i :9090 (Unix).

Design Evolution: When to Change the Architecture

The current design is suitable for low-to-medium traffic services with simple dependencies. You should evolve this architecture if the following conditions are met:

1. Cascading Failures: If retries are causing downstream systems to crash, replace the @retry annotation with a full Circuit Breaker pattern to fail fast without attempting requests.
2. Dynamic Scaling: If the service is deployed in a dynamic cloud environment, replace the hardcoded port 9090 with an environment variable or a service discovery mechanism.
3. Complex Payloads: If JSON validation becomes complex, move from the generic json type to record types to enforce strict schema validation at the boundary.

Rollback Procedure: To revert changes to the retry logic or port configuration, stop the process (Ctrl+C), modify the service.bal file, and restart using bal run service.bal.