Handle Flaky APIs in Ballerina with Declarative HTTP Client Retry

The Problem: Unreliable External Services

Microservices often call third‑party APIs that can return transient errors like 5xx or timeouts. A naïve approach is to wrap every call in a loop, but that scatters retry logic and makes the code hard to maintain.

Declarative Retries with @retry

Ballerina 2.0 introduces a @retry annotation that can be attached to an http:Client constructor. The annotation lets you specify the number of attempts, the initial delay, the back‑off strategy, and which error conditions should trigger a retry—all without writing boilerplate code.

import ballerina/http;

@http:ClientConfig {
    retry: @retry {
        attempts: 3,
        delay: 1s,
        backoff: exponential,
        on: {statusCode: 502, 503}
    }
}
service /client {
    @http:Client {baseUrl: "http://example.com"} var client = new;

    function getData() returns string|error {
        var response = client->get("/data");
        return response.payload.toString();
    }
}

In this example the client will automatically retry up to three times if the upstream returns a 502 or 503. The retry logic is part of the runtime, so it adds almost no overhead compared to a manual loop.

Combining Retries with Circuit Breakers

Retries are great for transient network glitches, but if the downstream service is persistently failing you may want to stop hammering it. Ballerina lets you stack the retry policy on top of a circuit‑breaker using the @circuitBreaker annotation.

@circuitBreaker {
    threshold: 5,
    timeout: 10s
}
@http:ClientConfig {
    retry: @retry {attempts: 3, delay: 1s}
}
service /client {
    @http:Client {baseUrl: "http://example.com"} var client = new;
    // ...
}

The circuit‑breaker opens after five consecutive failures, immediately returning an error for new requests while the breaker stays open for ten seconds. Once closed, normal retry logic resumes.

Verification & Practical Tips

1. Run the example. Install Ballerina 2.0+, then execute bal run retry_example.bal. Observe console logs showing retry attempts.
2. Mock the upstream. Spin up a local HTTP server that returns 502 for the first three requests and 200 thereafter. Verify that the client retries exactly three times before succeeding.
3. Check dependencies. Inspect bal.lock to ensure the ballerina/http module version supports @retry (>=2.0.0).
4. Measure latency. Time the request with and without the retry policy to quantify the added delay.
5. Validate idempotency. Only retry if the called API is safe to repeat; otherwise, duplicate side‑effects may occur.

Trade‑offs & When Not to Retry

Latency. Each retry adds a delay. If the upstream service is simply slow but healthy, retries can unnecessarily inflate response times.

Side‑effects. Retrying non‑idempotent operations (e.g., POST that creates resources) can lead to duplicates. Always confirm the operation is safe.

Circuit‑breaker tuning. Setting the threshold too low may block legitimate traffic during brief spikes; too high and you may keep hammering a failing service.

Take‑away

Ballerina’s declarative @retry annotation turns a noisy, error‑prone loop into clean, reusable code. Pair it with a circuit‑breaker for maximum resilience, but keep an eye on latency, idempotency, and threshold tuning. Test against a mock server to confirm the policy behaves as expected before deploying to production.