Choosing Between Ballerina Streaming and In‑Memory File Handling for Large Files

Decision: Use Streaming When File Size Exceeds a Few Megabytes

When building a Ballerina service that accepts or produces large files, the core decision is whether to stream the data or load it entirely into memory. The choice hinges on file size, memory constraints, downstream expectations, and the complexity you’re willing to manage.

Constraints to Consider

• File size threshold: Large files are typically >10 MB. For <10 MB, the memory overhead is negligible.
• Runtime memory budget: If the JVM heap is limited (e.g., <512 MB), streaming is safer for >1 GB files.
• Downstream service: Does the consumer need the entire payload at once? If so, streaming may still require buffering at the receiver.
• Network reliability: Unreliable links benefit from resumable chunked uploads.
• Ballerina version: Streaming APIs are stable from 1.2+. Older releases lack the stream type and related helpers.

Option Comparison Table

Feature	Streaming (stream type)	In‑Memory (byte[])
Peak Memory Usage	O(1) – only a small buffer	O(n) – entire file in heap
Complexity	Higher – async, backpressure, error propagation	Low – simple read/write
Error Handling	Propagate via error return; partial data may remain	Single exception; easier cleanup
Performance (throughput)	Potentially higher for very large files; depends on buffer size	Higher for small files; GC overhead for large files
Resumable Uploads	Supported via chunked transfer encoding	No inherent support
Downstream Buffering	May be required if downstream expects full payload	Not an issue
GC Pressure	Minimal; small buffers only	High for large files
Backpressure Handling	Built‑in; can pause upstream on slow consumer	None; data is already in memory
Implementation Effort	~50% more lines of code + async handling	~10% lines of code
Testing & Validation	Requires monitoring heap and simulating failures	Simple unit tests suffice

Trade‑Offs Explained

Memory vs. Simplicity – Streaming keeps heap usage low, which is critical for services that run in containers with strict memory limits. However, it forces you to write asynchronous code, manage backpressure, and implement robust error handling. For small payloads, the simplicity of reading into a byte[] often outweighs the memory benefit.

Backpressure vs. Latency – The stream API automatically pauses the producer when the consumer is slow, preventing buffer overrun. This can introduce latency if the consumer is a slow disk or network endpoint. In contrast, loading into memory can deliver data faster to a fast consumer but risks OOM if the consumer stalls.

Resumable Transfers – Streaming with chunked transfer encoding allows the client to resume uploads after a network hiccup. In‑memory uploads cannot resume; a failure requires the entire file to be re‑sent.

Error Propagation – Streams propagate errors via the error return type. If a failure occurs mid‑stream, the downstream may have partial data. You must decide whether to delete the partial file or keep it for debugging.

Concrete Implementation: Streaming File Upload

Below is a minimal Ballerina service that accepts a file via HTTP POST, streams it to a local file, and logs stream events. Replace /tmp/uploaded.bin with your desired destination.

import ballerina/io;
import ballerina/http;
import ballerina/log;

service /upload on new http:Listener(8080) {
    resource function post file(http:Request req) returns error? {
        // Destination file path
        string destPath = "/tmp/uploaded.bin";
        // Open a file output stream
        var fos = new io:FileOutputStream(destPath, true);
        if (fos is error) {
            log:printError("Failed to open file output stream", err = fos);
            return fos;
        }
        // Stream the request body directly to the file
        var result = req.getBody().pipe(fos);
        if (result is error) {
            log:printError("Streaming failed", err = result);
            // Optionally delete partial file
            _ = new io:File(destPath).delete();
            return result;
        }
        log:printInfo("File uploaded successfully to " + destPath);
        return;
    }
}

Key points:

• req.getBody() returns a stream of byte[] chunks.
• The pipe helper consumes the stream and writes to the output stream.
• All operations are asynchronous; the service returns immediately after the stream is drained.
• Error handling propagates via the error return.

Concrete Implementation: In‑Memory Upload (For Small Files)

For files <10 MB, the following simpler approach reads the entire payload into a byte[] and writes it to disk.

import ballerina/io;
import ballerina/http;
import ballerina/log;

service /upload on new http:Listener(8080) {
    resource function post file(http:Request req) returns error? {
        var payload = req.getBodyAsBytes();
        if (payload is error) {
            log:printError("Failed to read request body", err = payload);
            return payload;
        }
        string destPath = "/tmp/uploaded.bin";
        var result = io:fileWriteBytes(destPath, payload);
        if (result is error) {
            log:printError("Failed to write file", err = result);
            return result;
        }
        log:printInfo("File uploaded successfully to " + destPath);
        return;
    }
}

Notice the absence of asynchronous stream handling and the direct use of getBodyAsBytes.

Validation & Monitoring

1. Compile & Run
   ```bash bal run upload_service.bal ```
2. Send a 5 GB test file via curl
   ```bash curl -X POST --data-binary @largefile.bin http://localhost:8080/upload ```
3. Monitor JVM heap
   Use jcmd GC.heap_info or VisualVM to ensure the heap stays below the configured limit (e.g., 512 MB). In streaming mode, the heap should not exceed a few megabytes regardless of file size.
4. Compare latency
   Run the same upload with both implementations and record the time taken. Streaming may be slightly slower for small files due to async overhead.
5. Verify integrity
   Compute SHA‑256 of the original file and the stored file:
   ```bash sha256sum largefile.bin sha256sum /tmp/uploaded.bin ```
6. Simulate a mid‑stream failure
   Terminate the client after 1 GB and observe the error log. Ensure the service cleans up partial files if desired.

Practical Checklist Before Choosing Streaming

• Is the file size >10 MB or >1 GB?
• Do you have a container with limited heap?
• Does the client support chunked transfer encoding?
• Can the downstream service handle partial data or do you need to buffer the entire payload?
• Do you have the resources to handle asynchronous error propagation?

Answering these questions will guide you to the right approach.

Conclusion

For large files, streaming is the safer, scalable choice. It keeps memory usage low, supports resumable uploads, and handles backpressure automatically. For small payloads, the simplicity of in‑memory handling often wins. Always validate with real‑world file sizes and monitor heap usage to confirm your decision.