Filtering and Projecting JSON with Ballerina Table Comprehensions

The Problem: Shaping JSON without the Noise

You've received a JSON array from a third-party API containing user records, and you need to keep only active users under age 35, extracting just their full name and email address. Writing this out with nested for loops and if checks quickly becomes verbose, and juggling error paths adds more noise.

Ballerina table type offers a declarative alternative: a SQL-like comprehension syntax that can filter, project, and order in-memory record sets in just a few lines.

How Ballerina Table Comprehension Works

The table type stores ordered records with named fields. A comprehension block uses a from clause to iterate over a data source, an optional where clause to filter records, a select clause to project specific fields, and an optional order by clause to sort the result. All of this lives inside Ballerina's error-handling model: you can prefix the block with check and an else block to handle failures gracefully.

Key Clauses Explained

1. from record in data: Introduces the iteration variable and the table or array you're shaping.
2. where record.age > 30: Keeps only records that satisfy the condition. If a record does not have an age field, the comprehension may raise a runtime error or skip the record, depending on the Ballerina version.
3. select record.name, record.email: Projects just the named fields, dropping the rest. The result is a new table; the original data stays untouched.
4. order by record.name: Sorts the resulting table by the chosen field.

A Worked Example

Suppose you have a table named users populated from a JSON payload. The following Ballerina snippet demonstrates a comprehension that filters users older than 30 and projects only their name and email:

import ballerina/io;
import ballerina/json;

type User record {string name; int age; string email;};

function main() returns error? {
    // Simulated JSON payload
    json sourceData = [
        {"name": "Alice", "age": 28, "email": "[contact removed]"},
        {"name": "Bob", "age": 34, "email": "[contact removed]"},
        {"name": "Carol", "age": 45, "email": "[contact removed]"}
    ];
    
    // Convert JSON array to table
    table userTable = check from User u in sourceData select u;
    
    // Comprehension: filter and project
    table filtered = from User u in userTable
                           where u.age > 30
                           select {name: u.name, email: u.email};
    
    // Output the result
    foreach var record in filtered {
        io:println(record.name, " - ", record.email);
    }
}

Verification Step

To try this yourself:

1. Save the file as table_example.bal.
2. Run it with the Ballerina command-line tool: bal run table_example.bal. No special permissions are required beyond a standard development environment with Ballerina installed.
3. Check the console output. You should see the names and email addresses of records where age exceeds 30, printed in the order defined by the from clause. The original userTable remains unchanged.
4. If a record in your source data lacks an age field, the comprehension may produce a runtime error. In that case, consider adding a defensive check or normalizing the data beforehand.

Trade-offs and When to Look Elsewhere

Table comprehensions shine when you need quick in-memory shaping within a Ballerina service. They integrate directly with the language's type system and error handling, so you don't have to context-switch to a separate query language.

However, there are practical limits:

• Nested field access: Referencing deeply nested fields in the where clause can become verbose and error-prone if the source records have inconsistent shapes.
• Aggregation: While order by is supported, Ballerina's table type does not offer built-in GROUP BY or aggregate functions like COUNT within a comprehension block. For heavy aggregation, you might chain a reduce loop after the comprehension.
• Mutation: Comprehensions always produce a new table. If you need in-place modification of records, you'll work with individual map entries rather than the comprehension syntax.

If your pipeline already relies on DataWeave, jq, or a SQL database, those tools may be more appropriate. But for Ballerina-native services that stay within one language runtime, table comprehensions are a concise and type-safe choice.

Closing: Try It and Verify

The fastest way to decide if this approach fits your workflow is to swap your next JSON-shaping loop for a comprehension block and run it through bal run. Observe the output shape, check that the original data is untouched, and assess whether the verbosity drops compared to your current loop pattern. If the filter logic grows complex, consider extracting it into a separate function or using Ballerina's pattern-matching features alongside the comprehension.

Ballerina's table comprehension syntax won't replace every data-manipulation task, but for many in-memory JSON transformations, it's a practical library-free tool that stays out of your way.