Pagination and Query Building in Jeet: A Practical Decision Guide

The Pagination Performance Problem in Jeet

When building full-text search applications with Jeet, a common challenge emerges with deep pagination. Using standard offset-based pagination with large page numbers causes severe performance degradation because Lucene must score and store all preceding documents before returning results. This becomes critical when users navigate to page 1000 or beyond.

Supported Pagination Approaches

Method	Implementation	Max Offset	Memory Impact	Use Case
TopDocsCollector	IndexSearcher.search(query, offset, maxDocs)	Limited by RAM	High for deep pages	Shallow pagination
BooleanQuery.Builder	Dynamic query composition	Clause limit 1024	Moderate	Complex filtering
Custom collector	Extend Collector class	Unlimited	Configurable	Deep pagination

Trade-offs Analysis

Offset-based pagination (TopDocsCollector) is simple to implement but becomes exponentially slower. The search() method scores every document up to the requested offset. For example, requesting page 500 with 20 items per page requires scoring 10,000 documents regardless of actual results.

BooleanQuery max clause limit (default 1024) can be reached when building complex filters dynamically. Each MUST, SHOULD, or MUST_NOT clause counts toward this limit. When exceeded, TooManyClausesException is thrown.

Concrete Implementation: Cursor-Based Pagination

Instead of offset-based pagination, implement cursor-based navigation using document scores and IDs. This approach avoids re-scoring documents and maintains consistent performance regardless of page depth.

Setting Up the Query Parser

// Initialize query parser for your index
QueryParser parser = new QueryParser("content", new StandardAnalyzer());

// Build query programmatically
BooleanQuery.Builder builder = new BooleanQuery.Builder();
builder.add(new TermQuery(new Term("category", "tech")), BooleanClause.Occur.MUST);
builder.add(new RangeQuery("date", start, end), BooleanClause.Occur.MUST);

Query query = builder.build();

Implementing Cursor Pagination

public class PaginatedResult {
    private final ScoreDoc[] docs;
    private final int totalHits;
    private final String nextCursor;

    public PaginatedResult(ScoreDoc[] docs, int total, String cursor) {
        this.docs = docs;
        this.totalHits = total;
        this.nextCursor = cursor;
    }
}

public PaginatedResult searchWithCursor(IndexSearcher searcher, Query query,
        String cursor, int pageSize) throws IOException {

    int start = cursor != null ? Integer.parseInt(cursor) : 0;
    TopDocs results = searcher.search(query, start + pageSize);

    ScoreDoc[] pageDocs = Arrays.copyOfRange(results.scoreDocs,
        start,
        Math.min(start + pageSize, results.scoreDocs.length));

    String nextCursor = (start + pageSize) < results.totalHits ?
        String.valueOf(start + pageSize) : null;

    return new PaginatedResult(pageDocs, results.totalHits, nextCursor);
}

Validation and Testing

To verify your pagination implementation works correctly:

1. Test query parsing: Call parser.parse("field:value") with various query strings to ensure syntax compatibility
2. Benchmark performance: Measure response times for offsets 0, 100, 1000, 5000 to confirm performance degradation
3. Check clause limits: Build queries with 100+ clauses to verify no TooManyClausesException
4. Validate fuzzy matching: Test FuzzyQuery with edit distance of 1 and 2 to confirm typo tolerance

Key Limitations and Considerations

Memory consumption grows with result set size because TopDocs requires heap allocation for all matching documents. Monitor JVM heap usage during testing with large indexes.

Version compatibility varies between Lucene versions. Query syntax and available classes may change. Always test query strings after version upgrades.

Deep pagination alternative: For truly large result sets, consider implementing CustomCollector or using IndriPolicy for sampling-based approaches.

Practical Verification Steps

Run these commands to validate your setup:

# Check current Lucene version in classpath
mvn dependency:tree | grep lucene

# Test query parser with sample input
java -cp . QueryParserTest "category:tech AND date:[2023-01-01 TO *]"

# Benchmark pagination at different offsets
java -cp . PaginationBenchmark --offset=1000 --size=50

Expected output: Query parsing should succeed without exceptions, and pagination times should remain consistent across different offsets when using cursor-based approach.