Hibernate Pagination: Memory-side loading when using JOIN FETCH
0 reputation · 11 Dec 2021, 00:13 UTC
0 reputation · 11 Dec 2021, 00:13 UTC
Hibernate utilizes setFirstResult() and setMaxResults() to implement pagination, typically delegating the offset and limit logic to the database via the configured SQL dialect.
A specific behavioral conflict occurs when JOIN FETCH is applied to a collection within a paginated query. Because joining a one-to-many relationship can produce duplicate parent rows in the result set, the ORM cannot reliably calculate the correct row offset at the database level without risking data loss or incorrect page sizes.
In such scenarios, Hibernate may bypass the database-level pagination and load the entire result set into memory to perform the filtering manually. This behavior creates a significant risk of OutOfMemoryError when dealing with large datasets.
32775 reputation · 11 Dec 2021, 11:12 UTC
When you call setFirstResult() and setMaxResults() on a query that contains JOIN FETCH for a collection, Hibernate cannot push the pagination into the SQL. The join creates duplicate rows for the root entity, so Hibernate fetches the entire result set from the database and then performs the offset/limit in Java memory. This memory‑side filtering can cause OutOfMemoryError on large datasets.
LIMIT/OFFSET on the root table when a JOIN FETCH is present.DistinctRootEntityResultTransformer (or an equivalent) to deduplicate root entities.logging.level.org.hibernate.SQL=DEBUG
Run the query and confirm that the generated SQL does not contain LIMIT/OFFSET on the root table.SELECT COUNT(*) in a separate query) and compare with the expected page size multiplied by the average number of fetched associations.query.setResultTransformer(Criteria.DISTINCT_ROOT_ENTITY);
This keeps the root entities unique but does not eliminate the duplicate rows that are fetched.hibernate.query.passDistinctThrough=false
Available from Hibernate 5.2+. This forces Hibernate to add a SELECT DISTINCT clause, reducing the number of rows returned.JOIN FETCH with batch or subselect fetching:
@BatchSize(size=30)
private Set<Child> children;
or
fetch=FetchMode.SUBSELECT
These strategies load collections lazily or in separate queries, allowing pagination to be applied at the database level.SELECT id FROM Parent WHERE … ORDER BY … LIMIT ? OFFSET ? to get the IDs for the desired page.FROM Parent p JOIN FETCH p.children WHERE p.id IN (:ids) to fetch the full entities for those IDs.Prior to Hibernate 5.2 the hibernate.query.passDistinctThrough hint does not exist, and some older releases may ignore the distinct transformer altogether. Verify the behavior on your specific version by inspecting the generated SQL.
To tailor the recommendation precisely, please let us know the exact Hibernate version you are using. Certain strategies (e.g., the passDistinctThrough hint) are only available in newer releases.
Use comments to ask for clarification. Post a solution as an answer.
32,775 reputation · 11 Dec 2021, 09:53 UTC
To clarify the detection of this behavior, Hibernate typically emits a specific warning when it is forced to perform pagination in memory. In most versions (including Hibernate 5.x and 6.x), look for the log message HHH000104: firstResult/maxResults specified with collection fetch; applying in memory!
It is important to note that this risk is exclusive to collection-valued associations (One-to-Many or Many-to-Many). Using JOIN FETCH on a @ManyToOne or @OneToOne relationship does not trigger in-memory pagination because these joins do not multiply the number of root entity rows in the result set, allowing the database to safely apply LIMIT and OFFSET.
For verification, if you see this warning in your logs, you can confirm the performance hit by checking if the generated SQL lacks the pagination keywords despite setMaxResults() being called.