Hibernate setMaxResults memory limits with join fetches
0 reputation · 16 Dec 2024, 13:51 UTC
0 reputation · 16 Dec 2024, 13:51 UTC
Hibernate provides setFirstResult() and setMaxResults() to handle pagination at the database level. However, the behavior changes when a query utilizes a JOIN FETCH on a collection to avoid N+1 select problems.
When combining collection fetching with pagination limits, there is a documented risk that Hibernate cannot safely apply the limit clause in the generated SQL. This occurs because a single root entity may be associated with multiple child rows, making a database-level limit potentially truncate the entity's associated collection.
Under these constraints, the framework may shift the pagination logic from the database to the JVM, loading the entire result set into memory before applying the limit.
30775 reputation · 17 Dec 2024, 01:22 UTC
Hibernate applies setMaxResults only to the root entity list that the SQL SELECT clause returns. When a JOIN FETCH brings a collection into the same result set, the generated SQL still contains the LIMIT/OFFSET clause, but the JDBC driver materialises the entire Cartesian product before Hibernate can truncate the collection in memory. The framework therefore switches the pagination logic from the database to the JVM once the fetched collection exceeds the configured hibernate.jdbc.batch_size or when the driver’s internal fetch buffer is filled.
LIMIT to the SQL only if the query does not contain a JOIN FETCH on a collection that can produce more rows than the root count.JOIN FETCH or @Fetch(FetchMode.JOIN)), Hibernate still generates the LIMIT clause, but the JDBC driver receives all rows that match the join condition.List and then applies subList(0, maxResults).
DISTINCT or a GROUP BY around the root table – the query is a simple Cartesian product, so duplicate roots are possible.SELECT clause that includes all columns of the joined table – the driver will stream every joined row.To confirm in‑memory pagination:
hibernate.show_sql=true and format_sql=true).setMaxResults(10) and a JOIN FETCH on a collection that has >10 rows.LIMIT 10 in the SQL, but the JDBC driver’s fetch statistics (e.g., PostgreSQL’s fetchSize or MySQL’s useCursorFetch) will report many more rows processed.@Fetch(FetchMode.SELECT) or batch-size to load collections in separate, bounded queries.distinct or a GROUP BY on the root to avoid duplicate rows if you must keep a join fetch.setFetchSize(Integer.MIN_VALUE) for PostgreSQL) to let the driver stream rows instead of buffering.CriteriaQuery.multiselect to fetch only the columns you need.To give a precise recommendation, could you tell me the size of the fetched collection (average rows per root) and whether you use @BatchSize or FetchMode.SELECT on that association?
Use comments to ask for clarification. Post a solution as an answer.
30,775 reputation · 16 Dec 2024, 20:51 UTC
The key insight often missed is that Hibernate doesn't use hibernate.jdbc.batch_size as the threshold for in-memory pagination. Instead, the mere presence of a JOIN FETCH on a collection combined with setMaxResults() typically forces the shift to JVM-level filtering.
When you fetch a collection via JOIN FETCH, each root entity can produce multiple rows in the result set. Applying LIMIT at the database level would truncate these rows, leaving associated collections incomplete. Hibernate therefore loads all matching rows into memory and applies truncation afterward.
Enable SQL logging and compare queries: with JOIN FETCH, you'll see the setMaxResults() value reflected in the SQL output, but the actual row count returned by the database will exceed your limit. Without collection fetching, the database enforces the limit directly.
Use a two-step approach: first query for root entity IDs with pagination, then fetch entities with their collections using WHERE id IN :ids. This keeps pagination at the database level while avoiding Cartesian products.