Sequelize Pagination: Limit and Distinct behavior during Eager Loading
0 reputation · 27 Mar 2024, 07:36 UTC
0 reputation · 27 Mar 2024, 07:36 UTC
When implementing pagination in Sequelize (v6+), the limit and offset options are used to control the result set size. However, a specific complexity arises when utilizing include for one-to-many associations alongside these pagination constraints.
To prevent the joined result set from inflating the row count, Sequelize often generates a subquery. When multiple associations are involved, the interaction between the limit option and the distinct: true configuration becomes critical to ensure the main model is paginated correctly rather than the total number of joined rows.
There is uncertainty regarding how the distinct flag affects the generated SQL when joining across multiple levels of associations, specifically concerning whether the limit is applied to the unique primary keys of the main model or the resulting join table.
distinct: true consistently ensure the limit applies to the main model's primary keys across all supported SQL dialects?distinct is omitted during a paginated eager load of multiple one-to-many associations?34775 reputation · 27 Mar 2024, 09:45 UTC
Does distinct:true consistently ensure the limit applies to the main model's primary keys across all supported SQL dialects?
In practice, distinct:true is the mechanism Sequelize uses to keep limit and offset on the parent table, regardless of how many child rows are joined. With the default subQuery:true behaviour for a paginated include, Sequelize builds an inner query that selects unique parent primary keys with the limit/offset applied, then joins the associations in an outer query. The exact SQL is dialect-specific: PostgreSQL can use DISTINCT ON ("Parent"."id"), while MySQL, SQLite, and MSSQL rely on equivalent subquery/DISTINCT constructs. In all cases the intent is the same: the outer query returns a unique set of parent rows before the pagination limits are applied.
However, the guarantee only holds when the query keeps its default shape. If you set subQuery:false, use separate:true, or add options that alter the query (custom attributes that trigger a GROUP BY, for example), the distinct logic can be bypassed or weakened, and the limit may end up applied to the joined row set instead.
Therefore:
distinct:true is the documented way to paginate parents reliably when including one-to-many associations.ONLY_FULL_GROUP_BY).subQuery:false, separate:true, custom attributes), re-verify; the guarantee no longer holds automatically.distinct Is Omitted?Without distinct:true, Sequelize applies the limit to the raw joined result set. A parent with five children produces five joined rows, each consuming one slot of the limit. A page requesting limit: 10 can then contain fewer than ten distinct parents, and the count from findAndCountAll is inflated because it counts joined rows rather than unique parents.
Example: Parent.findAndCountAll({ limit: 2, include: [Child] }) against a dataset where each parent has three children can return just one parent (the first two joined rows) and a count of six (two parents × three children), which is misleading.
logging: console.log to the query (or set it on the instance) and inspect the generated SQL for DISTINCT, DISTINCT ON, GROUP BY, or a subquery wrapped around the primary key.await Parent.findAndCountAll({ limit: 1, include: [Child], distinct: true }); — a parent with three children should come back as one parent with all three children.distinct:true and observe that the limit is applied to the joined rows, truncating the association.When paginating over one-to-many associations, always include distinct:true (and use findAndCountAll with the same flag if you need a total count that matches the pages). If you use subQuery:false for performance, check the generated SQL; the safest fallbacks are keeping distinct:true or switching to separate:true, which runs a separate query per association and avoids join inflation entirely, at the cost of single-query atomicity.
Which SQL dialect are you targeting? That changes the verification steps — for example, MySQL with ONLY_FULL_GROUP_BY enabled behaves differently around GROUP BY/DISTINCT than PostgreSQL's DISTINCT ON.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.