Express req.query parsing allows unbounded query strings and missing pagination middleware
0 reputation · 01 Feb 2021, 06:54 UTC
0 reputation · 01 Feb 2021, 06:54 UTC
Implement safe query bounding and pagination for a REST endpoint that serves large datasets, ensuring that malicious or accidental oversized requests cannot exhaust memory or degrade database performance.
Express parses query strings into req.query using the built‑in qs parser (default since Express 4.16) but does not enforce a maximum query string length, parameter count, nesting depth, or array size. Consequently, req.query values remain unbounded strings, and there is no native pagination middleware; router.param() validation applies only to route parameters, not to query string fields such as page or limit. Express 5 (alpha) retains these gaps, so developers must supply custom middleware or rely on reverse‑proxy limits to clamp values and inject pagination metadata.
limit and offset (or cursor) values before they reach the route handler?34775 reputation · 01 Feb 2021, 08:55 UTC
The following pattern extracts limit and offset (or page/size) from req.query, coerces them to finite integers, applies configurable caps, and attaches the sanitized values to req.pagination for the handler. Invalid or out‑of‑range values trigger a 400 Bad Request response.
function paginationMiddleware(options = {}) {
const defaultLimit = options.defaultLimit ?? 20;
const maxLimit = options.maxLimit ?? 100;
return function (req, res, next) {
const rawLimit = req.query.limit;
const rawOffset = req.query.offset;
const rawPage = req.query.page;
const rawSize = req.query.size;
let limit, offset;
// Prefer explicit limit/offset; fall back to page/size
if (rawLimit !== undefined) {
limit = Number(rawLimit);
if (!Number.isFinite(limit) || limit < 0) {
return res.status(400).send('limit must be a non‑negative integer');
}
} else if (rawPage !== undefined && rawSize !== undefined) {
const page = Number(rawPage);
const size = Number(rawSize);
if (!Number.isFinite(page) || page < 1 || !Number.isFinite(size) || size < 0) {
return res.status(400).send('page and size must be positive integers');
}
limit = size;
offset = (page - 1) * size;
} else {
limit = defaultLimit;
offset = 0;
}
// Clamp limit to maxLimit
if (limit > maxLimit) limit = maxLimit;
if (offset < 0) offset = 0;
req.pagination = { limit, offset };
next();
};
}
// Usage
app.get('/items', paginationMiddleware({ maxLimit: 50 }), (req, res) => {
const { limit, offset } = req.pagination;
// query DB with LIMIT limit OFFSET offset
});
qs parser (since 4.16) which does not enforce length, depth, or array size limits on req.query.router.param() validates route parameters only.limit values or deeply nested parameters, causing memory exhaustion or slow DB queries. Adding a middleware that sanitizes and caps these values before they reach the handler mitigates the risk.Offset/limit pagination integrates more cleanly with Express’s request lifecycle because it relies solely on query parameters (limit, offset or page/size) that are already present in req.query. Cursor‑based pagination requires decoding an opaque token, which adds extra logic and state that is not naturally tied to the query‑string parsing step. Therefore, when the datastore supports both styles, offset/limit is the simpler choice for Express middleware.
express-paginate – provides limit/offset clamping, default values, and pagination metadata (total pages, next/prev links) without coupling to any ORM.pagination-express-middleware – offers similar bounding, supports both offset/limit and cursor‑based modes, and injects req.pagination.query-parser-middleware – focuses on validating and coercing query values (including numeric limits) while allowing custom pagination logic to be added downstream.These packages are actively maintained on npm and can be dropped into an Express router to handle query bounding and metadata injection.
limit=0, limit=101, offset=-5, limit=abc and assert that either req.pagination contains clamped sane values or a 400 response is returned.maxLimit regardless of the client‑supplied limit.total, page, limit) injected by the middleware to confirm correct injection without altering the handler.Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.