JAMstack build-time data fetching limits for large datasets

Answer: Typical limits and bottleneck scale

Confirmed platform limits:

• Netlify Free tier: builds stop after 15 minutes; Pro tier after 30 minutes.
• Vercel Hobby/Pro plans: builds limited to 45 minutes (Enterprise can raise this).
• Most CI‑based static‑site providers enforce similar soft limits (often 20‑60 min) unless you purchase higher‑tier minutes.

Confirmed API constraints:

• Many headless CMS APIs enforce request‑rate caps (e.g., Contentful ≈ 60 req/s) or GraphQL complexity limits that return HTTP 429 when exceeded.
• These limits add latency or cause failures when thousands of pages are fetched sequentially.

Likely explanation of the bottleneck

When the total build time (data fetch + page generation) approaches the platform timeout, or when API throttling adds significant delay, the build is killed or fails. Empirical observations indicate that sites with roughly >10 k pages or >100 k CMS records often see build times rise sharply, pushing them toward the timeout threshold unless pagination, caching, or incremental strategies are used.

Steps to diagnose and mitigate for your case

1. Measure the current full‑build duration locally or in CI with a representative data set.
2. Enable API request logging (or use a proxy) to count requests and watch for 429/throttling responses during a build.
3. If build time exceeds ~80 % of your platform’s limit, implement pagination/batching, cache fetched data between builds, or adopt Incremental Static Regeneration (ISR) / On‑Demand Builders for the most volatile content.
4. Re‑measure after changes; if the build now stays comfortably under the limit, the bottleneck is resolved.

Missing diagnostic detail: What is your observed average build time for a full data fetch? Knowing this tells us whether you are already near the timeout and whether ISR or caching is urgently needed.