Jekyll Incremental Build: Architecture Note

Requirements

Jekyll’s incremental build must satisfy four core requirements:

1. Detect changes to source files (Markdown, HTML, layouts, includes, data files) without scanning the entire tree on every rebuild.
2. Rebuild only the pages whose output could be affected by those changes, preserving overall site consistency.
3. Continue to work with existing plugins, collections, and custom generators that run in the same Ruby process.
4. Leave the generated _site directory as the sole output boundary, ensuring no stray files are left behind.

Smallest Suitable Design

The design that meets the above requirements with the fewest moving parts consists of three components:

• File‑system watcher – a lightweight listener (using listen gem) that queues the paths of modified source files.
• Dependency graph – a directed graph limited to layout files, include files, and data files (YAML/JSON/TOML in _data). Each page or post records which of these assets it depends on.
• Selective rebuild step – processes only the queued source files, runs the normal Jekyll pipeline for each, and writes the resulting HTML to _site while leaving untouched files unchanged.

When a queued file is a layout, include, or data file, the graph is consulted to determine which pages need regeneration; otherwise only the file itself is rebuilt.

Trust and Data Boundaries

The source tree (the project root) is treated as trusted input. Jekyll assumes that any file it reads is safe to execute within the Ruby process. The generated _site directory is the output boundary; nothing outside this directory is written during an incremental build. Plugins, however, execute with the same privileges as the Jekyll process and can run arbitrary Ruby code, so they must be vetted before use.

Operational Checks

To confirm that incremental mode is behaving as expected:

1. Start the server with the incremental flag:

jekyll serve --incremental

2. Edit a single Markdown file (e.g., _posts/2026-09-01-example.md) and save.
3. Observe that only the corresponding HTML file in _site (e.g., _site/2026/09/01/example.html) has a newer timestamp; all other files retain their original timestamps.
4. Inspect the .jekyll-metadata file: the entry for the edited file should show an updated timestamp or checksum, while entries for unchanged files remain identical.
5. Change a YAML file in _data (e.g., _data/navigation.yml) and verify that every page that references that data is rebuilt (their timestamps update).

These steps give a practical way to check that the watcher, dependency graph, and selective rebuild are functioning.

Failure Modes

Even with the minimal design, several failure modes can lead to stale or incorrect output:

• Watcher misses changes – certain editors perform atomic renames or write to temporary files; network‑mounted filesystems may not propagate events reliably. If a change is not queued, the affected page will not be rebuilt.
• Incomplete dependency graph – if a plugin reads a data file that is not declared as a dependency (e.g., via site.data accessed dynamically), the graph will not trigger a rebuild when that data changes, leaving pages stale.
• Plugin side‑effects – plugins that modify global site state (such as site.posts) or write to files outside the normal pipeline can cause incremental builds to produce incorrect results because the graph does not track those mutations.

Conditions That Would Change the Design

The current design assumes a relatively static site structure and shared‑process plugin execution. It would need to be revisited if:

1. The project adopts front‑end tooling that requires cross‑page JavaScript or CSS bundling (e.g., Webpack, Rollup). Incremental rebuilds of individual pages would no longer suffice; a broader build step would be necessary.
2. Jekyll moves to a sandboxed plugin model where plugins run in isolated processes or with restricted access. The trust boundary would shift, and the dependency graph might need to include plugin‑generated assets.

Either scenario would invalidate the assumption that a simple file‑watcher plus a layout/include/data graph is sufficient, prompting a more comprehensive dependency tracking system or a fallback to full rebuilds.

Limitations and Practical Verification

While the incremental build speeds up local development, it is not a substitute for a full build in production. To verify that your site is safe:

• Run a full build (jekyll build) after a series of incremental edits and compare the _site directories with a tool like diff -r or rsync --checksum.
• Check that no files are missing or have unexpected content, especially when using plugins that generate pages dynamically.
• If discrepancies appear, consider disabling incremental mode for that workflow or adding explicit dependencies to the graph (e.g., by listing required data files in a page’s front matter).

These checks give confidence that the incremental build has not introduced silent stale output.