Implementing Incremental Rebuilds with gulp.watch and lastRun

The Problem: Redundant Build Cycles

In large projects, running a full build pipeline on every file change is inefficient. Processing hundreds of unchanged files through transpilers or minifiers increases developer feedback loops and wastes CPU cycles. While gulp.watch can trigger tasks, simply triggering a full task is not an incremental build.

The goal is to process only the files that have changed since the last successful execution of a specific task, while ensuring that deleted source files do not leave "ghost" artifacts in the build directory.

Requirements

A reliable incremental system must meet these criteria:

• Version Compatibility: Must use Gulp 4, as the lastRun API is not available in Gulp 3.
• State Tracking: Must track the timestamp of the last successful task completion to filter source files.
• Resilience: The watcher must not crash when a plugin encounters a syntax error in a modified file.
• Cleanliness: Must provide a mechanism to handle file deletions, as modification-time filtering only identifies new or changed files.

Smallest Suitable Design

The minimal architecture for incremental builds relies on the since option in gulp.src. This tells Gulp to only emit files modified after a specific date.

Core Implementation

Run these commands in your project root. Ensure you have Gulp 4 installed (gulp --version). You will need through2 for logging and del for cleanup.

import { src, dest, watch, series, lastRun } from 'gulp';
import { deleteAsync } from 'del';
import through2 from 'through2';

// Task to process files incrementally
export function compileJS() {
  return src('src/**/*.js', { since: lastRun(compileJS) })
    .pipe(through2.obj((file, enc, cb) => {
      console.log('Processing:', file.relative);
      cb(null, file);
    }))
    // Replace with actual transformation plugins (e.g., babel)
    .pipe(dest('dist'));
}

// Task to remove orphaned files in dist
export async function cleanOrphans() {
  // Logic: In a production scenario, compare dist/ files against src/ files
  // For this minimal design, we provide a full clean as the primary recovery
  console.log('Checking for orphaned files...');
}

// The Watcher
export function watchFiles() {
  // ignoreInitial: false ensures the first run processes all files
  watch('src/**/*.js', { ignoreInitial: false }, series(compileJS, cleanOrphans));
}

// Full rebuild escape hatch
export async function clean() {
  return await deleteAsync(['dist/**']);
}

export const rebuild = series(clean, compileJS);
export default watchFiles;

Design Decisions

• lastRun(compileJS): This retrieves the timestamp of the last time compileJS completed successfully. If the task is renamed or called ad-hoc without this reference, Gulp treats it as a full build.
• series(): We use series instead of parallel to ensure the compilation finishes before any cleanup or notification logic begins.
• The Deletion Gap: src({ since: ... }) ignores deleted files. To solve this, the cleanOrphans step is required to synchronize the destination folder with the source folder.

Trust and Data Boundaries

Gulp plugins operate as middleware in a Node.js stream, meaning they have full access to the filesystem and environment variables. To maintain a secure boundary:

• Pin Plugin Versions: Use a package-lock.json to prevent automatic updates to third-party plugins that execute during every file save.
• Restrict Globs: Avoid using overly broad globs (e.g., **/*.js) that might accidentally pull in node_modules or configuration files from the project root.

Operational Checks

To verify the incremental logic is functioning, perform these three checks:

1. Initial Run: Start the watcher. Confirm that all files in src/ are logged as "Processing".
2. Incremental Update: Modify a single file. Confirm that only that specific file is logged. If all files are processed again, the lastRun reference is likely broken.
3. Deletion Test: Delete a source file. Check the dist/ folder. If the corresponding output file remains, your cleanOrphans logic needs to be expanded from a placeholder to a directory comparison tool.

Failure Modes and Design Changes

Certain conditions render this mtime-based (modification time) design obsolete:

1. Watch Storms

In monorepos with thousands of files, chokidar (Gulp's watch engine) may consume excessive CPU or hit OS file handle limits. If this happens, add ignored: /node_modules/ to the watch options or enable usePolling: true for Docker/Network mounts.

2. Stream Crashes

A plugin error (like a Babel syntax error) will kill the Node process and stop the watcher. To prevent this, attach an error handler to the stream:

src('src/**/*.js', { since: lastRun(compileJS) })
  .on('error', function(err) { 
    console.error('Error occurred:', err.message); 
    this.emit('end'); 
  })
  .pipe(...)

3. Dependency Graphs

If changing utils.js requires main.js to be rebuilt (because main imports utils), this design fails. lastRun only knows about the file that changed, not its dependents. If your project requires dependency-aware builds, migrate from Gulp to a bundler (Webpack, Esbuild) or a graph-based build system (Bazel).