Using Gulp’s `since` Option for Incremental Builds: An Architecture Note

Problem & Takeaway

Large JavaScript projects often spend a disproportionate amount of time rebuilding unchanged files. Gulp’s since option in gulp.src lets you process only files that have changed since the last run, cutting build times while keeping output deterministic. This note outlines the minimal design, trust boundaries, operational checks, failure modes, and when you should evolve the approach.

Requirements

• Detect and rebuild only files modified after the previous run of the task.
• Maintain full build reproducibility: unchanged inputs must not alter the output.
• Operate safely on a shared file system that may have varying timestamp precision.

Minimal Design

Define a single Gulp task that uses gulp.src with the since option, pipes the stream through the necessary transforms, and writes the result with gulp.dest. Wrap the task in gulp.series if order matters across multiple steps.

const { src, dest, series } = require('gulp');
const babel = require('gulp-babel');
const concat = require('gulp-concat');

function scripts() {
  return src('src/**/*.js', { since: gulp.lastRun(scripts) })
    .pipe(babel({ presets: ['@babel/preset-env'] }))
    .pipe(concat('bundle.js'))
    .pipe(dest('dist'));
}

exports.build = series(scripts);

Trust / Data Boundaries

• File System: The source tree is the only trusted boundary. Gulp relies on file timestamps to decide whether a file has changed.
• Transform Plugins: Each plugin (e.g., Babel, Concat) must be deterministic; they should not alter output based on external state.
• Cache: Avoid relying on external caching layers unless you explicitly manage them; otherwise you risk stale artifacts.

Operational Checks

1. Run the task with gulp scripts --reporter verbose to see which files are processed.
2. Verify that the console output lists only the files you modified.
3. After the run, inspect the dist/bundle.js timestamp; it should be newer than the most recent source file that was processed.
4. Optionally, use gulp.lastRun(scripts) in a custom log to confirm the stored timestamp matches the expected value.

Failure Modes

• Stale Timestamps: If the file system does not update timestamps (e.g., due to NFS sync delays), Gulp may skip files that actually changed.
• Transform Errors: Syntax errors in a source file abort the stream, leaving the destination partially updated.
• Missing Source: If the source glob matches no files, Gulp silently emits a no-op; the destination remains unchanged.
• Timestamp Precision: Some file systems have 1‑second precision; rapid successive edits may be treated as unchanged.

Conditions That Change the Design

• Large source trees (tens of thousands of files) may require an additional caching layer (e.g., gulp-cache) to avoid scanning the entire tree on each run.
• When you need real‑time feedback (e.g., live reload during development), switch to a watch‑based incremental build: add gulp.watch('src/**/*.js', scripts) and handle reload logic.
• If you want to ensure content changes (not just timestamps) trigger rebuilds, consider using a hash‑based cache or gulp-newer.
• On distributed build systems, you may need to propagate the last run timestamp to all agents to keep the incremental logic consistent.

Verification Checklist

• Run gulp scripts --reporter verbose before and after modifying a file; only the modified file should appear in the log.
• Check the output file timestamps: they should match or exceed the timestamps of the source files that were processed.
• If you add a new file, the task should rebuild the entire destination (or at least include the new file).
• When no files change, the task should finish quickly with no log entries indicating file processing.

Limitations & Mitigations

• Timestamp precision may vary across OSes; on systems with low precision, consider gulp-newer or a hash‑based approach.
• Large directories can increase memory usage during change detection; monitor node process memory and consider splitting the task into smaller streams.
• Plugins that modify file metadata (e.g., adding source maps) can unintentionally update timestamps; ensure such plugins are configured to preserve timestamps if needed.

Conclusion

Gulp’s since option offers a lightweight, reliable way to achieve incremental builds for most projects. By carefully defining trust boundaries, validating operational checks, and being aware of failure modes, you can keep build times short without sacrificing correctness. Upgrade to a more sophisticated incremental strategy only when the simple since approach no longer scales with your codebase size or build complexity.