Duplicate gulp.dest writes after Gulp 4 task retry

Answer

gulp.dest cannot skip a write when the destination file already exists; it always writes the vinyl file to disk. Gulp 4 core provides no built‑in idempotent retry mechanism, so you must filter source files before they reach gulp.dest or maintain your own write log. The way you signal task completion (async/await, returning a stream, promise, or callback) does not change which files are safe to exclude on a retry.

Confirmed facts

• gulp.dest performs a plain fs.createWriteStream (or equivalent) without checking for existing content.
• Gulp 4 has no native retry or transactional write log; idempotency must be added via plugins such as gulp-changed or gulp-newer, or via custom logic that records which files have been successfully written.
• The task completion signal only tells Gulp when the stream has ended; it does not influence source‑file filtering for a subsequent run.

Likely explanation

When a task fails after some files have already passed through gulp.dest, those files remain on disk. On retry the same source files flow through the stream again, and gulp.dest overwrites them because it has no memory of the previous successful writes.

Steps to achieve idempotent retry for this case

1. Add a source‑file filter that skips files whose destination already exists and is up‑to‑date. The simplest approach is to use gulp-changed (or gulp-newer) before gulp.dest:

const { src, dest, series } = require('gulp');
const changed = require('gulp-changed');

function build() {
  return src('src/**/*.js')
    .changed('dist')          // skips if dest file exists with same or newer mtime
    .pipe(/* your plugins */)
    .pipe(dest('dist'));
}

function buildWithRetry(done) {
  // Example retry wrapper – adjust attempts as needed
  let attempts = 0;
  function tryBuild() {
    return build().on('error', err => {
      attempts++;
      if (attempts < 3) {
        console.warn(`Build failed, retrying… (${attempts})`);
        return tryBuild();
      }
      done(err);
    });
  }
  return tryBuild();
}

exports.default = series(buildWithRetry);

1. If you need content‑based checks (not just timestamps), maintain a simple write log:

const fs = require('fs');
const path = require('path');
const logFile = path.join(process.cwd(), '.gulp-write-log.json');

function loadLog() {
  try { return JSON.parse(fs.readFileSync(logFile, 'utf8')); }
  catch { return {}; }
}
function saveLog(log) { fs.writeFileSync(logFile, JSON.stringify(log, null, 2)); }

function buildWithLog() {
  const log = loadLog();
  return src('src/**/*.js')
    .pipe(through2.obj(function(file, enc, cb) {
      const destPath = path.join('dist', file.relative);
      if (log[destPath] === file.stat.mtime.getTime()) {
        // already written with same timestamp – skip
        return cb();
      }
      this.push(file);
      cb();
    }))
    .pipe(/* plugins */)
    .pipe(dest('dist', {
      // record successful writes
      // using a post‑write hook via 'end' on the destination stream
    }));
}
// (Implementation of the post‑write hook omitted for brevity)

Missing diagnostic detail

If you are using a custom completion signal (e.g., returning a promise vs. a stream) and you suspect it influences which files are considered “already processed,” let me know which signal you are using; otherwise, the recommendation above applies regardless of the signal.