Connection timed out error during concurrent SFTP fetches in Harvester

Connection timed out during high‑concurrency SFTP fetches

Harvester’s fetcher pool is intended to limit concurrent downloads via the maxConcurrentFetchers setting. In practice, when many fetchers target the same remote directory, logs frequently show a Connection timed out message, suggesting that the underlying SFTP protocol is being throttled or that the thread pool is not being respected for SFTP sources.

Another uncertainty concerns how Harvester treats files that are partially written when a fetcher is interrupted. The current documentation does not specify whether the incomplete file is removed, retained, or renamed, leaving the ingestion pipeline vulnerable to duplicate or corrupted data.

Given these gaps, I need to understand:

• How is maxConcurrentFetchers enforced for SFTP sources, and does it differ from HTTP/FTP?
• What is Harvester’s policy for incomplete files left behind after a fetcher termination?
• Can adjusting the fetcher pool size or adding a retry strategy effectively mitigate the observed latency spikes?