Connection timed out error during concurrent SFTP fetches in Harvester

Answer first

No confirmed product documentation is available to verify how Harvester enforces maxConcurrentFetchers for SFTP, HTTP or FTP, nor its policy for incomplete files after fetcher interruption. With the information provided, the behavior cannot be confirmed from public sources. The guidance below is a conservative assessment based on general SFTP/SSH behavior and must be verified against your Harvester version.

Confirmed facts

• Training data does not contain confirmed specifics for a product named Harvester that exposes maxConcurrentFetchers or defines SFTP fetcher-pool behavior distinct from HTTP/FTP.
• Without version-specific release notes or source inspection, it is not possible to state whether maxConcurrentFetchers applies equally to SFTP, HTTP and FTP, or whether it is enforced per source type, per host, or globally.
• Incomplete-file handling after fetcher termination is product-specific. No confirmed policy for Harvester can be stated.

Likely explanation vs confirmed facts

Likely explanation: SFTP runs over SSH. Even if a client-side fetcher pool limit exists, each fetcher may open its own SSH session and TCP connection. Concurrent fetches to the same remote directory commonly exhaust server-side limits such as MaxSessions, open file descriptors, or local port availability, producing Connection timed out errors that appear unrelated to the pool setting.

Likely explanation: If maxConcurrentFetchers is enforced only at the application fetcher level, SFTP clients may still open separate SSH connections per file unless connection sharing or pooling is explicitly implemented. That would make SFTP more sensitive to server throttling than HTTP/FTP.

Likely explanation: Retry strategies can mask transient timeouts but risk duplicate or corrupt ingestion unless fetches are idempotent and use atomic rename/commit semantics. Increasing pool size often worsens timeouts when the remote SFTP server throttles sessions; lowering concurrency is usually more reliable.

Steps needed for this case

1. Verify enforcement scope. Inspect Harvester release notes and source for the fetcher implementation to confirm whether maxConcurrentFetchers applies to SFTP, HTTP and FTP equally, and whether it limits fetcher threads or actual SSH sessions.
2. Distinguish client pool vs server limits. Capture SFTP/SSH debug logs during a timed-out concurrent run and compare active SSH sessions against the configured pool. Run a controlled single-connection SFTP fetch against the same remote directory to isolate concurrency-related timeouts from persistent network issues.
3. Check incomplete-file policy. Run a controlled fetcher interruption and inspect the target directory immediately after. Determine whether partial files are removed, retained in place, or renamed to a temporary suffix. Do not enable retries until this behavior is known.

One missing diagnostic detail that changes the recommendation: Does Harvester reuse a single SSH connection per host across fetchers, or does each fetcher open a new SSH session per file? If connection reuse is not implemented, reducing maxConcurrentFetchers and adding server-side session limits tuning will be required; if reuse is implemented, the timeout is more likely network or server MaxSessions.