Missing per-request timeout for Git LFS chunk transfers in Forgejo
0 reputation · 19 Nov 2023, 22:22 UTC
0 reputation · 19 Nov 2023, 22:22 UTC
The goal is to bound Git LFS uploads with a server‑side timeout so stalled connections do not consume resources indefinitely.
Forgejo already propagates request‑level context.Context to HTTP handlers and honors SERVER_TIMEOUT and WEBHOOK_TIMEOUT, but the LFS chunk transfer handler does not respect these limits; there is no dedicated configuration option for LFS timeouts, and it is unclear whether adding one would require changes to the LFS protocol handling or middleware.
Is there a plan to expose a LFS_TIMEOUT setting in app.ini? Would implementing per‑request context cancellation for LFS transfers interfere with existing LFS client expectations? What impact could a server‑side LFS timeout have on mirroring or other LFS‑related background jobs?
33275 reputation · 20 Nov 2023, 06:42 UTC
As of Forgejo v1.22.x there is no LFS_TIMEOUT setting in app.ini and no merged pull request that adds one. The LFS chunk upload/download handlers in modules/lfs/server.go run on a plain net/http server and do not inherit the global SERVER_TIMEOUT or WEBHOOK_TIMEOUT values. Stalled client connections can therefore hold goroutines and file descriptors indefinitely.
Implementing a per-request timeout is technically feasible but requires wrapping the LFS handlers with a context.WithTimeout derived from a new config key and propagating that context through the underlying storage layer (local filesystem or S3). The Tus resumable‑upload protocol used by git-lfs and the GitHub CLI will retry a chunk if the connection is closed mid‑transfer, so a server‑side timeout that exceeds the typical chunk transfer time (default 1 MB chunks) is safe for clients.
Mirroring and background LFS sync jobs (PullMirrorLFS, scheduled pulls) use their own HTTP client timeouts (default 30 s) and are unaffected by an inbound LFS transfer timeout.
LFS.TIMEOUT (or similar) to app.ini.sample and the configuration loader.modules/lfs/server.go, wrap UploadHandler and DownloadHandler with a middleware that creates ctx, cancel := context.WithTimeout(r.Context(), cfg.LFS.Timeout) and passes ctx to the storage operations.context.Canceled / context.DeadlineExceeded as non‑fatal so the client can resume cleanly.LFS_TIMEOUT below ~10 s will break uploads on high‑latency or bandwidth‑constrained links, causing repeated chunk retries and increased load.Forgejo issue tracker shows open discussions requesting an LFS timeout, but no merged PR on the v1.22 release branch. The feature is therefore not on the immediate roadmap unless a contributor submits a PR with the changes outlined above.
If you need a bound right now, place a reverse proxy (nginx, Caddy, Traefik) in front of Forgejo and enforce proxy_read_timeout / proxy_send_timeout on the /lfs/ location. This kills stalled connections at the proxy layer without touching Forgejo code.
Are you seeing OOM kills or file‑descriptor exhaustion on the Forgejo host during large LFS uploads, or is this a preventive hardening request? If the former, the proxy workaround is the fastest mitigation; if the latter, a config‑driven timeout can wait for a proper upstream implementation.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.