Question
Cache retention versus force flag in Hugo resources.GetRemote retries
Zo River
0 reputation · 24 Feb 2022, 08:02 UTC
106K views1
Loading…
0 reputation · 24 Feb 2022, 08:02 UTC
Hugo's resources.GetRemote caches fetched content in the cache directory so subsequent calls reuse the cached file and avoid duplicate writes. When a transient failure occurs, Hugo retries the request up to three times internally, but the retry count is not configurable via the command line.
The Force flag available on resources.GetRemote bypasses the cache and forces a fresh download; using it on a retry overwrites the cached file rather than creating a new output, which prevents duplicate public files but may negate caching benefits. Running concurrent Hugo builds without isolated cache directories can introduce race conditions that might lead to duplicate writes.
Uncertainty remains about whether a retry that succeeds after cache use still results in a single file in the public directory, and whether the Force flag guarantees no duplicate output when retries happen in parallel environments.
resources.GetRemote with Force set to false produce a second copy of the resource in the published output?Force to true on a retry always overwrite the cached file without creating a duplicate public file, and how does cache directory isolation affect this in concurrent Hugo builds?34775 reputation · 24 Feb 2022, 10:37 UTC
The premise needs correcting before the questions can be answered: as of current Hugo releases, resources.GetRemote has no Force option and no documented built-in retry loop. It either returns the remote resource, serves it from the on-disk cache, or returns an error. Because of that, the "retry overwrites the cached file" scenario described in the question does not match how the function actually behaves, and duplicate files in public/ are not a normal outcome of any of these paths.
Well-established behavior:
hugoresources/_gen or the configured caches.getremote.dir), keyed by a hash of the URL plus the options map.caches.getremote.maxAge. A value of -1 keeps entries indefinitely; 0 effectively disables reuse, which is the closest thing to a "force" flag.Uncertain / version-dependent: exact defaults, config key names, and error-vs-non-2xx handling vary across Hugo versions. Verify against your installed version (hugo version) before relying on specifics.
public/, the more likely causes are calling GetRemote with different option maps (different cache keys, published under different hashed filenames) or publishing the resource under two different paths in templates.hugo --gc, or deleting the getremote cache directory) or setting maxAge: 0. These affect fetching, not output naming — the published filename derives from the resource path, so a re-fetch replaces the same output file rather than duplicating it. For concurrent builds, the real risk is two processes sharing one cache directory; give each build its own cache dir (via caches.getremote.dir or --cacheDir) to avoid contention. Even then, cache races affect fetch integrity, not the count of files in public/.hugo version
hugo --logLevel info # watch whether GetRemote hits the network or cache
hugo --gc # clear caches, then rebuild and compare outputIf you are genuinely observing duplicate files, the one missing diagnostic that would change this answer is the actual template call: share the resources.GetRemote invocation (URL and options map) and the two differing output paths, since distinct option maps are the most plausible duplicate source.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 24 Feb 2022, 18:56 UTC
To build on the previous point regarding the lack of a Force flag, it is important to clarify how Hugo actually handles cache invalidation. Because resources.GetRemote uses a hash of the URL and the options map as the cache key, any change to the request parameters—such as modifying a header or the request body—will result in a different key.
This behavior means that if you need to bypass the cache programmatically without a dedicated flag, you can introduce a dynamic variable into the options map. For example:
{{ $res := resources.GetRemote "https://example.com/image.jpg" (dict "headers" (dict "X-Cache-Bust" .Site.BuildDate)) }}While this achieves a fresh fetch, be aware that this creates a new entry in the cache directory rather than overwriting the old one. In high-frequency build environments, this can lead to cache bloat unless the caches.getremote.maxAge is configured to prune old assets.