Cypress version pinning: binary cache or Docker image for a repeatable test environment

1. Is a pinned package‑lock entry plus a controlled cache sufficient?

Yes. When you lock the Cypress npm version in package-lock.json and force a deterministic cache location, the same binary is downloaded every run, giving bit‑for‑bit repeatability.

Confirmed steps

1. Pin the Cypress version, e.g. npm install cypress@13.6.0 --save-exact (or use --save with an exact version in package.json).
2. Set a CI‑specific cache folder: export CYPRESS_CACHE_FOLDER=$RUNNER_TEMP/cypress-cache (or any directory that is cleared for each job).
3. Force a fresh download to avoid relying on a pre‑populated cache: export CYPRESS_INSTALL_BINARY=1. If you prefer to pre‑populate the cache, set CYPRESS_INSTALL_BINARY=0 and ensure the folder already contains the exact binary.
4. Disable any overrides that could skip the download: unset CYPRESS_DOWNLOAD_SKIP and unset CYPRESS_NODE_BINARY.
5. Run npm ci (or npm install) in a clean workspace.

2. Which cache‑related environment variables must be fixed in CI to avoid silent drift?

The variables that influence where and whether the binary is fetched are:

• CYPRESS_CACHE_FOLDER – must point to a dedicated, isolated directory per job.
• CYPRESS_INSTALL_BINARY – set to 1 to guarantee a download, or 0 only if you have verified the cache already holds the correct binary.
• CYPRESS_DOWNLOAD_SKIP – should be unset or explicitly 0.
• CYPRESS_NODE_BINARY – should be unset unless you intentionally want to use a custom Node binary.

Fixing these four ensures the same binary is used regardless of the worker’s global ~/.cache/Cypress state.

3. How should we handle Chrome version skew if we stay outside Docker?

Cypress bundles Electron, which has its own Chromium base. To match the Chrome version used by the official Docker images you have two practical options:

1. Install the exact Chrome version that the image pins (e.g., for cypress/included:13.6.0 the Chrome version is 114.0.5735.90). On Ubuntu you can do:

# Example: install Chrome 114.0.5735.90
wget -q https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_114.0.5735.90-1_amd64.deb
sudo dpkg -i google-chrome-stable_114.0.5735.90-1_amd64.deb
sudo apt-get install -f  # resolve dependencies

1. Alternatively, use a lightweight Chrome binary such as chrome-aws-lambda (or playwright-chromium) and launch Cypress with cypress open --browser chrome pointing to that binary via CYPRESS_browser or the browser configuration.

If you cannot lock Chrome, accept that Electron‑based tests may diverge in CSS, JavaScript engine, or security features. In that case run a subset of critical tests against real Chrome (installed as above) to validate fidelity.

Verification (optional but recommended)

• Check the binary version: npx cypress version and npx cypress info.
• Confirm the binary’s SHA‑256: shasum -a 256 $(npx cypress cache path)/cypress and compare to a stored hash.
• Verify Chrome version inside the test environment: google-chrome --version.
• Run a simple smoke test (e.g., visit https://example.com and assert an element) in both the npm‑cache setup and the Docker image; identical outcomes indicate repeatability.