Make Your Jupyter Notebook Prove It Runs: A CI Gate with nbconvert

The stale-state trap

You open a notebook, run a few cells to check a plot, tweak a parameter in an earlier cell, and save. The file on disk now contains outputs from a session you assembled by hand. Tomorrow, a colleague—or a CI runner—opens the same file and runs it top to bottom. It fails, or worse, it succeeds with different numbers.

The root cause is that a .ipynb file stores source code, outputs, and execution counts, but it does not store the kernel state that produced those outputs. Variables, imported modules, open file handles, and random seeds all live in memory. When you execute cells out of order, the saved outputs become a record of your exploration, not a guarantee that the notebook works as written.

What nbconvert --execute actually does

The jupyter nbconvert --to notebook --execute command treats the notebook as a script. It launches a fresh kernel, executes every cell from top to bottom in document order, and writes the resulting outputs back into a notebook file. Because the kernel starts empty, any hidden dependency on earlier interactive state disappears. If a cell raises an unhandled exception, nbconvert stops and exits with a non-zero status, which makes it usable as a pass/fail merge gate.

This is different from simply re-running the notebook in JupyterLab. The conversion step is headless and deterministic: no browser, no toolbar, no manual cell selection.

A worked CI command

Run the following on a CI runner or in a clean virtual environment where the notebook's dependencies are already installed. Replace analysis.ipynb with your notebook path and python3 with the kernel name declared in the notebook metadata.

jupyter nbconvert --to notebook --execute \
  --output-dir executed/ \
  analysis.ipynb \
  --ExecutePreprocessor.timeout=600 \
  --ExecutePreprocessor.kernel_name=python3

Here is what each part contributes:

• --to notebook --execute: convert the notebook to the same format, but run it first.
• --output-dir executed/: write the executed copy to a subdirectory instead of overwriting the source. In pull-request checks, avoid --inplace unless you intentionally want to mutate the tracked file.
• --ExecutePreprocessor.timeout=600: allow up to 600 seconds per cell. The default may be too short for data loading or model fitting.
• --ExecutePreprocessor.kernel_name=python3: pin the kernel so CI does not silently pick a different Python environment.

Expected checks: the process should exit successfully. Reopen the executed copy and confirm that execution counts increase monotonically from the first cell to the last. Run git diff -- analysis.ipynb before and after execution; if only outputs and execution counts changed, the gate is behaving as intended. Unexpected source edits mean the notebook is mutating its own code or files.

Risk: the --inplace flag overwrites the input notebook. If the original is the source of truth, copy it first or use --output-dir.

Trade-offs and limitations

A clean top-to-bottom run proves executability in one environment. It does not prove numerical correctness, model quality, or that hidden filesystem state was absent.

Cost and fragility

Long training loops, large downloads, and expensive preprocessing will make the gate slow. Keep heavy work behind a cached artifact or move it to a separate script. Widgets, input() prompts, and GUI windows will hang or fail under a headless kernel. Use absolute paths rather than relative paths, because the CI working directory may differ from your laptop.

Secrets in outputs

Executed notebooks embed their outputs. If a cell prints an API token from an environment variable, a private file path, or a large model dump, that data lands in the committed artifact. Scrub or strip outputs before committing, and run CI with sanitized environment variables.

Version drift

nbconvert is packaged separately from JupyterLab and the classic Notebook. CLI trait names such as ExecutePreprocessor.timeout can change across major releases. Pin the nbconvert version in your CI environment file so the gate does not break when a runner image updates.

Headless display assumptions

Notebooks that assume a display, clipboard, or browser may need adjustment. For plotting, set %matplotlib inline or a non-interactive backend such as Agg in the first cell, and verify that image outputs are embedded rather than opening windows.

An actionable checklist

1. Pin nbconvert and the kernel in your environment lockfile.
2. Run the execution command with --output-dir in pull-request checks; reserve --inplace for deliberate artifact updates.
3. Set --ExecutePreprocessor.timeout generously for your slowest cell.
4. Print sys.executable and key package versions in the first cell, then compare them with the lockfile.
5. Test the gate negatively: copy the notebook, add a cell that raises an exception, and confirm the pipeline fails with a traceback.
6. Review the executed diff for leaked secrets or unexpectedly large outputs before merging.

Treating the notebook as an executable artifact rather than a session recording changes the conversation from "it worked on my machine" to "it ran cleanly in the pinned environment."