Run Jupyter Notebooks Headlessly in CI: A Practical Guide with nbconvert --execute

Desired Outcome

Execute a Jupyter notebook non‑interactively, capture all outputs and tracebacks in a new notebook file, and use the exit status to gate continuous‑integration (CI) jobs.

Prerequisites

• Jupyter & nbconvert – Installed in the same virtual environment that will run the CI job. Verify with jupyter nbconvert --version.
• Kernel registration – The notebook’s kernelspec (e.g., python3) must be available: jupyter kernelspec list should list it.
• File paths – The notebook and any data files it references must be reachable from the working directory where the command runs. Relative paths resolve against the notebook’s location.
• Permissions – The CI runner must have read/write access to the notebook, output directory, and any dependent resources.

Procedure

1. Verify the installed tools:

   jupyter nbconvert --version
   jupyter kernelspec list
   

   The output should show a recent nbconvert (≥ 6.0) and the expected kernel.

2. Create a minimal test notebook (test.ipynb) containing:

   # test.ipynb
   print("Hello, world!")
   

   This notebook will serve as a sanity check.

3. Run the notebook headlessly:

   jupyter nbconvert \
       --to notebook \
       --execute \
       --output test_executed.ipynb \
       --ExecutePreprocessor.timeout=600 \
       test.ipynb
   

   Where: --to notebook keeps the Jupyter format; --execute runs all cells; --output names the resulting file; --ExecutePreprocessor.timeout sets a 10‑minute timeout per cell.

4. Validate the result:

   • Check exit code: echo $? should be 0 if all cells succeeded.
   • Open test_executed.ipynb and confirm the first cell’s outputs array contains the string Hello, world!.
   • If a cell fails, the traceback is embedded in its outputs and the process exits with a non‑zero status.

5. Integrate into CI (example GitHub Actions step):

   - name: Execute notebooks
     run: |
       jupyter nbconvert \
           --to notebook \
           --execute \
           --output-dir ./executed \
           --ExecutePreprocessor.timeout=600 \
           **/*.ipynb
     continue-on-error: false
   

   This step will fail the job if any notebook throws an error.

6. Optional: allow errors but still record tracebacks:

   jupyter nbconvert \
       --to notebook \
       --execute \
       --allow-errors \
       --output-dir ./executed \
       my_notebook.ipynb
   

   The command continues past failing cells, but the exit status remains non‑zero.

Expected Checks

• Non‑zero exit status indicates a cell raised an exception. Use --allow-errors if you want to proceed.
• Presence of output notebook in the specified directory.
• Each cell’s outputs array should contain either output data or a traceback.
• Verify that the kernel used matches the notebook’s kernelspec by inspecting the metadata.kernelspec field.

Recovery Options

• Increase timeout – If a cell hangs, raise --ExecutePreprocessor.timeout or set to -1 to disable the limit.
• Debug mode – Add --debug to get verbose logs that include the exact cell causing failure.
• Parameter injection – For notebooks that need input values, consider papermill instead of nbconvert.
• Sandboxing – Run the command in a container or virtual machine to avoid executing arbitrary code on the host.

Limitations

• nbconvert executes the notebook as-is; it does not support parameter passing or environment variable injection beyond what the notebook already uses.
• Execution state is not preserved across runs – each cell starts from a clean kernel.
• Untrusted notebooks will run arbitrary code with the CI runner’s privileges; always review or sandbox before execution.

Quick Comparison: nbconvert vs Papermill

Feature	nbconvert	papermill
Parameter injection	No	Yes (via --parameters)
Execution mode	Headless, single run	Headless, with parameterization
Output format	Notebook (.ipynb)	Notebook (.ipynb)
Exit status on error	Non‑zero	Non‑zero (unless --allow-errors)
Typical use case	CI gating, batch runs	Parameterized reports, pipelines

Practical Verification Checklist

• Run jupyter nbconvert --help to confirm flag names in your version.
• Create a notebook with a deliberate error (e.g., raise ValueError("test")) and ensure the exit code is non‑zero.
• Measure execution time of a long cell; if it exceeds 30 seconds, adjust --ExecutePreprocessor.timeout.
• After CI integration, inspect the job log for the CellExecutionError message if a failure occurs.