Extract Jupyter Notebook Code to Scripts Without Copy-Paste

The Problem: Copy-Paste Is Not a Deployment Strategy

Your notebook calculates a monthly cohort table. It works, but now the data engineering team needs the same logic in a scheduled script. You select cells, copy them into etl.py, and run it. Something breaks: a variable was defined two cells up, a display call fails outside the browser, or an import was hidden in a markdown note. You spend the afternoon debugging transcription errors instead of the actual pipeline.

Thesis: Extract and Verify Inside Jupyter

Jupyter's cell magic system lets you save code directly to disk and time it before you commit to the extraction. %%writefile writes a cell's contents to a file, while %timeit measures execution without manual timer code. You can move from exploration to a standalone script without leaving the notebook.

Step 1: Benchmark Before You Extract

Run %timeit on the expression you plan to move. Jupyter executes it repeatedly and reports a mean with standard deviation, so you know whether the code is fast enough for a scheduled job.

%timeit sum(range(10_000_000))

Where to run: Any Jupyter code cell. Permissions: None. Risk: None; it is read-only analysis.

Step 2: Save the Cell to a Python File

Use %%writefile as the first line of a cell. Everything below it in that cell is written to disk exactly as typed.

%%writefile cohort_etl.py
import pandas as pd

def build_cohort_table(df):
    return df.groupby('cohort_month').size()

if __name__ == '__main__':
    data = pd.read_parquet('s3://bucket/events.parquet')
    print(build_cohort_table(data))

This creates cohort_etl.py in the notebook's working directory. Permissions: Write access to the current directory. Risk: It silently overwrites an existing file with the same name.

Step 3: Run the Script Outside Jupyter

From a Jupyter cell, run the saved file with a shell escape:

!python cohort_etl.py

If the script fails, the error appears in the cell output. Fix the notebook cell, rewrite the file, and test again. Once it runs cleanly, you can copy cohort_etl.py into your repository or CI pipeline.

Trade-Offs and Limitations

• %%writefile captures only the current cell. Imports and helper functions defined in other cells must be included manually or imported inside the script.
• Jupyter magics such as %timeit do not run in standard Python. Remove them before deploying the script.
• Large DataFrames or plot objects inside the cell are saved as code, not output, so they will execute again when the script runs.

Practical Verification

After writing the file, list it to confirm persistence:

!ls -l cohort_etl.py

Then execute it and compare the output against the notebook result. If the numbers match, the extraction is trustworthy.

Next Step

Pick your longest-running notebook cell, time it with %timeit, and write it to a file with %%writefile. Running that file outside Jupyter is the fastest way to prove the logic is portable.