DataSpell's Notebook-as-Code Architecture: How Bidirectional Sync Works Without Dual-Execution Divergence

Requirements: Version Control and Refactoring for Notebooks

Jupyter notebooks excel at exploratory analysis but fail at software engineering basics: git diff on .ipynb produces JSON noise, refactoring tools can't traverse cell boundaries, and code search stops at cell edges. Teams need notebooks to behave like code — diffable, searchable, refactorable — without sacrificing the interactive kernel workflow.

DataSpell solves this by maintaining a single source of truth (.ipynb) while presenting a derived, synchronized Python script (.py) that participates in standard tooling. The architecture accepts a hard constraint: the kernel executes only the notebook. The script is a view, not a second runtime.

Smallest Suitable Design: Deterministic Cell-to-Script Mapping

The sync engine uses a lossless, deterministic transformation:

• Markdown cells → # %% [markdown] comment blocks
• Code cells → # %% delimited regions (optionally wrapped as functions for symbol navigation)
• Cell metadata (tags, execution count) → special comments: # %% tags=[\"hide_input\"]
• Cell UUIDs → embedded in comments to preserve identity across round-trips

This mapping is bijective for standard constructs. The .py file regenerates identically from the same .ipynb every time — no formatting drift, no timestamp noise.

Trust and Data Boundaries: Kernel Owns Execution

The critical boundary: the Jupyter kernel never executes the .py file. When you run a cell, DataSpell sends the cell's content from the .ipynb to the kernel. The .py exists solely for:

• Editing in a familiar script UI
• git diff and code review
• IDE code navigation (go-to-definition, find-usages across cells)
• Refactoring operations that span multiple cells
• External tooling (linters, type checkers) that expect .py

This avoids dual-execution divergence — a common failure mode when both representations are independently runnable.

Operational Checks: Verifying Sync Integrity

1. Round-Trip Fidelity Test

# In DataSpell terminal or external shell
# Assumes notebook.ipynb exists and is committed
dataspell export notebook.ipynb --to-script notebook_exported.py
# Import back (requires DataSpell UI: File → Import → Python Script)
# Compare AST structure: cell count, types, order, UUIDs

Run this after any major sync operation. The imported notebook must match the original in cell sequence, types, and UUIDs. Metadata like execution counts and outputs are intentionally lost — they belong to the kernel session, not source control.

2. Cell-ID Stability

Each cell carries a UUID in .ipynb metadata. The sync engine preserves these in .py comments:

# %% [markdown] uuid="a1b2c3d4-..."
# ## Section Title

# %% uuid="e5f6g7h8-..."
def process_data(df):
    return df.clean()

If a git diff on .py shows UUID changes without cell content changes, the sync is corrupt. Trigger Reload from Notebook (right-click .py tab → Reload from Notebook) to regenerate.

3. VCS Diff Sanity

git diff notebook.py

Expected: logical edits only (added lines, changed expressions). Failure signal: wholesale rewrite, marker reordering, or UUID churn. This indicates external .py modification or sync bug.

Failure Modes and Mitigations

Concurrent External Edit (Git Merge on .py)

Symptom: Teammate merges .py directly; your local .py diverges from .ipynb.

Resolution: Discard .py changes. In DataSpell: right-click the .py editor tab → Reload from Notebook. This regenerates .py from the authoritative .ipynb. Then commit the regenerated .py.

Policy: Never resolve conflicts in .py. Resolve in .ipynb using DataSpell's built-in notebook merger or nbdime, then regenerate.

Non-Round-Trippable Constructs

IPython magics (%timeit, %%bash), widget state, and multi-language cells (%%sql) have no Python equivalent. The sync engine embeds them as annotated comments:

# %% [magic] kind="timeit"
# %timeit expensive_computation()

# %% [widget] state="{\"value\": 42}"
# slider = IntSlider(value=42)

The UI flags these cells with a warning badge. Editing the comment does not update the notebook — you must edit the source cell in .ipynb.

Large Notebook Latency (>5k Lines)

Per-cell dirty tracking limits recomputation. Only changed cells and their script regions update. If latency appears, check Help → Diagnostic Tools → Sync Performance for per-cell timings. Splitting the notebook is the structural fix.

Conditions That Would Change the Design

The current architecture assumes:

1. Jupyter protocol remains notebook-first (JSON + kernel protocol)
2. Kernel-side execution of .py is non-standard
3. Collaborative editing is sequential (git-based), not real-time

Design would shift if:

• Jupyter adopts a native script-first format (e.g., Jupytext's py:percent becomes the canonical on-disk format with kernel protocol support). Then .ipynb becomes the derived view.
• Kernel-side .py execution standardizes (e.g., Jupytext nbfmt integration in IPython). The trust boundary dissolves; both representations become executable.
• Real-time collaborative editing requires CRDTs on both representations simultaneously. Current event-driven sync cannot merge concurrent edits from multiple users without a conflict-free replicated data type spanning both formats.

Practical Verification Checklist

After enabling View → Tool Windows → Python Script in DataSpell 2024.2+:

1. Edit a markdown cell → verify .py comment block updates instantly.
2. Edit a function signature in .py → return focus to notebook → verify cell content reflects change.
3. Run git diff on committed .py after notebook edits → confirm logical-only diff.
4. Trigger Reload from Notebook after external .py edit → verify byte-identical regeneration.
5. Export → Import round-trip → compare cell count, types, order, UUIDs.

Limitations to Accept

• .py is not runnable with python script.py — magics, widgets, and kernel bootstrap are absent.
• Cell outputs and execution counts live only in .ipynb; they are not synced.
• Cross-cell refactorings (rename, extract method) update .py and .ipynb but require kernel restart to refresh runtime state.
• Git merges on .py are an anti-pattern; the workflow assumes .ipynb as merge target.

The architecture trades script executability for notebook fidelity. That trade is intentional: it keeps the interactive workflow intact while giving version control and refactoring tools a usable surface.