DataSpell Virtual Environment ↔ Docker Container: Synchronizing Python Packages

Short answer

DataSpell does not automatically rebuild a local virtual environment when requirements.txt, pyproject.toml, or environment.yml change externally. Docker containers launched from the IDE also have no built‑in way to mirror the local venv’s package list on startup. There is no observer mechanism inside DataSpell that propagates package installations between the host venv and a running container. The reliable pattern is to keep a single source‑of‑truth file (requirements.txt or a lock file), create the local venv from it manually, and rebuild the Docker image whenever that file changes.

Why the IDE behaves this way (likely explanation)

• JetBrains IDEs use a virtual file system (VFS) watcher that only refreshes indexes and shows a “File changed externally” notification. It does not execute pip, conda, or Docker commands.
• Docker integration in DataSpell is limited to running docker compose commands via Run Configurations; it cannot inject the host venv into a container at container start.
• Containers are intended to be immutable images. The source of truth must remain the dependency file on the host, not a mutable venv inside a container.

Confirmed facts (from current DataSpell 2024.x behavior)

1. Creating a venv from requirements.txt or environment.yml is a one‑time action (Tools ▸ Python Packages ▸ “Add requirements.txt” or “Create Conda Environment”).
2. External edits to the dependency file trigger only a VFS refresh; you must manually run “Sync Python Packages” or conda env update --file environment.yml --prune.
3. A Dockerfile that COPY requirements.txt . and RUN pip install -r requirements.txt is the standard way to make the container reflect the same packages. The image must be rebuilt (docker compose build) after any change.
4. No built‑in file‑watcher → Docker rebuild pipeline exists. External tools (e.g., entr, watchexec, pre‑commit hooks) can automate the rebuild.
5. Mounting the host venv into a container (-v ~/.virtualenvs:/venv) breaks portability and isolation; avoid for production‑like workflows.

Practical workflow (only steps needed)

1. Define dependencies once – keep a requirements.txt (or pyproject.toml + poetry.lock/pip-tools lock) at the project root.
2. Create the local venv – in DataSpell: File ▸ Settings ▸ Project ▸ Python Interpreter ▸ Add Interpreter ▸ Virtualenv Environment and point to the same requirements.txt. Run “Sync Python Packages” after any manual edit.
3. Write a minimal Dockerfile:

   FROM python:3.11-slim
   WORKDIR /app
   COPY requirements.txt .
   RUN pip install --no-cache-dir -r requirements.txt
   COPY . .
   

4. Add a docker‑compose.yml (optional) that references the Dockerfile and any services.
5. Create a DataSpell Run Configuration of type “Docker‑Compose” with command up --build. Execute it whenever you change requirements.txt.
6. Automate rebuild (optional) – add a file watcher script, e.g.:

   # watch-rebuild.sh
   watchexec -w requirements.txt -w pyproject.toml "docker compose build"
   

   Run this script in a terminal while developing.

One diagnostic detail that could change the recommendation

Are you using conda (environment.yml) or pure pip/poetry for the local environment? Conda environments require conda env update --file environment.yml --prune and cannot be reproduced inside a container by simply copying the file; you would need a conda‑based base image or export a lock file (conda-lock) for Docker.

Verification checklist

• Open DataSpell → Settings → Build, Execution, Deployment → Docker → confirm daemon connection and Compose support.
• Create a test project with a requirements.txt, use Tools → Python Packages → “Add requirements.txt” → observe the manual sync prompt.
• Build the Dockerfile with docker compose build from a Run Configuration; verify the container has the expected packages (pip list inside the container).
• Edit requirements.txt externally → confirm DataSpell only shows “File changed externally” and does not run pip install.
• Check JetBrains YouTrack for open feature requests such as “Auto‑recreate venv on requirements.txt change” or “Docker compose auto‑rebuild on file watch”.