Spyder Variable Explorer Architecture: Design, Trust Boundaries, and Failure Modes

Requirements

The Variable Explorer must let a user inspect live Python objects from the IPython kernel without executing arbitrary code in the frontend. It needs to work for built‑in types, NumPy arrays, pandas objects, and user‑defined classes while keeping the UI responsive even when objects are large or costly to serialize. The solution must survive kernel restarts, crashes, and objects that raise exceptions during introspection.

Smallest Suitable Design

Three layers satisfy those requirements:

1. Kernel‑side plugin (spyder_kernels.plugins.variable_explorer) walks user_ns, applies user‑configured filters (e.g., hide private/uppercase names), and calls spyder_kernels.utils.serialize.serialize_object for each variable.
2. ZMQ transport uses the Jupyter protocol (variable_explorer_request/variable_explorer_reply). Each reply is limited to ~10 MiB by default; larger payloads are split only when the user explicitly requests a full export.
3. Qt frontend (VariableExplorerModel and VariableExplorerWidget) builds a tree/table view from the JSON‑safe payload. Child nodes for containers are loaded lazily: the first reply includes length and a preview of the first 100 items; expanding a node triggers a follow‑up request with a path index.

Trust/Data Boundary

The kernel is considered fully trusted because it executes the user’s code. The frontend treats every field received over ZMQ as untrusted display data: it shows type names, shapes, dtypes, and scalar previews but never evaluates or executes the received content. No executable code, pickles, or byte‑code crosses the boundary.

Operational Checks

• Liveness: the frontend periodically sends kernel_info_request and expects a heartbeat reply; missing replies trigger a “Kernel not responding” banner.
• Serialization errors: serialize_object wraps each object in a try/except block. On failure it returns a placeholder object with the exception type; the UI shows a tooltip like <serialization failed: TypeError>.
• Stale data detection: the kernel sends a monotonic protocol version with each reply. If the version changes (e.g., after a kernel restart), the frontend clears the view and shows a “Kernel restarted, variables cleared” banner.

Failure Modes

1. Non‑serializable or recursive objects: objects with a broken __repr__, __len__, or deep recursion can cause the serialization function to hang. A per‑object timeout (default 2 s) and a recursion depth limit (default 50) abort the attempt and return the failure placeholder.
2. Massive objects: a 10 GB NumPy array exceeds the ZMQ limit. The kernel sends only metadata (shape, dtype) and a preview; the full data is transferred only when the user clicks “Export to file”, which streams the array in chunks.
3. Kernel crash during introspection: if the kernel dies while the Variable Explorer is building a reply, the frontend receives no response and displays the stale‑data warning after the heartbeat timeout. A manual namespace refresh restores consistency.

Conditions That Would Change the Design

• Native Jupyter variable introspection: if the Jupyter protocol added a standard variable_list message, Spyder could drop the custom variable_explorer_request and rely on the kernel’s built‑in introspection, reducing maintenance overhead.
• Stricter kernel sandboxing: moving the kernel to a heavily sandboxed environment (e.g., gVisor) would prevent direct access to user_ns. The design would need capability‑based handles and a proxy service that serializes only whitelisted attributes.
• Frontend migration to a web stack: replacing Qt with a browser‑based UI would swap the Qt model/view for a virtualized JSON stream (e.g., React‑Virtualized). The core serialization and transport layers could remain unchanged, but the lazy‑loading heuristic would need to adapt to incremental DOM rendering.

Practical Engineering Decision: Lazy Child Loading

To keep the initial payload small, the kernel serializes containers as:

{
  "type": "ndarray",
  "shape": [1000000],
  "dtype": "float64",
  "preview": [0.0, 0.1, …, 0.9],   # first 100 items
  "len": 1000000
}

When the user clicks the expander for that node, the frontend sends a follow‑up request containing the variable name and a path index (e.g., ["my_array", "slice", 0, 1000]). The kernel then serializes only the requested slice. This approach guarantees that the first reply stays well under the 10 MiB limit even for very large nested structures.

Verification Steps

1. Version match: In Spyder’s internal IPython console run

   import spyder_kernels; print(spyder_kernels.__version__)
   

   and compare the first two components with the output of spyder --version in a terminal. A mismatch disables the Variable Explorer until you run conda install spyder-kernels=<version>.
2. Size limit: Open Preferences → IPython Console → Advanced → Variable Explorer and confirm that Max variable size (MB) is set to 10 (the default). You can raise it temporarily to test behavior, but note that values far above 10 MiB may cause UI freezes on slow networks.
3. Large array test: Execute

   import numpy as np
   big = np.ones((5_000_000,), dtype='float64')  # ~40 MiB

   The Variable Explorer should show type: ndarray, shape: (5000000,), dtype: float64 and a preview of the first 100 values. Clicking “Export to file” should produce a CSV/HDF5 without freezing the UI.
4. Broken __repr__ test: Define

   class Bad:
       def __repr__(self):
           raise ValueError('intentional')
   bad = Bad()
   

   The Variable Explorer will list bad with a tooltip reading <serialization failed: ValueError>, confirming per‑object error isolation.
5. Kernel restart detection: Click the restart button in the IPython console toolbar. After the kernel restarts, the Variable Explorer banner should read Kernel restarted, variables cleared and the view should be empty.

Limitations

The current design assumes that the kernel can safely import spyder_kernels and that the user’s environment permits ZMQ communication on the loopback interface. In highly restricted containers (e.g., strict SELinux policies) the ZMQ channel may be blocked, causing the Variable Explorer to stay empty. There is no automatic refresh on every variable assignment; users must rely on explicit refresh, cell execution completion, or debugger pauses to see updates, which can lead to stale reads if background threads modify user_ns.