SPSS Statistics ↔ Python Integration Plug-in: Where Does the Data-Materialization Bottleneck Sit When Crossing the Session Boundary?

Answer

The bottleneck lies in the materialization step where SPSS streams the active dataset into Python objects via spssdata.Spssdata(). The wall‑clock time and peak memory grow linearly with the number of cases that survive the current session state (filters, weights, TEMPORARY) multiplied by the number of variables, because each case is turned into a Python tuple/list and allocated in the interpreter’s heap. The SPSS side merely reads the case cache; the costly work is Python object creation and memory allocation.

Confirmed facts

• The spssdata.Spssdata iterator respects active FILTER, WEIGHT, and TEMPORARY selections, so the measured cost already reflects the exact subset that would cross the boundary.
• Memory measured as process RSS (e.g., via psutil.Process().memory_info().rss) includes interpreter overhead; subtracting a baseline RSS taken before the snippet isolates the pure data copy.
• Round‑trip cost adds a second crossing when results are written back with spss.Submit; time that separately if needed.

Likely explanation

SPSS internally streams cases from its case cache without building Python objects; the crossing to Python forces each case into a native Python object, which dominates the elapsed time and memory usage. Therefore the bottleneck is on the Python side of the boundary, not in the SPSS scanning phase.

Minimal measurement snippet

BEGIN PROGRAM PYTHON.
import time, spss, spssdata, psutil
start = time.perf_counter()
# materialize active dataset
data = [row for row in spssdata.Spssdata()]
elapsed = time.perf_counter() - start
peak_mem = psutil.Process().memory_info().rss
print('time_sec', elapsed, 'peak_mem_bytes', peak_mem)
END PROGRAM.

Accounting for session state

1. Apply any FILTER, WEIGHT, or TEMPORARY commands in the SPSS syntax **before** the BEGIN PROGRAM block.
2. Run the snippet immediately after those commands; the Spssdata iterator will yield only the cases that survive the current state, so the measured time and memory reflect the production workload.
3. If you need to isolate the effect of a particular filter, repeat the measurement with the filter toggled and compare the deltas.

Representative SPSS streaming baseline

Run a simple frequency table that does not request any statistics:

FREQUENCIES VARIABLES=all
  /FORMAT=NOTABLE
  /STATISTICS=NONE.

Capture its elapsed time via OMS OUTPUT EXPORT or the SPSS timer; this approximates the pure data‑scan cost without materializing rows into Python objects, providing a baseline for comparison.

Missing diagnostic detail

To refine the recommendation, please confirm the exact IBM SPSS Statistics version and the bundled Python interpreter version (e.g., Statistics 29.0 with Python 3.10). The measurement can shift between releases because the spssdata module and interpreter optimizations evolve.