Variable Explorer truncation threshold for large DataFrames in Spyder
0 reputation · 25 Jan 2020, 11:05 UTC
0 reputation · 25 Jan 2020, 11:05 UTC
Spyder's Variable Explorer displays a compact summary (shape, dtype, short preview) for large pandas DataFrames and NumPy arrays instead of full contents. The DataFrame and Array viewers then render rows lazily as the user scrolls, which keeps the IDE responsive.
No user-facing preference controls the size threshold at which the Value column switches from a full representation to the truncated summary. The Preferences dialog offers filters for variable names and types, but not a configurable row or byte limit for the preview. Consequently, users cannot tune the trade-off between immediate visibility and console responsiveness without resorting to manual slicing (for example, df.head(n)) and inspecting the derived variable.
The Variable Explorer was substantially rewritten in the Spyder 4 series, and viewer internals have continued to evolve. Exact truncation behavior may differ between releases and is not guaranteed by the bundled documentation.
What is the current default truncation threshold for the Value column in Spyder 5.x and 6.x? Is there a hidden configuration option or environment variable that adjusts this limit? Does the lazy viewer impose any upper bound on the number of rows it will attempt to render for a single variable?
34775 reputation · 25 Jan 2020, 22:11 UTC
In Spyder 5.x and 6.x the Value column of the Variable Explorer shows a short preview (shape, dtype and a few rows) instead of the full repr() when the underlying object is considered “large”. The IDE uses an internal heuristic that switches to this compact view when the rendered representation would exceed a few kilobytes or a few hundred rows. That number is not exposed in the Preferences dialog and there is no public documentation that pinpoints the exact threshold.
Neither the Preferences pane nor any documented configuration file or environment variable contains a key that lets you adjust the row/byte limit for the Value column. A quick scan of the default spyder.ini and the ~/.spyder-py3 configuration directories shows no such option.
When you double‑click a variable, Spyder opens the dedicated DataFrame/Array viewer. This viewer renders rows lazily – it only processes the rows that are currently visible in the scroll area. There is no hard cap on the number of rows that the viewer will eventually display; it will attempt to render the whole table, but the operation is throttled by the available RAM and the kernel’s ability to provide the data on demand.
df.head(1000)) and inspect the slice; the Variable Explorer will show the full repr() for the smaller object.df.info() or df.describe() in the console for metadata without pulling the whole table into the UI.df.to_csv()) and open it with an external viewer.If you notice a specific size at which the switch to the compact preview occurs in your installation, let us know the exact Spyder version and the DataFrame’s shape. That information can help refine the description of the internal heuristic.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.