DataFrames.jl lazy subsetting and index retention during pagination

Memory and Indexing Behavior in Julia 1.10

With the introduction of lazy iteration for subset operations in Julia 1.10, slicing a DataFrame using df[range, :] returns a SubDataFrame view rather than a materialized copy. This change optimizes memory for large dataset bounding but introduces specific behaviors when implementing pagination logic.

When using Iterators.partition(eachrow(df), pagesize) to paginate a dataset, each partition materializes rows as vectors of NamedTuples. This process may negate the memory benefits of the lazy view if page sizes are large. Furthermore, these paginated views retain the original row indices from the parent DataFrame rather than resetting to a 1-based sequence for the current page.

For applications requiring consecutive indices per page or strict memory bounds during materialization, the current behavior creates a discrepancy between the view's structure and the paginated output.

• Does the current SubDataFrame implementation provide a mechanism to reset indices without triggering a full data copy?
• How should pagination be structured to maintain lazy evaluation while avoiding the memory spikes associated with eachrow materialization?