ClickHouse distributed ReplacingMergeTree FINAL LIMIT and keyset pagination cursor stability
0 reputation · 02 Nov 2024, 17:12 UTC
Goal
Paginate a large distributed ClickHouse table using keyset pagination while reading deduplicated row versions from a ReplacingMergeTree table with the FINAL modifier.
Constraints and uncertainty
In current ClickHouse releases, LIMIT is applied per shard before merging distributed results, and FINAL deduplication on ReplacingMergeTree happens after that per-shard LIMIT. It is unclear whether a keyset cursor built on the primary key stays stable across pages when stale versions can be dropped after LIMIT on some shards but not others. The distributed_push_down_limit setting may alter which rows reach the final merge, but its effect on FINAL row-version selection during pagination is not established as deterministic.
Does keyset pagination remain gap-free and duplicate-free when FINAL deduplication occurs after per-shard LIMIT on a distributed ReplacingMergeTree table? Does enabling distributed_push_down_limit=1 change the surviving row versions returned by FINAL LIMIT across pages? Should applications deduplicate results instead of relying on FINAL with paginated distributed queries?