Spark AQE post-shuffle partition count target

Goal: Constrain the number of partitions produced after an Apache Spark Adaptive Query Execution shuffle coalescing stage in Spark 3.x.

spark.sql.adaptive.coalescePartitions.enabled treats spark.sql.adaptive.advisoryPartitionSizeInBytes as a size target, not a fixed partition count. The resulting partition count depends on runtime data statistics and skew and may differ from spark.sql.shuffle.partitions. Downstream workloads that require a stable partition count or exact output file count lack a deterministic boundary.

It is unclear whether a stable post-shuffle partition count can be enforced without disabling AQE entirely, and what the small-file and skew trade-offs are when AQE is turned off.

• Can spark.sql.adaptive.advisoryPartitionSizeInBytes alone produce a stable partition count across identical inputs?
• Is there a configuration that caps or fixes the AQE coalesced partition count while keeping AQE join and pruning optimizations enabled?