ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Learn how to implement exactly-once ETL pipelines using Apache Spark Structured Streaming, focusing on micro-batch architecture, checkpointing, and operational monitoring.
Static planning guesses at shuffle partition counts and join strategies. AQE re-decides after the shuffle, using real partition sizes — here's how to configure and verify it.
A decision‑by‑decision guide for picking Spark’s shuffle partition count, with a size‑based formula, a fixed‑high‑value skew option, and Adaptive Query Execution, plus steps to validate the choice using the Spark UI.
A technical decision guide for choosing between Parquet and Avro in Apache Spark, comparing read/write performance, schema evolution, and columnar vs. row-based storage.
I need to create a custom Apache Spark configuration in Azure Synapse Studio and make it available for use in notebooks and job definitions. The process involves entering a name, optional description, annotations, and adding key‑value configuration properties through the UI. I am uncertain about which fields are mandatory, how the validation step evaluates t
Enabling `spark.sql.ansi.enabled` in Apache Spark 3.4 (and the experimental 3.3.x backport) enforces strict SQL standard compliance, rejecting implicit casts and ambiguous column references that legacy code often depends on. The compatibility boundary becomes uncertain when the Spark runtime is paired with a Hive metastore earlier than 3.1.0, because version
A Delta Lake table on Apache Spark receives frequent schema changes: new columns in most releases, and occasionally a column drop or type change that later has to be undone. The goal is to choose one documented rollback strategy that keeps the table recoverable without breaking downstream Spark jobs. Two documented approaches fit. Time travel with RESTORE re
Goal: Ensure that a user‑defined class used in a Spark transformation is serializable when the application runs on a cluster with spark.serializer set to KryoSerializer. In local mode the job completes because Spark falls back to the JVM’s default Java serializer for driver‑side objects, masking the missing Kryo registration. When submitted to a cluster the
Goal Determine whether Catalyst physical optimization can change the number or placement of shuffle stage boundaries, or whether those boundaries are fixed by the logical DAG before physical planning begins. Uncertainty Spark documentation describes stage boundaries as shuffle-determined, while Catalyst chooses physical operators such as join strategies. It