ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Developer tools · #apachespark
I'm designing a Spark Structured Streaming pipeline (assume Spark 3.4/3.5) that writes micro-batches to an external store via foreachBatch , with checkpointing and the write-ahead log enabled. The hard constraint is that when a batch fails partway and Spark retries it, the target must not end up with duplicate records. Two documented patterns seem applicable
Goal: Determine whether Apache Spark Structured Streaming’s internal checkpointing and write‑ahead log (WAL) mechanism provides sufficient fault tolerance for recovery from HDFS‑based checkpoint storage, or whether augmenting it with external HDFS snapshots is necessary to guard against metadata corruption or accidental deletion. Constraints: The checkpoint
Goal: determine if Spark applies a global sort when LIMIT/OFFSET used without ORDER BY and what performance implications arise. Constraints: Spark does not guarantee partition order; without ORDER BY, LIMIT/OFFSET may trigger a shuffle to sort the entire dataset before returning subset, leading to nondeterministic results and potential memory pressure on ske
Migration goal Consider a small Structured Streaming job that reads from Kafka and writes to a relational sink. The plan is to move it to a newer Spark 3.x release, possibly with a revised aggregation, without pausing the pipeline. The open decision is whether the existing checkpoint directory can carry the query across that boundary, or whether cutover need
Goal: configure a Spark Structured Streaming job so that retrying a failed micro-batch does not produce duplicate rows in the downstream sink. The job uses foreachBatch to write to a table, and the checkpoint directory retains committed offsets. If a task fails after the sink acknowledges part of a batch but before the checkpoint advances, the next attempt m
Spark 4.0 changes the Scala binary compatibility boundary compared with Spark 3.5. Spark 3.5 ships Scala 2.12 and 2.13 builds, while Spark 4.0 is documented as Scala 2.13 only and requires Java 17 at runtime. The goal is to determine the impact on existing codebases compiled against Spark 3.5 with Scala 2.12 and Java 8/11 toolchains. The constraints include
Evaluating Structured Streaming state management and checkpointing for fault tolerance in a test integration without production credentials. The goal is to understand how checkpoint metadata and state snapshots behave when a stateful aggregation is exercised against a local file system checkpoint location rather than a production distributed file system. The
Executor memory limits that are sufficient on a developer workstation do not map consistently to container limits in production. In Spark 3.x, spark.executor.memory defines heap while spark.executor.memoryOverhead adds a non-heap allowance, and the sum is enforced by YARN and Kubernetes cgroups. Local[*] mode runs driver and executors in one JVM and does not
Goal Maintain exactly‑once semantics across all Structured Streaming sinks during retries, avoiding duplicate writes when a task restarts. Current Landscape Apache Spark relies on checkpointing and write‑ahead logs to recover from failures. Exactly‑once guarantees are only available for sinks that implement idempotent writes (Kafka, Delta Lake, HDFS/Parquet)
yarn
kubernetes