Spark Structured Streaming vs DStream: A Decision Guide

The decision and constraints

You are adding a real-time ingestion pipeline to an existing Spark platform and must choose between Spark Structured Streaming and the older Spark Streaming API built on DStreams. The choice affects how you write transformations, how the job recovers after failure, and whether your pipeline will still be supported when you upgrade Spark.

Useful takeaway: start new pipelines with Structured Streaming on Spark 3.x unless you need latency below Structured Streaming's typical micro-batch range or you must maintain legacy RDD-based code. Treat DStream as maintenance mode and confirm your vendor's support policy before building new production workloads on it.

Constraints to check first

• Spark version on the target cluster. Structured Streaming reached broad stability in Spark 3.x; older clusters may lack required sources and fixes.
• Latency target. Structured Streaming uses micro-batches by default, though continuous processing mode exists for limited sources.
• Transformation complexity. Joins, aggregations, and Spark SQL integration are simpler with DataFrames.
• Sink requirements. Exactly-once guarantees depend on the sink implementation, not the streaming API alone.
• Team skills. DataFrame/Dataset code differs from RDD and DStream callbacks.

Supported options at a glance

Capability	Structured Streaming	Spark Streaming (DStream)
API style	Declarative DataFrame/Dataset operations	Imperative RDD transformations on DStream objects
Fault tolerance	Checkpointed offsets and state with source-provided replay	Manual checkpointing with RDD lineage replay
State management	Built-in mapGroupsWithState and aggregations with watermarking	updateStateByKey or mapWithState; more manual handling
Schema handling	Schema from source; schema evolution support depends on source/sink	No enforced schema; records are RDD rows
Typical latency	Milliseconds to seconds, controlled by trigger interval	Sub-second possible with small batch intervals
Spark 3.x vendor support	Primary streaming API; active development	Maintenance mode; some distributions mark it deprecated
Exactly-once output	Supported with compatible sinks and checkpointing	Supported with compatible sinks; operator must manage idempotency

Some Spark distributions list DStream as deprecated or maintenance-only starting around Spark 3.x. This is not uniform across vendors, so check your cluster's release notes before treating DStream as a long-term option.

Trade-offs

Choose Structured Streaming when

• You need joins between streams or between a stream and a static dataset.
• You want SQL-like aggregations with event-time windows and watermarks.
• You need stable checkpoint formats across Spark upgrades.
• Your team already uses DataFrames and Spark SQL.

Consider DStream only when

• You are extending an existing RDD-based pipeline and a rewrite is not practical.
• You require latency at the edge of what micro-batching can achieve and your source supports continuous processing.
• You are maintaining legacy workloads on a cluster where Structured Streaming is unavailable.

Concrete implementation: Kafka to Parquet with Structured Streaming

This example reads JSON events from Kafka, parses a timestamp field, and writes micro-batches to Parquet. It is a template, not a tested production job. Replace every placeholder with values from your environment.

Version assumptions: Spark 3.2 or later, PySpark, and the Kafka source package matching your Spark version.

from pyspark.sql import SparkSession
from pyspark.sql.functions import from_json, col
from pyspark.sql.types import StructType, StructField, StringType, TimestampType

spark = (
    SparkSession.builder
    .appName("KafkaToParquetStructuredStreaming")
    .getOrCreate()
)

spark.sparkContext.setLogLevel("WARN")

event_schema = StructType([
    StructField("event_id", StringType(), True),
    StructField("event_time", TimestampType(), True),
    StructField("payload", StringType(), True)
])

raw = (
    spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "<KAFKA_BOOTSTRAP_SERVERS>")
    .option("subscribe", "<SOURCE_TOPIC>")
    .option("startingOffsets", "earliest")
    .load()
)

events = (
    raw.selectExpr("CAST(value AS STRING) as json")
    .select(from_json(col("json"), event_schema).alias("data"))
    .select("data.*")
)

query = (
    events.writeStream
    .format("parquet")
    .option("path", "<OUTPUT_PATH>")
    .option("checkpointLocation", "<CHECKPOINT_DIR>")
    .trigger(processingTime="30 seconds")
    .start()
)

query.awaitTermination()

Where to run and required permissions

• Run spark-submit on an edge node, gateway, or laptop with network access to the Kafka brokers and write access to the storage system holding <OUTPUT_PATH> and <CHECKPOINT_DIR>.
• The Spark service account must be able to read from the Kafka topic and create files under the output and checkpoint directories.
• Submit with the matching Kafka package, for example:

spark-submit \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.12:<SPARK_VERSION> \
  kafka_to_parquet.py

Replace <SPARK_VERSION> with your cluster's Spark version. Do not reuse a checkpoint directory for a query with a different schema; Structured Streaming will fail on restart if the checkpoint metadata does not match.

Expected checks

1. Confirm the query is active: in the Spark UI or driver logs, look for the running micro-batch and numInputRows greater than zero after you publish test records.
2. Publish a small batch of JSON records to <SOURCE_TOPIC>.
3. Wait for at least one trigger interval, then read <OUTPUT_PATH> with Spark and verify row counts match the published test records.
4. Inspect <CHECKPOINT_DIR> for offset logs to confirm the engine is recording progress.

Risks

• Schema changes in Kafka can fail the query until the DataFrame schema is updated.
• Exactly-once semantics depend on the sink; the Parquet file sink supports micro-batch idempotency, but downstream consumers must handle possible file rewrites during recovery.
• Long-running stateful operations require watermark configuration; otherwise state can grow without bound.

Limitations and how to validate

• Structured Streaming works only with sources that implement the Structured Streaming source API. Custom sources require custom provider code.
• Continuous processing mode offers lower latency for a limited set of sources and is not a general replacement for micro-batches.
• To validate recovery, stop the query, kill the Spark application, restart it with the same checkpoint location, and confirm it resumes from the last committed offset rather than reprocessing from earliest.

Stopping and cleaning up state

Starting the query writes checkpoint state. To stop it cleanly:

query.stop()

If you need to rerun from scratch, stop the Spark application and remove <CHECKPOINT_DIR> and <OUTPUT_PATH>. Deleting these directories is destructive; do not do it on a production query unless you intend to discard processed state.

Bottom line

For new real-time ingestion on Spark 3.x, Structured Streaming is the safer default because it unifies batch and streaming code, offers more robust state handling, and receives active support. Keep DStream only for legacy workloads or ultra-low-latency experiments, and verify your vendor's deprecation schedule before committing to it.