Explore

Articles
General
Community
Search

Community

Members
Leaderboards
Achievements
Community guidelines

The essentials

About ReadMeFeed
Privacy
Terms of use
Moderation policy

Home Community Create Inbox Profile

ReadMeFeed / Community knowledge

#etl-pipeline

Real questions. Useful conversations. Find the people who know your stack.

All Unanswered Has answers Resolved Unresolved

Cards Compact list

Guide143.9K views1,039 votes1 reply1 comment548 following

Architecting Exactly-Once ETL with Spark Structured Streaming Micro-batches

Learn how to implement exactly-once ETL pipelines using Apache Spark Structured Streaming, focusing on micro-batch architecture, checkpointing, and operational monitoring.

etl-pipeline structured-streaming data-engineeringTasadduq Burneyowner34.8K rep · 29 Nov 2025, 01:25 UTC

Guide10.2K views3,495 votes1 reply1 comment

1,368 following

Choosing Between Parquet and Avro for Spark ETL Pipelines

A technical decision guide for choosing between Parquet and Avro in Apache Spark, comparing read/write performance, schema evolution, and columnar vs. row-based storage.

etl-pipeline parquet avroTasadduq Burneyowner34.8K rep · 18 Jul 2026, 13:25 UTC