Day 3 - Open Table Formats: Iceberg, Hudi, Delta Lake

Traditional data lakes (Parquet + Hive metastore) struggle with ACID transactions, row-level updates, and time travel. Open table formats (Apache Iceberg, Apache Hudi, Delta Lake) add a transaction layer to Parquet files on S3, solving these problems. Today we cover three formats and AWS integration.

Why Open Table Formats Are Needed

Hive-style tables treat "directory = table" and read based on file lists. This approach has limitations:

• Consistency guarantees during concurrent writes are difficult (lack of atomicity).
• Single record UPDATE/DELETE is very expensive (entire file rewrite).
• Schema changes and historical point-in-time queries are inconvenient.
• Row-level deletion for GDPR "right to be forgotten" is difficult.