The Problem With Data Lakes Nobody Wanted to Talk About
For years, the data lake promised a future where organisations could store everything and query anything, at any scale, without the rigid constraints of a traditional data warehouse. The reality was considerably messier. Teams built pipelines on top of Hive and raw object storage like Amazon S3, only to discover that the combination was far more brittle than anyone had advertised. Jobs failed mid-write and left tables in inconsistent states. Schema changes broke downstream consumers without warning. Querying yesterday's data required clumsy workarounds. The data lake had a reliability problem, and for a long time, the industry quietly worked around it rather than fixing it at the source.
Apache Iceberg changed that. Originally developed at Netflix to solve the company's own data engineering headaches at scale, Iceberg is an open table format designed to sit on top of object storage and give it the kind of transactional guarantees that databases have offered for decades. Since becoming an Apache top-level project, it has become one of the most consequential pieces of infrastructure in the modern data stack, and teams across the industry are now actively migrating away from Hive to adopt it.
What a Table Format Actually Does
To understand why Iceberg matters, it helps to understand what a table format is and what problem it is solving. Object storage systems like S3, Google Cloud Storage, and Azure Data Lake Storage are fundamentally just file stores. They have no concept of a table, a row, or a transaction. When query engines like Spark or Trino want to treat a collection of Parquet files as a table, they need a layer on top that tracks which files belong to the table, what the schema looks like, and how the data is partitioned. That layer is the table format.
Hive's approach to this problem, the Hive Metastore, was designed in an era when Hadoop clusters were the norm and the workloads were more predictable. It stores table metadata centrally, but it was never built for the concurrency, scale, or correctness demands of modern cloud data platforms. Iceberg takes a fundamentally different approach by treating metadata as a first-class, versioned artifact stored alongside the data itself in object storage, rather than in a separate central service.
ACID Transactions Over Object Storage
One of Iceberg's most important contributions is bringing genuine ACID transaction support to object storage. In a traditional data lake built on Hive, two jobs writing to the same table at the same time could corrupt each other's output. A reader could observe a table mid-write and get incomplete results. There was no safe way to do an atomic replace of a partition without risking data loss or a window of inconsistency.
Iceberg solves this through its metadata layer. Every write to an Iceberg table produces a new snapshot — a complete, immutable description of the table's state at that moment, referencing exactly which data files are part of the table. Writers create new snapshots atomically, meaning a new snapshot either fully commits or it does not commit at all. Readers always see a consistent, complete view of the table based on a snapshot, and they are never exposed to partially written data. This is a dramatic improvement in reliability for any team running concurrent pipelines against shared tables.
Time Travel and Snapshot Isolation
Because every change to an Iceberg table produces a new snapshot rather than overwriting the old one, the full history of a table is preserved as long as those snapshots are retained. This enables time travel queries — the ability to read a table exactly as it existed at a previous point in time by referencing an older snapshot. For data engineers, this is enormously useful for debugging pipeline failures, auditing data changes, and reproducing past results without maintaining separate backup copies of data.
Snapshot isolation also means that long-running analytical queries are protected from concurrent writes. A query that begins reading a table will continue reading from its initial snapshot even if a new write commits while the query is running. This eliminates a whole class of subtle, difficult-to-reproduce bugs that plague Hive-based systems where a write mid-query could cause unexpected results or failures.
Schema Evolution Without Breaking Downstream Consumers
Schema evolution — the ability to change a table's structure over time as business needs shift — has historically been one of the most painful aspects of managing data lakes. Adding a column to a Hive table seems straightforward until a downstream job that reads that table by column position rather than column name suddenly starts returning wrong data. Renaming a column, dropping one, or reordering them could silently corrupt downstream pipelines in ways that took days to diagnose.
Iceberg handles schema evolution safely by design. Columns in Iceberg are tracked by a unique identifier, not by their name or position. This means a column can be renamed, reordered, or have its type promoted without affecting existing readers, because those readers are referencing the column by its stable ID. New columns added to a table return null for existing files that predate the addition, and dropped columns are simply hidden from new readers. Schema changes in Iceberg are non-breaking by default, which removes an entire category of operational risk from data platform teams.
Hidden Partitioning and Partition Evolution
Partitioning is one of the primary mechanisms through which query engines achieve performance on large datasets — by dividing data into logical chunks that can be skipped entirely if they are not relevant to a query. In Hive, partitioning is explicit and exposed to the user. Queries must be written with awareness of the partition scheme, and if the partitioning strategy needs to change, it typically means rewriting the entire table and updating every query that references it.
Iceberg introduces the concept of hidden partitioning, where the partition scheme is tracked in the table's metadata and applied transparently by the query engine. A user querying an Iceberg table does not need to know how it is partitioned to benefit from partition pruning. The engine reads the metadata, determines which partitions are relevant to the query's filter predicates, and skips everything else automatically. Crucially, Iceberg also supports partition evolution — the ability to change a table's partitioning strategy over time without rewriting existing data, with old and new partitions coexisting safely within the same table.
Why Teams Are Migrating From Hive
The practical result of all of these capabilities is that Iceberg removes a substantial amount of operational complexity from running a data lake at scale. Teams that previously maintained elaborate workarounds for Hive's reliability problems — custom locking mechanisms, partition management scripts, manual snapshot management — can replace that bespoke infrastructure with the guarantees Iceberg provides out of the box. The reduction in pipeline failures, the elimination of schema-change incidents, and the availability of time travel for debugging all represent meaningful improvements in day-to-day data engineering productivity.
The ecosystem support for Iceberg has also matured rapidly. Major query engines including Apache Spark, Trino, Flink, Hive, and Dremio all support reading and writing Iceberg tables. Cloud data warehouses and managed services from major providers have added native Iceberg support, allowing organisations to use Iceberg as a common, open table format that works across multiple compute engines without vendor lock-in. This interoperability is a significant part of the format's appeal — data stored in Iceberg tables is not tied to any single vendor's proprietary format.
For teams evaluating the migration, the effort is not trivial. Existing Hive tables need to be converted, pipelines need to be updated, and teams need to build familiarity with Iceberg's operational model. But the consensus emerging from the data engineering community is that the long-term reliability and capability gains make the investment worthwhile. Iceberg did not just improve the data lake — it addressed the foundational weaknesses that had always limited what data lakes could reliably deliver.