Iceberg Format: Best Metadata Governance 2026 Guide

Apache Iceberg is an open table format that turns a set of data files in object storage into a database-style table with ACID transactions, schema evolution, hidden partitioning, and time travel. Its metadata governance, meaning lineage, access control, classification, and audit that hold across engines and teams, lives in the catalog you pair Iceberg with, not in the table format itself. Choosing and operating that catalog is the governance decision.

Key Takeaways

• Apache Iceberg is the de facto open table format in 2026; the governance decision is which catalog you pair it with [1][14][32].

• The format provides ACID transactions, schema evolution, and time travel. It does not provide lineage, classification, cross-engine policy, or a unified audit trail [1][2][11].

• Access policy does not transfer between catalogs. A grant set in one catalog does not apply when the same table is read through another engine [11][32].

• For new multi-engine builds, choose a catalog that implements the Iceberg REST Catalog spec with role-based access control and credential vending, such as Apache Polaris [12][16].

• Enterprise-grade governance is a layer: one identity, one policy engine, a REST catalog with credential vending, a metadata and lineage platform, and a unified audit trail across every engine.

This guide is the definition-first, catalog-by-catalog counterpart to the editorial Apache Iceberg Is the New Standard. Align Your Data Strategy or Fall Behind. The editorial argues why Iceberg won; this guide answers how to govern what you build on it.

What Is Apache Iceberg and Why Governance Matters Now

Direct answer: Apache Iceberg is a table format, not a file format and not a storage engine. It keeps data in open files such as Parquet on object storage and adds a metadata tree that lets any compatible engine read and write the same table with ACID guarantees [1][3].

Iceberg was created at Netflix to fix the correctness and scale limits of Hive tables, donated to the Apache Software Foundation, and is now the most widely adopted open table format in the industry [1][5]. Governance matters more in 2026 than it did in 2022 because the number of engines touching a single table has multiplied: one finance table can be written by Spark, queried by Trino, read by DuckDB, and federated into BigQuery in the same week [14][32]. Governance configured per engine drifts the moment a new engine is added.

The metadata tree has five layers. Each is a place where governance can be enforced or, more often, where it is not. The table below maps each layer to its governance relevance.

Time travel lets you query a table as of a past snapshot or timestamp; schema evolution lets you add, drop, rename, or reorder columns without rewriting data [1][2]. These make a table auditable in principle. They do not record who changed it, who read it, what in it is sensitive, or whether Spark, Trino, and a Python job honored the same access rule. Those are governance functions supplied by the catalog and the systems around it.

Where Iceberg's Native Metadata Governance Falls Short

Direct answer: Apache Iceberg does not natively provide catalog-wide lineage, data classification, cross-engine policy enforcement, column masking, a unified audit trail, or an identity model. The specification defines how tables behave, not how organizations secure them, which is why so many vendors adopted it [1][2].

When Does Iceberg's Native Metadata Fall Short?

The hardest gap to close is cross-engine policy enforcement, the guarantee that the same access rule applies no matter which engine reads the table. Access policies live in the catalog, and there is no industry standard for sharing them across catalogs [11]. Namespace grants defined in a REST catalog such as Apache Polaris do not apply when the same physical table is read through AWS Glue [11][32]. The table is one object in storage; the policy is many, and they do not stay in sync on their own.

A second gap is the boundary between table policy and object-store permissions. Credential vending, where the catalog hands an engine short-lived, scoped storage credentials so the engine never needs broad bucket access, is how mature catalogs close it. Apache Polaris and the managed REST catalogs support credential vending; the older Hive Metastore and basic Glue setups generally do not [12][15]. Without it, every engine needs standing access to the bucket and the real security boundary becomes IAM rather than the catalog.

Catalog Options for Iceberg Metadata Management

Direct answer: The Iceberg catalog tracks where each table's current metadata file lives and mediates create, read, update, and commit operations. It is the most consequential governance choice because every engine must pass through it. For new multi-engine builds, choose a catalog that implements the Iceberg REST spec with role-based access control and credential vending [12][16].

Which Iceberg Catalog Should I Use: Glue Data Catalog, PyIceberg, or BigQuery Iceberg?

Use the Glue Data Catalog with Lake Formation for AWS-only estates; the BigLake / Lakehouse Runtime Catalog REST endpoint for Google Cloud; and Apache Polaris or another REST-spec catalog for open, multi-engine builds. PyIceberg is the Python client, not a catalog of its own, and connects to any of these. The table compares the options on governance.

Two distinctions trip up evaluations. The Iceberg REST Catalog is a protocol, not a product: Polaris, Unity Catalog, BigLake, and Glue (via S3 Tables) are implementations that speak it, which lets an engine swap catalogs without code changes [12][16][32]. PyIceberg is the Python implementation of the Iceberg client, and its catalog is whatever you configure (REST, a SQL catalog on Postgres or SQLite, Glue, or Hive); a local SqlCatalog on SQLite is for development and has no real governance [27][28][29].

Google renamed its offering. BigLake metastore added Iceberg REST Catalog support in late 2025 and was repositioned as the Lakehouse Runtime Catalog, a serverless managed catalog with a standard REST interface shared by Spark and BigQuery [25][26]. For a BigQuery Iceberg catalog, that REST endpoint is the supported path for new workflows. For background on why open formats underpin governance, see vendor-neutral enterprise data platforms built on open formats.

Engine-Specific Metadata Behavior: Athena, DuckDB, and BigQuery

Direct answer: Engines expose and modify Iceberg metadata differently, but all resolve tables through a catalog, so the governance you get depends on the catalog rather than the engine.

Can I Query Iceberg Metadata With Athena or DuckDB Without a Dedicated Catalog?

You can inspect metadata in both, but both still resolve tables through a catalog. Amazon Athena exposes Iceberg's internal metadata as queryable tables: $history, $snapshots, $files, $partitions, and $manifests [19]. A statement such as SELECT * FROM "orders$snapshots" returns snapshot IDs, parent IDs, commit timestamps, and the operation type, the closest thing Iceberg has to a built-in change log [19][20]. When AWS Lake Formation row or cell filters are present on the base table, or you lack permission to view all columns, querying the $files, $partitions, $manifests, and $snapshots metadata tables fails with an AccessDeniedException, so metadata inspection is itself governed [19].

DuckDB is now a read-write Iceberg engine. The DuckDB Iceberg extension shipped full read and initial write support in v1.4.0, delete and update for v2 tables in v1.4.2, and in v1.5.3 (May 2026) added MERGE INTO, ALTER TABLE, partition transforms, and Iceberg v3 support, and it can attach to Iceberg REST catalogs directly, including in the browser via DuckDB-Wasm [21][22][23][24]. Because DuckDB Iceberg write can commit to a governed table from a laptop, the catalog's authentication and credential vending are load-bearing for DuckDB too. A DuckDB Iceberg catalog connection inherits exactly the governance of the REST catalog it points at.

BigQuery reads and writes Iceberg through the BigLake / Lakehouse Runtime Catalog REST endpoint, bringing lakehouse tables under the same IAM model as native BigQuery datasets [25][26]. Spark and Trino remain the workhorses for production writes and federated reads and honor whatever catalog and policy layer sits in front of them. For a broader engine comparison, see the 2026 query layer across Snowflake, Databricks, and Starburst.

Iceberg vs. Delta Lake vs. Hudi: Governance Capabilities Compared

Direct answer: Iceberg wins on vendor-neutral governance and the broadest multi-engine reach, which is why its governance is a catalog decision. Delta Lake offers the most complete out-of-the-box governance, tied to Databricks Unity Catalog. Hudi offers the richest native operational audit trail, with a narrower engine ecosystem [30][31][32].

Delta Lake records table state in a _delta_log of JSON transactions and Parquet checkpoints, with governance centered on Unity Catalog [30][32]. Apache Hudi keeps an append-only timeline in .hoodie that logs every commit, compaction, and cleaning action, giving it the strongest native change-data-capture and operation-level audit trail [31]. Apache Iceberg uses the hierarchical metadata tree above and is governed by the Apache Software Foundation through consensus, which prevents any single vendor from steering the format [32].

How Does Iceberg Format Compare to Delta Lake for Metadata Management?

For retiring legacy Delta tables, the delta archive decision in migrations is usually handled by interoperability layers such as Delta UniForm or Apache XTable rather than a full rewrite, which lets a mixed estate converge on Iceberg reads over time [31][32].

Building an Enterprise-Grade Iceberg Governance Architecture

Direct answer: Governance for Iceberg is a layer assembled across catalogs, engines, and storage, because no single component covers the whole surface. A reference architecture that holds up under audit has five components.

1. Unified identity. One identity provider, federated into every engine and catalog, so a user or service is the same principal everywhere. Keycloak is the common open choice.

2. A single policy engine. One place to define access, masking, and row-filtering rules that every engine consults. Apache Ranger is the established open option, and the goal is to define a policy once and have it apply to Trino, Spark, and any other engine at the same time.

3. A REST catalog with credential vending. A catalog that speaks the Iceberg REST spec and hands engines short-lived, scoped storage credentials, so the object store is not a separate, broader boundary [12][15].

4. A metadata and lineage platform. A system such as DataHub that stores classification tags, ownership, and lineage, and that drives policy. Tag a dataset as sensitive once, and policy generation propagates from that tag rather than being hand-configured per table [11].

5. A unified audit pipeline. Every read, write, and denial, across every engine, landing in one log store an auditor can query. Snapshot history alone does not provide this.

The organizing pattern is to define identity and policy once and enforce them everywhere. Tag a column as PII in the metadata platform, map the tag to a role rule, and have that rule generate the right grants in the policy engine for every engine and storage path automatically. When that loop is closed, adding an engine does not open a gap, because the engine inherits the same identity, policy, and audit trail.

How NexusOne Implements This Layer

NexusOne builds this governance layer for enterprises that have standardized on Iceberg and need to govern it at scale. It superimposes one identity model through Keycloak and one policy model through Apache Ranger across every engine, so a policy defined once applies to Trino, Spark, and Kyuubi at the same time, and to object storage on a per-object basis [11]. Catalog and lineage run through Gravitino and DataHub, where a tag on a dataset drives policy generation across Ranger at the bucket, schema, and table level. This works because NexusOne sits horizontally across the catalogs and engines rather than inside any one of them, which is the position from which a grant in one place can be made to hold everywhere [11][32]. For regulated environments, see compliance-ready data platforms for DoD IL5 and HITRUST; for deployment topology, see Iceberg on Kubernetes and hybrid and multi-cloud data integration.

Enterprise Iceberg Governance Readiness Checklist

Start Governing Your Iceberg Lakehouse With Confidence

Direct answer: Treat the catalog as the governance control point, require REST-spec portability with role-based access control and credential vending, and assemble one identity, policy, lineage, and audit layer across every engine. That combination passes audits a per-engine setup cannot.

Iceberg gives you a portable, transactional, time-travelable table that every engine can read, and it leaves identity, cross-engine policy, classification, lineage, and unified audit to the layer around it [1][11]. The catalog sets the ceiling on how good that layer can be. To structure an evaluation, use the 2026 AI and Data Buyer's Guide; to fix governance that is already fragmented across systems, start here; and to pressure-test your Iceberg governance architecture against the gaps above, book an expert consultation.

Frequently Asked Questions

What Is the Iceberg Format and How Does It Work?

Apache Iceberg is an open table format for large analytic datasets. It keeps data in open files such as Parquet on object storage and adds a hierarchical metadata tree, a catalog pointer, a metadata file, manifest lists, and manifest files, that lets any compatible engine read and write the same table with ACID transactions, schema evolution, hidden partitioning, and time travel [1][2]. Engines such as Spark, Trino, DuckDB, and BigQuery operate on the same tables without proprietary connectors [1][32].

What Is the Difference Between Iceberg Format and Parquet?

Parquet is a file format that stores columnar data in a single file. Iceberg is a table format that organizes many Parquet, ORC, or Avro files into one logical table with transactional metadata [1][3]. Parquet knows the rows and columns inside one file; Iceberg knows which files make up a table, what schema and partition spec apply, and what the table looked like at every past snapshot. They are used together, since Iceberg tables are usually backed by Parquet data files [1].

Which Iceberg Catalog Should I Use: Glue Data Catalog, PyIceberg, or BigQuery Iceberg?

For new multi-engine deployments, choose a catalog that implements the Iceberg REST specification with role-based access control and credential vending, such as Apache Polaris [12][16]. For AWS-only estates, the Glue Data Catalog with Lake Formation is the natural fit and leads adoption [14]. For Google Cloud, use the BigLake / Lakehouse Runtime Catalog REST endpoint [25][26]. PyIceberg is the Python client that connects to any of these, with a local SqlCatalog useful only for development [27][29].

How Does Apache Iceberg Handle Metadata Governance at Enterprise Scale?

Iceberg handles the table side, consistent schema, snapshots, and time travel, but it does not provide identity, cross-engine access policy, classification, lineage, or unified audit on its own [1][11]. At enterprise scale those come from the catalog plus a surrounding layer: an identity provider, a shared policy engine, a metadata and lineage platform, and a centralized audit pipeline. The hardest part is enforcing one policy consistently across every engine, because a grant in one catalog does not automatically transfer to another [11][32].

Can I Query Iceberg Metadata With Athena or DuckDB Without a Dedicated Catalog?

You can inspect Iceberg metadata in both, but both still resolve tables through a catalog. In Amazon Athena, you query metadata tables directly with SQL, such as SELECT * FROM "table$snapshots" or "table$files", and Lake Formation filters can restrict access to those metadata tables [19][20]. DuckDB reads and now writes Iceberg through its extension and can attach to Iceberg REST catalogs, including in the browser via DuckDB-Wasm [21][22][23]. A local development setup without a real catalog has no meaningful governance and should not become a production pattern [29].

What Is Apache Iceberg v3 and Should I Upgrade My Iceberg Tables?

Iceberg v3 is the specification update that adds deletion vectors for more efficient row-level deletes, row lineage, and a nanosecond-precision timestamp type [6][7][8]. Row lineage tracks a unique _row_id and a _last_updated_sequence_number for every row, improving change tracking and incremental processing [6][7]. Engines including Snowflake, Spark, and DuckDB added v3 support through 2025 and 2026 [9][22]. Upgrade when your engines support v3 and you need its delete performance or row-level lineage, and verify every engine that writes your tables understands v3 first, since mixed-version writers can cause problems [8][10].

How Does Iceberg Format Compare to Delta Lake for Metadata Management?

Iceberg uses a hierarchical metadata tree and is governed neutrally by the Apache Software Foundation, with a broad REST catalog ecosystem spanning Polaris, Unity Catalog, Glue, and BigLake [12][32]. Delta Lake uses a _delta_log of JSON transactions and checkpoints and offers the most complete built-in governance through Databricks Unity Catalog, strongest inside Databricks [30][32]. Iceberg favors engine portability and vendor independence; Delta favors out-of-the-box governance within the Databricks ecosystem. For retiring legacy Delta tables, interoperability layers such as Delta UniForm or Apache XTable let a mixed estate converge on Iceberg reads without a full rewrite [31][32].

References

1. Apache Iceberg: official project site. https://iceberg.apache.org/

2. Apache Iceberg: Table Spec. https://iceberg.apache.org/spec/

3. Amazon Web Services: What is Apache Iceberg? https://aws.amazon.com/what-is/apache-iceberg/

4. Snowflake: What Are Apache Iceberg Tables? https://www.snowflake.com/en/fundamentals/apache-iceberg/

5. Wikipedia: Apache Iceberg. https://en.wikipedia.org/wiki/Apache_Iceberg

6. Google Open Source Blog: What's new in Apache Iceberg v3 (Aug 2025). https://opensource.googleblog.com/2025/08/whats-new-in-iceberg-v3.html

7. AWS Big Data Blog: Apache Iceberg V3 deletion vectors and row lineage. https://aws.amazon.com/blogs/big-data/accelerate-data-lake-operations-with-apache-iceberg-v3-deletion-vectors-and-row-lineage/

8. Starburst: Iceberg v3: Getting Started. https://www.starburst.io/blog/iceberg-v3/

9. Snowflake: Announcing Apache Iceberg v3 Support on Snowflake. https://www.snowflake.com/en/blog/apache-iceberg-v3-support/

10. Dremio: Apache Iceberg V2 vs V3: What Changed. https://www.dremio.com/blog/apache-iceberg-v2-vs-v3-what-changed-and-what-it-means-for-your-tables/

11. Atlan: Apache Iceberg Tables Governance: A Practical Guide. https://atlan.com/know/iceberg/apache-iceberg-table-governance/

12. Alex Merced (iceberglakehouse): Apache Iceberg Catalogs Explained. https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-catalogs-explained/

13. Alex Merced (DEV): The State of Apache Iceberg Catalogs in June 2026. https://dev.to/alexmercedcoder/the-state-of-apache-iceberg-catalogs-in-june-2026-265e

14. DataLakehouseHub: The 2025 State of the Apache Iceberg Ecosystem. https://datalakehousehub.com/blog/2026-02-state-of-the-apache-iceberg-ecosystem/

15. RisingWave: Apache Iceberg Catalogs Explained. https://risingwave.com/blog/apache-iceberg-catalogs/

16. RisingWave: Iceberg Catalog Comparison: Hive vs Glue vs REST vs Nessie. https://risingwave.com/blog/iceberg-catalog-comparison-guide/

17. Conduktor: Iceberg Catalog Management: REST, Hive, Glue, and Nessie. https://www.conduktor.io/glossary/iceberg-catalog-management-hive-glue-and-nessie

18. e6data: Iceberg Catalogs 2025: Emerging Metadata Solutions. https://www.e6data.com/blog/iceberg-catalogs-2025-emerging-catalogs-modern-metadata-management

19. Amazon Athena: Querying Iceberg table metadata. https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-table-metadata.html

20. Amazon Athena: Query Apache Iceberg tables. https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html

21. DuckDB: Iceberg Extension overview. https://duckdb.org/docs/current/core_extensions/iceberg/overview

22. DuckDB: New DuckDB-Iceberg Features in v1.5.3 (May 2026). https://duckdb.org/2026/05/29/new-iceberg-features

23. DuckDB: Iceberg REST Catalogs. https://duckdb.org/docs/lts/core_extensions/iceberg/iceberg_rest_catalogs

24. DuckDB: Writes in DuckDB-Iceberg (Nov 2025). https://duckdb.org/2025/11/28/iceberg-writes-in-duckdb

25. Google Cloud Blog: BigLake metastore now supports Iceberg REST Catalog. https://cloud.google.com/blog/products/data-analytics/biglake-metastore-now-supports-iceberg-rest-catalog

26. Google Cloud Documentation: Use the BigLake metastore Iceberg REST catalog. https://docs.cloud.google.com/biglake/docs/blms-rest-catalog

27. PyIceberg: official documentation. https://py.iceberg.apache.org/

28. PyIceberg: Configuration. https://py.iceberg.apache.org/configuration/

29. PyIceberg: SQL catalog reference. https://py.iceberg.apache.org/reference/pyiceberg/catalog/sql/

30. Dremio: Apache Iceberg vs Delta Lake. https://www.dremio.com/blog/apache-iceberg-vs-delta-lake/

31. Onehouse: Apache Iceberg vs Delta Lake vs Apache Hudi. https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison

32. RisingWave: Apache Iceberg vs Delta Lake vs Hudi (2026). https://risingwave.com/blog/apache-iceberg-vs-delta-lake-vs-hudi-2026/