Iceberg Won the Format War. Your Data Platform Still Is Not Ready for Agents

What Actually Got Settled This Year?

For roughly three years, the enterprise data platform conversation ran on a question that was never really an engineering question: Iceberg, Delta or Hudi. It absorbed an enormous amount of architectural attention, generated a remarkable volume of vendor content, and gave a lot of organisations a respectable reason to defer a decision. That question is now closed. The more consequential question that replaced it is getting a fraction of the attention.

Apache Iceberg v3 reached general availability at Snowflake Summit in early June 2026, bringing deletion vectors, row lineage tracking and VARIANT types into the production spec, with v4 proposals restructuring metadata still under community discussion. In February 2026, Apache Polaris graduated to a top-level Apache project after eighteen months of incubation, implementing the full Iceberg REST specification with credential vending, server-side commit deconflicting and role-based access control. A June 2026 review of the catalog landscape is the clearest practitioner summary of where both of those left the ecosystem.

The adoption picture caught up fast. In the State of Apache Iceberg in the Enterprise study, based on a January 2026 survey of 252 senior data and IT leaders already running Iceberg in production, 58% reported using it for business-critical analytics, 95% were using or planning AI and machine learning workloads on it, and 93% said adoption had unlocked use cases they could not previously support. The accompanying announcement of the study describes respondents managing environments ranging from hundreds of terabytes to multiple petabytes, which is to say this is not a survey of early adopters kicking tyres.

So the storage layer is a settled question, and that is genuinely good news. It also means the excuse is gone. When the format was contested, "we are waiting for the standard to shake out" was a defensible position on a roadmap. It is not defensible now, and what it was covering for is becoming visible: the layers above and below the table format, which are where the difficult, unglamorous, load-bearing work has always been.

Why Is AI-Ready a Higher Bar Than Analytics-Ready?

There is a comfortable assumption inside a lot of engineering organisations that a data platform good enough for the BI team is therefore good enough for AI. It is wrong in a specific and expensive way, and the failure mode it produces looks like a model problem right up until somebody traces it back.

Gartner's assessment is blunt. In its research on the risk that a lack of AI-ready data poses to AI projects, the firm predicted that through 2026 organisations will abandon 60% of AI projects that are not supported by AI-ready data, and reported that in a third-quarter 2024 survey of 248 data management leaders, 63% either did not have the right data management practices for AI or were unsure whether they did. The second number is the more damning one, because uncertainty at that level of seniority means nobody has audited it.

The distinction that matters is cadence and blast radius. Analytics-ready data has to be correct by the time a human reads a dashboard, and a human reading a dashboard is a remarkably effective anomaly detector: a revenue figure that is 40% off gets questioned before it reaches a board deck. AI-ready data has to be correct at the moment a model or an agent consumes it, with lineage, active metadata, freshness guarantees and permission semantics attached, because nothing downstream of that consumption will notice that it was wrong. An agent handed a stale snapshot does not hesitate. It writes a confident paragraph, files a ticket, or updates a record.

That raises the bar in four concrete places. Freshness stops being a nightly SLA and becomes a per-query property. Lineage stops being a compliance artefact and becomes a runtime requirement, because when an agent produces a wrong answer the first question is which version of which table it read. Access control stops being enforced at the BI tool and has to be enforced at the storage and catalog layer, because the agent is not using your BI tool. And semantics have to be written down somewhere machine-readable, because a schema carries structure and not meaning.

What Is the Pipeline Layer Actually Costing You?

The most useful recent number on this is not about models at all. Fivetran's enterprise data infrastructure benchmark, published in March 2026, surveyed 500 senior data and technology leaders at organisations with more than 5,000 employees across the United States, United Kingdom, EMEA and APAC in the fourth quarter of 2025. It found that 97% reported disruptions to AI or analytics initiatives caused by data infrastructure and pipeline problems, and that 53% of engineering time was going to pipeline maintenance.

The operational detail underneath those headline figures is worth sitting with. The published benchmark findings record an average of 4.7 pipeline failures per month at large enterprises, incidents taking close to 13 hours to resolve, more than 60 hours of monthly downtime delaying analytics delivery, and an estimated $3 million in average monthly business exposure against annual data spend averaging $29.3 million.

Read those two figures together and the strategic picture is uncomfortable. Half the engineering capacity of the data organisation is being consumed keeping the existing plumbing upright, and almost every organisation is already experiencing AI initiatives being delayed by that plumbing. The AI roadmap is not competing with other product work for headcount. It is competing with maintenance of the substrate it depends on, and maintenance is winning because maintenance is on fire.

This is also why the standard executive response makes things worse. Faced with an AI mandate and a fixed headcount, the instinct is to move the strongest data engineers onto the AI initiative, which removes exactly the people who were keeping the failure rate down. Six months later the pipeline incident count is up, the AI project is blocked on data quality, and the conclusion drawn is that the AI project was too ambitious. It was not. It was under-resourced at the wrong layer.

The Format Is Open. Is the Catalog?

Here is the part that most architecture reviews have not caught up with. Choosing an open table format was supposed to be the anti-lock-in decision. It was, for exactly one layer. The lock-in moved up to the catalog, and the catalog is now where the interesting commercial pressure lives.

The functional picture is genuinely encouraging. The Iceberg REST catalog specification removed the need for engine-specific connectors per catalog and enabled credential scoping, server-side commit deconflicting and, most importantly, scan planning, where the catalog filters rows and columns before returning a plan and therefore enforces governance across every engine that connects to it. Practitioner comparisons of how to choose an Iceberg control plane now treat the catalog rather than the format as the architecturally significant decision, and a July 2026 industry write-up put it plainly: Iceberg won the format war, and now the catalog counts.

The options have real differences and they are not marketing differences. Apache Polaris is vendor-neutral, governed by the Apache Foundation, and runs in production behind managed offerings from more than one vendor. Databricks Unity Catalog has Iceberg v3 support and cross-engine attribute-based access control via server-side scan planning, and independent analysis of what Databricks Iceberg support actually requires is worth reading before assuming interoperability claims are symmetric. Project Nessie offers Git-style branching that is excellent for data CI/CD but lacks built-in access control and credential vending. Apache Gravitino federates metadata across Iceberg, Delta and Hudi plus non-table assets and ships an MCP server for agent connectivity. Lakekeeper is a lightweight Rust implementation with no JVM dependency and strong authorisation via OpenFGA.

The unsolved problem in all of it is governance portability. Policies defined in one catalog do not transfer to another. There is no export format for an access model, no standard for a row-level policy, no way to lift a decade of accumulated grants and masking rules and set them down somewhere else. In practice organisations designate one catalog as the governance boundary and route every engine through it, which is a sound pattern and also, precisely, a lock-in. There is a second asymmetry worth naming: vendor-managed distributions of open catalogs frequently exceed their open-source counterparts in capability, so "we run the open one" and "we run the managed one" are not the same architecture wearing different billing.

The practical consequence for anyone signing a data platform contract this year is that the exit-cost analysis has to be done at the catalog layer, not the storage layer. Your Parquet files in object storage are portable. The three hundred access policies, the masking rules, the row filters and the lineage graph sitting in the catalog are not, and they are the artefact that took four years to build.

Why Does a Table Format Need an On-Call Rotation?

The second thing the format war concealed is that Iceberg tables are not files you write and forget. They degrade, predictably, in ways that are cheap to fix early and expensive to fix late, and the degradation is a direct function of how you write to them.

The mechanism is simple. Every insert creates new data files rather than modifying existing ones, so streaming ingestion, incremental writes and high-frequency updates accumulate small files continuously. A practical guide to fixing small files in Iceberg sets out the consequences: query planning latency climbs as the planner enumerates more files, scan performance degrades, and object storage cost rises because these systems bill per request. Alongside compaction sit snapshot expiration and orphan file cleanup, covered in detail in a walkthrough of Iceberg table maintenance operations, and both of them accumulate wasted storage silently if nobody owns them.

None of that is difficult in isolation. Every team starts the same way: Spark procedures, an Airflow DAG, some monitoring off the Iceberg metadata tables. It works beautifully for twenty tables. A comparison of nine production compaction approaches makes the scaling problem explicit: at hundreds of tables the scripts themselves become the engineering toil, and teams that build event-driven compaction end up with a custom distributed system that needs its own monitoring, alerting and on-call rotation.

This is the finding in the enterprise Iceberg study that got the least coverage and deserves the most. Most organisations running Iceberg at business-critical scale are handling optimisation, access control, compliance and disaster recovery through internally built scripts and manually orchestrated workflows, which is exactly the configuration that produces inconsistency and operational risk as the estate grows. The format was standardised. The operations around it were not, and every organisation is independently reinventing them.

The economics here are unusually clear for infrastructure work, which is rare and worth exploiting when making the case internally. One engineer whose actual job is table maintenance typically saves more in compute and storage than they cost, before you count the queries that stop timing out and the incidents that stop happening. That is a rare thing to be able to say about a platform hire, and it is a much easier argument to win than a general plea for data platform headcount.

What Changes When the Consumer Is an Agent?

Everything above was true before agents. Agents change the consequences rather than the mechanics, and they change them in one particular direction: they remove the human who used to absorb the errors.

The catalog is quietly becoming the control plane for autonomous consumption. The direction of travel across Polaris, Unity and Gravitino is toward catalogs that carry semantic definitions and machine-readable context alongside metadata, and that enforce policy during query planning rather than trusting the client. Gravitino already ships an MCP server so agents can connect to the metadata layer directly. That is the right architecture, and it is right for a reason worth stating explicitly: an agent will happily query anything it is permitted to query, so permission enforcement has to live below the point where the agent connects, not above it.

The failure modes are specific and they compound. An agent that reads a table with an expired snapshot gets an answer that is silently out of date. An agent connecting through a service account that inherited broad grants during a migration reads data its requester should never see, and produces a summary that now contains that data. An agent that joins two tables with different late-arriving-data behaviour produces a number nobody can reproduce a day later. Every one of these is a data platform defect presenting as an AI defect, and in our experience the diagnosis takes weeks longer than it should because the investigation starts with the model.

This is also where the layer above the catalog matters, and the two problems are the same problem viewed from either end. The catalog answers who may read this and what version they get. The semantic layer answers what this column means and how this metric is defined. Skip either and the agent will confidently fill the gap with an invention, which is the behaviour that turns a promising internal pilot into a permanent internal pilot.

Why Is This a Staffing Problem Before It Is a Tooling Problem?

Everything described here is solvable with known techniques. None of it is research. The reason so many organisations are stuck is not that the engineering is hard in an intellectual sense, it is that the engineering is unglamorous, continuous, and competing for attention with work that demos better.

The skills side compounds it. Analysis of how data and skills gaps threaten enterprise AI success identifies shortages that cut across three distinct capabilities: the technical data engineering expertise itself, the ability to translate business requirements into AI-ready data products, and responsible AI practice. Those are three different people in most organisations, and the market for all three tightened at once.

There is also a structural mismatch nobody plans for. This work is permanent. Table maintenance, pipeline reliability, catalog governance and access model hygiene do not have a completion date, which makes them a poor fit for project-based outsourcing and an even poorer fit for a contractor engaged for a quarter. The person who designed your access model needs to still be reachable when it needs to change, and the person who wrote your compaction strategy needs to be the person who tunes it when ingestion patterns shift. Continuity is not a nice-to-have in data platform work. It is most of the value.

That combination, permanent ownership, senior skills, and a scarce talent market, is exactly the shape that a dedicated nearshore team fits well. At Stepto we staff data platform work as long-running dedicated teams rather than as projects, with engineers in Serbia working the same hours as our clients in Western Europe and the UK, which matters more here than in most disciplines: pipeline incidents happen during the working day and the fix conversation needs both the data engineer and the analyst who noticed the number was wrong. Working inside the EU legal perimeter matters too, because production data access, GDPR obligations and processing agreements are all live constraints the moment somebody needs to debug a pipeline against real records.

The economic argument follows from the maintenance economics rather than from a rate card. A dedicated data engineer who owns compaction, snapshot policy, catalog governance and pipeline reliability displaces a meaningful share of that 53% maintenance burden and frees your in-house team for the work only they can do, which is the modelling, the domain semantics, and the product decisions that depend on knowing your business. The cost comparison that matters is not hourly rate against hourly rate. It is a stable owned platform against another year of half your engineering capacity spent firefighting.

How Do You Scope This Into Ninety Days?

The instinct with platform debt is to propose a rebuild, and the instinct is wrong, because a rebuild cannot be justified against a roadmap and will lose. What works is a bounded diagnostic followed by two or three narrow, high-leverage fixes that produce visible numbers.

Weeks one to three are measurement, and they are non-negotiable because the argument for everything after depends on them. Count your pipeline incidents over the last two quarters and the hours spent resolving them. Take a file-size histogram across your largest twenty tables. Measure query planning time separately from scan time, because planning time is the small-file signal. Inventory snapshot retention and orphan files, and price the storage. Most teams discover a number here that makes the rest of the conversation easy.

Weeks four to eight are the catalog decision, and it should be made deliberately rather than inherited from whichever engine somebody adopted first. Write down which catalog is your governance boundary, what your access model actually is as opposed to what it has accumulated into, and what leaving would cost. If the answer to the last one is unknown, that is the finding. Fix the access model before you connect any agent to it, because retrofitting permission semantics after an agent has been reading data is a very different conversation with your legal team.

Weeks six to twelve are the maintenance layer, running in parallel. Move compaction, snapshot expiration and orphan cleanup off ad hoc scripts and onto something with a real schedule, real monitoring and a named owner. Set retention policies per table class rather than globally. Instrument the maintenance jobs themselves so a silently failing compaction job is an alert rather than a discovery six weeks later when a query times out.

And staff it as ongoing capacity from the start rather than as a project with an end date, because the failure pattern here is depressingly consistent: a team does the ninety-day cleanup, achieves a genuine improvement, reassigns everyone to feature work, and reconstructs the same debt over the following eighteen months. The estate grows, ingestion patterns change, new tables arrive without policies, and the histogram goes back to what it was. This is a maintained system, not a remediated one.