You Are Paying More to Watch the System Than to Run It: The Observability Bill of the Agent Era

The Line Item That Grew Faster Than the System

There is a specific moment that arrives in most engineering organisations somewhere around the third year of a cloud-native build, and it usually arrives as a question from finance rather than from engineering. The question is some version of: why does monitoring cost more than the thing being monitored?

It is not a rhetorical question any more. Grafana Labs' fourth annual observability survey, drawn from 1,363 respondents across 76 countries between October 2025 and January 2026, found that observability spend averages around 17% of total compute infrastructure spend, with 10% the single most common answer. Other industry measures put the median nearer 8 to 12 percent, and Kubernetes-heavy estates consistently land at the top of that range. Whichever number you believe, the shape is the same: a tenth or more of your infrastructure budget is now spent on knowing what the other nine tenths are doing.

The absolute figures have moved with it. A Futurum survey reported this year found that well over a third of organisations, 36%, plan to spend more than $1 million on observability in 2026, and 7% plan to spend in excess of $5 million. For a mid-market engineering group of 50 to 100 engineers running a few hundred services, monthly monitoring bills in the $40,000 to $120,000 range are now unremarkable rather than scandalous.

What makes this different from ordinary cloud cost inflation is the direction of travel inside the survey data. Half of respondents expect to spend more on observability next year, and when asked why, 63% cite broader adoption rather than price increases. The bill is not growing because vendors got greedy. It is growing because teams instrumented more things, and because the things they instrumented in the last eighteen months emit telemetry in a fundamentally different pattern than the ones before them.

Why Agents Emit Telemetry Differently

A conventional request-response service produces a predictable telemetry footprint. One request, one trace, a handful of spans, a latency histogram, an error counter. You can multiply requests per second by spans per request and forecast the bill to within a rounding error. That predictability is the assumption every observability pricing model was built on, and agentic systems break it.

An agent run is not one request. It is a loop. The model is called, decides to invoke a tool, the tool calls a service, the result comes back, the model is called again, a retry fires because of a rate limit, a sub-agent is spawned, and the whole thing terminates somewhere between two and forty steps later depending on inputs nobody controls. Each of those steps is a span if you want to be able to debug it, and you do want to be able to debug it, because there is no stack trace for a model that made a bad decision on step seven.

The volume multiplier is not subtle. Industry estimates for AI workloads put telemetry generation at ten to fifty times that of the traditional services they sit alongside, with teams adding AI capabilities to existing products commonly seeing observability data volumes rise by two to five times overall. Those are the kind of multipliers that turn a comfortable line item into an emergency inside a single quarter, and they typically land without a corresponding budget request, because nobody models telemetry when they scope an agent feature.

The payload itself is heavier too. Datadog's State of AI Engineering 2026, built from LLM telemetry across thousands of production customers, found that average tokens per request more than doubled year over year for median organisations and quadrupled for the heaviest users, with system prompts alone accounting for 69% of all input tokens observed in customer traces. If you log prompts and completions for debugging and evaluation, and most teams building anything serious do, your log volume scales with context window growth, and context windows have been growing for three years straight.

Then there is the failure traffic, which is the part teams forget entirely. The same dataset recorded roughly 5% of LLM call spans reporting errors in February 2026, with rate limit errors accounting for about 60% of those failures that month, amounting to nearly 8.4 million rate limit errors across traced systems in March alone. Every one of those is a retry, and every retry is another span, another log line, another set of dimensions. Failure modes in agentic systems are not just reliability problems. They are billing events.

Cardinality Is the Bill, Not Volume

Here is where most cost-reduction efforts go wrong. Teams look at the invoice, see a large number next to logs or custom metrics, and conclude they have a volume problem. They then spend a quarter compressing, truncating and dropping data, and the bill moves by fifteen percent. The actual driver is usually cardinality, and cardinality behaves multiplicatively rather than additively.

Cardinality is the number of unique time series produced by a metric, and it is the product of every dimension you attach. A latency metric tagged with service, endpoint, region and status code might produce a few thousand series. Add customer tenant on a multi-tenant SaaS and you multiply by your customer count. Add model name, then prompt version, then tool name, then execution path, which is exactly the set of dimensions an agentic system needs in order to be debuggable, and you are into millions of series generated by a single metric definition. Vendors bill custom metrics per series. The invoice does not grow linearly with traffic; it grows with the product of your tag dimensions.

This is why the AI-era instrumentation instinct is so expensive. The dimensions teams add when they start observing agents are precisely the high-cardinality ones: tenant, user, session, device, model, region, execution path. Each is individually defensible and each is individually catastrophic for a per-series pricing model. The engineer who adds user_id as a metric tag to answer one incident question can add five figures to a monthly bill and will not find out for six weeks.

The vendors have noticed. Datadog used its DASH 2026 keynote to introduce Infinite Cardinality Metrics, a pricing option explicitly built for the dimension explosion of agentic systems and for the fact that agents themselves now query metrics. That is a genuine acknowledgement that the old model does not fit, and it is also a reminder of where the leverage sits: pricing structure, not data hygiene. A team that fixes its cardinality discipline and renegotiates its contract will save far more than a team that spends six months tuning log verbosity.

The Sampling Trap

When the bill arrives, the standard remedy is sampling. Keep 1% of traces, or 10%, or head-sample at the ingest point and move on. For a conventional stateless service, this is defensible: requests are largely interchangeable, and a random sample of a million similar traces tells you what the population looks like.

For agentic systems it is close to self-defeating. The traces that matter are, by construction, the rare ones. The agent run that looped nineteen times before terminating, the one where a tool returned malformed JSON and the model improvised around it, the one where a prompt injection changed the plan mid-execution. These are not representative samples of your traffic; they are the entire reason you are paying for tracing. Head-based sampling deletes them with the same probability as everything else, and the incident where you need one is the incident where you discover it is gone.

The consequence is a genuinely perverse loop. Teams ship non-deterministic systems, which raises the value of full-fidelity traces to its highest point in the history of the discipline. The telemetry cost of those systems then forces sampling, which lowers trace fidelity to its lowest point in years. You end up least observable exactly when you are least predictable, which is the opposite of what the last decade of operational practice was trying to achieve.

The better answer is tail-based sampling combined with tiering, and it is more work, which is why it is less common. Tail sampling holds the trace until it completes, then decides: keep everything that errored, everything that exceeded a latency or step-count threshold, everything from a flagged tenant, everything that touched a new prompt version, plus a small random baseline of the boring successes. Route the high-value set to the expensive queryable store with long retention, and the boring baseline to cheap columnar storage with short retention. This is exactly the pattern that has driven interest in treating observability as an analytics problem over a columnar database rather than as a stream of documents in a managed SaaS index. Teams that have made this move report genuinely large savings alongside better coverage; the case studies presented around OpenTelemetry's 2026 milestones include organisations moving from 5% sampled traces to 100% coverage across all environments while cutting cost by roughly 72%, because the change was architectural rather than a matter of throwing data away.

The Second Bill: Attention

There is a cost in this story that never appears on an invoice, and in most organisations it is larger than the one that does. Telemetry that nobody reads still costs money to collect, but telemetry that generates alerts nobody can act on costs engineering capacity, which is more expensive per unit than storage has ever been.

The Grafana survey puts hard numbers on it. Alert fatigue is the single biggest obstacle to faster incident response, cited by 30% of respondents, nearly double the next most common answer. Complexity and overhead top the list of overall observability concerns at 38%, ahead of signal-to-noise problems at 34% and ahead of cost itself at 31%. Read that ordering carefully: practitioners rank the operational burden of their observability stack as a bigger problem than what it charges them.

This is what the industry has started calling telemetry debt, and the framing is useful because it behaves like technical debt in the specific ways that matter. It accumulates through individually reasonable decisions, a dashboard added during an incident, an alert added after a postmortem, a metric added for a launch. It is never anyone's job to remove. And its cost is paid in a currency, engineer attention during incidents, that is not tracked on any dashboard. Analysis of the pattern consistently finds the same inventory: noisy alerts firing on thresholds nobody has revisited in two years, dashboards with single-digit annual view counts, and retention policies set to the vendor default because nobody wanted to own the decision.

The agent era compounds this from both directions. More telemetry produces more alerts, and the systems being alerted on fail in less legible ways. It also raises the stakes on the underlying data quality, because AI agents are increasingly the first responder to those alerts, and an agent triaging on noisy, poorly labelled, inconsistently structured telemetry will produce confident nonsense faster than a human would produce uncertain sense. Practitioners appear to know this instinctively: 95% of Grafana's respondents said it is important for AI to show its reasoning, which is what people say when they do not fully trust the input data.

OpenTelemetry Graduated, and the Negotiation Changed

The most consequential structural development in this space happened in May 2026, when the CNCF announced OpenTelemetry's graduation, confirming its status as the de facto standard for telemetry collection. Graduation is a governance milestone rather than a technical one, but its practical effect on cost is direct and underappreciated.

Adoption is now past the tipping point. Roughly 48.5% of organisations use OpenTelemetry with another quarter planning to, and the Grafana survey breaks that down by signal at 57% for metrics, 50% for traces and 48% for logs, with 65% of respondents investing in both Prometheus and OpenTelemetry together. Meanwhile 77% say open source or open standards are important to their observability strategy and 61% call them essential or very important.

The reason this matters commercially is that it separates instrumentation from destination. When your services are instrumented with a vendor's proprietary agent, switching backends means re-instrumenting your entire estate, which is a multi-quarter project nobody approves, which means your renewal conversation happens with no credible alternative. When your services emit OTLP through a collector you control, the backend is a configuration change. That does not automatically reduce your bill, but it changes who has leverage in the room, and 65% of Grafana's respondents now name cost as the single most important tool selection criterion.

The collector layer is also where the cheapest engineering wins live. A telemetry pipeline, whether the OpenTelemetry Collector using OTTL transformations or a dedicated pipeline product, lets you drop, aggregate, redact and route before data reaches a per-gigabyte meter. Aggregate high-cardinality dimensions into histograms at the edge. Strip debug attributes in production. Route audit logs to object storage and application logs to the queryable index. These are configuration decisions in a component you own, not a support ticket with a vendor, and they compound: the consensus across 2026 trend analyses is that GenAI and OpenTelemetry together are reshaping the landscape faster than any pricing change.

One caveat worth stating, because it is the most common way this goes wrong: OpenTelemetry semantic conventions for GenAI are still stabilising. If you are instrumenting agents today, pin your convention version, keep the attribute mapping in one place in the collector rather than scattered across services, and expect to migrate. Teams that hardcode today's attribute names into a hundred services will pay for it in 2027.

Telemetry as a Product With an Owner and a Budget

The organisations that have this under control did not find a cheaper vendor. They changed the ownership model, and the change is uncomfortable enough that most teams avoid it until finance forces the issue.

The first move is attribution. Observability cost is almost universally billed to a single platform or infrastructure cost centre, which makes it nobody's problem at the point where the spend is actually created, in a service team adding a metric tag. Attributing telemetry cost back to the team and service that generates it, and showing it on the same dashboard as that service's other costs, changes behaviour faster than any policy document. This is the same discipline that engineering organisations have been forced to learn about model API spend, and for the same reason: an invoice at the org level cannot tell you whether the increase came from one customer's runaway agent loop, a retry storm during an upstream outage, or a prompt change that added four hundred tokens to every request.

The second move is a telemetry budget expressed in engineering terms rather than currency. A service gets an allowance of active time series, log volume per request and trace retention, published like an SLO. Exceeding it is a design review, not an invoice surprise. This sounds bureaucratic and is in practice the opposite, because it pushes the decision to the moment of instrumentation where it costs a five-minute conversation rather than to the quarterly review where it costs a project.

The third is a deletion practice. Every dashboard, alert and metric gets a review cadence, and anything with no views, no fires and no owner in six months is removed rather than grandfathered. Nobody enjoys this work and it never gets prioritised against feature delivery, which is exactly why it needs to sit with a team whose mandate includes it rather than being volunteered by whoever is least busy that sprint.

The fourth is treating instrumentation as a reviewed artefact. High-cardinality tags should be as reviewable in a pull request as a database index, because they have a comparable cost profile and a worse feedback loop. A linting rule on metric dimensions and a cardinality budget check in CI cost a day to build and prevent the class of six-week-delayed billing surprise entirely.

Who Actually Does This Work

The awkward truth about everything above is that it is real engineering work, it is not glamorous, and it competes directly with the roadmap. Building a tail-sampling collector configuration, migrating a hundred services to OpenTelemetry semantic conventions, standing up a columnar store for cold telemetry, writing the cardinality budget check, and running a quarterly deletion pass are each a few weeks of focused effort by someone who understands the tradeoffs. None of them will ever win the prioritisation argument against a customer-facing feature in a given sprint, which is why the bill keeps growing in organisations that are perfectly aware of the problem.

It is also work with a particular shape. It is continuous rather than project-bounded, because instrumentation drifts back toward chaos the moment nobody is watching. It requires deep context on your specific service topology, which means it does not decompose well into a fixed-scope engagement handed to people who have never seen your incident history. And it demands genuine seniority, because the judgement call about which traces are worth keeping is a judgement about how your system fails, which is not a question a checklist answers.

That combination, continuous, context-heavy and senior, is precisely what a dedicated development team is good at and what a project-based vendor is structurally bad at. Stepto has run senior-led engineering teams out of Serbia since 2014, and this category of work, platform and reliability engineering that sits alongside product delivery rather than competing with it, is one of the more common reasons clients extend a team rather than close an engagement. A standing team can own the telemetry pipeline continuously, absorb the semantic convention migrations as they land, and run the unglamorous deletion cadence, without any of it entering the negotiation over feature scope every quarter.

Timezone overlap matters more here than on an average build, for a reason specific to this domain: most of the decisions are made during or immediately after incidents. An engineer who was awake for the postmortem writes a better alert than one reading the transcript nine hours later. For European and UK clients, a nearshore team in the same working day participates in that loop; an offshore team receives its conclusions. We have written before about how that gap is priced into quotes without being named, and observability is the clearest case of it we know.

There is a broader pattern here that connects to something we have argued repeatedly: as AI raises the volume and speed of code production, the constraint moves downstream to verification capacity. Observability is verification in production. Underfunding it while accelerating code generation is not a cost saving, it is a decision to find out about failures from customers instead of from instruments, and that trade has never once come out favourably in the long run.

What to Do in the Next Ninety Days

If this describes your situation, the sequencing matters more than the individual tactics, because two of these produce savings quickly and fund the rest.

Start with an audit that answers three questions in numbers: which metrics generate the most active time series, which dashboards and alerts have not been viewed or fired in six months, and what proportion of ingested log volume is ever queried. In most estates the third answer is under five percent, and stating it out loud is what unlocks the budget for the rest of the work. This is a week of effort and it is the highest-leverage week available.

Then get a collector in the path if you do not have one. Even with no transformation rules enabled, owning the collector layer means every subsequent decision, dropping an attribute, aggregating a dimension, routing a log class to cheap storage, becomes a config change you control rather than a vendor conversation. It is also the precondition for having any leverage at renewal.

Third, fix cardinality at the source for the top offenders identified in the audit, and add the CI check so the same class of mistake cannot recur. Fourth, and only after the first three, move to tail-based sampling with tiered retention. It is the biggest win and the most invasive change, and it is much easier to justify and to reason about once you have the audit numbers and the pipeline already in place.

Do not start with a vendor migration. It is the most disruptive option, it consumes a quarter of engineering time, and teams that migrate before doing the cardinality and pipeline work simply reproduce the same cost structure on a new invoice within a year. The problem is almost never which vendor you chose. It is that nobody owns what you send them.