Your Agent Died at Step 41. It Had Already Charged the Customer.
Agents now run for hours, not seconds, and the runtimes they were deployed on assume a request finishes in milliseconds. The missing piece is not a better model, it is a state layer, and every major platform has now shipped one.
Why Does the Request/Response Habit Break Agents?
Almost every AI feature in production today was built on an assumption inherited from web development: the unit of work fits inside one request. A chat completion returns in a couple of seconds, a retrieval-augmented answer in five, and the whole thing lives happily inside an HTTP handler, a Lambda invocation or a background job with a generous timeout. That assumption held for two years because the work really was that small.
It does not hold any more, and the measurements are unambiguous about the direction. METR's task-length benchmark, which scores agents against tasks calibrated by how long human professionals take to complete them, now puts the strongest models well past the hour mark: its January 2026 revision estimates a 50% time horizon of roughly 320 minutes for Claude Opus 4.5, with a doubling time of about 196.5 days across the whole measurement period and roughly 88.6 days when the fit is restricted to progress since 2024. The same revision had to expand its suite from 170 to 228 tasks and lift the number of eight-hour-plus tasks from 14 to 31, simply because the old suite was running out of headroom.
Translate that from benchmark language into operational language and you get a sentence most engineering teams have not yet internalised: the thing you deployed as a function is now a process that runs for a working day. It makes sixty tool calls instead of one. It accumulates a context window that grows monotonically. It sometimes needs to stop and ask a person whether to proceed, and that person is asleep. And it does all of this on infrastructure whose failure modes were tuned for workloads that finish in 200 milliseconds.
The infrastructure reality is worth spelling out, because it is where the pain arrives. Serverless functions have execution ceilings, which is precisely why AWS added a durable execution primitive to Lambda on 2 December 2025 that checkpoints progress automatically and can suspend a function for up to a year without incurring compute charges while it waits. Containers get restarted by deploys, evicted by the scheduler, reclaimed as spot capacity, and killed by the OOM reaper when a context window grows faster than anyone modelled. Model providers return timeouts and overload responses. Any single one of those events, at any point in a six-hour run, ends the run. The question is what state survives it.
This matters at scale rather than at the margins. Gartner expects 40% of enterprise applications to be integrated with task-specific AI agents by 2026, up from less than 5% in 2025. That is a very large number of long-running, side-effecting processes about to be deployed on runtimes that assume short, stateless ones.
What Actually Fails When a Six-Hour Run Dies at Step 41?
Four separate things are lost when a long agent run dies, and teams tend to notice them in ascending order of expense.
The first is the reasoning trace: the plan, the intermediate conclusions, the retrieved documents, the scratchpad. The second is the money already spent generating it, which for a multi-hour run with a large accumulated context is not a rounding error. The third is time, both the wall-clock time of the run and the human time of whoever was waiting on the output. The fourth, and the only one that reaches your customers, is the record of which side effects already happened. Steps 1 through 40 sent emails, opened tickets, wrote rows, called payment APIs, posted to Slack, opened pull requests. If the only place that record lived was the process memory of a container that no longer exists, then restarting the run does all of it again.
The arithmetic of why long runs fail at all is unforgiving, and it is not primarily about model quality. Compose steps and reliability multiplies: as Inngest puts it, five steps at 99% reliability each give you 95% overall, and ten steps give you 90%. Extend that to a sixty-step agent run and the happy path becomes a minority outcome. Philosopher Toby Ord formalised the same intuition against the METR data in a paper proposing that agents fail at a roughly constant rate per minute of human-equivalent task time, which makes success decay exponentially with task length and gives every agent a characteristic half-life. It is an uncomfortable model precisely because it predicts that better models raise the half-life without ever removing the decay.
The empirical literature has converged on the same shape from the other direction. An Oxford synthesis published in July 2026 pulled together 27 benchmark, taxonomy and audit papers spanning 19 benchmarks and grouped agent failures into six clusters, including tool invocation and parameter errors, planning failures, and long-horizon degradation from context accumulation. Two of its conclusions deserve to be pinned above every agent team's desk: failures compound nonlinearly with task length, and strong performance on individual subtasks does not reliably translate into end-to-end success. It also notes that piling on more scaffolding does not consistently improve reliability, which should temper the instinct to solve this with another orchestration layer of prompts.
The reason this stays invisible until production is that staging exercises the wrong distribution. Demos run the short happy path on a warm process with a cooperative model and no deploy in flight. Production runs the long path, on a Tuesday, during a rolling update, against a rate-limited provider, with a human approval step that nobody answered until Thursday.
Key Takeaways
- Four things die with a long run: the reasoning trace, the tokens already paid for, the elapsed time, and the record of side effects already executed
- Only the last one is customer-visible, and it is the one most teams have no durable record of
- Reliability multiplies across steps, so a sixty-step run makes the happy path a minority outcome even with strong per-step reliability
- An Oxford synthesis of 27 papers found failures compound nonlinearly with task length and that extra scaffolding does not reliably help
Why Did Four Platforms Ship the Same Primitive in Twelve Months?
When independent vendors converge on an identical abstraction inside a year, it is usually because their customers have all hit the same wall. That is what happened with durable execution, and the timeline is tight enough to be worth laying out.
Inngest dates the crossing into the early majority to late 2025, when AWS released durable functions, Cloudflare took Workflows to general availability, and Vercel launched its Workflow DevKit, on top of a category Temporal had spent years establishing and Azure Durable Functions had already brought to serverless. On the Cloudflare side, the general availability post is explicit that the AI use case drove the feature set: the April 2025 GA release added a waitForEvent API so a workflow can pause indefinitely for human approval or an incoming webhook, alongside higher concurrency and better metrics, with workflow state retained for 3 days on the free plan and 30 on paid. AWS shipped its version with Python and Node support at launch and brought the Java SDK to general availability in April 2026, which is the sort of follow-through that indicates enterprise pull rather than a launch-week experiment.
The clearest market signal came in February. Temporal raised a $300M Series D at a $5B valuation on 17 February 2026, reporting revenue up more than 380% year over year, weekly active usage up 350%, more than 20 million monthly installs, and 9.1 trillion lifetime executions on its cloud of which 1.86 trillion came from AI-native companies. The same announcement carries a line from OpenAI's VP of App Infrastructure describing durable execution as a core requirement for modern AI systems, and arguing that as those systems become longer-running, durability matters as much as performance. You can discount vendor framing and the underlying ratio still says something: roughly a fifth of all executions on that platform now come from companies whose product is AI.
Set that beside the failure statistics and a specific reading emerges. Gartner predicts more than 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. Notice what is not on that list: model capability. Projects are dying of operational unreliability and the cost of firefighting it, which is exactly the failure mode a state layer addresses. We made the adjacent argument at the architecture level in why your AI agents are production microservices; durable execution is the concrete answer to the state half of that problem.
There is a public before-and-after worth citing, because it is rare to get one with numbers attached. Salesforce's engineering team documented migrating a synchronous AI column-execution path to Temporal-backed durable orchestration: under a load test of ten concurrent users at 200 rows per worksheet, the synchronous version failed roughly 90% of the time with retry storms that took over ten minutes to surface a permanent failure, while the durable version failed 0% of the time, and a related prompt-template migration improved P95 completion time by roughly 60%. The mechanism they credit is mundane and reusable: checkpoint after every successful row, cap retries at ten with exponential backoff, and resume only the incomplete work.
What Durable Execution Actually Buys, and What It Does Not
The mechanic underneath every implementation is the same. Your orchestration code is executed against an append-only history. Each step's result is persisted as it completes, and when the process dies the engine replays the code against that recorded history, feeding back the results it already has, until it reaches the first step that never finished. From the code's point of view nothing happened; from the outside, a six-hour run survived three deploys and a node failure. Suspension falls out of the same design, which is why waiting becomes nearly free rather than a held thread.
That buys four concrete things. Crash recovery without bespoke state machines. Retries with backoff at the step boundary rather than the whole-run boundary. Long, cheap waits for humans, webhooks and rate limits. And an execution history that doubles as an audit trail, which turns out to matter for reasons that have nothing to do with reliability, as anyone building under the oversight and logging expectations we covered in the EU AI Act as an engineering problem will recognise.
It is equally important to be precise about what it does not buy, because the marketing language around exactly-once semantics does real damage here. A durable engine can guarantee that a step's recorded result is used exactly once by your workflow. It cannot guarantee that the outside world experienced the effect exactly once. If the process dies after your code called the payments API but before the result was checkpointed, the engine has no way to know whether the charge went through, and its only honest options are to retry or to fail. Retries give you at-least-once execution. Turning at-least-once into effectively-once is your job, and the tool is an idempotency key.
Replay also imposes a discipline that surprises teams retrofitting existing agents: the orchestration body has to be deterministic. Reading the clock, generating random values, hitting the network directly or branching on ambient state inside the replayable path all break the guarantee, because replay must reproduce the same decisions. Non-deterministic work belongs behind step boundaries where its result gets recorded. This is usually good news for LLM calls, which are the least deterministic thing in the system and which replay handles perfectly well because their outputs are persisted rather than regenerated. It is less good news for prompt and tool changes: editing the shape of a workflow while instances of it are mid-flight is a versioning problem, and the teams that learn this learn it during an incident.
One further limit deserves stating plainly. Durability makes a failed run resumable; it does not make a wrong run right. An agent that confidently completes the wrong task will now do so with excellent checkpointing. Correctness lives in evaluation, which is a separate build we treated in the eval gap, and the two are complementary rather than substitutable.
Key Takeaways
- Durable execution replays your workflow against a persisted history, so crash recovery, step-level retries and free long waits come for free
- It guarantees exactly-once use of a recorded result, not exactly-once effect in the outside world
- Replayable code must be deterministic: clocks, randomness and direct I/O belong behind step boundaries
- Resumability is not correctness; a well-checkpointed agent can still complete the wrong task flawlessly
The Idempotency Arithmetic Nobody Runs Before Launch
Here is the calculation worth doing on a whiteboard before your agent touches a production system. Take a modest deployment: 20,000 agent runs a day, six side-effecting tool calls per run, so 120,000 calls. Assume a 0.5% transient failure rate on those calls, which is optimistic for anything crossing a network to a third party, and a retry policy of up to three attempts. You are now performing on the order of 1,800 retried calls every day. Each of those is a coin flip on whether the first attempt actually landed. If even 1% of them duplicate a customer-visible action, that is eighteen incidents a day, arriving as double charges, duplicate emails, duplicate tickets and duplicate deployments.
The organisational consequence of that arithmetic is worse than the engineering one, because duplicates get misdiagnosed. An agent that creates two orders did not hallucinate; it retried a non-idempotent operation and lost track of the outcome. But the incident report says the AI made a mistake, trust in the system drops, and the team responds by adding validation prompts rather than an idempotency key. We have watched this pattern play out often enough to write it up for a business audience in why AI automation silently gets things wrong, and the technical root cause is nearly always distributed systems hygiene rather than model behaviour.
The fix is well understood and almost never retrofitted cheaply. Every side-effecting operation gets a stable key derived from durable workflow state, typically the workflow instance ID plus the logical step, so that a retry after any kind of restart presents the same key. The receiving system deduplicates on it. Where you control the endpoint that is a database constraint and a stored response. Where you do not, you inherit whatever the vendor offers: many payment and messaging APIs accept an idempotency key header, and many internal REST endpoints and MCP tools accept nothing at all, in which case the deduplication has to move into a gateway you own that records attempted calls before making them. Auditing your tool surface for this property is the single highest-value hour of work available to a team about to put an agent into production, and it pairs naturally with the repository and interface hygiene we described in making a codebase agent-ready.
Checkpoint granularity is the other half of the answer, and the Salesforce write-up above states the rule better than most textbooks: the right retry boundary is usually the smallest meaningful unit of recoverable work. Checkpointing per batch means a failure at row 199 of 200 redoes 199 rows and every side effect they carried. Checkpointing per row means it redoes one. Choose the boundary at the same place you would put a database transaction, and the retry story mostly writes itself.
Finally, be honest about the actions you cannot compensate. Sagas and compensating transactions handle refunds, reversals and deletions, and they are the right pattern for anything with an inverse. An email that has been read has no inverse. A message posted into a customer's Slack has no inverse. A file deleted from a system with no soft-delete has no inverse. For that category the only real protection is a hard gate before execution: an idempotency check that cannot be bypassed, and often a human approval, which is its own state problem.
Human-in-the-Loop Is a State Problem, Not a UI Problem
Every serious agent deployment ends up with approval gates: before a payment, before an outbound message to a customer, before a production change, before anything a regulator would want a named person attached to. Teams tend to treat this as a front-end feature, build a review queue, and then discover that the hard part is on the other side. The agent has to stop, keep everything it knows, survive an arbitrary wait, and resume with new information injected into the middle of its reasoning.
The frameworks have converged on the same shape as the platforms. LangGraph pairs a checkpointer with an interrupt: calling interrupt() persists graph state and hands control back to your application, and the run waits indefinitely until you resume the same thread with a resume command carrying the human's decision. Cloudflare's waitForEvent does the same job at the platform level, with the workflow parked until an HTTP request or API call delivers the event it is waiting for. The pattern is identical in both: suspend, persist, wait for a signal, resume deterministically.
Two operational consequences follow, and both are usually discovered late. First, an approval step converts a compute problem into a coverage problem. If your median approval latency is fourteen hours because the only people authorised to click approve are in one timezone, then your agent's throughput is governed by a rota, not by tokens. That is a staffing question rather than an architecture question, and it is one of the more concrete reasons we argue for real working-hours overlap in the hidden cost of timezone overlap: a nearshore development team sharing four to six hours with your own engineers can hold an approval queue inside the same business day, which an offshore rota structurally cannot.
Second, somebody has to own suspended workflows as an operational surface. A run parked for three days waiting on an approval that will never come is not an error in any dashboard, it is a silent liability holding a lock on a customer record. Stuck-workflow age, approval latency, and the count of instances waiting on each event type belong on the same board as error rates, and someone needs to be on call for them. That is the same organisational question we worked through in what happens when an agent takes the page, and it is why we usually put a senior SRE-minded engineer inside agent delivery pods rather than adjacent to them.
There is a compliance dividend here that is easy to under-sell internally. A durable workflow history is, incidentally, a complete record of what the agent did, in what order, with which inputs, and which human approved which step at which time. If you operate anywhere near the reporting clocks we described in the Cyber Resilience Act's 24-hour reporting window, that history is the difference between reconstructing an incident from logs over two days and answering the question in an hour.
Key Takeaways
- Approval gates are a persistence problem: suspend, persist, wait for a signal, resume deterministically
- Approval latency becomes a throughput ceiling, which makes working-hours overlap an architecture-adjacent staffing decision
- Stuck-workflow age and approval latency need dashboards and an on-call owner; they never surface as errors
- Workflow history doubles as the audit trail regulators and incident reports demand
What Does the State Layer Cost to Run?
Four line items, and the first one is smaller than people fear. Replay is cheap, because it feeds recorded results back into your code rather than regenerating them; you are not paying for inference twice when a container restarts. What you do pay for is genuine retries of failed model calls, which is why step-level retry beats run-level retry on the invoice as well as on the clock. Re-running one failed extraction step costs a fraction of re-running the forty steps that preceded it.
The second is state storage and retention, and it has a sharper edge than it looks. Workflow histories for agent runs are large, because the interesting content is prompts, tool payloads and model outputs. Retention windows are finite by default, as with the 3-day free and 30-day paid retention Cloudflare documents for Workflows, so if your audit or debugging requirements exceed that window you need an export path and a place to put it, which is a data-retention and residency decision as much as a storage one. Note the pleasant asymmetry on the compute side: AWS charges no compute while a durable function is suspended, and allows suspension for up to a year, so long human waits are close to free where they used to burn a worker.
The third is observability, and it is where budgets get surprised. Workflow histories are not traces, and most teams end up wanting both: spans to understand latency, and histories to understand what happened. Emitting full prompt and tool payloads into your telemetry pipeline as well as into workflow state is how you end up paying twice for the same bytes, at the rates we picked apart in the observability bill of the agent era. Sample aggressively, link from trace to workflow instance rather than duplicating the payload, and put the cost per completed agent run on the same dashboard as the token spend, in the spirit of giving your AI feature a real cost of goods.
The fourth is engineering time, and it is the only one that is genuinely large. Retrofitting durability into an agent that was written as a loop over tool calls is two to six weeks of senior work for a single meaningful workflow: identifying step boundaries, extracting non-deterministic work behind them, adding idempotency keys and a deduplication store, building replay tests, versioning in-flight instances, and instrumenting the result. The reason it needs senior work is that none of it is AI work. It is distributed systems work, of exactly the kind that has been unglamorous and well understood for fifteen years, which is a large part of why it goes missing on teams staffed entirely for model integration.
Against that, price the alternative honestly. The cost of not building it is measured in duplicated customer-visible actions, manual reconciliation performed in spreadsheets by people whose job that is not, engineer-hours lost to unreproducible incidents, and eventually the quiet cancellation of an agent programme that worked fine in the demo. That last outcome is the one Gartner's cancellation forecast is describing, and it is considerably more expensive than a month of infrastructure work.
How Do You Retrofit This Without Rewriting the Agent?
The good news is that this is a mechanical migration with a well-defined order, and it does not require throwing away working prompts. The order matters more than the tooling choice.
Start with an inventory nobody has: every tool your agent can call, marked as read-only or side-effecting, and for each side-effecting one, whether the receiving system supports an idempotency key. That single table usually reorders the roadmap, because it shows which actions are currently one retry away from a customer-visible duplicate. Next, pick the smallest workflow that has real business consequences and give it durable orchestration, choosing step boundaries where you would put a transaction. Then add idempotency keys derived from workflow instance and step, with a deduplication store you own for the endpoints that offer nothing. Only then move approval gates into suspension rather than polling loops, because an approval gate on a non-durable workflow is a liability dressed as a control.
Test it the way you would test any distributed system, which is to say adversarially and in CI. Kill the worker mid-run and assert the run resumes without repeating a side effect. Replay a recorded history against modified workflow code and assert the versioning behaves. Force a tool call to fail after the external effect has landed but before the checkpoint, which is the exact scenario idempotency exists for, and assert no duplicate. If your test suite has never simulated a deploy landing in the middle of a live agent run, then that scenario is currently being tested in production. This is also the point where the verification capacity problem we described in twice the code, half the QA stops being abstract.
On staffing, the competence surface here is narrower than it looks and does not require a new hiring category. A workable pod for retrofitting durability across an agent estate is four people: a senior backend engineer who owns workflow decomposition and idempotency, a platform engineer for the execution engine, observability and secrets, an AI engineer who owns prompts, tools and evaluation so that durability work does not silently change behaviour, and a part-time architect to arbitrate the versioning and data-retention trade-offs. At Central and Eastern European rates that is a fraction of what the same four skill sets cost in London or the Bay Area, and we publish the actual arithmetic in what a dedicated development team costs in 2026. This is precisely the shape of work our dedicated development teams take on, and the reason we tend to argue for a standing team rather than a project: durability is not a phase you finish, it is a property you maintain as the tool surface grows.
If you are buying this capability rather than building it, three questions separate teams that have done it from teams that have read about it. Ask them to describe their idempotency strategy for a tool whose vendor does not support idempotency keys, and listen for a gateway that records intent before acting. Ask what happens to in-flight workflow instances when they change a prompt, and listen for versioning rather than a shrug. Ask who gets paged for a workflow suspended for seventy-two hours, and listen for a name and a threshold. The broader checklist lives in how to vet a software development agency, and if you are weighing where in Europe to put the team, we compared the realistic options in Serbia, Poland and Romania as nearshore destinations. Teams building this alongside broader automation work can see how we structure it under AI automation solutions.
Key Takeaways
- Inventory tools as read-only or side-effecting first, and record which receiving systems support idempotency keys
- Sequence the work: durable orchestration, then idempotency keys, then approval suspension, then versioning
- Test adversarially in CI: kill workers mid-run, replay modified code, and fail a call after its effect has landed
- A four-person pod covers the surface; the scarce skill is distributed systems engineering, not prompt engineering
The Missing Piece Is a State Layer, Not a Better Model
The pattern of 2026 is that agent programmes are not failing on intelligence. They are failing on the parts of software engineering that have been well understood since long before anyone put a model in the loop: what happens when a process dies halfway through, how you avoid doing the same thing twice, how a system waits for a human without leaking resources, and how you prove after the fact what actually happened. The reason four platforms shipped the same durable execution primitive inside twelve months, and the reason the market put a $5B valuation on the company that pioneered it, is that the industry has collectively discovered where the wall is. Agents got long enough that the request/response habit broke, and the fix is an architectural one: persist every step, retry at the smallest recoverable boundary, derive idempotency keys from durable state, make waiting free, and keep the history as your audit trail. None of that is exotic, and none of it will be delivered by the next model release. It does require a team that has run distributed systems in production and is still there in month nine when the tool surface has tripled and someone needs to reason about in-flight workflow versions. That is a staffing decision as much as a technology one, and it is the difference between an agent programme that quietly moves into production and one that shows up in the 40% that get cancelled.
Building a team in Eastern Europe?
StepTo helps European and US companies build senior-led nearshore engineering teams in Serbia. Let's talk about what your next engagement could look like.
Start a conversationWritten by
Igor GazivodaCo-founder & CEO · StepTo
Igor has 15+ years in software engineering and business development. Former CTO at a Series A fintech startup, he specializes in scaling engineering teams, nearshore strategy, and AI-driven product development. He holds a Master's in Computer Science from the University of Belgrade and has published on distributed systems architecture.
LinkedIn →