The Agent With a Mouse: Automating the Enterprise Systems That Never Got an API
Computer-use agents now beat the human baseline on OSWorld. That says almost nothing about whether one should be driving your ERP, and the gap between the two is where the money goes.
The Long Tail of Software That Never Got an API
Every automation conversation of the past decade quietly assumed an API. Integrate the systems, orchestrate the calls, put a workflow engine in the middle. It is good advice, and it fails precisely where the money is, because the systems holding the most valuable process knowledge in most European enterprises are the ones nobody can integrate with. A 2004 ERP module with a Java applet front end. A freight carrier's booking portal. A regional bank's payments console that exists only behind a session cookie and a hardware token. A national tax filing site that has never published an interface and never will. These systems expose a screen and nothing else, and somewhere in your organisation there is a person whose actual job is to be a human API adapter for them.
Robotic process automation was built for exactly this. It is not a dead category either, the global RPA market generated roughly $3.8 billion in revenue in 2024 on healthy year-over-year growth, and enterprises are still buying it. What everyone who has operated it at scale knows is that the licence cost was never the real cost. The real cost was maintenance: a bot records a click path against specific selectors and coordinates, the vendor ships a UI refresh on a Tuesday, and forty automations break at once. RPA teams ended up as a permanent repair crew for a system whose entire premise was eliminating manual work.
Computer-use agents are the same premise with different machinery. Instead of recording a selector path, a multimodal model takes a screenshot, decides where to click, moves the mouse, types, takes another screenshot, and repeats. It does not need to know that the button moved, because it is looking at the screen the way a person does. Every frontier lab now ships a version of this, and Gartner projects that 40% of enterprise applications will embed task-specific AI agents by the end of 2026, up from under 5% in 2025.
The pitch writes itself and the demo is genuinely impressive. The engineering question is much narrower than the pitch: for which class of work does driving a graphical interface with a probabilistic model actually beat the alternatives, and what does it cost when it is wrong? That question has answers now, from benchmarks, from security research, and from published cost data. Most of them are less flattering than the demo.
The Benchmark Crossed the Human Line. That Matters Less Than You Think.
OSWorld is the benchmark that matters here, because it is not a sandbox. It is 369 tasks executed on real operating systems against real applications: file management, spreadsheet manipulation, multi-application workflows, browser work, the sort of thing an operations analyst does before lunch. When the paper was published in April 2024, the human baseline was 72.36% and the best agent scored 12.24%, and the authors were blunt that models failed on GUI grounding and operational knowledge, meaning they could neither reliably hit the right pixel nor understand what the application was for.
Two years later the public OSWorld leaderboard has top entries in the mid-80s, comfortably above that human baseline. Whatever else you think about AI progress narratives, going from 12% to 85% on unmodified desktop software in twenty-six months is one of the steepest capability curves in the field, and anyone still arguing that models cannot operate a GUI is working from 2024 information.
Here is the part that gets skipped. OSWorld tasks are short. They take a competent human somewhere between thirty seconds and a few minutes. The capability being measured is grounded, multi-step, but bounded. And bounded is doing enormous work in that sentence, because METR's time-horizon research has found that task duration is the strongest single predictor of whether an agent succeeds, with a correlation strong enough to explain most of the variance in outcomes. Frontier models complete close to 100% of tasks a skilled human would finish in under four minutes, and under 10% of tasks that take a human more than four hours. METR's updated time-horizon work puts the doubling rate of the 50%-success horizon at roughly seven months historically and faster since 2023, while noting that its own measurements above sixteen hours are not reliable with the current task suite. The trend is real and the ceiling is not where the demo suggests.
Now apply that to an actual back-office process. Reconciling a batch of supplier invoices across an ERP, a document store and a banking portal is not a six-step benchmark task, it is forty or more steps with state carried between them, and per-step reliability compounds ruthlessly. An agent that is 99% reliable on each individual step completes the whole chain correctly about 67% of the time. At 98% per step, it is 45%. Nobody sells you the per-step number, and the end-to-end number is the only one your finance team will ever experience.
Key Takeaways
- OSWorld: 369 real desktop tasks, 72.36% human baseline, best agent 12.24% in April 2024, mid-80s by mid-2026
- METR finds task duration is the strongest predictor of failure, with success under 10% beyond four human-hours
- Benchmark tasks are minutes long, enterprise processes are hours long and stateful
- 99% per-step reliability across a 40-step process yields roughly 67% end-to-end success
RPA Failed Loudly. Agents Fail Quietly. That Is Worse.
The failure mode is the thing that changes, and it is the single most important design consideration in this entire category. When an RPA bot broke, it broke loudly and stopped. The selector was not found, the script threw, the queue backed up, somebody got an alert. It was expensive and annoying and it was also, in a specific sense, safe: a halted bot has not done anything wrong to your data.
An agent driving the same screen does not halt. It reasons about what it sees and it proceeds. When the layout changes, that resilience is exactly what you wanted. When it has misread the situation, the same behaviour means it types the amount into the adjacent field, approves the record above the one it was asked about, marks a ticket resolved because the screen looked like resolution, or reruns a step it had already completed because the confirmation banner scrolled out of view. The process completes. The log says success. The damage is discovered by a human three weeks later during a reconciliation.
This is the same dynamic we described for autonomous remediation in AI SRE agents and the on-call rota, and it generalises: with nondeterministic automation, silent success is more dangerous than loud failure, because failure gets noticed. The operational consequence is that you cannot rely on the agent's own report of what it did. Verification has to inspect the resulting system state independently, through a database query, a report export, a reconciliation job, something outside the loop that produced the action. Teams that skip this build an automation whose error rate is structurally unknowable.
There is a related trap in vendor positioning. RPA platforms have absorbed agents rather than being displaced by them, and the marketing now describes a converged product that reads documents, reasons over policy and executes through existing connectors. Convergence at the platform layer does not draw the boundary between the deterministic and the probabilistic parts of your process for you. That boundary is an architecture decision, it belongs to your engineers, and no vendor is incentivised to draw it conservatively.
The Security Problem That Does Not Have a Patch
Indirect prompt injection is not a bug in a particular product. It is a structural property of systems where instructions and data arrive through the same channel. An attacker embeds a hostile instruction in a web page, a calendar invitation, a PDF, a support ticket or a product review, the agent reads it as part of doing its job, and the model has no reliable mechanism for deciding that this text is content to be processed rather than a command to be followed.
The published research on agentic browsers over the past year is not reassuring. Brave's security team documented indirect prompt injection in Perplexity's Comet browser, where instructions planted in ordinary page content could steer the agent. Zenity Labs went further in March 2026, demonstrating zero-click agent hijacking including a chain that reached into an authenticated password manager session and exfiltrated stored credentials. At Black Hat USA 2026 a Brave engineer reported that every AI browser examined was vulnerable, and the industry read on this, summarised well by Dark Reading, is that there is no complete fix available, only mitigation.
The reason this hits computer-use automation harder than it hits a chatbot is the confused deputy problem. A GUI agent works because it inherits a real human session: the SSO cookie, the open tabs, the saved credentials, the permissions of whichever operator's desktop it is running on. That inheritance is the entire value proposition and the entire attack surface, and they cannot be separated. An agent that can see your ERP the way Marta in accounts payable sees it can also do everything Marta can do, including the things Marta would never do.
Anthropic's own computer use documentation is unusually direct about this, recommending a dedicated virtual machine or container with minimal privileges, allowlisted network access, keeping sensitive credentials out of the model's reach entirely, human confirmation for consequential or consent-bearing actions, and review of action logs. Read that as a deployment specification rather than a liability disclaimer, because it is a fair description of the minimum viable architecture. The practical translation for an enterprise process is a scoped service account per automation with only the permissions that one process needs, credentials injected by the harness and never present in the model's context, and an egress allowlist. We covered the wider version of this failure in the overprivileged agent problem and its supply-chain cousin in prompt injection through the toolchain.
Key Takeaways
- Indirect prompt injection is structural, not a product defect, and has no complete fix today
- Documented 2026 attacks include zero-click hijacking and credential extraction from an authenticated session
- A GUI agent inherits the operator's full session, which is simultaneously the value and the attack surface
- Minimum viable deployment: isolated VM, scoped service account, credentials held by the harness, egress allowlist
Screenshots Are Not Free: The Unit Economics Nobody Models
The cost profile of a GUI agent is unlike anything else in your automation estate, because every single step ships a high-resolution image into a model. Published measurements from the WebMall web agent benchmark put a representative task at roughly $0.28 with a frontier model, built from around 765 image tokens per screenshot plus text context and output, across about seven actions. Configuration matters enormously: cheaper open-weight setups landed between $0.07 and $0.12 per task while the most capable configurations ran from $0.34 to $0.56, with total input token counts between roughly 120,000 and 320,000 per task and wall-clock runtimes from two to six minutes.
Set that against the alternative. The same state change executed through an API is a few milliseconds and a rounding error on your cloud bill, executed identically every time, with an error that is a status code rather than a probability. A GUI agent is two to three orders of magnitude more expensive per operation and less reliable. That is not an argument against it, it is an argument about where it belongs.
The arithmetic decides it for you. Fifty thousand invoice lines a month at thirty cents each is $15,000 a month, $180,000 a year, before you count the cost of the failures and the humans checking for them. A one-off integration project against the same system, even an ugly one involving screen scraping, a headless session and a fortnight of reverse engineering, is cheaper by the end of the first quarter and cheaper by an order of magnitude over three years. Invert the volume and the answer inverts too: a process that runs eleven times a month, against a supplier portal you will stop using next year, is never worth an integration and is a perfectly sensible thing to hand to an agent.
Prompt caching helps at the margin, with providers advertising input cost reductions up to around 90% on cached prefixes, and it should absolutely be part of any production deployment. It does not change the ordering. The metric to instrument from day one is cost per successfully completed task, not cost per token and definitely not cost per agent run, because a run that completes and does the wrong thing is negative value that a token dashboard will happily report as throughput. We have written about the broader version of this discipline in the AI invoice nobody budgeted for.
Key Takeaways
- Published benchmarks put a representative GUI agent task around $0.28, ranging from $0.07 to $0.56 by configuration
- Input consumption runs 120,000 to 320,000 tokens per task with two to six minute runtimes
- High-volume stable processes favour integration, low-volume high-variance processes favour agents
- Instrument cost per successfully completed task, not cost per token or per run
Agent at the Edge, Determinism at the Core
The architecture that survives contact with production puts the probabilistic component where variance lives and keeps deterministic code where consequences live. The agent perceives, interprets and handles the messy edge, then hands a structured result to ordinary software that does the actual writing, the actual posting, the actual money movement. This is the same separation that makes multi-agent systems behave like production microservices rather than like a demo.
Start read-only, and stay there longer than feels necessary. An agent that opens the portal, reads the status of two hundred shipments and writes them into a table delivers most of the value of the automation, carries almost none of the risk, and produces the evidence you need to decide whether to grant write access at all. Grade its extractions against a human-verified sample for a few weeks and you will know its real accuracy rate, which is the number that should drive every subsequent decision. An agent that is right 85% of the time is genuinely useful when you know it is 85%, and dangerous when someone in the steering committee believes it is 99%.
Then record and promote. Let the agent run the process, capture the full trace of what it did, and once a stable path emerges, promote that path into scripted, deterministic code. The agent stops being the daily driver and becomes the fallback for the day the script breaks, which is precisely the failure the old RPA model had no answer for. This inverts the vendor pitch, since it deliberately reduces agent usage over time, and it is the pattern that makes the economics work: you pay agent prices only for genuine novelty.
Everything that writes needs tiering by blast radius and reversibility. Reading a status, downloading a report and populating a draft are one tier. Submitting a form, approving a record, sending money and anything that touches a customer are another, and those need either human confirmation in the loop or a hard-scoped, reversible, fully audited path. Underneath all of it sits the thing most teams build last and should build first, an evaluation set: a fixed bank of recorded task instances with known correct outcomes, graded automatically, tracking success rate, cost and step count on every model or prompt change. Anthropic's guidance on evals for agents is a reasonable starting point, and the reason it matters here more than almost anywhere else is that your automation's behaviour changes every time the provider ships a model update you did not ask for. We made the general case in the eval gap.
Key Takeaways
- Put the agent where variance lives and deterministic code where consequences live
- Run read-only for weeks and grade against human-verified samples to learn the real accuracy rate
- Record and promote stable paths into scripted integrations, keeping the agent as the fallback
- Tier write actions by blast radius, and build the eval set before the automation, not after
Why 40% of These Projects Will Be Cancelled
Gartner's forecast that over 40% of agentic AI projects will be cancelled by the end of 2027 came from a poll of more than 3,400 organisations actively investing in the technology, and the stated causes are escalating costs, unclear business value and inadequate risk controls. The same analysis notes that a large share of the market is agent washing, with Gartner estimating that only around 130 of the thousands of vendors claiming agentic capability are building anything that meets the definition. The adoption gap is just as telling: a modest minority of organisations have agents genuinely in production while a large majority expect to within two years, which is the exact shape of a category about to produce a great many disappointing pilots.
The projects that die rarely die of capability. They die of scoping. A team looking for executive sponsorship picks the flagship process, which is by definition the highest-volume, most complex, most regulated and most integrated one in the company, and discovers eight months in that it has taken on a long-horizon multi-system workflow with compounding per-step error, an unbounded token bill and an auditor asking who authorised the payment run. The boring process nobody wanted to present to the board, the one where a contractor logs into a supplier portal twice a week to pull a report, would have shipped in a month and worked.
The compliance layer is arriving on its own schedule regardless. Automated actions taken against regulated systems need an audit record that shows what was proposed, what was approved, by whom or by what policy, and what actually changed, and retrofitting that after a year of unlogged autonomous operation means a year you cannot account for. For European organisations this sits alongside the obligations we walked through in the EU AI Act as an engineering problem, and any vendor that cannot describe its audit schema and its action-tier model before it describes its accuracy rate is selling the wrong half of the product. If you want the shorter version of that filter, we wrote it up as how to spot AI washing before you hire anyone.
There is one question that settles most of these business cases in an afternoon. If you removed the agent entirely and simply built the integration, would the project still be worth doing? If the answer is yes, build the integration, because you have just discovered that the value was in connecting the systems rather than in the model. If the answer is no, because the target system genuinely has no interface and the volume genuinely does not justify reverse-engineering one, you have found a real use case for a computer-use agent, and you should scope it as narrowly as that answer implies.
This Is Integration Engineering, and Most Teams Staff It Wrong
Notice what the work actually consists of. Reverse-engineering an undocumented legacy interface and learning how it misbehaves. Designing a scoped service account model and a credential path that keeps secrets out of the model's context. Writing verification that inspects end state rather than trusting a summary. Building an eval harness and keeping it current. Instrumenting cost per successful outcome and tracing every action for audit. Knowing which parts of the process to promote out of the agent and into code, and having the discipline to do it. Not one of those is prompt engineering. All of them are ordinary, senior, unglamorous integration and platform work of exactly the kind that decides whether an AI initiative ships, which is the pattern we described in why your AI is not working and in why AI agents fail at business automation.
Most organisations staff it in one of two ways that predict the outcome. Either an innovation or AI team owns it, strong on models and thin on the integration, security and operations experience that this work is mostly made of, or a vendor owns it whose commercial model is denominated in agent runs and who therefore has no reason at all to help you promote work out of the agent. The staffing that works is a small senior team that owns the process end to end, treats the agent as one component among several, and is measured on the business outcome rather than on how much of the solution is AI.
This is the shape of engagement we build at StepTo. We have run senior-led dedicated development teams out of Serbia since 2014, with engineers assigned by name rather than rotated off a bench, which matters more in automation work than in almost anything else we do. The operational knowledge of how a twenty-year-old logistics system actually behaves when a session expires mid-transaction takes months to accumulate and one staff rotation to lose entirely. Process automation is also unusually conversation-dependent: half the real questions are things only the person who does the job today can answer, and resolving them needs live overlap rather than an overnight ticket round trip. A team in Belgrade shares the full European working day and the start of the US East Coast one, which is the practical case for nearshore delivery here, over and above the rate difference. The underlying numbers are in what a dedicated team actually costs and in the hidden cost of timezone overlap.
If you want a first engagement that tells you something true, scope it like this: one process, chosen for being boring rather than strategic, four to six weeks, read-only for the first half, with a graded accuracy figure, a cost per successful task and a written recommendation on which parts should be promoted into deterministic code at the end. That is a small enough commitment to be worth making on an unproven category, and large enough that the answer is real. If the recommendation that comes back is to build an integration and drop the agent, the engagement has done its job and saved you a year.
The Bottom Line
Computer-use agents are one of the genuinely new capabilities of 2026 rather than a repackaging of something older, and the benchmark progress is not marketing: going from 12.24% to the mid-80s on 369 real desktop tasks, past a 72.36% human baseline, in twenty-six months is a serious result. What it does not license is the conclusion that an agent should now be operating your ERP unsupervised. The benchmark measures short tasks, real processes are long, per-step reliability compounds, and the failure mode has shifted from a bot that stops to an agent that confidently does the wrong thing and reports success. Layer on indirect prompt injection that nobody has a complete fix for, in a system whose entire design inherits a human's authenticated session, and a per-task cost two to three orders of magnitude above the API call it is imitating, and the shape of a sane deployment becomes fairly obvious. Point the agent at the boring, low-volume, high-variance corners where no interface exists and none is coming. Keep it read-only until you have measured what it actually gets right. Hold the credentials outside the model, verify against system state rather than the agent's own account of itself, and promote every stable path into deterministic code as soon as it stabilises. Done that way, this is a genuinely useful tool for the ugliest part of the enterprise estate. Done the way it is currently being sold, it is a well-funded way to automate your mistakes.
Building a team in Eastern Europe?
StepTo helps European and US companies build senior-led nearshore engineering teams in Serbia. Let's talk about what your next engagement could look like.
Start a conversationWritten by
Igor GazivodaCo-founder & CEO · StepTo
Igor has 15+ years in software engineering and business development. Former CTO at a Series A fintech startup, he specializes in scaling engineering teams, nearshore strategy, and AI-driven product development. He holds a Master's in Computer Science from the University of Belgrade and has published on distributed systems architecture.
LinkedIn →