The Model Ships With the App: What Moving AI Inference On-Device Actually Demands From Your Engineering Team

Why Is Inference Moving Back Onto the Device?

For roughly three years, the default architecture for any AI feature was an HTTPS call to somebody else's GPU. That was the right call at the time: the capability gap between a frontier model and anything you could run locally was enormous, the integration was an afternoon's work, and the bill was small enough that nobody in finance asked about it. Every one of those three conditions has weakened in 2026, and two of them have reversed.

The cost condition reversed first, and not in the way most teams expected. Per-token prices really are falling. What is rising is the amount of inference a single unit of user-visible work consumes. Gartner's August 2026 forecast, which it labels the inference paradox, expects token costs to fall by around 95% by 2030 while inference cost per agentic workflow rises more than fivefold over the next two years, on the mechanism that routing a task to a reasoning agent instead of a simple chat completion multiplies provider inference cost at least five times before task complexity is accounted for. The same firm's infrastructure numbers show where the money is going: AI-optimised infrastructure-as-a-service spending is forecast to grow 96% in 2026, with inference rather than training taking the majority of it. Cheaper tokens have not produced cheaper products.

The capability condition reversed second, and it happened on hardware you do not own and do not pay for. Counterpoint Research puts AI-advanced PC penetration at roughly 59% of global shipments in 2026, up from about 39% in 2025, and Gartner's tracking of the same transition had AI PCs at 31% of the market at the end of 2025 on the way to a majority share. The definitional bar is not marketing: a Copilot+ designation requires a neural processing unit rated at 40 TOPS or better, and that class of part is now standard rather than premium. On the phone side, the equivalent silicon has been shipping for several generations and is now addressable through supported platform APIs rather than through a research SDK.

It is worth being precise about what this article is and is not about, because there is an adjacent decision that gets conflated with it constantly. Deciding to host a fine-tuned open-weights model on infrastructure you control instead of calling a frontier API is a server-side economics question, and we worked through it in detail in the case for small language models in production. This is the other question: whether the model runs on hardware you neither own, monitor, nor control, in the hands of a user who may be offline, on a metered connection, on a three-year-old handset, or in a jurisdiction whose regulator would rather the data never left it.

What Actually Runs on a Phone or Laptop in 2026?

The honest answer, and the one that surprises teams who last evaluated this in 2024, is: more than you need for most of what you are currently paying an API to do. A 4-bit quantised model in the 7B to 8B range sustains roughly 15 to 30 tokens per second on current mobile and laptop silicon inside a power budget under ten watts, which is comfortably above reading speed and well inside the latency envelope for an inline assist feature. Comparisons of the 2026 edge runtime stack make the same point differently: the meaningful metric stopped being peak TOPS and became sustained tokens per second under thermal throttling, because a chip that hits its rated number for eight seconds and then halves is useless in a product.

Below the phone and laptop tier there is a second, quieter market that matters for anyone building connected products. Embedded NPUs in 2026 typically deliver 2 to 10 TOPS at 2 to 6 watts, which is not going to run a general assistant but is more than enough for on-device vision, anomaly detection, wake-word and intent classification, and the structured-extraction step that used to require a round trip to a backend. If you build industrial, medical, retail or automotive hardware, the inference-placement decision is no longer theoretical; we covered the surrounding cost structure in our piece on what an IoT product actually costs across its supported lifetime.

The runtime layer consolidated faster than most teams noticed. ExecuTorch, the PyTorch-native on-device runtime, reached 1.0 with a base footprint around 50KB and a delegate system spanning a dozen hardware backends, and it is not a demo: it runs AI features in consumer applications at a scale measured in billions of users. ONNX Runtime remains the framework-agnostic alternative, reaching NPUs through Core ML, NNAPI and QNN execution providers, and a direct comparison of the two puts roughly 80% of the most popular edge LLMs on Hugging Face as having working ExecuTorch exports. For teams who want a single binary and no build pipeline, llama.cpp and its GGUF ecosystem still cover an enormous amount of ground.

The bigger change is that you may not need a runtime at all, because the platform vendors now ship one. Apple's Foundation Models framework was opened up in 2026 to reach the on-device model and third-party cloud models, including Gemini, through one Swift interface, and the hybrid mode in Firebase AI Logic lets an SDK prefer or fall back to the on-device model instead of always hitting the cloud. On Android, Gemini Nano is reached through system-level GenAI APIs rather than as a raw model handle. Requests served by the on-device model carry no per-call charge, which is the entire economic argument in one sentence.

What this class of model is genuinely good at is narrow and useful: classification, entity and field extraction, summarisation of text already on the device, rewriting and tone adjustment, proofreading, image description, intent routing, and producing structured output against a schema. What it is not good at is long-horizon reasoning, broad world knowledge, multi-step tool use, and anything where being subtly wrong is expensive. That split is the design input for everything in the rest of this article.

The Three Arguments That Actually Get This Signed Off

In practice, on-device inference gets approved for one of three reasons, and it is worth knowing which one you are arguing, because they imply different architectures and different definitions of success.

The first is unit economics, and it is the least interesting technically and the most persuasive commercially. An on-device call has a marginal cost of zero to you. That does not make it free, it makes it capital rather than operating expense, paid once in engineering and thereafter borne by the user's battery. The features where this matters are the high-frequency, low-complexity ones: the classification that runs on every message, the extraction that runs on every uploaded document, the suggestion that fires on every keystroke pause. Those are precisely the calls that ruin a gross margin, because they scale with usage rather than with revenue. If you have not instrumented which of your AI calls are cheap-and-constant versus expensive-and-rare, that is the prerequisite work, and it is the same instrumentation we argued for in pricing AI features you cannot measure and in our piece on AI FinOps and the invoice nobody budgeted for.

The second is latency, and specifically tail latency. A local inference call avoids network transit, provider queueing, and the retry you did not plan for. The gap is roughly an order of magnitude, and it is the difference between a feature that feels like part of the interface and one that feels like a web request. It also changes what is buildable: inline completion, live transcription and real-time redaction are all latency-bound rather than capability-bound problems, and a cloud round trip prices them out regardless of how good the model is.

The third is data residency, and for European clients this is increasingly the argument that ends the discussion. If the personal data never leaves the handset, there is no international transfer to assess, no additional processor to add to your Article 28 arrangements, and no transfer-impact analysis to keep current. That is not a small saving in paperwork; it removes an entire class of ongoing compliance obligation from the roadmap. It sits alongside the sovereignty pressure we described in what EU cloud sovereignty actually requires from engineering, and it interacts directly with the access-control questions in giving outsourced developers access to production data. A model that runs locally cannot leak a prompt to a third party, because there is no third party in the path.

There is a fourth reason that rarely appears in the business case and frequently appears in the incident review: the feature keeps working when the network does not. For field service, logistics, manufacturing, maritime, healthcare and anything installed in a building with thick walls, that is not a nice-to-have. It is the difference between a product and a demo.

The Bill Nobody Budgets: You Are Now Running a Fleet

Here is the trade every team makes and few teams price. A cloud model is a configuration value. You change a string, you deploy, and within minutes every user is on the new model. If it regresses, you change the string back. An on-device model is a binary artefact in the hands of people who did not ask for it and may never update it.

The first consequence is a combinatorial testing problem. You are no longer validating one model against one set of prompts. You are validating a matrix of model version by quantisation scheme by device tier by operating system version, and, critically, every upgrade path into the current version. Practitioner accounts of fleet updates in 2026 make the point bluntly: each new version must be tested not only on its own but as an upgrade from multiple prior versions, because devices inevitably lag, and without proper observability something like 15% of a fleet ends up running a version that was deprecated six months earlier. That is a release engineering discipline, not an ML one, and it is usually missing from the team that proposed the feature.

The second consequence is delivery weight. Model weights are hundreds of megabytes even after aggressive quantisation. That collides with app store size limits, with users on metered connections, with enterprise MDM policies that throttle background downloads, and with the perfectly reasonable expectation that an app update should not consume a gigabyte. The answers exist, on-demand resources, delta updates, tiered model bundles where the small model ships and the larger one downloads on first use over unmetered connections, but each one is engineering you have to build and then support.

The third consequence is the one that catches people out: rollback is not symmetric. If a new model version has already been used to generate embeddings, summaries, classifications or local index entries on the device, reverting the model does not revert the artefacts it produced. You need versioned local state, a migration path, and a policy for what happens to data produced by a model version you have withdrawn. Teams that treat the on-device model as a stateless function discover this in production, usually via support tickets about search results that stopped matching.

None of this is exotic. It is the same discipline that connected-hardware teams have practised for years, which is exactly why the skills to do it are concentrated in embedded and mobile engineering rather than in the application teams currently shipping AI features. Staged rollout with a real kill switch, per-cohort telemetry, forced-update thresholds and a supported-version window are the baseline, and they need to exist before the first model ships, not after the first bad one does.

Quantisation Is a Product Decision, Not a Build Flag

The single most common way this project goes wrong is treating quantisation as a packaging step. Somebody exports the model at 4-bit because that is what fits, the demo works, and the accuracy question is never asked in a form that could produce a 'no'.

The mechanics are well understood. Post-training quantisation methods such as GPTQ and AWQ retain most of a model's quality while cutting memory roughly fourfold, and the failure mode is specific rather than general: outlier activations blow up under naive low-bit quantisation, which is what techniques like SmoothQuant and SpinQuant exist to reshape. A comprehensive survey of on-device AI models catalogues the trade-offs across compression, memory and latency for exactly this class of deployment, and work on quantisation-aware training at inference cost is motivated explicitly by the accuracy loss that post-training methods still suffer in low-bit regimes. The lesson is not that 4-bit is bad. It is that the accuracy cost is task-dependent and cannot be inferred from a benchmark average.

The third axis is energy, and it is the one that has no cloud analogue at all. Evaluations of quantised LLMs on edge hardware measure energy efficiency, output accuracy and inference latency together, because on a battery-powered device they are a single trade-off rather than three independent ones. A quantisation scheme that gains you four points of accuracy and costs the user 20% of their afternoon is a product regression, and it will be reported as one, in the app store, by people who will not mention the model.

What this means practically is that your evaluation suite has to fan out. One pass against one model is not evidence. You need the same task suite run against each candidate quantisation on each supported device tier, with accuracy, p95 latency and energy per inference recorded, and you need it in CI on real hardware rather than on a simulator. This is the on-device instance of a gap we have written about at length in the eval gap that costs more than teams expect, and it is worse here, because the cloud version of this mistake is a bad week and the on-device version is shipped to a million handsets.

Hybrid Routing Is the Architecture That Actually Ships

Almost nobody ends up fully on-device, and the teams that try usually ship something disappointing. The architecture that works is a router with an explicit policy, and the policy has three inputs: how sensitive the data is, how hard the task is, and whether the network and the device are currently able to serve it. Architecture guides for hybrid cloud-local deployments converge on precisely those three dimensions, which is a good sign that the shape is real rather than fashionable.

It is also shipping in products rather than sitting in slide decks. Perplexity demonstrated a hybrid local-cloud inference system at Computex 2026 that routes tasks automatically between the user's machine and the cloud, and the platform SDKs described earlier expose the same pattern as a first-class option rather than something you assemble yourself. The interesting engineering is no longer in making local inference work. It is in the routing policy and everything downstream of it.

Downstream is where the cost lands. You now have two prompt paths with two different capability profiles, which means two prompt sets to maintain, two output distributions to validate, and a contract between them if the feature is supposed to behave consistently. Structured output helps enormously here, because a schema gives you something to assert against on both paths; free-form generation does not. You need an escalation policy that is explicit about when a local answer is insufficient, and you need to decide, as a product question rather than an engineering one, whether the user is told which path served them. Teams that leave this implicit ship a feature whose quality varies by handset and cannot explain why.

Then there is telemetry, and this is the constraint people miss until late. You have just promised that certain data never leaves the device. That promise also applies to your logs. You cannot debug an on-device regression by shipping the prompt and the completion to your observability stack, which means investing in on-device aggregation, sampled and consented capture, and metric-level signals such as schema-validation failure rates, fallback rates and latency distributions rather than raw traces. It is less convenient and considerably cheaper, which is a rare combination, and it connects to the broader problem we covered in the observability cost crisis in AI telemetry.

One design note worth stating early: build the router around a capability interface, not around a vendor's SDK. The on-device model you ship in 2026 will not be the one you ship in 2028, and the platform APIs are moving quickly. The reasoning is identical to the argument in designing an exit strategy for model portability, with the added twist that on-device you cannot deprecate the old path on your own schedule, because old devices keep running it.

The Security Surface That Comes With Shipping a Model

Moving inference to the device removes an entire category of risk and creates a smaller, less familiar one. It is worth naming the second category before someone else does.

The model is now on hardware an adversary can own outright. Weights can be extracted, inspected and tampered with, which matters if the model embeds proprietary behaviour, a fine-tune on sensitive corpora, or classification logic you would prefer competitors not read. The mitigations are ordinary supply-chain hygiene applied to a new artefact type: signed model bundles, integrity verification before load, platform attestation where available, and treating the model file with the same seriousness as an executable, because in every meaningful sense it is one.

The on-device AI subsystem is itself an attack surface. Security research examining on-device AI deployment has demonstrated concrete risks in system-level AI components that applications call into implicitly, and the practical implication is that adopting a platform's local model inherits that platform's exposure. That is usually an acceptable trade, but it should be a decision with a named owner rather than a default.

Prompt injection does not go away, and some teams assume it does. A local model summarising a document the user just received is reading untrusted input, and if that model can call tools, touch local files or trigger network requests, the injection has somewhere to go. The same containment thinking applies as in the agent context we described in supply chain attacks on AI coding agents: constrain the tool surface, treat model output as untrusted input to whatever consumes it, and never let a local model's output reach a privileged action without a check that a human or a deterministic rule performs.

Finally, the credentials. A hybrid architecture means the device holds something that lets it reach the cloud path, and an API key in a shipped binary is an API key that is public. Short-lived tokens brokered by your backend, per-device identity and revocation, and rate limits scoped to a device rather than to an application are the baseline, and the wider failure pattern is the one we set out in secrets sprawl in the age of AI agents.

This Needs a Team Shape Most Companies Do Not Have

Look back over the previous five sections and notice what the work actually is. Model selection and quantisation with a real evaluation harness. Mobile or embedded engineering close to memory, thermal and power budgets. Release and fleet engineering with staged rollout, version windows and a kill switch. Backend work for the cloud path, the token broker and the aggregation pipeline. Product judgement about when a local answer is good enough. That is four disciplines and a decision-maker, and almost no organisation has all of them sitting in the same squad.

What usually happens is that the AI team builds a proof of concept that works beautifully on a current-generation development machine, and it then stalls for two quarters against problems that are entirely unglamorous: an accuracy regression on the two-year-old device tier that represents 40% of the installed base, a model download that trips enterprise MDM policies, an evaluation suite that cannot run on real hardware in CI, and a rollback story that nobody wrote down. None of those are research problems. All of them are delivery problems, and delivery problems are what a standing team is for.

This is a strong fit for a dedicated development team rather than a fixed-scope project engagement, for a structural reason. On-device inference is not a feature you finish. It is a capability with a long tail: a new model generation every few months, a new device tier every year, an OS release that changes NPU scheduling behaviour, a quantisation improvement worth re-evaluating, and a fleet that needs someone watching version distribution indefinitely. You want the people who chose the quantisation scheme in month two to still be there in month fourteen when it needs revisiting, which is the argument we made more generally in dedicated teams versus project-based outsourcing.

It is also a strong argument for nearshore development specifically, because the hardest decisions in this programme are conversations rather than tickets. Whether to accept a two-point accuracy drop for a 30% battery improvement is a joint product, engineering and commercial call, and it does not survive being written into a ticket and answered nine hours later. Our engineering teams in Serbia work a full business day overlapping European hours and most of the US morning, which is the difference between settling a trade-off in an afternoon and settling it next sprint. For clients weighing that geography against the alternatives, we compared them directly in Serbia, Poland and Romania as nearshore destinations.

Where the domain knowledge already sits in-house and what is missing is capacity in a specific discipline, usually embedded or mobile engineers who are comfortable with model runtimes, staff augmentation into your existing squad is the faster route. Where the AI capability itself is being built alongside the product, our AI automation and engineering practice treats inference placement, evaluation and fleet update as part of the initial architecture rather than as a retrofit, and if you want to start with the people rather than the programme, you can hire AI developers directly into a team you already run.