Your Fallback Plan Is a Retry Loop: What 2026's Outage Year Should Change About How You Build
Third-party providers now account for roughly two-thirds of publicly reported outages, fewer than a third of organisations run any failover test, and the AI layer is the least reliable dependency most products have. The engineering work that fixes it, and why nobody gets to it.
What Actually Broke in 2026?
On 16 September 2026, Salesforce spent roughly four and a half hours in a global incident that its own status note attributed to requests stalling on an internal login service. In the same week, ThousandEyes counted 531 global network outage events, of which 187 were public cloud outages, up 7% on the week before. None of that made the front page. It was an ordinary week.
The aggregate picture is more useful than any single incident. A review of first-half 2026 cloud and SaaS reliability logged 4,723 incidents across 86 cloud providers, 4,589 across 187 developer-tooling providers, and 2,730 across 34 AI and LLM providers. May was the worst month, at 6,070 incidents. The report's own summary of the half-year is one word long: dependency risk.
The individual failures are where the lesson lives, because almost none of them look like the disaster scenarios in a business continuity plan. Railway lost eight hours on 19 and 20 May 2026 when an automated account suspension at Google Cloud disabled its API and control plane, which then took down customer workloads that were running on AWS, a provider that was working perfectly at the time. Railway's own assessment, quoted in the same report, is the most honest sentence any vendor wrote this year: they took full responsibility for the architectural decisions that allowed a single upstream provider action to cascade into a platform-wide outage. A GitHub pull request search outage over 28 April to 1 May ran two days and thirteen hours after an Elasticsearch repair job deleted 1.8 billion PR documents without safety flags. A datacentre fire degraded Google Cloud's India regions for 21 days and 12 hours.
The template for all of it was set the previous autumn. AWS's own post-event summary describes a latent race condition between two DNS Enactors that left the regional DynamoDB endpoint with an empty DNS record, which the automation could not then repair. DynamoDB was down, and because EC2, Lambda, ECS, STS and IAM authentication lean on it, most of us-east-1 went with it. Instance launches did not fully recover until 1:50 PM PDT the following afternoon, about fifteen hours after the first failure. No component was badly built. The dependency graph was simply deeper than anyone had drawn it.
The uncomfortable part is that this is the forecast, not the anomaly. Forrester principal analyst Lee Sustar expects at least two major multiday hyperscaler outages during 2026, on the reasoning that the providers are diverting investment away from legacy x86 and ARM environments toward GPU-centric datacentres while the older infrastructure ages under growing complexity. Your capacity plan assumes the platform underneath is getting better every year. That assumption is now contested by the people who study the platforms for a living.
Key Takeaways
- A single mid-September week produced 531 tracked global outage events, 187 of them in public cloud
- First-half 2026 logged 4,723 cloud incidents, 4,589 developer-tooling incidents and 2,730 AI provider incidents
- The Railway outage took down workloads on a healthy cloud because the control plane lived on a different one
- A fifteen-hour region-wide failure began with an empty DNS record, not a datacentre fire
Why Most of Your Downtime Now Starts Outside Your Code
There is a genuine paradox in the reliability data, and reading only half of it produces the wrong strategy. On a per-site basis, infrastructure is getting more reliable: the Uptime Institute's 2026 annual outage analysis records the fifth consecutive year of declining outage rates per site, although the pace of improvement has slowed. The same report notes that third-party IT and datacentre service providers, meaning the cloud and internet giants, the telcos and the colocation companies, account for about two-thirds of publicly reported outages, and that external infrastructure failures linked to fibre and connectivity are becoming more prominent and more likely to cause extended disruption.
Put those two findings together and you get the actual shape of 2026. Individual facilities fail less often. The systems built on top of them fail in bigger, wider, more correlated ways, because everyone is standing on the same few slabs. The reliability of the component improved; the blast radius of its failure grew faster.
Concentration is the mechanism. Synergy Research's second-quarter 2026 numbers, reported at 28% for AWS, 20% for Microsoft and 15% for Google, against a $143.4 billion quarter, put roughly 63% of worldwide cloud infrastructure spending through three companies. Add the CDN and edge layer, the identity providers, the payment processors and now the model APIs, and the median European SaaS product has perhaps eight organisations that can independently stop it from working. Your uptime is the product of theirs, and you do not get a vote on any of the multiplicands.
The cost side has moved in the same direction. In the Uptime survey underlying that 2026 analysis, 57% of respondents said their most recent major outage cost more than $100,000, and for the second consecutive year one in five put the cost above $1 million. Those are not numbers that justify a quarterly backlog ticket labelled 'improve resilience'. They are numbers that justify a named workstream with an owner.
This is also the honest counterweight to the cloud-exit conversation. We have argued elsewhere that repatriation decisions should be run on numbers rather than ideology, and that holds here too. Leaving the hyperscalers does not remove dependency risk, it re-shapes it, usually into a smaller provider with a thinner operations team. The goal is not independence. It is knowing precisely which failures you have accepted and what your product does while they last.
Is Your AI Layer the Least Reliable Dependency You Have?
Almost certainly, yes, and the measurement is not close. Across more than 215 tracked services, AI and ML APIs rank as the least reliable API category, well behind payments infrastructure, where Stripe sits around 99.99%. The same analysis counts eleven OpenAI incidents in the 28 days of January 2026, roughly one every two and a half days, most resolved within 30 to 90 minutes.
Early September made the point publicly. Over 2 and 3 September 2026, OpenAI, Anthropic and Google all posted disruptions inside roughly twenty-four hours of each other: OpenAI from 10:58 to 11:22 UTC across fifteen ChatGPT components and four Codex components, Anthropic with about twenty-five minutes of elevated error rates the day before, and Gemini with the longest of the three at roughly two hours. Consumer interfaces, developer APIs, image generation and account creation were all in scope. Teams whose product routes every request through one model provider discovered what their fallback path actually was, which for most of them was an exponential backoff that eventually gave up.
The arithmetic of agentic workloads is what turns a good-sounding SLA into a bad user experience. At 99.9% availability, a service is entitled to be down for roughly 8.8 hours a year, which sounds tolerable for a single request. Chain forty tool calls together in an agent run at that same per-call availability and the probability that the whole run completes is about 96%, meaning one run in twenty-five dies partway through. If that run had side effects, which is the entire point of agents, you now own a partial-completion problem rather than an availability problem, which is why StepTo keeps arguing for a durable execution layer underneath long-running agents rather than a retry decorator on top of them.
Infrastructure leaders have priced this in even where their architectures have not. In Cockroach Labs' 2026 survey of 1,125 senior architects, engineers and executives, 77% expect AI to drive at least 10% of all service disruptions in the year ahead, 36% name cloud infrastructure as the first thing that breaks under AI-driven load and 30% name the database, and 83% believe their data infrastructure will fail without major upgrades within twenty-four months, with 34% putting that inside eleven months.
The mitigations are not exotic, they are just work nobody scheduled: a provider-abstraction layer so a model swap is configuration rather than a refactor, a defined secondary model with its own prompt and eval baseline, and a tested route between them. We have written separately about why model portability is an architecture decision rather than a procurement one; availability is simply the version of that argument that arrives on a Thursday morning instead of at contract renewal.
Key Takeaways
- AI and ML APIs measure as the least reliable API category across 215+ tracked services
- One provider logged eleven incidents in 28 days, roughly one every two and a half days
- A forty-step agent run at 99.9% per-call availability completes about 96% of the time
- A second model provider is only a fallback if something has actually routed traffic to it
Nobody Tests the Failover, and the Invoice Still Arrives
The gap between what organisations believe about their resilience and what they have demonstrated is the single most consistent finding in this literature. Cockroach Labs' survey of 1,000 senior cloud architects and technology executives found fewer than one third conducting any failover testing, 100% reporting outage-related revenue loss in the prior year, an average of 86 outages per organisation per year at an average length of 196 minutes, and average outage-related losses of $495,000 at large enterprises.
Read that again with an engineer's eye rather than a marketer's. Eighty-six outages a year is one every four and a half days, and 196 minutes is more than three hours of degraded or absent service per event. And the failover path that is supposed to contain those events is, for most organisations, an untested assertion in a document. A failover you have never executed is not a capability. It is a hypothesis with a wiki page.
The cost of the untested hypothesis has gone up as AI has moved into the transaction path. In the 2026 edition of the same research, 98% of companies estimate one hour of AI-related downtime at $10,000 or more, and nearly two thirds put it above $100,000 per hour. When an AI feature sits in the middle of underwriting, support triage or order processing, its unavailability is no longer a degraded convenience. It is a stopped process with a queue building behind it.
Why does the work not get done? Because resilience engineering has the worst incentive profile in software. It is invisible when it succeeds, it competes against roadmap features that have named customers attached, and the person who proposes a game day is proposing to deliberately break production on a Wednesday. The 2026 survey found 63% of respondents saying their leadership teams underestimate how quickly AI demand will outpace their existing data infrastructure, which is a polite way of describing a funding conversation that has not happened yet.
The framing that works with a board is not technical. It is the one the numbers above hand you for free: what is our hourly cost of being down, how many hours of it did we buy last year without noticing, and what would a tested second path have cost by comparison? Almost every organisation that runs that calculation honestly finds the resilience work is cheaper than the outages it already paid for.
Key Takeaways
- Fewer than a third of organisations conduct any failover testing at all
- The average surveyed organisation absorbed 86 outages a year at 196 minutes each
- Nearly two thirds put an hour of AI-related downtime above $100,000
- Frame the funding ask as hours of downtime already paid for, not as insurance
What Resilience Engineering Actually Means in Practice
Strip away the vocabulary and there is a concrete, finishable list of work here. It starts with a dependency inventory that is honest about transitivity. Not the vendor list procurement keeps, but the graph: every external call in the request path, what happens to a user when it fails, whether the failure is detectable, and how long the product can run without it. Most teams discover two or three dependencies they did not know were in the critical path, typically an analytics SDK blocking first render or an identity provider that gates a health check.
Then the control plane question that Railway's incident put on the table for everyone. Where does your ability to respond live? If your deployment pipeline, your feature flags, your DNS, your secrets and your incident tooling all sit inside the provider that is currently failing, you do not have an incident response capability, you have an incident observation capability. The test is blunt: can you roll back, flip a flag and reroute traffic while the primary vendor's console is unreachable? If the answer requires a login to the thing that is down, that is the first thing to fix, and it is usually a week of work rather than a quarter.
The request-path mechanics are well-trodden and still frequently absent: timeouts on every outbound call, a total time budget per request rather than per hop, circuit breakers that fail fast instead of queueing, retries with exponential backoff and jitter, and hard caps on retry volume. That last one matters more than it used to, because a fleet of autonomous agents retrying aggressively against a degraded provider is a self-inflicted denial of service, and it will extend your own outage after the provider recovers. Idempotency keys and a replayable queue turn a class of failures from data loss into delay, which is the single highest-leverage change available to most transactional systems.
Isolation is where the money decisions live. Multi-region active-active is expensive and complicated and most products do not need it; what they do need is an explicit, written decision about which regional failures are survivable and which are accepted, with an RTO attached that someone has actually measured rather than aspired to. The measurement is the point. It is common for the recovery time a team achieves in its first real test to sit well past the recovery time its documentation promises, and it is much better to discover that in a scheduled exercise than during a fifteen-hour region event.
Finally, the schedule. A game day every quarter, with a failure injected on purpose, a runbook followed as written and the gaps recorded, converts all of the above from documentation into capability. Observability has to survive the exercise too, which means checking that your telemetry does not route through the same provider you are simulating the loss of, a constraint that sits awkwardly with the cost pressure on telemetry budgets but is not negotiable.
Degraded Mode Is a Product Decision, Not an Engineering One
The most common reason a resilience project stalls is not technical difficulty. It is that nobody with product authority has decided what the software should do when a dependency is gone, so engineers default to the only behaviour that requires no decision: propagate the error and show a spinner.
Every externally dependent feature needs an explicit answer. If the model API is unavailable, does the support assistant serve a cached answer, fall back to deterministic keyword routing, queue the request for later processing, or tell the user honestly that the feature is unavailable and hand them a human? Those are four different products with four different support costs, and picking one is a commercial decision about the user relationship, not an implementation detail an engineer should be making alone at 3 AM.
The pattern generalises past AI. If the payment provider is down, can you authorise a low-risk transaction optimistically and reconcile later? If search is degraded, can you serve a stale index rather than an empty page? If the recommendation service is gone, is a curated static list acceptable for an hour? Each answer converts a hard failure into a soft one, and soft failures are what separate a product that users trust from one they abandon during the third incident of the quarter.
This work also needs a verification story, because degraded paths rot silently. A fallback that nobody exercises will be broken by an unrelated refactor within two quarters, and you will find out at exactly the wrong moment. The same discipline StepTo has argued for around evaluating AI features properly applies here: the degraded path needs a test that runs on every build, or it is decoration.
The artefact that makes all of this tractable is small. One table, one row per external dependency, four columns: what breaks, what the user sees, what the system does, who decides. Most teams can fill it in a single workshop, and the argument it provokes is the valuable part.
In Europe, Resilience Is Now Evidence You Have to Produce
For a growing set of European companies, the tested-failover question has stopped being a matter of engineering taste. It is an audit finding waiting to happen, and the people asking are not your users.
The financial sector felt it first. We have covered how DORA turns an ICT provider into a supervised dependency, complete with a register of information, concentration risk assessment, documented exit plans and a resilience testing programme that has to be evidenced rather than asserted. The regime's logic is precisely the one this article has been describing: a financial entity's operational risk includes the operational risk of everyone it depends on, whether or not the contract says so.
The same logic flows down through supply chains under NIS2, where security and continuity obligations land contractually on suppliers of essential and important entities, as we set out in our piece on the supply chain flow-down clause. If you build software for a European bank, hospital, utility or logistics operator, your continuity arrangements are now part of their compliance posture, and the questionnaire will ask when you last tested them.
Readiness is not where it needs to be. The 1,000-executive resilience survey referenced above found 79% admitting their organisations are not fully prepared to comply with mandates such as DORA and the NIS2 directive. That is a large gap between regulatory expectation and demonstrated capability, and it is closing on a legal timetable rather than an engineering one.
There is a pragmatic upside worth naming. The artefacts a regulator wants, a dependency register, a tested recovery procedure, an exit plan, evidence of exercises, are the same artefacts that make a real incident shorter. Teams that build them for compliance reasons usually discover they have bought genuine operational capability by accident. It is one of the rare cases where the compliance work and the engineering work are the same work.
Key Takeaways
- DORA requires evidenced resilience testing, concentration risk assessment and documented exit plans
- NIS2 pushes continuity obligations contractually onto software suppliers of essential entities
- 79% of surveyed executives say they are not fully prepared for these mandates
- The compliance artefacts and the incident-shortening artefacts are the same artefacts
Who Actually Does This Work, and Why It Never Gets Staffed
Resilience work fails to happen for a reason that has nothing to do with whether engineering leaders believe in it. It is senior work, it is not urgent until it is catastrophic, and it competes for exactly the people who are already fully allocated to the roadmap. The engineer who can reason about a dependency graph, design a degraded mode and run a credible game day is the same engineer three feature teams are currently fighting over.
It is also hard to hire for on a permanent requisition. What the work needs is someone who has operated systems through real incidents, which is a background you cannot interview for with a coding exercise and cannot grow internally in a quarter. Most organisations end up in a holding pattern: the work is acknowledged, assigned to a senior person as a part-time responsibility, and deferred every sprint until an outage promotes it to the top of the list at the least convenient possible moment.
This is a good fit for the engagement model StepTo is built around. We are a senior-led nearshore partner based in Serbia, working with European and US clients in overlapping hours, and for reliability work that overlap is not a comfort feature. Dependency mapping, failover design and game days are collaborative, high-context activities that involve arguing with your architects in real time and watching dashboards together while something is deliberately broken. That is a working session across a two-hour gap and a week of asynchronous misunderstanding across a twelve-hour one, which is the practical case we have made before about the hidden cost of timezone overlap in outsourcing quotes.
The capability split is clean enough to scope separately. Dependency mapping, failover architecture, circuit breakers and degraded-path implementation are DevOps and reliability engineering work, and where the gap is a specific skill rather than a whole workstream, bringing in SRE engineers or cloud architects through staff augmentation puts the capability inside your existing team rather than beside it. Multi-region or cross-provider redesign is closer to a cloud migration engagement, and it should be scoped and estimated as one instead of hidden inside a feature sprint.
For ongoing ownership we default to a dedicated development team rather than fixed-scope delivery, because resilience is a property you maintain rather than a project you finish: dependencies change, providers change their failure modes, and last year's runbook quietly stops matching the architecture. How we run engagements is on the process page, rates are on the pricing page, and the broader case for nearshore development is set out in our Serbia overview. If you would rather start with the diagnosis than the delivery, a scoped dependency and failover review is a two to three week engagement that produces the dependency table, a measured RTO and a prioritised gap list, and it is usually the cheapest thing on the roadmap relative to what it prevents.
Buy the Second Path Before the Invoice Teaches You To
The through-line of 2026's incident record is that the failures did not come from bad code in the systems that went down. They came from an empty DNS record, an automated account suspension, a repair job without a safety flag, a datacentre fire and a model provider under scaling pressure, every one of them in somebody else's estate. You cannot engineer those away, and you cannot procure your way out of them either, because the concentration that creates the risk is the same concentration that makes the platforms good. What you can do is decide, deliberately and in advance, what your product does while they are happening, and then prove the decision works by exercising it on a date you chose. Start with the smallest version: one afternoon, one table listing every external dependency and what the user sees when it fails, and one honest answer to the question of whether you could roll back with the vendor's console unreachable. That afternoon will tell you more about your actual resilience posture than any architecture diagram in the repository, and if it reveals more work than your team has capacity to absorb, that is precisely the kind of bounded, senior engagement a dedicated nearshore team is well suited to take on alongside you.
Building a team in Eastern Europe?
StepTo helps European and US companies build senior-led nearshore engineering teams in Serbia. Let's talk about what your next engagement could look like.
Start a conversationWritten by
Igor GazivodaFounder & CEO · StepTo
Igor has 15+ years in software engineering and business development. He specializes in scaling engineering teams, nearshore strategy, and AI-driven product development. He holds a Master's in Computer Science from the University of Belgrade.
LinkedIn →