How Uber Protects Against Retry Storms

Skip to main content

Uber BlogExploreProductsAdvertisingLearn more about advertising on Uber. Reach consumers as they go anywhere and get anything.EarnResources for driving and delivering with UberRideExperiences and information for people on the moveEatOrdering meals for delivery is just the beginning with Uber EatsMerchantsPutting stores within reach of a world of customersBusinessTransforming the way companies move and feed their peopleHealthMoving care forward together with medical providersHigher EducationEnhancing campus transportationTransitExpanding the reach of public transportationCompanyEngineeringThe technology behind Uber EngineeringCommunity supportDoing the right thing for cities and communities globallyNewsroomUber news and updates in your countryUber.comProduct, how-to, and policy content—and moreMoreNo resultsENHelpLog inSign upRideDrive & deliverUber EatsBusinessUber BlogLog inSign upRideDrive & deliverUber EatsBusiness,September 17, 2026How Uber Protects Against Retry StormsDMDeepanshu MehndirattaSenior Staff EngineerASAlok SrivastavaPrincipal EngineerVDVibhor DhingraSr Software Engineer1+Share this articleFacebookLinkedinX socialLinkIntroductionRetry storms historically impact business operations and brand trust. While retry configuration tuning and retry budgets provide meaningful mitigation at the service level, they’re manually configured and lack visibility into cross-service amplification caused by deep dependency chains and fan-out patterns. As a result, it can be difficult to shield infrastructure against the domino effect triggered by a single service outage deeper in the stack. A key reason is that retry behavior today isn’t context-aware. While we can control how many retries occur, we can’t precisely control when they occur. This stems from the challenge of reliably distinguishing between errors generated by a service and those merely propagated through it.As a result, retries are applied uniformly rather than conditionally.This approach works for transient or low-rate failures. However, during moderate or severe degradation, it becomes counterproductive. Aggressively retrying against an already struggling service increases load, accelerates failure, and amplifies retry traffic across upstream dependencies. What begins as a localized outage can quickly escalate into a stack-wide incident—ultimately degrading, or in the worst case, completely breaking, the end user experience.One might argue that error codes from downstream services could be translated upstream to provide context for retries. While theoretically possible, this approach doesn't scale at Uber due to large fan-in and fan-out, evolving call flows, and the need for frequent adaptive changes. Therefore, we developed a context-aware mechanism in shared infrastructure to handle errors more efficiently. This blog explains the mechanism.BackgroundConsider a simple call chain as shown in Figure 1, where the total number of requests arriving at NodeA is Ƞ. By deduction, all nodes B, C, D, E, F, and G serve Ƞ requests in the steady state (when no node errors out).Figure 1: Call-chain with 1:1 fan-out, where a node calls its downstream exactly once for any incoming request.If service D starts erroring out and each service is configured to retry once (1 regular attempt and another attempt if the downstream fails), let’s look at the total number of requests served by each node.Figure 2: Call chain where a service errors out. NodeABCDEFGDepth0123456Requests ServedȠ 2 × Ƞ 4 × Ƞ 8 × Ƞ 8 × Ƞ8 × Ƞ8 × ȠThis can be distilled down to a simple formula, assuming the number of retries R is the same at every hop. ɗ denotes the depth of the node in the call chain, the number of requests served by the node if the node creates or passes through an error:Rɗ × ȠRetry BudgetsWe can optimize this by introducing retry budgets. Let’s assume the same retry budget at every hop represented by B. The new formula becomes:(1+B)ɗ × ȠNow, let’s try to see the number of requests served with a retry budget of 10%:NodeABCDEFGDepth0123456Requests ServedȠ 1.1 × Ƞ 1.21 × Ƞ 1.33 × Ƞ 1.33 × Ƞ1.33 × Ƞ1.33 × ȠIn the above example, the error originates at NodeD, and if we limit the retry to only between NodeD and NodeC , and restrict entirely NodeA and NodeB  from retrying on this error, we can guarantee a similar availability of the call-path without overburdening NodeD, NodeE,  NodeF , and  NodeG.Error OwnershipConsider the same example of retry budgets while restricting retries between the edge from NodeC  to NodeD, where the error originates.NodeABCDEFGDepth0123456Requests ServedȠ Ƞ Ƞ 1.1 × Ƞ 1.1 × Ƞ1.1 × Ƞ1.1 × ȠHere, we clamp down the total number of requests served by all nodes from D till the leaf node G to just 10% over baseline, while allowing at least once retry for up to 10% of errors when they’re first returned. But what about the availability of NodeD as seen by NodeC? Let’s run some numbers for various availability scenarios, and try to calculate ‌availability after retry.Base Availability %Base Error Rate %Retry BudgetError Rate after Retries %Availability after Retries %99.90.110%0.000199.999999110%0.0199.9995510%0.2599.75901010%199802010%1288703010%2377As shown in the table above, for availability drops up to 10% in the callee node, even a single retry is helpful in getting the perceived availability by the caller node up to 99%. Beyond this, perceived availability drops significantly as a good chunk of requests are never retried due to the retry budget in place.This calculation assumes the errors from the callee are independent and that retries will lead to recovery. However, in many real-world scenarios like service overload, bad database hosts, database overload, or sharding issues, the probability of retries remains high even with retries. This contradicts the idea that retries to callee always increase perceived availability to the caller. It’s also this intuition that forms the basis of error ownership. During periods of high error rates from a service, the errors are less likely to be randomized, and wouldn’t benefit from a higher number of retries, and instead might be responsible for further degradation.ArchitectureThe solution is about establishing error ownership, which can be explained using the symptom versus cause analogy.If a service calls N outbounds for fulfilling a request, and if an outbound error-out causes it to return an error, then the error returned by that service is only a symptom. Simultaneously, in the context of the service, the cause is the incoming error from its downstream.However, if no outbound of the service errors out while fulfilling the request and it still returns an error, the service is the cause of the returned error, and is the owner. In the next section, we discuss some possible solutions that can leverage this.Simple CorrelationClaiming Error OwnershipWe use the Service Dependency Analysis Solution to correlate an inbound failure with an outbound failure and use the ruleset shown in Figure 3 for making or refuting error claims.Figure 3: Decision logic for claiming error ownership. Retrying with Error OwnershipThe caller uses the logic shown in Figure 4 to determine if it should retry the request. Figure 4: Decision logic for allowing retries.While the scenario of missing error claim headers is an uncooperative environment, it could occur because the downstream service doesn’t have the service dependency analysis solution, and is unable to correlate outbound and inbound errors. Or, the downstream service has missing context propagation, resulting in an incomplete correlation between outbound and inbound errors.Here, the first node to see a missing error claim from a downstream unclaims the error, limiting the impact radius of the retry disturbance (it’s no longer a storm), while still allowing sufficient retries to the error-returning service.Decision MatrixCallee ErrorCaller ErrorCallee Error ClaimCaller Should Retry (Retry Middleware)Caller Propagated Error Claim NoYesNANAClaimYesYesMissingYesUnclaimYesYesClaimedYesUnclaimYesYesUnclaimedNoUnclaimFigure 5: 3 nodes used to demonstrate caller errors. When the edges A -> B and B -> C are fail-close, and the Node C returns an internal error that it’s claimed, Node B upon seeing the claimed error from C should retry to C. However, if the retry fails, it’d propagate the error to Node A, but while returning the error it must unclaim it. Node A upon seeing the error from B and the unclaimed error header shouldn’t retry the request to Node B.Figure 6: Decision logic for Error claim propagation.Coincidental Errors and Why We Need Service Dependency AnalysisThe decision matrix above covers cases where the downstream call fails or there’s an internal server error. There could also be scenarios where both happen simultaneously, as shown in Figure 7.Figure 7: Example where Node A has 2 fail-open dependencies, Node B and Node C.In this example, Node A is also experiencing a 10% error rate because of an overloaded cache/database that isn’t tracked via the service dependency analysis solution. This makes it an internal server error of Node A, and Nodes B and C also have a 10% error rate for unrelated reasons.The service dependency analysis solution tries to attribute errors to downstreams first and itself last, so even when Nodes B and C are fail-open, it assigns the blame to them when the failures are colocated. This unclaims the error and prevents the upstream of Node A from retrying to it and possibly recovering.The impact of missed legitimate retry opportunities in such a scenario is significant. If Node A serves 100 requests, out of which 10 experience internal server errors originating at Node A, 1.9 of those 10 requests would be incorrectly unclaimed by Node A as an error not originating from itself, resulting in a missed legitimate retry opportunity.Based on the analysis of the 6 months of service dependency analysis solution data, coincidental errors like a legitimate server error or a fail-close dependency error happening during a fail-open dependency error are extremely rare. In the worst case scenario, for 80% of edges with over 100 callee failures in a minute, around 2% of the times the caller failed too (a coincidental failure). Using the example above, we’d get 0.396 out of 10 requests that’d be incorrectly unclaimed by Node A.However, even if we fixate on the worst case, since the service dependency analysis solution creates a memory of failure patterns, it can leverage this memory to only unclaim errors when inbound failures correlate with outbound failures in fail-close dependencies. This eliminates the risk of retry suppression during coincidental errors.Figure 8: Decision logic for determining when to retry.Edge Case ScenariosGuaranteeing At-Least-Once RetriesMany services don’t have retries configured for their fail-close outbounds. Eliminating retries from callers of these services would lead to availability drops along the incoming caller chain. This is solved by introducing a flag to signal whether retry criteria is satisfied for an error returned by a downstream. When the retry middleware sees an error from the downstream, it can compute whether the retry criteria is satisfied and pass that along to the service dependency analysis solution. Figure 9: Retry logic to guarantee at-least-once retries.The Retry Middleware marks retry criteria satisfied if any of the following conditions shown in Figure 10 are met.Figure 10: Decision logic to determine whether the retry criteria is satisfied. The service dependency analysis solution leverages this retry criteria satisfied signal from the retry middleware to decide at the inbound whether it needs to own the error returned by the fail-close downstream.Figure 11: Decision logic for service error ownership.Here’s the end-to-end flow:Figure 12: End-to-end flow.We don’t introduce any new retries along the call chain. The retry-at-least-once behavior only works if at least one service along the call chain has retries configured.By introducing an at-least-once-retry guarantee, we eliminated availability drops in our call chains. At the same time, we safeguarded them against retry storms by leveraging error ownership.Context Drop HandlingFigure 13: 4 nodes used to demonstrate context drop handling.For the service dependency analysis solution and error ownership, intra-service context drops shift the retryable error left. Consider the example in Figure 14, where Node D returns an internal error to Node C. The intra-service context in Node C is broken, so the service dependency analysis solution can’t correlate the outbound request from Node C to D with the incoming request from Node B to C.As a result, Node C claims the error, while decoupled retries at the retry middleware continue to happen between Node C and D. A set of retries also happens between Node B and C, as Node C must claim the error it’s returning. However, when the retries fail and Node B returns an error to Node A, it also returns the unclaimed error header, which should prevent Node A from retrying the request to Node B. In this case, we cut down the total number of requests to Nodes B through D by half, assuming a single retry (1 attempt and 1 retry) configuration at all nodes. If there were 5 nodes to the left of Node A, the worst case would’ve had 32 times more requests without error ownership propagation.Figure 14: Description.Error Ownership in ProductionEverything described so far isn’t theoretical—error ownership is fully implemented and operational across Uber’s service mesh today. The scheme runs in the retry middleware and the Service Dependency Analysis Solution that sit in the request path of our user-facing APIs, continuously claiming and unclaiming errors as traffic flows through deep dependency chains. Because it’s embedded in shared infrastructure, services inherit retry-storm protection without bespoke per-service error-handling logic. The following real-world incident demonstrates how this production deployment behaved under a genuine large-scale degradation.Use Cases at UberOn November 18th, 2025, Uber had a major outage due to an issue with a Core Entity service. The service lives over 5 levels deep in our call chain and is critical for business operations. It started returning a very high error rate due to an underlying infrastructure issue. This error was quickly propagated up the call chain. During this time, many upstream caller services had enough opportunity to retry the failed requests returned by these services. With simple retry budgets, this would’ve resulted in a 46%-135% traffic increase on the degraded service, prolonging the outage by diminishing chances of recovery. Because error ownership was already enabled in production, as described above, the system contained the blast radius automaticallyImmediate retry attempts were stopped to the immediate callers of the degraded service, where some callers were stopped from making up to 200,000 additional requests. We also calculated the retries that were stopped at the ancestors of these immediate callers, and aggregated them at the root node. We learned that we could stop a staggering 9.5 million spurious requests in our service mesh. Those could’ve easily prolonged the outage by constantly hammering the degraded service.ConclusionOur approach has dramatically reduced the total request volume flowing through the call graph during degradation events.To quantify this, we define the max retry storm radius after error ownership as the max depth of call path where a retry storm could happen after error ownership was enabled. It’s computed for the call graph of every root node. Across all our user-facing APIs, we got this value down to a maximum of 3, where the earlier value of max retry storm radius was up to 25. We also got the average down to 2 from 20.Figure 15: Max retry storm radius after error ownership.By limiting retries exclusively to error-owning services, where they can genuinely resolve the issue, we can prevent the exponential fan-out of requests characteristic of retry storms. This has helped safeguard Uber’s infrastructure from cascading failures triggered by single-point degradations. AcknowledgmentsCover Photo Attribution: Generated with ChatGPT by OpenAI; no external images, logos, or third-party assets used.Stay up to date with the latest from Uber Engineering—follow us on LinkedIn for our newest blog posts and insights.Written byDMDeepanshu MehndirattaSenior Staff EngineerDeepanshu Mehndiratta is a Senior Staff Engineer in Uber's Business Platform org, where he leads Reliability and AI Engineering. His AI work spans the MCP Gateway and Uber's frontier deep-agent ecosystem, connecting all Uber services to AI agents and leveraged by tens of thousands of employees.ASAlok SrivastavaPrincipal EngineerAlok Srivastava is a Principal Engineer on Uber's Business Platform team. He leads Uber's Edge Platform, the ingress and egress tier for Uber's business traffic, spanning APIs, content, and push messaging across all mobile and web surfaces.VDVibhor DhingraSr Software EngineerVibhor Dhingra is a Senior Software Engineer at Uber. Previously, as part of Uber's Business Platform team, he owned Uber’s core caching proxy microservice, redesigning it to serve over 4M QPS from 1.2M QPS in 2 years.ASAnkit SrivastavaDistinguished EngineerAnkit Srivastava is a Distinguished Engineer at Uber, where he works on the development of core business platforms that scale to millions of people who use Uber across the world.CategoryEngineeringBackendRelated Articles6 articlesBackendEngineeringLarge-Scale Automated Dependency Analysis Across Uber's Service MeshSeptember 15, 2026BackendEngineeringFrom Signals to Context: Lessons from Lavaredo Ultra TrailSeptember 11, 2026BackendEngineeringWebHalving the Time: How Uber Eats Rebuilt Its Search PipelineSeptember 10, 2026BackendEngineeringFrom One Horizontal Scaling Controller to Many: Evolving Uber's Compute PlatformSeptember 9, 2026BackendEngineeringFrom Chaos to Control: Addressing Shard Distribution Challenges in M3DB with SubclustersSeptember 1, 2026AI / MLEngineeringRunning a Software Factory Efficiently at Uber ScaleAugust 27, 2026UberVisit Help CenterCompanyAbout usOur offeringsNewsroomInvestorsBlogCareersProductsRideDriveDeliverEatUber for BusinessUber FreightGift cardsUber HealthUber AdvertisingMerchantsGlobal citizenshipSafetySustainabilityTravelReserveAirportsCitieslinkedinyoutubeinstagramtwitterGlobeEnglish© 2026 Uber Technologies Inc.PrivacyAccessibilityTermsSelect your preferred languageEnglish, EnglishChinese, 简体中文Spanish, Español (Internacional)ProductsAdvertisingLearn more about advertising on Uber. Reach consumers as they go anywhere and get anything.EarnResources for driving and delivering with UberRideExperiences and information for people on the moveEatOrdering meals for delivery is just the beginning with Uber EatsMerchantsPutting stores within reach of a world of customersBusinessTransforming the way companies move and feed their peopleFreightTaking shipping logistics in a new directionHealthMoving care forward together with medical providersHigher EducationEnhancing campus transportationTransitExpanding the reach of public transportationCompanyEngineeringThe technology behind Uber EngineeringCommunity supportDoing the right thing for cities and communities globallyNewsroomUber news and updates in your countryUber.comProduct, how-to, and policy content—and moreHelpENSelect your preferred languageEnglish, EnglishChinese, 简体中文Spanish, Español (Internacional)RideDrive & deliverUber EatsBusinessDrive & deliverRideUber EatsUber for BusinessManage accountSign out

Retry storms historically pose significant risks to business operations and brand trust, stemming from the fact that traditional retry configuration and budgets operate at the service level and lack visibility into the compounding effect caused by deep dependency chains and fan-out patterns across the infrastructure. This limitation arises because current retry behavior is not context-aware; it applies retries uniformly rather than conditionally based on the true source of the error, which is difficult to distinguish from errors merely propagated through subsequent services. When services aggressively retry against an already struggling dependency, it increases load, accelerates failure, and amplifies the retry traffic across upstream services, potentially escalating a localized outage into a system-wide incident that degrades the end-user experience.

The authors developed a context-aware mechanism within shared infrastructure to address this by introducing retry budgets and establishing a concept of error ownership. Retrying uniformly is counterproductive during moderate or severe degradation because it fails to account for the cascading effect of retries across the call chain. The initial analysis, based on a simple model where $R$ is the number of retries and $d$ is the depth in the call chain, suggested that the number of requests served by a node increases exponentially with depth, modified by the retry budget $B$, as $(1+B)^d \times Q$. Optimizing this requires controlling where retries occur, specifically by limiting retries to error-owning services, which prevents the exponential fan-out characteristic of retry storms.

Error ownership is the core architectural solution, conceptually based on the symptom versus cause analogy. A service can either be the cause of an error or merely a symptom of a downstream failure. The system leverages a Service Dependency Analysis Solution to correlate inbound failure signals with outbound failures to determine which service owns the error. This involves a decision logic that dictates whether a caller should retry based on error claims, factoring in whether the error originated from the callee service or was merely propagated. This mechanism allows the system to selectively manage retry behavior, ensuring that retries only occur where genuine recovery is possible, thus safeguarding the integrity of the call path.

To handle complex scenarios, such as coincidental errors where multiple downstream services fail independently or context drops within a single service, the system must accurately attribute the error. The service dependency analysis solution attributes errors to downstreams first, preventing upstream nodes from making unnecessary retries based on unrelated failures. This involves a flow where a retry middleware signals whether retry criteria are met to the service dependency analysis solution, which then informs the inbound service whether it needs to claim the error returned by a fail-close dependency. This process guarantees at-least-once retries where necessary while simultaneously suppressing spurious retries characteristic of a storm.

Furthermore, the solution addresses intra-service context drops, where information about the request context is lost within a service, which can prevent accurate correlation between outbound and inbound errors. By allowing the service to claim the error in such situations, the system decouples the retries at the retry middleware from the error ownership propagation, leading to a reduction in the total volume of requests flowing through the call graph. This approach has been fully operationalized across the service mesh, embedding retry-storm protection directly into shared infrastructure rather than relying on bespoke per-service logic. In a real-world incident, this system successfully contained a degradation event, stopping millions of spurious requests and drastically reducing the maximum retry storm radius in the call graph from values up to twenty-five down to a maximum of three, demonstrating the effectiveness of leveraging error ownership to prevent cascading failures.