Understanding the Recent DDoS Attack Against Read the Docs
Recorded: Sept. 9, 2026, 5:01 p.m.
| Original | Summarized |
Understanding the Recent DDoS Attack Against Read the Docs - Read the Docs Product Pricing Blog Docs Log in Read the Docs Community Read the Docs Business Sign up Choosing a platform Product Pricing Blog Docs Log in Log in Read the Docs Community Read the Docs Business Choosing a platform Sign up Get started with Read the Docs Which site best fits your project? Read the Docs Community Clone public repositories Public documentation Ads supported hosting Free for open-source software. Sign up Read the Docs Business Clone private and public repositories Public and private documentation Team management for your organization Plans starting at $50/month. Sign up Choosing a platform
Home Understanding the Recent DDoS Attack Against Read the Docs Sep 8, 2026 Tagged ddos security infrastructure performance cloudflare terraform David Fischer Analytics from the June 2026 attack In mid-to-late June 2026, Read the Docs experienced the largest Massive volume: At peak, we received 5.5 million requests per minute, compared to our normal daily peak of under 100k requests per minute. Global distribution: We saw malicious requests originating from millions of unique IP addresses across hundreds of networks (ASNs) globally. This included residential IP blocks as well as major and minor hosting providers. Header & TLS randomization: The attackers systematically randomized HTTP request headers and TLS connection parameters to evade signature-based filters (JA3/JA4). Limitations of automated CDN defenses: Read the Docs uses Cloudflare and while Cloudflare's automated DDoS protection mitigated some traffic originating from what they called "known botnets", a big part of the attack passed that first check and got through to our rate limiting and WAF rules. Cache evasion: Attackers found and deliberately targeted URLs that resulted in cache misses, such as non-existent pages with unique paths (404s) as well as temporary redirects (302s). Adaptive behavior: When we implemented blocks or rate limits, the botnet adjusted its request rates, rotating through different target paths and spreading traffic across broader IP pools to probe our defense boundaries. Scale and breadth “There's no way we could have handled this attack without Cloudflare.” Adapting to our defenses Analytics showing oscillating "Yo-Yo" traffic levels during the attack. Attackers would ramp up to discover our rate limit thresholds and then back off to let the rate limit windows expire. A Slack notification when Read the Docs is getting 45k uncached reqs/min. Without caching and rate limiting, auto-scaling infrastructure will just scale out to handle the load at our expense. Rate limiting and fingerprinting Cipher suite and TLS anomalies: Automated scrapers and bot clients frequently present abnormal TLS connections different from browsers. Cloudflare's bot detection has specific tools to detect these. Too many bad requests: Legitimate users and good bots are almost always served successful (200) responses, not redirects or 404s. Since 200s are always cached, they almost never present a problem. When we see too many more expensive requests like redirects or 404s, we begin rate limiting the browser fingerprint, the ASN, or even possibly the specific domain as a whole. Adding these rules, which we call the "penalty box", probably made the biggest difference in automatically mitigating the attack as it changed over time. Protocol inconsistencies: Malicious tools often declare modern User-Agent strings while using older HTTP/1.1 connections. Unfortunately, this inconsistency wasn't very useful in this attack, which was entirely HTTP/2 and HTTP/3. Client fingerprinting: Everyone using the Golang HTTP client or the Python requests module with the same TLS cipher suites will have the same JA4 fingerprint. This fingerprinting is specific to a browser or tool, not specific to a user. These fingerprints also weren't very helpful in this attack as the attackers were randomizing their TLS parameters. IP block classification: One area we are still working on is to classify more IP blocks into different categories with their own limits. For a service like Read the Docs which receives lots of automated traffic and wants to allow bots, we know we're going to get a lot of traffic from major cloud ASNs like Amazon, Google Cloud, and Azure. They should have higher limits than most residential or minor hosting providers. Give users an escape hatch IP blocking is obsolete for distributed attacks: Botnets or large scrapers use proxy services which make simple IP blocks useless. We already knew that, but this incident underscored it. Defenses need to have broader rate limits across more than just IPs (ASNs, hostnames, etc.). Aggressively cache: Cache everything, whether it's a simple static file, a 404, or a temporary redirect. Even setting a short cache window of a few minutes will ensure that these resources can't be used to attack our infrastructure. The default settings on the CDN and in most web frameworks are not what a service like Read the Docs wants. Protect cache-miss surfaces: Attackers actively search for non-cacheable paths (e.g. dynamic redirects, search endpoints, and 404s). Cache where possible, and if caching isn't feasible, try to handle as much on the edge as possible. Targeted challenges beat blunt instruments: Combining bot management heuristics with rate limits allowed us to mitigate the attack with minimal impact on legitimate users. Infrastructure as Code is essential: Managing edge and WAF rules via Terraform enabled us to review, test, version-control, and roll out complex filtering rules quickly and safely. We're still seeing low levels of background traffic from the attack IP blocks, See more posts from our blog Stay updated Learn more Product Company Copyright 2025, Read the Docs, Inc & contributors |
In mid-to-late June 2026, Read the Docs experienced its most significant distributed denial-of-service attack, reaching a peak of over 5.5 million requests per minute, which was more than one hundred times the normal baseline traffic. This incident lasted nearly ten days and served to rigorously test the platform’s infrastructure, edge defenses, and incident response capabilities. The attack differed significantly from previous traffic floods because it was highly distributed, rapidly adapted to existing defenses, and specifically targeted areas that bypassed caching mechanisms. The attack demonstrated several key characteristics. It involved massive volume, originating from millions of unique IP addresses across hundreds of global Autonomous System Numbers (ASNs), including residential blocks and hosting providers. Furthermore, the attackers employed header and Transport Layer Security (TLS) randomization to effectively evade signature-based filtering methods like JA3/JA4. Attackers deliberately focused on URLs that resulted in cache misses, such as non-existent pages (404s) and temporary redirects (302s), as well as overwhelming hardcoded Nginx redirect directives with excessive traffic. This distributed nature meant that traditional rate limiting applied per Cloudflare colocation location was insufficient, as the attack spread globally, making it difficult to limit malicious traffic without impacting legitimate scrapers. Read the Docs relied heavily on Cloudflare for protection, which mitigated some traffic from known botnets, but a significant portion of the attack successfully bypassed initial checks and reached the application layer defenses. The attackers utilized an adaptive behavior, employing a yo-yo pattern where they would escalate request rates to find rate limit thresholds and then back off, maximizing the financial impact on auto-scaled infrastructure and causing intermittent service degradation. To effectively defend against volumetric DDoS attacks, a defense-in-depth strategy encompassing edge caching, web application firewalls, rate limiting, local caching, and request fingerprinting is essential. Edge caching serves as the first line of defense by serving documentation from a Content Delivery Network (CDN), which accelerates fetching for geographically distant users. However, attackers quickly discovered which requests were cached versus uncached. Therefore, aggressive caching must be applied to all responses, including temporary items like redirects and 404s, often utilizing short cache windows specified by the Cache-Control header, to prevent these surfaces from being exploited. Since relying solely on IP-based blocking proved ineffective against distributed attacks, defenses needed to shift focus from the source of the request to the characteristics of the request itself. The team implemented targeted rate limiting rules that combined bot probability scores with per-IP limits. The focus was placed on identifying expensive, non-cached requests, such as redirects or 404s, to apply penalties based on these outcomes rather than focusing solely on the origin IP or country. This approach, often referred to as the "penalty box," proved crucial in automatically mitigating the attack. Further fingerprinting techniques were introduced, focusing on request attributes rather than just IP addresses. Anomalies in cipher suites and TLS connections were monitored to detect abnormal client behavior. Additionally, the team analyzed protocol inconsistencies and client fingerprinting based on TLS parameters, noting that inconsistencies in User-Agent strings were less informative in this specific HTTP/2 and HTTP/3 environment. While client fingerprinting based on TLS cipher suites provided some insight, the attackers' ability to randomize these parameters limited their utility. The team also recognized the need to classify IP blocks more granularly, assigning higher limits to major cloud ASNs like Amazon and Google Cloud compared to residential providers. A critical operational decision involved maintaining an "escape hatch" for legitimate users. Instead of imposing outright blocks, the platform opted for a JavaScript challenge as the strictest measure. This approach balances security needs with user experience, as users who successfully complete the challenge are unlikely to be challenged again soon after. Overall, the experience underscored that IP blocking for distributed threats is obsolete; defenses must incorporate broader limits across hostnames, ASNs, and request characteristics. Successful defense required aggressively caching all response types, actively protecting cache-miss surfaces, employing targeted challenges, and managing complex filtering rules safely through Infrastructure as Code tools like Terraform. |