PyPI Blog: Incident File Hosting Errors

Recorded: Sept. 8, 2026, 3:08 p.m.

Original

Summarized

Incident Report: File Hosting Errors - The Python Package Index Blog

The Python Package Index Blog

Incident Report: File Hosting Errors

Initializing search

GitHub

The Python Package Index Blog

GitHub

The PyPI Blog

Archive

2026

2025

2024

2023

Authors

Mike Fiedler

Jacob Coffee

William Woodruff

Seth Larson

Nicole Harris

Maria Ashna

Dustin Ingram

Ee Durbin

Facundo Tuesca

Deb Nicholson

Shamika Monahan

Donald Stufft

Table of contents

Executive summary

Background: How PyPI's file hosting cache works

Timeline

August 15

August 17

August 18

August 19

August 20

August 21

August 24

August 28

Contributing factors

POP goes the canary

Bugs at home

5xx volume over time

Going forward

Thanks

Back to index

Mike Fiedler

PyPI Admin, Safety & Security Engineer (PSF)

Metadata

September 8, 2026

8 min read

infrastructure
transparency

Incident Report: File Hosting Errors
Executive summary
For about two weeks in August 2026,
some PyPI users hit intermittent 502 and 503 errors
downloading files from files.pythonhosted.org,
triggering failures during installation from PyPI.
Thanks to our users filing reports in the support tracker;
one report in particular narrowed the problem to a single cache node.
Two separate problems were uncovered.
A canary deployment inside Fastly's network
had triggered a misconfiguration at one cache node,
causing Fastly's routing layer to return 502 responses
for traffic reaching the affected cache node.
Separately, I found and fixed several bugs in our own Fastly configuration
around origin fallback and range request behavior.
Those had been there for a while,
and only surfaced while digging into these reports.
Both intermittent problems are now fixed,
and downloads are back to full function since August 28.

Background: How PyPI's file hosting cache works 1
Most people run pip install (or your installer of choice) and it just works:
a request goes out, the desired file comes back.
Behind the scenes, files.pythonhosted.org is a Fastly CDN service
in front of three origins (aka 'backends').
When a file is uploaded, PyPI writes it to Amazon S3 first
as the main, durable copy.
A background job syncs it to Backblaze B2, because Fastly and Backblaze have a
zero-cost egress agreement:
serving files out of B2 through Fastly doesn't cost anything beyond storage fees.
On the reader (installer) path, Fastly tries B2 first.
If B2 doesn't answer, or answers with something we don't expect,
Fastly falls back to S3.
PyPI uses the Amazon S3 Glacier Instant Retrieval storage class
to balance storage and retrieval costs.
When Fastly calls S3 as a fallback, this costs more than from B2,
but will continue to work for consumers as a stop-gap.
Once cached, Fastly no longer has to check B2 or S3 - the file should never change,
and the Cache-Control header
sets max-age=365000000, immutable, public - about 11.5 years.
A third backend named Conveyor handles everything that isn't a package file request,
like predictable URLs,
plus a handful of legacy redirects.
flowchart TD
Client([Client request]) --> Edge{Fastly edge}
Edge -->|package file| B2[(B2: egress-free cache)]
Edge -->|everything else| Conveyor[Conveyor]
B2 -->|200 or 206| Response([Response to client])
B2 -.->|404, timeout, or 5xx| Archive[(S3: origin, fallback)]
Archive --> Response
Conveyor --> Response
That fallback path only works if the edge notices B2 has failed.
One of the bugs I fixed was that it didn't always notice correctly,
which added to the "normal" error noise.
Timeline
August 15
Fastly's report places the start of the problem on this Saturday,
at a single Seattle-area cache node.
This is corroborated by #11876.
August 17
The first two reports of persistent 502s from files.pythonhosted.org are opened,
#11895/#11897.
August 18
#11908
adds a detailed reproduction, showing
88 recorded 502s over six hours,
across 32 unrelated packages,
including the small .whl.metadata range requests installers use to read PEP 658 metadata.
With three open network reports,
I go digging in Datadog Logs for the corresponding PyPI Files errors
and find nothing of consequence.
infra#237 merges,
fixing the B2-to-archive failover
for the case where B2 doesn't respond at all,
rather than responding with an error.
August 19
#11925
isolates the problem to one Fastly cache node, cache-pae2080020:
502 for every request routed to it, for over 19 hours,
confirmed by x-served-by headers on the failing responses.
Fastly's network operations removes a routing override
sending a slice of our traffic to the affected point of presence.
infra#238 merges,
improving the logging configuration for the file hosting service.
infra#239 merges,
rejecting HTTP methods that have no business hitting a file host.
August 20
Fastly observes recovery at the affected point of presence,
and later confirms the elevated error rate has stopped.
August 21
infra#241 merges,
exempting suffix and multi-range requests from segmented caching,
after a morning spike traced to a single client sending logically invalid ranges.
August 24
infra#243 merges,
after two broken parallel downloaders generated 41,315 more of the same class of error in a single day.
August 28
Fastly patches the underlying canary configuration bug on their side,
and excludes all PSF traffic, including PyPI, from their canary cohort.
infra#245 merges,
fixing a URL-normalization ordering bug
that let a bad segmented-caching response get cached
and served to every subsequent request for the same file.
Contributing factors
POP goes the canary 2
Fastly runs a canary cohort,
a subset of their fleet running caching and routing software ahead of a full rollout.
PyPI's traffic had been part of that cohort for a number of years,
helping Fastly engineering validate changes.
A partial rollback during a canary deployment
left the caching configuration on one Seattle point of presence (POP) reverted
while the routing configuration in front of it was not.
The mismatch caused that routing layer to return 502s.
Fastly has since removed all PSF traffic, including PyPI, from the canary program.
We'd like to get back to participating eventually,
once clearer controls and notifications exist around this traffic.
It's a reasonable way to help Fastly validate infrastructure changes before they hit everyone,
and it hasn't cost us much before now.
Bugs at home
While that was going on,
I found unrelated bugs in our own configuration that produced the same symptom:
elevated 502s, and in a few cases 501s that looked like 502s from outside.
The archive fallback in the diagram above
only ran when B2 answered with an error status.
If B2 didn't answer at all (a timeout, a refused connection, a TLS failure),
Fastly synthesized its own 503
and skipped straight past the code that would have tried the archive.
Package files are immutable,
so there was never a "try the archive" path for that case
until infra#237 added one.
Separately, we use segmented caching
to avoid pulling a full gigabyte-sized wheel into cache
only to serve a Range request for partial content.
That feature has narrower support for range syntax than HTTP does in general:
it can't answer a suffix range (bytes=-1024, read the last N bytes)
or a request with the start of the range past the end,
and returns a synthetic 501 for either.
Some installers use exactly this kind of suffix range
to read wheel metadata without downloading the whole file,
so those reads were failing outright until I started exempting them:
flowchart LR
subgraph Before["Before infra#241 and infra#243"]
direction TB
C1([Client: suffix or inverted range]) --> E1{Fastly edge}
E1 --> S1[Segmented caching]
S1 --> R1([501 Not Implemented])
end
subgraph After["After"]
direction TB
C2([Client: suffix or inverted range]) --> E2{Fastly edge}
E2 -->|range shape not supported| N2[Segmented caching skipped]
N2 --> R2([Normal range handling: 206 or 416])
end
Before ~~~ After
A 501 for a malformed client request is the wrong status class.
I've opened a support ticket with Fastly about the segmented-caching range handling.
Once fixed, my handling code can probably be reverted.
Another bug: an existing segmented caching exemption check
ran before the request URL was normalized,
so a .metadata request with a query string still on it didn't match,
kept segmented caching enabled,
and got a 501 back from the archive backend for what should have been a normal fetch:
flowchart LR
subgraph Before["Before infra#245"]
direction TB
C3([Client: <code>GET name.whl.metadata<b>?token=x</b></code>]) --> E3{Fastly edge}
E3 -->|exemption checked before URL is normalized| S3[Segmented caching stays on]
S3 --> B23[(B2)]
B23 -->|501 for the 1MiB segment, cached at the edge| R3([Every later request: 501])
end
subgraph After2["After"]
direction TB
C4([Client: <code>GET name.whl.metadata<b>?token=x</b></code>]) --> E4{Fastly edge}
E4 -->|URL normalized first| X4[Exemption matches: segmented caching off]
X4 --> B24[(B2)]
B24 --> R4([Normal response, cached correctly])
end
Before ~~~ After2
infra#245
moved the exemption check after URL normalization to close that hole.
None of these were new bugs.
They'd been in the configuration already,
and it took real traffic on large files with range requests
to trigger investigation and resolution.
5xx volume over time
Fastly's own real-time analytics for the file hosting service
show B2 errors (blue) climbing from August 15 onward
while the S3 archive backend (pink) stays flat at zero,
because the fallback that should have been routing failures there wasn't firing yet:

Our own Datadog metric shows a fuller story,
from before the incident to past the end of it:

Note the log scale.
The baseline was already noisy before any of this started,
thousands to tens of thousands of 5xx responses,
which is why the increase starting August 15 is easy to miss.
The August 21 spike is the one that isn't:
a single client sending logically invalid ranges,
pushing the count to close to a million.
The cliff immediately after it is
infra#241.
Traffic past that point sits two to three orders of magnitude below the pre-incident baseline,
tens to hundreds of errors rather than thousands.
Some of what we'd been treating as background noise was this bug running the whole time.
Going forward
The Python Software Foundation is hiring an infrastructure engineer
to add to the engineering staff of four.
Part of their remit will be PyPI,
which should help us catch conditions like this earlier and prevent rather than react.
The continued financial support from our community - individuals and companies alike - makes that possible.
A lot of the traffic hitting files.pythonhosted.org
is CI jobs installing the same dependencies they installed last run,
and a large share of that can be traced back to GitHub Actions runs.
If you're not already caching those downloads, it's worth turning on:
setup-python's
pip, pipenv, and poetry caching
is opt-in via the cache input, off by default,
and setup-uv's
caching
defaults to on for most GitHub-hosted runner events, but is worth checking.
An unchanged dependency that's cached doesn't touch us on the next run at all,
which means fewer requests for us to serve,
and one less thing that can break your build if we or Fastly have a rough day.
Thanks
Thanks to everyone who took the time to file an issue and capture the details
instead of just working around the problem.
If you run into file-hosting issues in the future,
pypi/support is still the right place,
and the more detail you can include (especially x-served-by headers and timestamps),
the faster we can act on it.
This work would not be possible without generous donations, please consider
supporting the PSF to keep this kind of
infrastructure running.
Thanks to Alpha-Omega, which sponsors my role.

or, how it's supposed to work ↩

while not perfectly accurate, I couldn't resist. ↩

Metadata requests no longer tracked in PyPI download counts

Made with

Material for MkDocs

For approximately two weeks in August 2026, users experienced intermittent 502 and 503 errors when downloading files from files.pythonhosted.org, which consequently caused installation failures from PyPI. The investigation, spurred by user reports, eventually narrowed the issue down to a single cache node. This incident involved two main areas of failure: a misconfiguration within Fastly’s canary deployment and several bugs in the software's own configuration related to origin fallback and range request handling. Following these findings, both intermittent problems were resolved, and download functionality returned to normal by August 28.

The background of the system involves how PyPI handles file hosting via a caching mechanism relying on Fastly as a Content Delivery Network (CDN) in front of multiple backends. When a file is uploaded, PyPI stores the main copy in Amazon S3 Glacier Instant Retrieval and syncs it to Backblaze B2 for zero-cost egress. The reader path prioritizes B2; if B2 fails or provides unexpected results, Fastly falls back to S3. This system is further managed by a Conveyor backend for non-package file requests. A critical area of complexity involves segmented caching, which aims to reduce load by serving only partial content for range requests, but this feature had limitations regarding suffix ranges and URL normalization checks.

The timeline of the incident began on August 15th when Fastly reported an issue on a single Seattle-area cache node, which was corroborated by user reports. Subsequent reports indicated persistent 502 errors starting August 17th. Detailed reproduction demonstrated that the failure included requests for small .whl.metadata range requests used by installers, leading to over eighty recorded 502s across unrelated packages. Engineers investigated Datadog logs and found no immediate evidence of a widespread issue. Progress was made when engineers fixed the B2-to-archive failover mechanism to correctly handle complete failures rather than just error responses, which was crucial for the fallback path. Isolation efforts pinpointed a specific Fastly cache node exhibiting 502 errors for over nineteen hours, confirming where the routing layer malfunctioned.

The contributing factors were multi-faceted. First, the issue stemmed from a canary deployment within Fastly’s network that caused a mismatch between caching configuration and routing configuration on one point of presence, leading to 502 responses. Second, several bugs existed in the system’s own implementation. One bug related to the archive fallback mechanism only triggering when B2 returned an error status, meaning it failed to handle total connection failures gracefully. Additionally, limitations in segmented caching, specifically concerning how suffix ranges and URL normalization were processed before checking cache exemptions, caused issues where requests for metadata files resulted in incorrect 501 responses instead of correct cached content. An existing exemption check running before URL normalization led to subsequent requests incorrectly failing against the archive backend.

Real-time monitoring showed B2 errors increasing from August 15th, while the S3 archive remained stable, indicating that the expected failure routing was not properly engaging the fallback mechanism at that time. The largest spike in errors occurred on August 21st due to a single client sending logically invalid range requests, pushing error counts significantly higher before stabilizing after related fixes were implemented.

Moving forward, steps were taken to improve infrastructure transparency and resilience. The Python Software Foundation is expanding its engineering staff to include infrastructure expertise for PyPI, aiming to enhance proactive detection of such conditions. Furthermore, the report advises users to enable caching mechanisms in tools like pip, pipenv, and poetry, which are opt-in by default, as cached dependencies reduce load on the hosting infrastructure. The authors emphasize the importance of users providing detailed reports, including x-served-by headers and timestamps, to expedite future incident resolution.