Proton had enough redundant capacity to survive the loss of its Frankfurt datacenter. Frankfurt refused to fail cleanly.
A cooling system collapse on August 26 drove room temperature from roughly 21.8°C to 51.9°C in under half an hour, with some probes reaching 60°C. Servers and network equipment died one by one. Around midnight, both switches serving a critical rack failed. That rack held several primary database copies. The system had entered a state between normal operation and total site loss, where Proton’s fast whole-site contingency did not fit the shape of the damage.
The outage lasted roughly two hours. Most services returned by 01:30 CEST, with push notifications and payments following around 02:00. Proton says no user content or email was lost, although mail delivery was delayed. The more useful fact sits inside the recovery: the company had automated failover for many components and a human gate around primary databases, then watched a physical failure attack those assumptions in random order.
Heat created a partial-death state
The incident began after the datacenter operator replaced air filters on both redundant air compressors powering the cooling system. Proton says the work happened overnight without prior notice, and the operator failed to report the cooling failure when it occurred. By the time Proton’s service alarms exposed the damage, the room was already consuming its own redundancy.
The thermal clock was savage. Proton says denser, higher-power CPUs and GPUs have compressed the time from cooling loss to critical conditions from three or four hours to about twenty minutes. During the event, some network cards reached 105°C, compared with a normal operating temperature near 45°C. Their protection logic disabled them until a cold system reset.
That behavior matters. A card can remain physically present while becoming operationally absent. A rack can retain some live machines while losing both network paths. A site can contain valid database replicas while operators no longer trust its next few minutes of thermal life. Infrastructure diagrams usually present these states as boxes with clean green or red borders. Frankfurt produced brownout geometry.
Whole-site failover was the easier case
Proton’s published reliability material says its infrastructure can survive an entire datacenter loss. Traffic rerouting is automated for ordinary server failures, data is replicated across sites, and multiple backups protect customer content. The August incident did preserve that final property. Availability failed while data integrity held.
The awkward middle state exposed a different contract. Proton does not automatically promote primary database replicas without human supervision. That restraint protects against split brain, where an unreachable primary remains alive and accepts writes while another replica is promoted elsewhere. Two divergent histories can preserve uptime briefly and corrupt truth permanently.
Human supervision therefore carried real value. Operators had to establish which primary copies were dead, which replicas were current, whether Frankfurt was becoming safe again, and whether moving primaries to Zurich would create a larger traffic shift. Once cooling returned, they chose surviving Frankfurt replicas where possible and Zurich replicas where necessary. That reduced immediate topology churn, but the assumptions underneath the decision were wrong. Frankfurt’s overheated network cards still required cold resets before the site could recover normally.
The distinction kills a comforting redundancy myth. Extra copies answer the question of whether data and capacity exist elsewhere. Recovery also needs authority to promote those copies, trustworthy failure detection, reachable control channels, enough network capacity for the new topology, and a safe destination. Every one of those conditions can be uncertain during progressive physical failure.
Security tightened the recovery path
The same controls that protect administrative infrastructure slowed part of the restart. Proton limits access to out-of-band controllers, the management interfaces used when an operating system or primary network path is unavailable. When overheated network cards demanded cold resets, additional staff had to be awakened to access those controllers.
That was a defensible security boundary. Broad, always-available management access would turn hardware recovery into a delicious attack surface. The incident still shows the operational cost of restricting it. Emergency access has to survive the failure domain, remain auditable, and reach enough authorized humans inside the thermal deadline.
Hardware scarcity sharpened the choice. Proton says the team initially prioritized restoring cooling and powering down servers because replacement equipment is difficult to obtain amid AI-driven demand. This makes the outage part of a larger infrastructure economy. A server has a live service value, a data-custody role, a replacement lead time, and a thermal survival window. Incident response now includes deciding which of those values can be sacrificed first.
The operator saved almost all of the infrastructure, though some servers suffered what Proton called “heat death.” The company also warned that surviving equipment may have lost useful lifespan. Recovery metrics rarely price that invisible damage. A service can return to green while capacitors, network cards, power supplies, and drives carry the history of an hour spent far outside their intended environment.
Redundancy needs state coverage
The postmortem identifies the partial-failure mode as known. Proton had already scheduled database resilience work for completion by the end of 2026, but had ranked the scenario as extremely unlikely. The remaining work is now under review for acceleration, while additional datacenter capacity is due within weeks.
Adding capacity will reduce dependence on one site. It will not automatically solve the state-machine problem. The harder work is enumerating transitions between healthy, degraded, progressively dying, isolated, and fully lost. Each state needs promotion rules, health signals, operator authority, management access, traffic budgets, rollback paths, and tests that prove the transition works while components fail out of order.
The cleanest improvement may look ordinary beside a new building. Rack placement should keep database primaries, replicas, and their network paths from sharing one thermal or switching fate. Site health needs a threshold that can force deliberate evacuation before random hardware loss scrambles the topology. Database tooling needs a supervised failover path fast enough for a twenty-minute physical clock. Out-of-band access needs break-glass custody that preserves security without depending on one sleeping person.
Proton’s candid report deserves credit because it names the bad tradeoffs instead of hiding behind “redundant cooling failed.” The company protected user data, recovered the service, and published exact temperatures, equipment behavior, operator choices, and unfinished resilience work. That transparency also makes the gap impossible to ignore.
The system survived the disaster it was designed to recognize. Service failed inside the disaster it could not classify quickly enough.