Ethereum Prysm client experiences mainnet incident: resource exhaustion leads to large-scale missing blocks and witnesses.

CoinLive
2025.12.14 03:40
The Prysm team released a mainnet incident recap report stating that during the Ethereum mainnet Fusaka session on December 4th, almost all Prysm beacon nodes ran out of resources while processing specific attestations, resulting in their inability to respond to validator requests in a timely manner and causing a large number of missing blocks and witnesses. The incident affected epochs 411439 to 411480, a total of 42 epochs, with 248 blocks missing out of 1344 slots, a missing rate of approximately 18.5%. Network participation dropped to 75% at one point, and validators lost approximately 382 ETH in witness rewards. The root cause was that Prysm received attestations from nodes that may have been out of sync with the mainnet. These attestations referenced the block root of the previous epoch. To verify their legitimacy, Prysm repeatedly replayed the old epoch state and performed high-cost epoch transitions, causing nodes to run out of resources under high concurrency. The defect originated from Prysm PR 15965, which had been deployed to the testnet a month prior but did not trigger the same scenario. The official temporary solution was to enable the `--disable-last-epoch-target` parameter in version v7.0.0; subsequent releases v7.0.1 and v7.1.0 included a long-term fix, using head state to verify attestations and avoid repeatedly replaying historical states. Prysm stated that the issue gradually subsided after 4:45 UTC on December 4th, with network participation recovering to over 95% by epoch 411480. The Prysm team pointed out that this incident highlights the importance of client diversity; if a single client accounts for more than one-third, it may lead to a temporary inability to terminate; exceeding two-thirds poses a risk of an invalid termination chain. They also reflected on the unclear communication regarding feature switches and the failure of the test environment to simulate a large number of asynchronous nodes, and will improve testing strategies and configuration management in the future.