Beam Blockchain Stop Event 01/21/2019 — Post-Mortem
Today, on January 21st, 2019, Beam Blockchain experienced a temporary stop. The issue was identified by Team Beam and eventually fixed.
The root cause of the issue was that two cloned wallets (most likely created by copying the same wallet.db file) both sent the same cloned UTXO to the blockchain, which resulted in incorrect cut-through processing and ultimately to an invalid block.
The technical details of the issue and the fix are available in our Github Issue #386.
Timeline of events
- 9:13 AM (GMT): Beam Blockchain stopped producing blocks at block 25709
- 9:20 AM (GMT): Team Beam and some people in the community identified that there is an issue. Beam developers immediately started looking into it.
- 10:22 AM(GMT): After confirming the issue, Team Beam issued a tweet and notified the community that the issue is under investigation.
- 11:02 AM (GMT): Beam’s Dev Team succeeded to reproduce the issue. The root cause of the issue was identified.
- 11:42 AM (GMT): Sparkpool and possibly other pools have independently created a fix and continued creating blocks without processing the transactions (empty blocks) and mining continued.
- 12:00 PM (GMT): Beam’s Dev Team coded a fix and started testing.
- 01:59 PM (GMT): After testing, the code was committed to Github. Community and mining pools were notified.
- 02:29 PM (GMT): Sparkpool independently built Beam nodes from source and deployed to production. Transaction processing resumed (see block 25820 with 1019 outputs.)
- 03:58 PM (GMT): Team Beam released the Agile Atom 1.1.4202 Hotfix that solves the issue. The community was notified.
Aftermath
- No Beam blocks were produced from 9:13 AM until 11:42 AM GMT
- No Beam transactions were processed between 9:13 AM until 2:29 PM GMT
- No funds were lost
- By the time of this writing, blocks are processed normally and hashrate returned to similar levels as before the incident
Acknowledgments
- We would like to thank the Beam community for understanding and support
- We would like to thank Sparkpool and Beepool for cooperation and swift decision-making
- We would like to thank the following community members for having helped us to handle this situation with humor and self-depreciation.
If you find any kind of bug, issue or vulnerability, related to the one we faced today or not, please make sure to reach us as soon as possible via email: security@beam.mw or submit an issue on Github.
Thanks for your patience and your understanding. Team Beam will continue with your help to build a confidential, comprehensive and secure ecosystem.
