Tl;dr: This blog shares insights on how Coinbase is investing in new tools and processes to scale its node operations.
By Min Choi, Senior Engineering Manager — Crypto Reliability
Blockchain nodes power almost every user experience at Coinbase. We use them to monitor fund movements, help our customers earn their staking rewards, and build the analytics needed to support popular features within our applications. As such, being able to effectively manage blockchain nodes is vital to our core business and we are continuing to invest in ways to scale our node operations.
One of the most difficult aspects of node management is keeping up with the constant, and sometimes unpredictable, changes to the node software. Asset developers are consistently releasing new code versions and some blockchains, such as Tezos, leverage an on-chain governance model to take a community vote on all proposed changes. A decentralized governance model such as this makes it difficult to predict when a change will be introduced and prepare our internal systems in advance. An example of such a scenario is depicted in the below Messari alert.
Data provided by https://messari.io/
The consequences of not keeping up with these changes can be severe to our customers. They could cause long delays to balance updates in our core wallets or slashed staking rewards. To help minimize these incidents from occurring, we’re focusing investments into the following areas:
This service gives us an extra pair of hands (or should I say “ARM”) to process common node upgrades. All puns aside, the ARM service monitors Github release activity for dozens of critical blockchains and automates the deployment of new node binaries to our non-production environments. This frees up our engineers to focus on service validations and work proactively with asset developers to resolve problems prior to production release.
The below diagram shows the high level data flow for ARM.
Here’s a recent example of how the ARM service was leveraged to process a node upgrade for Algorand.
- On May 9 at 12:44 PM PDT, Algorand version 3.6.2 was released.
- On May 9 at 1:13 PM PDT, the ARM service filed a ticket to notify our engineers and track the incoming change.
- On May 9 at 1:43 PM PDT, the required code change was automatically generated for build and deployment.
- On May 9 at 2:13 PM PDT, the change was automatically deployed to all our non-production environments for Algorand.
- On May 9 at 2:43 PM PDT, an error in one of the three…
Click Here to Read the Full Original Article at The Coinbase Blog – Medium…
