Tl;dr: Our recent evaluation of Kubernetes underscored its suitability for scaling Coinbase into the future. In the past, a migration to Kubernetes raised concerns due to the operational burden of running and securing the control plane in-house. We’ve now concluded that managed Kubernetes offerings reduce this operational burden without compromising our stack security.
By Clare Curtis, Coinbase Staff Software Engineer
Almost two years ago we released a blog post detailing why Kubernetes is not part of our technical stack. At the time, migrating to Kubernetes would have created a whole new set of problems that outweighed any near-term benefits. However, as these technologies have matured, our newly-formed Compute Team devised a strategy for leveraging Kubernetes in a way that can deliver a more flexible and scalable version of our current system.
Coinbase has grown substantially since we first considered migrating to Kubernetes. With any growth of this kind, it is important to prioritize scalability concerns. As we continue to scale, one of the main areas in need of future-proofing is Coinbase’s compute platform. In mid-2020, our largest service was configured to run a relatively small number of hosts, whereas today it’s running 10x that number.
In this same period, we quadrupled the size of our engineering organization causing a substantial increase in the number of deployments — each needing completely new hosts. The increase in the number of deployments have raised concerns over future scalability as we are already running into technical limitations of current APIs and resources. Recurring issues with getting enough capacity and having it delivered in a reasonable timeframe, caused an increase in failed deployments and required our largest services to dramatically slow down their release process.
While these issues are solvable, we decided to take this opportunity to evaluate whether it made sense to continue investing in a homegrown system or consider an…
Click Here to Read the Full Original Article at The Coinbase Blog – Medium…
