medium.com/@magda7817/why-has-figma-reinveted-the-wheel-with-postgresql-3a1cb2e9297c
Why Has Figma Reinvented the Wheel With Postgres?
Its really a cautionary tale of building your software on proprietary technology and services.
The powers that be hired a big wig consultant to come in and make us a more enticing acquisition target now that we're cash flow positive. One of the first things he did was explode about how tightly coupled we had become to AWS and install a new devops czar who immediately started migrating us into kubernetes. I initially was of the opinion that that was maybe reflexive and mostly unnecessary, but he had tales just like this one.
If you ask around, as I did, I'm sure you can find that you know a bunch of people with either first or second hand knowledge of a case where a small startup team (<20 engineers) was absolutely fucked (financially, technologically, expertise-wise, there's a bunch of different ways in can go wrong) by their coupling to one of the cloud providers and had to derail the business development plan for months to decouple. Turns out about a third of the CKAs in my personal sphere got their certification in such a circumstance.
I'm now firmly in the camp that it's never too early to be cloud agnostic, and it's obviously much less expensive to start with it than migrate into it later.
To be fair, you can be totally fucked by any part of the stack you don’t 100% control yourself (or fuck yourselves on a any part you do). Nobody can afford to run everything so you have to just manage risk. In the past couple years even FOSS standards have started flipping and fuck you over like Hashicorp, or Opscode, or Redhat, or Docker, etc.
To be honest, your story doesn't sound like the consultant was reflexive, it sounds like he was making a payday for himself. The move to K8s (EKS), from what I assume was ECS, sounds ridiculous at your size if ECS + Fargate/EC2 met your requirements.
You got taken for a ride and the company spent a whole lot of time/money on busywork for nothing.
it's obviously much less expensive to start
it's way easier to spin up managed services from a vendor than to have to service your own. It's quite involved to just setup the infrastructure. Sure you'll be cloud agnostic, but you'll also be a year behind to launch if you're a startup.
AWS also gives you a lot of start up credits that make it inexpesnive to start with on AWS.
You could also over-index on premature optimization because of scars from the past. Every rule is subject to discretionary criticism.
We've been using on-prem Kubernetes for a few years now, and I'm hearing grumblings of moving towards AWS. The grass is always greener I suppose. (Also, that decision isn't necessarily made by technical resources)