What job interviews taught me about Kubernetes

Pretty sure if there was a simple alternative, people would hate it.

Everyone initially wants thing A. But then they want to customize it to do all permutations and combinations n of A, B, C. They want it to be extensible. They want redundancy. They want orchestration. They want integration.

It’s why practically every config file format eventually becomes its own scripting language. Even HTML started off simple — now ridiculously complex — all the more ironic since practically nobody writes it by hand. Instead of CSS simplifying it, it became more complex.

There is another thing that is extremely customizable and extensible. It’s called a programming language. People write programs to solve specific problems.

There seems to be a perverse trend of cobbling together a Byzantine mesh of libraries, plugins, and services with complex configuration files to make it do practically everything possible. We just used to write software for such purposes…

And for anyone who thinks HTML is simple… the A (anchor) tag has an “ping” attribute that results in POST requests to a list of URLs when a link is clicked ! The list of attributes and resulting variations in behavior is quite mind boggling. It was supposed to be a damn link! https://html.spec.whatwg.org/multipage/links.html

I really wish there was an 80% kubernetes. I think you could get there with some changes:

1. No overlay networks. 1 IP per machine. pods use dynamically allocated ports, and the kubelet enforces pods listen only on their assigned ports using seccomp.

2. No kube-proxy or equivalent Layer-4 "load-balancer". It's not good, but it's often used. You should use some kind of Layer-7 load balancing instead. Also you need to look up the port number from (1). This also greatly lessens the need for DNS.

3. A better config language. YAML and helm templates are terrible. kustomize is built into kubectl, but it's frustratingly limiting and also still very complicated. Something like nix would have been great. This can make it easier to upgrade third party configs since you can have more logic to validate and merge your settings with upstream defaults or templates.

4. Maybe an EBF-like for the api server? If the built-in k8s objects don't have a setting for something, then you need to write an operator or control loop yourself and then run that too, which is a big lift. Over time, k8s just keeps adding more and more built-in things and then revising them, which creates a ton of churn. If you could easily script simple operations, then they wouldn't have to build in every permutation ahead of time. E.g. the HorizontalPodAutoscaler has 24 config object types with several fields each, but all it does is set replicas based on data read from the api-server, so it could be replaced by some kind of flexible script that runs in the control plane.

I don't know... running a startup sized kubernetes is relatively easy and pain free these days (k3s). Especially when it comes to scaling up.

CNPG is an absolute monster (in a good way). cert-manager is easier than the docker alternative, calico has never failed me (except in bgp mode which has some footguns like not being able to come back from a dead state since it has a chicken and an egg problem unless you point it to the external load balancer which I would have known if I read the documentation). trafeik is all you need. talosos largely mitigates the bare metal problems and comes pre-hardened and pre-optimized.

I solo most of my development projects and have used k3s for all of them. The only complaint is that cert-manager by default will fail silently and your certificates will expire. I largely mitigated this by having proper visibility setup via grafana and automated alerts (warns if certificates are about to expire) which should have been done by me anyway.

Two years ago I'd agree, today with LLMs everything I have runs talos with fully automated updates and I haven't had to be on-call for almost a year.

  MDomain blog.kronis.dev

Docker swarm is that simple solution you're looking for. But people don't need simple solutions. They want scalable solutions and Kubernetes fits this niche perfectly. You can deploy it on single server today and scale to 100 servers managed cluster tomorrow.

Just to provide a similar example. Linux system is insanely complicated. Kernel alone has thousands of options. Distos have tens of thousands of packages. Wherever you look at, everything is hard and complicated. Firewall, containers, init system, filesystem hierarchy, storage layers. One would think that some people desire simpler operating system. But everybody uses Linux despite all these complexities. Try to find OpenBSD in production, for example. It's not easy.

> One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start).

And if you can do this again, what's your solution to reverse proxy, certificate management, DNS...etc? I guess you can docker-compose some custom stack on a single machine, maybe add one more machine then you can say it's HA enough for small scale. But you can also spend the same amount of time to install those kubernetes controllers with zero customization. In my experience, if you go with the default configuration, most of the well-maintained k8s components are boring as hell these days.

> (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS)

If load to your service increases, you need to scale up/out your service. This is universally true. Do you have a proprietary solution that's easier and more reliable than bumping up the replicas count in kubernetes?

There are lots of design decisions in Kubernetes that I hate. But if you want me to choose between Kubernetes and any proprietary stack, in 2026, I would definitely choose Kubernetes.

> I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions

Hashicorp's Nomad basically is just that, supports various way of running stuff too which is neat. Shame about the license change which basically killed all my interest in it, so seems the hole is indeed still unfilled.

Dear friend, you have built a Kubernetes

https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...

k8s is not a pain, I would never return to something like Pupet / Ansible / etc ... to deploy bare ec2 instances, it's just re-inventing the wheel badly.

Just use ECS / Fargate with an ALB in front if you need a simpler use case.

>I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity

That's how I see it as well but it's really tough to go against the grain. I have a small enthusiastic community of users around Uncloud (https://github.com/psviderski/uncloud) who went full circle - fed up with k8s and came back to simple, boring declarative Compose deployments across a handful of interconnected hosts.

Uncloud is essentially a cluster version of Docker Compose without a control plane and cluster management overhead.

We started out core product on ECS, which is a declarative way to run a containerized service. It has been nice and reliable, but it has limitations (slow scaling, weird AWS Quotas if you have ephemeral tasks).

We're moving our non-critical components onto EKS (pipelines, tooling, etc). We had one outage from runaway IP allocation in a subnet, but otherwise it's been pretty stable.

I do hear vague horror stories so I'm really not excited about moving our prod stack to it, but it's actually been really good for installing 3rd party software so far.

The pattern is pervasive. Big corp promotes a solution that fits their need. People read about it, think adopting big corp solution means they are doing the right thing. Few people have big corp need, let alone everyone big corps are different. And then endless hours spent fighting big corp solution to not so big corp problem.

Working with k8s myself I'm somewhere in between you and the article on opinion. I think k8s is good when you can afford to hire a person dedicated to managing it (or at least find someone with experience in running it that can make it part of their MO)

That is, k8s is probably best considered when you are beginning to consider having an infrastructure department, or if one of your early hires knows Kubernetes and is opinionated in a way that is less "throw cool and complex stuff at the wall"* and more "the 5 things I want in a k8s cluster that I don't want to spend much time on and should just work"

My understanding of the 2000s and 2010s was that there was a big focus on inventing self service deployment systems for developers, and k8s is that solution(!), for the same scale that would begin considering re-inventing the wheel internally anyways

Is the simpler thing not Lambda/Cloud Run (with terraform)?

On the application developer side k8s is awesome fo, but the you look inside the box and it melts your face off.

I'm not sure a middle ground exists unfortunately. It's either full service like Lambda or bag of knives like k8s.

We also had similar problems at our small scale startup. We tried k8s for a pilot project, and observation was the same: the complexity was not worth it for us. We needed something simpler instead we adopted Nomad, which actually fit our use case. It had its own issues and bugs, but overall, it was much more straightforward to work with.

Without kubernetes you end up gluing a bunch of plumbing together anyways. It’s just that people don’t see the glue because they’re so used to it and learning something outside of that framework creations friction.

This isn’t so different from say Linux vs BSD. You can roll your own things and call it a system. Or you can just use something that targets a spec to provide a (mostly) cohesive and consistent layer to build upon.

As the article mentions towards the end, AWS EKS, GCP GKE, and other competitors have made k8s setup turnkey. You can deploy a new cluster with all the controllers you mentioned in a single click / Terraform.

> I think there is a real hole in the market for a simple solution

Unless of course, all of the busywork that comes with kubernetes IS the value (to the engineer). Perhaps a bunch of engineers know at some level that locking the company into an overcomplicated cloud-within-a-cloud setup that has all sorts of weekly issues and requires constant work gives them a lot of job safety that they wouldn't get if they just used an AWS autoscaling group and you're done for the next 5 years.

Because simpler solutions DO exist (like a loadbalancer in front of an autoscale group, and not making a giant SOA for an app that orders you taxis, or books you a bnb or whatever nonsense).

Isn't fargate or ecs that simple service?

Kamal is somewhere in the middle. Probably a little closer to a bunch of bash scripts. But it’ll get your container going pretty quick. Can take a bit of fiddling with SSH/docker-login. Plus it handles deployments very well.

I built canine.sh for exactly that reason, gives you a sensible deployment platform on top of k8s with one install, and you can customize it once you outgrow it.

> there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions

There's Nomad for this; I wish more teams would run Nomad.

There was: Heroku

It was glorious.

to what extent would AWS EKS auto mode solve those problems?

>I think there is a real hole in the market for a simple solution that lets you deploy some containers

Containers? In this climate? What's the kernel LPE rate at after copyfail and copyfail2? No containers, VM or harden. No half measures.

If there's going to be something new, it needs to be topical, and containers are out.

> I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.

I mean, it's CDK and whatever equivalents other providers have, isn't it? If you fully embrace all the stuff they give you then it's straightforward to declare everything and it all works together. The downside is the vendor lock-in but unless you actively deploy to multiple environments, which most people don't, you're probably locked in in various ways without knowing about it.

I had exactly the same experience, and we went into the same direction (ansible + ec2). No regrets.

Docker swarm has been providing the simple solution for over a decade but nobody wants to use it because it's not k8s.

Dokku?

> k8s has really sucked up all the oxygen.

Because anything else involves making opinionated decisions that will be wrong for many users.

People who don’t understand why k8s is so widespread don’t understand all the problems it’s solving.

GCP cloud run is pretty close to this. We’re using it now and I’ve a lot of experience with gke.

They’ve announced persistent “instances” recently which solves a big problem for us - sometimes you want continual long running workloads.

Nomad, Consul and Vault all running on VMs that you manage with Terraform.

The problem is that when you run this long enough you want K8s features anyway.

Strong agree, if there's one thing LLMs are excellent at, it's writing Terraform and Kubernetes deployments (and/or helm charts). What used to be half a day of research, trial and error, is now 20 seconds of AI churn and 98% of the time it nails it on the first try. And then point it at grafana and tell it to write you a dashboard for the new service/s. Easy peasy lemon squeezy. What used to require a team of 4 devops/SRE to support a medium sized company, can now be collapsed down into a a single part time SRE.

> One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances.

Using Kubernetes because you're unable to grok docker's networking enough so you can't run multiple containers using their own ports and not conflicting with other stuff sounds like a recipe for disaster, even (especially?) if you use agents for this. Particularly if you let them manage a production environment, you're bound to lose important data eventually.

> pretty much free lunch.

Aah, famous last words of the young :)

Finally just bought a piece of my own hardware and got LLM to deploy k3s cluster on it.

I think diy homelab/hosting is more accessible than ever.

Cut costs on cloud spend and invest into AI spend.

For a solo dev on a budget, I think it just makes sense.

This cuts both ways.

At any stage of https://www.macchaffee.com/blog/2024/you-have-built-a-kubern... a SOTA model can repackage it into Kubernetes.

If you're feeling extra spicy you don't even need the deploy scripts. Just a `llm` user account with the right permissions & ssh keys on all your servers.

> Ask your favorite GPT to generate manifests, ... > Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch.

Writing manifests seems like a trivial thing to focus on. Who operates the k8s cluster in production? Who runs upgrades? Who's on call to monitor the system? Of course if someone else is doing all the work for you, it feels like free lunch!

Interesting. I have just started reading about Kubernetes. Is there an reading material that goes over this process you just described?

100% agree with you.

You can actually treat kubernetes as a glorified docker compose engine. Deploy pods, deploy nginx instead of ingress controller, deploy certbot cronjob instead of cert-manager, and believe it or not, it'll work! On a single server!

People often compare Kubernetes with thousands of additional services to a simple VPS, but that's not apples to apples comparison.

> many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update

I just want to point out that you can totally still do this with Kubernetes. Of course it's not correct, but you can save that unencrypted secret in a .env file right into your container while you're building it - no need to use Kubernetes's support for supplying environment variables from the manifest. And of course, you don't even need a Dockerfile to build that container - you can just exec into a running container, paste it in, and then docker save.

Kubernetes doesn't save you from making stupid decisions, it just makes it easier to make better ones.

Sounds like you had specific issues with openbap and your cluster provider. Whatever the tech stack it's all presented the same way and easily discoverable. Kubernetes is a cloud operating system- this is exactly the point about standardized knowledge.

Spot on. I have a lot of trouble convincing cloud folks that for durable state, you probably don’t want kubernetes. It’s not that e.g. the CSI drivers and operators for clustered databases aren’t top notch—they are; the era of “avoid stateful kube services” is long behind us—it’s that the cloud provider managed services for e.g. blob stores or databases are so much more reliable. The S3s and Auroras of the world are expensive for a reason: no matter how good your kube native database operator is, it still doesn’t assume responsibility for a ton of the failure points that managed services do. And that’s true even at modest scale (e.g. upgrades are just harder when you’re running your own DB) and in cost conscious environments (sure, the Elasticache bill is steep, but the salary and velocity cost of fixing memory-leak-caused kube memcached crashes is steeper).

I thought that was the point of the article, right?

That the tech benefits may not be there, but they’re using it for the non-tech benefits

I think a big part of this whole discussion (and why it's always such a divisive topic) is that there are so many factors to consider that there is no one single golden bullet. An industry standard just makes this issue go away; also it makes the decision making easier since you're not taking risks going for "VM-with-systemd" or "plain docker" or "bare metal" over $STANDARD.

> I would not advise asking the majority of CTOs these questions either. Many got to that position by saying what people want to hear, which is the "average" safe answer.

Agree; this is the same as asking people why they're not having kids: they either a) don't know or b) don't want to / are not willing to say the truth.

This blurb gave me the idea to try and quantize this. Scrape the top HN blogs over the last few years and see how occurences of common phrases change.

I'd expect to see a huge increase in "solving real problems" over the last months.

> less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated

I’d like to gently push back on that. ;-D

Terraform, when committed to git, provides organisational memory. But less so uniformity, since all providers are different (and you should expect different things when applying). No tracing besides git. And tfstate is hard to share between developers, unlike kube state.

Kubernetes is more the same across providers. And it manages drift after something is applied, which is not a direct argument of OP, but a strong reason over other IAC.

And yes, I also enjoy how well deploying works. And how things generally fit together. Liking the networking complexity less so.

The upgrade cycle is a feature, not a bug. If (when) you need to do a big lift and shift, or there's some 0 day CVE, push buttan, get security update. You CAN drift behind but there's a real $$$ cost to that now. Every three months I toss opus at my k8s stack and verify it's compliant with k8s v1.xx.y and then push the upgrade button on my staging cluster, and then a week later I push the upgrade button on my prod cluster. What used to be two days of maintenance every quarter is now more like 2-5 minutes spread across the two upgrades.

I'll admit I'm dreading switching over to the gateway api, but by the time I get forced off ingresses it should be a stable/mature ecosystem. That's still a ways out though.

I don't know anyone still dealing with VMs anymore, except our IT guy who manages a couple of pet servers for random executives from the before times. In the last year k8s has started absorbing executive pet processes and the number of VMs our IT guy manages has dropped by about half.

While I'm here spouting stuff, yeah hiring for k8s is real easy, if our SRE gets hit by a bus, he can be replaced in a week, and we can probably struggle through using opus until that happens. K8s being he lingua franca of git ops IaC makes it real easy for the new guy to parachute in and start working. Every VM thing is going to be totally bespoke and have the personality of the guy who designed it, which is rarely a good thing.

I somewhat agree with you... but it's not like you don't need some actual experts who know what they're doing, especially when stuff goes bonkers and it will go bonkers.

Even on AWS EKS, you will run into bullshit with their network overlay. Egress policies are a mess (at least half a year ago, you were not able to say something like "allow pod A to egress traffic to service (!) B" despite a service resolving down to an IP address in the end.

And that's before going into the unholy mess that is getting connectivity to and from the external world to your cluster. Cloudfront, ACM certificates, ALB, ALB-EKS integration, Route53, Route53-EKS integration, EFS, EFS-EKS integration, EBS, EBS-EKS integration, RDS, RDS-EKS integration, IAM-EKS integration, SSM, SSM-EKS integration, autoscaling... and if you want more pain and don't already wince, try setting that up across regions or, as I had to do once, across account boundaries.

Kubernetes is powerful. But do not make the mistake of assuming it's easy to get started with, at least on the admin side. Even if you got prior AWS experience, getting it all integrated into EKS so you don't have to deal with Terraform and helm/k8s for a full deployment of a piece of software will take you an awful lot of time.

For users though? It's a breeze, I will admit as much. Everything down to the firewall rules can be encoded in k8s spec files.

What I've seen more than anything else is that Kubernetes built an ecosystem (of contributors and users, but also of companies invested in its success) that none of its competitors could or would. There was apparently a faction within Google that believed open-sourcing Kubernetes was a mistake because Google would have made more money keeping it in-house, but in terms of the success of the project I think it was entirely the right call, as was creating a foundation to maintain and promote it. Look at the history of its competition:

* DC/OS was always its own thing and as time went on, eventually Mesosphere was basically the sole maintainer of the underlying Mesos. Very little external contribution.

* OpenShift was different from Mesos and basically maintained only by Red Hat from the Makara acquisition (sometime in 2010 I think) to about mid-2015 (i.e. the point where they ripped out most of the OpenShift-native process isolation and orchestration and replaced it with Docker and Kubernetes). Pre-Kubernetes OpenShift frankly struggled to catch on and again, basically everybody who cared about developing it worked for one company.

* CoreOS was developing fleet in the open but dropped it outright when Kubernetes was released. The phrase I heard there was "We started to say something and Google finished our sentence." They pivoted to Kubernetes for orchestration so hard it was kind of awkward talking to customers who used fleet after that. In theory somebody could have picked it up like Kinvolk picked up rkt for awhile (and later CoreOS Linux as Flatcar), but as far as I know nobody ever made a serious effort to do so.

* Docker released Docker Swarm shortly after Kubernetes was released -- yet another one-company product. (I still don't really understand why they released Swarm -- for simple workloads, Docker Engine and Docker Compose were enough, and for more complex ones Docker Engine was, at that time, still the sole underlying runtime in Kubernetes. There were already two distinct orchestrators on the market, one from a much larger company with a lot more operational experience running containerized workloads than Docker had. What was their thought process?)

* HashiCorp released Nomad well after Kubernetes but not only was it another sole-corporate-maintainer orchestrator, it deliberately omitted a lot of the basics Kubernetes included like service discovery in an effort to stay simple -- so in very few cases was Nomad alone actually enough to orchestrate workloads (nor was it intended to be, as the Nomad engineers in the ~1.0 days would have been first to tell you). Past a point this made Nomad more work to get running and keep running than Kubernetes was.

The flip side is, I don't think a purely community-developed orchestrator would have won, even with a foundation backing it. It's not the corporate backing that's the issue, it's the lack of diversity in that corporate backing.

Well, maybe especially as a solo dev. Things like Heroku and other tools that were largely called PaaS at the time were very popular with individuals and small teams but they had limitations for enterprise development--and even ran into barriers once anyone ran into those limitations.

From what I've seen, Azure container apps (ACA) sits exactly there in the middle: a managed Kubernetes, opinionated and full of defaults you may or may not like, but makes the deployment and maintenance a breeze.

The setup may be simple, but what about the maintenance?

There are compile-to-yaml config languages

"linux+nginx+postgres+favourite-lang" - I agree here. Yes, don't over-complicate things.

Yeah this was ,ynexperienve as Wellfleet. Nó one réaluadar knew what was going on and we had 5 figure cloud spend bills per month.

Honestly I felt the same way for a while but the more I'm exposed to both Fortune 500 companies and ones who have a handful of employees I see Kubernetes as just a good starting point rather than adopting it later.

It removes the overhead of a lot of what sysadmins and devs of yesteryear did by hand or had to have a career's worth of experience to do quickly.

That's not to say that people don't need to know what they're getting into when they adopt kubernetes but especially when you're using a managed offering and not on the bleeding edge of what it supports it's pretty easy in terms of overhead and maintenance.

But if you just build bash scripts on top of vms, the talent pool is much larger, necessarily.

I feel the motivation goes a bit in the opposite direction. Yes a commoditized worker who speaks a common k8s language, but not too common a language, there has to be some selection here, a talent pool of 100K engineers that know linux is too big, but a pool of 5k engineers that know k8s apparently is just right.

Gotta filter those resumes somehow!

Tangential but IME those engineers do very very well, as they're good at marketing themselves, but more importantly they often align with the CTO/CEO who wants to present the company as the hottest thing since sliced bread – they signal they're on the same page.

Who is going to criticise the engineer who is open to new things / appears innovative / loves MVPs / 110% supportive of the latest bs the CTO is spewing? Yeah

The linked article discusses very different reasons for preferring kube. CTOs and hiring managers like it for reasons totally different from the cargo cult/hype-driven engineers.

that probably depends on how much security and resource isolation you need. Multi-Tenant security in Kubernetes is not a simple thing, for a wide variety of reasons, and noisy neighbour problems are also potentially a headache.