Why we moved to k3s
After three years on full Kubernetes we switched to a lightweight distribution. Reasons, surprises and numbers.
Three years ago we set up full Kubernetes on a set of our own servers. It was the right call — at the time. We had a team that knew what it was doing and applications that benefited.
Over time it became clear we were using a fraction of what Kubernetes offered. And maintaining the control plane itself started eating more time than running the applications on top of it.
Why k3s
k3s is a Kubernetes distribution from Rancher Labs that fits an entire cluster into a single binary. No separate etcd, no standalone kube-proxy, no hour-long install.
# single-node install
curl -sfL https://get.k3s.io | sh -
sudo k3s kubectl get node
# NAME STATUS ROLES AGE VERSION
# infra01 Ready control-plane,master 42s v1.30.2+k3s1
Forty-two seconds. Previously kubeadm init with etcd preparation took a solid twenty minutes — and that was with an Ansible role doing it for us.
What we had to change
It isn’t entirely painless. Three things slowed us down:
- Traefik instead of ingress-nginx. k3s ships Traefik out of the box. You can disable it (
--disable traefik), but we chose to migrate the annotations. Half a day. - local-path-provisioner instead of Ceph RBD. For stateful workloads we had to think about what genuinely needs networked storage. Turned out: only databases — and we moved those outside the cluster.
- SQLite instead of etcd. Fine for single-node. For HA you need embedded etcd or an external DB. We went with embedded etcd across three nodes.
The best code is the code you don’t have to write. The best infrastructure is the kind you don’t have to maintain.
Numbers
Measured on a production cluster, three nodes, same hardware before and after.
| Metric | Before (kubeadm) | After (k3s) |
|---|---|---|
| RAM, control plane idle | 2.1 GB | 480 MB |
| Node boot time | ~120 s | ~55 s |
Pods in kube-system | 14 | 6 |
| Monthly CPU minutes | baseline | −38 % |
The RAM difference was the biggest surprise. Freeing 1.6 GB per node meant we fit half again as many application pods on the same machines.
When k3s is not the right call
- More than ~100 pods per cluster and still growing.
- You need cloud-provider integrations (AWS ELB controller, EBS CSI, etc.).
- You have a compliance requirement for a specific certified distribution.
- Someone else is paid to run your control plane. Then EKS/GKE is cheaper in terms of your time.
Conclusion
For us it was a clear win. Fewer moving parts, faster recovery, lower consumption. And crucially — we can rebuild the whole cluster from one shell script and a backup, which was never true of the previous setup.