K3s homelab platform

Problem

Side projects usually die in deployment. Every new app needs hosting, TLS, a database, backups, monitoring and a way to try changes before they go live. Doing that by hand per app is how projects stall. The goal was a platform where shipping a new app is a template and a pull request, and where the deploy process itself stops mistakes.

Approach

Everything is declared in one Git repository and reconciled by Flux. Nothing is changed in the cluster by hand.

Architecture

Shipping

• Review apps for every PR. Opening a pull request in any app repo builds an image and dispatches an event. The platform repo then generates an isolated namespace, hostname and TLS certificate for it. Closing the PR tears it all down.
• Gates built into the workflows. The org is on GitHub’s free plan, so there are no server-side required reviewers. The gates live in the workflows instead: promotions poll the CI check before merging, and a post-merge job verifies that the new image actually rolled out.
• One shared, backed-up Postgres. CloudNativePG runs a single cluster. Apps get isolated databases and roles from a manifest, with nightly volume snapshots kept 14 days, plus continuous WAL archiving and weekly base backups to object storage for 30 days of point-in-time recovery.
• CI inside the cluster. Ephemeral GitHub Actions runners and a shared remote BuildKit, so there’s no Docker daemon anywhere, including on the dev machine.
• Safe for coding agents. Agents do much of the day-to-day work here, so their access is scoped. A read-only MCP server gives them pods, logs, events, Flux status and PromQL through a tailnet identity that RBAC limits to get/list/watch, with no Secrets. Anything that changes the cluster still has to be a pull request.
• Observability. Prometheus, Grafana, Loki (14 days of logs for every pod), Alertmanager pushing to phones, and an external dead-man’s switch that reports when the whole lab is down.

Hard parts

• Upgrading Flux safely. Each Flux minor version drops old API versions, so manifests have to be migrated before the controllers are swapped. Otherwise anything still on a removed version silently stops applying.
• Automated dependency updates without footguns. Renovate automerges only patch-level image updates and minor chart updates, and only after CI is green. Everything else waits for review.
• Hostnames live in the apps. ALLOWED_HOSTS, CSRF origins and CSP host lists are app code, so adding a domain means the app change deploys before the ingress change, or the new host returns 400 until it does.
• Public ingress from home. A small cloud front end passes TLS through to the cluster over a tunnel, with PROXY protocol so apps still see real client IPs.

Outcome

The strip on the home page is live data from this cluster. As of this writing it runs 4 nodes (three bare-metal, one GPU), ~85 deployments and ~150 pods, at 99.9%+ availability over 30 days. It hosts 12 apps wired for review apps. The platform repo has 1,300 merged pull requests, about 150 of them review-app deploys. Every cluster change goes through Git.

It is also more expensive and less reliable than the cloud.

Stack