Kubernetes¶
The platform runs on a hybrid k3s cluster: lightweight Kubernetes with workers in two worlds — AWS EC2 for the reliable, scalable tier, and a Proxmox VM in Bogotá as the self-hosted, low-cost tier.
What runs where¶
| Tier | Purpose |
|---|---|
| Control plane (AWS) | Kubernetes API, scheduling policy, storage coordination |
| Infra worker (AWS) | Platform services: ingress (Caddy), SSO (Authentik), secrets operator (Infisical), provider broker, LiteLLM, Matrix, Vikunja, Plane, observability — and the only AWS Longhorn data node |
| Proxmox worker (Bogotá) | The whole agent gateway fleet on self-hosted hardware — the "your server, your data" tier |
| Self-hosted browser worker | The self-hosted browser service, pinned to its local volume |
The former AWS gateway worker was retired in August 2026; every agent gateway now runs on the Proxmox tier.
Storage strategy¶
- RDS PostgreSQL — one managed instance, twelve logical databases, one per stateful service (Authentik, Matrix, Vikunja, Firefly, LiteLLM, Plane, Infisical, …). Centralized backups, point-in-time recovery, and no in-cluster database pods to babysit.
- local-path — agent gateway state lives on node-local volumes on the Proxmox tier, with scheduled backups to S3.
- Longhorn — block storage for the AWS-hosted platform volumes, with recurring snapshots and offsite backups. Its volumes run degraded on a single data node since the gateway worker was retired; a third data node restores the replica count.
- S3 — backups for the self-hosted tier and Longhorn volumes.
Deployment discipline¶
- GitHub Actions deploys from the private repository: reviewed PRs, kustomize overlays per environment, dry-run validation in CI.
- SealedSecrets — plaintext secrets never enter git; the live Secret is re-sealed into an encrypted manifest after every change.
- Canary-first rollouts — new runtime images and configs roll out to internal agents first, then a low-risk tenant, then the fleet.
- Immutable images — runtime images are published to GHCR with immutable tags and digests; manifests pin the exact artifact.
Operational posture¶
- Scheduled and on-demand backups with restore drills (the platform regularly proves it can restore from scratch).
- Placement audits: workloads land on the right tier by policy, not by accident; the control plane never runs tenant workloads.
- The cluster self-monitors: health cron jobs, alerting, and a mission-control panel aggregate gateway, storage, and service state.
Why k3s and not a managed Kubernetes
k3s gives a full Kubernetes API with a fraction of the resource and management overhead, runs equally well on a cloud VM and on self-hosted hardware, and keeps the whole platform reproducible from plain manifests — exactly what a small platform team needs.