A private AI platform I built and run at home
The endpoint you just used isn't a cloud service. It's a Kubernetes cluster in my homelab that I designed, deployed, and keep running. Short tour below.
What it is
I started this to learn how modern AI infrastructure actually works by running it, not by reading about it.
Inference runs on two GPU nodes. An RTX 5060 Ti handles most of the work and a GTX 1650 Super takes the overflow. LiteLLM sits in front of both and exposes one OpenAI-compatible API, so requests can move between GPU and CPU backends without the caller having to care.
There's also a retrieval layer on Qdrant, so models can answer from my own documentation and notes instead of training data alone. When a question needs current information, it can pull in live web results too.
Everything else exists to keep that running. Config lives in Git and deploys through Argo CD, CI checks it before it lands, and Prometheus and Grafana show me what the cluster and the GPUs are doing. A daily job compares public CVE feeds against the software actually installed on my hosts and in the cluster, drafts remediation notes, and opens an issue for me to review.
What's running
GPU inference
Two GPU nodes with tier-aware scheduling, so heavy models land on the RTX and everything else has somewhere to go.
vLLM, KubernetesOne gateway
A single OpenAI-compatible API in front of every backend, with routing and fallback handled for you.
LiteLLMRetrieval and search
A vector store plus a private search layer, so answers can come from my corpus or from the live web.
Qdrant, SearXNGGit-driven deploys
Every change is committed, validated against the Kubernetes schema in CI, then reconciled onto the cluster.
Argo CD, Forgejo Actions, kubeconformMonitoring
Metrics, logs, traces, and GPU utilization in one place, so I can tell what broke and when.
Prometheus, GrafanaPatch pipeline
A daily job matches CVE feeds against a live software inventory and files issues with suggested fixes.
CISA KEV, NVD, GHSAHow a request reaches it
A public edge takes the HTTPS request and forwards it through an encrypted tunnel into the house. The cluster itself, including the models, databases, and everything supporting them, isn't reachable from the internet. Addressing and the security details are left off this page on purpose.
Stack