Two GPU tiers + CPU
vLLM serves the main 14B model on the RTX 5060 Ti; the smaller GTX 1650 Super runs a slower hybrid Ollama fallback. A CPU-only tier is available too.
The endpoint you just used isn't a cloud AI service. It's backed by infrastructure I designed, deployed, and operate in my homelab — Kubernetes, two different GPUs, a retrieval system, and a lot of supporting services. Here's the short tour.
What it is
I started this project to understand how modern AI infrastructure works by building and operating a complete platform, not just running a model in a container.
The primary inference tier is an RTX 5060 Ti running vLLM with Qwen3-14B-AWQ. A second machine with a GTX 1650 Super runs Qwen3.5-9B through Ollama, sharing work between its 4 GB of VRAM and system RAM. I also have a CPU-only Ollama tier. LiteLLM provides one OpenAI-compatible API and handles model routing and fallback.
Qdrant and a custom RAG service let the platform answer from my documentation and its own security findings, while SearXNG gives Open WebUI optional live web search. Forgejo, Argo CD, Prometheus, Grafana, and Tempo are there to make the system deployable, understandable, and maintainable.
The seven-node K3s cluster includes three control-plane VMs using embedded etcd and kube-vip. It tolerates a control-plane VM failure, but all three VMs run on the same physical host — it's deliberately not physical-host high availability.
What's running
vLLM serves the main 14B model on the RTX 5060 Ti; the smaller GTX 1650 Super runs a slower hybrid Ollama fallback. A CPU-only tier is available too.
One OpenAI-compatible endpoint exposes the GPU, CPU, and RAG paths, with routing and configured fallbacks behind the same interface.
Private documents and security findings are retrieved from Qdrant. Optional SearXNG search adds current information to conversations.
Argo CD reconciles Kubernetes manifests from Forgejo. CI checks manifests on each push, but currently runs alongside deployment rather than gating it.
Internal metrics, GPU utilization, distributed traces, and public-endpoint probes from an independent monitor VPS help me spot and diagnose failures.
A daily workflow matches CISA, NVD, and GitHub advisories against live software inventory, drafts LLM remediation notes, and files issues for my review.
How a request reaches it
Public HTTPS traffic terminates on a separate VPS running NGINX. Approved requests cross a WireGuard tunnel to the appropriate service inside the homelab. There is no direct public ingress to the home network, and each exposed service has its own authentication boundary.
The control plane uses three etcd members and a floating kube-vip API address. All control-plane VMs share the homelab hypervisor, so a complete host outage is still a platform outage.
Behind the endpoint
See it in action
Sanitized prompts and recorded output excerpts from the running platform. No credentials or infrastructure secrets included.
The stack