Running now · self-hosted

A private AI platform I built and run at home.

The endpoint you just used isn't a cloud AI service. It's backed by infrastructure I designed, deployed, and operate in my homelab — Kubernetes, two different GPUs, a retrieval system, and a lot of supporting services. Here's the short tour.

What it is

Built to learn by actually running it

I started this project to understand how modern AI infrastructure works by building and operating a complete platform, not just running a model in a container.

The primary inference tier is an RTX 5060 Ti running vLLM with Qwen3-14B-AWQ. A second machine with a GTX 1650 Super runs Qwen3.5-9B through Ollama, sharing work between its 4 GB of VRAM and system RAM. I also have a CPU-only Ollama tier. LiteLLM provides one OpenAI-compatible API and handles model routing and fallback.

Qdrant and a custom RAG service let the platform answer from my documentation and its own security findings, while SearXNG gives Open WebUI optional live web search. Forgejo, Argo CD, Prometheus, Grafana, and Tempo are there to make the system deployable, understandable, and maintainable.

The seven-node K3s cluster includes three control-plane VMs using embedded etcd and kube-vip. It tolerates a control-plane VM failure, but all three VMs run on the same physical host — it's deliberately not physical-host high availability.

What's running

A working platform, not just a model

Two GPU tiers + CPU

vLLM serves the main 14B model on the RTX 5060 Ti; the smaller GTX 1650 Super runs a slower hybrid Ollama fallback. A CPU-only tier is available too.

vLLM · Ollama · K3s

One gateway

One OpenAI-compatible endpoint exposes the GPU, CPU, and RAG paths, with routing and configured fallbacks behind the same interface.

LiteLLM · OpenAI-compatible API

Retrieval and search

Private documents and security findings are retrieved from Qdrant. Optional SearXNG search adds current information to conversations.

Qdrant · FastAPI · SearXNG

Git-driven deploys

Argo CD reconciles Kubernetes manifests from Forgejo. CI checks manifests on each push, but currently runs alongside deployment rather than gating it.

Argo CD · Forgejo Actions · kubeconform

Observability

Internal metrics, GPU utilization, distributed traces, and public-endpoint probes from an independent monitor VPS help me spot and diagnose failures.

Prometheus · Grafana · Tempo · OTel

Security pipeline

A daily workflow matches CISA, NVD, and GitHub advisories against live software inventory, drafts LLM remediation notes, and files issues for my review.

Forgejo Actions · CVE feeds · RAG
The pipeline is intentionally human-reviewed: it recommends and records findings, but does not autonomously patch systems. Object storage uses SeaweedFS on the LAN only, without a public S3 endpoint.

How a request reaches it

A public edge, a private homelab

Public HTTPS traffic terminates on a separate VPS running NGINX. Approved requests cross a WireGuard tunnel to the appropriate service inside the homelab. There is no direct public ingress to the home network, and each exposed service has its own authentication boundary.

01 · PUBLIC EDGENGINXTLS, service authentication, rate limits
02 · ENCRYPTED TUNNELWireGuardPrivate transport into the homelab
03 · AI PLATFORMLiteLLM / RAG / GPUK3s inference, retrieval, and CPU backends

The control plane uses three etcd members and a floating kube-vip API address. All control-plane VMs share the homelab hypervisor, so a complete host outage is still a platform outage.

Behind the endpoint

Full architecture

AI infrastructure architecture: public edge and monitor VPS, WireGuard tunnels, homelab Docker services, seven-node K3s cluster with three etcd control-plane VMs, two GPU tiers, RAG, GitOps, observability, and automated CVE workflow
Current architecture — seven K3s nodes, two GPU tiers, independent monitoring, and a daily security workflow.Open full size ↗

See it in action

Recorded API demonstrations

Inference, retrieval, web search, and fallback

Sanitized prompts and recorded output excerpts from the running platform. No credentials or infrastructure secrets included.

View examples ↗

The stack

Built from open tools and a lot of operational work

  • Debian 13
  • K3s
  • Embedded etcd
  • kube-vip
  • Terraform
  • WireGuard
  • NGINX
  • NVIDIA
  • vLLM
  • Ollama
  • LiteLLM
  • Qdrant
  • FastAPI
  • Open WebUI
  • SearXNG
  • Argo CD
  • Forgejo Actions
  • Prometheus
  • Grafana
  • Tempo
  • OpenTelemetry
  • SeaweedFS