02 Inference · primary GPU
vLLM · Qwen3-14B-AWQ
RTX 5060 Ti · 16 GB VRAMThe primary, high-capacity inference path running Qwen3-14B-AWQ on the homelab GPU.
Recorded prompt
Recorded response
Recorded platform demonstration · October 25, 2026
These are saved prompts, model responses, observed timings, search results, and an infrastructure-inventory snapshot from my self-hosted AI platform. The demonstration covers primary and fallback GPUs, CPU inference, retrieval, live search, and gateway routing.
Model identities below reflect the actual configured platform models from the current project documentation. The demo script's stale model labels have been corrected; recorded answer text and timing measurements are retained. Exact software, container, and kernel versions, internal monitoring ports, and implementation identifiers that would increase reconnaissance value are withheld.
01 Platform introspection
The inventory exporter tracks host packages, container images, and Kubernetes workloads. At recording time it reported 985 components. Workload names and roles are retained below; version tags, precise package releases, and the kernel build are omitted.
| Kind | Area | Component | Exact version |
|---|---|---|---|
| DaemonSet | Cluster services | kube-vip | Withheld |
| DaemonSet | GPU management | NVIDIA device plugin | Withheld |
| DaemonSet | Cluster services | K3s service load balancer | Withheld |
| Deployment | Inference | LiteLLM | Withheld |
| Deployment | Inference | Ollama | Withheld |
| Deployment | Inference | vLLM | Withheld |
| Deployment | RAG | Python RAG service | Withheld |
| Deployment | GitOps | Dex | Withheld |
| Deployment | GitOps | Redis | Withheld |
| Deployment | GitOps | Argo CD server | Withheld |
| Deployment | Cluster services | Local-path storage provisioner | Withheld |
| Deployment | Cluster services | CoreDNS | Withheld |
| Deployment | Cluster services | Traefik | Withheld |
| Deployment | Cluster services | Metrics server | Withheld |
| Deployment | Observability | OpenTelemetry Collector | Withheld |
| StatefulSet | RAG | Qdrant | Withheld |
| StatefulSet | GitOps | Argo CD controller | Withheld |
Git · NVIDIA Container Toolkit · NVIDIA Container Toolkit Base · OpenSSL · sudo
One reporting system in this sample; exact kernel release and patch revision withheld.
02 Inference · primary GPU
The primary, high-capacity inference path running Qwen3-14B-AWQ on the homelab GPU.
Recorded prompt
Recorded response
03 Inference · secondary GPU
The secondary node runs a hybrid GPU/CPU model, trading latency for a capable fallback inference tier.
Recorded prompt
Recorded response
04 Inference · large CPU model
A larger, CPU-only mixture-of-experts model provides a slow quality-oriented path. This request took approximately 2 minutes 51 seconds.
Recorded prompt
Recorded response (excerpt as captured by the demo script)
The CAP theorem, proposed by Eric Brewer in 2000, states that a distributed data store can only guarantee two out of the following three properties simultaneously:
The Core Premise: In a distributed system, network partitions are inevitable. Therefore, you cannot build a true distributed database that avoids partition tolerance ($P$). When a network split occurs and communication is severed, the system must make a choice: either halt operations to maintain Consistency (CP) or keep responding with potentially outdated data to maintain Availability (AP).
Designers primarily choose between Consistency and Availability when a partition ($P$) occurs.
When a partition occurs, the system prioritizes data integrity. If nodes cannot verify with others that data is safe to write, they reject the request.
05 Retrieval-augmented generation
A custom retrieval service searches the private corpus and sends relevant context to the model. Both recorded queries are shown, including the one that did not produce an answer.
Recorded prompt · monitoring
Recorded response · internal ports redacted
Recorded prompt · CVE findings
Recorded response
The second result is intentionally shown as recorded. It indicates this particular query did not retrieve the requested findings; it is not a verified statement that the platform has no actively exploited vulnerabilities.
06 SearXNG + local model
The search engine retrieves external research excerpts, which the locally served model uses to produce a consolidated response. Search snippets are shown as recorded, including their original truncation.
Recorded search query
Four research results passed to the model
https://arxiv.org/abs/2306.00978v6
Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce the cloud computing cost and protect users' privacy. However, the astronomical model size and the limited hardware resource pose s
https://www.ncbi.nlm.nih.gov/pubmed/42062362
In motorcycle motorsport, engineering decisions are constrained by mass, stiffness, fatigue life, and thermal margins. In a comparable way, deploying large language models (LLMs) for edge or trackside assistance is limited by memory capacity, bandwidth, latency, and power. This paper presents a syst
https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html
… We compare TinyChat against existing edge LLM inference systems AutoGPTQ, llama.cpp and exllama in Figure 10. Our system achieves up to 1.7× speedup over llama.cpp on Orin. …
https://arxiv.org/pdf/2605.15152.pdf
LLM quantization has become essential for memory-efficient deployment. Recent work has shown that quantization schemes can pose critical security risks: an adversary may release a model that appears benign in full precision but exhibits malicious behavior once quantized by users. However, existing q
Recorded model answer
07 Gateway routing
A controlled availability test removes the secondary Ollama service, sends a request through its public alias, then restores the service.
Outpost Ollama scaled to zero replicas.
Request targets qwen-outpost, with qwen-gpu configured as fallback.
The secondary Ollama service is restored after the test.
Recorded response
The recorded result shows a successful response while the outpost service was stopped. The response alias is still qwen-outpost; this excerpt does not independently expose the specific upstream that answered.
08 Gateway model discovery
The gateway advertised these four aliases in the recorded model-list response. The concrete model names here reflect the current deployment documentation.
qwen-gpuQwen3-14B-AWQ · vLLM · primary RTX 5060 Ti
qwen-outpostQwen3.5-9B · Ollama · secondary GTX 1650 Super + RAM
qwen-cpuQwen3.6-35B-A3B · CPU-only Ollama on homelab
infra-ragCustom RAG service · Qdrant retrieval + Qwen3-14B generation