AI Platform Demo – Queries & Outputs

Sanitized demonstration page. Exact software versions have been generalized by replacing numeric version components with x. Infrastructure explanations are intentionally omitted; this page focuses on live query examples and results.

Prompt:
What are the core tradeoffs when choosing between AWQ and GPTQ quantization for production LLM deployment?

Output (excerpt):
• AWQ generally preserves model quality well while remaining efficient.
• GPTQ often provides strong compression and compatibility.
• Tradeoffs include deployment complexity, throughput, memory usage,
  accuracy retention, and operational simplicity.

Latency: ~5.3s
Model: qwen-gpu
Runtime: vLLM + Qwen2.5-7B-AWQ

Prompt:
Explain the benefit of heterogeneous GPU scheduling in a self-hosted AI inference cluster.

Output:
Heterogeneous scheduling allows workloads to be matched to the most
appropriate GPU. Large requests can be routed to higher-capacity GPUs
while smaller requests use lower-cost resources, improving utilization
and preserving premium capacity.

Latency: ~5.9s
Model: qwen-outpost

Prompt:
Explain the CAP theorem and its practical implications for distributed database design.

Output:
Successful response returned from the CPU inference tier.

Latency: ~24.1s
Model: qwen-cpu

Prompt:
What monitoring ports does the platform expose, and what does each one serve?

Output (excerpt):
The platform identified externally exposed monitoring endpoints,
HTTPS services, and private connectivity paths using information
retrieved from the RAG corpus.

Latency: ~5.5s
Model: infra-rag

Prompt:
What vulnerabilities are currently flagged as actively exploited on this platform?

Output:
The platform identified matches against known exploited vulnerability
feeds and generated remediation guidance. Exact CVE identifiers,
affected versions, and package fingerprints are intentionally omitted.

Latency: ~12.2s
Model: infra-rag

Prompt:
Latest developments in open-source LLM quantization techniques

Output (excerpt):
The platform combined web search results with an LLM summary covering:
• INT8 and INT4 quantization
• AWQ and GPTQ approaches
• VRAM optimization strategies
• Recent open-weight model developments

Latency: ~3.5s
Model: qwen-gpu + SearXNG

Scenario:
Primary target unavailable.

Result:
Request automatically routed through configured fallback chain and
completed successfully.

Latency: ~5.7s
Requested model: qwen-outpost

qwen-gpu
qwen-outpost
qwen-cpu
infra-rag

kube-vip:vx.x.x
nvidia-device-plugin:vx.xx.x
litellm:vx.xx.xx
ollama:latest
vllm-openai:nightly
qdrant:vx.xx.x
argocd:vx.x.x
redis:x.x.x-alpine
coredns:x.xx.x
traefik:x.x.x

Host packages:
git x:x.xx.x
openssl x.x.x
sudo x.x.x
nvidia-container-toolkit x.xx.x