The Self-Hosted AI Tier: Private Inference with Ollama, OpenWebUI, and ComfyUI
August 26, 2026
I wanted a private ChatGPT and a private image generator, a place to ask questions and generate images where not one token leaves my infrastructure. The same instinct that makes me self-host search makes me self-host inference: if the queries are mine, the model serving them should be mine too. This tier is where the lab spends its heaviest compute, and it is built around one rule: one clean front door, and everything expensive behind it.
Why run models locally at all
Three reasons, in order. Privacy: prompts, documents, and generated output stay inside the lab, full stop, with no third-party retention policy to trust. Control: I pick the models, the versions, and when they change, instead of waking up to a silently "improved" endpoint. Cost: once the hardware exists, inference is just electricity, and I already own the hardware.
The trade is that I am now the capacity planner. That constraint shaped every placement decision below.
Architecture
┌─────────────────────────────┐
client ──HTTPS──▶ reverse proxy ──▶ │ OpenWebUI (the front door) │
└───────┬──────────────┬──────┘
│ OpenAI-compat │ workflows
▼ ▼
┌────────────┐ ┌──────────────┐
│ Ollama │ │ ComfyUI ×2 │
│ LLM serving│ │ diffusion │
└────────────┘ └──────────────┘
│
▼
┌────────────┐
│ Qwen3-TTS │ speech synthesis
└────────────┘
heavy compute pinned to the two beefiest nodes:
a Ryzen 9 5950X (32 threads) and a 36-core Xeon E5-2697 v4 (256GB RAM)Everything a user touches goes through OpenWebUI. Behind it, Ollama serves language models, two ComfyUI instances handle image generation, and Qwen3-TTS does speech. The expensive workloads are deliberately concentrated on the two most capable nodes in the cluster.
Ollama: the model server
Ollama is the engine room. It pulls and manages models, keeps them warm, and, most usefully, exposes an OpenAI-compatible API, which means anything that speaks the OpenAI protocol can point at it with a base-URL change and nothing else. It runs as a container on the Ryzen 9 node, where 32 threads of Zen 3 do the heavy lifting.
# ollama environment (excerpt): expose on the internal network, keep models warm
OLLAMA_HOST=0.0.0.0:11434
OLLAMA_KEEP_ALIVE=30m # don't reload weights between every message
OLLAMA_MAX_LOADED_MODELS=2 # cap concurrent models so RAM doesn't blow up
OLLAMA_NUM_PARALLEL=2By default Ollama binds 127.0.0.1, so it has to be bound to the internal address (as above) before OpenWebUI on another container can reach it. From there the reverse proxy and firewall, not the bind, are what control who gets to it.
OpenWebUI: the front door
OpenWebUI is the single interface over everything: chat, model switching, multi-user accounts, and document RAG. It is the heaviest-provisioned container in this tier for a reason (it runs on the 36-core Xeon node, which has 256GB of RAM): it is doing embeddings and retrieval on top of proxying chat.
# OpenWebUI environment (excerpt): point the front door at Ollama, keep it internal
OLLAMA_BASE_URL=http://ollama.internal:11434
WEBUI_AUTH=true # accounts on, even internally
ENABLE_SIGNUP=false # I add users; the internet does notOnly OpenWebUI is published, behind the reverse proxy and authentication; Ollama and ComfyUI stay reachable only from inside the tier. That is the point of a single front door rather than three: an unauthenticated inference endpoint left on the network is a free GPU for whoever finds it first.
ComfyUI: diffusion on demand
Image generation lives in two separate ComfyUI containers, each provisioned generously on the Ryzen node. Two instances, not one, so I can pin different model sets and workflows without them fighting over the same memory: one can be mid-render on a big SDXL graph while the other stays clean for quick jobs.
They are the only guests in this tier I run stopped by default. Diffusion is bursty and heavy, and there is no reason to hold that much allocation hostage 24/7 for something I use in sessions. I spin them up when I need them and shut them down after. Left running but unused, two large containers make the cluster look full when it is idle, which is exactly the high RAM allocation with no usage data that the posture scanner flags, and the nudge to keep on-demand workloads actually off when they are off.
Qwen3-TTS: giving it a voice
Qwen3-TTS rounds out the tier with local text-to-speech, so the "assistant" can talk without a cloud voice API in the loop. Same principle as the rest: the audio is generated in the lab and stays there.
Why it's designed this way
- One front door. OpenWebUI is the only published surface. Every model backend sits behind it on the internal network, so there is exactly one thing to authenticate and one thing to watch.
- Compute follows the workload. LLM and diffusion land on the Ryzen 9 and the 36-core Xeon because that is where the threads and RAM are. The small low-power nodes run DNS and telemetry, not transformers.
- On-demand beats always-on for the heavy stuff. Chat is cheap enough to keep warm; diffusion is not, so it stays stopped until asked for.
- Privacy is the whole point. The tier exists so that prompts, documents, and outputs never become someone else's training data.
What's next
- GPU passthrough. Inference is CPU-bound today, and the clear upgrade is passing a discrete GPU into the Ollama and ComfyUI containers for an order-of-magnitude speedup on both token generation and diffusion steps.
- Shared model storage. Pull each model once to a shared volume instead of duplicating weights per container.
- Wire the tier into the lab's own assistant. The inventory platform already runs a cluster-aware assistant, so pointing internal tooling at the self-hosted Ollama endpoint instead of a hosted API is the natural convergence.