Add smart_image_gen Tool for per-prompt checkpoint routing

Open WebUI Tool the LLM invokes instead of the built-in image action.
Auto-routes among the seven SDXL checkpoints (photo / juggernaut /
pony / general / furry-{nai,noob,il}) based on either an explicit
`style` arg or first-match-wins regex over the prompt. Constructs the
ComfyUI workflow inline, submits via /prompt, polls /history, returns
the result as a base64 data-URI markdown image so no extra hosting is
needed. Per-style default negatives. ComfyUI URL / steps / CFG /
timeout are admin-tunable Valves.

Filters can't see image-gen requests in Open WebUI (the routers skip
the filter chain), so the LLM-driven Tool is the only path that
gives intent-aware routing without changing the chat UX.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
57_WolveandClaude Opus 4.7 committed 2026-04-19 12:17:02 -05:00
1 parent 704bcfdf13
commit 392b26167f
2 files changed
+258 -7

No files matched your search

+41 -7
View File
@@ -10,13 +10,14 @@ production `srvno.de` deployment.
## Files
| File | Purpose |
| -------------------------- | -------------------------------------------------------- |
| `docker-compose.yml` | Service definitions, volumes, GPU reservations |
| `Caddyfile` | TLS + reverse proxy config (one site block per hostname) |
| `init-models.sh` | LLMs to preseed into Ollama on first boot |
| `comfyui-init-models.sh` | Checkpoints/VAEs/LoRAs to preseed into ComfyUI on first boot |
| `.env.example` | Secrets and image-tag pins. Copy to `.env` |
| File | Purpose |
| --------------------------------------- | -------------------------------------------------------- |
| `docker-compose.yml` | Service definitions, volumes, GPU reservations |
| `Caddyfile` | TLS + reverse proxy config (one site block per hostname) |
| `init-models.sh` | LLMs to preseed into Ollama on first boot |
| `comfyui-init-models.sh` | Checkpoints/VAEs/LoRAs to preseed into ComfyUI on first boot |
| `openwebui-tools/smart_image_gen.py` | Tool that auto-routes image generation to the right SDXL checkpoint |
| `.env.example` | Secrets and image-tag pins. Copy to `.env` |
## 1. Host prerequisites
@@ -146,6 +147,39 @@ Open WebUI submits the workflow to ComfyUI; the result drops back into
the chat when KSampler finishes. To test img2img, attach an image and
use the edit action.
## 8. (Optional) Install the smart-routing Tool
The image-button path always uses the admin's **Default Model**. To get
per-prompt checkpoint routing — e.g. "draw me a cyberpunk city" picks
CyberRealistic, "anthro fox warrior" picks one of the furry checkpoints —
install the `smart_image_gen.py` Tool. The LLM calls it instead of the
built-in image action and chooses the right SDXL checkpoint per request.
1. **Workspace -> Tools -> +** (top-right).
2. Paste the contents of
[`openwebui-tools/smart_image_gen.py`](openwebui-tools/smart_image_gen.py).
3. Save. Optionally adjust the Valves (ComfyUI URL, default steps, CFG,
timeout) via the gear icon.
4. **Workspace -> Models** (or pick an existing chat model) -> edit ->
under **Tools**, enable `smart_image_gen` -> save.
5. Make sure the model has **native function calling** enabled
(Workspace -> Models -> the model -> Advanced Params -> Function
Calling: Native). Mistral, Qwen, and Llama 3.1+ all support this.
In a chat with that model, ask for an image — "make me a photoreal
portrait of a cyberpunk samurai" — the LLM should call
`generate_image(prompt=..., style="photo")`. The status bar shows
"Routing to photo (CyberRealisticXLPlay…)" while it generates.
To extend (new checkpoint, new style):
- Add the filename to `comfyui-init-models.sh` so it gets pulled.
- Add a key to the `CHECKPOINTS` dict in `smart_image_gen.py`.
- Optionally add style-specific negatives to `NEGATIVES`.
- Optionally add keyword routing rules to `ROUTING_RULES` for the
auto-detect path.
- Re-paste the Tool source in Workspace -> Tools.
## Enabling Anubis (later)
The `anubis-owui` service is defined in compose but no Caddy site block