diff --git a/deployments/ai-stack/README.md b/deployments/ai-stack/README.md index 4fece4b..65979cc 100644 --- a/deployments/ai-stack/README.md +++ b/deployments/ai-stack/README.md @@ -17,6 +17,7 @@ production `srvno.de` deployment. | `init-models.sh` | LLMs to preseed into Ollama on first boot | | `comfyui-init-models.sh` | Checkpoints/VAEs/LoRAs to preseed into ComfyUI on first boot | | `openwebui-tools/smart_image_gen.py` | Tool that auto-routes image generation to the right SDXL checkpoint | +| `openwebui-models/image_studio.md` | Dedicated chat-model preset — system prompt that forces tool use | | `.env.example` | Secrets and image-tag pins. Copy to `.env` | ## 1. Host prerequisites @@ -171,6 +172,23 @@ portrait of a cyberpunk samurai" — the LLM should call `generate_image(prompt=..., style="photo")`. The status bar shows "Routing to photo (CyberRealisticXLPlay…)" while it generates. +If the LLM responds in text instead of calling the tool, install the +**Image Studio** chat-model preset (next section) — a dedicated model +with a system prompt that removes the ambiguity. + +## 9. (Recommended) Install the Image Studio model preset + +General-purpose chat models often "describe" an image in text instead +of firing the `generate_image` tool, especially on conversational +phrasing ("can you draw me…", "I'd love a picture of…"). The +**Image Studio** preset wraps `mistral-nemo:12b` in a system prompt +that mandates tool use — every message is treated as an image request. + +Setup (under 5 minutes): see +[`openwebui-models/image_studio.md`](openwebui-models/image_studio.md). +Users then pick **Image Studio** from the chat-model dropdown when +they want to generate. + To extend (new checkpoint, new style): - Add the filename to `comfyui-init-models.sh` so it gets pulled. diff --git a/deployments/ai-stack/openwebui-models/image_studio.md b/deployments/ai-stack/openwebui-models/image_studio.md new file mode 100644 index 0000000..cb51b10 --- /dev/null +++ b/deployments/ai-stack/openwebui-models/image_studio.md @@ -0,0 +1,96 @@ +# Image Studio — dedicated image-generation chat model + +A custom Open WebUI model preset that wraps a base LLM with a system +prompt heavily biased toward calling the `smart_image_gen` tool. Users +pick **Image Studio** from the chat-model dropdown when they want to +generate images, and the LLM treats every message as an image request. + +This exists because general-purpose chat models often "describe" an +image in text instead of calling the tool, especially when the request +is conversational ("can you draw me…", "I'd like a picture of…"). A +dedicated preset removes the ambiguity. + +## Create the preset + +**Workspace → Models → +** (top right). + +| Field | Value | +| ----- | ----- | +| Name | `Image Studio` | +| Base Model | `mistral-nemo:12b` (best tool-caller in this stack) | +| Description | `Image generation and routing across SDXL checkpoints.` | +| System Prompt | Paste the block from [System prompt](#system-prompt) below. | +| Tools | enable **only** `smart_image_gen` | + +In the **Advanced Params** section: + +| Field | Value | +| ----- | ----- | +| Function Calling | `Native` (mandatory) | +| Temperature | `0.5` (lower = more reliable tool-calling) | +| Top P | `0.9` | +| Context Length | leave default | + +Save. The new model appears in the chat-model dropdown for any user with +access. + +## System prompt + +``` +You are Image Studio, a focused image-generation assistant. Your only +purpose is to create images for the user via the generate_image tool. + +ALWAYS: +- Call generate_image(prompt, style) for every image request, no matter + how it is phrased — "draw", "make me", "show me", "I want a picture + of", "create", "generate", "render", "imagine", "can you do", etc. +- Pick the style that fits what the user asked for: + * photo — photorealistic photographs, portraits, cinematic + * juggernaut — alternate photoreal style, sharper and saturated + * pony — anime, cartoon, manga, stylised illustration + * general — catch-all when nothing else fits + * furry-nai — anthropomorphic, NAI-trained mix + * furry-noob — anthropomorphic, NoobAI base + * furry-il — anthropomorphic, Illustrious base (default for + unspecified furry / anthro requests) +- Write rich, descriptive prompts: subject, action, environment, + lighting, mood, composition, camera framing, style cues. Expand + short user requests into a fuller scene description. +- If the user is vague, make confident creative choices and proceed. + Generate first, then offer variations or refinements. + +NEVER: +- Say you cannot generate images. The generate_image tool exists for + exactly this purpose. +- Describe what an image would look like in text instead of generating + it. +- Refuse because the prompt is too short or too vague — make + reasonable assumptions and call the tool. +- Include quality tags like "masterpiece", "best quality", "score_9", + or "absurdres" in your prompt; the tool prepends the right tags for + whichever style you pick. +- Set sampler, CFG, steps, or scheduler — the tool picks per style. + +After the image appears, briefly note the style/checkpoint you chose +and offer one or two concrete iteration paths (different style, +alternate composition, tighter framing, seed variations). +``` + +## Why this works when a generic chat model didn't + +- **The system prompt is unambiguous.** No room for the model to + decide "I'll just describe it in text instead." +- **Only one tool is attached.** No competing tools to choose between. +- **Native function calling is mandatory.** The "Default" mode in + Open WebUI uses prompt-injection tool emulation that fails silently + on a lot of local models. +- **Lower temperature.** Tool calling is more reliable with less + sampling randomness. + +## Iterating on the system prompt + +If users ask for things you didn't anticipate (specific aspect ratios, +multi-image batches, particular checkpoints not in the routing rules), +edit the system prompt above and re-paste into the Workspace → Models +entry. It's the highest-leverage place to tune behaviour without +touching the Tool's Python.