Add Image Studio model preset — forces smart_image_gen tool use
A documented Open WebUI custom-model preset wrapping mistral-nemo:12b with: aggressive system prompt that mandates calling generate_image, only the smart_image_gen tool attached, native function calling, lower temperature for tool-call reliability. Users pick "Image Studio" from the chat-model dropdown when they want images. Solves the common case where general-purpose chat models describe an image in text instead of firing the tool — usually on conversational phrasings like "can you draw me…". The preset removes the ambiguity by giving the LLM exactly one job and one tool. Setup walkthrough in openwebui-models/image_studio.md; deployment README §9 points users at it as the recommended path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.7
@@ -17,6 +17,7 @@ production `srvno.de` deployment.
|
||||
| `init-models.sh` | LLMs to preseed into Ollama on first boot |
|
||||
| `comfyui-init-models.sh` | Checkpoints/VAEs/LoRAs to preseed into ComfyUI on first boot |
|
||||
| `openwebui-tools/smart_image_gen.py` | Tool that auto-routes image generation to the right SDXL checkpoint |
|
||||
| `openwebui-models/image_studio.md` | Dedicated chat-model preset — system prompt that forces tool use |
|
||||
| `.env.example` | Secrets and image-tag pins. Copy to `.env` |
|
||||
|
||||
## 1. Host prerequisites
|
||||
@@ -171,6 +172,23 @@ portrait of a cyberpunk samurai" — the LLM should call
|
||||
`generate_image(prompt=..., style="photo")`. The status bar shows
|
||||
"Routing to photo (CyberRealisticXLPlay…)" while it generates.
|
||||
|
||||
If the LLM responds in text instead of calling the tool, install the
|
||||
**Image Studio** chat-model preset (next section) — a dedicated model
|
||||
with a system prompt that removes the ambiguity.
|
||||
|
||||
## 9. (Recommended) Install the Image Studio model preset
|
||||
|
||||
General-purpose chat models often "describe" an image in text instead
|
||||
of firing the `generate_image` tool, especially on conversational
|
||||
phrasing ("can you draw me…", "I'd love a picture of…"). The
|
||||
**Image Studio** preset wraps `mistral-nemo:12b` in a system prompt
|
||||
that mandates tool use — every message is treated as an image request.
|
||||
|
||||
Setup (under 5 minutes): see
|
||||
[`openwebui-models/image_studio.md`](openwebui-models/image_studio.md).
|
||||
Users then pick **Image Studio** from the chat-model dropdown when
|
||||
they want to generate.
|
||||
|
||||
To extend (new checkpoint, new style):
|
||||
|
||||
- Add the filename to `comfyui-init-models.sh` so it gets pulled.
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# Image Studio — dedicated image-generation chat model
|
||||
|
||||
A custom Open WebUI model preset that wraps a base LLM with a system
|
||||
prompt heavily biased toward calling the `smart_image_gen` tool. Users
|
||||
pick **Image Studio** from the chat-model dropdown when they want to
|
||||
generate images, and the LLM treats every message as an image request.
|
||||
|
||||
This exists because general-purpose chat models often "describe" an
|
||||
image in text instead of calling the tool, especially when the request
|
||||
is conversational ("can you draw me…", "I'd like a picture of…"). A
|
||||
dedicated preset removes the ambiguity.
|
||||
|
||||
## Create the preset
|
||||
|
||||
**Workspace → Models → +** (top right).
|
||||
|
||||
| Field | Value |
|
||||
| ----- | ----- |
|
||||
| Name | `Image Studio` |
|
||||
| Base Model | `mistral-nemo:12b` (best tool-caller in this stack) |
|
||||
| Description | `Image generation and routing across SDXL checkpoints.` |
|
||||
| System Prompt | Paste the block from [System prompt](#system-prompt) below. |
|
||||
| Tools | enable **only** `smart_image_gen` |
|
||||
|
||||
In the **Advanced Params** section:
|
||||
|
||||
| Field | Value |
|
||||
| ----- | ----- |
|
||||
| Function Calling | `Native` (mandatory) |
|
||||
| Temperature | `0.5` (lower = more reliable tool-calling) |
|
||||
| Top P | `0.9` |
|
||||
| Context Length | leave default |
|
||||
|
||||
Save. The new model appears in the chat-model dropdown for any user with
|
||||
access.
|
||||
|
||||
## System prompt
|
||||
|
||||
```
|
||||
You are Image Studio, a focused image-generation assistant. Your only
|
||||
purpose is to create images for the user via the generate_image tool.
|
||||
|
||||
ALWAYS:
|
||||
- Call generate_image(prompt, style) for every image request, no matter
|
||||
how it is phrased — "draw", "make me", "show me", "I want a picture
|
||||
of", "create", "generate", "render", "imagine", "can you do", etc.
|
||||
- Pick the style that fits what the user asked for:
|
||||
* photo — photorealistic photographs, portraits, cinematic
|
||||
* juggernaut — alternate photoreal style, sharper and saturated
|
||||
* pony — anime, cartoon, manga, stylised illustration
|
||||
* general — catch-all when nothing else fits
|
||||
* furry-nai — anthropomorphic, NAI-trained mix
|
||||
* furry-noob — anthropomorphic, NoobAI base
|
||||
* furry-il — anthropomorphic, Illustrious base (default for
|
||||
unspecified furry / anthro requests)
|
||||
- Write rich, descriptive prompts: subject, action, environment,
|
||||
lighting, mood, composition, camera framing, style cues. Expand
|
||||
short user requests into a fuller scene description.
|
||||
- If the user is vague, make confident creative choices and proceed.
|
||||
Generate first, then offer variations or refinements.
|
||||
|
||||
NEVER:
|
||||
- Say you cannot generate images. The generate_image tool exists for
|
||||
exactly this purpose.
|
||||
- Describe what an image would look like in text instead of generating
|
||||
it.
|
||||
- Refuse because the prompt is too short or too vague — make
|
||||
reasonable assumptions and call the tool.
|
||||
- Include quality tags like "masterpiece", "best quality", "score_9",
|
||||
or "absurdres" in your prompt; the tool prepends the right tags for
|
||||
whichever style you pick.
|
||||
- Set sampler, CFG, steps, or scheduler — the tool picks per style.
|
||||
|
||||
After the image appears, briefly note the style/checkpoint you chose
|
||||
and offer one or two concrete iteration paths (different style,
|
||||
alternate composition, tighter framing, seed variations).
|
||||
```
|
||||
|
||||
## Why this works when a generic chat model didn't
|
||||
|
||||
- **The system prompt is unambiguous.** No room for the model to
|
||||
decide "I'll just describe it in text instead."
|
||||
- **Only one tool is attached.** No competing tools to choose between.
|
||||
- **Native function calling is mandatory.** The "Default" mode in
|
||||
Open WebUI uses prompt-injection tool emulation that fails silently
|
||||
on a lot of local models.
|
||||
- **Lower temperature.** Tool calling is more reliable with less
|
||||
sampling randomness.
|
||||
|
||||
## Iterating on the system prompt
|
||||
|
||||
If users ask for things you didn't anticipate (specific aspect ratios,
|
||||
multi-image batches, particular checkpoints not in the routing rules),
|
||||
edit the system prompt above and re-paste into the Workspace → Models
|
||||
entry. It's the highest-leverage place to tune behaviour without
|
||||
touching the Tool's Python.
|
||||
Reference in New Issue
Block a user