User saw the LLM's chat response include a literal 'edit_image(prompt="...", mask_text="...", style="furry-il", denoise=0.85)' line after the image rendered — Default function- calling mode tends to make the model 'narrate' its tool call by re-typing it as Python-style syntax. Added an explicit NEVER block: no echoing the call, no JSON, no listing arguments, no enumerating styles/denoise/mask_text. The same info is in the collapsible 'View Result from edit_image' block that Open WebUI renders alongside the message — there's no need for the LLM to also paste it as prose. Follow-up text is for human conversation, not bookkeeping. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
239 lines
10 KiB
Markdown
239 lines
10 KiB
Markdown
# Image Studio — dedicated image-generation chat model
|
||
|
||
A custom Open WebUI model preset that wraps a base LLM with a system
|
||
prompt heavily biased toward calling the `smart_image_gen` tool. Users
|
||
pick **Image Studio** from the chat-model dropdown when they want to
|
||
generate or edit images, and the LLM treats every message as an image
|
||
request — calling `generate_image` for new images and `edit_image` for
|
||
modifications to attached ones.
|
||
|
||
This exists because general-purpose chat models often "describe" an
|
||
image in text instead of calling the tool, especially when the request
|
||
is conversational ("can you draw me…", "I'd like a picture of…"). A
|
||
dedicated preset removes the ambiguity.
|
||
|
||
## Two ways to install
|
||
|
||
### Option A: Import the JSON (fast)
|
||
|
||
Workspace → Models → **Import** (top right) → upload
|
||
[`image_studio.json`](image_studio.json).
|
||
|
||
This drops the preset in fully configured: base model, system prompt,
|
||
tool attachment, function-calling mode, temperature, suggestion
|
||
prompts. Verify after import:
|
||
|
||
- The `smart_image_gen` tool is actually attached (Tools list under the
|
||
model's edit screen). If not, the tool ID Open WebUI assigned doesn't
|
||
match the `toolIds: ["smart_image_gen"]` in the JSON — re-attach
|
||
manually.
|
||
- Base Model is set to `mistral-nemo:12b`. Adjust if you want a
|
||
different LLM (Qwen3.6 or Llama 3.1 also work well; smaller
|
||
parameter counts may struggle with native tool calling).
|
||
|
||
### Option B: Create manually (table below)
|
||
|
||
**Workspace → Models → +** (top right).
|
||
|
||
| Field | Value |
|
||
| ----- | ----- |
|
||
| Name | `Image Studio` |
|
||
| Base Model | `huihui_ai/qwen3.5-abliterated:9b` (vision-capable, 256K context, abliterated). Pull via `init-models.sh` first. |
|
||
| Description | `Image generation and routing across SDXL checkpoints.` |
|
||
| System Prompt | Paste the block from [System prompt](#system-prompt) below. |
|
||
| Tools | enable **only** `smart_image_gen` |
|
||
|
||
In the **Advanced Params** section:
|
||
|
||
| Field | Value |
|
||
| ----- | ----- |
|
||
| Function Calling | `Default` — `Native` leaks Qwen 3.5's tool-call XML to chat as text instead of executing it. `Default` uses Open WebUI's own prompt-injection wrapper, which the parser reliably handles for any base model. Use `Native` only if you've swapped the base model to one with proven Open-WebUI-side parser support (e.g. `mistral-nemo:12b`). |
|
||
| Temperature | `0.5` (lower = more reliable tool-calling) |
|
||
| Top P | `0.9` |
|
||
| Context Length | leave default |
|
||
| Custom Parameters | `tool_choice: required` (forces the model to call a tool — bypasses planning behaviour on stubborn models like the abliterated Qwen 3.5) |
|
||
|
||
Save. The new model appears in the chat-model dropdown for any user with
|
||
access.
|
||
|
||
## System prompt
|
||
|
||
```
|
||
/no_think
|
||
|
||
You are an image-tool dispatcher. You do not respond in prose. Every
|
||
user message MUST result in exactly one tool call.
|
||
|
||
ROUTING:
|
||
- If the user attached an image (including images you previously
|
||
generated in this chat) → call edit_image(prompt=..., ...)
|
||
- Otherwise → call generate_image(prompt=..., ...)
|
||
Both tools take `prompt` as the first argument — same name on both.
|
||
Do NOT invent `edit_instruction`.
|
||
|
||
Fire the tool on the FIRST message, with no preamble. Do not write a
|
||
'plan', 'approach', 'steps', 'breakdown', or any explanation before
|
||
calling. Do not ask clarifying questions. Do not say what you are
|
||
about to do. If the request is vague, pick reasonable defaults and
|
||
call the tool — the user iterates after.
|
||
|
||
STYLES (pick one):
|
||
photo photorealistic photo / portrait / cinematic
|
||
juggernaut alternate photoreal — sharper, more saturated
|
||
pony anime, cartoon, manga, stylised illustration
|
||
general catch-all when nothing else fits
|
||
furry-nai anthropomorphic, NAI-trained mix
|
||
furry-noob anthropomorphic, NoobAI base
|
||
furry-il anthropomorphic, Illustrious base (default for any
|
||
furry/anthro request)
|
||
|
||
STYLE FOR edit_image — pick in this order:
|
||
|
||
- The image was generated by you earlier in this chat → omit `style`,
|
||
the tool auto-inherits from the previous call.
|
||
- The user just UPLOADED an image → look at it and pick the style
|
||
that matches what you see (anthropomorphic furry/scaly/feathered
|
||
character → furry-il, pony score-tag art → pony, photo / portrait
|
||
→ photo or juggernaut, anime → pony, ambiguous → general). Keep
|
||
using that style for subsequent edits in the same chat.
|
||
- Always pick for the DESIRED OUTPUT, but for normal edits the
|
||
desired output IS the input style — only override when the user
|
||
explicitly wants a style change ("turn this anime into a photo").
|
||
|
||
edit_image has TWO MODES — pick based on whether the change is local
|
||
or global:
|
||
|
||
- LOCAL ("change the ball to a basketball", "add a hat to the dog",
|
||
"remove the bird", "recolor the car red") → set `mask_text` to a
|
||
brief noun phrase naming the region ("the ball", "the dog", "the
|
||
bird", "the car"). Only that region is repainted; rest stays
|
||
pixel-perfect.
|
||
- GLOBAL ("make this a sunset", "turn this into anime", "restyle as
|
||
oil painting") → leave mask_text unset. The whole image is
|
||
reimagined.
|
||
|
||
ALWAYS prefer LOCAL when the user names a specific object, person,
|
||
or region. GLOBAL is only for whole-image style/lighting
|
||
transformations.
|
||
|
||
Denoise:
|
||
- LOCAL (mask_text set): default 1.0. Drop to 0.6–0.8 only for
|
||
subtle local edits that should retain some original structure.
|
||
- GLOBAL (no mask_text): default 0.7. Use 0.3–0.5 for subtle
|
||
restyle, 0.85–1.0 for radical reimagining.
|
||
|
||
Pick style for the DESIRED OUTPUT, not the input image.
|
||
|
||
Write rich, descriptive prompts (subject, action, environment,
|
||
lighting, mood, framing). Do NOT add quality tags like 'masterpiece',
|
||
'best quality', 'score_9', 'absurdres' — the tool prepends the
|
||
correct tags per style. Do NOT set sampler, CFG, steps, scheduler —
|
||
the tool picks them.
|
||
|
||
AFTER the tool returns, write at most one short PLAIN-ENGLISH
|
||
sentence noting your style/mode choice and offering one iteration
|
||
idea. The image is already shown to the user.
|
||
|
||
NEVER, after the tool returns:
|
||
- echo or repeat the tool call (no `edit_image(prompt=..., ...)`,
|
||
no `<function=...>`, no JSON, no parameter listings)
|
||
- describe what's in the image
|
||
- list the arguments you used
|
||
- enumerate styles, denoise, mask_text, etc.
|
||
Those details are visible in the collapsible 'View Result from
|
||
edit_image' tool-result block — the user can expand it if they
|
||
care. Your follow-up message is for HUMAN conversation, not
|
||
bookkeeping.
|
||
```
|
||
|
||
The first line `/no_think` disables Qwen 3.x's reasoning phase. If
|
||
your base model isn't Qwen 3, leaving it in is a no-op (other models
|
||
ignore it). Drop it only if it actually causes problems.
|
||
|
||
## Set a separate Task Model (required after install)
|
||
|
||
`tool_choice: required` is what makes Image Studio reliably fire the
|
||
tool, but it has a side effect: Open WebUI uses the same model with
|
||
the same params for **title generation**, **tag generation**, and
|
||
**autocomplete**. With every response forced to be a tool call, those
|
||
text-only background tasks can't produce text, so chats stay named
|
||
"New Chat" forever and tag suggestions go silent.
|
||
|
||
Fix: point Open WebUI at a different model for those tasks.
|
||
|
||
**Admin Settings → Interface → Task Model** → pick any of the
|
||
non-Image-Studio models you have pulled. `mistral-nemo:12b`,
|
||
`llama3.1:8b`, `qwen3.6:latest`, or `dolphin3:8b` all work. The Task
|
||
Model only handles short background calls (titles, tags, autocomplete,
|
||
search-query rewriting) — it doesn't need to be vision-capable or
|
||
particularly large. Smaller is faster and cheaper.
|
||
|
||
Save. New Image Studio chats now get descriptive titles, tag
|
||
suggestions return, and autocomplete lights up.
|
||
|
||
## Vision capability
|
||
|
||
The shipped preset sets `meta.capabilities.vision: true` so Open WebUI
|
||
allows users to attach images to chats with this model. Two paths:
|
||
|
||
### Default — `huihui_ai/qwen3.5-abliterated:9b`
|
||
|
||
The shipped preset uses Qwen 3.5 abliterated 9B as the base — vision-
|
||
capable, 256K context, no censorship hedging. Preseed via
|
||
`init-models.sh` (an `s3_pull` line is already in place; see
|
||
[Mirroring models to S3](../README.md#mirroring-models-to-s3) for the
|
||
mirror workflow).
|
||
|
||
**Important Qwen 3.x quirk:** thinking mode is on by default and
|
||
breaks native function calling — the model "thinks" about how to use
|
||
the tool instead of just calling it. The shipped system prompt starts
|
||
with `/no_think` to suppress this. If the model still plans instead
|
||
of firing the tool, also set `enable_thinking: false` in **Advanced
|
||
Params → Custom Parameters** (API-level enforcement).
|
||
|
||
### Alternatives
|
||
|
||
If Qwen 3.5 isn't a fit (size, language preferences, abliteration
|
||
caveats), other vision-capable Ollama tags worth trying:
|
||
|
||
- `qwen2.5vl:7b` — smaller, no thinking mode, very reliable tool-caller
|
||
- `llama3.2-vision:11b` — Meta's vision variant, ~7 GB
|
||
- `minicpm-v:8b` — fast, capable
|
||
|
||
To swap, change `base_model_id` in `image_studio.json` (or the Base
|
||
Model field if you imported manually) and pull the model via
|
||
`init-models.sh` or the Open WebUI model UI.
|
||
|
||
### Non-vision base model
|
||
|
||
If you'd rather use a text-only LLM (e.g. `mistral-nemo:12b`),
|
||
keep `vision: true` in the preset so Open WebUI still permits image
|
||
attachments; the image flows through to `edit_image` via
|
||
`__messages__` / `__files__` and ComfyUI does the visual work. The
|
||
LLM can't see the image, but for explicit edit instructions ("change
|
||
the background to a sunset") that doesn't matter.
|
||
|
||
## Why this works when a generic chat model didn't
|
||
|
||
- **The system prompt is unambiguous.** No room for the model to
|
||
decide "I'll just describe it in text instead."
|
||
- **Only one tool is attached.** No competing tools to choose between.
|
||
- **Function Calling: Default** is the safer choice for Qwen 3.x
|
||
abliterated. Native mode expects the parser to recognise the
|
||
model's structured tool-call format, which currently leaks Qwen
|
||
3.5's `<function=...><parameter=...>` XML to chat as plain text on
|
||
the published Open WebUI / Ollama versions. Default mode uses Open
|
||
WebUI's own prompt-injection wrapper that round-trips reliably.
|
||
Try Native only after swapping the base model to one known to work
|
||
end-to-end (mistral-nemo, qwen2.5vl).
|
||
- **Lower temperature.** Tool calling is more reliable with less
|
||
sampling randomness.
|
||
|
||
## Iterating on the system prompt
|
||
|
||
If users ask for things you didn't anticipate (specific aspect ratios,
|
||
multi-image batches, particular checkpoints not in the routing rules),
|
||
edit the system prompt above and re-paste into the Workspace → Models
|
||
entry. It's the highest-leverage place to tune behaviour without
|
||
touching the Tool's Python.
|