57_WolveandClaude Opus 4.7
d935e24624
release / Build & Push Docker Image (push) Successful in 44s
Five pieces:
1. Dockerfile installs storyicon/comfyui_segment_anything (GroundingDINO
+ SAM-HQ in one bundle) into custom_nodes and pip-installs its
requirements at build time. Model weights auto-download to the
comfyui-models volume on first inpaint (~3 GB one-time cost).
2. install-custom-node-deps.sh — entrypoint wrapper that pip-installs
requirements.txt for any custom_node present at startup. Lets users
add custom nodes via ComfyUI-Manager (or by git-cloning into the
volume) and have the deps picked up on the next restart, without
editing the Dockerfile.
3. smart_image_gen v0.6: edit_image gains a `mask_text` param. When
set, builds an inpainting workflow (LoadImage → GroundingDinoSAM
Segment → SetLatentNoiseMask → KSampler) so only the named region
is repainted. When unset, falls through to the existing img2img
path. Denoise default switches: 1.0 with mask_text (full repaint
within mask), 0.7 without.
4. Image Studio system prompt teaches the LLM the LOCAL vs GLOBAL
distinction — set mask_text whenever the user names a specific
object/region ('the ball', 'the dog', 'the sky'); leave it unset
only for whole-image style/lighting transformations.
5. Deployment README documents the new mode + the first-inpaint
weight-download caveat.
Image rebuild required — bump tag to pick up the Dockerfile change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
186 lines
7.5 KiB
Markdown
186 lines
7.5 KiB
Markdown
# Image Studio — dedicated image-generation chat model
|
||
|
||
A custom Open WebUI model preset that wraps a base LLM with a system
|
||
prompt heavily biased toward calling the `smart_image_gen` tool. Users
|
||
pick **Image Studio** from the chat-model dropdown when they want to
|
||
generate or edit images, and the LLM treats every message as an image
|
||
request — calling `generate_image` for new images and `edit_image` for
|
||
modifications to attached ones.
|
||
|
||
This exists because general-purpose chat models often "describe" an
|
||
image in text instead of calling the tool, especially when the request
|
||
is conversational ("can you draw me…", "I'd like a picture of…"). A
|
||
dedicated preset removes the ambiguity.
|
||
|
||
## Two ways to install
|
||
|
||
### Option A: Import the JSON (fast)
|
||
|
||
Workspace → Models → **Import** (top right) → upload
|
||
[`image_studio.json`](image_studio.json).
|
||
|
||
This drops the preset in fully configured: base model, system prompt,
|
||
tool attachment, function-calling mode, temperature, suggestion
|
||
prompts. Verify after import:
|
||
|
||
- The `smart_image_gen` tool is actually attached (Tools list under the
|
||
model's edit screen). If not, the tool ID Open WebUI assigned doesn't
|
||
match the `toolIds: ["smart_image_gen"]` in the JSON — re-attach
|
||
manually.
|
||
- Base Model is set to `mistral-nemo:12b`. Adjust if you want a
|
||
different LLM (Qwen3.6 or Llama 3.1 also work well; smaller
|
||
parameter counts may struggle with native tool calling).
|
||
|
||
### Option B: Create manually (table below)
|
||
|
||
**Workspace → Models → +** (top right).
|
||
|
||
| Field | Value |
|
||
| ----- | ----- |
|
||
| Name | `Image Studio` |
|
||
| Base Model | `huihui_ai/qwen3.5-abliterated:9b` (vision-capable, 256K context, abliterated). Pull via `init-models.sh` first. |
|
||
| Description | `Image generation and routing across SDXL checkpoints.` |
|
||
| System Prompt | Paste the block from [System prompt](#system-prompt) below. |
|
||
| Tools | enable **only** `smart_image_gen` |
|
||
|
||
In the **Advanced Params** section:
|
||
|
||
| Field | Value |
|
||
| ----- | ----- |
|
||
| Function Calling | `Native` (mandatory) |
|
||
| Temperature | `0.5` (lower = more reliable tool-calling) |
|
||
| Top P | `0.9` |
|
||
| Context Length | leave default |
|
||
| Custom Parameters | `tool_choice: required` (forces the model to call a tool — bypasses planning behaviour on stubborn models like the abliterated Qwen 3.5) |
|
||
|
||
Save. The new model appears in the chat-model dropdown for any user with
|
||
access.
|
||
|
||
## System prompt
|
||
|
||
```
|
||
/no_think
|
||
|
||
You are an image-tool dispatcher. You do not respond in prose. Every
|
||
user message MUST result in exactly one tool call.
|
||
|
||
ROUTING:
|
||
- If the user attached an image → call edit_image
|
||
- Otherwise → call generate_image
|
||
|
||
Fire the tool on the FIRST message, with no preamble. Do not write a
|
||
'plan', 'approach', 'steps', 'breakdown', or any explanation before
|
||
calling. Do not ask clarifying questions. Do not say what you are
|
||
about to do. If the request is vague, pick reasonable defaults and
|
||
call the tool — the user iterates after.
|
||
|
||
STYLES (pick one):
|
||
photo photorealistic photo / portrait / cinematic
|
||
juggernaut alternate photoreal — sharper, more saturated
|
||
pony anime, cartoon, manga, stylised illustration
|
||
general catch-all when nothing else fits
|
||
furry-nai anthropomorphic, NAI-trained mix
|
||
furry-noob anthropomorphic, NoobAI base
|
||
furry-il anthropomorphic, Illustrious base (default for any
|
||
furry/anthro request)
|
||
|
||
edit_image has TWO MODES — pick based on whether the change is local
|
||
or global:
|
||
|
||
- LOCAL ("change the ball to a basketball", "add a hat to the dog",
|
||
"remove the bird", "recolor the car red") → set `mask_text` to a
|
||
brief noun phrase naming the region ("the ball", "the dog", "the
|
||
bird", "the car"). Only that region is repainted; rest stays
|
||
pixel-perfect.
|
||
- GLOBAL ("make this a sunset", "turn this into anime", "restyle as
|
||
oil painting") → leave mask_text unset. The whole image is
|
||
reimagined.
|
||
|
||
ALWAYS prefer LOCAL when the user names a specific object, person,
|
||
or region. GLOBAL is only for whole-image style/lighting
|
||
transformations.
|
||
|
||
Denoise:
|
||
- LOCAL (mask_text set): default 1.0. Drop to 0.6–0.8 only for
|
||
subtle local edits that should retain some original structure.
|
||
- GLOBAL (no mask_text): default 0.7. Use 0.3–0.5 for subtle
|
||
restyle, 0.85–1.0 for radical reimagining.
|
||
|
||
Pick style for the DESIRED OUTPUT, not the input image.
|
||
|
||
Write rich, descriptive prompts (subject, action, environment,
|
||
lighting, mood, framing). Do NOT add quality tags like 'masterpiece',
|
||
'best quality', 'score_9', 'absurdres' — the tool prepends the
|
||
correct tags per style. Do NOT set sampler, CFG, steps, scheduler —
|
||
the tool picks them.
|
||
|
||
AFTER the tool returns, write at most one short sentence noting your
|
||
style/mode choice and offering one iteration idea. The image is
|
||
already shown to the user; do not describe it.
|
||
```
|
||
|
||
The first line `/no_think` disables Qwen 3.x's reasoning phase. If
|
||
your base model isn't Qwen 3, leaving it in is a no-op (other models
|
||
ignore it). Drop it only if it actually causes problems.
|
||
|
||
## Vision capability
|
||
|
||
The shipped preset sets `meta.capabilities.vision: true` so Open WebUI
|
||
allows users to attach images to chats with this model. Two paths:
|
||
|
||
### Default — `huihui_ai/qwen3.5-abliterated:9b`
|
||
|
||
The shipped preset uses Qwen 3.5 abliterated 9B as the base — vision-
|
||
capable, 256K context, no censorship hedging. Preseed via
|
||
`init-models.sh` (an `s3_pull` line is already in place; see
|
||
[Mirroring models to S3](../README.md#mirroring-models-to-s3) for the
|
||
mirror workflow).
|
||
|
||
**Important Qwen 3.x quirk:** thinking mode is on by default and
|
||
breaks native function calling — the model "thinks" about how to use
|
||
the tool instead of just calling it. The shipped system prompt starts
|
||
with `/no_think` to suppress this. If the model still plans instead
|
||
of firing the tool, also set `enable_thinking: false` in **Advanced
|
||
Params → Custom Parameters** (API-level enforcement).
|
||
|
||
### Alternatives
|
||
|
||
If Qwen 3.5 isn't a fit (size, language preferences, abliteration
|
||
caveats), other vision-capable Ollama tags worth trying:
|
||
|
||
- `qwen2.5vl:7b` — smaller, no thinking mode, very reliable tool-caller
|
||
- `llama3.2-vision:11b` — Meta's vision variant, ~7 GB
|
||
- `minicpm-v:8b` — fast, capable
|
||
|
||
To swap, change `base_model_id` in `image_studio.json` (or the Base
|
||
Model field if you imported manually) and pull the model via
|
||
`init-models.sh` or the Open WebUI model UI.
|
||
|
||
### Non-vision base model
|
||
|
||
If you'd rather use a text-only LLM (e.g. `mistral-nemo:12b`),
|
||
keep `vision: true` in the preset so Open WebUI still permits image
|
||
attachments; the image flows through to `edit_image` via
|
||
`__messages__` / `__files__` and ComfyUI does the visual work. The
|
||
LLM can't see the image, but for explicit edit instructions ("change
|
||
the background to a sunset") that doesn't matter.
|
||
|
||
## Why this works when a generic chat model didn't
|
||
|
||
- **The system prompt is unambiguous.** No room for the model to
|
||
decide "I'll just describe it in text instead."
|
||
- **Only one tool is attached.** No competing tools to choose between.
|
||
- **Native function calling is mandatory.** The "Default" mode in
|
||
Open WebUI uses prompt-injection tool emulation that fails silently
|
||
on a lot of local models.
|
||
- **Lower temperature.** Tool calling is more reliable with less
|
||
sampling randomness.
|
||
|
||
## Iterating on the system prompt
|
||
|
||
If users ask for things you didn't anticipate (specific aspect ratios,
|
||
multi-image batches, particular checkpoints not in the routing rules),
|
||
edit the system prompt above and re-paste into the Workspace → Models
|
||
entry. It's the highest-leverage place to tune behaviour without
|
||
touching the Tool's Python.
|