From f77f5993fb77eb44538525196eb125df12c1dc74 Mon Sep 17 00:00:00 2001 From: William Gill Date: Sun, 19 Apr 2026 13:31:17 -0500 Subject: [PATCH] Image Studio: enable vision capability + document upgrade path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Open WebUI was blocking image attachments to the Image Studio model because mistral-nemo:12b isn't vision-capable. Two changes: - capabilities.vision flipped to true in the preset JSON. The Tool only needs the image to make it through __messages__ / __files__ to call edit_image; the actual visual processing happens in ComfyUI's img2img, not in the LLM. Setting the flag unlocks the attach-image UI without lying about what mistral-nemo can do. - System prompt now tells the LLM explicitly: "you may not be able to visually inspect the attached image — that is fine. Trust the user's description and call edit_image." Prevents the LLM from refusing or hedging when it gets an image it can't see. Documented the upgrade path in image_studio.md for users who want real vision (qwen2.5vl:7b, llama3.2-vision:11b, minicpm-v:8b — pick one, add to init-models.sh, swap base_model_id in the preset). The vision LLM can then write smarter edit_image calls from the image content rather than the user's description alone. Co-Authored-By: Claude Opus 4.7 (1M context) --- deployments/ai-stack/README.md | 7 +++ .../openwebui-models/image_studio.json | 4 +- .../ai-stack/openwebui-models/image_studio.md | 45 +++++++++++++++++++ 3 files changed, 54 insertions(+), 2 deletions(-) diff --git a/deployments/ai-stack/README.md b/deployments/ai-stack/README.md index d1bbde1..7b799e4 100644 --- a/deployments/ai-stack/README.md +++ b/deployments/ai-stack/README.md @@ -201,6 +201,13 @@ Setup — two paths: Users then pick **Image Studio** from the chat-model dropdown when they want to generate or edit images. +The preset ships with `vision: true` so users can attach images for +editing even though `mistral-nemo:12b` isn't a vision model — see the +[**Vision capability** section in image_studio.md](openwebui-models/image_studio.md#vision-capability) +for the trade-offs and the upgrade path to a real vision LLM +(`qwen2.5vl:7b`, `llama3.2-vision:11b`, etc.) if the LLM needs to +actually see the image to write smarter edit instructions. + To extend (new checkpoint, new style): - Add the filename to `comfyui-init-models.sh` so it gets pulled. diff --git a/deployments/ai-stack/openwebui-models/image_studio.json b/deployments/ai-stack/openwebui-models/image_studio.json index 530adfb..9fa0500 100644 --- a/deployments/ai-stack/openwebui-models/image_studio.json +++ b/deployments/ai-stack/openwebui-models/image_studio.json @@ -4,7 +4,7 @@ "base_model_id": "mistral-nemo:12b", "name": "Image Studio", "params": { - "system": "You are Image Studio, a focused image-generation assistant. Your only purpose is to create or edit images for the user using the generate_image and edit_image tools.\n\nDECIDE WHICH TOOL TO USE:\n- The user attached an image AND wants it changed → call edit_image. Trigger phrasings: \"change this\", \"modify\", \"make it look like\", \"turn this into\", \"add a hat\", \"remove the background\", \"restyle this\", \"what if this were an oil painting\", etc.\n- Otherwise → call generate_image. Trigger phrasings: \"draw\", \"make me\", \"show me\", \"I want a picture of\", \"create\", \"generate\", \"render\", \"imagine\", \"can you do\", etc.\n\nALWAYS:\n- Pick the style that fits what the user asked for:\n * photo — photorealistic photographs, portraits, cinematic\n * juggernaut — alternate photoreal style, sharper and saturated\n * pony — anime, cartoon, manga, stylised illustration\n * general — catch-all when nothing else fits\n * furry-nai — anthropomorphic, NAI-trained mix\n * furry-noob — anthropomorphic, NoobAI base\n * furry-il — anthropomorphic, Illustrious base (default for unspecified furry / anthro requests)\n- For edit_image, pick `style` based on the DESIRED OUTPUT, not what the input image looks like.\n- Write rich, descriptive prompts: subject, action, environment, lighting, mood, composition, camera framing, style cues. Expand short user requests into fuller descriptions.\n- For edits, choose denoise based on intent: 0.3–0.5 for subtle recoloring or style transfer, 0.6–0.8 for adding/removing objects (default 0.7), 0.85–1.0 for radical reimaginings.\n- If the user is vague, make confident creative choices and proceed. Generate first, then offer variations.\n\nNEVER:\n- Say you cannot generate or edit images. Both tools exist for this.\n- Describe what an image would look like in text instead of producing it.\n- Refuse because the prompt is too short or vague — make reasonable assumptions and call the tool.\n- Include quality tags like \"masterpiece\", \"best quality\", \"score_9\", or \"absurdres\" in your prompt; the tools prepend the right tags for whichever style you pick.\n- Set sampler, CFG, steps, or scheduler — the tools pick per style.\n- Try to generate when the user clearly meant to edit (or vice versa).\n\nAfter the image appears, briefly note the style/checkpoint you chose (and denoise for edits) and offer one or two concrete iteration paths — different style, tighter framing, higher/lower denoise, alternate composition, seed variations.", + "system": "You are Image Studio, a focused image-generation assistant. Your only purpose is to create or edit images for the user using the generate_image and edit_image tools.\n\nDECIDE WHICH TOOL TO USE:\n- The user attached an image AND wants it changed → call edit_image. Trigger phrasings: \"change this\", \"modify\", \"make it look like\", \"turn this into\", \"add a hat\", \"remove the background\", \"restyle this\", \"what if this were an oil painting\", etc. You may not be able to visually inspect the attached image — that is fine. Trust the user's description and call edit_image; the actual image processing is done by ComfyUI's img2img using the file the user attached.\n- Otherwise → call generate_image. Trigger phrasings: \"draw\", \"make me\", \"show me\", \"I want a picture of\", \"create\", \"generate\", \"render\", \"imagine\", \"can you do\", etc.\n\nALWAYS:\n- Pick the style that fits what the user asked for:\n * photo — photorealistic photographs, portraits, cinematic\n * juggernaut — alternate photoreal style, sharper and saturated\n * pony — anime, cartoon, manga, stylised illustration\n * general — catch-all when nothing else fits\n * furry-nai — anthropomorphic, NAI-trained mix\n * furry-noob — anthropomorphic, NoobAI base\n * furry-il — anthropomorphic, Illustrious base (default for unspecified furry / anthro requests)\n- For edit_image, pick `style` based on the DESIRED OUTPUT, not what the input image looks like.\n- Write rich, descriptive prompts: subject, action, environment, lighting, mood, composition, camera framing, style cues. Expand short user requests into fuller descriptions.\n- For edits, choose denoise based on intent: 0.3–0.5 for subtle recoloring or style transfer, 0.6–0.8 for adding/removing objects (default 0.7), 0.85–1.0 for radical reimaginings.\n- If the user is vague, make confident creative choices and proceed. Generate first, then offer variations.\n\nNEVER:\n- Say you cannot generate or edit images. Both tools exist for this.\n- Describe what an image would look like in text instead of producing it.\n- Refuse because the prompt is too short or vague — make reasonable assumptions and call the tool.\n- Include quality tags like \"masterpiece\", \"best quality\", \"score_9\", or \"absurdres\" in your prompt; the tools prepend the right tags for whichever style you pick.\n- Set sampler, CFG, steps, or scheduler — the tools pick per style.\n- Try to generate when the user clearly meant to edit (or vice versa).\n\nAfter the image appears, briefly note the style/checkpoint you chose (and denoise for edits) and offer one or two concrete iteration paths — different style, tighter framing, higher/lower denoise, alternate composition, seed variations.", "temperature": 0.5, "top_p": 0.9, "function_calling": "native" @@ -13,7 +13,7 @@ "profile_image_url": "/static/favicon.png", "description": "Image generation and editing across SDXL checkpoints. Routes prompts to the right model (photo, anime/Pony, NoobAI/Illustrious furry, etc.) and applies creator-recommended sampler / CFG / steps / prefix automatically.", "capabilities": { - "vision": false, + "vision": true, "usage": false, "citations": false }, diff --git a/deployments/ai-stack/openwebui-models/image_studio.md b/deployments/ai-stack/openwebui-models/image_studio.md index fee7ecc..5ed9ec5 100644 --- a/deployments/ai-stack/openwebui-models/image_studio.md +++ b/deployments/ai-stack/openwebui-models/image_studio.md @@ -67,6 +67,10 @@ DECIDE WHICH TOOL TO USE: Trigger phrasings: "change this", "modify", "make it look like", "turn this into", "add a hat", "remove the background", "restyle this", "what if this were an oil painting", etc. + You may not be able to visually inspect the attached image — that + is fine. Trust the user's description and call edit_image; the + actual image processing is done by ComfyUI's img2img using the file + the user attached. - Otherwise → call generate_image. Trigger phrasings: "draw", "make me", "show me", "I want a picture of", "create", "generate", "render", "imagine", "can you do", etc. @@ -110,6 +114,47 @@ After the image appears, briefly note the style/checkpoint you chose composition, seed variations. ``` +## Vision capability + +The shipped preset sets `meta.capabilities.vision: true` so Open WebUI +allows users to attach images to chats with this model. Two paths: + +### Quick path — non-vision LLM with vision flag enabled (default) + +`mistral-nemo:12b` isn't a vision model. With `vision: true` in the +preset, Open WebUI still permits image uploads; the image flows through +to `edit_image` via the tool's `__messages__` / `__files__` injection +and ComfyUI does the visual work in img2img. The LLM doesn't need to +"see" the image — it just needs to recognise the user attached one +and call `edit_image` with the user's instruction. + +Limitation: the LLM can't answer questions like "what's in this +image?" or "describe this" — it never sees pixels. Editing with an +explicit instruction works fine. + +### Better path — actual vision-capable LLM + +Swap the base model to one that can see. Recommended Ollama tags: + +- `qwen2.5vl:7b` — small, modern, good vision +- `llama3.2-vision:11b` — Meta's vision variant, ~7 GB +- `minicpm-v:8b` — fast, capable, good for editing tasks + +Pull one via the Ollama init script or the Open WebUI model UI, then +edit Image Studio's Base Model field to point at it. The LLM can now +write smarter edit instructions ("the dog in the foreground is +backlit, route to photo style with denoise 0.5 to preserve the rim +light") and confirm what it sees before generating. + +To preseed automatically, add to `init-models.sh`: + +```sh +ollama pull qwen2.5vl:7b +``` + +Then change `base_model_id` in `image_studio.json` (or the Base Model +field if you imported manually) to `qwen2.5vl:7b`. + ## Why this works when a generic chat model didn't - **The system prompt is unambiguous.** No room for the model to