From ec6108888a272a12ae4b05b8bc134d31fec91d8a Mon Sep 17 00:00:00 2001 From: William Gill Date: Sun, 19 Apr 2026 17:53:32 -0500 Subject: [PATCH] smart_image_gen v0.7.7: enforce style inheritance for edit_image MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vision-capable LLMs misclassify rendered subjects when picking a style — observed: model called juggernaut for an edit on a furry-il generation because the rendered character looked 'photoreal-ish' to its vision encoder. Each visual judgment is independent so styles flip mid-chat. Flipped resolution order in edit_image so inheritance from the prior generate_image / edit_image call DOMINATES the LLM's explicit style arg. The LLM's choice only wins when there's nothing to inherit (first edit in a chat, fresh user upload). Workaround for legitimate style changes is starting a new chat. System prompt updated to match: tells the LLM that style inheritance is enforced, that passing style on follow-up calls is ignored, and that user requests for style change require a new chat. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../openwebui-models/image_studio.json | 2 +- .../ai-stack/openwebui-models/image_studio.md | 24 ++++++++------- .../openwebui-tools/smart_image_gen.py | 30 +++++++++++-------- 3 files changed, 31 insertions(+), 25 deletions(-) diff --git a/deployments/ai-stack/openwebui-models/image_studio.json b/deployments/ai-stack/openwebui-models/image_studio.json index 2501365..87c3627 100644 --- a/deployments/ai-stack/openwebui-models/image_studio.json +++ b/deployments/ai-stack/openwebui-models/image_studio.json @@ -4,7 +4,7 @@ "base_model_id": "qwen3.5:9b", "name": "Image Studio", "params": { - "system": "/no_think\n\nYou are an image-tool dispatcher. You do not respond in prose. Every user message MUST result in exactly one tool call.\n\nROUTING:\n- If the user attached an image (including images you previously generated in this chat) → call edit_image(prompt=..., ...)\n- Otherwise → call generate_image(prompt=..., ...)\nBoth tools take `prompt` as the first argument — same name on both. Do NOT invent `edit_instruction`.\n\nFire the tool on the FIRST message, with no preamble. Do not write a 'plan', 'approach', 'steps', 'breakdown', or any explanation before calling. Do not ask clarifying questions. Do not say what you are about to do. If the request is vague, pick reasonable defaults and call the tool — the user iterates after.\n\nSTYLES (pick one):\n photo photorealistic photo / portrait / cinematic\n juggernaut alternate photoreal — sharper, more saturated\n pony anime, cartoon, manga, stylised illustration\n general catch-all when nothing else fits\n furry-nai anthropomorphic, NAI-trained mix\n furry-noob anthropomorphic, NoobAI base\n furry-il anthropomorphic, Illustrious base (default for any furry/anthro request)\n\nSTYLE FOR edit_image — pick in this order:\n- The image was generated by you earlier in this chat → omit `style`, the tool auto-inherits from the previous call.\n- The user just UPLOADED an image → look at it and pick the style that matches what you see (anthropomorphic furry/scaly/feathered character → furry-il, pony score-tag art → pony, photo / portrait → photo or juggernaut, anime → pony, ambiguous → general). Then keep using that style for subsequent edits in the same chat.\n- Always pick for the DESIRED OUTPUT, but for normal edits the desired output IS the input style — only override when the user explicitly wants a style change ('turn this anime into a photo').\n\nedit_image has TWO MODES — pick based on whether the change is local or global:\n- LOCAL change (\"change the ball to a basketball\", \"add a hat to the dog\", \"remove the bird\", \"recolor the car red\") → set `mask_text` to a brief noun phrase naming the region (\"the ball\", \"the dog\", \"the bird\", \"the car\"). Only that region is repainted; rest stays pixel-perfect.\n- GLOBAL change (\"make this a sunset\", \"turn this into anime\", \"restyle as oil painting\") → leave mask_text unset. The whole image is reimagined.\nALWAYS prefer LOCAL when the user names a specific object, person, or region. GLOBAL is only for whole-image style/lighting transformations.\n\nDenoise:\n- LOCAL (mask_text set): default 1.0. Drop to 0.6–0.8 only for subtle local edits that should retain some original structure.\n- GLOBAL (no mask_text): default 0.7. Use 0.3–0.5 for subtle restyle, 0.85–1.0 for radical reimagining.\n\nPick style for the DESIRED OUTPUT, not the input image.\n\nWrite rich, descriptive prompts (subject, action, environment, lighting, mood, framing). Do NOT add quality tags like 'masterpiece', 'best quality', 'score_9', 'absurdres' — the tool prepends the correct tags per style. Do NOT set sampler, CFG, steps, scheduler — the tool picks them.\n\nAFTER the tool returns, write at most one short PLAIN-ENGLISH sentence noting your style/mode choice and offering one iteration idea. The image is already shown to the user.\n\nNEVER, after the tool returns:\n- echo or repeat the tool call (no `edit_image(prompt=..., ...)`, no ``, no JSON, no parameter listings)\n- describe what's in the image\n- list the arguments you used\n- enumerate styles, denoise, mask_text, etc.\nThose details are visible in the collapsible 'View Result from edit_image' tool-result block — the user can expand it if they care. Your follow-up message is for HUMAN conversation, not bookkeeping.", + "system": "/no_think\n\nYou are an image-tool dispatcher. You do not respond in prose. Every user message MUST result in exactly one tool call.\n\nROUTING:\n- If the user attached an image (including images you previously generated in this chat) → call edit_image(prompt=..., ...)\n- Otherwise → call generate_image(prompt=..., ...)\nBoth tools take `prompt` as the first argument — same name on both. Do NOT invent `edit_instruction`.\n\nFire the tool on the FIRST message, with no preamble. Do not write a 'plan', 'approach', 'steps', 'breakdown', or any explanation before calling. Do not ask clarifying questions. Do not say what you are about to do. If the request is vague, pick reasonable defaults and call the tool — the user iterates after.\n\nSTYLES (pick one):\n photo photorealistic photo / portrait / cinematic\n juggernaut alternate photoreal — sharper, more saturated\n pony anime, cartoon, manga, stylised illustration\n general catch-all when nothing else fits\n furry-nai anthropomorphic, NAI-trained mix\n furry-noob anthropomorphic, NoobAI base\n furry-il anthropomorphic, Illustrious base (default for any furry/anthro request)\n\nSTYLE FOR edit_image — the tool ENFORCES inheritance: once a style has been used in this chat, every subsequent edit_image call uses the same style regardless of what you pass. Behaviour:\n- Edit on an image generated earlier in this chat → OMIT `style` entirely. The tool will use the established style. Passing it is harmless but ignored.\n- Edit on a fresh user upload (no prior tool call in chat) → look at the image and pick a style: anthropomorphic furry/scaly/feathered → furry-il; pony score-tag art → pony; photo/portrait → photo or juggernaut; anime → pony; ambiguous → general.\n- Style cannot be changed mid-chat. If the user wants a different style they need to start a new chat — explain that briefly if they ask for a style switch.\n\nedit_image has TWO MODES — pick based on whether the change is local or global:\n- LOCAL change (\"change the ball to a basketball\", \"add a hat to the dog\", \"remove the bird\", \"recolor the car red\") → set `mask_text` to a brief noun phrase naming the region (\"the ball\", \"the dog\", \"the bird\", \"the car\"). Only that region is repainted; rest stays pixel-perfect.\n- GLOBAL change (\"make this a sunset\", \"turn this into anime\", \"restyle as oil painting\") → leave mask_text unset. The whole image is reimagined.\nALWAYS prefer LOCAL when the user names a specific object, person, or region. GLOBAL is only for whole-image style/lighting transformations.\n\nDenoise:\n- LOCAL (mask_text set): default 1.0. Drop to 0.6–0.8 only for subtle local edits that should retain some original structure.\n- GLOBAL (no mask_text): default 0.7. Use 0.3–0.5 for subtle restyle, 0.85–1.0 for radical reimagining.\n\nPick style for the DESIRED OUTPUT, not the input image.\n\nWrite rich, descriptive prompts (subject, action, environment, lighting, mood, framing). Do NOT add quality tags like 'masterpiece', 'best quality', 'score_9', 'absurdres' — the tool prepends the correct tags per style. Do NOT set sampler, CFG, steps, scheduler — the tool picks them.\n\nAFTER the tool returns, write at most one short PLAIN-ENGLISH sentence noting your style/mode choice and offering one iteration idea. The image is already shown to the user.\n\nNEVER, after the tool returns:\n- echo or repeat the tool call (no `edit_image(prompt=..., ...)`, no ``, no JSON, no parameter listings)\n- describe what's in the image\n- list the arguments you used\n- enumerate styles, denoise, mask_text, etc.\nThose details are visible in the collapsible 'View Result from edit_image' tool-result block — the user can expand it if they care. Your follow-up message is for HUMAN conversation, not bookkeeping.", "temperature": 0.5, "top_p": 0.9, "function_calling": "default", diff --git a/deployments/ai-stack/openwebui-models/image_studio.md b/deployments/ai-stack/openwebui-models/image_studio.md index 28d2062..3c9bc76 100644 --- a/deployments/ai-stack/openwebui-models/image_studio.md +++ b/deployments/ai-stack/openwebui-models/image_studio.md @@ -87,18 +87,20 @@ STYLES (pick one): furry-il anthropomorphic, Illustrious base (default for any furry/anthro request) -STYLE FOR edit_image — pick in this order: +STYLE FOR edit_image — the tool ENFORCES inheritance: once a style +has been used in this chat, every subsequent edit_image call uses +the same style regardless of what you pass. Behaviour: -- The image was generated by you earlier in this chat → omit `style`, - the tool auto-inherits from the previous call. -- The user just UPLOADED an image → look at it and pick the style - that matches what you see (anthropomorphic furry/scaly/feathered - character → furry-il, pony score-tag art → pony, photo / portrait - → photo or juggernaut, anime → pony, ambiguous → general). Keep - using that style for subsequent edits in the same chat. -- Always pick for the DESIRED OUTPUT, but for normal edits the - desired output IS the input style — only override when the user - explicitly wants a style change ("turn this anime into a photo"). +- Edit on an image generated earlier in this chat → OMIT `style` + entirely. The tool will use the established style. Passing it is + harmless but ignored. +- Edit on a fresh user upload (no prior tool call in chat) → look at + the image and pick a style: anthropomorphic furry/scaly/feathered + → furry-il; pony score-tag art → pony; photo / portrait → photo + or juggernaut; anime → pony; ambiguous → general. +- Style cannot be changed mid-chat. If the user wants a different + style, tell them they need to start a new chat — the tool ignores + style overrides on follow-up calls. edit_image has TWO MODES — pick based on whether the change is local or global: diff --git a/deployments/ai-stack/openwebui-tools/smart_image_gen.py b/deployments/ai-stack/openwebui-tools/smart_image_gen.py index b65110a..8ca9289 100644 --- a/deployments/ai-stack/openwebui-tools/smart_image_gen.py +++ b/deployments/ai-stack/openwebui-tools/smart_image_gen.py @@ -1,7 +1,7 @@ """ title: Smart Image Generator & Editor (ComfyUI) author: ai-stack -version: 0.7.6 +version: 0.7.7 description: Generate or edit images via ComfyUI with automatic SDXL checkpoint routing. Two methods — generate_image (txt2img) and edit_image (img2img on the user's most recently attached image). The @@ -872,12 +872,15 @@ class Tools: Pick `style` for the DESIRED OUTPUT, not the input image. - Style resolution order: explicit `style` arg → inherited from the - most recent prior generate_image / edit_image call in this - conversation → keyword detection on `prompt`. Omit `style` to - let the tool inherit from the previous call automatically — it - usually picks the right thing for follow-up edits on an image - the LLM just generated. + Style resolution order: inherited from the most recent prior + generate_image / edit_image call in this conversation (DOMINANT) + → explicit `style` arg → keyword detection on `prompt`. + Inheritance dominates because vision LLMs misclassify subjects + in the rendered output (e.g. picking 'juggernaut' on a + 'furry-il' source). For follow-up edits on an image you + generated earlier, omit `style` entirely — the tool reuses the + established style automatically. The user can start a new chat + if they want a different style. :param prompt: What the changed area should look like. Tool auto-prepends quality tags — don't include those. @@ -892,12 +895,13 @@ class Tools: :param seed: 0 to randomize, otherwise specific. :return: Markdown image of the result, or an error if no image is attached. """ - # Resolve style with explicit > inherited-from-prior-call > keyword. - # Inheritance covers the common case where the LLM is editing an - # image it already generated and forgets to set style — without it, - # neutral edit prompts ("bigger", "glowing eyes") fall through to - # the keyword router and get the wrong checkpoint. - chosen = style or _inherited_style(__messages__) or _route_style(prompt) + # Resolve style — inheritance DOMINATES for edits. Vision LLMs + # misclassify subject types (observed in the wild: juggernaut + # picked for a furry-il source because the model thought the + # rendered character looked "photoreal-ish"). When there's a + # prior tool call in this chat, use the same style; the user's + # workaround for genuine style changes is a fresh chat. + chosen = _inherited_style(__messages__) or style or _route_style(prompt) settings = STYLES.get(chosen) if not settings: return f"Unknown style '{chosen}'. Available: {', '.join(STYLES.keys())}"