Add text-targeted inpainting via GroundingDINO+SAM (mask_text param)

Five pieces:

1. Dockerfile installs storyicon/comfyui_segment_anything (GroundingDINO
   + SAM-HQ in one bundle) into custom_nodes and pip-installs its
   requirements at build time. Model weights auto-download to the
   comfyui-models volume on first inpaint (~3 GB one-time cost).

2. install-custom-node-deps.sh — entrypoint wrapper that pip-installs
   requirements.txt for any custom_node present at startup. Lets users
   add custom nodes via ComfyUI-Manager (or by git-cloning into the
   volume) and have the deps picked up on the next restart, without
   editing the Dockerfile.

3. smart_image_gen v0.6: edit_image gains a `mask_text` param. When
   set, builds an inpainting workflow (LoadImage → GroundingDinoSAM
   Segment → SetLatentNoiseMask → KSampler) so only the named region
   is repainted. When unset, falls through to the existing img2img
   path. Denoise default switches: 1.0 with mask_text (full repaint
   within mask), 0.7 without.

4. Image Studio system prompt teaches the LLM the LOCAL vs GLOBAL
   distinction — set mask_text whenever the user names a specific
   object/region ('the ball', 'the dog', 'the sky'); leave it unset
   only for whole-image style/lighting transformations.

5. Deployment README documents the new mode + the first-inpaint
   weight-download caveat.

Image rebuild required — bump tag to pick up the Dockerfile change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Dockerfile

@@ -52,8 +52,24 @@ RUN git clone --depth 1 https://github.com/ltdrdata/ComfyUI-Manager.git \
         ${COMFYUI_HOME}/custom_nodes/ComfyUI-Manager && \
     pip install -r ${COMFYUI_HOME}/custom_nodes/ComfyUI-Manager/requirements.txt
 
+# comfyui_segment_anything — GroundingDINO + SAM-HQ in one bundle. Required
+# by the smart_image_gen Tool's text-targeted inpainting (edit_image with the
+# mask_text parameter). Model weights auto-download on first use into
+# /opt/comfyui/models/{sams,grounding-dino}/ — first inpaint takes ~3 GB of
+# downloads, subsequent runs are instant.
+RUN git clone --depth 1 https://github.com/storyicon/comfyui_segment_anything.git \
+        ${COMFYUI_HOME}/custom_nodes/comfyui_segment_anything && \
+    pip install -q -r ${COMFYUI_HOME}/custom_nodes/comfyui_segment_anything/requirements.txt
+
+# Entrypoint wrapper — auto-installs requirements.txt for any custom_node
+# present at startup (covers Manager-installed nodes and nodes cloned
+# directly into the comfyui-custom-nodes volume).
+COPY install-custom-node-deps.sh /usr/local/bin/install-custom-node-deps.sh
+RUN chmod +x /usr/local/bin/install-custom-node-deps.sh
+
 EXPOSE 8188
 
 # --listen 0.0.0.0 binds to every interface so the Open WebUI container on the
 # shared compose network can reach it. --port is explicit for clarity.
+ENTRYPOINT ["/usr/local/bin/install-custom-node-deps.sh"]
 CMD ["python", "main.py", "--listen", "0.0.0.0", "--port", "8188"]

deployments/ai-stack/README.md

@@ -17,7 +17,7 @@ production `srvno.de` deployment.
 | `init-models.sh`                        | LLMs to preseed into Ollama on first boot                |
 | `mirror-ollama-model.sh`                | Helper — mirror an Ollama model into a tarball you can host on S3 |
 | `comfyui-init-models.sh`                | Checkpoints/VAEs/LoRAs to preseed into ComfyUI on first boot |
-| `openwebui-tools/smart_image_gen.py`    | Tool that auto-routes image generation AND editing to the right SDXL checkpoint |
+| `openwebui-tools/smart_image_gen.py`    | Tool that auto-routes image generation, img2img, and text-targeted inpainting to the right SDXL checkpoint |
 | `openwebui-models/image_studio.md`      | Dedicated chat-model preset — manual setup walkthrough               |
 | `openwebui-models/image_studio.json`    | The same preset as an importable Open WebUI model JSON               |
 | `.env.example`                          | Secrets and image-tag pins. Copy to `.env`               |
@@ -167,10 +167,19 @@ calls:
 
 - **`generate_image`** for new images from scratch (txt2img).
 - **`edit_image`** for modifying an image the user attached to the
-  chat (img2img).
+  chat. Two modes:
+    - With `mask_text` — text-targeted inpainting via GroundingDINO+SAM
+      (e.g. "the dog's collar"). Only the named region is repainted.
+    - Without `mask_text` — full img2img which reimagines the whole
+      image at the requested denoise.
 
 Both auto-route to the right SDXL checkpoint per request.
 
+> **First inpaint takes a few minutes**: SAM-HQ (~2.5 GB) and
+> GroundingDINO (~700 MB) auto-download into the `comfyui-models`
+> volume on the very first call to `edit_image` with `mask_text`.
+> Subsequent inpaints are instant.
+
 1. **Workspace -> Tools -> +** (top-right).
 2. Paste the contents of
    [`openwebui-tools/smart_image_gen.py`](openwebui-tools/smart_image_gen.py).

deployments/ai-stack/openwebui-models/image_studio.json

@@ -4,7 +4,7 @@
     "base_model_id": "huihui_ai/qwen3.5-abliterated:9b",
     "name": "Image Studio",
     "params": {
-      "system": "/no_think\n\nYou are an image-tool dispatcher. You do not respond in prose. Every user message MUST result in exactly one tool call.\n\nROUTING:\n- If the user attached an image → call edit_image(edit_instruction, style, denoise)\n- Otherwise → call generate_image(prompt, style)\n\nFire the tool on the FIRST message, with no preamble. Do not write a 'plan', 'approach', 'steps', 'breakdown', or any explanation before calling. Do not ask clarifying questions. Do not say what you are about to do. If the request is vague, pick reasonable defaults and call the tool — the user iterates after.\n\nSTYLES (pick one):\n  photo         photorealistic photo / portrait / cinematic\n  juggernaut    alternate photoreal — sharper, more saturated\n  pony          anime, cartoon, manga, stylised illustration\n  general       catch-all when nothing else fits\n  furry-nai     anthropomorphic, NAI-trained mix\n  furry-noob    anthropomorphic, NoobAI base\n  furry-il      anthropomorphic, Illustrious base (default for any furry/anthro request)\n\nFor edit_image, pick style for the DESIRED OUTPUT, not the input image. Default denoise 0.7. Use 0.3–0.5 for subtle changes (recolor, style transfer) and 0.85–1.0 for radical reimaginings.\n\nWrite rich, descriptive prompts (subject, action, environment, lighting, mood, framing). Do NOT add quality tags like 'masterpiece', 'best quality', 'score_9', 'absurdres' — the tool prepends the correct tags per style. Do NOT set sampler, CFG, steps, scheduler — the tool picks them.\n\nAFTER the tool returns, write at most one short sentence noting your style choice and offering one iteration idea. The image is already shown to the user; do not describe it.",
+      "system": "/no_think\n\nYou are an image-tool dispatcher. You do not respond in prose. Every user message MUST result in exactly one tool call.\n\nROUTING:\n- If the user attached an image → call edit_image\n- Otherwise → call generate_image\n\nFire the tool on the FIRST message, with no preamble. Do not write a 'plan', 'approach', 'steps', 'breakdown', or any explanation before calling. Do not ask clarifying questions. Do not say what you are about to do. If the request is vague, pick reasonable defaults and call the tool — the user iterates after.\n\nSTYLES (pick one):\n  photo         photorealistic photo / portrait / cinematic\n  juggernaut    alternate photoreal — sharper, more saturated\n  pony          anime, cartoon, manga, stylised illustration\n  general       catch-all when nothing else fits\n  furry-nai     anthropomorphic, NAI-trained mix\n  furry-noob    anthropomorphic, NoobAI base\n  furry-il      anthropomorphic, Illustrious base (default for any furry/anthro request)\n\nedit_image has TWO MODES — pick based on whether the change is local or global:\n- LOCAL change (\"change the ball to a basketball\", \"add a hat to the dog\", \"remove the bird\", \"recolor the car red\") → set `mask_text` to a brief noun phrase naming the region (\"the ball\", \"the dog\", \"the bird\", \"the car\"). Only that region is repainted; rest stays pixel-perfect.\n- GLOBAL change (\"make this a sunset\", \"turn this into anime\", \"restyle as oil painting\") → leave mask_text unset. The whole image is reimagined.\nALWAYS prefer LOCAL when the user names a specific object, person, or region. GLOBAL is only for whole-image style/lighting transformations.\n\nDenoise:\n- LOCAL (mask_text set): default 1.0. Drop to 0.6–0.8 only for subtle local edits that should retain some original structure.\n- GLOBAL (no mask_text): default 0.7. Use 0.3–0.5 for subtle restyle, 0.85–1.0 for radical reimagining.\n\nPick style for the DESIRED OUTPUT, not the input image.\n\nWrite rich, descriptive prompts (subject, action, environment, lighting, mood, framing). Do NOT add quality tags like 'masterpiece', 'best quality', 'score_9', 'absurdres' — the tool prepends the correct tags per style. Do NOT set sampler, CFG, steps, scheduler — the tool picks them.\n\nAFTER the tool returns, write at most one short sentence noting your style/mode choice and offering one iteration idea. The image is already shown to the user; do not describe it.",
       "temperature": 0.5,
       "top_p": 0.9,
       "function_calling": "native",

deployments/ai-stack/openwebui-models/image_studio.md

@@ -65,8 +65,8 @@ You are an image-tool dispatcher. You do not respond in prose. Every
 user message MUST result in exactly one tool call.
 
 ROUTING:
-- If the user attached an image → call edit_image(edit_instruction, style, denoise)
-- Otherwise → call generate_image(prompt, style)
+- If the user attached an image → call edit_image
+- Otherwise → call generate_image
 
 Fire the tool on the FIRST message, with no preamble. Do not write a
 'plan', 'approach', 'steps', 'breakdown', or any explanation before
@@ -84,9 +84,29 @@ STYLES (pick one):
   furry-il      anthropomorphic, Illustrious base (default for any
                 furry/anthro request)
 
-For edit_image, pick style for the DESIRED OUTPUT, not the input
-image. Default denoise 0.7. Use 0.3–0.5 for subtle changes (recolor,
-style transfer) and 0.85–1.0 for radical reimaginings.
+edit_image has TWO MODES — pick based on whether the change is local
+or global:
+
+- LOCAL ("change the ball to a basketball", "add a hat to the dog",
+  "remove the bird", "recolor the car red") → set `mask_text` to a
+  brief noun phrase naming the region ("the ball", "the dog", "the
+  bird", "the car"). Only that region is repainted; rest stays
+  pixel-perfect.
+- GLOBAL ("make this a sunset", "turn this into anime", "restyle as
+  oil painting") → leave mask_text unset. The whole image is
+  reimagined.
+
+ALWAYS prefer LOCAL when the user names a specific object, person,
+or region. GLOBAL is only for whole-image style/lighting
+transformations.
+
+Denoise:
+- LOCAL (mask_text set): default 1.0. Drop to 0.6–0.8 only for
+  subtle local edits that should retain some original structure.
+- GLOBAL (no mask_text): default 0.7. Use 0.3–0.5 for subtle
+  restyle, 0.85–1.0 for radical reimagining.
+
+Pick style for the DESIRED OUTPUT, not the input image.
 
 Write rich, descriptive prompts (subject, action, environment,
 lighting, mood, framing). Do NOT add quality tags like 'masterpiece',
@@ -95,8 +115,8 @@ correct tags per style. Do NOT set sampler, CFG, steps, scheduler —
 the tool picks them.
 
 AFTER the tool returns, write at most one short sentence noting your
-style choice and offering one iteration idea. The image is already
-shown to the user; do not describe it.
+style/mode choice and offering one iteration idea. The image is
+already shown to the user; do not describe it.
 ```
 
 The first line `/no_think` disables Qwen 3.x's reasoning phase. If

deployments/ai-stack/openwebui-tools/smart_image_gen.py

@@ -1,7 +1,7 @@
 """
 title: Smart Image Generator & Editor (ComfyUI)
 author: ai-stack
-version: 0.5.0
+version: 0.6.0
 description: Generate or edit images via ComfyUI with automatic SDXL
     checkpoint routing. Two methods — generate_image (txt2img) and
     edit_image (img2img on the user's most recently attached image). The
@@ -254,6 +254,58 @@ def _build_txt2img(positive: str, negative: str, settings: dict,
     }
 
 
+def _build_inpaint(positive: str, negative: str, settings: dict,
+                   image_filename: str, mask_text: str,
+                   denoise: float, seed: int) -> dict:
+    """
+    SDXL inpainting workflow with text-driven masking. Uses
+    comfyui_segment_anything (GroundingDINO + SAM-HQ — installed by the
+    Dockerfile) to derive a mask from `mask_text` (a noun phrase like
+    "the dog's collar"), then SetLatentNoiseMask + KSampler repaint
+    only that region. Everything outside the mask stays pixel-perfect.
+
+    First inpaint downloads ~3 GB of SAM/GroundingDINO weights into
+    /opt/comfyui/models/{sams,grounding-dino}/ — subsequent runs reuse
+    them.
+    """
+    return {
+        "3": {"class_type": "KSampler", "inputs": {
+            "seed": _seed_value(seed),
+            "steps": settings["steps"], "cfg": settings["cfg"],
+            "sampler_name": settings["sampler"], "scheduler": settings["scheduler"],
+            "denoise": denoise,
+            "model": ["4", 0], "positive": ["6", 0],
+            "negative": ["7", 0], "latent_image": ["13", 0],
+        }},
+        "4": {"class_type": "CheckpointLoaderSimple",
+              "inputs": {"ckpt_name": settings["ckpt"]}},
+        "6": {"class_type": "CLIPTextEncode", "inputs": {"text": positive, "clip": ["10", 0]}},
+        "7": {"class_type": "CLIPTextEncode", "inputs": {"text": negative, "clip": ["10", 0]}},
+        "8": {"class_type": "VAEDecode", "inputs": {"samples": ["3", 0], "vae": ["4", 2]}},
+        "9": {"class_type": "SaveImage",
+              "inputs": {"filename_prefix": "smartinpaint", "images": ["8", 0]}},
+        "10": {"class_type": "CLIPSetLastLayer",
+               "inputs": {"stop_at_clip_layer": -settings["clip_skip"],
+                          "clip": ["4", 1]}},
+        "11": {"class_type": "VAEEncode", "inputs": {"pixels": ["12", 0], "vae": ["4", 2]}},
+        "12": {"class_type": "LoadImage", "inputs": {"image": image_filename}},
+        "13": {"class_type": "SetLatentNoiseMask",
+               "inputs": {"samples": ["11", 0], "mask": ["16", 1]}},
+        "14": {"class_type": "SAMModelLoader (segment anything)",
+               "inputs": {"model_name": "sam_hq_vit_h (2.57GB)"}},
+        "15": {"class_type": "GroundingDinoModelLoader (segment anything)",
+               "inputs": {"model_name": "GroundingDINO_SwinT_OGC (694MB)"}},
+        "16": {"class_type": "GroundingDinoSAMSegment (segment anything)",
+               "inputs": {
+                   "sam_model": ["14", 0],
+                   "grounding_dino_model": ["15", 0],
+                   "image": ["12", 0],
+                   "prompt": mask_text,
+                   "threshold": 0.3,
+               }},
+    }
+
+
 def _build_img2img(positive: str, negative: str, settings: dict,
                    image_filename: str, denoise: float, seed: int) -> dict:
     """
@@ -595,7 +647,8 @@ class Tools:
         self,
         edit_instruction: str,
         style: Optional[StyleName] = None,
-        denoise: float = 0.7,
+        mask_text: Optional[str] = None,
+        denoise: Optional[float] = None,
         negative_prompt: Optional[str] = None,
         seed: int = 0,
         __request__=None,
@@ -607,31 +660,44 @@ class Tools:
     ) -> str:
         """
         Edit, modify, transform, or restyle an image the user has ATTACHED
-        to the chat. Use this whenever the user uploads or attaches an
-        image and asks to change it — "make this a sunset", "add a hat",
-        "turn this into anime", "remove the background", etc. If no image
-        is attached, use generate_image instead.
+        to the chat. Use whenever the user uploads an image and asks to
+        change it. If no image is attached, use generate_image instead.
 
-        The tool finds the most recent attached image automatically, picks
-        the right SDXL checkpoint via `style` (same options as
-        generate_image), and applies the edit at the requested `denoise`.
+        TWO MODES — choose based on whether the change is local or global:
+
+        - LOCAL change ("change the ball to a basketball", "make the dog
+          wear a hat", "remove the bird") → set `mask_text` to a brief
+          noun phrase describing the region ("the ball", "the dog", "the
+          bird"). The tool uses GroundingDINO+SAM to find that region
+          automatically and only that area is repainted; the rest of the
+          image stays pixel-perfect.
+
+        - GLOBAL change ("make this a sunset", "turn this into anime",
+          "restyle this as oil painting") → leave `mask_text` unset. The
+          whole image is reimagined via img2img.
+
+        Always prefer LOCAL mode when the user names a specific object,
+        person, or region. GLOBAL mode is for whole-image style/lighting
+        transformations.
 
         Denoise tuning:
-        - 0.3 to 0.5 → subtle (style transfer, color tweaks, preserve composition)
-        - 0.6 to 0.8 → moderate (add/remove objects, lighting changes — default)
-        - 0.85 to 1.0 → heavy (basically reimagining; loses original details)
+        - LOCAL (mask_text set): default 1.0 — full repaint within mask.
+          Drop to 0.6–0.8 for subtle local edits that should retain some
+          original structure.
+        - GLOBAL (no mask_text): default 0.7 — moderate edit. Use 0.3–0.5
+          for subtle restyling, 0.85–1.0 for radical reimagining.
 
-        Pick `style` based on the desired output, NOT the input image.
-        Same options as generate_image: photo, juggernaut, pony, general,
-        furry-nai, furry-noob, furry-il.
+        Pick `style` for the DESIRED OUTPUT, not the input image.
 
-        :param edit_instruction: What to change. The tool auto-prepends the
-            style's quality tags so don't include those.
-        :param style: One of the StyleName values. Omit to auto-detect from
-            edit_instruction.
-        :param denoise: 0.0 = no change, 1.0 = ignore source. Default 0.7.
+        :param edit_instruction: What the changed area should look like.
+            Tool auto-prepends quality tags — don't include those.
+        :param style: One of the StyleName values. Omit to auto-detect.
+        :param mask_text: Noun phrase describing the region to edit. Set
+            for LOCAL changes; omit for GLOBAL.
+        :param denoise: 0.0 = no change, 1.0 = ignore source. Defaults to
+            1.0 with mask_text, 0.7 without.
         :param negative_prompt: Extra terms to exclude. Usually unneeded.
-        :param seed: 0 to randomize, otherwise a specific seed.
+        :param seed: 0 to randomize, otherwise specific.
         :return: Markdown image of the result, or an error if no image is attached.
         """
         chosen = style or _route_style(edit_instruction)
@@ -639,6 +705,10 @@ class Tools:
         if not settings:
             return f"Unknown style '{chosen}'. Available: {', '.join(STYLES.keys())}"
 
+        # Denoise default depends on mode: 1.0 (full repaint within mask)
+        # for inpainting, 0.7 for img2img.
+        if denoise is None:
+            denoise = 1.0 if mask_text else 0.7
         denoise = max(0.0, min(1.0, denoise))
 
         async def emit(msg: str, done: bool = False):
@@ -665,8 +735,10 @@ class Tools:
             if not uploaded_name:
                 return "Failed to upload source image to ComfyUI."
 
+            mode = "inpaint" if mask_text else "img2img"
             await emit(
-                f"Routing to {chosen} ({settings['ckpt']}), denoise {denoise:.2f}"
+                f"Routing to {chosen} ({settings['ckpt']}), {mode}, denoise {denoise:.2f}"
+                + (f", mask='{mask_text}'" if mask_text else "")
             )
 
             positive = f"{settings['prefix']}{edit_instruction}"
@@ -674,14 +746,25 @@ class Tools:
             if negative_prompt:
                 negative = f"{negative}, {negative_prompt}"
 
-            workflow = _build_img2img(
-                positive=positive,
-                negative=negative,
-                settings=settings,
-                image_filename=uploaded_name,
-                denoise=denoise,
-                seed=seed,
-            )
+            if mask_text:
+                workflow = _build_inpaint(
+                    positive=positive,
+                    negative=negative,
+                    settings=settings,
+                    image_filename=uploaded_name,
+                    mask_text=mask_text,
+                    denoise=denoise,
+                    seed=seed,
+                )
+            else:
+                workflow = _build_img2img(
+                    positive=positive,
+                    negative=negative,
+                    settings=settings,
+                    image_filename=uploaded_name,
+                    denoise=denoise,
+                    seed=seed,
+                )
 
             raw_out, err = await _submit_and_fetch(
                 session, base, workflow, self.valves.TIMEOUT_SECONDS, emit, settings,

install-custom-node-deps.sh

@@ -0,0 +1,21 @@
+#!/bin/sh
+# Entrypoint wrapper. Pip-installs requirements.txt for any custom_node
+# present in /opt/comfyui/custom_nodes/, then exec's the CMD.
+#
+# This makes the container self-healing for custom nodes that get added
+# at runtime — either via ComfyUI-Manager from the web UI, or by
+# git-cloning directly into the comfyui-custom-nodes volume. Pip skips
+# already-satisfied requirements quickly, so the boot-time cost on
+# subsequent restarts is negligible.
+
+set -e
+
+if [ -d /opt/comfyui/custom_nodes ]; then
+    for req in /opt/comfyui/custom_nodes/*/requirements.txt; do
+        [ -f "$req" ] || continue
+        echo "[entrypoint] installing $req"
+        pip install -q -r "$req" || echo "  (install failed — continuing)"
+    done
+fi
+
+exec "$@"