Add text-targeted inpainting via GroundingDINO+SAM (mask_text param)
release / Build & Push Docker Image (push) Successful in 44s
release / Build & Push Docker Image (push) Successful in 44s
Five pieces:
1. Dockerfile installs storyicon/comfyui_segment_anything (GroundingDINO
+ SAM-HQ in one bundle) into custom_nodes and pip-installs its
requirements at build time. Model weights auto-download to the
comfyui-models volume on first inpaint (~3 GB one-time cost).
2. install-custom-node-deps.sh — entrypoint wrapper that pip-installs
requirements.txt for any custom_node present at startup. Lets users
add custom nodes via ComfyUI-Manager (or by git-cloning into the
volume) and have the deps picked up on the next restart, without
editing the Dockerfile.
3. smart_image_gen v0.6: edit_image gains a `mask_text` param. When
set, builds an inpainting workflow (LoadImage → GroundingDinoSAM
Segment → SetLatentNoiseMask → KSampler) so only the named region
is repainted. When unset, falls through to the existing img2img
path. Denoise default switches: 1.0 with mask_text (full repaint
within mask), 0.7 without.
4. Image Studio system prompt teaches the LLM the LOCAL vs GLOBAL
distinction — set mask_text whenever the user names a specific
object/region ('the ball', 'the dog', 'the sky'); leave it unset
only for whole-image style/lighting transformations.
5. Deployment README documents the new mode + the first-inpaint
weight-download caveat.
Image rebuild required — bump tag to pick up the Dockerfile change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
7c7897818e
commit
d935e24624
6 files changed
+189
-40
No files matched your search
@@ -17,7 +17,7 @@ production `srvno.de` deployment.
|
||||
| `init-models.sh` | LLMs to preseed into Ollama on first boot |
|
||||
| `mirror-ollama-model.sh` | Helper — mirror an Ollama model into a tarball you can host on S3 |
|
||||
| `comfyui-init-models.sh` | Checkpoints/VAEs/LoRAs to preseed into ComfyUI on first boot |
|
||||
| `openwebui-tools/smart_image_gen.py` | Tool that auto-routes image generation AND editing to the right SDXL checkpoint |
|
||||
| `openwebui-tools/smart_image_gen.py` | Tool that auto-routes image generation, img2img, and text-targeted inpainting to the right SDXL checkpoint |
|
||||
| `openwebui-models/image_studio.md` | Dedicated chat-model preset — manual setup walkthrough |
|
||||
| `openwebui-models/image_studio.json` | The same preset as an importable Open WebUI model JSON |
|
||||
| `.env.example` | Secrets and image-tag pins. Copy to `.env` |
|
||||
@@ -167,10 +167,19 @@ calls:
|
||||
|
||||
- **`generate_image`** for new images from scratch (txt2img).
|
||||
- **`edit_image`** for modifying an image the user attached to the
|
||||
chat (img2img).
|
||||
chat. Two modes:
|
||||
- With `mask_text` — text-targeted inpainting via GroundingDINO+SAM
|
||||
(e.g. "the dog's collar"). Only the named region is repainted.
|
||||
- Without `mask_text` — full img2img which reimagines the whole
|
||||
image at the requested denoise.
|
||||
|
||||
Both auto-route to the right SDXL checkpoint per request.
|
||||
|
||||
> **First inpaint takes a few minutes**: SAM-HQ (~2.5 GB) and
|
||||
> GroundingDINO (~700 MB) auto-download into the `comfyui-models`
|
||||
> volume on the very first call to `edit_image` with `mask_text`.
|
||||
> Subsequent inpaints are instant.
|
||||
|
||||
1. **Workspace -> Tools -> +** (top-right).
|
||||
2. Paste the contents of
|
||||
[`openwebui-tools/smart_image_gen.py`](openwebui-tools/smart_image_gen.py).
|
||||
|
||||
Reference in new issue
Block a user