smart_image_gen v0.3: add edit_image (img2img) method

The Tool now exposes two methods the LLM picks between based on whether
the user attached an image:

  generate_image — txt2img (existing, unchanged behavior)
  edit_image     — img2img on the most recently attached image

edit_image extracts the source image from __messages__ (base64 data
URIs in image_url content blocks) or __files__ (local path or URL),
uploads to ComfyUI's /upload/image, runs an img2img workflow at the
caller-specified denoise (default 0.7), and returns the edited result.
Same per-style routing / sampler / CFG / prefix logic as generation.

Refactored the submit-and-poll loop into _submit_and_fetch shared by
both methods. Image extraction is defensive — tries messages first,
then files (path then URL), returns a clear "no image attached"
message rather than silently generating from scratch.

Image Studio system prompt rewritten to teach the LLM when to call
edit_image vs generate_image and how to pick denoise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
57_WolveandClaude Opus 4.7 committed 2026-04-19 12:59:13 -05:00
1 parent 41d571d8d1
commit d4e2058859
3 files changed
+342 -117

No files matched your search

+9 -3
View File
@@ -16,7 +16,7 @@ production `srvno.de` deployment.
| `Caddyfile` | TLS + reverse proxy config (one site block per hostname) |
| `init-models.sh` | LLMs to preseed into Ollama on first boot |
| `comfyui-init-models.sh` | Checkpoints/VAEs/LoRAs to preseed into ComfyUI on first boot |
| `openwebui-tools/smart_image_gen.py` | Tool that auto-routes image generation to the right SDXL checkpoint |
| `openwebui-tools/smart_image_gen.py` | Tool that auto-routes image generation AND editing to the right SDXL checkpoint |
| `openwebui-models/image_studio.md` | Dedicated chat-model preset — system prompt that forces tool use |
| `.env.example` | Secrets and image-tag pins. Copy to `.env` |
@@ -153,8 +153,14 @@ use the edit action.
The image-button path always uses the admin's **Default Model**. To get
per-prompt checkpoint routing — e.g. "draw me a cyberpunk city" picks
CyberRealistic, "anthro fox warrior" picks one of the furry checkpoints —
install the `smart_image_gen.py` Tool. The LLM calls it instead of the
built-in image action and chooses the right SDXL checkpoint per request.
install the `smart_image_gen.py` Tool. It exposes two methods the LLM
calls:
- **`generate_image`** for new images from scratch (txt2img).
- **`edit_image`** for modifying an image the user attached to the
chat (img2img).
Both auto-route to the right SDXL checkpoint per request.
1. **Workspace -> Tools -> +** (top-right).
2. Paste the contents of