BrushEdit is an open-source AI agent for advanced, unified image inpainting and editing through intelligent, instruction-based manipulation.
Developed by a collaborative effort between Tencent, Peking University, The Chinese University of Hong Kong, and Tsinghua University, this AI agent addresses critical challenges faced by content creators, designers, and digital artists who struggle with complex image editing tasks.
BrushEdit is great for a variety of image-editing scenarios. Here are some examples:
How It Works
BrushEdit operates through four main steps to achieve instruction-based image editing.
First, it classifies the required editing type, such as adding, deleting, or modifying content.
Next, the primary object within the image that needs editing is identified.
After this, BrushEdit acquires both the editing mask and the corresponding target caption, using pre-trained MLLMs and detection models.
Finally, BrushEdit performs the actual image inpainting, utilizing a dual-branch inpainting model called BrushNet. This model inpaints target areas based on both the target caption and editing masks, leveraging the generative potential of inpainting models while preserving background details.
Features
- Aspect Ratio Selection: Choose from various aspect ratios, with a maximum resolution of 1024px to avoid memory issues.
- VLM Model Selection: Select from preloaded VLM models for faster processing. Custom models can be used by downloading and uncommenting lines in
vlm_template.pyfrom the GitHub repository. - Mask Generation: Generate masks based on input instructions, identifying areas needing modification.
- Square/Circle Masking: Create coarse-grained masks for squares and circles for more flexible editing options.
- Mask Inversion: Invert existing masks to focus on different areas of an image.
- Mask Dilation/Erosion: Expand or shrink masks to include or exclude specific image regions.
- Mask Movement: Move masks to new positions for precise adjustments.
- Target Prompt Generation: Automatically generate a target prompt from instructions. Manual modification is available.
- Blending: Combine the edited output with the original input to preserve details in unedited areas, particularly helpful for complex edits.
- Control Length: Adjust the intensity of editing and inpainting.
- Base Model Selection: Select from preloaded models. Custom models can be used by downloading and uncommenting lines in
vlm_template.pyfrom the GitHub repository. - Number of Samples: Generate multiple samples, allowing users to choose the best result.
- Negative Prompt: Input negative prompts for classifier-free guidance to refine results.
- Guidance Scale: Adjust the guidance scale for classifier-free guidance to influence the edit’s intensity.
Playground
How To Use It
1. Access BrushEdit’s playground on this page or through its Hugging Face Space.
2. The demo currently offers two modes: fully automated and interactive.
Fully Automated Editing:
- Upload an image or select an example.
- Enter your editing instructions, including adding, deleting, or modifying content. For example, “remove the xxx”.
- Click the “Run” button to automatically apply edits.
Interactive Editing:
- Upload an image or select an example.
- Use the brush tool to outline the area you want to edit. You can use the eraser to restore parts of the mask.
- Input your instructions.
- Click the “Run” button to automatically edit the image.
Additional Tips:
- After entering instructions, use the “Generate Mask” button to preview the mask, which can be further customized.
- Perform operations such as randomization, dilation, erosion, and movement to fine-tune the mask.
- Use the “Generate Target Prompt” to preview the automatically generated prompt and modify it as needed.
- For optimal results, use GPT-4o with an API key or the Qwen2VL model.
Related Resources
- Paper: https://arxiv.org/abs/2412.10316
- Hugging Model: https://huggingface.co/TencentARC/BrushEdit
- Github Repo: https://github.com/TencentARC/BrushEdit
FAQs
Q: What is the maximum resolution for images in BrushEdit?
A: The maximum resolution is 1024px to prevent out-of-memory errors.
Q: Can I use my own VLM models with BrushEdit?
A: Yes, you can download custom VLM models and uncomment the relevant lines in vlm_template.py from the GitHub repository.
Q: Does BrushEdit support both automated and interactive editing?
A: Yes, BrushEdit offers both fully automated and interactive command-based editing modes.
Q: Is BrushEdit free to use?
A: Yes, BrushEdit is completely free.
Q: What if the VLM generated target prompt does not meet expectations?
A: You can manually modify the generated target prompt to better suit your needs.
