Turn Sketches into 3D Models with AI – Vibe Draw

Vibe Draw is an open-source AI-driven sketching and illustration tool that assists artists, designers, and hobbyists in creating high-quality digital artwork. It bills itself as a “Cursor for 3D Modeling,” aiming to make 3D creation more intuitive, kind of like how AI code assistants speed up development.

This tool converts rough 2D sketches into 3D models using AI, then lets you assemble them into interactive scenes. If you’ve ever wanted to prototype a game level, visualize an idea, or just experiment with 3D without learning Blender, this is worth a look.

Features

• 2D-to-3D Conversion: Sketch loosely, refine with AI, and generate 3D models in a few clicks.
• Live Editing: Tweak models by redrawing or using text prompts (e.g., “make this building taller”).
• Export to Standard Formats: .glTF support means you can pull models into Unity, Unreal, or other tools.
• Real-Time Collaboration: Multiple users can edit the same 3D world simultaneously.
• Trustworthy Model: Powered by Microsoft TRELLIS, a large 3D asset generation model. 

Who’s It For?

• Indie game devs mocking up level designs
• Designers creating quick 3D mockups without CAD software
• Educators teaching basic 3D concepts without complex tooling
• Hobbyists exploring 3D modeling casually

How to use it

You’ll need Node.js (18+) and Python (3.10+) installed. The frontend is Next.js/React, standard npm install and npm run dev. The backend uses FastAPI and Celery, managed via Docker Compose.

1. Clone the repo from GitHub and navigate to backend.

cd backend

2. Copy .env.example to .env, and then add your actual API keys for Claude, Gemini, Cerebras, and TRELLIS.

ANTHROPIC_API_KEY=
GOOGLE_API_KEY=
CEREBRAS_API_KEY=
TRELLIS_API_KEY=

3. Run docker compose up gets the backend services (API, task queue, Redis) going.

4. Setup frontend:

cd frontend
npm install
npm run dev

5. Open your browser to http://localhost:3000 (or whatever port Next.js tells you).

6. Start Creating:

• Use the 2D Canvas tab to draw something.
• Click “Improve Drawing” (optional, experiment with/without it).
• Click “Make 3D”. Wait for the process to complete (watch for updates).
• Go to the 3D World tab. Your model should appear in a sidebar; drag it into the scene.
• Click a model in the scene to select it. Use the sketch overlay or the text prompt input at the bottom to iterate.
• When ready, click the Export button (usually top right) to get your .glTF file.

See It In Action

Related Resources

• Vibe Draw Source Code: This is where you’ll get the code and installation instructions. https://github.com/martin226/vibe-draw
• Three.js Documentation: Useful if you plan to use the exported .glTF files in web projects. https://threejs.org/docs/
• TLDraw: The 2D drawing library used. Interesting to see what it can do. https://www.tldraw.com/
• FastAPI Documentation: For understanding the backend API structure if you want to dig deeper. https://fastapi.tiangolo.com/
• AGPL License Text: Understand the terms before using or modifying the code, especially for commercial purposes. https://www.gnu.org/licenses/agpl-3.0.en.html

FAQs

Q: Is Vibe Draw completely free to use?
A: The Vibe Draw software itself is free and open-source under the AGPL license. However, running it requires API keys for external AI services (Claude, Gemini, etc.). Depending on your usage volume and the pricing tiers of those services, you might incur costs from the API providers. So, the code is free, but running it might not be.

Q: How good are the 3D models it generates? Can I use them directly in a game?
A: The quality varies. For simple, cartoonish, or low-poly styles, the results can be quite usable, especially for prototypes or background elements. Expect to do some iteration (using sketches or prompts within Vibe Draw) or cleanup in another tool for more refined results. The topology might not always be optimal for complex animation, but for static meshes or simple movements, it could work.

Q: What does the AGPL license mean for my project?
A: AGPL is a copyleft license. If you modify the Vibe Draw source code and make that modified version accessible over a network (like running it as a web service for others), you’re generally obligated to share your modified source code under the same AGPL license. Using the tool internally or using its output (.glTF files) in your project usually doesn’t trigger this sharing requirement, but consult the license or legal advice for specifics, especially if you plan commercial distribution.

Q: Do I need to be an artist or a pro developer to use Vibe Draw?
A: Not really. Basic sketching ability helps get your initial idea down. On the technical side, you need to be comfortable with command-line tools, installing software (Node, Python), managing API keys, and running Docker Compose. It’s aimed more at tech-savvy users than complete beginners, but you don’t need to be a senior dev or a 3D expert.