Build Custom GPTs From Any Websites – GPT Crawler

Turn any websites into AI knowledge with GPT Crawler. Create tailored GPTs effortlessly. Start now!

GPT Crawler is a TypeScript library that transforms any website into a valuable knowledge base.

It crawls the site(s) you specify to extract key information and formats it into a JSON file that can be used in your own custom GPTs or OpenAI Assistant API. You will get an intelligent agent with domain-specific knowledge from any site.

Subscribe to our newsletter and get the top 10 AI tools and apps delivered straight to your inbox. Subscribe now!

How to use it:

1. To get started, make sure you have Node.js (>= 16) and Playwright installed.

2. Clone the GPT Crawler repo from GitHub and install necessary dependencies.

git clone https://github.com/builderio/gpt-crawler
npm i

3. Edit the config.ts file to specify the URL and selectors for the content you wish to crawl.

export const config: Config = {
  url: "/path/to/",
  match: "/path/to/docs/**",
  selector: `.main-container`,
  maxPagesToCrawl: 50, // max number of pages to crawl
  outputFileName: "knowledge.json",
};

4. Run the crawler using npm start.

5. Upload the knowledge.json to your custom GPTs or OpenAI Assistant.

GPT Crawler Knowledge Upload

6. Here is a custom GPT demonstrating how to scrape data from builder.io and create an AI assistant.

Leave a Reply

Your email address will not be published. Required fields are marked *

Get the latest & top AI tools sent directly to your email.

Subscribe now to explore the latest & top AI tools and resources, all in one convenient newsletter. No spam, we promise!