WebMCP
webmcp is an MCP server that connects MCP clients to web search, page fetching, and local LLM-based extraction.
It’s ideal for research agents, coding assistants, and local automation stacks that need live search results, cleaned page text, and structured output from one or more URLs.
The MCP server runs on Python 3.10 or newer and expects a local OpenAI-compatible endpoint for extraction calls.
Features
- Searches the web through DuckDuckGo or SearXNG.
- Extracts structured data from one or more URLs.
- Fetches JavaScript-heavy pages through headless Chromium.
- Fetches simpler pages through a lightweight HTTP path.
- Sends cleaned page content to a local OpenAI-compatible chat completions endpoint.
Use Cases
- Run a multi‑step research task where the main agent searches for relevant pages, fetches their full text, and then extracts specific facts or fields using a smaller local extractor model.
- Build a custom news aggregator that searches for headlines, pulls article bodies, and structures the output as JSON records with title, date, author, and summary.
- Scrape product pages behind JavaScript‑heavy storefronts while providing the extractor LLM with a precise JSON schema to return only price, SKU, and availability status.
- Feed the extracted text from a batch of URLs directly into a summarization prompt without writing any intermediate parsing code.
- Audit search result quality or page fetch success rates by reviewing the
tool_calls.log.jsonfile after a long‑running agent session. - Replace DuckDuckGo with a private SearXNG instance to keep all search traffic inside a local network or custom search configuration.
How To Use It
1. Prepare the host components
| Component | Requirement | Notes |
|---|---|---|
| Python | 3.10+ | Required to run app.py. |
| Local extraction endpoint | OpenAI-compatible | Examples include llama.cpp, LM Studio, vLLM, and Ollama. |
| Main research host | llama.cpp WebUI | This is the documented tested host path. |
| Browser runtime | Playwright Chromium | Required for use_browser=True. |
2. Install the MCP server. The first command installs the Python dependencies from the pinned requirements file. The second command downloads Chromium for browser-mode page fetches.
pip install -r requirements.txt
python -m playwright install chromium3. Copy .env.example to .env, then set the runtime values:
LLM_URL=http://localhost:1234
LLM_MODEL=your-model-name
SEARCH_PROVIDER=ddg
# Optional when SEARCH_PROVIDER=searxng
SEARXNG_URL=http://localhost:8080| Variable | Required | Default | Purpose |
|---|---|---|---|
LLM_URL | Yes | None | Base URL for the local OpenAI-compatible endpoint. The extraction request posts to LLM_URL/v1/chat/completions. |
LLM_MODEL | Yes | None | Model name sent in the extraction request payload. |
SEARCH_PROVIDER | No | ddg | Selects ddg or searxng for search_web. |
SEARXNG_URL | Only for searxng | Empty | Base URL for the SearXNG instance. The server calls <SEARXNG_URL>/search with format=json. |
5. Start the main researcher host. The main llama.cpp researcher process must include --webui-mcp-proxy in its launch parameters. This workflow does not function correctly if that flag is missing. The documented workflow uses research_prompt.txt as the system prompt for the main research host.
6. Run webmcp:
python app.pyThe server starts on:
http://0.0.0.0:8642The entry point uses Uvicorn and serves the MCP app over Streamable HTTP on port 8642.
7. Connect your MCP client to the server. Use the Streamable HTTP endpoint at http://0.0.0.0:8642. The documented tested host path is llama.cpp WebUI. The code exposes the app with permissive CORS and no built-in authentication layer, so production exposure needs your own network controls or reverse proxy policy.
8. Available tools:
| Tool | Parameters | Return shape | Notes |
|---|---|---|---|
get_current_date() | None | UTC date string | The return format is YYYY-MM-DD (Weekday). |
search_web(query, limit=10) | query: str, limit: int = 10 | JSON string | Each item includes title, url, and description. |
extract(urls, prompt=None, schema=None, use_browser=True) | urls: list[str], prompt: str | None, schema: dict | None, use_browser: bool = True | String | The call requires at least one of prompt or schema. |
FAQs
Q: What breaks if I skip --webui-mcp-proxy?
A: The main researcher workflow does not function correctly if that flag is missing from the llama.cpp server launch parameters.
Q: Do I need both prompt and schema for extract?
A: No. extract accepts either one. The tool rejects the call only when both values are empty. A focused prompt works well for free-form extraction. A schema works well for strict JSON output.
Q: When should I switch use_browser to false?
A: Set use_browser=false for simple static pages such as plain docs, changelogs, or lightweight marketing pages. Keep use_browser=true for pages that load key content through JavaScript after initial navigation.
Q: Does webmcp include authentication or network hardening?
A: The default server code does not implement authentication. The app binds to 0.0.0.0, enables permissive CORS, and disables DNS rebinding protection. A public deployment needs external access controls.
Q: Why does extraction return weak or empty results on some pages?
A: The weak result may start at the fetch layer, the readability pass, the target page markup, or an anti-bot block. The current project notes put the rough unusable-content rate for JS page rendering in the 25 to 30 percent range, and some targets still return 400-range errors. A tighter prompt or stricter schema can help when the page text itself is usable.
Q: Where can I inspect recent tool activity?
A: Check tool_calls.log.json in the server directory. The log stores the last 10 entries with UTC timestamp, tool name, arguments, and result.
Latest MCP Servers
WebMCP
webmcp is an MCP server that connects MCP clients to web search, page fetching, and local LLM-based extraction. It’s ideal…
Google Meta Ads GA4
Token Savior Recall
Featured MCP Servers
Notion
Claude Peers
Excalidraw
FAQs
Q: What exactly is the Model Context Protocol (MCP)?
A: MCP is an open standard, like a common language, that lets AI applications (clients) and external data sources or tools (servers) talk to each other. It helps AI models get the context (data, instructions, tools) they need from outside systems to give more accurate and relevant responses. Think of it as a universal adapter for AI connections.
Q: How is MCP different from OpenAI's function calling or plugins?
A: While OpenAI's tools allow models to use specific external functions, MCP is a broader, open standard. It covers not just tool use, but also providing structured data (Resources) and instruction templates (Prompts) as context. Being an open standard means it's not tied to one company's models or platform. OpenAI has even started adopting MCP in its Agents SDK.
Q: Can I use MCP with frameworks like LangChain?
A: Yes, MCP is designed to complement frameworks like LangChain or LlamaIndex. Instead of relying solely on custom connectors within these frameworks, you can use MCP as a standardized bridge to connect to various tools and data sources. There's potential for interoperability, like converting MCP tools into LangChain tools.
Q: Why was MCP created? What problem does it solve?
A: It was created because large language models often lack real-time information and connecting them to external data/tools required custom, complex integrations for each pair. MCP solves this by providing a standard way to connect, reducing development time, complexity, and cost, and enabling better interoperability between different AI models and tools.
Q: Is MCP secure? What are the main risks?
A: Security is a major consideration. While MCP includes principles like user consent and control, risks exist. These include potential server compromises leading to token theft, indirect prompt injection attacks, excessive permissions, context data leakage, session hijacking, and vulnerabilities in server implementations. Implementing robust security measures like OAuth 2.1, TLS, strict permissions, and monitoring is crucial.
Q: Who is behind MCP?
A: MCP was initially developed and open-sourced by Anthropic. However, it's an open standard with active contributions from the community, including companies like Microsoft and VMware Tanzu who maintain official SDKs.
