io.github.Decodo/mcp-web-scraper
搜索与获取by decodo
Enable your AI agents to scrape and parse web content dynamically, including geo-restricted sites
什么是 io.github.Decodo/mcp-web-scraper?
Enable your AI agents to scrape and parse web content dynamically, including geo-restricted sites
README
Decodo MCP Server
<p align="center"> <a href="https://dashboard.decodo.com/integrations?utm_source=github&utm_medium=social&utm_campaign=mcp_server"> <img src="https://github.com/user-attachments/assets/a1e52a9e-3da1-4081-b3c6-053aafb8f196"/></a>Connect LLMs and AI agents to live web data using MCP (Model Context Protocol). The Decodo MCP Server lets you scrape websites, search engines, eCommerce platforms, and social media directly from AI tools like Claude, Cursor, and Windsurf, all without the need to build scraping infrastructure from scratch.
- Structured outputs in JSON, Markdown, and screenshots
- Server-side JavaScript rendering and anti-bot handling
- 125M+ IPs across 195+ locations
What is Decodo MCP server?
The Decodo MCP Server is a web scraping layer for AI agents. It connects MCP-compatible clients to Decodo's Web Scraping API, enabling:
- Web scraping for LLMs
- Real-time data retrieval for RAG
- AI agent browsing and research
- Structured data extraction from dynamic websites
Instead of maintaining proxies, parsers, and retry logic, you get a single integration point for reliable web data access.
Why use MCP for web scraping?
Model Context Protocol (MCP) is the emerging standard for connecting AI agents to external tools and data sources. With MCP:
- Agents can call tools dynamically
- Integrations stay standardized
- Workflows scale across environments
The Decodo MCP Server gives your agents reliable, production-ready web access through this standard.
Key features
Web scraping for AI agents, no infrastructure required. Scrape any website, including JavaScript-heavy pages, without handling proxy rotation, CAPTCHA solving, or anti-bot systems.
Structured outputs for LLM workflows. Markdown (LLM-ready), JSON (for structured pipelines), and screenshots (for visual context), built for RAG pipelines, AI research agents, and automation flows.
Built-in support for popular targets. Ready-made tools for Google and Bing (SERPs), Amazon, Walmart, and Target (eCommerce), Reddit, TikTok, and YouTube (social media), and ChatGPT and Perplexity (AI search).
Global proxy infrastructure. 125M+ residential IPs, 195+ geo-locations, and a 99.99% success rate on even the most protected targets.
Modular MCP toolsets. Enable only what you need: web, search, ecommerce, social_media,
ai for cleaner tool selection and better agent performance.
Fast time to value. From API key to first scrape in minutes, no setup overhead.
Use cases
Use the Decodo MCP Server when you need web scraping for AI agents, structured data extraction at scale, reliable access to dynamic websites, real-time data for RAG, or an alternative to building scraping infrastructure from scratch. Common scenarios:
- AI-powered web scraping – give LLMs the ability to collect fresh data instead of relying on static training data.
- RAG with live data – pull real-time Google, Bing, and AI search results into retrieval pipelines.
- eCommerce intelligence – track product prices, listings, and sellers across marketplaces without getting blocked.
- Social media data collection – gather posts, channels, and engagement data from Reddit, TikTok, and YouTube.
- Travel and price aggregation – build tools that collect live pricing and availability across websites.
Quick start
- Create a free account at dashboard.decodo.com – up to 2K free requests, no credit card required.
- Get your API key. Obtain a Web Scraping API basic authentication token from the dashboard.
- Download Node.js 18+ from https://nodejs.org.
- Get MCP client like Claude Desktop, Curson, Windsurf or other MCP-compatible tools..
- Configure the MCP server in your AI client (see configuration examples below).
Connecting to Decodo's MCP server
Open your preferred MCP client and add the following configuration (see examples for Claude Code, Cursor, Windsurf bellow):
{
"mcpServers": {
"Decodo": {
"url": "https://mcp.decodo.com/mcp",
"headers": {
"Authorization": "Basic <basic_auth_token>"
}
}
}
}
Claude Desktop
- Open Claude Desktop → Settings → Developer → Edit Config.
- Add to claude_desktop_config.json:
{
"mcpServers": {
"Decodo": {
"command": "npx",
"args": ["-y", "@decodo/mcp-server"],
"env": {
"SCRAPER_API_TOKEN": "<basic_auth_token>",
"TOOLSETS": "web,ai"
}
}
}
}
- Save and restart Claude Desktop.
Cursor
- Open Settings → MCP.
- Click Add a new global MCP server (opens mcp.json).
- Add the same configuration as above.
- Save — look for a green status indicator next to Decodo.
Windsurf
- Open Settings → Windsurf Settings.
- Scroll to Cascade → Add custom server + (opens mcp_config.json).
- Add the same configuration as above.
- Save and restart Windsurf.
Test your setup
Once connected, try this prompt in your client:
▎ "Scrape the titles of the top 5 articles from Hacker News"
You should get a structured list back within seconds. If you see an auth error, double-check your token from the dashboard.
Optional: enable specific toolsets
Running the MCP server locally
<details>Prerequisites
- Node.js 18.0+
- An MCP client - popular choices are Claude Desktop and Cursor
Step-by-step guide
- Clone this repository:
git clone https://github.com/Decodo/mcp-server
- Run the following commands in the terminal:
cd decodo-mcp-server
npm install
npm run build
- Take note of your build location:
cd build/
pwd
Adding index.js to the end of this directory, your build file location should look something like
this:
/Users/your.user/projects/decodo-mcp/build/index.js
- Update your MCP client with the server information:
{
"mcpServers": {
"decodo-mcp": {
"command": "node",
"args": ["/Users/your.user/projects/decodo-mcp/build/index.js"],
"env": {
"SCRAPER_API_TOKEN": "<web_scraping_api_base64_token>"
}
}
}
}
Toolsets
Tools are organized into toolsets. You can selectively enable specific toolsets by passing a
comma-separated list via the toolsets query parameter:
"Decodo MCP Server": {
"url": "https://mcp.decodo.com/mcp?toolsets=web,ai",
"headers": {
"Authorization": "Basic <your_auth_token>"
}
}
When no toolsets are specified, all tools are registered.
| Toolset | Tools |
|---|---|
web | scrape_as_markdown, screenshot |
search | google_search, google_ads, google_lens, google_travel_hotels, bing_search |
ecommerce | amazon_search, amazon_product, amazon_pricing, amazon_sellers, amazon_bestsellers, walmart_search, walmart_product, target_search, target_product, tiktok_shop_search, tiktok_shop_product, tiktok_shop_url |
social_media | reddit_post, reddit_subreddit, reddit_user, tiktok_post, youtube_metadata, youtube_channel, youtube_subtitles, youtube_search |
ai | chatgpt, perplexity, google_ai_mode |
Tools
The server exposes the following tools:
| Tool | Description | Example prompt |
|---|---|---|
scrape_as_markdown | Scrapes any target URL, expects a URL to be given via prompt. Returns results in Markdown. | Scrape peacock.com from a US IP address and tell me the pricing. |
screenshot | Captures a screenshot of any webpage and returns it as a PNG image. | Take a screenshot of github.com from a US IP address. |
google_search | Scrapes Google Search for a given query, and returns parsed results. | Scrape Google Search for shoes and tell me the top position. |
google_ads | Scrapes Google Ads search results. | Scrape Google Ads for laptop and show me the top ads. |
google_lens | Scrapes Google Lens image search results. | Search Google Lens for this image: https://example.com/image.jpg |
google_ai_mode | Scrapes Google AI Mode (Search with AI) results. | Ask Google AI Mode: What are the top three dog breeds? |
google_travel_hotels | Scrapes Google Travel Hotels search results. | Search Google Travel Hotels for hotels in Paris. |
amazon_search | Scrapes Amazon Search for a given query, and returns parsed results. | Scrape Amazon Search for wireless keyboard. |
amazon_product | Scrapes Amazon Product page. | Scrape Amazon product B09H74FXNW and show me the details. |
amazon_pricing | Scrapes Amazon Product pricing information. | Get pricing for Amazon product B09H74FXNW. |
amazon_sellers | Scrapes Amazon Seller information. | Get information about Amazon seller A1R0Z7FJGTKESH. |
amazon_bestsellers | Scrapes Amazon Bestsellers list. | Show me Amazon bestsellers in electronics. |
walmart_search | Scrapes Walmart Search for a given query, and returns parsed results. | Scrape Walmart Search for camping tent. |
walmart_product | Scrapes Walmart Product page. | Scrape Walmart product 15296401808. |
target_search | Scrapes Target Search for a given query, and returns parsed results. | Scrape Target Search for kitchen appliances. |
target_product | Scrapes Target Product page. | Scrape Target product 92186007. |
tiktok_post | Scrapes a TikTok post URL for structured data (e.g. engagement, caption, hashtags). | Scrape this TikTok post: https://www.tiktok.com/@nba/video/7393013274725403950 |
tiktok_shop_search | Scrapes TikTok Shop Search for a given query, and returns parsed results. | Scrape TikTok Shop Search for phone cases. |
tiktok_shop_product | Scrapes TikTok Shop Product page. | Scrape TikTok Shop product 1731541214379741272. |
tiktok_shop_url | Scrapes TikTok Shop page by URL. | Scrape this TikTok Shop URL: https://www.tiktok.com/shop/s?q=HEADPHONES |
youtube_metadata | Scrapes YouTube video metadata. | Get metadata for YouTube video dFu9aKJoqGg. |
youtube_channel | Scrapes YouTube channel videos. | Scrape YouTube channel @decodo_official. |
youtube_subtitles | Scrapes YouTube video subtitles. | Get subtitles for YouTube video L8zSWbQN-v8. |
youtube_search | Search YouTube videos. | Search YouTube for "How to care for chinchillas". |
reddit_post | Scrapes a specific Reddit post. | Scrape the following Reddit post: https://www.reddit.com/r/horseracing/comments/1nsrn3/ |
reddit_subreddit | Scrapes Reddit subreddit results. | Scrape the top 5 posts on r/Python this week. |
reddit_user | Scrapes a Reddit user profile and their posts/comments. | Scrape this Reddit user: https://www.reddit.com/user/IWasRightOnce/ |
bing_search | Scrapes Bing Search results. | Search Bing for laptop reviews. |
chatgpt | Search and interact with ChatGPT for AI-powered responses and conversations. | Ask ChatGPT to explain quantum computing in simple terms. |
perplexity | Search and interact with Perplexity for AI-powered responses and conversations. | Ask Perplexity what the latest trends in web development are. |
Parameters
The following parameters are inferred from user prompts:
| Parameter | Description |
|---|---|
jsRender | Renders target URL in a headless browser. |
geo | Sets the country from which the request will originate. |
locale | Sets the locale of the request. |
tokenLimit | Truncates the response content up to this limit. Useful if the context window is small. |
prompt | Prompt to send to AI tools (chatgpt, perplexity). |
search | Activates ChatGPT's web search functionality (chatgpt only). |
xhr | When true, includes XHR or fetch responses in the scrape result where supported (e.g. tiktok_post). |
deviceType | Device type to emulate for the request (desktop, mobile, tablet). |
domain | Domain to use for the request (e.g., amazon.com, amazon.co.uk, bing.com). |
pageFrom | Starting page number for pagination. |
deliveryZip | ZIP code for delivery location (Target, Walmart). |
storeId | Store ID for local inventory (Target, Walmart). |
country | Country for TikTok Shop requests. |
limit | Maximum number of results to return (e.g., YouTube channel videos). |
language_code | Language code for subtitles (e.g., en, es). |
Examples
Scraping geo-restricted content
Query your AI agent with the following prompt:
Scrape peacock.com from a German IP address and tell me the pricing.
This prompt will say that peacock.com is geo-restricted. To bypass the geo-restriction:
Scrape peacock.com from a US IP address and tell me the pricing.
Limiting number of response tokens
If your agent has a small context window, the content returned from scraping will be automatically truncated, in order to avoid context-overflow. You can increase the number of tokens returned within your prompt:
Scrape hacker news, return 50k tokens.
If your agent has a big context window, tell it to return full content:
Scrape hacker news, return full content.
Related repositories
Web Scraping API, Decodo OpenClaw skill
Try it
Plug Decodo MCP Server into your AI workflow in just a few clicks and equip your AI agents with real-time data from any website.
Start for free | Docs | Discord
License
All code is released under the MIT License.
常见问题
io.github.Decodo/mcp-web-scraper 是什么?
Enable your AI agents to scrape and parse web content dynamically, including geo-restricted sites
相关 MCP Server
网页抓取
编辑精选by Anthropic
Fetch 是 MCP 官方参考服务器,让 AI 能抓取网页并转为 Markdown 格式。
✎ 这个服务器解决了 AI 直接处理网页内容时格式混乱的问题,适合需要让 Claude 分析在线文档或新闻的开发者。不过作为参考实现,它缺乏生产级的安全配置,你得自己处理反爬虫和隐私风险。
Puppeteer 浏览器控制
编辑精选by Anthropic
Puppeteer 是让 Claude 自动操作浏览器进行网页抓取和测试的 MCP 服务器。
✎ 这个服务器解决了手动编写 Puppeteer 脚本的繁琐问题,适合需要自动化网页交互的开发者,比如抓取动态内容或做端到端测试。不过,作为参考实现,它可能缺少生产级的安全防护,建议在可控环境中使用。
Brave 搜索
编辑精选by Anthropic
Brave Search 是让 Claude 直接调用 Brave 搜索 API 获取实时网络信息的 MCP 服务器。
✎ 如果你想让 AI 助手帮你搜索最新资讯或技术文档,这个工具能绕过传统搜索的限制,直接返回结构化数据。特别适合需要实时信息的开发者,比如查 API 更新或竞品动态。不过它依赖 Brave 的 API 配额,高频使用可能受限。