JiamingShi-JsPhys/web-forager ↗★ 2
web-forager
Clean-room, robots-aware web information gathering: multi-engine search, HTML/PDF/Office/feed extraction, Bilibili/Zhihu/Xiaohongshu reading, sitemaps and resumable surfing, plus a browser mode with seven read-only playbooks. CLI + two MCP servers.
安装
npx -p @deepseek-ai/dsh dsh plugin --profile web add github:JiamingShi-JsPhys/web-forager说明文档
阅读完整 README ↗web-forager
A clean-room, robots-aware web information gathering toolkit for agents and analysts: one engine behind a CLI, two MCP servers and a Claude Code skill.
Disclaimer (read this first)
This plugin is for learning and information gathering only. Users must comply with the Terms of Service of target websites and applicable copyright law. Malicious concurrent attacks are prohibited. Collecting, aggregating or inferring the personal information of an identifiable individual — or using it for doxxing, harassment, stalking or any other invasion of privacy — is prohibited; the user bears full legal responsibility for such use.
English · 简体中文
Version: 1.2.2 | CLI actions: 9 | MCP tools: 10 | Browser playbooks: 7
What this tool is for
web-forager automates reading publicly reachable pages for research, monitoring and personal knowledge work. It is provided for educational and research purposes only.
What it must not be used for
- Identifying, locating, profiling or contacting a private individual, or assembling scattered records into a picture of a person.
- Harassment, stalking, intimidation, doxxing, or any invasion of another person's privacy.
- Bypassing logins, CAPTCHAs, paywalls or access controls, or placing load on a site beyond the built-in rate limit.
- Any use that breaks the law where you are, or the terms of the site you are reading.
Your responsibility
It is the end user's responsibility to obey all applicable local, state and federal laws and the Terms of Service and copyright of every site reached. The authors do not grant permission to access anything you are not entitled to read, and do not verify that a given use is lawful.
No warranty
The software is provided "as is", without warranty of any kind, express or implied, as stated in the MIT License. It may fail, return incomplete or outdated content, or change behaviour between versions.
Limitation of liability
To the maximum extent permitted by law, the authors and contributors assume no liability and are not responsible for any misuse of this software or for any damage, loss or legal consequence arising from it. Fetched text comes from third-party sites and is not endorsed or checked by this project.
If you redistribute it
Keep this disclaimer and the licence notice with any copy you publish or fork.
Scope and compliance
The core engine is an original clean-room implementation: no source is copied from Crawlee, Scrapy, Colly, Katana or any other crawling framework, and no GPL/AGPL dependency is used. The tool never bypasses a login, CAPTCHA, Cloudflare challenge or paywall, does not run concurrent request floods and does not target personal data. robots.txt (RFC 9309 subset) is checked before every target page, each request carries a contactable information-gathering User-Agent, and the default rate is one request per second per origin — search, sitemap and surf hops included. Fetched text comes from third-party sites; this plugin is not responsible for that content or for any consequence of using it. A query that asks to identify a private person — an explicit doxxing phrase, a name paired with a contact field, or someone's address, phone or ID — is refused with prohibited_query (exit code 10) before any request is sent.
Two modes, one record
| HTTP mode | Browser mode | |
|---|---|---|
| Reads | what the server returns | what a person sees once scripts run and login state applies |
| Entry points | run.py, mcp_server.py | browser/run.js, browser/mcp_server.js |
| Needs | Python 3.10+ | Node.js 18+ and a bridge command |
| Covers | search, HTML/PDF/Office/feeds, sitemaps, surf, Bilibili, scholarly APIs | seven read-only playbooks, rendered pages, danmaku |
Both modes agree on four core fields — metadata, markdown, tables and warnings — and extend them differently. --core on either CLI, or core: true on browser_run, folds a record onto one documented shape (Records).
Only Bilibili has a site-specific HTTP adapter. Zhihu, Xiaohongshu, Weibo and Douban are read as ordinary pages in HTTP mode; Zhihu and Xiaohongshu add browser playbooks, and Weibo and Douban have none.
How a run flows
Every request enters through the same HttpFetcher: robots policy, cache, retry/backoff, per-origin rate limiter and size cap. The chosen action then routes the payload to the matching extractor, and all extractors converge on the same Markdown plus metadata record.
flowchart LR
IN["URL / keyword / DOI / file"] --> SEL{"choose action"}
SEL -->|one page| F["fetch"]
SEL -->|one site| FD["find / feed"]
SEL -->|local file| C["clean"]
SEL -->|papers| S["scholar"]
SEL -->|site inventory| M["map"]
SEL -->|exploration| W["surf"]
SEL -->|Bilibili| B["bili"]
F --> HTTP["robots.txt check
per-origin rate limit"]
FD --> HTTP
C --> EXT["HTML / PDF / DOCX / XLSX / EPUB / CSV"]
S --> API["arXiv / Crossref / PubMed"]
M --> SM["robots Sitemap / sitemap index / .xml.gz"]
W --> HTTP
B --> HTTP
HTTP --> EXT
EXT --> MD["Markdown + metadata + tables + chunks"]
MD --> OUT["JSON / --agent envelope / --root workspace / MCP"]
surf only follows `` links that already exist in the page; it never constructs guessed URLs, submits forms or executes page JavaScript. map reads only published sitemaps, and scholar calls documented APIs instead of scraping result pages.
Browser mode is a second loop beside this one: the bridge supplies a page, browser/run.js reads a DOM snapshot, the session decides what is new, the playbook picks the next navigation step and a capture is written. Only one state-changing action runs per observation cycle, and write-like controls are denied by default.
Installation
Install it as one plugin: the CLI, both MCP servers and SKILL.md share a single engine, so splitting the tree breaks it.
Method 1: install from the plugin marketplace by URL
Every host below has its own plugin marketplace. Register this repository's Git URL, then install the plugin it lists (registering the URL twice is harmless — the second call reports it is already known).
https://github.com/JiamingShi-JsPhys/web-forager
Method 2: install with commands
Claude Code, Codex, dsh and CodeBuddy take a one-line plugin command; Cursor, Windsurf, Cline, Continue, Roo Code, GitHub Copilot and opencode read a rule file, and one script writes all of them. Every command, every flag and the optional dependencies live on one page: docs/INSTALL_COMMANDS.md.
Method 3: let your AI install it
Please visit the GitHub repository `https://github.com/JiamingShi-JsPhys/web-forager` and clone it, then run `python install.py --project . --global`. If your host has a plugin system, install it as a single plugin instead; the script covers the hosts that read rule files.
Warning: when an AI performs the installation it will very likely scatter
scripts/,run.pyandSKILL.mdinto unrelated folders. The modules import each other and the MCP server assumes the plugin-relative layout, so a scattered copy stops working. Use Method 1 if you need the full plugin.
Quick start
Install the plugin, then stay in the chat box: type / and pick a skill or a command. No shell, no absolute path — the folder you are working in is the workspace.
Skills
The skill is registered under the web-forager namespace, so it is one / away:
/web-forager:web-forager collect https://www.bilibili.com/video/BV1xx411c7mD as Markdown
Say what you want after the name; the skill picks the action and returns Markdown, metadata, tables and warnings. A page that robots.txt disallows is refused, not fetched.
Commands
| Command | Does |
|---|---|
/web-forager:fetch | Fetches one URL, or searches a keyword and reads the results |
/web-forager:surf | Searches or opens a seed URL, then follows ranked in-page links with a resumable session |
/web-forager:scholar | Queries arXiv, Crossref or PubMed, or resolves a DOI |
/web-forager:map | Discovers page URLs from robots.txt Sitemap lines, sitemap indexes and .xml.gz files |
/web-forager:browse | Runs a read-only browser playbook through the browser MCP server |
/web-forager:fetch https://example.org/article
/web-forager:scholar 10.1038/nature14539
When the engines are blocked
Search defaults to auto. A mostly-CJK query goes to 360, Sogou and Sogou WeChat — the article index is the one channel that normally carries a Chinese person's or institution's name — and is not handed to Bing, which splits such a phrase into single characters and answers with dictionary pages. Those pages are dropped from any engine's results. An agent that wants Bing for a Chinese query can still ask with --engine bing.
Two routes need no search engine at all:
python run.py --action find --site the-decoder.com --query "OpenAI watermark" --max-pages 3
python run.py --action feed --url https://the-decoder.com/ --num-results 10
find tries the site's own search URLs (/?s=, /search?q= and two more), ranks the links by the query and fetches the best ones. feed discovers the site's RSS/Atom `` and reads it. doctor --probe asks every engine once and reports which ones answer from this network; doctor also reports the proxy in use with its credentials removed. Requests honour HTTPS_PROXY / ALL_PROXY, and --proxy URL overrides them for one run.
Browser mode
The browser layer lives in browser/ and needs Node.js 18 or newer. It is read-only by default: like, coin, favourite, follow, share, comment, danmaku and publish are denied unless --allow-write is passed after the user authorises that single action.
node browser/run.js --list
node browser/run.js --doctor
node browser/run.js --playbook bili-feed --session bili-01 --max-items 8 --agent
node browser/run.js --playbook bili-video --url "https://www.bilibili.com/video/BV1xx411c7mD"
node browser/run.js --playbook zhihu-dig --query "reinforcement learning" --agent
node browser/run.js --playbook bili-feed --dry-run
| Playbook | Does | Requires |
|---|---|---|
bili-feed | Home feed, scroll and capture unseen cards | none |
bili-video | One video and its visible comments | --url or --bvid |
bili-chain | Follow the next-episode control and resume from the session | --url, --bvid, or a chain |
bili-danmaku | Timestamped danmaku, preferring bridge observe over DOM text (experimental) | --url or --bvid |
zhihu-dig | Zhihu search, rank links, capture the best unseen page | --query |
xhs-dig | Xiaohongshu public notes, rank /explore links, capture one | --query |
search-dig | Bilibili search, rank cards, open the best unseen result | --query |
The bundled bridge is picked up automatically once playwright and its Chromium binary are installed; --bridge or WEB_FORAGER_BRIDGE points at a bridge elsewhere:
npm install playwright && npx playwright install chromium
node browser/run.js --playbook bili-feed --doctor # says what is still missing
WEB_FORAGER_BROWSER_HEADFUL=1 shows the window, WEB_FORAGER_BROWSER_PROFILE=/path reuses a persistent profile (this is what carries login state for the extension backend), and WEB_FORAGER_BROWSER_CDP=http://127.0.0.1:9222 attaches to a browser you already control (cdp). A bridge declares its real capabilities through an optional describe call; one that does not implement it is reported as declared: false with every capability false.
--strict-robots fetches robots.txt before every navigation and stops with robots_denied (exit 3) on a disallowed path. Without it the record says so: compliance.robots_checked is false. A run that reaches the site but finds nothing readable is degraded (exit 8) with its records attached; an anti-bot or error interstitial is blocked (exit 9); a site that asks for credentials is login_required (exit 6).
A browser run is the most expensive thing the plugin does, so its defaults are small: records go to disk with --out result.json (or out: in MCP), no screenshots are taken unless asked, --max-screens 5 and --min-interval 3000 keep a walk polite, and a long walk is better split into runs that resume from --session.
MCP servers
{
"mcpServers": {
"web-forager": {
"command": "python",
"args": ["/absolute/path/to/web-forager/mcp_server.py"]
},
"web-forager-browser": {
"command": "node",
"args": ["/absolute/path/to/web-forager/browser/mcp_server.js"]
}
}
}
HTTP tools: web_fetch, web_search, web_surf, web_scholar, web_map, web_bilibili, web_clean, web_doctor, web_find, web_feed. Browser tools: browser_list, browser_run, browser_session, browser_doctor.
Two switches keep a search cheap: max_pages: 0 returns the result list with no page bodies (fetch one with web_fetch and a small max_chars), and respect_robots: false reads a page robots.txt disallows, only for resources you are authorised to read. web_search also returns engine_report — one entry per engine tried — and warnings when the results do not mention the query.
Test the protocol directly:
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | python mcp_server.py
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | node browser/mcp_server.js
CLI examples
python run.py --action fetch --url "https://www.bilibili.com/video/BV1xx411c7mD"
python run.py --action fetch --url "https://www.zhihu.com/" --render auto
python run.py --action fetch --query "city walk guide" --num-results 8 --max-pages 3
python run.py --action surf --url "https://www.zhihu.com/" --depth 1 --max-pages 5 --session zhihu-01
python run.py --action clean --input notes.pdf --format markdown
python run.py --action map --url "https://example.org" --limit 500
python run.py --action bili --mode feed --max-items 10
Workspace output
python run.py --action fetch --url "https://example.org/article" --root
Writes only:
/process/web-forager/web-forager__.json
/process/web-forager/web-forager__.md
/_broadcast/web-forager__.json
material/ and results/ belong to the user and the orchestrator; web-forager never writes there. Browser captures follow the same discipline under /process/web-forager/browser/ and /_broadcast/.
Records
| Field | HTTP mode | Browser mode |
|---|---|---|
metadata, markdown, tables, warnings | yes | yes |
url, title | yes (url and final_url) | yes |
status, content_type, document_kind, cached, rendered, fetched_at, chunks, links | yes | no |
schema_version, captured_at, playbook, backend, compliance, viewport_evidence, interaction_trace | no | yes |
--core returns the shared shape instead:
{
"schema_version": "core-1",
"source": "web-forager",
"mode": "http",
"url": "https://example.org/article",
"title": "An article",
"collected_at": "2026-11-07T10:00:00Z",
"metadata": {},
"markdown": "...",
"tables": [],
"warnings": [],
"extra": { "status": 200, "cached": false }
}
mode is http or browser, source names the producer, and extra keeps every field the two modes do not share. The same fold is available as a library call: engine.normalize_record() in Python and normalizeRecord() in browser/lib/record.js.
results is an index, not a second copy o