GreyRaphael/dsh-web-fetch-moli ↗★ 2
dsh-web-fetch-moli
利用轻量级Moli浏览器渲染并爬取网页 适合需要在低内存占用下抓取复杂单页应用和动态网页的用户。
安装
$
npx -p @deepseek-ai/dsh dsh plugin --profile web add github:GreyRaphael/dsh-web-fetch-moli说明文档
阅读完整 README ↗Configuration
The settings card (设置 → 插件 → 插件配置 → Moli 网页爬取) configures web-fetch-moli in real time:
| Field | Default | Description |
|---|---|---|
backend | local | local (managed Moli CDP daemon) or cdp (remote CDP endpoint). |
moliPath | (auto) | Path to moli binary. Blank = auto-discover on $PATH and standard locations. |
cdpEndpoint | 127.0.0.1:9222 | Remote CDP endpoint (host:port or URL, used in cdp mode). |
bypassCsp | true | Bypasses Content Security Policy via CDP Page.setBypassCSP, unlocking micro-frontend dynamic script loading. |
autoScrollSentinel | true | Programmatically triggers IntersectionObserver load-more sentinels for structure-first geometry. |
shareBrowserContext | true | Preserves cookies and localStorage across fetches in cdp mode. Unchecked (and always in local mode): fresh isolated context per fetch. |
denoise | true | Runs LinkeDOM + Readability + mdream to strip ads, navbars, and footers before LLM-optimized Markdown conversion. |
timeoutMs | 90000 | Total deadline budget (ms) for one fetch; heavy micro-frontend pages may need several layout + scroll rounds. The bundled layer ships 60000, the schema default is 90000. |
maxConcurrency | (auto) | Max concurrent rendering slots (auto: local 20, remote CDP 50). |
challengeWaitMs | 5000 | Bounded wait (ms) for Cloudflare verification to clear naturally (0 disables). |
challengeRetries | 0 | Same-page retry attempts after challenge wait. |