dsh-read-url

Usage

Just talk to the agent:

Read https://example.com/article and summarize the key points
Read https://docs.example.org/guide in markdown mode
Read these URLs together and compare their viewpoints:   

Real examples (measured)

1. Token economy — return only what the model needs

read_url on a portal returns cleaned body capped at maxChars (default 6,000) — not the raw page with nav/ads/footers. Repeat reads hit the 5-min cache ((cached)), so the agent never re-fetches:

title: 新闻中心首页_新浪网
charset utf-8
(chars 800/12398 — 截断，offset 续读)

2. Overseas sites — direct + proxy race

When a proxy is configured, the direct fetch and the proxy curl start together and the first to complete wins. A blocked overseas site that used to cost ~11s (direct-connect timeout + fallback) now reads in under a second:

BBC 中文: OK (633ms) — clean 4,000+ chars of headline news

3. Long-article continuation (offset)

A 12,000-char article is read in slices; offset resumes from cache without repeating earlier text — the model keeps exactly what it needs in context:

chars 800+800/12398 · cached

4. Batch research across pages

read_url_batch reads up to 10 pages in parallel (concurrency 4), each cleaned individually, failures isolated:

读取 2/4 页成功，2 页失败
--- 阮一峰的网络日志 (491 字符) ---
--- Example Domain (127 字符) ---
[失败] https://zh.wikipedia.org/... — Fetch failed: HTTP 403 ...

Tools

read_url(url, maxChars?, offset?, mode?, includeLinks?) — fetch and extract clean body. Handles HTML pages, JSON APIs (fully compact render with long-value clipping) and RSS/Atom feeds (entry list + feedCount); paginated articles (novel/news/forum) are auto-joined up to paginateMax pages; page metadata (published, author) is harvested automatically — meta → schema.org JSON-LD → byline, three-tier fallback (v1.3.0: JSON-LD fills publication time/author on news sites); shell pages (link hops / anti-hotlink relays / no-JS SPA entries) are followed to the real page automatically (≤3 hops, loop-safe, fails open on fetch errors, v1.3.0); pages declaring (frame sites / legacy forums) resolve relative links against it (v1.3.0)

Param	Type	Default	Description
url	string	required	http(s) URL
maxChars	number	6000	Max body characters returned (500–20000)
offset	number	0	Resume reading from this character offset (long-article continuation; served from cache without repeating earlier text)
mode	string	text	text = plain (most token-efficient); markdown = structured
includeLinks	boolean	false	Also return up to 20 page links (title+URL)

read_url_batch(urls, maxChars?, mode?, includeLinks?) — read multiple URLs (1–10) in parallel, each cleaned individually, merged into one compact report

Param	Type	Default	Description
urls	string[]	required	http(s) URL list (1–10)
maxChars	number	3000	Max body characters per page (500–20000)
mode	string	text	text = plain; markdown = structured
includeLinks	boolean	false	Also return links per page (title+URL)

• Concurrency capped at 4 (avoids rate-limiting); a failing page is isolated ([失败] + reason in the output) and does not affect the others;
• Reuses every read_url capability and the session cache (encoding, cleaning, SPA rendering, 5-min cache — repeat batches hit the cache).

read_url_site(url, maxPages?, maxDepth?, includeContent?, maxCharsPerPage?) — recursive site crawl: BFS from the entry URL across same-host pages, returns a compact site map

Param	Type	Default	Description
url	string	required	http(s) entry URL
maxPages	number	15	Maximum attempted pages (2–50, including failures; fractional values round down)
maxDepth	number	2	Max link depth from entry (1–5)
includeContent	boolean	false	Attach a short body summary per page (default off — structure first, token-efficient)
maxCharsPerPage	number	500	Summary length per page when includeContent=true (200–2000)

• Same-host only; login/API/static-asset paths are skipped; URLs deduped (fragment stripped);
• Concurrency 2 (gentle on the target site); per-page failures recorded as [失败] without aborting;
• Output is an indented tree: [depth] title (chars) URL;
• No SPA rendering here (crawling favors speed/breadth) — use read_url for JS-only pages.

read_url_links(url, limit?) — list the page's links without returning body text (lighter; good for sourcing / mapping a site)

Param	Type	Default	Description
url	string	required	http(s) URL
limit	number	20	Max links returned (1–50)

Configuration (optional)

Override via the profile's cordis.patch.yml (defaults in the plugin's own cordis.patch.yml):

- id: dsh-read-url
  config:
    timeoutMs: 15000      # per-request timeout (500-120000, clamped)
    maxBytes: 3145728     # response body cap (bytes)
    maxChars: 6000        # default body truncation
    maxLinks: 20          # read_url_links default count
    spaRender: true       # SPA rendering enhancement (needs playwright installed; degrades with a hint otherwise)
    paginate: true        # auto-join paginated articles
    paginateMax: 3        # pages per article, incl. the first (1-10, clamped)
    userAgent: '...'      # request UA
    cacheTtlMs: 300000    # success-cache TTL
    cacheMax: 32          # cache entry cap

Values are coerced and clamped to sane ranges at load — quoted numbers in YAML work, garbage falls back to defaults.

Output (compact)

{
  "url": "...",
  "title": "...",
  "siteName": "...",
  "lang": "zh-CN",
  "charset": "gbk",
  "mode": "text",
  "truncated": true,
  "charsTotal": 12990,
  "charsReturned": 6000,
  "charsStart": 0,
  "text": "...",
  "rendered": true,        // only when extracted after SPA rendering
  "paginated": 3,          // pages auto-joined (only when > 1)
  "feedCount": 20,         // RSS/Atom entries (feeds only)
  "published": "2026-08-01",  // when the page declares it (incl. byline fallback)
  "author": "...",           // when the page declares it (incl. byline fallback)
  "links": []               // only when includeLinks=true
}

PTC mode

Output is pure JSON and composable; orchestrate parallel multi-URL reads in PTC mode:

const results = await Promise.all([
  read_url({ url: 'https://a.example.com', maxChars: 4000 }),
  read_url({ url: 'https://b.example.com', maxChars: 4000 }),
])

Configuration (optional)

Override via the profile's cordis.patch.yml (defaults in the plugin's own cordis.patch.yml):

- id: dsh-read-url
  config:
    timeoutMs: 15000      # per-request timeout (500-120000, clamped)
    maxBytes: 3145728     # response body cap (bytes)
    maxChars: 6000        # default body truncation
    maxLinks: 20          # read_url_links default count
    spaRender: true       # SPA rendering enhancement (needs playwright installed; degrades with a hint otherwise)
    paginate: true        # auto-join paginated articles
    paginateMax: 3        # pages per article, incl. the first (1-10, clamped)
    userAgent: '...'      # request UA
    cacheTtlMs: 300000    # success-cache TTL
    cacheMax: 32          # cache entry cap

Values are coerced and clamped to sane ranges at load — quoted numbers in YAML work, garbage falls back to defaults.