Docs & Knowledge · JimLiu/baoyu-skills

baoyu-url-to-markdown

Name: baoyu-url-to-markdown
Author: JimLiu/baoyu-skills

Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.

L5 Restricted · Isolated sandbox & strict auditV2 Human-AuditedReverse APITerms / ComplianceCredential / SessionExecute CommandNetwork AccessWrite FileDual-pass Audit

Control levels describe skill capabilities and suggested control measures; they do not represent quality or safety certifications, nor do they dictate usability. Usage decisions rest entirely with users and administrators. Different agent hosts apply varying constraints; do not assume hosts provide automatic protection. Third-party skill copyrights belong to their original authors. Use at your own risk.

1How It Works & Mechanism

The entry point is a shell wrapper + vendored TS CLI: baoyu-fetch directly runs lib/cli.ts via bun, and the CLI selects an adapter (x / youtube / hn / generic) based on the URL.

skills/baoyu-url-to-markdown/scripts/baoyu-fetch

exec bun "$script_dir/lib/cli.ts" "$@"

SKILL.md requires the agent to run bun install first to set up dependencies before invoking ${READER}; the CLI defaults to non-headless mode (see the discrepancy item under verification) and launches a visible Chrome instance.

Scraping relies on real Chrome + a custom-built CDP client (launches browser via chrome-launcher and connects directly to DevTools via ws), rather than puppeteer.

skills/baoyu-url-to-markdown/scripts/lib/browser/cdp-client.ts

import WebSocket from "ws";

JS execution within the page uses Runtime.evaluate (awaitPromise + returnByValue); scrolling, clicking, and DOM reads all go through it.

The X adapter does not merely read the DOM; it features a built-in session mechanism: it first reads x.com/twitter.com cookies via CDP (requiring auth_token and ct0), writes the cookies in plaintext into a sidecar file under the profile directory, and re-injects them on subsequent runs.

skills/baoyu-url-to-markdown/scripts/lib/browser/cookie-sidecar.ts

await writeFile(filePath, JSON.stringify(data, null, 2));

The file is named x-session-cookies.json and resides inside the Chrome profile directory (unencrypted, without tightened permissions); it is used to restore login state when cookies cannot be retrieved through the normal path.

The data source for X comes from the page's own GraphQL responses: the tool extracts and parses /graphql/ response bodies such as TweetDetail and TweetResultByRestId from CDP network logs, and automatically scrolls and clicks 'Show replies' to load thread content.

skills/baoyu-url-to-markdown/scripts/lib/adapters/x/payloads.ts

entry.url.includes("TweetDetail") ||

Thus, it does not reverse-engineer the signing logic of private Twitter APIs, but it does read response payloads from those non-public endpoints.

The YouTube adapter uses custom-built InnerTube calls: it extracts INNERTUBE_API_KEY from ytcfg on the page, sends a POST to /youtubei/v1/player within the page context (credentials: 'include', carrying page cookies), and explicitly impersonates an ANDROID client.

skills/baoyu-url-to-markdown/scripts/lib/adapters/youtube/transcript.ts

context: { client: { clientName: 'ANDROID', clientVersion: '20.10.38' } },

This shares the same architectural pattern as baoyu-youtube-transcript from the same repo: bypassing the official Data API and impersonating clients, following a reverse-engineering/spoofing approach.

Can reuse an existing running Chrome debugging session: scans ps aux first for processes with the same profile and --remote-debugging-port, or reads the DevToolsActivePort file in the profile, and attaches upon discovery.

skills/baoyu-url-to-markdown/scripts/lib/browser/profile.ts

const activePort = parseDevToolsActivePort(path.join(options.profileDir, "DevToolsActivePort"));

It also supports --cdp-url ws://… to connect directly to any debug endpoint; this represents an intentional session takeover capability.

When encountering login walls or CAPTCHAs, it avoids brute-forcing and provides wait modes: when Cloudflare/reCAPTCHA or similar challenges are detected on the page, it pops up visible Chrome for manual user solving, resuming upon detecting progress or on pressing Enter.

skills/baoyu-url-to-markdown/scripts/lib/browser/interaction-gates.ts

text.includes("verify you are human") ||

Defaults to a 10-minute timeout with 1.5-second polling; this is the concrete implementation of 'Handles login/CAPTCHA via interaction wait modes' described in SKILL.md.

The primary pipeline for generating markdown relies on local Defuddle/Readability + quality scoring; only when quality falls short is the target URL sent to the third-party service https://defuddle.md to obtain cleaner markdown.

skills/baoyu-url-to-markdown/scripts/lib/extract/html-to-markdown.ts

const DEFUDDLE_API_ORIGIN = "https://defuddle.md";

This is why the 'Quality Gate' exists in SKILL.md: headless fetching can silently return low-quality content, requiring agent re-examination (--debug-dir can dump document.json/markdown.md/page.html/network.json).

2Core Capabilities

01Any URL → clean markdown (defaults to stdout, supports --output to save to file, --format json for structured documents)

02Site adapters: X/Twitter (including long threads and articles), YouTube (transcripts), Hacker News (nested comment trees), generic web pages

03YouTube transcript extraction (page-context InnerTube + caption XML parsing)

04Login/CAPTCHA interactive wait modes (interaction automatic detection / force manual control)

05Media downloading and link rewriting (--download-media saves to imgs/ and videos/, configurable via --media-dir)

06Debug forensics: --debug-dir dumps document.json / markdown.md / page.html / network.json (including request headers and response bodies)

07X session persistence (cookie sidecar export/re-injection) and CDP debug session reuse

08Quality Gate workflow (headless results treated as provisional, allowing fallback to interactive mode for re-fetching after human/agent re-inspection)

3External Dependencies

Kind	Dependency / Endpoint
package	@mozilla/readability ^0.6.0（正文提取）
package	defuddle ^0.17.0（本地 markdown 提取）
package	jsdom ^30.0.1（HTML 清洗/解析，未开 runScripts）
package	turndown + turndown-plugin-gfm（HTML→markdown）
package	chrome-launcher ^1.2.1（浏览器启动）
package	ws ^8.18.3（CDP WebSocket 客户端）
package	unified + remark-parse + remark-gfm（媒体链接收集与重写）
cli	bun（CLI 运行时；shell 包装 exec bun）
cli	Chrome/Chromium 二进制（chrome-launcher，可用 --browser-path 指定）
cli	ps aux（POSIX 进程扫描）与 osascript（macOS 窗口激活）
network	Chrome DevTools HTTP/WebSocket 端点
api	第三方 markdown 服务 defuddle.md（质量不足时把目标 URL 发过去）
network	YouTube watch 页、InnerTube player、caption 轨与 i.ytimg.com 缩略图
network	Hacker News 条目页（jsdom 解析）
network	X/Twitter 页面与页面自身发出的 /graphql/ 请求（工具只观察，不构造请求）
network	用户指定 URL 与页面内媒体 URL（下载时带伪造的桌面 Chrome UA）

4Recommended Governance & Evidence

Suggested Controls for Safe Operation

For Agent Runtime

Isolated environment, per-use explicit authorization

For Administrators

Strict authorization, full audit, compliance signoff

Evidence & Triggers (File Paths & Excerpts)

SKILL.md / 脚本: Calls undocumented or reverse-engineered private APIs / spoofs client identifiers
SKILL.md / 脚本参数: Contains logic to bypass automation detection, disable host sandboxing, or skip SSL certificate validation
SKILL.md / 会话存储: Reads or reuses browser user login states and session cookies
本地凭据 / 浏览器配置: Reads user login cookies/sessions/keyrings: reads x.com/twitter.com cookies via CDP Network.getCookies, requiring auth_token and ct0 to be present
scripts/: Executes local scripts or commands: {baseDir}/scripts/baoyu-fetch <url> … (shell exec bun → lib/cli.ts; requires bun i on first run)
网络端点: Declares external network access: user-specified target URLs (real browser pages; generic pages routed through Readability/Defuddle/defuddle.md)
本地工作目录: File read/write and artifact output: markdown/JSON specified by --output (parent directories created automatically)

Audit Notes: This is a capability/policy-level determination, not malicious injection: the codebase is clean, un-obfuscated, lacks hardcoded secrets, and has no TLS/sandbox downgrades. Three layers of reasoning: ① [Reverse engineering/Impersonation] The YouTube adapter bypasses official Data APIs, constructing InnerTube requests inside page context ('const response = await fetch(\'/youtubei/v1/player?key=\' + apiKey + \'&prettyPrint=false\', {' with credentials: \'include\', where key is taken from window.ytcfg), while explicitly impersonating an official client ('context: { client: { clientName: \'ANDROID\', clientVersion: \'20.10.38\' } },'); this falls into the same category of boundary violations as baoyu-youtube-transcript from the same repository. ② [Plaintext credential storage & debug session takeover] The X adapter writes x.com/twitter.com cookies including auth_token/ct0 in plaintext to <chrome-profile-dir>/x-session-cookies.json ('await writeFile(filePath, JSON.stringify(data, null, 2));', without encryption) and re-injects them as needed; it also takes over already running Chrome debug sessions via DevToolsActivePort / ps aux (supports connecting to arbitrary ws debug endpoints via --cdp-url); --debug-dir dumps entire request headers and response bodies to disk ('const networkDump = await network.toJSON({ includeBodies: true });'). ③ [Third-party outbound transfer & request spoofing] When local extraction quality is inadequate, target URLs are sent to third-party https://defuddle.md to fetch markdown; media download requests carry spoofed desktop Chrome UA headers; page contents dictate subsequent crawl targets and local artifact filenames. Context note: cookies are used strictly for their stated purpose (fetching login-required X content), without TLS degradation, unescaped interpolation, or script execution via jsdom, and with wait modes for login/CAPTCHA rather than automated bypass.

5Second-Pass Verification

[ok] YouTube 适配器的逆向/冒充 — transcript.ts 有 window.ytcfg?.data_?.INNERTUBE_API_KEY、页面内 POST /youtubei/v1/player?key=…（credentials: 'include'）与 context.client.clientName='ANDROID' 三处，与 Main 终裁的『逆向 youtubei ＋冒充官方客户端』一致。
[ok] X cookie 明文 sidecar 落盘 — cookie-sidecar.ts 用 Network.getCookies 取 cookie 后 JSON.stringify 明文写入 <profile>/x-session-cookies.json（x/index.ts 指定 filename），无加密无 chmod；convert.ts 在开头尝试回灌、finally 里导出。
[ok] 接管已运行调试会话 — profile.ts 解析 DevToolsActivePort 并 spawnSync('ps', ['aux']) 匹配 profileDir + --remote-debugging-port；chrome-launcher.ts 支持 --cdp-url ws:// 直连与复用日志 'Reusing Chrome debugger at …'。
[ok] 网络日志含请求头与响应体（--debug-dir 落盘） — network-journal.ts 记录 requestHeaders / requestBody / 响应体（Network.getResponseBody），convert.ts 在写 debug 前调用 toJSON({ includeBodies: true }) 并写入 network.json。
[ok] 第三方 defuddle.md 兜底 — DEFUDDLE_API_ORIGIN=https://defuddle.md，构建 URL 后 fetch；通用适配器仅在 outputFormat==='markdown' 时启用该兜底，JSON 模式不会外发目标 URL。
[discrepancy] 文档与实现不一致（默认无头） — SKILL.md 第 93 行写 '# Default: headless capture, markdown to stdout'、quality-gate.md 亦写 'Default | Headless Chrome'，但 cli.ts 默认 headless: false（convert.ts 在非交互模式传该值），实际会启动可见 Chrome（--no-startup-window）。使用前应知道默认并非无头。
[discrepancy] 文档与实现不一致（默认 profile 路径） — SKILL.md 写默认 ./baoyu-skills/chrome-profile，代码实际解析为按 OS 的绝对路径（XDG_DATA_HOME / Application Support / APPDATA 下的 baoyu-skills/chrome-profile），从不使用 cwd 相对路径。
[discrepancy] 文档未披露 cookie 导出 — SKILL.md 只写 BAOYU_CHROME_PROFILE_DIR 为『Chrome user data directory』，从未提及会导出 X cookie 到 sidecar 文件；该行为只在代码中可见——这是本次侦查中风险披露落差最大的一处。

6Verdict & Scenarios

A single CLI covers four categories of websites, featuring a quality gate workflow rather than assuming 'any fetch is a success'.

Proper handling for auth-walled pages: interactive wait mode lets users solve login/CAPTCHA manually, resuming automatically upon detecting progress.

Custom CDP client has minimal dependencies and high controllability: chrome-launcher + ws, avoiding the full puppeteer stack.

Debug-friendly: --debug-dir produces four types of forensic artifacts in one shot: document, markdown, page.html, and network.

Complete media localization: --download-media downloads and rewrites links, keeping imgs/videos self-contained in the same directory alongside markdown.

Best for: Ideal for individual users and content creators who need to save web pages, X threads, YouTube transcripts, or HN discussions as markdown: those requiring local archiving, bundling media assets to disk, handling login walls (via one-time manual pass), and comfortable using real Chrome with a custom CDP pipeline.

Not for: Not suitable for corporate compliance or privacy-sensitive environments (plaintext X cookie sidecars, forensic debug dumps to disk, third-party outbound calls to defuddle.md); not suitable for unattended large-scale batch scraping (high overhead from interactive waits and real browsers); not suitable for pipelines requiring stable official APIs; and not suitable for targeting intranet/private URLs or third-party authenticated sessions.

Install Agent-Ready

① Verified Mirror 2026-09-15

Option A · Direct DownloadDownload baoyu-url-to-markdown.tar.gz

sha256: 15be54497a8d6c30…

Option B · Copy Guide for Agent

install.json

Agents automatically download and verify mirrors from this JSON guide.

② Upstream GitHub

GitHub Repo ↗

Pinned to commit 1567581c26.