Home / All Skills / Content & Media Creation / canghe-url-to-markdown
English (EN)简体中文 (ZH)日本語 (JA)
Content & Media Creation · freestylefly/canghe-skills

canghe-url-to-markdown

Fetch any URL and convert to markdown using Chrome CDP. Supports two modes - auto-capture on page load, or wait for user signal (for pages requiring login). Use when user wants to save a webpage as markdown.

L4 High-Privilege · Graded auth & full auditV1 Commit-PinnedExecute CommandNetwork AccessWrite FileDual-pass Audit
Author freestyleflyGitHub freestylefly/canghe-skills ↗License MIT（仓库根 LICENSE）Commit dd0bf35595
Control levels describe skill capabilities and suggested control measures; they do not represent quality or safety certifications, nor do they dictate usability. Usage decisions rest entirely with users and administrators. Different agent hosts apply varying constraints; do not assume hosts provide automatic protection. Third-party skill copyrights belong to their original authors. Use at your own risk.

1How It Works & Mechanism

The core mechanism drives a local real Chrome instance via CDP to load any URL, then converts the rendered HTML into clean markdown.

skills/canghe-url-to-markdown/SKILL.md
Fetches any URL via Chrome CDP and converts HTML to clean markdown.
What it does: Uses a real browser to solve JS rendering challenges rather than static HTTP scraping.

Provides two capture modes: auto (captures once the network is idle) and --wait (captures after a user keypress signal).

skills/canghe-url-to-markdown/SKILL.md
- Two capture modes: auto or wait-for-user
What it does: Leaves the capture timing to the user for pages involving login states, lazy loading, or paywalls.

In wait mode, it prints a prompt and blocks waiting for a newline on stdin, triggering extraction once received.

skills/canghe-url-to-markdown/scripts/main.ts
console.log("Page opened. Press Enter when ready to capture...");
What it does: Uses readline to wait for user confirmation that the page is ready before executing the CDP extraction script.

The script launches Chrome itself (enumerating common installation paths by platform, overridable via URL_CHROME_PATH).

skills/canghe-url-to-markdown/scripts/cdp.ts
export function findChromeExecutable(): string | null {
What it does: Locates the browser executable, spawns it, and exposes the debugging port using --remote-debugging-port.

Discovers the CDP endpoint via the local debugging port (http://127.0.0.1:<port>/json/version).

skills/canghe-url-to-markdown/scripts/cdp.ts
const res = await fetchWithTimeout(`http://127.0.0.1:${port}/json/version`, { timeoutMs: 5_000 });
What it does: Waits for Chrome's debugging port to be ready, then establishes a WebSocket CDP connection.

Converts HTML to markdown using sanitize + turndown, falling back to plain text extraction on failure.

skills/canghe-url-to-markdown/scripts/html-to-markdown.ts
return turndown.turndown(sanitized);
What it does: Sanitizes the DOM before converting to markdown to strip noise such as scripts and styles as much as possible.

Outputs markdown with YAML front matter saved to url-to-markdown/<domain>/<slug>.md, appending a timestamp on collision.

skills/canghe-url-to-markdown/SKILL.md
YAML front matter with `url`, `title`, `description`, `author`, `published`, `captured_at` fields, followed by converted markdown content.
What it does: Separates file metadata from body text for easy archiving and reprocessing.

Chrome path, data directory, and profile directory can all be overridden via environment variables.

skills/canghe-url-to-markdown/SKILL.md
| `URL_CHROME_PATH` | Custom Chrome executable path |
What it does: Handles cases where Chrome is not found or an isolated profile is needed; variables are purely path-related, not secrets.

2Core Capabilities

01Capture any URL (including JS-rendered pages) as markdown
02Two capture modes: auto / wait
03Full JS rendering support
04Generate output paths by domain/title and resolve collisions
05Auto-scroll to trigger lazy loading before capture
06Output markdown with YAML metadata
07Customize Chrome / data directory / profile via environment variables
08Configure preferences via EXTEND.md (output directory/mode/timeout)

3External Dependencies

KindDependency / Endpoint
clibun / npx -y bun（运行 scripts/main.ts）
cli本机 Chrome/Chromium/Edge（由 cdp.ts 定位并 spawn，用 CDP 驱动）
network用户指定的目标 URL（浏览器真实加载该页面及其子资源）

4Recommended Governance & Evidence

Suggested Controls for Safe Operation
For Agent Runtime
Graded/per-use authorization, sandbox recommended
For Administrators
Approval workflow, rollback support, full audit
Evidence & Triggers (File Paths & Excerpts)
  • scripts/: Executes local script or command: npx -y bun ${SKILL_DIR}/scripts/main.ts <url> [--wait] [-o out.md] [--timeo
  • 网络端点: Declares external network access: User-specified target URL: real browser loads the page (including its sub-resources and page-initiated requests)
  • 本地工作目录: File read/write and artifact output: url-to-markdown/<domain>/<slug>.md (body + YAML metadata)
Audit Notes: The audit target is this skill's own directory: five local scripts (main/cdp/html-to-markdown/paths/constants) collaborate to launch local Chrome + CDP, load the user-specified URL, and convert HTML to markdown on disk. Regular network requests exist (target URL and 127.0.0.1 debug port), but outbound destinations are user-specified and expected, so it is not Blue; it does not read credentials/cookies, nor does it perform reverse engineering, impersonation, bypasses, security downgrades, or arbitrary code execution, so it is not Orange/Red. Compared to its upstream repository baoyu-url-to-markdown (rated Red due to reverse engineering YouTube InnerTube, writing plain-text credentials to disk, and debug session hijacking), this fork is a general-purpose scraper without site adapters or credential persistence, thus downgraded to Yellow.

5Second-Pass Verification

  • [ok] 网络调用定位 — 风险扫描『網路請求』命中 cdp.ts 的 fetch/WebSocket 与 http://127.0.0.1:${port}/json/version；均为本机调试端口，目标页网络由 Chrome 自身发起。
  • [ok] 不读凭证 — 风险扫描『憑證/環境變數』命中项均为路径/目录类（URL_CHROME_PATH/URL_DATA_DIR/URL_CHROME_PROFILE_DIR/APPDATA/XDG_DATA_HOME）与 ${SKILL_DIR}/$HOME 文档文本，无密钥/cookie 读取。
  • [ok] 独立 profile 而非用户主 profile — paths.ts resolveUrlToMarkdownChromeProfileDir 指向用户数据根/canghe-skills/url-to-markdown/chrome-profile。
  • [ok] 无站点适配器/无冒充 — §1 清单只有 6 个文件，无 X/YouTube 等适配器；cdp.ts 无 InnerTube/UA 伪装类代码，与上游 baoyu-url-to-markdown 的红档成因不同。
  • [ok] 无安全降级 — 风险扫描『危險旗標/防護繞過』与『沙箱/未受控執行』均无命中；未见 remote-allow-origins=*/--no-verify 等。
  • [ok] 两模式与写盘路径属实 — main.ts 的 waitForUserSignal 打印 'Press Enter when ready to capture...'；main.ts 以 writeFile(outputPath,...,'utf-8') 落盘。
  • [ok] 档位复核 — 有常规网络调用、无凭证、无绕过 → yellow；相较上游红档按证据差异降档，已在 reason 说明。
  • [ok] 元数据与 commit — commit=dd0bf355955b4c82b764740b4183c86a72ba0e0c，license=MIT（仓库根），installs=null。

6Verdict & Scenarios

  • Captures using real Chrome CDP to retrieve JS-rendered content, outperforming pure HTTP fetching.
  • Wait mode hands capture timing over to the user, handling login states, lazy-loaded content, and paywalls.
  • Fully local with zero cloud dependency; outputs self-contained markdown with YAML metadata for easy archiving.
  • Provides configurable paths, directories, modes, and timeouts via environment variables and EXTEND.md, with an isolated profile that will not interfere with the user's primary browser.
    • Best for: Ideal for knowledge workers, researchers, and content collectors who need to archive webpages offline as markdown (especially JS-rendered or login-required pages) and are comfortable capturing with a local Chrome instance.
    Not for: Not suitable for environments without Chrome or scenarios requiring avoidance of browser automation; not suitable for scraping pages containing sensitive login credentials or violating target site terms of service; not suitable for users expecting structured multi-site adapters (e.g., dedicated site parsers).
    Install Agent-Ready
    ① Verified Mirror 2026-10-10
    Option A · Direct DownloadDownload canghe-url-to-markdown.tar.gz
    sha256: bc953895075e756e…
    Option B · Copy Guide for Agent
    install.json
    Agents automatically download and verify mirrors from this JSON guide.
    ② Upstream GitHub
    GitHub Repo ↗
    Pinned to commit dd0bf35595.