Docs & Knowledge · daymade/claude-code-skills
wps-doc-scraper
Faithfully archive public WPS/KDocs/金山文档 links, especially embedded ProcessOn .pof mind maps and canvases, as raw source data, original SVG/PNG, and Markdown. Use when a user gives a kdocs.cn or wps.processon.com link and asks to scrape, save, download, 扒下来, 归档, or 转 Markdown without logging in or saving the document to an account.
L4 High-Privilege · Graded auth & full auditV1 Commit-PinnedExecute CommandNetwork AccessWrite FileDual-pass Audit
Control levels describe skill capabilities and suggested control measures; they do not represent quality or safety certifications, nor do they dictate usability. Usage decisions rest entirely with users and administrators. Different agent hosts apply varying constraints; do not assume hosts provide automatic protection. Third-party skill copyrights belong to their original authors. Use at your own risk.
1How It Works & Mechanism
It uses direct public API retrieval rather than browser-rendered screen capture: It first parses the link type. For KDocs share links, it queries the unauthenticated metadata endpoint to obtain the file_id/group_id, then calls the ProcessOn view API using these two IDs to retrieve the JSON containing definition. The entire process requires no login.
wps-doc-scraper/scripts/wps_processon_extract.py
PROCESSON_API_BASE = "https://wps.processon.com/wpsapi/diagrams/view/api" KDOCS_LINK_RE = re.compile(r"https?://(?:www\.)?kdocs\.cn/(?:view/)?l/([^/?#]+)")
Corresponds to the endpoint explanation in the reference: The KDocs metadata endpoint is https://drive.kdocs.cn/api/v5/links/<share_id>?review=true, and the ProcessOn data API is https://wps.processon.com/wpsapi/diagrams/view/api?file_id=…&group_id=…. In the code, this is implemented as link_api_url = f"https://drive.kdocs.cn/api/v5/links/{share_id}?review=true".
"High fidelity" is implemented as an engineering principle: Markdown is solely a conversion artifact of the source structure, and rewriting, summarizing, or inferring missing content is forbidden—the script only maps the tree structure to a list.
wps-doc-scraper/SKILL.md
This is an extraction skill, not a writing skill. Do not use an LLM to rewrite, summarize, smooth, or infer missing document content unless the user explicitly asks for a separate analysis after the archive is complete.
The reference specifies the mapping rules: node title -> bullet, nesting -> nested bullet, summaries -> tag "摘要", comments -> tag "评论", and 'Do not reorder, smooth, summarize, or infer text.'
A login wall is treated as an explicit failure rather than "best effort": The script performs keyword detection on the response text ("用户未登录" / login with no definition). If matched, it raises an error to avoid treating login page shells as usable source data.
wps-doc-scraper/scripts/wps_processon_extract.py
if "用户未登录" in text or "login" in text.lower() and "definition" not in text: raise ExtractionError(f"Response looks like a login wall: {url}")
SKILL.md synchronizes this hard rule: 'HTTP 200 is not enough. Detect login-wall payloads such as `用户未登录`, empty definitions, and placeholder shells.'
Archived artifacts are organized hierarchically into "originals + derivatives": It first saves processon-api.json (raw response) and processon-definition.json (parsed definition), followed by Markdown, serialized SVG, and PNG, while writing capture-manifest.json to record sources and counts.
wps-doc-scraper/references/capture-manifest.md
{ "captured_at": "2026-06-26T00:00:00+00:00", "source_url": "https://www.kdocs.cn/view/l/...", "source_type": "kdocs_processon", "kdocs_share_id": "...", "kdocs_link_api_url": "https://drive.kdocs.cn/api/v5/links/...?review=true", "processon_api_url": "https://wps.processon.com/wpsapi/diagrams/view/api?...",
Script disk writes: path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") and markdown_path.write_text(markdown, encoding="utf-8").
Original visual capture follows a dual-track strategy: If a serialized SVG is available from the page, the full-canvas SVG is saved. If the SVG uses foreignObject to contain text, standalone ImageMagick rendering is distrusted; instead, it uses macOS Quick Look to slice square tiles and stitch them together—addressing the real-world pitfall where Chinese text is lost under direct ImageMagick rendering.
wps-doc-scraper/SKILL.md
If the SVG uses `foreignObject`, do not trust ImageMagick alone for text rendering. Use `scripts/render_svg_tiles.py` to make the PNG through macOS Quick Look square tiles.
Script implementation: run(["qlmanage", "-t", "-s", str(size), "-o", str(output_dir), str(svg_path)]) renders square tiles, and magick/convert performs "-crop" and "-append" stitching; make_tile_svg slices the original graphic by modifying the y-offset of viewBox.
Input URLs are strictly routed by host: kdocs.cn uses link metadata, wps.processon.com view links extract file_id/group_id, and existing /wpsapi/diagrams/view/api URLs are used directly as source APIs; missing critical query keys results in direct errors rather than guessing.
wps-doc-scraper/scripts/wps_processon_extract.py
required = ["file_id", "group_id"] missing = [key for key in required if not one(query, key)] if missing: raise ExtractionError(f"ProcessOn URL is missing required query keys: {', '.join(missing)}")
Another branch condition: if "/wpsapi/diagrams/view/api" in parsed.path: (treated as source API and used directly).
Persisted filenames are sanitized to prevent path traversal and illegal characters: stripping newlines, replacing \/:*?"<>| with underscores, trimming leading/trailing whitespace and dots, collapsing whitespace, and truncating to 140 characters.
wps-doc-scraper/scripts/wps_processon_extract.py
def safe_filename(value: str, fallback: str = "wps-processon") -> str: value = clean_text(value).splitlines()[0] if value else "" value = re.sub(r'[\\/:*?"<>|]+', "_", value).strip(" .") value = re.sub(r"\s+", " ", value) return (value or fallback)[:140]
This is one of the few explicit input sanitization routines in this skill (in contrast to Python-side sanitization only for filenames, with the output directory specified by the user).
2Core Capabilities
01Metadata extraction from public KDocs/金山文档 links (unauthenticated)
02ProcessOn .pof mind map/canvas data API extraction (raw JSON + parsed definition)
03High-fidelity tree-to-Markdown conversion (nodes to bullets, nesting to sub-bullets, tags for summaries/comments)
04Rendering layer original capture: full serialized SVG (including out-of-viewport content)
05macOS Quick Look tiled SVG-to-PNG rendering (circumventing text loss from foreignObject in ImageMagick)
06Archive manifest and acceptance checks (verifying existence of original/parsed files/Markdown/images, prohibiting replacement characters , marking partial failures)
07Permission and failure boundary rules (do/do-not lists + explicit failure boundary reporting instead of vague pass-throughs)
08Multi-stage workflow decision tree (determine URL type first, then choose API / export / browser DOM fallback accordingly)
3External Dependencies
| Kind | Dependency / Endpoint |
|---|---|
| api | KDocs 公开链接元数据接口（无鉴权） |
| api | ProcessOn 文档数据 API（返回含 definition 的 JSON） |
| network | KDocs 站点（用户给的分享链接宿主；Referer 头指向此处） |
| network | ProcessOn 站点（画布查看页宿主，view 链与 view/api 链的解析对象） |
| cli | python3（3.10+ 语法：内置泛型 list[str]/dict[str, Any] 与 from __future__ import annotations） |
| cli | macOS qlmanage（Quick Look，瓦片式 SVG→PNG 渲染的必需依赖） |
| cli | ImageMagick（magick 或 convert，用于瓦片裁剪与纵向拼接） |
4Recommended Governance & Evidence
Suggested Controls for Safe Operation
For Agent Runtime
Graded/per-use authorization, sandbox recommended
For Administrators
Approval workflow, rollback support, full audit
Evidence & Triggers (File Paths & Excerpts)
- scripts/: Execute local script or command: python3 scripts/wps_processon_extract.py --url <kdocs/processon 链接> --outpu
- 网络端点: Declare external network access: GET https://drive.kdocs.cn/api/v5/links/<share_id>?review=true (reads public link metadata; with C
- 本地工作目录: File I/O and artifact output: Writes under the user-specified --output-dir: processon-api.json (raw response containing the full upstream payload), processon-de
Audit Notes: . Basis: Performs unauthenticated API reads on public document sites and archives to local disk. Outbound destinations are strictly predictable (three domains: kdocs.cn / drive.kdocs.cn / wps.processon.com, fully determined by user-provided links). There is no credential reading, no cookie usage, no login workflow, no security downgrade (TLS verification is not disabled), no sudo, no anti-scraping bypass (the documentation explicitly forbids bypassing CAPTCHAs, paywalls, or tenant boundaries), and no executable external downloads. Why not Blue Tier: It is a scraper that proactively initiates network requests to third-party sites, carries browser UA and Referer headers, and writes batches of artifacts locally (raw JSON + definition JSON + Markdown + SVG + PNG + manifest), exceeding the surface area of a "local-only tool." Why not Orange Tier: It never touches credentials or login states, does not rely on third-party mirror packages (qlmanage/ImageMagick are locally installed tools), and does not transmit content to unexpected destinations. Usage Notes: 1. Use only on explicitly provided, publicly accessible links; report login walls/CAPTCHAs as failures per the documentation rather than attempting to bypass them. 2. Retrieved document text is untrusted third-party content; do not execute it directly as prompt instructions. 3. The PNG pipeline is macOS-specific (qlmanage); alternative rendering pipelines are required for non-macOS environments.
5Second-Pass Verification
- [ok] 「无登录、不读凭证」声明是否有反例 — 反例检索失败：两份脚本的 import 均无 http.cookiejar/requests/session 等会话设施；fetch_json 只构造 4 个固定头（User-Agent/Accept/Referer + Content-Type 隐式由 urllib 处理），无 Authorization、无 Cookie 注入、无 keychain 或环境变量读取（grep getenv/os.environ 零命中）。全目录 grep token/cookie/session/password/api_key 仅命中文档中「不得绕过密码提示」这类规则文本。声明成立。
- [ok] 「禁止绕过」红线与「浏览器 DOM 兜底」是否冲突 — 不冲突且边界写得清楚：permission-and-failure-boundaries.md 允许「Use the user's already-open browser session only to view content the user can legitimately access.」，同时禁止「Logging into WPS/KDocs without explicit user instruction」与「Bypassing CAPTCHAs, tenant restrictions, paywalls...」。SKILL.md 的浏览器兜底也限定在「公开链接」与「已渲染出的内容」，并要求失败时按边界上报。即兜底用于取公开可见内容，不是用来越权。
- [discrepancy] 写盘是否有路径/文件名风险 — 部分防护、部分缺口（如实记录）：文件名做了 safe_filename 净化（剥离换行、把 \/:*?"<>| 替换为 _、strip(' .')、压缩空白、截断 140 字符），可防非法字符；但**输出路径本身未做任何约束**——--output-dir 与 <title>-全画布.svg 的拼接结果没有 resolve/越界检查，用户（或上游传入的路径参数）若给出 / 或系统目录，脚本会直接在其下建目录写文件。与 repomix-unmixer 的关键区别：本处的路径来自用户命令行与本地文件名，**不可由远端文档内容控制**（远端只影响 title 字符串，而 title 已经过 safe_filename 清洗），因此不构成「外部内容 → 任意写入」的攻击链。风险等级低，但「无路径校验」这一事实应予记录。
- [ok] 是否漏记外部端点（域名全量回查） — 提取结果：drive.kdocs.cn（元数据 API）、wps.processon.com（view 页与 wpsapi 数据 API）、www.kdocs.cn（Referer 与分享页宿主）、www.w3.org（SVG 命名空间字符串，非网络请求）、example.com（文档占位）。逐条已归入 external_deps 的 7 个条目；无未记载的隐蔽外发。
- [ok] 是否有安全降级或隐蔽执行 — 检索结果：无 CERT_NONE/verify=False/--no-verify（urllib 默认走系统 CA 校验）、无 shell=True、无 eval/exec/os.system、无 base64 大块（render_svg_tiles 的 svg 处理是文本拼接而非编码载荷）、无 >200 字符无空格长行。两份脚本全文可读、逻辑线性。
- [ok] 「保真」宣称与实际输出是否相符（有无夸大） — 基本相符，且作者把限制写进文档：manifest 验收检查包含「No text output contains �」「Partial extraction is marked partial in notes」，SKILL.md 也要求「Report extraction gaps plainly. Do not silently produce a polished Markdown file from partial data.」。脚本确实先落原始 JSON 再落派生件（processon-api.json → processon-definition.json → <title>.md），符合「保留原始件」的宣称。唯一需注意：Markdown 的完整度上限取决于上游 definition 是否含全部节点，文档第 4 条硬规则已承认「Do not infer hidden nodes or missing text」。
- [ok] 平台依赖是否被如实声明 — 如实：SKILL.md 明确 'Use scripts/render_svg_tiles.py to make the PNG through macOS Quick Look square tiles'，脚本内也有 shutil.which("qlmanage") is None → raise RenderError("macOS qlmanage was not found") 的硬失败，reference 的失败边界清单亦列出「PNG renderer missing Quick Look or ImageMagick」。即非 macOS 环境下该渲染路径不可用是被声明的，不是隐藏缺陷。
- [ok] pin commit 与 skill.path 是否与任务书表格一致 — git rev-parse HEAD = d5c4678cb5d4fd6acc9c922690df035dbd33d247，与任务书表格『库内 HEAD』逐字一致；目录位于仓库根，相对路径 wps-doc-scraper；本目录最后一次提交 878f947（2026-09-07）。GitHub API 复核 MIT / stars 1392 / pushed 2026-09-15T09:30:04Z。
6Verdict & Scenarios
Best for: Best for: Users who need long-term archiving of publicly shared content from WPS/金山文档 (especially embedded ProcessOn mind maps/canvases)—for knowledge management, competitor research retention, or team documentation backup—and who want to retain raw data and original visual assets for future audit. Also suitable for scenarios requiring the conversion of mind maps into diff-friendly Markdown trees.
Not for: Not suitable for: Documents that require login or restricted permissions to access (the documentation enforces this as a hard boundary and will not log in or bypass restrictions on your behalf); scenarios requiring rewriting or summarizing document content (this is an extraction tool; LLM rewriting is explicitly forbidden); scenarios requiring PNG captures in non-macOS environments (depends on qlmanage); and large-scale bulk scraping (designed for faithful archiving of single public links without bulk scheduling or rate-limiting strategies).
Install Agent-Ready
① Verified Mirror 2026-09-15
Option A · Direct DownloadDownload wps-doc-scraper.tar.gz
sha256:
Option B · Copy Guide for Agentdc2d17f32517aa55…Agents automatically download and verify mirrors from this JSON guide.
② Upstream GitHub
GitHub Repo ↗
Pinned to commit d5c4678cb5.