Workflow & Core Tools · daymade/claude-code-skills

scrapling-skill

Name: scrapling-skill
Author: daymade/claude-code-skills

Install, troubleshoot, and use Scrapling CLI to extract HTML, Markdown, or text from webpages. Use this skill whenever the user mentions Scrapling, `uv tool install scrapling`, `scrapling extract`, WeChat/mp.weixin articles, browser-backed page fetching, or needs help deciding between static and dynamic extraction.

L5 Restricted · Isolated sandbox & strict auditV2 Human-AuditedTerms / ComplianceExecute CommandNetwork AccessWrite FileDual-pass Audit

Control levels describe skill capabilities and suggested control measures; they do not represent quality or safety certifications, nor do they dictate usability. Usage decisions rest entirely with users and administrators. Different agent hosts apply varying constraints; do not assume hosts provide automatic protection. Third-party skill copyrights belong to their original authors. Use at your own risk.

1How It Works & Mechanism

It does not implement scraping itself: its entire capability lies in invoking the external CLI 'scrapling' properly. The essence of the skill is a three-stage SOP consisting of 'installation diagnosis + scraping tier selection + output verification', with the scraping engine completely outsourced to the external tool.

scrapling-skill/SKILL.md

Use Scrapling through its CLI as the default path. Start with the smallest working command, validate the saved output, and only escalate to browser-backed fetching when the static fetch does not contain the real page content.

This is the core answer to 'why it can achieve its claimed features': capabilities originate from the independent open-source package scrapling, while this skill provides judgment and failure recovery pathways.

Diagnose-first before use is a strict prerequisite: the skill adopts the default assumption that 'the user's installation may be broken', requiring running the built-in diagnostic script first and treating its output as the sole basis for the next step.

scrapling-skill/SKILL.md

Run the bundled diagnostic script first: ```bash python3 scripts/diagnose_scrapling.py ``` Use the result as the source of truth for the next step.

Companion note: 'Do not assume the user's Scrapling install is healthy. Verify it first.'

The diagnostic script uses shutil.which alongside the actual exit code and stderr keywords of `scrapling --help` ('no module named click' / 'install scrapling with any of the extras') to distinguish between 'not installed' and 'installed but missing CLI extras', and directly provides the corresponding remediation commands.

scrapling-skill/scripts/diagnose_scrapling.py

print("status: broken") if "install scrapling with any of the extras" in output.lower() or "no module named 'click'" in output.lower(): print("cause: installed without CLI extras") print("fix: `uv tool uninstall scrapling` then `uv tool install 'scrapling[shell]'`")

Docstring self-description: 'Diagnose a local Scrapling CLI installation and optionally run a smoke test.' All external calls use subprocess argument lists without shell=True.

Scraping intensity follows a three-tier ladder (static get → browser fetch → anti-detection stealthy-fetch), with an explicit rule 'do not use the strongest tier by default'—this is the core methodology of this skill compared to generic 'CLI-invoking' documentation.

scrapling-skill/SKILL.md

- Start with `extract get` for normal pages, article pages, and most WeChat public articles. - Use `extract fetch` when the static HTML does not contain the real content or the page depends on JavaScript rendering. - Use `extract stealthy-fetch` only after `fetch` still fails because of anti-bot or challenge behavior. Do not make it the default.

stealthy-fetch is a first-class entry point for anti-bot / detection evasion capabilities—one of the key bases for security classification.

Output verification is established as strict discipline: declaring success based solely on the exit code is forbidden; files written to disk must be opened to inspect their size and content. This treats 'false success' as the primary failure mode to guard against.

scrapling-skill/SKILL.md

- Do not claim success from exit code alone. Inspect the saved file.

Companion commands: 'wc -c article.md', "sed -n '1,40p' article.md", and 'rg -n '<title>|js_content|rich_media_title|main' page.html'.

It provides a recovery recipe to 'disable verification' (--no-verify) for TLS certificate verification failures, but attaches two verbal prerequisites: 'must first confirm the failure matches curl:(60) pattern' and 'must not disable by default'.

scrapling-skill/SKILL.md

If `extract get` fails with `curl: (60) SSL certificate problem`, treat it as a local trust-store problem first, not a Scrapling content failure. Retry the same command with: ```bash --no-verify ``` Only do this after confirming the failure matches the local certificate verification error pattern. Do not silently disable verification by default.

The same recipe appears repeatedly in references/troubleshooting.md under 'Verified Failure Modes → 3. Static fetch TLS trust-store failure'; additionally, the 'WeChat article smoke test' example command in SKILL.md directly includes --no-verify (see security.injection_surface).

2Core Capabilities

01Installation health check: distinguishes between not installed / missing CLI extras / missing browser runtimes

02Browser runtime probing (Playwright's chromium and chrome-headless-shell caches)

03Reproducible smoke testing: runs static/dynamic scraping on demand, dumps output files to a temporary directory, and prints size and preview

04Targeted scraping of WeChat Official Account article bodies (#js_content selector pattern)

05Three export formats: HTML / Markdown / Text (including -s main content selector)

06Verified failure mode library (3 categories: missing extras / browser not ready / TLS trust store issues) + recovery paths

07Anti-bot evasion escalation path (stealthy-fetch) as a controlled final tier

08Anti-hallucination discipline package (no blind installs, no treating exit code as success, no hardcoded personal absolute paths)

3External Dependencies

Kind	Dependency / Endpoint
cli	scrapling（外部开源抓取 CLI；本 skill 的核心依赖，未随包分发）
cli	uv（工具安装器）
cli	python3（运行自带诊断脚本）
cli	rg / wc / sed（输出验收用，均为常规 POSIX 工具）
package	Playwright 浏览器运行时（由 `scrapling install` 下载，落 ~/Library/Caches/ms-playwright 或 ~/.cache/ms-playwright）
network	任意目标站点（用户给定的 URL；skill 不限定域名，示例用 example.com 与 mp.weixin.qq.com）
network	example.com（文档中的占位目标，非固定端点）

4Recommended Governance & Evidence

Suggested Controls for Safe Operation

For Agent Runtime

Isolated environment, per-use explicit authorization

For Administrators

Strict authorization, full audit, compliance signoff

Evidence & Triggers (File Paths & Excerpts)

SKILL.md / 脚本参数: Contains bypassing automated detection, disabling host sandboxes, or skipping SSL certificate verification
scripts/: Executes local scripts or commands: python3 scripts/diagnose_scrapling.py [--url U] [--selector S] [--dynamic]
网络端点: Declares external network access: scrapling CLI initiates HTTP scraping (static get or browser fetch) against arbitrary user-provided URLs; the skill's scripts themselves make no network calls
本地工作目录: File I/O and artifact output: scraping outputs are saved to user-specified paths (examples: page.html / article.md)

Audit Notes: Flagged as Red at the 'policy/capability level' rather than 'malicious injection level'—the codebase is clean: only 3 files in the entire directory, diagnostic script uses purely standard libraries, subprocess calls use argument lists without shell=True, no credential access, no hardcoded outbound endpoints, and no obfuscation. The Red rating is driven by two capabilities taught as standard operational paths in the documentation: (1) [Security Downgrade] Disabling TLS verification. Both '3. Static fetch TLS trust-store failure' in references/troubleshooting.md and the corresponding section in SKILL.md present `--no-verify` as a recovery recipe; crucially, the 'Diagnose and smoke test a WeChat article body' example command in SKILL.md bundles `--no-verify` alongside `--selector '#js_content'` as a copy-pasteable command; the two verbal prerequisites ('verify curl:(60) pattern first' and 'do not disable by default') exist only in text and cannot be strictly enforced. (2) [Anti-Bot / Evasion] stealthy-fetch is explicitly positioned as an escalation path for anti-bot / challenge scenarios (stated in both Step 3 and Step 6), treating website anti-scraping evasion as a first-class documented capability. Combined with 'arbitrary URL scraping + targeted WeChat Official Account extraction', this falls into the Red category per classification paradigms. Mitigating factors must also be noted: stealthy-fetch is mandated 'not to be the default tier', --no-verify is instructed to be used 'only when matching specific errors', and the skill contains no exfiltration channels (scraped content is saved locally only). Therefore, the Red rating should be interpreted as 'capabilities and guidance are powerful; use with informed consent', rather than 'suspected malicious'.

5Second-Pass Verification

[ok] 每条外部依赖的调用点是否真实存在（而非推测） — 六条依赖全部回到原文：scrapling（Step 2 的 uv tool install 'scrapling[shell]'、Step 4 的 scrapling extract …）、uv（同处）、python3（Step 1 的 python3 scripts/diagnose_scrapling.py）、Playwright 运行时（Step 2 的 scrapling install + 诊断脚本的 ms-playwright 缓存探测代码）、mp.weixin.qq.com（Step 4 的公众号示例）、example.com（Step 4 占位示例）。无推测项。
[ok] 代码层面是否存在被遗漏的网络/凭证行为（找反例） — 反例检索失败：diagnose_scrapling.py 的 import 仅 argparse/shutil/subprocess/sys/tempfile/pathlib/typing；无 urllib/requests/socket；无环境变量或 token 读取（grep os.environ/getenv 零命中）；唯一的网络行为全在被调用的 scrapling CLI 里，本仓库不自带该 CLI。故『脚本自身不发网络请求、不读凭证』成立。
[ok] 『外部 CLI 才是能力本体』这一实现原理是否夸大——skill 自身是否另有抓取实现 — 三文件全部读毕：诊断脚本只做 which/--help/缓存目录探测与一次可选 subprocess 冒烟；troubleshooting.md 全是命令与症状；SKILL.md 全是 SOP。无任何自研抓取/解析代码，原理陈述成立且不夸大。
[discrepancy] 安全降级（--no-verify）是否只出现在文档说明、未进入可执行示例 — 第一遍按上一条思路复核后发现更强的事实：SKILL.md 的『### Diagnose and smoke test a WeChat article body』代码块是把 --no-verify 与 --selector '#js_content' 写在一起的三行命令（python3 scripts/diagnose_scrapling.py \ --url 'https://mp.weixin.qq.com/s/ARTICLE_ID?scene=1' \ --selector '#js_content' \ --no-verify），而诊断脚本确实会把它透传：`if args.no_verify: cmd.append("--no-verify")`。即降级路径不仅被说明、还被脚本实现并示范，不是纯叙述。
[ok] references/troubleshooting.md 与 SKILL.md 是否一致 — 三档抓取顺序（get → fetch → stealthy-fetch）、--no-verify 的限定条件、浏览器缓存两处路径、微信抓取模式在三档文档中表述一致；troubleshooting.md 额外给出 Contents 与更细的症状清单，属补充而非冲突。
[discrepancy] 本 pin 下目录内容与历史（含 .security-scan-passed 标记）是否变化 — git diff 63a65c1fccae9f32f7d1e0e99670321ae09d4f53..HEAD -- scrapling-skill/ 显示唯一变化是删除 scrapling-skill/.security-scan-passed（'1 file changed, 4 deletions(-)'），三个实质文件字节未变。该标记文件是 gitleaks 类密钥扫描的通过记录，与本 skill 的能力/风险结论无关，但意味着『曾有过的安全扫描通过标记在当前 pin 已不存在』，引用旧记录时不应再把它算作现状。另：同 pin 的 data/analysis 目录内已有一份本 skill 的旧侦查记录（commit 63a65c1…），本次为按新 pin 的独立重做，结论方向一致（均判红）而证据面按当前 pin 重新采集。
[ok] pin commit 与 skill.path 是否与任务书一致 — git rev-parse HEAD = d5c4678cb5d4fd6acc9c922690df035dbd33d247；该目录位于仓库根，相对路径 scrapling-skill；本目录最后实质提交 878f947（2026-09-07）。GitHub API 复核 MIT / stars 1392 / pushed 2026-09-15T09:30:04Z。

6Verdict & Scenarios

Makes 'environment verification first' the initial workflow step with machine-readable determinations, avoiding the most common false failure (mistaking missing CLI extras for target site unreachability).

The tier ladder (get → fetch → stealthy-fetch) provides an explainable cost/benefit decision framework rather than jumping straight to 'use a browser if it fails'.

Output verification rules are specific and actionable (wc -c / sed preview / rg matching key containers), directly addressing 'exit code 0 with an empty shell' as the primary hallucination in scraping tasks.

Scraping WeChat Official Account articles includes clear default selectors and verification checkpoints, crystallizing a high-frequency specific task into a reusable recipe.

Best for: Best for: daily tasks requiring reliable local retrieval of webpage content, especially archiving WeChat Official Account article bodies, scenarios requiring a judgment of 'only use browser if static fetch fails', and users who just installed scrapling but are unsure if it was set up correctly (the diagnostic script is a single command). Also suitable as an exemplary study on 'how to properly use an external scraping CLI'—its verification disciplines are worth copying verbatim.

Not for: Not suitable for: scraping tasks requiring compliance reviews (the skill does not verify site authorization and provides an anti-bot evasion escalation tier); scenarios requiring scraping authenticated/paywalled content (scrapling's capability boundaries and site authorizations are out of scope for this skill); environments expecting the skill to include a built-in scraping engine (it requires external installation of scrapling + Playwright runtimes); scenarios with strict TLS verification requirements (the documentation provides a recipe to disable verification).

Install Agent-Ready

① Verified Mirror 2026-09-15

Option A · Direct DownloadDownload scrapling-skill.tar.gz

sha256: 65c575aca652cc60…

Option B · Copy Guide for Agent

install.json

Agents automatically download and verify mirrors from this JSON guide.

② Upstream GitHub

GitHub Repo ↗

Pinned to commit d5c4678cb5.