Workflow & Core Tools · daymade/claude-code-skills
github-sensitive-data-cleanup
Scan and remove sensitive data (secrets, API keys, private domains/IPs, PII) from GitHub repository history. Use this skill whenever the user says "scan sensitive data", "clean git history", "remove secrets from repo", "sanitize GitHub history", "清理敏感数据", "历史重写", "force push", "泄露", or needs to repair a public repo after accidental secret/private context leakage. Also use before any force push to a public repository to verify visibility, backup, and scan results.
L4 High-Privilege · Graded auth & full auditV1 Commit-PinnedExecute CommandNetwork AccessWrite FileDual-pass Audit
Control levels describe skill capabilities and suggested control measures; they do not represent quality or safety certifications, nor do they dictate usability. Usage decisions rest entirely with users and administrators. Different agent hosts apply varying constraints; do not assume hosts provide automatic protection. Third-party skill copyrights belong to their original authors. Use at your own risk.
1How It Works & Mechanism
The entire skill is structured as an orchestration workflow (SKILL.md) combined with four executable Python scripts and three reference documents. SKILL.md specifies only execution steps and operational constraints, delegating all mechanical operations to scripts under `scripts/`, which invoke `git`, `gitleaks`, `git-filter-repo`, and `gh` via `subprocess`.
github-sensitive-data-cleanup/SKILL.md
The bundled scripts automate the mechanical parts:
Operational mechanics: SKILL.md contains no shell logic directly; the agent interprets "Steps 0–7 plus 7 non-negotiable safety rules", while execution is delegated to four standalone CLI scripts. The practical ceiling of this skill is strictly bounded by the capabilities of these scripts, not by prompt phrasing.
Scanning uses a four-layer architecture: Layer 1 for gitleaks (secrets/tokens), Layer 2 for custom regexes (intranet IPs, phone numbers, repo-specific PII), Layer 3 for private domain/IP rules extracted from user-supplied gitleaks configs plus an optional identities file, and Layer 4 for agent-executed manual AI semantic reviews. The scripts handle Layers 1–3 and tag Layer 4 for manual review.
github-sensitive-data-cleanup/scripts/scan_repo.py
Layer 4: AI semantic review flag (must be performed manually by an agent)
Documented in the script header docstrings. Key design rationale: regex engines cannot reliably capture unlisted private contexts (e.g., real personal names, project codenames, transcript snippets), requiring the scripts to flag these tasks for manual/model review via `ai_semantic_review_required: True` in the report dictionary.
Full history scans do not rely solely on gitleaks: Layers 2 and 3 collect all commits using `git rev-list --all` and stream them in chunks (`batch_size=500`) to `git grep --perl-regexp`, aggregating matching commit hashes to prevent command-line length overflows and avoid scoping issues limited to shallow commit depths.
github-sensitive-data-cleanup/scripts/scan_repo.py
and to ensure the entire history is searched (not just the newest N).
Implementation detail: `grep_all_commits()` invokes `git grep -n --perl-regexp -e <pattern> <batch...>` per pattern. A `returncode == 1` with empty stdout denotes no matches for the batch (non-error); it returns a deduplicated `set` of commit hashes. `verify_cleanup.py` imports `grep_all_commits` directly from `scan_repo.py` to prevent logic drift between scan and verification steps.
Private rules are decoupled from the public repository: Layer 3 avoids hardcoding private domain lists by parsing regex patterns corresponding to two specific rule IDs (`private-domain-context` and `private-ip-context`) directly from the user's local gitleaks TOML file at runtime, alongside an optional line-by-line identities file (treated as literals via `re.escape`).
github-sensitive-data-cleanup/scripts/scan_repo.py
LAYER3_RULE_IDS = ["private-domain-context", "private-ip-context"]
Operational mechanics: Resolves the paradox of distributing a public skill without embedding proprietary identifiers. Rules remain locally stored (`~/scripts/git-pii-guard/gitleaks.toml` or `.pii-patterns`), with the skill acting as an extractor. `DEFAULT_PATTERNS` strictly includes generic patterns (10./172.16-31./192.168. subnets and regional phone numbers), explicitly commenting `Do NOT hardcode real private domains here`. TOML parsing prioritizes standard `tomllib` (Python 3.11+), falling back to a minimal parser handling triple- and double-quoted regex strings.
Mandatory backup gating precedes history rewriting: the pipeline asserts that a) `git-filter-repo` is executable on PATH, b) the working tree is clean (aborting on any stdout from `git status --short`), and c) a full backup created via `git bundle create --all` passes immediate validation with `git bundle verify`. Any failure halts execution prior to rewriting.
github-sensitive-data-cleanup/scripts/rewrite_history.py
raise RuntimeError(f"Backup verification failed: {verify.stderr}")
Execution sequence: `is_dir(.git)` → verify replacements file exists → verify message-replacements file exists → `shutil.which(git-filter-repo)` → `--version` check → `check_clean_working_tree()` → print DESTRUCTIVE OPERATION banner → require `--yes` → create backup → execute rewrite. Backup errors catch `CalledProcessError`/`RuntimeError` and trigger `exit(1)`. Rewrite interruptions print an explicit notice indicating the backup location.
History rewriting executes via `git-filter-repo --force --replace-text <file>`, with an optional `--replace-message <file>` flag to sanitize commit metadata channels, mitigating vulnerabilities where sensitive identifiers persist inside commit messages.
github-sensitive-data-cleanup/scripts/rewrite_history.py
cmd += ["--replace-message", str(message_replacements_path)]
Post-rewrite, the script generates a matching `.json` report (`backup_path.with_suffix(".json")`) tracking `old_heads`/`new_heads` via `git show-ref --heads` snapshots, outputting instructions to run `verify_cleanup.py` followed by `safe_push.py`. The `--force` flag is passed directly to `git-filter-repo` to bypass fresh-clone constraints, not to `git push`.
Verification adopts dual-channel validation to prevent blind-spot correlation: the blob channel executes `git grep` across all commits, while the message channel parses `git log --all --format=%H%x1f%B%x1e` output to run pattern matching against commit messages, outputting the first 10 matching commit hashes on failure.
github-sensitive-data-cleanup/scripts/verify_cleanup.py
"--format=%H%x1f%B%x1e", "--no-color"],
Patterns are reverse-extracted directly from the rewrite replacements file (`extract_patterns_from_replacements` parses `literal:`, `regex:`, and plain strings) to ensure isolation from post-rewrite edits to `.pii-patterns`. Any pattern hit or command failure outputs `VERIFICATION FAILED` and triggers `exit(1)`. Literal patterns are sanitized via `re.escape` before being evaluated by PCRE grep.
Push operations enforce fail-closed gating: `gh repo view --json visibility,isPrivate,stargazerCount,forkCount,owner,name` inspects repository metadata directly via the current working directory without string guessing. If `gh` fails or omits expected keys, execution aborts immediately. Public repositories with `forkCount > 0` emit warnings, omitting `--yes` aborts execution, and pushes attempt `--force-with-lease` before single-attempt `--force` fallbacks triggered only by stale info errors.
github-sensitive-data-cleanup/scripts/safe_push.py
if "stale info" in result.stderr.lower():
Operational mechanics: Mandates verifiable proof that a target repository's history has not been forked prior to a force push. Metadata validation occurs strictly before the confirmation prompt (`--yes`) and the subsequent push command. Missing keys trigger `Aborting rather than using fallback values.` to fail closed. The `--no-verify` flag is explicitly omitted across all scripts.
2Core Capabilities
01Four-layer sensitive data scanning (gitleaks + custom regex + private context extraction + AI semantic review flags)
02Exhaustive Git history searching across `--all` commits using batched `git grep --perl-regexp`, returning distinct commit hashes
03Decoupled private rules: extracts target rule regexes directly from local user gitleaks configurations without embedding private identifiers into public skills
04Automated repository history rewriting with mandatory backup and verification gates (`git bundle create --all` and `git bundle verify`)
05Dual-channel commit message and blob rewriting/verification (`--replace-message` and `git log --all` parsing)
06Pre-push repository visibility and fork count checks, aborting if metadata validation fails
07Graduated force pushing: prefers `--force-with-lease`, falling back to `--force` exclusively on stale ref state, while rejecting `--no-verify`
08Incident-derived operational guidelines and manual escalation triggers (e.g., active secrets exposed across repositories with >50 forks)
3External Dependencies
| Kind | Dependency / Endpoint |
|---|---|
| cli | gitleaks（Layer 1 密钥扫描） |
| cli | gitleaks detect（子进程实参：--source / --report-format json / --report-path / --verbose） |
| cli | git-filter-repo（历史重写，须在 PATH 且 --version 可执行） |
| cli | git（bundle create --all / bundle verify / status --short / show-ref --heads / rev-list --all / grep --perl-regexp / log --all / push --force-with-lease|--force） |
| cli | gh（GitHub CLI，用于仓库可见性与 fork 数核查） |
| api | GitHub REST API（由 gh repo view 隐式访问；源码内无字面 URL，端点归属 GitHub） |
| network | PyPI（文档命令 `uv run --with gitpython ...` 会要求 uv 从包索引获取 gitpython；四个脚本实际零 gitpython import，属文档声明依赖——[INFERENCE] 端点归属，源码内无 URL） |
| package | gitpython（SKILL.md 与四个脚本 docstring 的命令示例中声明；代码无 import，属未使用的声明依赖） |
| cli | uv（文档规定的运行器：uv run --with gitpython / uv run scripts/...） |
| cli | date（取 UTC 时间戳写入报告 scanned_at） |
| cli | brew（仅安装说明：git-filter-repo / gitleaks / gh） |
| cli | BFG Repo-Cleaner（参考文档里的备选工具，需 Java；未被脚本调用） |
| network | 用户侧的私有规则/身份清单文件（非网络端点，但属外部运行时输入）：私有 gitleaks.toml、identities.txt、repo 根 .pii-patterns |
4Recommended Governance & Evidence
Suggested Controls for Safe Operation
For Agent Runtime
Graded/per-use authorization, sandbox recommended
For Administrators
Approval workflow, rollback support, full audit
Evidence & Triggers (File Paths & Excerpts)
- scripts/: Local script execution: `scripts/scan_repo.py` detects binary via `shutil.which("gitleaks")` and executes `gitleaks detect` using `subprocess`.
- 网络端点: External network access: `safe_push.py` calls `gh` CLI targeting GitHub REST API (`api.github.com`) to query repository `visibility`/`isPrivate` metadata.
- 本地工作目录: File I/O and artifact output: `scan_repo.py` writes JSON reports to `--output` (creating parent directories as needed); temporary gitleaks files are cleaned up post-scan.
Audit Notes: Escalated due to three significant operational surfaces: 1) Inherently irreversible destructive operations: `rewrite_history.py` executes `git filter-repo --force` against target repositories, and `safe_push.py` issues force pushes to remote GitHub repositories (`--force-with-lease` falling back to single `--force` on stale info), meaning remote forks and upstream caches cannot be invalidated. 2) Ingestion of sensitive local configurations: `--identities-file`, `--gitleaks-config`, `.pii-patterns`, and replacement files process plaintext secrets, internal domains, and personal identities. 3) Remote package management requirements: documentation templates call `uv run --with gitpython`, resolving dependencies from remote indexes (though scripts do not import gitpython directly). However, comprehensive gating is present: clean working tree enforcement, bundle creation with verification, mandatory `--yes` flags, fail-closed `gh` visibility checks, lease-first pushes, and total absence of `--no-verify`. Destructive actions require deliberate operator invocation following SKILL.md. Read-only scan usage (`scan_repo.py` / `verify_cleanup.py`) operates with standard blue-tier surface boundaries; executing history rewrites or pushes warrants orange-tier operational controls requiring prior validation of `--repo`, `--branch`, and `--backup` parameters.
5Second-Pass Verification
- [ok] 文件清单全量核对（有无未列出的隐藏脚本/数据文件） — glob（hidden=true, gitignore=false）结果仅 9 个文件：SKILL.md、evals/evals.json、references/{ai_semantic_review_prompt,incident-lessons,tooling_notes}.md、scripts/{rewrite_history,safe_push,scan_repo,verify_cleanup}.py。无隐藏文件、无二进制、无 .pii-patterns 样本、无内置 hook；仓库级的 PII Guard hook（~/scripts/git-pii-guard）与 .githooks/ 属仓库/机器环境，不在 skill 目录内，skill 只是要求不得绕过它。
- [ok] external_deps 逐条反查调用点 — 全部命中：gitleaks → scan_repo.py `gitleaks_bin = shutil.which("gitleaks")` + verify_cleanup.py 同名调用；git-filter-repo → rewrite_history.py `shutil.which("git-filter-repo")` 且随后 `--version` 探测；git → show-ref/status/bundle create/bundle verify/rev-list/grep/log/push 各自 subprocess.run 均可定位；gh → safe_push.py 的 `["gh", "repo", "view", "--json", ...]`；date → scan_repo.py `["date", "-u", "+%Y-%m-%dT%H:%M:%SZ"]`；uv/brew 只出现在文档命令与 Prerequisites，无对应代码调用（已在条目 note 中说明）。
- [ok] 安全结论反例搜索：是否存在漏掉的网络调用、隐藏外发或混淆内容 — 未找到反例，安全结论成立（记录检索范围以免过度外推）：对 skill 目录 grep `requests|urllib|socket|curl|wget|http://|https://` 零命中，且全目录无任何 http(s):// 字面 URL（连「联系 GitHub Support」一段也只用文字）；无 base64/fromhex/codecs/eval/exec/__import__ 等混淆或动态执行构造，无 60 字符以上的不透明长 token；对 git / git-filter-repo / gitleaks / gh / date 之外的二进制调用为零。唯一出网路径是 safe_push.py 经 gh 访问 api.github.com（uv --with gitpython 的包索引访问属 uv 行为而非脚本行为，已在 external_deps 标注 [INFERENCE] 端点）。证明边界：这是对该 commit 下 9 个文件的静态核对，不含运行时行为。
- [ok] 实现原理是否夸大：SKILL.md「What the Bundled Scripts Do」逐条对代码 — 四条声明全部落地：scan_repo.py 确实同跑 gitleaks 与自定义 grep 层并输出 JSON；rewrite_history.py 确实先备份再 `git filter-repo --replace-text`，且 `--message-replacements` 确实映射到 `--replace-message`；verify_cleanup.py 确实同时查 blob（git grep 全 commit）与 commit message（`git log --all --format=%H%x1f%B%x1e`），使漏改 message 的重写必然失败；safe_push.py 确实调 gh repo view 并走 lease-first。SKILL.md 未出现代码做不到的承诺。
- [ok] 7 条不可协商安全规则在 SKILL.md 的位置与代码落地情况 — SKILL.md 第 64~77 行「Safety Rules (Non-Negotiable)」编号 1~7：scan-before-decide / backup-before-rewrite / gh repo view 验可见性且不得从 URL 或目录名推断 / 绝不 --no-verify / --force-with-lease 优先且仅在 remote ref stale 时回退 --force / 重写后必须重新扫描 + AI 语义审查 / 公开且带 forks 需格外小心。落地核对：规则 2、4、5 有代码门禁（备份+verify、脚本从不加 --no-verify、lease→stale info→force 一次）；规则 3 有硬门禁（gh 失败或元数据缺 key 即 abort）；规则 1、6、7 依赖 agent 遵守文档与人工动作（scan 结果、--yes、fork 通知），不由代码强制——这是规则与实现的唯一不对称处。
- [ok] safe_push.py 的 public+forks 判定与 force-with-lease 回退条件 — 判定式为 `if not info["isPrivate"] and info["forkCount"] > 0:`，命中时只向 stderr 打印 `WARNING: This is a PUBLIC repository with {forkCount} forks.`，不阻断流程；唯一的硬门禁是缺 --yes 时 exit(1)。回退条件为 `if "stale info" in result.stderr.lower():`（对应 git 对 force-with-lease 拒绝时的 (stale info) 文案），回退仅一次 (--force)，再失败即 exit(1)。注意：脚本不校验 remote/branch 是否与默认分支一致，也不在回退前复跑验证——--repo/--remote/--branch 参数的正确性由操作者负责。
- [discrepancy] 文档声明的依赖 gitpython 是否真实被使用 — 四处 docstring 与 SKILL.md 多处命令写 `uv run --with gitpython scripts/...`，但四个脚本的 import 块逐行核对后只有 argparse/json/re/shutil/subprocess/sys/tempfile/tomllib/pathlib，零 `import gitpython` 或 `from git import`。即文档要求 uv 从包索引拉一个不使用的第三方包（多余供应链接触面），去掉 `--with gitpython` 不影响任何功能；此外同一份 SKILL.md 内部不一致：Step 4/5/6 的命令写 `uv run scripts/...`（不带 --with），Step 1/1.5 与「What the Bundled Scripts Do」写 `uv run --with gitpython`。
- [discrepancy] --gitleaks-config 是否被传给 gitleaks 本体 — run_gitleaks 的 cmd 只有 `detect --source --report-format --report-path --verbose`，没有 `--config`；用户私有 gitleaks.toml 仅被 parse_gitleaks_rules 解析出 private-domain-context / private-ip-context 两条 regex 供 Layer 3 的 git grep 使用。SKILL.md 的说法（该 flag 用于从私有配置读取这两条规则）与代码一致，不存在夸大；但若用户以为私有配置会整体驱动 gitleaks 扫描（例如自定义 allowlist/其它私有规则），实际不会——gitleaks 走自身默认配置与仓库本地 .gitleaks.toml。此项为「文档未声明但易误读」的行为差异，非虚假声明。
6Verdict & Scenarios
Best for: Suitable for developers and security teams who need an end-to-end remediation pipeline (scan → backup → rewrite full history including commit messages → dual-channel verify → check visibility → push) after leaking secrets, private domains, internal IPs, or PII to GitHub repositories. It also functions as a structured force-push gatekeeper ensuring backups and visibility audits occur before destructive updates. For safest execution, run `scan_repo.py` and `verify_cleanup.py` first in read-only local mode, inspect the replacement mapping files and target flags (`--repo`, `--remote`, `--branch`, `--backup`), and then execute rewrite and push operations.
Not for: Not intended as an authoritative certification that a leak has been permanently erased: it cannot modify cached GitHub infrastructure, revoke forks, rotate compromised credentials, or guarantee complete semantic coverage of unstructured text (Layer 4 remains non-deterministic). Not suitable for unattended or bulk multi-repo rewriting operations (no remote/branch allowlisting exists; `--yes` acts as the sole gating check). Not recommended for minimal-intervention fixes where new commits can be handled by credential rotation plus standard revert/gitignore updates, nor for environments without force-push privileges or where invalidated PR branches cannot be tolerated.
Install Agent-Ready
① Verified Mirror 2026-09-15
Option A · Direct DownloadDownload github-sensitive-data-cleanup.tar.gz
sha256:
Option B · Copy Guide for Agentaa1ae4b1f420e74a…Agents automatically download and verify mirrors from this JSON guide.
② Upstream GitHub
GitHub Repo ↗
Pinned to commit d5c4678cb5.