Workflow & Core Tools · juliusbrussee/caveman

caveman-compress

Name: caveman-compress
Author: juliusbrussee/caveman

Compress a memory file such as CLAUDE.md or a todo list into caveman format to save input tokens, keeping a readable backup. Trigger: /caveman-compress.

L4 High-Privilege · Graded auth & full auditV1 Commit-PinnedExecute CommandNetwork AccessWrite FileDual-pass Audit

Author juliusbrusseeGitHub juliusbrussee/caveman ↗Stars 105691License MIT（LICENSING.md 表格：`skills/` | MIT | Existing Caveman skill stays MIT and untouched.）。仓库为拆分授权：engine/ proxy/ rewriter/ browse/ mcp/ shrink/ 等 Engine-linked 目录为 BSL-1.1（LICENSE.BSL）；GitHub API license.spdx_id = NOASSERTION。Commit 15581d1400

Control levels describe skill capabilities and suggested control measures; they do not represent quality or safety certifications, nor do they dictate usability. Usage decisions rest entirely with users and administrators. Different agent hosts apply varying constraints; do not assume hosts provide automatic protection. Third-party skill copyrights belong to their original authors. Use at your own risk.

1How It Works & Mechanism

This is a rare skill in this batch that contains 'real code': SKILL.md has only one core instruction—execute `python3 -m scripts <absolute_path>` within the directory where SKILL.md resides, with all compression logic implemented in the bundled Python package.

skills/caveman-compress/SKILL.md

The compression scripts live in `scripts/` (adjacent to this SKILL.md). If the path is not immediately available, search for `scripts/__main__.py` next to this SKILL.md. 2. From the directory containing this SKILL.md, run: python3 -m scripts <absolute_filepath>

What is happening here: 'LLM compression' is packaged into a rerun-safe CLI. scripts/__main__.py contains only two lines (from .cli import main; main()), and cli.py performs argument, existence, and type checks before calling compress_file() in compress.py.

The compression action sends the file body to Claude: it prefers the Anthropic SDK (requires ANTHROPIC_API_KEY) and falls back to a local `claude --print` CLI subprocess (using desktop user authentication).

skills/caveman-compress/scripts/compress.py

api_key = os.environ.get("ANTHROPIC_API_KEY") if api_key: try: import anthropic client = anthropic.Anthropic(api_key=api_key, timeout=CLAUDE_CALL_TIMEOUT_SECONDS) msg = client.messages.create( model=os.environ.get("CAVEMAN_MODEL", "claude-sonnet-4-5"),

Decisive evidence: This is the skill's only network egress. The fallback branch uses fixed argument arrays and passes the prompt via stdin with no shell involved—claude_bin = shutil.which("claude") or "claude", args are [claude_bin, "--print", "--setting-sources", "", "--strict-mcp-config"]. The model can be overridden with CAVEMAN_MODEL (default: claude-sonnet-4-5); single call timeout = LOCK_WAIT_SECONDS/(MAX_RETRIES+1) = 300s.

Code blocks are replaced with opaque markers before being sent to the model, and are restored afterward under the rule that 'each marker must appear exactly once'; otherwise, writing is rejected entirely—a core mechanism to prevent compression from corrupting commands.

skills/caveman-compress/scripts/compress.py

def restore_code_blocks(text: str, blocks: List[Tuple[str, str]]) -> str: """Restore markers exactly; fail closed if model removed, copied, or altered one."""

Markers follow the format @@CAVEMAN_PRESERVED_CODE_<n>_<sha256_first_16_hex>@@ (CODE_MARKER_PREFIX); if the input itself contains this prefix, a ValueError is raised immediately; residual unknown markers after restoration also trigger a rejection. The model's only allowed action on code blocks is verbatim relocation without altering content.

Writing does not mean completion: a dedicated structural validator (validate.py) compares headers, code blocks, URLs, paths, bullet points, and inline code between the original and compressed texts; if it fails, it asks Claude to 'fix only the listed errors' and retries up to 2 times.

skills/caveman-compress/scripts/validate.py

validate_headings(orig, comp, result) validate_code_blocks(orig, comp, result) validate_urls(orig, comp, result) validate_paths(orig, comp, result) validate_bullets(orig, comp, result) validate_inline_codes(orig, comp, result)

Extremely strict validation: header text changes count as errors (to avoid breaking in-document anchor links), while mere level changes are warnings; 4-space indented code blocks are also compared (code comments cite a counterexample where `kubectl delete pod --all -n prod` was modified to `-n dev` yet passed validation, directly overwriting user files). The compressed draft is first written to <file>.caveman-staged and only overwrites the target after validation succeeds.

Writing follows a chain of 'backup first, read-back verification, atomic replacement, and preserving permission bits & line-ending styles', with many details engineered to fix past data loss incidents.

skills/caveman-compress/scripts/compress.py

write_bytes_atomic(backup_path, original_raw) if backup_path.read_bytes() != original_raw: print(f"❌ Backup write verification failed: {backup_path}")

The backup is a byte-for-byte exact replica (not a re-render) and is read back and verified after creation; write_bytes_atomic uses mkstemp+fsync+os.replace and preserves original permission bits (comments specifically reference the 0-byte incident in issue #655); read_source enforces strict UTF-8 decoding, rejecting non-UTF-8 inputs directly (issue #686), and preserves majority CRLF/LF endings on write-back (issue #762).

Hard input gates exist: files >500KB are rejected, file names suspected of being credentials/keys are rejected, empty files are rejected, and existing backups cause an abort (to prevent overwrites).

skills/caveman-compress/scripts/compress.py

MAX_FILE_SIZE = 500_000 # 500KB

The sensitive name denylist (SENSITIVE_BASENAME_REGEX) covers .env*, .netrc, credentials*, secret(s)*, password(s)*, id_rsa/ed25519, authorized_keys, *.pem/key/p12/pfx/crt/cer/jks/keystore/asc/gpg; path components and token normalization are also matched (.ssh/.aws/.gnupg/.kube/.docker, api-keys → apikeys, etc.). The comment states the rationale: 'Compressing them ships raw bytes to the Anthropic API — a third-party data boundary'.

Concurrency uses OS-native file locks (POSIX fcntl.flock / Windows msvcrt.locking), with lock keys derived from the first 16 characters of the backup path's sha256; filesystems that do not support locks degrade to uncoordinated execution, and symlinks encountered in lock directories/files are strictly rejected.

skills/caveman-compress/scripts/compress.py

return _state_base_dir("locks") / f"{digest}.lock"

LOCK_WAIT_SECONDS = 900 (comments state this must exceed the worst-case runtime: MAX_RETRIES+1 calls × 300s each); EOPNOTSUPP/ENOSYS are treated as 'filesystem unsupported', logging a warning and continuing, whereas ENOLCK is deliberately excluded (comments explain it may indicate transient lock exhaustion, and misjudging it could lead to lock-free execution under real contention).

Non-expansion invariant: the compressed output, evaluated on body text excluding frontmatter, must be strictly shorter, otherwise execution aborts without touching the original file; this rule also holds for repair runs.

skills/caveman-compress/scripts/compress.py

if not _is_smaller_than_body(compressed_body, body): print(" Original file is untouched (no backup created).") return False

Comments reference issue #776: older versions only checked structure, allowing longer 'compressed' outputs to pass validation and overwrite the original. Frontmatter is stripped separately and reattached verbatim (since models tend to alter frontmatter).

Compressible scope is governed by detect.py extension allowlists/denylists plus content sniffing solely for extensionless files; .original.md backups are never re-compressed.

skills/caveman-compress/scripts/detect.py

COMPRESSIBLE_EXTENSIONS = {".md", ".mdc", ".txt", ".markdown", ".rst", ".typ", ".typst", ".tex"}

SKIP_EXTENSIONS covers py/js/ts/json/yaml/toml/env/lock/css/html/go/sh, etc.; for files without extensions, it checks shebang (→code), JSON/YAML sniffing (→config), and code line ratio >40% (→code), otherwise treating it as natural_language. Comments clarify that any extension not explicitly on the allowlist or denylist is marked unknown and never compressed (fail-closed).

2Core Capabilities

01In-place compression of natural language memory files (CLAUDE.md / todo / preferences) into caveman style to reduce input tokens per session

02Verbatim preservation contract for code blocks, inline code, URLs, paths, headings, and list structures (dual assurance via masking + validation)

03Automatic backup outside the source directory ($XDG_DATA_HOME/caveman-compress/backups/<parent-dir-name>/, or %LOCALAPPDATA% on Windows) to prevent automatic skill loaders from re-reading backups

04Fail-safe operation: validation failure → targeted repair retries ≤2 times → persistent failure aborts with an error, leaving the original file intact (and cleaning up staged backups)

05Pre-rejection of sensitive files: file names suspected of being credentials/keys/PII are rejected prior to reading (transmitting zero bytes to third-party APIs)

06Concurrency safety: cross-session mutual exclusion for the same target file (OS-native locking + 900s timeout with clear notifications)

07Detailed compatibility handling: CRLF/LF preservation, permission bits retention, atomic writes (avoiding 0-byte corruptions on interruption), and strict UTF-8 validation rather than silently dropping invalid bytes

08Built-in benchmark tooling: benchmark.py uses tiktoken (optional) to measure token savings between original and compressed pairs and executes the same validation suite

3External Dependencies

Kind	Dependency / Endpoint
api	Anthropic Messages API（经 anthropic Python SDK）
cli	claude CLI（Claude Code / Claude desktop 认证）
package	anthropic（Python SDK，可选）
package	tiktoken（可选，仅 benchmark.py）
cli	python3（运行时不带第三方依赖；仅标准库 contextlib/errno/hashlib/os/re/shutil/stat/subprocess/tempfile/time/pathlib，Windows 用 msvcrt、POSIX 用 fcntl）

4Recommended Governance & Evidence

Suggested Controls for Safe Operation

For Agent Runtime

Graded/per-use authorization, sandbox recommended

For Administrators

Approval workflow, rollback support, full audit

Evidence & Triggers (File Paths & Excerpts)

scripts/: Executes local script or command: python3 -m scripts <absolute_filepath> (the sole execution mechanism specified in SKILL.md)
网络端点: Declares external network access: Anthropic API via anthropic SDK (client.messages.create), active when ANTHROPIC_API_KEY is present
本地工作目录: File I/O and artifact output: modifies the target file directly (in-place overwrite with compressed content)

Audit Notes: Rationale: (1) Executes bundled Python scripts and spawns subprocesses; (2) Accesses the ANTHROPIC_API_KEY environment variable (credential exposure surface); (3) Dispatches raw file contents to a third-party model API (Anthropic, an external data boundary explicitly noted in the skill's source as a 'third-party data boundary'); (4) Overwrites user files in place (buffered by backups, atomic operations, and validations). Optional dependencies involve remote packages (anthropic, tiktoken). Mitigating factors (kept from lower risk ratings): no shell=True injection paths, static argument arrays, pre-emptive rejection of sensitive file names, backup read-back verification, non-expansion invariants, and rollback on failure; its own SECURITY.md notes a High Risk rating from Snyk. Trigger conditions: actions only occur when the user explicitly triggers /caveman-compress or requests compression of a memory file.

5Second-Pass Verification

[ok] SKILL.md 提到的脚本路径与运行方式 — `python3 -m scripts <file>` 成立：scripts/__main__.py 导入 cli.main 并调用；cli.py 校验参数个数与文件存在/类型后执行 compress_file。SKILL.md 提示的 'search for scripts/__main__.py' 与实际文件名一致。
[ok] Anthropic 为唯一网络出口 — 六个脚本中无 urllib/requests/socket/curl/wget 命中；网络仅经 anthropic SDK client.messages.create 或 claude CLI 子进程。SDK 端点在源码中未硬编码，故 external_deps.endpoint 留空（不臆测）。
[ok] 「subprocess 无 shell 注入」这一说法 — subprocess.run([claude_bin, "--print", "--setting-sources", "", "--strict-mcp-config"], input=prompt, ...) 为列表参数 + stdin，无 shell=True；SECURITY.md 亦声明 'Does not use shell=True or string interpolation in subprocess calls'。
[ok] 「备份在源目录之外」是否属实（SKILL.md / Purpose 段） — backup_dir_for() 返回 _state_base_dir("backups")/filepath.parent.name，其中 _state_base_dir 为 $XDG_DATA_HOME/caveman-compress 或 ~/.local/share/caveman-compress（Windows 用 %LOCALAPPDATA%），确实不在源文件旁；注释给出理由（skill auto-loader 会二次读入 .original.md）。
[ok] 敏感文件拒绝覆盖面的自评（第一遍称「覆盖 .env/credentials/密钥」） — 逐条核对 SENSITIVE_BASENAME_REGEX 与 SENSITIVE_PATH_COMPONENTS/SENSITIVE_NAME_TOKENS：.env*、.netrc、credentials*、secret(s)*、password(s)*、id_rsa/dsa/ecdsa/ed25519(.pub)、authorized_keys、known_hosts、*.pem|key|p12|pfx|crt|cer|jks|keystore|asc|gpg；路径组件含 .ssh/.aws/.gnupg/.kube/.docker 与 credential(s)/secret(s)；名字 token 含 secret/credential/password/passwd/apikey/accesskey/token/privatekey。属启发式 denylist（可被改名绕过），已在 analysis 注记，未夸大。
[ok] 固定大小上限 500KB — compress_file 内 MAX_FILE_SIZE = 500_000 且在任何 API 调用之前检查；SECURITY.md 亦声明 'Files larger than 500KB are rejected before any API call is made'。
[ok] 「校验不过则原文件不动」 — 压缩候选先写 <file>.caveman-staged 并对其跑 validate；只有 is_valid 才 _write_target(filepath, ...)，两次重试用尽后 unlink 暂存与备份并打印 'Failed after retries: original left untouched'。
[ok] SKILL.md 压缩规则 vs 脚本提示词是否一致（有无文档吹牛） — SKILL.md 的 Remove/Preserve/Compress 规则与 build_compress_prompt 的 STRICT RULES 同向（保代码块、保行内代码、保 URL、保标题、保路径命令），脚本还把「不要给整体套一层 ```markdown 围栏」写进提示词，并有 strip_llm_wrapper 兜底剥离。无夸大。

6Verdict & Scenarios

Engineering rigor far surpassing standard text skills: masking + validation + atomic writes + backup read-back + non-expansion invariants establish a robust 'do no harm' default posture.

Clear boundaries and disclosures regarding credential egress: sensitive file names are blocked before read, and SECURITY.md explicitly documents subprocess, I/O, auth boundaries, and the 500KB limit.

Well-considered concurrency and cross-platform support: OS-native locks (auto-released on crash), 900s wait with notifications, graceful degradation for unsupported filesystems, Windows encoding, and .cmd resolution.

Verifiable performance metrics: README includes real-world benchmark tables (average 898 → 481 tokens, 46% reduction), reproducible via benchmark.py.

Documented failure modes: source comments cite issue numbers explaining why each validation exists, allowing users to understand boundary triggers.

Best for: Ideal for teams or individuals maintaining long-lived memory files read on every session (e.g., CLAUDE.md, todo, preferences) who accept content processing via Anthropic API; especially beneficial for users already leveraging caveman-style prompting to cut token costs on the input side. For frequently iterated files, out-of-tree backups and non-expansion invariants significantly mitigate degradation risks.

Not for: Not suitable for: (1) files containing credentials, customer data, or compliance-restricted contents (payload is transmitted to external third-party APIs); (2) environments lacking Anthropic credentials or the claude CLI (scripts will fail); (3) users expecting semantic-lossless compression (only structural integrity is strictly guaranteed); (4) compressing code or config files (non-allowlisted files are rejected); (5) files exceeding 500KB. Note that compression triggers only upon explicit /caveman-compress invocations and never runs automatically.

Install Agent-Ready

① Verified Mirror 2026-09-15

Option A · Direct DownloadDownload caveman-compress.tar.gz

sha256: 52e12dda463f9c55…

Option B · Copy Guide for Agent

install.json

Agents automatically download and verify mirrors from this JSON guide.

② Upstream GitHub

GitHub Repo ↗

Pinned to commit 15581d1400.