Content & Media Creation · heygen-com/hyperframes
embedded-captions
Add captions or subtitles to an existing single-subject talking-head video without editing the footage. Use for plain verbatim captions, cinematic captions embedded behind the subject, VFX captions, “炸/特效/酷炫字幕,” or a named identity from the 35-style catalog. Route by visual identity, not by backend engine. The quiet `anchor` rail is the default; embed every word only when the user explicitly wants a fully cinematic treatment. The workflow runs locally end to end, including transcription and subject matting; split multi-shot footage before applying it.
L4 High-Privilege · Graded auth & full auditV1 Commit-PinnedExecute CommandNetwork AccessWrite FileDual-pass Audit
Control levels describe skill capabilities and suggested control measures; they do not represent quality or safety certifications, nor do they dictate usability. Usage decisions rest entirely with users and administrators. Different agent hosts apply varying constraints; do not assume hosts provide automatic protection. Third-party skill copyrights belong to their original authors. Use at your own risk.
1How It Works & Mechanism
The actual selection surface is a 35-row identity catalog (10 classic DNA + 25 themed), where the engine and authoring files are derived via table lookup; SKILL.md explicitly prohibits exposing backend engine names as questions to users.
skills/embedded-captions/SKILL.md
**Never surface "Standard vs Cinematic vs Theme" as a question** — those are backend names (a product has one UX even with several engines). The catalog encodes everything routing needs: reading surface, voice, recommend-for, scene needs, adjacency notes for the genuinely-close pairs (loud↔ordnance, neon↔neonsign, cream↔stardust).
What is happening here: Encapsulating the complexity of 'three engine systems' inside the product. Users only select a visual identity (such as anchor/cream/ordnance), and the skill derives the authoring file and compilation pipeline from rows in CATALOG.md. Verification: CATALOG.md actually contains 35 identity rows starting with `| \``; 10 JSON files under dna/; 25 JSON files under themes/—all three are consistent.
Subtitles are modeled into three states: drop (filler words, not displayed) / rail (verbatim lower-third subtitles in front, carrying most text) / embed (promoted peak words, composited behind the subject, occluded via subject matting). Embed is defined as a scarce asset.
skills/embedded-captions/SKILL.md
| **rail** | the default — ordinary spoken content (verbatim) | clean lower-third subtitle, **in front**, readable. A punch word can get an inline `emphasis` highlight (accent colour / active-word pop) — it stays on the rail. | | **embed** | a promoted peak — the headline beat | one big word composited **behind the subject** (matte occlusion), designed entrance + exit |
What is happening here: Using 'occlusion' as the VFX source—first matting the subject from the video, then placing peak words behind the subject layer so text is occluded by the person before revealing. Scarcity is a hard rule: '≤1 hero per block (thought), never two co-visible, ≥ a beat of air between hero windows (the compiler warns under 0.6s)', and among multiple heroes, only the largest (APEX) receives full lockup treatment.
Preprocessing runs via a single parallel command: matting ∥ transcription ∥ audio envelope run concurrently, followed by serial calculation of safe-zones—turning 'three manual steps prone to omission' into a single call.
skills/embedded-captions/scripts/prepare.sh
# matte.cjs (CPU-heavy ONNX) ∥ transcribe.cjs (whisper) → then safe-zones.cjs # (matte and transcribe are independent; safe-zones needs the matte's frames_fg.) # Replaces hand-running steps 2 / 3 / 3b — one call, nothing forgotten, ~the cost # of the slower of the two instead of their sum.
What is happening here: Three background processes (matte/transcribe/audio-envelope) redirect logs and wait, followed by running safe-zones.cjs; on failure, each echoes tail logs and exits with a non-zero code. Output includes frames_fg/, frames_bg/, matte.fps, transcript.json, and safe-zones.json—the complete input set for downstream authoring.
Transcription runs locally via WhisperX (via uvx, pinned version), using wav2vec2 forced alignment to obtain word-level timestamps to satisfy downstream 80ms strict alignment gating; falls back to repository-local whisper.cpp on failure.
skills/embedded-captions/scripts/transcribe.cjs
// Pin whisperx so `uvx` fetches a reproducible build instead of resolving // "latest" on every run (a supply-chain + determinism foot-gun). Override // with $WHISPERX_VERSION if you've validated a different release. const whisperxSpec = `whisperx==${process.env.WHISPERX_VERSION || "3.8.6"}`;
What is happening here: Third-party package versions are pinned (preventing supply-chain issues and non-reproducibility), invoked as `uvx --python 3.12 --from whisperx==3.8.6 whisperx <wav> --model small --device cpu --compute_type int8 --output_format json`; defaults to the multilingual `small` model rather than `small.en` (code comments explain that .en models mis-transcribe non-English and heavily accented speech).
Matting does not bundle weights; it shells out to the repository CLI's remove-background (u2net_human_seg), downloading ~168MB to the user cache directory on first run; VFR sources are normalized to CFR prior to matting.
skills/embedded-captions/scripts/matte.cjs
const r = cp.spawnSync("node", [hfCli(), "remove-background", matteSrc, "-o", mov], { stdio: ["ignore", "pipe", "pipe"], encoding: "utf8", });
What is happening here: Offloading 'model assets' to CLI cache management, with zero bundled weights in the script. Comments explain why: legacy implementation bundled a 34MB PP-MattingV2 + onnxruntime inference loop. VFR normalization is backed by empirical data ('observed: 2251 fg frames vs 902 bg on one clip').
The compositing stage uses ffmpeg with the matte as an occlusion layer: embed renders 'background + captions first, then overlays subject alpha'; rail renders 'transparent WebM, stacked to the very front'; frame rate strictly adheres to matte.fps for frame alignment.
skills/embedded-captions/scripts/render-and-composite.sh
# FPS: matte.fps (written by matte.cjs at the source's NATIVE rate) is authoritative # so the matte overlay stays frame-aligned with the render. Falls back to plan.fps / # frame-count inference / 24. Warn if plan.json fps disagrees with the matte.
What is happening here: The renderer uses plan.duration while the matte frame rate comes from the source native frame rate; mismatches cause subject floating or trailing black frames. The script clamps output duration to min(matte frames/fps, source duration) and includes a 'Bug-1 guard' against foreground-only tails. It also snapshots index.html + plan.json into history/ for rollback.
Geometric gating uses headless Chromium DOM bounding rects × matte alpha for pixel-level verification (occlusion, frame clipping, rail/climax collision), with --strict enforcing 80ms timing alignment; fast preview frames (~2s/frame) catch layout issues before paying for full rendering.
skills/embedded-captions/SKILL.md
`node scripts/preview-frames.cjs <project> [t…]` composites **faithful preview frames in ~2s each** (caption layers screenshotted at seek-time + real video frame + matte occlusion + rail overlay = what the final composite will look like at that moment).
What is happening here: Turning 'rendering is expensive → inspect cheaply first' into procedural discipline ('A full render costs minutes — never use it to _discover_ layout problems.'). Gate checklist: check-timing.cjs --strict (80ms), check-occlusion + measure-layout (subject occlusion and frame bounds), check-overflow (warning only), and check-rail-climax (duplicate reveals between rail and climax, hard failure).
2Core Capabilities
01Caption overlay without re-editing original footage: Dual-track Standard/rail + peak embed model (drop/rail/embed tri-state allocation)
0235-identity visual catalog + scene-parameterized DNA (accent color sampling, contact shadow aligned to light direction, depth-of-field matched blur, hero amplitude scaling with RMS audio loudness)
03Cinematic pure-embed compilation: Authors only write blocks (word lines + planes + CSS + single hero); compiler auto-generates timeline, intra-block accumulation, inter-block transitions, hero three-act structure, and apex/minor layering
04Theme mode: Complete theme framework (body pattern × hero setpiece × front fx × plate reaction), verifying word-level integrity upon compilation and yielding final_fx.mp4 upon rendering
05Local word-level transcription (WhisperX via uvx) with near-silence guards (preventing Whisper hallucinations during silence) and gibberish transcription rejection
06Subject matting and occlusion compositing (CPU-only ONNX, ~2fps@1080p, automatic weights download)
07Decision gates: Multi-speaker/hard cuts, no subject, short clips/no speech/invisible face, pre-existing burned-in captions, handheld shake—each with explicit rejection or split handling
08Cost-effective previews (~2s/frame) and dual QA checklists: Negative checklist blocks layout breaks; positive 5-point reference-bar checklist evaluates design quality, with recommendation to use fresh-eyes sub-agents for blind preview reviews
3External Dependencies
| Kind | Dependency / Endpoint |
|---|---|
| cli | ffmpeg / ffprobe（系统） |
| cli | hyperframes CLI（init / remove-background / render / lint / check / snapshot / preview） |
| cli | uvx（Python 运行器，按需拉取 whisperx==3.8.6） |
| network | 抠像权重下载（u2net_human_seg.onnx，约 168MB，仅首次） |
| network | GSAP CDN（仅示例渲染与引擎模板 HTML 里引用） |
| package | sharp（图像/alpha 数学）与 puppeteer（版面度量、截图） |
| package | python3（Theme 的 drawon setpiece 在编译期调用 gen-stroke-path.py） |
| cli | hyperframes skills update（自刷新，需网络） |
4Recommended Governance & Evidence
Suggested Controls for Safe Operation
For Agent Runtime
Graded/per-use authorization, sandbox recommended
For Administrators
Approval workflow, rollback support, full audit
Evidence & Triggers (File Paths & Excerpts)
- scripts/: Local script/command execution: bash scripts/prepare.sh <project> (spawns node matte.cjs / transcribe.cjs / aud in parallel)
- 网络端点: Declared external network access: uvx fetches whisperx==3.8.6 and dependencies on demand from PyPI (pinned version to prevent 'latest' non-reproducibility)
- 本地工作目录: File I/O and artifact output: <project>/frames_fg/, frames_bg/, matte.fps, audio.mp3, _wx_audio.wav, _wx_out/ (
Audit Notes: Behavioral analysis: 18 .cjs + 3 .sh + 1 .py local execution chain involving extensive local file I/O and ffmpeg/Chromium invocations. External network access is strictly targeted and pinned: whisperx==3.8.6 via uvx on PyPI, hyperframes CLI-managed u2net matting weights (~168MB, one-time), gsap@3.14.2 on jsdelivr, and `skills update` self-refresh. **No credential, cookie, or keychain reading detected** (unlike HeyGen-dependent skills); no TLS downgrades, anti-scraping bypasses, or arbitrary code execution vectors (subprocesses execute fixed binaries/scripts with arguments derived from local files). Safe from orange tier. Key operational notes: ① First run downloads third-party Python packages and model weights over network (supply chain surface); ② Processes local raw user video, persisting segmented matte frames (containing subject likeness) to the project directory, requiring proper directory permissions and lifecycle cleanup; ③ Transcribed text enters model context, forming a prompt injection surface at the content level.
5Second-Pass Verification
- [ok] 是否隐藏凭证读取（与同批 HeyGen 系 skill 的差异） — 对整目录 grep `\.heygen|HEYGEN|api[_-]?key|TOKEN|SECRET|keychain|Bearer|Authorization` → 0 命中；process.env 命中 12 处全部是 HYPERFRAMES_ROOT / WHISPER_MODEL / WHISPER_LANG / TRANSCRIBE_ENGINE / WHISPERX_VERSION 等行为开关。『全本地、无凭证』成立。
- [ok] 网络外发是否全部可预期 — 逐条回查：uvx 拉 whisperx（版本钉死 3.8.6，注释明说是为防供应链与不可复现）；模型下载由 hyperframes CLI 自管且路径固定在 ~/.cache/hyperframes/；CDN 引用仅在 engine.html / example-renders / _archive 模板内；无其它域名。未发现隐藏上报或遥测。
- [ok] 实现原理与功能声明是否相符 — 'runs locally end to end, including transcription and subject matting' 成立（本地 WhisperX + 本地 ONNX 抠像）；'without editing the footage' 成立（Standard/Cinematic 只叠加，脚本注释与 SKILL.md 均强调不调色/不覆盖）；唯一被允许改原片的是 Theme 的 PLATE 反应预算，SKILL.md 已显式标注为例外。
- [discrepancy] SKILL.md 的三引擎描述与脚本实际状态不一致（文档漂移） — SKILL.md 开篇仍把 Standard 描述为默认模式（'**Standard** (default) builds a clean verbatim **rail** … + an **embed** climax'），并在多处按 rail.html + index.html 描述两轨产出；但同 commit 的 scripts/render-and-composite.sh 第 35-37 行写着 'Standard mode retired 2026-06-12 — rail-surface needs are served by theme DNAs like "anchor"'，dna/README.md 第 15-16 行同样写 'Standard/rail mode was retired 2026-06-12'，make-composition.cjs 第 253-257 行则在检测到 standard 派生的 plan.json 时直接报错退出。然而 scripts/ 下仍保留完整的 STANDARD 合成分支（第 392 行起 'STANDARD mode (rail + embed) — detected by rail.html'）、modes/standard/ 目录与 44 个字体仍在包内。结论：**rail + peak embed 的实现仍在，但文档层（SKILL.md）已落后于引擎层（dna/README、render 脚本注释）**，两处关于『默认模式』的说法互相矛盾。这是对 pin commit 有效的真实出入，非第一遍误读。
- [ok] 『35 身份』等数量声明的准确性 — 实测 CATALOG.md 身份行 = 35；dna/*.json = 10（chrome/cream/documentary/editorial/glitch/ink/keynote/loud/neon/velocity）；themes/*.json = 25；references/*.md = 14。与 SKILL.md 自述一致。
- [ok] 门禁是否真的会执行（而非只写在文档里） — render-and-composite.sh 内确实依次调用 check-timing（plan+transcript 存在时）、measure-layout + check-occlusion（plan + frames_fg 存在时）、check-overflow（无 plan 的 custom 模式告警）、check-rail-climax（rail.html 存在时，支持 RAIL_CLIMAX_SKIP=1 覆盖），并把结果汇总进 _gates.txt；与 SKILL.md 的门禁描述对应。
6Verdict & Scenarios
Best for: Suitable for: Users with single-speaker footage (talking-head, interviews, explainers) seeking finished-grade captions without cutting or color grading the raw footage; creators needing to balance verbatim lower-third readable subtitles with peak words embedded behind the subject; and teams seeking ready-made visual identities (35 styles, ranging from understated anchor to VFX styles like ordnance, terminal, vhs, and arcade) without authoring custom CSS/GSAP. Explicit triggers and heuristics cover Chinese queries for “炸/特效/酷炫字幕”.
Not for: Not suitable for: ① Multi-speaker footage or footage with hard cuts (SKILL.md requires pre-splitting or rejection); ② Non-talking-head footage (lacking a human subject); ③ Clips under 3 seconds, lacking speech, or where faces are persistently invisible; ④ Raw footage with pre-existing burned-in subtitles or heavy text graphics (avoids double-stacking subtitle layers); ⑤ Degraded transcriptions (heavy accents or non-native speech producing gibberish) where switching to the medium model still fails—SKILL.md mandates rejection over delivering inaccurate captions; ⑥ Requirements involving footage modifications (color grading, video effects, cropping/re-framing)—this skill strictly leaves raw footage untouched; ⑦ Budget/low-overhead scenarios seeking plain readable text without peak effects—lightweight subtitle tools are better suited.
Install Agent-Ready
① Verified Mirror 2026-09-15
Option A · Direct DownloadDownload embedded-captions.tar.gz
sha256:
Option B · Copy Guide for Agent9cdab9186d817c12…Agents automatically download and verify mirrors from this JSON guide.
② Upstream GitHub
GitHub Repo ↗
Pinned to commit b8328f9573.