music-to-video
Turn a music track (an audio file, a video to pull audio from, or a track generated from a mood brief) into a beat-synced video — lyric video, slideshow, or kinetic promo. The music drives all pacing; any user-supplied images/videos are cut onto the same beat grid, and a complete video needs zero assets. Narrated pieces → the input-matched workflow (see /hyperframes). Unclear → /hyperframes.
1How It Works & Mechanism
An end-to-end 7-step gated workflow skill: Step 0 Initialize project (hyperframes.json + assets/bgm.mp3) → Step 1 Music analysis (audiomap.json) → Step 2 Frame skeleton (STORYBOARD.md) → Step 3 User-gated plan population → Step 4 Parallel build with one frame-worker sub-agent per frame → Step 5 Deterministic assembly into index.html → Step 6 User-gated check + render.
Workflow: Step 0 setup → `hyperframes.json` + `assets/bgm.mp3`; Step 1 analyze → `audiomap.json`; Step 2 skeleton → `STORYBOARD.md` (frames, groups `TBD`); Step 3 plan → complete `STORYBOARD.md` + `frame.md`; Step 4 build → `compositions/frames/NN-*.html`; Step 5 assemble → `index.html`; Step 6 render → `renders/video.mp4`.
Beat analysis is executed by a single authoritative script: analyze-beatgrid.py (Python: librosa beat tracker + 16th-note bar grid + drum kit band-split classification + riser/glitch/hard-stop events + energy narrative), producing a deterministic audiomap.json; the SKILL strictly forbids re-detecting beats using other tools or by ear.
`analyze-beatgrid.py` is the **only** beat analyzer — never re-measure beats with another tool or by ear. Its energy / density / rolls / onsets / silences are always reliable.
The script remains honest about unreliable fields: bpm/beats_sec are only trustworthy when music truly has rhythm—on calm music, the grid is merely a metronome imposed by the tracker; the pacing (beat_cut/phrase_flow) for each frame is decided by the orchestrator in Step 2.
Its `bpm` and `beats_sec` are reliable **only when the music is genuinely rhythmic**; on calm music the grid is a metronome the tracker imposed, so pace by phrases and energy instead and never hard-cut to it.
Workforce allocation model: The orchestrator (this skill) only plans and assembles, delegating each frame to a frame-worker sub-agent to build a single composition file based on fixed context (PROJECT_DIR, frame_id, Storyboard block, audiomap, frame.md, asset paths, and contracts); frame count tracks 'distinct treatments' rather than beat count (packing density within frames) to prevent fast-tempo music from exhausting sub-agent counts.
Extra density goes _inside_ a group, so **frame count tracks distinct treatments, not beats** — a fast track does not blow up the number of sub-agents.
Deterministic assembly is guaranteed by scripts: assemble-index.mjs runs without sub-agents or heuristic judgment, mounting each frame by the accumulated data-start from the STORYBOARD, BGM on track 11, with hard cuts between frames (no transition injector needed—frames already fill tracks seamlessly); validate-plan.mjs validates the plan in Step 3 (duration mismatches, uncovered frames, and missing src are treated as hard errors).
`assemble-index.mjs` is deterministic — no subagent, no judgment. It references each frame file at its cumulative `data-start`, mounts `assets/bgm.mp3` on track 11, and hard-cuts frame → frame (frames tile the track with no gaps, so there is **no transition injector**).
Clear asset/BGM provenance chain: User supplies audio/video → extract; none provided → route to /media-use for mood→track generation (provider auth belongs to media-use; the SKILL only relays output via `npx hyperframes auth status` and states 'Never write keys into a per-repo .env'); local offline providers act as fallbacks.
If signed out, apply one branch: - **Collaborative:** wait for sign-in or an explicit choice to continue offline with the local provider. - **Autonomous:** state the status and continue through the available local provider.
2Core Capabilities
3External Dependencies
| Kind | Dependency / Endpoint |
|---|---|
| package | Python3 + librosa/numpy/soundfile（analyze-beatgrid.py；缺失时 pip install） |
| cli | ffmpeg/ffprobe（analyze-beatgrid.py 解码） |
| cli | hyperframes CLI（init/update/auth status/check/render；render 带 --skill=music-to-video） |
| network | （间接）pip/npm 官方注册表首次拉包；BGM 生成走 /media-use provider（HeyGen/Lyria 等，auth 属 media-use） |
| network | 产物运行期 CDN（模板引用 jsdelivr gsap/three；作品级依赖） |
4Recommended Governance & Evidence
- scripts/: Executes local scripts or commands: python3 analyze-beatgrid.py assets/bgm.mp3 -o audiomap.json (local decoding and analysis; ffmpeg s
- scripts/ 或 SKILL.md: Declares dynamic installation of external language package dependencies
- 网络端点: Declares external network access: pip install (official PyPI packages, only when dependencies are missing), npx hyperframes initial/update package pulls (official npm/GitHub channels)
- 本地工作目录: File I/O and artifact output: Complete set of project artifacts under videos/<project>/: hyperframes.json, assets/bgm.mp3, audiomap.json, STOR
5Second-Pass Verification
- [ok] 『analyze-beatgrid.py 是唯一分析器』 — SKILL.md Step 1/Two ideas 两处声明 + 目录实测无其它分析脚本；audio 相关生成/转写均指向 media-use。
- [ok] 帧数与子代理数解耦声称 — SKILL.md 'frame count tracks distinct treatments, not beats' + frame-worker.md 'One frame = one file; groups live inside' 复核一致。
- [ok] worker 不跑 CLI 的纪律 — SKILL.md Step 4 'The worker never runs the `hyperframes` CLI…' + frame-worker.md 'Writing the file is your terminal action; the orchestrator runs lint/check' 双重复核。
- [discrepancy] auth status 是否触碰凭证 — 首轮注记『无凭证面』需细化：Step 0 确实运行 `npx hyperframes auth status` 并中继输出（登录/用量状态，账户级而非明文 token），且 BGM 生成委托 media-use 时其凭证面在 media-use。已在 credential_reads/reason 如实表述并维持 yellow。
- [ok] 渲染门禁与时长一致性判据 — Gate 文本 'the user approved … `renders/video.mp4` exists with audio, duration == audiomap.audio.duration_sec' 与 assemble 的确定性设计自洽。
6Verdict & Scenarios
7806505645ff0114…