Content & Media Creation · iart-ai/javascript-animation-skills
soundtrack
This skill should be used when the user asks to "add music to the animation", "make the music in code", "synthesize a soundtrack", "compose background music that fits the cuts", "sync the music to the video", "check if the audio lines up with the picture", or "the music is too sudden / startling". Covers a code-synthesized score that lives inside the animation page (Web Audio API, rendered offline to WAV), AI-generated or user-supplied music, and an automatic check that cuts land on the beat without a startling jump in loudness.
L4 High-Privilege · Graded auth & full auditV1 Commit-PinnedExecute CommandNetwork AccessWrite FileDual-pass Audit
Control levels describe skill capabilities and suggested control measures; they do not represent quality or safety certifications, nor do they dictate usability. Usage decisions rest entirely with users and administrators. Different agent hosts apply varying constraints; do not assume hosts provide automatic protection. Third-party skill copyrights belong to their original authors. Use at your own risk.
1How It Works & Mechanism
The default route is "synthesize via code inside the page": paste templates into the animation page's <script>, where the soundtrack and visuals share the same BPM/SECTIONS/EVENTS so that sound for every action lands on the exact same frame—zero keys, zero audio files.
skills/soundtrack/SKILL.md
Paste `templates/score.js` into the animation page's `<script>`, before the live-preview block at the bottom (the starter marks the spot), and set:
What this is doing: turning the soundtrack into a native part of the page (Web Audio), rendered offline to WAV by javascript-animation's render.mjs and then muxed, requiring neither external assets nor API keys.
The mechanism for music tracking visuals is a tempo map rather than manual cue matching: each scene is allocated an integer number of bars (roughly 70–110 bpm), naturally aligning every cut onto a downbeat.
skills/soundtrack/SKILL.md
- **Tempo map.** Each scene gets a whole number of bars (tempo flexes per scene, roughly 70-110 bpm), so every cut lands on a downbeat.
What this is doing: score.js divides sections by cues, snaps each section to integer bars, resolves harmonic progressions to the tonic, and solves alignment issues through bar integerization.
Cuts are signaled by harmony rather than volume (scenes move toward the dominant chord and cuts land on the tonic), backed by measured values explaining why volume spikes should not be used: +7~10 dB jumps are perceived by users as startling, whereas +3~5 dB with harmonic changes reads as a clean cut.
skills/soundtrack/SKILL.md
- **Mark cuts with harmony, not volume.** Each scene walks home to the dominant (G) and the cut arrives on the tonic (C). A loud accent on a cut (bass + bright chime at once) reads as a jump scare: users flagged a +7 to +10 dB jump as startling; +3 to +5 dB with a chord change reads as a clean edit.
What this is doing: codifying mixing decisions into reproducible empirical values—the most valuable part of this skill, with each guideline mapped to specific code branches.
Automated verification turns "alignment" and "non-startling" into two fail-safe criteria: reporting event-by-event offset to the nearest strong onset and entrance loudness jump; flagging OFF-BEAT if offset exceeds ±100 ms and STARTLE if loudness jumps over +6 dB, skipping the initial one second and final fade-out.
skills/soundtrack/SKILL.md
It fails on OFF-BEAT (more than ±100 ms, ~3 frames) or STARTLE (more than +6 dB versus the second before). The opening second and the final fade are skipped (no "before" to compare against; a fade-out is not a cut).
What this is doing: sync-check.mjs runs custom analyses (64×64 grayscale frames to detect cuts and blank baselines, 10ms energy rise to detect onsets, ±1s neighborhood to evaluate "strong" onsets), specially handling "cutting into silence counts as landing" and "comparing post-rest levels against pre-rest levels".
When using non-synthesized music, the direction reverses: visuals follow the music—render, mix, run the check, and shift scene boundaries in the page's SCENES according to the reported offsets before re-rendering; since scene functions only take local time, shifting boundaries does not touch scene code, making this iteration loop very cheap.
skills/soundtrack/SKILL.md
With music you don't control, reverse the direction: **the picture follows the music.**
What this is doing: transforming the constraint of "unalterable music" into the engineering conclusion of "very low-cost visual alignment", providing an iterative loop.
Capability boundaries are explicitly declared: code synthesis excels at music boxes, piano-like plucks, pads, chimes, and light percussion, but cannot produce realistic vocals, orchestral arrangements, or polished genre productions—switch to AI music or real tracks when needed.
skills/soundtrack/SKILL.md
Synthesized in code: music box, piano-like plucks, pads, bells, soft percussion are fine. Realistic voices, orchestras and genre production are not; use AI music or a real track for those.
What this is doing: establishing boundaries early to avoid forcing synthesizers into unsuited genres, while explaining why an alternative AI music branch is provided.
The AI music branch requires a user-provided API key, where the agent curls an external generation service directly (Lyria on OpenRouter) with music descriptions—this is the only branch in this skill that reads credentials and sends materials externally, positioned as "only when asked".
skills/soundtrack/references/ai-music.md
Use when the user wants something richer than a synthesized score. Needs an API key.
What this is doing: ai-music.md provides a curl command with stream:true (Authorization: Bearer $OPENROUTER_API_KEY) and base64 audio concatenation scripts, followed by ffmpeg mixing commands; the default path does not involve it.
2Core Capabilities
01Code-synthesized score (default): two templates—a beat-grid groove (sections/drums/bass/pad/lead/optional breaks + around 20 synth SFX) and a music-box score (tempo map + optional soft chimes on cuts)
02Four instrument kits (electro/acoustic/keys/percussion) and multiple harmonic moods (bright/wistful/dreamy/tense/folk/blues or custom chords + key transposition), with customizable layering (drums can be omitted entirely)
03Offline rendering and multiplexing: rendered via OfflineAudioContext into WAV by javascript-animation's render.mjs and muxed into MP4 (deterministic, faster than real-time)
04Audio-visual sync check (sync-check.mjs): detects cut points from a CUES file or directly from visual frames, reports per-event offset and loudness jumps, exiting non-zero on OFF-BEAT/STARTLE
05Mixing and arrangement discipline: introduce at most 1–2 layers per measure, allow breaks (full silence) as the strongest downbeat, avoid extra accents on cut beats, and include in-scene hard cuts in CUES
06External music integration: AI-generated (Lyria via OpenRouter, receiving base64 audio in stream mode) or user-supplied tracks, using ffmpeg for trimming/fading followed by sync-check to back-solve visual alignment
07Explicit reporting for missing audio tracks/SCORE: non-zero exit with explicit warnings when a video has no audio stream instead of silently passing
3External Dependencies
| Kind | Dependency / Endpoint |
|---|---|
| cli | ffmpeg / ffprobe（分析视频与音频：抽灰度帧、抽 PCM；AI 支路里还用于混音） |
| api | OpenRouter 上的 Lyria（可选 AI 音乐生成，stream:true 收 base64 音频） |
| api | 该端点的凭证：从环境变量读取并放入 Authorization 头（仅 AI 音乐支路） |
| package | 兄弟 skill javascript-animation 的 render.mjs（离线渲染 SCORE 为 WAV 并 mux 进 MP4） |
| network | Web Audio API（浏览器内置，用于合成与离线渲染；非外部服务） |
4Recommended Governance & Evidence
Suggested Controls for Safe Operation
For Agent Runtime
Graded/per-use authorization, sandbox recommended
For Administrators
Approval workflow, rollback support, full audit
Evidence & Triggers (File Paths & Excerpts)
- scripts/: Execute local script or command: node scripts/sync-check.mjs piece.mp4 [--cues-file piece.cues.json] [--tol
- 网络端点: Declare external network access: https://openrouter.ai/api/v1/chat/completions — AI music branch only (when explicitly requested by the user), Authori
- 本地工作目录: File read/write and artifact output: Assets and intermediate artifacts: bgm.mp3 (curl streaming audio written to disk), piece_music.mp4 (ffmpeg muxed output)
Audit Notes: Evaluation target = the skill's own directory. The Orange tier is triggered by references/ai-music.md within its directory: 1) Reads credentials/environment variables—the command retrieves `$OPENROUTER_API_KEY` directly from the environment and injects it into the `Authorization: Bearer` header (analogous to home-assistant reading $HASS_TOKEN in the paradigm); 2) Sends user materials to an external generation/service backend—POSTs to https://openrouter.ai/api/v1/chat/completions (modalities:[audio], stream:true), sending music descriptions to a third-party model to generate audio, where returned base64 data is written directly to disk as bgm.mp3 and mixed into the final video, consuming account quota and storing generated artifacts. Trigger conditions are explicit: this branch is only used when the user requests richer music than synthesis can provide (SKILL.md lists it under 'Other sources (only when asked)'; the default path is in-page code synthesis with zero keys and zero endpoints). Apart from this branch, all behaviors are strictly local: templates are Web Audio synthesis code, and sync-check.mjs only invokes ffprobe/ffmpeg to inspect local files and print reports. There is no sandbox degradation, no TLS interference, no automation evasion, and no arbitrary code execution surface, ruling out the Red tier; the default path itself contains no network actions beyond Yellow tier.
5Second-Pass Verification
- [ok] $OPENROUTER_API_KEY 是否真的被读取并外发（橙档判定关键） — references/ai-music.md 原文给出 `curl -s -N https://openrouter.ai/api/v1/chat/completions -H "Authorization: Bearer $OPENROUTER_API_KEY" …`，且开头写明 'Needs an API key.'；端点与凭证同处一条可复制命令，调用点真实存在。
- [ok] 该支路是否真的是「可选、非默认」（避免过度判级） — SKILL.md 正文写 'Don't ask the user to choose a music source up front.'，并把 AI 音乐放在 '## Other sources (only when asked)' 段下；默认段标题为 '## Default: synthesize it in the page (no key, no files)'。触发条件为用户显式要求。
- [ok] sync-check 的判据是否与文档一致（能力声明可信度） — 代码内 TOL = Number(opt('--tol', 100)) / 1000 对应 ±100ms；`const jump = 20 * Math.log10(rms(v.t, v.t + .3) / ref), startle = jump > 6;` 对应 +6 dB；无音轨时 console.error 并 process.exit(1)；与 SKILL.md 逐条吻合，无夸大。
- [ok] 模板是否真的与画面共享数据（而非各自维护时间） — groove.js 头注释给出 schema（sections/events/kit/harmony/key）并写 'The picture reads the same BPM / SECTIONS / EVENTS, so sound and image can't drift apart.'；score.js 同样由 cues（即 window.CUES）切段并做小节取整。声明与实现一致。
- [ok] 是否有比橙更高的行为（红档特征） — 全仓扫描 `sandbox|automationcontrolled|no-verify|stealth|bypass|ignoreHTTPSErrors|dangerously` 零命中；curl 未使用 -k/--insecure，无绕防护动作，无任意代码执行面（模板是页面内 Web Audio 代码）。
- [ok] 确定性声明（每次渲染一致）是否可在模板代码内定位 — score.js 的混响脉冲用固定种子 LCG 生成并注释 'deterministic noise tail (seeded, so every render is identical)'；groove.js 同样用 `let s = 99 + c` 起手的 LCG 与固定种子噪声缓冲。声明属实。
6Verdict & Scenarios
Best for: Ideal for scoring code-generated short animations (default zero keys, zero assets, deterministically reproducible), acceptance verification of final cuts needing "on-beat cuts without startling jumps", and works with music box/pad/pluck/light percussion aesthetics; also suitable for routing "AI music or user-supplied tracks" through the same sync-check and visual alignment loop (requires self-provided OpenRouter key). Outputs in-page scoring code + WAV/MP4 rendered by render.mjs.
Not for: Not suitable for: 1) Soundtracks requiring realistic vocals, orchestral arrangements, or polished genre production (explicitly bounded in documentation); 2) Deployments disallowing any credential reading or outbound asset transfer (should strictly avoid the ai-music.md branch); 3) Environments lacking ffmpeg/ffprobe (checks and muxing depend on them); 4) Videos without window.CUES or SCORE (the script will fail with "no audio stream, nothing to check" rather than reaching a conclusion); 5) Workflows expecting scripts to fully replace human listening (thresholds are only baseline safeguards).
Install Agent-Ready
① Verified Mirror 2026-09-28
Option A · Direct DownloadDownload soundtrack.tar.gz
sha256:
Option B · Copy Guide for Agent7cd2fd4b7c5af0a2…Agents automatically download and verify mirrors from this JSON guide.
② Upstream GitHub
GitHub Repo ↗
Pinned to commit deb46e0eff.