create-sound

# Create Sound > Generated from `rules/*.md` by `src/build.mjs`. Do not edit by hand. Pick a generation path with `pipeline-detect-input`, then walk the matching section. ## 1. Generation Pipeline _Procedural steps the agent runs end-to-end. Start here when handling any create-sound request._ ### 1.1 Detect input mode and route the request _(CRITICAL)_ Decide which path to run based on what the user provided. | Input | Path | | ---------------------------------------- | ------------------------------------------------------------ | | Prompt only (no audio attachment) | Skip `interpret-*`. Go to `pipeline-pick-base-layer`. | | Audio file only | Run all `interpret-*` rules. Skip `event-*` / `mood-*`. | | Both prompt and audio | Run `interpret-*` first, then treat the prompt as a refinement layer over the measured `SoundDefinition`. | #### Detecting audio Look for attached files matching `*.wav`, `*.mp3`, `*.flac`, `*.ogg`, or any path the user references that resolves to an audio file. A JSON manifest (`*.json` next to a sprite) is also an audio-path signal. #### Refinement examples (prompt + audio) | Prompt qualifier | Refinement on measured definition | | -------------------------- | -------------------------------------------------------------- | | "warmer" | add `filter: { type: "lowpass", frequency: 2500 }` | | "shorter" / "punchier" | clamp `envelope.decay` to `<= 0.06` | | "brighter" | drop or raise any lowpass cutoff | | "with reverb" | append `effects: [{ type: "reverb", decay: 0.5, mix: 0.15 }]` | | "lower octave" | halve `source.frequency` (or both `start`/`end`) | #### Output of this step Produce an internal note like: ``` Input: prompt + audio Plan: run interpret-* on out/click.wav, then refine with mood-warm. ``` Then proceed to the next pipeline step. ### 1.2 Pick a base layer from the prompt's event class _(CRITICAL)_ Tokenize the prompt and find the strongest event-class signal. Match against the `event-*` rules. #### Token map | Tokens in prompt | Event rule | | ------------------------------------------------------------- | ----------------------- | | click, tap, key, press, button | `event-click` / `event-tap` | | tick, scroll, snap, focus | `event-tick` | | success, complete, win, achievement, level-up, confetti | `event-success` / `event-complete` | | error, fail, wrong, invalid, delete, destroy | `event-error` | | modal, dialog, popup, drawer, sheet, sidebar, dropdown, menu | `event-modal-open` / `event-modal-close` | | swoosh, slide, transition, page, tab | `event-swoosh` / `event-whoosh` | | notification, alert, ding, bell, mention, badge | `event-notification` | | toggle, switch, on, off | `event-toggle` | #### Direction tokens (open vs close) - "open", "appear", "in", "show", "expand", "confirm" -> ascending pitch. - "close", "dismiss", "out", "hide", "collapse", "cancel" -> descending pitch. #### Output A starting `SoundDefinition` literal copied from the chosen event rule's `example`. The next step (`pipeline-apply-mood`) will mutate it. If no event class fires confidently, default to `event-click` and let mood adjectives do the work. ### 1.3 Apply mood adjectives onto the base layer _(HIGH)_ After `pipeline-pick-base-layer` produces a starting `SoundDefinition`, scan the prompt for adjective tokens and apply each `mood-*` rule's mutation in order. #### Order of application 1. Source-shape adjectives (`warm`, `bright`, `glassy`, `metallic`, `lofi`, `retro`, `organic`) - mutate `source.type`, `source.fm`, or add `filter`. 2. Envelope adjectives (`punchy`, `airy`) - mutate `envelope.attack` / `envelope.decay`. 3. Effect adjectives (`reverby`, `delayed`, `crushed`) - append to `effects`. #### Conflict resolution - `warm` + `bright` -> the later token wins. - `lofi` + `glassy` -> apply both, but cap `effects` at 2 entries. - `punchy` + `airy` -> they're orthogonal (envelope vs source); both apply. #### Refinement on existing definition (audio + prompt path) When the input mode is `prompt + audio`, treat each adjective as a refinement on the measured definition rather than from scratch: | Adjective | Refinement | | --------- | ----------------------------------------------------------------------------- | | warmer | add or lower `filter.frequency` (lowpass at ~2500 Hz) | | brighter | remove lowpass or raise its cutoff above 6 kHz | | punchier | clamp `envelope.decay <= 0.06`, set `envelope.attack: 0` | | longer | extend `envelope.decay` and add `release` if missing | | crisper | raise `gain` slightly and add `fm: { ratio: 0.5, depth: 50 }` | #### Output A mutated `SoundDefinition`. Hand off to `pipeline-decide-layering`. ### 1.4 Decide single-layer vs multi-layer _(MEDIUM-HIGH)_ | Event class | Default | | ------------------------------------------ | ---------------------------------------- | | click, tap, tick, hover, focus, swoosh | 1 layer (`Layer`) | | toggle, copy, send, sync | 2 layers (paired pitches with `delay`) | | success, complete, level-up, confetti | 3+ layers (chord with cascading `delay`) | | error, delete | 2 layers (`sawtooth` + `square`) | See `layer-single`, `layer-octave-pair`, `layer-ascending-chord`, `layer-click-plus-body` for the concrete shapes. #### Promoting a single Layer to MultiLayerSound If the prompt or refinement requires more than one layer, wrap: ```ts { layers: [<existing layer>, <new layer>], // optional global effects, e.g. sidechain compressor, master EQ } ``` Per-layer `gain` values should sum to no more than ~0.6 (see `validate-gain-budget`). #### Demoting MultiLayerSound to a single Layer If only one layer survives mood application, emit the inner `Layer` directly rather than a one-element `MultiLayerSound`. Both validate, but the single-layer form is the canonical compact shape. ### 1.5 Emit, optionally render, optionally round-trip _(HIGH)_ #### 1. Emit Always return a TypeScript snippet ready to paste into a `.web-kits/<patch>.ts` file: ```ts import type { SoundDefinition } from "@web-kits/audio"; export const myClick: SoundDefinition = { source: { type: "sine", frequency: 1300, fm: { ratio: 0.5, depth: 60 } }, envelope: { decay: 0.012, release: 0.004 }, gain: 0.18, }; ``` Plus a one-line rationale that names the prompt tokens you acted on: > "click" -> base from `event-click`; "warm" -> kept default sine, no extra filter needed at 1.3 kHz. #### 2. Optional preview render If the user asked for a WAV (or you want to grade your own output), use [`packages/audio/src/offline.ts`](../../../packages/audio/src/offline.ts): ```ts import { renderToWav } from "@web-kits/audio"; import { writeFile } from "node:fs/promises"; const blob = await renderToWav(myClick, { duration: 0.3 }); await writeFile("preview.wav", Buffer.from(await blob.arrayBuffer())); ``` `duration` should be `attack + decay + release + 0.05` (small tail) or longer if reverb is present. #### 3. Optional round-trip validation If you generated from a prompt and want to confirm the result matches intent, run the `interpret-*` rules against the rendered WAV and diff measured vs intended values: | Field | Acceptable drift | | ----------------- | ------------------------------------------- | | Fundamental Hz | ±5% | | Attack | ±2 ms | | Decay | ±10% | | Spectral centroid | ±20% of expected for the chosen waveform | If drift exceeds tolerance, refine the definition (often by raising/lowering `gain`, tightening `envelope`, or adjusting `filter.frequency`) and render again. ## 2. Audio Interpretation _FFT analysis sub-steps that fire when the user shares an audio file._ ### 2.1 Acquire and split source audio _(HIGH)_ The user shared a single file or a sprite (one file containing many sounds). Before any FFT work, get one mono WAV per sound on disk. #### Sprite from an npm package ```bash npm pack <package-name> --pack-destination /tmp tar -xzf /tmp/<package-name>-*.tgz -C /tmp ``` Look for the MP3/WAV plus any JSON manifest mapping sound names to time offsets. #### Manifest-driven slicing ```bash ffmpeg -i sprite.mp3 \ -ss <start_seconds> -t <duration_seconds> \ -acodec pcm_s16le -ar 44100 \ output/<name>.wav ``` #### Silence-detection slicing (no manifest) ```bash ffmpeg -i sprite.mp3 -af silencedetect=noise=-40dB:d=0.05 -f null - ``` Read the `silence_start`/`silence_end` lines and slice between gaps. #### Output convention Per-sound WAVs go in `out/<name>.wav` (mono, 44.1 kHz, 16-bit PCM). Downstream interpret rules call `analyze.load_mono(path)` from [src/analyze.py](../src/analyze.py). ### 2.2 Extract fundamental frequency and pitch sweep _(HIGH)_ Sample the spectrum at multiple time slices to detect both the static pitch and any sweep. ```python from analyze import load_mono, analyze_slice sample_rate, data = load_mono("out/click.wav") slices = [0, 5, 10, 20, 50] # ms freqs_over_time = [analyze_slice(data, sample_rate, t) for t in slices] ``` #### Mapping | Observation | Output |