| name | algo-rhythmic-narrative |
| description | Translate a music video 'shooting schedule' into frame-by-frame color-coded pixel grids for XMVP's cell-based painting pipeline. Given a song's BPM, section map, and scene descriptions, this skill generates per-frame JSON instruction files where each frame is a small pixel grid (e.g. 64x64) with per-cell color and annotation data, synced to musical beats and sections. Use whenever the user wants to plan a music video as pixel grids, create a frame-by-frame color map for animation, convert a storyboard or shooting schedule into cell-painting instructions, generate beat-synced frame data for XMVP's frame_canvas.py, or bridge narrative scene descriptions to per-pixel color grids. Also trigger for 'algo-rhythmic', 'rhythmic narrative', 'shooting schedule to frames', 'beat-synced pixel grids', 'music video frame map', 'cell painting instructions', or 'narrative-to-grid'. Works with cartoon_producer.py's music-video and music-agency modes. |
Algo-Rhythmic Narrative
Translate a music video shooting schedule into frame-by-frame, cell-by-cell color-coded pixel grids — one grid per frame of a long narrative sequence, synced to a song's BPM and sections.
Before Starting: Read References
Load these before generating any frame data:
references/pipeline-integration.md — How outputs connect to XMVP's frame_canvas.py, cartoon_producer.py, and unicode_visualizer.py. Data formats, file conventions, and the handoff protocol.references/grid-language.md — The "Grid Language" specification: how to encode narrative meaning, mood, motion, and emphasis into color values and cell annotations at the pixel level.
These files contain the essential data contracts. Do not proceed without reading them.
Core Concept
A music video is a temporal narrative synchronized to musical structure. This skill is the translation layer between:
- Input: A "shooting schedule" — scene descriptions, timings, moods, camera directions, character positions — organized by musical section (intro, verse, chorus, bridge, etc.)
- Output: A sequence of small pixel grids (JSON), one per frame, where every cell carries color data and optional annotation data that downstream painting/generation tools can interpret.
The grids are NOT the final visual output. They are instruction maps — structured blueprints that tell frame_canvas.py, an LLM image generator, or any cell-based renderer what to paint where and when. Think of them as the musical score to the orchestra's performance.
The Three Clocks
Every frame in an algo-rhythmic narrative is governed by three simultaneous temporal systems:
- Musical Clock — BPM, beats, bars, sections. The rhythmic skeleton.
- Narrative Clock — Scene progression, character arcs, emotional trajectory. The story.
- Visual Clock — Camera moves, color transitions, composition shifts. The eye.
The skill's job is to synchronize all three clocks into a single grid-per-frame output stream.
User Input
The user provides:
Required
- Song metadata: BPM, duration, and a section map (timestamps for intro/verse/chorus/bridge/outro or equivalent).
- Shooting schedule: A list of scenes with descriptions. Each scene specifies:
- Time range (or musical section reference)
- Visual description (what we see)
- Mood/energy level
- Optional: camera direction, character positions, color palette hints
Optional
- Grid dimensions: Default 64x64. Can be any power-of-2 square or rectangular grid.
- FPS: Default derived from BPM (4 frames per beat). Can be overridden.
- Audio file: If provided, run
analyze_audio() and analyze_audio_profile() for automatic BPM/section detection.
- XMVP manifest (
.xml): If provided, extract Bible/Story/Portions/Manifest data directly.
- Style vocabulary: Named palette or color-coding convention (see Grid Language reference).
Pipeline Overview
INPUT PHASE 1 PHASE 2
┌──────────────┐ Temporal Scene
│ Song Metadata │──┐ Scaffold Decomposition
│ BPM/sections │ │ ┌──────────────────┐ ┌──────────────────────┐
└──────────────┘ ├───▶│ Beat grid │───▶│ Zones per scene │
┌──────────────┐ │ │ Section boundaries│ │ Color palettes │
│ Shooting │──┤ │ Scene boundaries │ │ Motion vectors │
│ Schedule │ │ │ Frame count │ │ Evolution curves │
└──────────────┘ │ └──────────────────┘ └──────────┬───────────┘
┌──────────────┐ │ │
│ Audio File │──┘ ▼
│ (optional) │ PHASE 3: Grid Synthesis
└──────────────┘ ┌──────────────────────┐
│ For each frame: │
│ Musical position │
│ Active scene(s) │
│ Per-cell RGB │
│ Beat modulation │
│ Transition blend │
└──────────┬───────────┘
│
▼
PHASE 4: Export
┌──────────────────────┐
│ manifest.json │
│ frames/*.json │
│ preview.html │
│ XMVP .xml (optional) │
└──────────────────────┘
Phase 1: Temporal Scaffold
The temporal scaffold is the rhythmic skeleton that everything else hangs on.
Building the Scaffold
fps = (bpm / 60.0) * frames_per_beat
total_frames = int(duration * fps)
frames_per_beat_exact = fps * (60.0 / bpm)
beat_boundaries = [int(i * frames_per_beat_exact) for i in range(total_beats)]
beats_per_bar = time_signature[0]
bar_boundaries = beat_boundaries[::beats_per_bar]
Scaffold JSON Structure
{
"song": {
"title": "Track Name",
"bpm": 120,
"duration_sec": 210.5,
"time_signature": [4, 4],
"fps": 8.0,
"frames_per_beat": 4,
"total_frames": 1684,
"total_beats": 421,
"total_bars": 105
},
"sections": [
{
"name": "intro",
"start_sec": 0.0,
"end_sec": 16.0,
"start_frame": 0,
"end_frame": 128,
"start_beat": 0,
"end_beat": 32,
"energy": "low",
"mood": "mysterious"
}
],
"scenes": [
{
"id": 1,
"description": "Wide shot: empty highway at dawn, heat shimmer",
"section": "intro",
"start_frame": 0,
"end_frame": 64,
"palette": "dawn_warm",
"camera": "slow_push",
"zones": {
"sky": { "y_range": [0.0, 0.4], "base_color": "#2B1B4E" },
"horizon": { "y_range": [0.4, 0.5], "base_color": "#FF6B35" },
"road": { "y_range": [0.5, 1.0], "base_color": "#1A1A2E" }
}
}
]
}
Phase 2: Scene Decomposition
Each scene from the shooting schedule gets decomposed into spatial zones and temporal curves.
Spatial Zones
A zone is a rectangular region of the grid that carries consistent semantic meaning:
- Background zones: sky, ground, walls, environment — large areas, slow-changing
- Subject zones: characters, objects — smaller areas, more dynamic
- Accent zones: highlights, effects, overlays — sparse, beat-reactive
Zones are defined as fractional y-ranges (and optionally x-ranges) of the grid. They can overlap (later zones composite over earlier ones).
Temporal Curves
Each zone has evolution curves that define how its properties change frame-to-frame:
- Color curve: Base color to target color over the scene's duration (linear, ease-in/out, step)
- Brightness curve: Modulated by beat phase (pulse on downbeats)
- Position curve: For subject zones that move (character walks, camera pans)
- Opacity curve: For fade-in/fade-out, crossfades between scenes
Beat Modulation
The Grid Language (see reference) defines how beats affect the grid:
- Downbeat (beat 1 of bar): +15% brightness across all zones, optional color shift
- Backbeat (beats 2, 4): +8% brightness in accent zones
- Off-beat: Neutral
- Fill/syncopation: Brief complementary color flash in accent zones
Phase 3: Grid Synthesis
This is the core generation loop. For each frame:
- Determine musical position: Which beat, which bar, which section, beat phase (0.0-1.0 within the beat)
- Determine active scene: Which scene(s) are active, and if transitioning, the blend factor
- For each cell (row, col):
- Determine which zone(s) it belongs to
- Sample the zone's color at this frame's temporal position
- Apply beat modulation
- Apply transition blending if between scenes
- Write the final RGB value
Frame JSON Structure
Each frame outputs a JSON file:
{
"frame_index": 42,
"time_sec": 5.25,
"beat": 10,
"beat_phase": 0.5,
"bar": 2,
"bar_beat": 2,
"section": "intro",
"scene_id": 1,
"transition": null,
"grid": {
"width": 64,
"height": 64,
"cells": [
[{"r": 43, "g": 27, "b": 78}, {"r": 44, "g": 28, "b": 79}],
"..."
]
},
"annotations": {
"zones_active": ["sky", "horizon", "road"],
"beat_emphasis": 0.0,
"narrative_weight": 0.2,
"camera_offset": [0.0, -0.02]
}
}
Compact Grid Format (for large frame counts)
For videos with thousands of frames, use the compact format — flat hex strings instead of nested objects:
{
"f": 42,
"t": 5.25,
"b": 10,
"bp": 0.5,
"s": "intro",
"sc": 1,
"px": "2B1B4E2C1C4F2D1D50..."
}
Where px is a hex string of concatenated RGB values, row-major order. This reduces file size by ~80% vs. verbose format.
Phase 4: Export
Output Files
algo-rhythmic-output/
├── narrative_grid_manifest.json # Master index
├── temporal_scaffold.json # Beat/section map
├── scene_plans.json # Decomposed scenes
├── frames/
│ ├── frame_00000.json # Per-frame grids (compact)
│ ├── frame_00001.json
│ └── ...
├── palettes.json # Named color palettes used
└── preview.html # Animated grid viewer
Manifest Structure
{
"version": "1.0",
"generator": "algo-rhythmic-narrative",
"song": { "...from scaffold..." },
"grid_dimensions": [64, 64],
"total_frames": 1684,
"format": "compact",
"frame_pattern": "frames/frame_{:05d}.json",
"xmvp_compat": {
"bible_style": "cell-painting instruction grids",
"portions_count": 8,
"manifest_segments": 1684
}
}
XMVP Integration
To feed this into XMVP directly, generate an XMVP XML manifest where each segment references the corresponding frame grid:
<XMVP version="3.03">
<Bible>{"constraints": {"width": 64, "height": 64, "fps": 8, ...}, ...}</Bible>
<Story>{"title": "...", "synopsis": "...", ...}</Story>
<Manifest>{"segs": [
{"id": 0, "start_frame": 0, "end_frame": 1,
"prompt": "GRID_INSTRUCTION: Paint per frames/frame_00000.json RGB values.",
"grid_ref": "frames/frame_00000.json"},
...
]}</Manifest>
</XMVP>
The grid_ref field is a non-standard extension. XMVP's cartoon_producer.py can be patched to recognize it, or the prompt text itself can encode the instruction for LLM-based painters.
Implementation Strategy
For Short Sequences (< 500 frames)
Generate all frame JSONs directly in Claude. Write a Python script that:
- Parses the shooting schedule
- Builds the temporal scaffold
- Synthesizes each grid frame-by-frame
- Writes to
/mnt/user-data/outputs/
For Long Sequences (500+ frames)
Generate a standalone Python generator script that the user runs locally:
- Generate
temporal_scaffold.json and scene_plans.json in full
- Generate
grid_synthesizer.py — a standalone script that reads the scaffold + plans and outputs all frames
- The synthesizer script should:
- Accept
--start-frame and --end-frame for partial generation
- Support
--preview mode (renders every Nth frame)
- Output to a configurable directory
- Include a progress bar
- Require only numpy and standard library (no XMVP dependencies)
Preview HTML
Always generate a preview HTML artifact that:
- Loads the manifest and a subset of frames (or inline data for short sequences)
- Animates them at the correct FPS
- Shows beat markers and section labels
- Allows scrubbing through the timeline
- Displays the current frame's grid as colored cells on a canvas
- Can be opened standalone in a browser
Quality Checklist
Before outputting, verify:
Anti-Patterns
- Do not generate identical grids for consecutive frames (even subtle evolution should be visible)
- Do not ignore beat phase (grids should breathe with the music)
- Do not hard-cut between scenes without transition blending unless the schedule explicitly specifies jump cuts
- Do not produce monochrome grids with no spatial zoning (every frame should have compositional structure)
- Do not generate frame files larger than 100KB each (use compact format for grids > 32x32)
- Do not generate thousands of files directly in Claude — use the generator script pattern for long videos
