clypra-dev

# Clypra Engineering Skill — Full System Reference > This skill covers the ENTIRE Clypra system. Read the section(s) relevant to your task. > For subsystem-specific docs, see `docs/preview/` for native preview deep dives. --- ## 1. What Clypra Is Clypra is a **native desktop video editor** (v1.6.0) — macOS/Windows. - Non-linear editing (NLE) with a track-based timeline - Multi-track video, audio, text, sticker, caption, and effects support - Native GPU rendering via wgpu/Metal/DX12 - AI-assisted features: auto-captions (Whisper), auto-reframe, background matting (ClyMatte) - Export via FFmpeg (MP4, MOV, WebM, GIF) with cloud export option - Device transfer (AirDrop-style local transfer) --- ## 2. Technology Stack | Layer | Technology | Location | |---|---|---| | **Desktop shell** | Tauri 2.x | `src-tauri/` | | **Frontend** | React 18 + TypeScript + Vite | `src/` | | **State management** | Zustand stores | `src/store/` | | **GPU renderer** | wgpu + Metal (macOS) / DX12 (Win) | `src-tauri/src/wgpu_compositor/` | | **Video decode** | FFmpeg (via ffmpeg-next) + VideoToolbox (macOS HW accel) | `src-tauri/src/engine/` | | **Audio engine** | CPAL (cross-platform audio) | `src-tauri/src/audio/`, `src-tauri/src/native_audio.rs` | | **AI / ML** | ONNX Runtime (ort), Whisper.cpp | `src-tauri/src/commands/whisper.rs`, `src-tauri/src/commands/ai.rs` | | **Thumbnail engine** | Custom Rust tile pyramid | `src-tauri/src/thumbnail_engine/` | | **Export** | FFmpeg H.264/H.265/WebM + cloud | `src/lib/export/`, `src-tauri/src/commands/export.rs` | | **IPC bridge** | Tauri `invoke()` + binary push channel | `src/lib/platform/tauri.ts` | | **Background matting** | ClyMatte (custom ML model) | `src-tauri/src/clymatte/` | --- ## 3. Repository Layout ``` clypra/ ├── src/ # TypeScript/React frontend │ ├── main.tsx # App entry point │ ├── components/ │ │ ├── editor/ │ │ │ ├── preview/ # Preview engine (NativeProgramPreview, etc.) │ │ │ ├── timeline/ # Timeline UI (tracks, clips, playhead) │ │ │ ├── properties/ # Inspector panel (transform, audio, effects) │ │ │ ├── sidebar/ # Asset browser (media, text, stickers, etc.) │ │ │ ├── toolbar/ # Top toolbar │ │ │ ├── transform/ # On-canvas transform overlay │ │ │ ├── scopes/ # Video scopes (waveform, vectorscope) │ │ │ └── viewport/ # Viewport controls (zoom, safe zones) │ │ ├── screens/ │ │ │ ├── EditorScreen.tsx # Main editor layout │ │ │ └── LaunchScreen.tsx # Project picker │ │ └── ui/ # Design system primitives │ ├── core/ # NLE engine (no React deps) │ │ ├── animation/ # Keyframe interpolation, motion presets, spring physics │ │ ├── audio/ # Native audio controller, AudioEngine (Web Audio API) │ │ ├── compositor/ # Semantic scene graph (JS-side) │ │ ├── evaluation/ # Timeline → EvaluatedScene (canonical rendering currency) │ │ ├── fonts/ # Deterministic font loading │ │ ├── history/ # Command pattern undo/redo │ │ ├── media/ # Audio detection, crop math, focal point │ │ ├── playback/ # PlaybackClock (single time source) │ │ ├── render/ # Render engine, raster surfaces, schedulers │ │ ├── runtime/ # ProjectSession, AppLifecycleCoordinator │ │ ├── storage/ # Project serialization/deserialization │ │ ├── telemetry/ # Client-side performance metrics │ │ ├── timeline/ # Audio clip builders, gap engine │ │ └── workers/ # Web Worker pool management │ ├── store/ # Zustand stores (see §5) │ ├── types/ # Shared TypeScript type definitions │ ├── lib/ # Pure utility libraries │ │ ├── export/ # Export pipeline (video, audio, OTIO) │ │ ├── filmstrip/ # Thumbnail filmstrip (tiles, layout, tiers) │ │ ├── platform/ # Tauri bridge, device capabilities │ │ ├── renderEngine/ # WebGL raster surface, render scheduler │ │ ├── timeline/ # Placement engine, gap manager, snap targets │ │ └── text/ # Text clip builders, text animations │ ├── features/ # Self-contained features │ │ ├── captions/ # AI captions (Whisper-based) │ │ ├── text-templates/ # Animated text template system │ │ ├── transitions/ # Transition renderer │ │ └── video-effects/ # Video effect picker + API │ ├── hooks/ # React hooks (usePlayback, useTimeline, etc.) │ ├── services/ # App-level services (telemetry, perf logging, updater) │ └── workers/ # Web Workers (compute, mediaAnalysis, rasterizers) ├── src-tauri/src/ # Rust backend │ ├── engine/ # Core render + decode engine │ │ ├── graph/ # Render graph (DAG executor, resource pool) │ │ ├── planner/ # Frame planning + lookahead queue │ │ ├── playback/ # Playback controller state machine │ │ ├── qos/ # Quality-of-service, adaptive decode │ │ ├── temporal/ # Temporal cache (decoded frame reuse) │ │ ├── timeline/ # Rust-side timeline model + evaluator │ │ ├── decoder.rs # VideoToolbox / FFmpeg decoder │ │ ├── presenter.rs # wgpu surface present │ │ └── telemetry.rs # Engine-side metrics │ ├── wgpu_compositor/ # GPU compositing pipeline │ │ ├── multi_track_composer.rs # Multi-track blend & composite │ │ ├── text_effect_pipeline.rs # GPU text effect rendering │ │ ├── transition_pipeline.rs # GPU transition rendering │ │ ├── chroma_key.rs # Chroma key shader │ │ ├── lut_texture.rs # LUT (color grading) support │ │ ├── frame_scheduler.rs # Frame deadline management │ │ └── yuv_ring_buffer.rs # YUV frame ring buffer │ ├── audio/ # CPAL audio graph │ ├── thumbnail_engine/ # Tile pyramid + mmap cache │ ├── clymatte/ # Background matting ML pipeline │ ├── commands/ # Tauri IPC command handlers │ ├── native_core/ # Native preview session lifecycle │ ├── transfer/ # Device-to-device transfer server │ └── sync_metrics.rs # Frame pacing + AV drift accumulator └── docs/preview/ # Deep-dive docs for the preview subsystem ``` --- ## 4. Core Data Model ### Project ```typescript interface Project { id: string; name: string; aspectRatio: AspectRatio; // "16:9" | "9:16" | "1:1" | "4:3" | "21:9" frameRate: number; // e.g. 30, 60 tracks: Track[]; assets: MediaAsset[]; creatorThumbnail?: CreatorThumbnail; } ``` ### Track ```typescript interface Track { id: string; type: TrackType; // "video" | "audio" | "text" | "sticker" | "filter" | "caption" | ... clips: Clip[]; } ``` ### Clip (base) ```typescript interface Clip { id: string; kind: ClipKind; // "video" | "audio" | "image" | "text" | "sticker" | ... startTime: number; // seconds on project timeline duration: number; // seconds sourceOffset: number; // seconds into the source asset playbackMapping: PlaybackMapping; // normal | reverse | freeze | speed-ramp effects: ClipEffect[]; keyframes: Record<VisualPropertyKey, VisualPropertyKeyframe[]>; // ...plus kind-specific fields on VideoClip, AudioClip, TextClip, etc. } ``` ### MediaAsset ```typescript interface MediaAsset { id: string; path: string; // native FS path — may be "" until hydrated streams: MediaStreamInfo[]; width?: number; height?: number; // ... } ``` **Critical**: `asset.path` may be empty (`""`) when a project is first loaded — the asset hasn't been probed/hydrated yet. Always guard for this before passing paths to Rust IPC. --- ## 5. State Management — Zustand Stores All stores live in `src/store/`. Each store has a clear ownership contract: | Store | Owns | Persisted | |---|---|---| | `timelineStore` | Track/clip composition, zoom, scroll, selection | Yes (via projectStore) | | `projectStore` | Active project metadata, save/load lifecycle | Yes (disk) | | `historyStore` | Undo/redo command stack | No (session only) | | `uiStore` | Panel sizes, active tabs, modal visibility | Partially | | `exportStore` | Current export job state + progress | No | | `exportHistoryStore` | Past export records | Yes | | `settingsStore` | User preferences (theme, quality, shortcuts) | Yes | | `mediaJobStore` | Background media analysis jobs (probe, waveform) | No | | `captionStore` | Caption segments and style | Yes (via projectStore) | | `cameraStore` | Camera recording state | No | | `recordingStore` | Screen/voiceover recording state | No | | `dragStateStore` | Active drag operation (timeline drag, media drop) | No | | `favoritesStore` | Starred effects/templates | Yes | | `presetStore` | Export presets | Yes | **Rules:** - Never put runtime/ephemeral state in a persisted store - Timeline mutations go through `timelineStore` actions — never mutate clip arrays directly - `historyStore` wraps mutations in commands (see `src/core/commands/`) - `projectStore` owns serialization — do not call Tauri file APIs from other stores --- ## 6. Rendering Architecture ### The canonical rendering pipeline ``` timelineStore (source of truth) ↓ evaluateTimelineSceneCached() ← core/evaluation/evaluator.ts ↓ EvaluatedScene (universal currency) ← typed scene graph for a given time T ↓ ┌─────────────────────────────────────────────────────────────────┐ │ Render targets (consume EvaluatedScene): │ │ • Preview (NativeProgramPreview) → wgpu Metal surface │ │ • Export (videoExport.ts) → FFmpeg frame sequence │ │ • Thumbnail (thumbnail_engine) → mmap tile cache │ │ • Filmstrip (filmstripTileCache) → timeline filmstrip tiles │ └─────────────────────────────────────────────────────────────────┘ ``` **Rule**: All rendering paths consume `EvaluatedScene`. Never build ad-hoc renderers that re-read `timelineStore` directly — always go through `evaluateTimelineSceneCached()`. ### Preview path selection ``` EMBEDDED_PREVIEW_ONLY (defaults to false unless VITE_CLYPRA_NATIVE_SURFACE=0)? ├── Yes (disabled) → web-canvas (IPC readback path) └── No (default) → native Metal/DirectX 12 surface path: nativeAudioClockReady = !isTauriRuntime() || clock.hasNativeClockPosition nativePlaybackPath = isTauriRuntime() && playing && nativeAudioClockReady nativeDirectSurface = nativeSurfaceUsable && nativeRequest && (playing || paused) ``` Full preview architecture: `docs/preview/NATIVE_SURFACE_ARCHITECTURE.md` ### WebGL render engine (web-canvas path) `src/lib/renderEngine/renderEngine.ts` — manages the WebGL context, texture cache, and compositing pipeline for the web-canvas preview and export fallback. Key types: `RasterSurface`, `RenderScheduler`, `ISM` (incremental scene model). --- ## 7. Timeline System ### Placement engine (`src/lib/timeline/placementEngine.ts`) Handles where clips land when dragged/dropped. Implements magnetic snapping, ripple-insert, and gap-aware placement. Depends on `snapTargets.ts` for snap point computation. ### Gap engine (`src/lib/timeline/gapEngine.ts`) Gaps are first-class timeline objects (`Gap` type). Operations: - `detectGaps()` — scan track for implicit gaps - `insertGapWithRipple()` — push clips forward - `removeGapWithRipple()` — pull clips back - `packTrack()` — collapse all gaps ### History / undo-redo (`src/core/history/`) Command pattern. Every timeline mutation that should be undoable is wrapped in a `Command` object with `execute()` / `undo()`. Commands are dispatched via `historyStore.dispatch(command)`. ### Keyframe system (`src/types/keyframes.ts`) Clips can have per-property keyframe tracks (`VisualPropertyKeyframe[]`). Evaluated by `src/core/animation/keyframeTrackOps.ts` at render time. Supported easing: linear, bezier, spring, step. --- ## 8. Audio Architecture ### Dual audio mode | Mode | When | Stack | |---|---|---| | **Web Audio API** | Browser / web-canvas preview | `AudioEngine.ts`, `AudioPlaybackAdapter.ts`, `AudioFXNodeChain.ts` | | **CPAL native audio** | Tauri runtime + native surface | `NativeAudioPreviewController`, `native_audio.rs`, `audio/mod.rs` | ### Web Audio path (`src/core/audio/AudioEngine.ts`) - Uses the Web Audio API `AudioContext` for decoding and mixing - `AudioFXNodeChain` — per-clip gain + pan + fade - `AudioBufferPool` — decoded buffer reuse - Clock: `AudioContext.currentTime` (high-resolution, drift-free) ### Native CPAL path (`src/core/audio/nativeAudioPreviewController.ts`) - Drives CPAL (Rust) via IPC: `replaceNativeAudioClips`, `seekNativeAudio`, `nativePlayFromAudio` - `syncNativeAudioTimeline()` — builds clip layout from timeline, installs into Rust graph - `getActiveAudioClips()` — filters clips to those with resolvable paths (guards empty `""` paths) - Clock: CPAL hardware clock → `setNativeClockPosition()` → `PlaybackClock.hasNativeClockPosition` - **Gate**: `nativeAudioClockReady = !isTauriRuntime() || clock.hasNativeClockPosition` ### PlaybackClock (`src/core/playback/PlaybackClock.ts`) Single source of truth for playback position. Singleton per app session. - `state`: `"idle" | "playing" | "paused"` - `time`: current playback position in seconds - `hasNativeClockPosition`: true when CPAL position received OR `markNativeAudioUnavailable()` called - `markNativeAudioUnavailable()` / `clearNativeAudioUnavailable()` — for projects with no audio clips - `nativeAudioWasUnavailable` getter — read-before-clear for transition detection - After any change to this file: **restart the dev server** (it's a module-level singleton, HMR won't update it) --- ## 9. Export Pipeline ### Flow ``` ExportDialog → exportPreflight() → verifyExportDependencies() → evaluateTimelineSceneCached() per frame → renderNativeFrame() (Tauri) OR WebGL fallback → FFmpeg encoder (via native_export Tauri command) → output file ``` ### Key files | File | Role | |---|---| | `src/lib/export/videoExport.ts` | High-level export orchestration | | `src/lib/export/exportSequence.ts` | Frame-by-frame render loop | | `src/lib/export/exportPreflight.ts` | Pre-flight dependency check | | `src/lib/export/exportPresets.ts` | Preset configs (H.264 1080p, 4K, etc.) | | `src/lib/export/frameBatching.ts` | Batched frame submission for throughput | | `src/lib/export/nativeTimelineExport.ts` | Native (Rust-side) export path | | `src/lib/export/cloudExport.ts` | Cloud export API integration | | `src/lib/export/otioExporter.ts` | OpenTimelineIO export | | `src-tauri/src/commands/export.rs` | Rust FFmpeg encoding commands | | `src-tauri/src/commands/native_export.rs` | Native frame rendering for export | ### Rules for export - Always call `exportPreflight()` before starting — it checks for missing text effects, unsupported codecs, disk space - Frame batching is critical for throughput on high-fps exports — use `frameBatching.ts` - Export must drain the `historyStore` undo stack before starting (prevents mid-export state mutation) - Progress is reported via `ExportProgress` callbacks — never block the UI thread --- ## 10. Thumbnail & Filmstrip System ### Thumbnail engine (Rust)