unreal-ad

name

unreal-ad

description

Unreal Assistant Director (Unreal AD) is a non-roster execution layer that helps Muse render simulation-based video using Unreal Engine. Muse remains the creative director and decides when Unreal is the right tool. UA Agent is the Unreal specialist — handles asset sourcing from user's downloaded libraries + external APIs, 3D asset prep, compatibility conversion, scene graph generation, and capability gating (manifest vs AMI rebuild). Unreal AD uses a C++ Commandlet (not Blueprint) for headless rendering — a compiled generic scene assembler that reads UnrealScenePlan JSON at runtime. The commandlet code stays fixed; only the scene data changes per job.

Unreal Assistant Director (Unreal AD)

Role Definition

Unreal AD is not on the agent roster. It has no DynamoDB agent-status entry. It does not receive mission briefings. It does not write lessons.

It is an execution layer — exactly how Claude Code is to Claude. Muse directs. UA Agent specialises. Unreal AD executes. Muse's scene plan JSON is the input. Rendered MP4 segments are the output.

Muse is responsible for creative decisions — including when to invoke Unreal. UA Agent is responsible for asset sourcing, compatibility, and capability gating. Unreal AD (the EC2 commandlet) is responsible for deterministic rendering.

The Three-Layer Model

Muse (director)
  Decides: use Unreal? → yes, if quality/complexity warrants simulation
  Generates: UnrealScenePlan JSON
  Aware of: what the commandlet can do generically (no need to know C++ details)

UA Agent (Unreal specialist — Lambda, runs before EC2 launches)
  Sourcing: user's downloaded asset libraries + external asset API calls
  Conversion: FBX/GLB/USD → Unreal-compatible .uasset
  ECS Fargate: heavy asset processing (meshes > 500MB, bulk conversion batches)
  Capability gate: does this scene need a new engine-level subsystem?
    → NO  → generate UnrealScenePlan JSON manifest, send to commandlet
    → YES → trigger AMI rebuild automatically, pipeline proceeds after

Unreal AD commandlet (EC2 g5.2xlarge — C++ Commandlet, not Blueprint)
  Receives: UnrealScenePlan JSON
  Executes: generic scene assembly (spawn, place, material, light, camera, render)
  Knows nothing about content — geometry is just data to it

UA Agent never makes content decisions. If an asset is missing and substitution judgment is required, it escalates to Muse — it does not guess.

---

Why Unreal Engine

Standard pipeline output (TONY 2D charts, Wan2.2 diffusion b-roll) has two hard limits:

1. Diffusion fingerprint — Wan2.2 output is detectable as AI-generated by YouTube's detection systems. Unreal renders are deterministic — no latent space sampling, no diffusion artifacts, no frame-to-frame noise signature.

2. Control ceiling — Diffusion models predict plausible frames. Unreal builds scenes from explicit geometry, physics, and lighting. A neural network explainer, a missile trajectory, a car crash reconstruction — these require controlled simulation, not prediction.

Unreal AD output is broadcast-quality simulation. The bar is educational clarity, not Hollywood VFX realism.

---

Visual Routing — Which System Gets Which Beat

Muse assigns every visual beat a renderEngine before production starts.

Beat Type	Assigned To	Reason
Charts, data tables, tickers	TONY Lambda	2D, data-driven, fast
Small UI overlays, lower thirds	TONY Lambda	CSS-rendered, zero EC2
Infographic cards, stat callouts	TONY Lambda	On-brand, Remotion
Atmospheric b-roll, mood footage	Wan2.2 EC2	Generated video, no assets needed
Talking head, presenter segments	SkyReels EC2	Avatar animation
3D simulation, physics explainers	Unreal AD	Deterministic, no diffusion
HUD overlays, comparison flicker	Unreal AD	Frame-perfect, data-driven
Studio segments (presenter in studio)	Unreal AD	Virtual broadcast studio
Neural network visualisation	Unreal AD	Particle systems + Blueprint
Historical reconstruction	Unreal AD	Asset library + simulation
Sports track + crash reconstruction	Unreal AD	Physics + camera control

Muse never assigns a beat to two engines. The routing is decided at MuseBlueprint generation time, before any EC2 launches.

---

Virtual Broadcast Studio

Two Modes

RRQ Default Studio — ships with the platform.

• Branded as "RRQ Studios"
• Dark, cinematic aesthetic — matches Mission Control design language
• Amber accent lighting, clean desk, monitor wall behind presenter
• No channel-specific branding
• Available immediately on account creation — zero configuration required

User Custom Studio — configured in Settings page.

• User uploads channel logo (SVG or PNG, transparent background preferred)
• User sets studio name (displayed as on-screen graphic)
• Unreal AD renders channel logo on monitor wall + studio signage
• Logo rendering guardrails:
  • Complex logos with faces or detailed illustrations → system renders channel name as text instead (with user notification)
  • Logos with more than 4 colours → simplified to 2-colour version
  • Aspect ratio must be between 1:1 and 4:1 — wider logos cropped to 4:1
• User can select studio layout from 3 presets: News Desk / Tech Lab / Open Stage
• After 10+ videos: Muse may recommend a studio variation based on content type (user approves — never auto-applied)

Studio Display Modes

Muse assigns studioMode per beat in the scene plan:

type StudioMode =
  | "studio-screen"           // presenter visible, content on studio screen/wall
  | "studio-fullscreen-expand" // studio fades out, simulation fills full frame
  | "simulation-only";        // no studio — pure simulation (faceless / autopilot)

studio-screen: Presenter at desk. Rendered simulation or video plays on the monitor wall behind them. Studio frame stays visible throughout. Used for: news-style commentary, product comparisons, segment intros.

studio-fullscreen-expand: Studio anchor shot → transition → full-frame simulation → transition back to studio. Muse specifies transition type: hard-cut | zoom-wipe | dissolve | flash. Duration of expand controlled by beat duration field in scene plan.

simulation-only: No studio layer at all. Pure rendered scene. Used in: Autopilot Mode (faceless), standalone explainer sequences, beats where studio presence would undermine immersion.

For faceless / Autopilot Mode channels: studio-screen beats automatically skip the presenter layer and render only the studio environment + screen content. Studio still provides broadcast framing without a presenter.

---

Scene Plan JSON — Contract Between Muse and Unreal AD

Muse outputs a UnrealScenePlan for every beat assigned to Unreal AD. This is the complete contract. Unreal AD receives this and nothing else.

interface UnrealScenePlan {
  beatId: string;                     // maps back to ScriptSection.id
  jobId: string;
  duration: number;                   // seconds
  studioMode: StudioMode;

  // Studio config (only when studioMode !== "simulation-only")
  studio?: {
    studioId: "rrq-default" | string; // string = user custom studio ID
    layout: "news-desk" | "tech-lab" | "open-stage";
    screenContent: "simulation" | "hud-overlay" | "chart" | "static-image";
  };

  // Transition into this beat (from previous beat)
  entryTransition: "hard-cut" | "zoom-wipe" | "dissolve" | "flash" | "none";
  // Transition out of this beat
  exitTransition: "hard-cut" | "zoom-wipe" | "dissolve" | "flash" | "none";

  // Camera direction
  camera: {
    type: "static" | "slow-push" | "orbit" | "tracking" | "overhead" | "first-person";
    focalTarget?: string;             // asset ID or semantic label ("missile", "building_01")
    depthOfField: boolean;
    motionBlur: boolean;
  };

  // Scene composition
  scene: {
    environment: string;             // semantic label: "desert-ruins", "space-station", "racing-circuit"
    timeOfDay: "dawn" | "day" | "dusk" | "night" | "interior";
    weatherFx: "none" | "dust" | "rain" | "snow" | "smoke" | "fog";
    assets: UnrealAssetRef[];        // see below
    physics: PhysicsDirective[];     // see below
    particles: ParticleDirective[];  // see below
  };

  // HUD overlays (rendered on top of scene, TONY-style data but Unreal-composited)
  hudOverlays: HUDOverlay[];

  // Audio sync points (Unreal AD times camera + physics to these)
  audioSyncPoints: {
    atSecond: number;
    cue: "impact" | "explosion" | "reveal" | "transition" | "emphasis";
  }[];

  // Quality/output spec
  output: {
    resolution: "720p" | "1080p";
    fps: 30 | 60;
    codec: "h264";                   // always h264 for Lambda stitching compatibility
    s3Key: string;                   // where to write the rendered MP4
  };
}

interface UnrealAssetRef {
  assetId: string | null;            // null = needs lookup or generation
  semanticQuery: string;             // "crumbling concrete wall" — used for catalog lookup
  placementRole: string;             // "background", "focal-point", "prop"
  position: { x: number; y: number; z: number };
  rotation: { pitch: number; yaw: number; roll: number };
  scale: { x: number; y: number; z: number };
  materialOverride?: string;         // "burnt", "weathered", "clean" — Blueprint material param
  animationBlueprint?: string;       // Blueprint ID from catalog: "BP_BuildingCollapse"
}

interface PhysicsDirective {
  targetAssetId: string;
  type: "collapse" | "projectile" | "fluid" | "cloth" | "rigid-body";
  triggerAtSecond: number;
  parameters: Record<string, number | string>; // Blueprint-exposed params
}

interface ParticleDirective {
  system: "fire" | "smoke" | "dust" | "sparks" | "debris" | "shockwave";
  attachTo?: string;                 // asset ID or world position
  position?: { x: number; y: number; z: number };
  scale: number;                     // 1.0 = default, 2.0 = double size
  triggerAtSecond: number;
  duration: number;
}

interface HUDOverlay {
  type: "stat-callout" | "comparison-bar" | "timeline-ticker" | "label-pointer" | "data-panel";
  content: Record<string, unknown>;  // type-specific data
  position: "top-left" | "top-right" | "bottom-left" | "bottom-right" | "center";
  appearsAtSecond: number;
  disappearsAtSecond: number;
  animationType: "fade" | "slide" | "flicker" | "typewriter";
}

---

UA Agent — Asset Sourcing

UA Agent runs in Lambda before any EC2 instance launches. It is the bridge between Muse's scene plan and the Unreal commandlet.

Asset Source Priority

UA Agent resolves assets in this order:

1. Existing catalog match (semantic search — fastest path, < 500ms)
2. User's downloaded asset libraries — user pushes downloaded Fab.com / Sketchfab / KitBash3D packs to s3://rrq-unreal-assets/user-library/. UA Agent indexes these at upload time. Always prefer owned assets.
3. External asset API calls — UA Agent calls Fab.com API, Sketchfab API, Smithsonian 3D API, NASA 3D API for matches. Free/CC-licensed only. Downloads, converts, catalogs — asset is owned forever after first use.
4. TripoSR generation — when Muse approves GENERATE for a scene gap. No external API — generated from text prompt.
5. Primitive fallback — sphere/cylinder/plane with material shader.

Asset Conversion Pipeline

User libraries contain assets in multiple formats (FBX, GLB, USD, OBJ). UA Agent handles all conversion:

User asset lands in s3://rrq-unreal-assets/user-library/{filename}
  → UA Agent detects new upload (S3 event trigger)
  → File < 500MB: Lambda conversion (FBX/GLB → .uasset via Unreal commandlet)
  → File >= 500MB or batch: ECS Fargate asset-converter task
  → Converted .uasset stored in s3://rrq-unreal-assets/{assetId}/
  → DynamoDB asset-catalog record written
  → Bedrock Titan v2 embedding generated (semantic search ready)
  → Oracle Domain 12 classifies contentDomains
  → Asset immediately available for future scenes

UA Agent also validates Unreal compatibility before storing — incompatible assets (wrong LOD, unsupported shader type) are flagged in the catalog with unrealCompatible: false and never used in renders.

CloudWatch Monitoring

UA Agent emits CloudWatch metrics throughout:

• Asset resolution time (catalog hit vs API fetch vs generation)
• Conversion success/failure rate per format
• ECS Fargate task duration + memory
• Catalog miss rate by contentDomain (feeds Oracle Domain 12 gap analysis)

---

Asset Catalog

Schema

Every asset in the library has a DynamoDB record in unreal-asset-catalog.

interface UnrealAssetRecord {
  assetId: string;                   // UUID — PK
  assetType: "static-mesh" | "blueprint" | "particle-system" | "material" | "animation" | "environment";
  name: string;                      // human-readable: "Desert Ruins Wall Section"
  s3Key: string;                     // s3://rrq-unreal-assets/{assetId}/
  fileFormat: "uasset" | "fbx" | "glb" | "usd";
  polyCount?: number;
  semanticTags: string[];            // ["desert", "ruins", "wall", "concrete", "damaged"]
  contentDomains: string[];          // ["war-explainer", "architecture-history", "disaster"]
  embeddingKey: string;              // S3 key to Titan v2 embedding vector
  source: "library" | "sketchfab" | "fab" | "nasa" | "smithsonian" | "tripo-sr" | "manual";
  sourceUrl?: string;
  license: "CC0" | "CC-BY" | "CC-BY-SA" | "royalty-free" | "generated";
  generatedFromPrompt?: string;      // if source = tripo-sr
  usageCount: number;                // how many videos have used this asset
  lastUsedAt: string;
  createdAt: string;
  unrealCompatible: boolean;         // validated import into Unreal project
  blueprintId?: string;              // if asset has associated animation Blueprint
}

Lookup Strategy — Fastest Path First

Unreal AD queries the catalog in this order for every UnrealAssetRef:

1. Exact assetId match — if Muse specified a known assetId, use it directly
2. Semantic vector search — Bedrock semantic search on embeddingKey using the semanticQuery string from the scene plan
3. Tag keyword match — DynamoDB GSI on semanticTags for keyword overlap
4. Oracle asset request queue — if no match found, see Scene Gap Handling below

Design goal: Muse should never wait for a catalog query. Lookup must complete in < 500ms. Bedrock semantic search is the primary path because it handles natural language like "crumbling concrete structure near water" without Muse needing to know asset names.

Catalog Indexing — Arranged for Muse

Assets are clustered into contentDomains at write time. When a new asset is added (from any source), Oracle Domain 12 runs a Haiku classification call to assign it to one or more content domains.

Domain examples:

• war-military
• natural-disaster
• sports-motorsport
• technology-hardware
• science-biology
• architecture-urban
• space-astronomy
• finance-markets
• history-ancient
• politics-governance

This means Muse can say "I'm building a technology explainer" and Unreal AD pre-filters the catalog to technology-hardware assets before running the semantic query — dramatically faster lookups.

---

Scene Gap Handling

When Unreal AD cannot find a suitable asset in the catalog:

Step 1 — Escalate to Muse

Unreal AD sends Muse a ASSET_GAP signal with:

• The original semanticQuery that failed
• The top 3 closest catalog matches (with similarity scores)
• TripoSR generation estimate (time + quality note)

Muse decides between:

• USE_CLOSEST — accept the closest catalog match and add a HUD label overlay to contextualise the substitution ("Symbolic representation")
• GENERATE — trigger TripoSR asset generation (pipeline pauses)
• SYMBOLIC — replace with a particle/primitive representation (sphere for planet, cylinder for building, etc.) — Muse explicitly approves this

Pipeline never makes this decision autonomously.

Step 2 — TripoSR Generation (if Muse selects GENERATE)

Muse approves GENERATE
  → Unreal AD sends generation job to TripoSR Lambda
  → TripoSR Lambda runs on g5.xlarge spot (same AMI, TripoSR loaded separately)
  → Generation time: ~2-4 minutes for a single mesh
  → Output: .glb mesh → validated → converted to .uasset
  → Stored in S3: s3://rrq-unreal-assets/{assetId}/
  → DynamoDB asset-catalog record written with source="tripo-sr"
  → Bedrock embedding generated and stored
  → Oracle Domain 12 classifies into contentDomains
  → Pipeline resumes with new asset

Asset is permanently in the library. The next video that needs a similar 3D object will find it via semantic search and never trigger generation again. This is the compounding effect: every gap filled makes the library richer.

Step 3 — Fallback (if TripoSR fails or Muse selects SYMBOLIC)

Unreal AD uses primitives + materials:

• Sphere with atmosphere shader → planets, orbs, cells, atoms
• Cylinder cluster → buildings, towers, silos
• Plane with displacement → terrain, ocean, desert
• Particle system only → explosions, energy fields, abstract concepts

These are always available — no external dependency. Quality is lower but never blocks production.

---

AWS Architecture

C++ Commandlet — Why Not Blueprint

Blueprint headless rendering requires booting the full Unreal editor, loading a level, and triggering Blueprint logic via command line flags or a wrapper. That is heavy — boot time matters a lot on a cold-started spot instance.

A C++ Commandlet (UCommandlet subclass) registers with a minimal subsystem set. It boots in seconds. It has better error handling and easier debugging. Unreal skips all editor subsystems it doesn't need.

The commandlet is a generic scene assembler:

• Import any FBX/USD mesh and place it at any position/rotation/scale
• Apply any material by path reference
• Set up any light (type, position, intensity, color, temperature)
• Move any actor along a keyframed path
• Configure any camera (FOV, aperture, position, motion blur, DoF)
• Render any resolution/quality pass to MP4

Muse can change absolutely everything — assets, lighting, layout, camera — just by generating a different JSON manifest. The commandlet doesn't care if the FBX is a neuron, a tree, or a racing car. Geometry is data.

New engine-level capabilities = AMI rebuild (automatic, no user prompt).

Engine-level means touching a different Unreal subsystem:

• MetaHuman + animations → AnimBlueprint subsystem
• Niagara particle systems → VFX subsystem
• Sequencer cinematic camera → Sequencer subsystem
• Procedural road generation → PCG subsystem
• Cloth simulations → Chaos physics subsystem

UA Agent is the capability gatekeeper. Before launching EC2:

• Can the current commandlet handle this scene? → generate JSON, proceed
• Does it need a new subsystem? → trigger AMI rebuild, then proceed automatically

EC2 Unreal Render Instance

Instance type: g5.2xlarge (NVIDIA A10G, 24GB VRAM)
Billing:       Spot — 2 attempts, then on-demand fallback (2-strike rule)
               Spot fail #1 → retry spot once
               Spot fail #2 → launch on-demand, pipeline continues uninterrupted
AMI size:      ~80-100GB (Unreal Engine 5 headless + C++ Commandlet + assets)
Self-terminate: yes — instance writes "RENDER_COMPLETE" to DynamoDB then halts

AMI contents:

• Ubuntu 22.04
• NVIDIA driver (CUDA 12)
• Unreal Engine 5 (headless mode — no display manager, no editor GUI)
• RRQ C++ Commandlet (compiled, registered as RRQSceneCommandlet)
• RRQ Unreal Project (materials, studio scene, base meshes pre-loaded)
• Asset manifest (DynamoDB delta sync on boot — downloads any new assets since AMI build)

Boot sequence:

1. Instance starts, reads job from DynamoDB unreal-render-jobs
2. Runs asset-sync.py — downloads delta assets from S3 (< 30s typical)
3. Runs Unreal headless commandlet:

   UnrealEditor-Cmd RRQProject.uproject -run=RRQSceneCommandlet \
     -ScenePlan=/tmp/scene_plan.json \
     -OutputPath=/tmp/render_output.mp4
   

4. Commandlet reads UnrealScenePlan JSON, assembles scene, renders
5. Uploads MP4 to S3
6. Writes RENDER_COMPLETE to DynamoDB
7. Calls aws ec2 terminate-instances on itself

Render time estimate: ~3-8 minutes per beat. Complex beats (rigid-body physics + particles + camera animation) approach 8 min. Static HUD-only beats take < 2 min.

ECS Fargate — Heavy Asset Processing

When UA Agent receives assets that exceed Lambda limits (meshes > 500MB, bulk conversion batches, complex FBX rigs), it dispatches to ECS Fargate instead of processing inline:

UA Agent detects heavy asset
  → Writes task to unreal-asset-requests with type="fargate-convert"
  → Triggers Fargate task: asset-converter container
  → Container: FBX/GLB → .uasset conversion + validation + embedding generation
  → CloudWatch monitoring throughout (task timeout, memory, CPU)
  → Output written to s3://rrq-unreal-assets/{assetId}/
  → DynamoDB asset-catalog record written
  → UA Agent resumes scene graph generation with new assetId

Fargate is also used for bulk library imports when the user pushes a new batch of downloaded assets to S3 — UA Agent doesn't block on these.

Spot → On-Demand Fallback (2-Strike Rule)

async function launchUnrealRenderInstance(jobId: string): Promise<string> {
  for (let attempt = 1; attempt <= 2; attempt++) {
    const instanceId = await launchSpotInstance(jobId);
    if (instanceId) return instanceId;
    // Spot failed — log to CloudWatch, try again
    await logCloudWatch(`Spot attempt ${attempt} failed for job ${jobId}`);
  }
  // Both spot attempts failed — fall back to on-demand
  await logCloudWatch(`Falling back to on-demand for job ${jobId}`);
  return await launchOnDemandInstance(jobId);
}

On-demand costs ~3× more per hour but guarantees capacity. The pipeline never tells the user which billing mode was used — it just renders.

AMI Rebuild — Automatic (UA Agent Triggered)

UA Agent detects a new subsystem is required for the requested scene. No user prompt. No wait. Rebuild fires and pipeline proceeds after.

UA Agent flags: scene requires [Niagara particle subsystem]
  → Checks DynamoDB system-config: is a newer AMI already being built?
    → YES → wait for AMI build completion, then proceed
    → NO  → triggers unrealAmiRebuildWorkflow immediately:
        1. Launches EC2 builder instance (on-demand, m5.4xlarge)
        2. Installs new capability module (Niagara plugin + headers)
        3. Recompiles RRQSceneCommandlet with new subsystem registered
        4. Runs validation render (standard test scene)
        5. Creates new AMI → updates UNREAL_AMI_ID env var
        6. Old AMI kept for 14 days (rollback)
  → After AMI ready → EC2 render instance launches with new AMI
  → Render proceeds normally

Build time: ~60-90 min (EC2 compile + Unreal cook). Jobs queued during this window wait in DynamoDB with status WAITING_FOR_AMI. Inngest polls and resumes them once the new AMI is ready.

Oracle Domain 12 also triggers AMI rebuild for new Unreal Engine versions — same pipeline, Zeus logs the version bump after completion.

TripoSR Lambda

Runs on g5.xlarge spot (lighter than full Unreal render instance). Fires only when Muse approves a GENERATE gap fill. Self-terminates after mesh output.

Input: text prompt (from semanticQuery in scene plan) Output: .glb mesh in S3

Asset Library S3 Bucket

s3://rrq-unreal-assets/
  {assetId}/
    mesh.(uasset|glb|fbx)
    thumbnail.jpg           — low-res preview for Oracle UI
    embedding.json          — Titan v2 vector
    metadata.json           — mirrors DynamoDB record
  user-library/             — user's downloaded asset packs (Fab, Sketchfab, KitBash3D, etc.)
    {filename}              — raw uploads — UA Agent indexes + converts on S3 event trigger
  commandlet/
    RRQSceneCommandlet.cpp  — C++ source (version controlled)
    build-manifest.json     — subsystem capabilities in current compiled version
  studio/
    rrq-default/            — RRQ Studios default scene files
    custom/{channelId}/     — user custom studio assets (logo, config)
  environments/
    {environmentId}/        — pre-built environment packs

DynamoDB Tables

unreal-render-jobs PK: jobId | SK: beatId Fields: status, scenePlanJson, startedAt, completedAt, outputS3Key, errorMessage, instanceId GSI: status-createdAt (Inngest polls for RENDER_COMPLETE)

unreal-asset-catalog PK: assetId Fields: (full UnrealAssetRecord schema above) GSI 1: source-createdAt — Oracle queries by source for catalog maintenance GSI 2: contentDomain-usageCount — Muse pre-filters by content domain

unreal-asset-requests PK: requestId Fields: semanticQuery, requestedBy (jobId), status (PENDING/GENERATING/COMPLETE/FAILED), generatedAssetId, createdAt GSI: status-createdAt — Oracle tracks pending generation requests

Add all three tables to CLAUDE.md DynamoDB table list.

---

Oracle Domain 12 — Unreal Engine Knowledge Base

Oracle Domain 12 runs on the standard Oracle Tuesday/Friday schedule.

What It Monitors

1. Unreal Engine release notes — new engine versions, deprecations, API changes
2. Fab.com free monthly drops — Unreal Marketplace free asset releases
3. Blueprint catalog freshness — checks if any RRQ Blueprints use deprecated APIs
4. Asset library gaps — reviews unreal-asset-requests table for recurring gap patterns (same semantic query failing multiple times = high-priority generation)
5. Render performance log — tracks which beat types are slowest, recommends Blueprint optimisations to Zeus
6. TripoSR model updates — checks for new TripoSR / Shap-E releases on HuggingFace

What It Updates

• Bedrock KB injection — new Unreal API docs + Commandlet patterns injected into the UA Agent knowledge context
• Commandlet capability catalog — new subsystem capabilities documented as UA Agent uses them (so future scenes can reference them correctly)
• Asset library auto-download — free Fab.com assets matching active content domains downloaded and catalogued automatically (within storage budget)
• AMI rebuild trigger — if Unreal Engine minor/patch version available, Oracle triggers unrealAmiRebuildWorkflow directly (same automatic path as UA Agent)

See AWS Architecture → AMI Rebuild Pipeline above for the full rebuild flow.

---

Inngest Workflow Integration

Unreal AD fits inside createVideoWorkflow as a parallel step alongside existing parallel media steps.

// In runParallelMediaStep() — production-steps.ts
// Unreal AD fires in parallel with existing workers

const unrealBeats = scriptOutput.sections.filter(
  s => s.renderEngine === "unreal-ad"
);

if (unrealBeats.length > 0) {
  // Each Unreal beat is a separate render job
  await Promise.allSettled(
    unrealBeats.map(beat =>
      invokeUnrealRender({
        jobId,
        beatId: beat.id,
        scenePlan: beat.unrealScenePlan,  // set by Muse during script step
      })
    )
  );
}

invokeUnrealRender is added to @rrq/lambda-client:

• Writes render job to DynamoDB unreal-render-jobs
• Launches EC2 g5.2xlarge spot with Unreal AMI
• Polls DynamoDB for RENDER_COMPLETE (Inngest step — 10 minute timeout)
• Returns S3 key of rendered MP4

---

Scene Gap Escalation Flow (Full)

Unreal AD cannot find asset
  → Checks top 3 closest catalog matches (similarity score)
  → If best match score > 0.85: auto-use + add HUD label (no escalation needed)
  → If best match score 0.60-0.85: sends ASSET_GAP signal to Muse
  → If best match score < 0.60: sends ASSET_GAP signal to Muse (strong gap)

Muse receives ASSET_GAP:
  → Evaluates: is this beat critical to the narrative?
  → CRITICAL: selects GENERATE → pipeline pauses → TripoSR fires → resumes
  → NON-CRITICAL: selects USE_CLOSEST or SYMBOLIC → render continues
  → If GENERATE selected:
      → TripoSR Lambda generates mesh (~3 min)
      → Asset validated + catalogued
      → Oracle Domain 12 classifies into contentDomains
      → Muse resumes scene plan with new assetId filled in
      → Render proceeds

Zeus receives gap report after render completes:
  → Writes to agent-decision-log: gap type, resolution chosen, outcome
  → If same gap occurs 3+ times: Zeus flags to Oracle for priority generation

---

Muse Integration — What Changes

Muse's MuseBlueprint gains a new section: unrealBeats.

For every visual beat Muse assigns renderEngine: "unreal-ad":

1. Muse generates the full UnrealScenePlan JSON
2. The scene plan is stored in ScriptSection.unrealScenePlan
3. buildSegments() in segment-builder.ts handles Unreal beats (output MP4 S3 key is known after render — same as other parallel workers)

Muse reads the asset catalog (via Oracle's Bedrock KB) before writing scene plans. This prevents Muse from requesting assets that don't exist.

Muse system prompt gains Oracle Domain 12 knowledge injection:

• Current asset library summary (contentDomains + asset count per domain)
• Blueprint catalog (what physics/animation Blueprints are available)
• Studio configurations available (RRQ default + any user custom studios)

This ensures Muse writes scene plans against assets that actually exist, minimising gap escalations.

---

Environment Variables to Add

# Unreal AD
UNREAL_AMI_ID=                        # AMI ID of current Unreal render AMI
UNREAL_INSTANCE_TYPE=g5.2xlarge
UNREAL_PROJECT_S3_KEY=s3://rrq-unreal-assets/project/rrq-unreal-project.zip
TRIPO_SR_LAMBDA_ARN=                  # TripoSR mesh generation Lambda
UNREAL_ASSET_BUCKET=rrq-unreal-assets
UNREAL_RENDER_TIMEOUT_MS=600000       # 10 minutes per beat

---

Environment Variables to Add (Additional)

# Unreal AD — EC2 + Fargate
UNREAL_EC2_BUILDER_INSTANCE_TYPE=m5.4xlarge  # AMI rebuild builder (on-demand)
UNREAL_ASSET_CONVERTER_CLUSTER=              # ECS Fargate cluster ARN
UNREAL_ASSET_CONVERTER_TASK_DEF=             # ECS task definition ARN
UNREAL_SPOT_MAX_ATTEMPTS=2                   # spot retry count before on-demand

---

Implementation Checklist

[ ] Read skills/muse/SKILL.md — understand MuseBlueprint schema before extending it [ ] Read skills/oracle/SKILL.md — understand Domain pattern before adding Domain 12

Types + Schema

[ ] Add unrealScenePlan?: UnrealScenePlan to ScriptSection in pipeline.ts [ ] Add renderEngine?: "tony" | "wan2" | "skyreels" | "unreal-ad" to ScriptSection [ ] Create lib/unreal/types.ts — all interfaces: UnrealScenePlan, UnrealAssetRef, PhysicsDirective, etc.

UA Agent Lambda

[ ] Create lambdas/ua-agent/handler.ts — UA Agent Lambda entry point [ ] Create lambdas/ua-agent/asset-sourcer.ts — resolveAsset() — catalog → user-library → external API → TripoSR [ ] Create lambdas/ua-agent/capability-gate.ts — canCommandletHandle() → true | { requiresSubsystem: string } [ ] Create lambdas/ua-agent/asset-converter.ts — convertAsset() — Lambda path (< 500MB) [ ] Create lambdas/ua-agent/fargate-dispatcher.ts — dispatchFargateConversion() — heavy asset path [ ] Create lambdas/ua-agent/external-apis.ts — fetchFabAsset(), fetchSketchfabAsset(), fetchNasaAsset() [ ] Add CloudWatch metric emission throughout UA Agent

Asset Catalog

[ ] Create lib/unreal/asset-catalog.ts — lookupAsset(), addAsset(), searchByDomain(), indexUserLibrary() [ ] Create lib/unreal/scene-gap-handler.ts — handleAssetGap(), escalateToMuse()

Render Job

[ ] Create lib/unreal/render-job.ts — invokeUnrealRender(), launchUnrealRenderInstance() [ ] Implement 2-strike spot → on-demand fallback in launchUnrealRenderInstance() [ ] Create lib/unreal/studio-config.ts — getRRQDefaultStudio(), getUserStudio(), validateChannelLogo()

Lambda Client + Types

[ ] Add invokeUnrealRender() to @rrq/lambda-client [ ] Add invokeUAAgent() to @rrq/lambda-client [ ] Add UnrealRenderOutputType to @rrq/lambda-types

Pipeline Integration

[ ] Add Unreal beat handling to runParallelMediaStep() in production-steps.ts [ ] Add Unreal beat handling to buildSegments() in segment-builder.ts [ ] Add studio config UI to Settings page (logo upload, layout selection, name override)

DynamoDB Tables

[ ] Create unreal-render-jobs DynamoDB table (PK: jobId, SK: beatId, GSI: status-createdAt) [ ] Create unreal-asset-catalog DynamoDB table (PK: assetId, 2× GSI) [ ] Create unreal-asset-requests DynamoDB table (PK: requestId, GSI: status-createdAt) [ ] Add all three tables to CLAUDE.md DynamoDB list [ ] Add ami-build-in-progress key to system-config DynamoDB table

AMI Rebuild Workflow

[ ] Create Inngest unrealAmiRebuildWorkflow — UA Agent trigger + Oracle trigger [ ] Implement WAITING_FOR_AMI job queue — Inngest polls system-config, resumes jobs on AMI ready [ ] Add unrealAmiRebuildWorkflow to Inngest client registration

Oracle Domain 12

[ ] Add Oracle Domain 12 section to skills/oracle/SKILL.md [ ] Implement Oracle Domain 12 runner — UE version check, Fab.com drops, catalog gaps, render perf log

EC2 AMI Build (ops task — not code)

[ ] Launch g5.2xlarge builder instance (on-demand) [ ] Install Unreal Engine 5 (headless — no display manager) [ ] Write + compile RRQSceneCommandlet (C++ UCommandlet subclass) - Registers with minimal subsystem set (skips editor subsystems) - Reads UnrealScenePlan JSON from -ScenePlan= argument - Generic handlers: SpawnMesh(), ApplyMaterial(), SetupLight(), MoveActor(), ConfigureCamera() - Renders to MP4 via Movie Render Queue CLI [ ] Build + validate RRQ Unreal Project (studio scene + base material library) [ ] Run validation render (standard test scene — must pass before AMI create) [ ] Create AMI → set UNREAL_AMI_ID env variable

ECS Fargate Asset Converter

[ ] Write Dockerfile: asset-converter (FBX/GLB/OBJ/USD → .uasset via Unreal commandlet) [ ] Push to ECR, create ECS task definition [ ] CloudWatch logging + alarms for failed tasks [ ] Test: large FBX (> 500MB) → successful .uasset in S3

TripoSR Lambda

[ ] Package TripoSR model weights via EFS mount (not Lambda layer — too large) [ ] Handler: text prompt → .glb mesh → S3 upload → catalog write [ ] Test: "crumbling concrete wall" → valid .glb asset

End-to-End Tests

[ ] Test: studio-screen beat → rendered MP4 with studio + presenter screen [ ] Test: studio-fullscreen-expand → dissolve transition → simulation → dissolve back [ ] Test: scene gap GENERATE path → TripoSR → asset added → render resumes [ ] Test: scene gap USE_CLOSEST path → HUD label overlay composited correctly [ ] Test: spot fails twice → on-demand launch → render completes [ ] Test: UA Agent detects new subsystem → AMI rebuild fires → job queued → resumes on AMI ready [ ] Test: user uploads FBX library → UA Agent indexes → asset appears in catalog [ ] Test: large FBX → ECS Fargate dispatch → conversion → catalog → render uses it [ ] Test: faceless mode → studio-screen beat → no presenter layer, studio + screen only

Verification

[ ] Verify: UA Agent never makes content decisions without Muse approval [ ] Verify: commandlet code does not change per job — only scene JSON changes [ ] Verify: on-demand fallback is never surfaced to user in UI [ ] Verify: AMI rebuild does not block other pipeline steps — only Unreal beats wait