run-longcat-avatar-video

name

run-longcat-avatar-video

description

Generate, benchmark, validate, and troubleshoot LongCat-Video-Avatar 1.5 videos with draw-things-cli. Use when Codex needs to prepare LongCat q8p or i8x checkpoints, drive an avatar from a reference image and audio file, generate a fixed 4k+1 frame clip, run local segmented AVC for long audio with the canonical 93/13 configuration, compare i8x against q8p on Apple silicon, estimate full-video runtime, or verify LongCat MP4 frame count, duration, codec, and audio muxing.

Run LongCat Avatar Video

Use the local draw-things-cli generate command for LongCat-Video-Avatar 1.5. Keep i8x and q8p comparisons identical except for the model checkpoint and output path.

Build And Inspect The CLI

Work from the Draw Things repository root. Build once, then reuse the same optimized binary for all runs in one comparison:

bazel build --compilation_mode=opt //Apps:DrawThingsCLI
CLI=bazel-bin/Apps/DrawThingsCLI
"$CLI" generate --help

Prefer the app model directory on macOS unless the user provides another one:

MODELS_DIR="${DRAWTHINGS_MODELS_DIR:-$HOME/Library/Containers/com.liuliu.draw-things/Data/Documents/Models}"

Do not switch builds, power modes, resolutions, segment sizes, or preview settings during a benchmark.

Prepare Models

Choose one of these DiT checkpoints:

• longcat_video_avatar_1.5_dmd_i8x.ckpt: 8-bit S model; prefer it for throughput on supported Apple silicon.
• longcat_video_avatar_1.5_dmd_q8p.ckpt: q8p baseline and fallback; use it for matched quality and performance comparisons.

Ensure the selected model and its registered dependencies:

"$CLI" models ensure \
  --models-dir "$MODELS_DIR" \
  --model longcat_video_avatar_1.5_dmd_i8x.ckpt

LongCat also needs these files in MODELS_DIR:

umt5_xxl_encoder_q8p.ckpt
wan_v2.1_video_vae_f16.ckpt
whisper_large_v3_f16.ckpt

The current model dependency list covers UMT5 and the Wan VAE. Verify Whisper separately because --audio-encoder-file defaults to whisper_large_v3_f16.ckpt, but it is not a registered LongCat model dependency:

test -f "$MODELS_DIR/whisper_large_v3_f16.ckpt"

Pass --audio-encoder-file NAME.ckpt only when using a different compatible Whisper checkpoint.

Audio API Boundary

Keep model audio separate from ControlNet hints. The reusable input and model-specific encoders live in Libraries/AudioConverter; LocalImageGenerator only consumes finished conditioning. LongCat audio follows this internal path:

AudioInput -> LongCatAudioConditioningEncoder -> LongCatAudioFeatures
           -> LongCatAudioConditioning -> AudioConditioning.longCat

AudioInput owns decoded PCM and builds the waveform used for output audio muxing. Run the Whisper encoder once to produce LongCatAudioFeatures; derive one LongCatAudioConditioning for a normal generation or one per AVC segment. AudioConditioning is the model-dispatch boundary where a future LTX audio-conditioning case can be added. Do not represent model audio as ControlHintType.audio or route it through ControlModel.

Validate Inputs

Use both a reference image and driving audio. The image is aspect-scaled and center-cropped to the requested output size. Width and height must be multiples of 64.

Confirm that AVFoundation can decode the audio before starting a long run:

afinfo "$AUDIO"

Reject an input that reports zero packets or zero duration even if its file size is nonzero. Use a valid CAF, MP3, M4A, or WAV container that afinfo and AVAudioFile can decode; do not fix a bad container by renaming its extension.

The DMD checkpoint is step-distilled. Use these baseline values unless the task explicitly changes them:

steps = 8
cfg = 1
shift = 7
fps = 25

LongCat temporal counts must be 4k + 1. Canonical values are 93 generated frames and 13 AVC condition frames. Values such as 77 and 109 are valid experiments, but do not mix them into a canonical 93/13 comparison.

Generate An AVC Video

Use AVC for audio longer than one generated clip. AVC is local-only and currently supports only LongCat-Video-Avatar 1.5. Its output duration follows the audio; do not pass --frames with --avc.

Use 93/13 explicitly even though they are the current defaults:

IMAGE=/path/to/reference.png
AUDIO=/path/to/driving-audio.caf
OUTPUT=/path/to/longcat_avc_i8x.mp4
PROMPT='A person speaks naturally to the camera with stable posture and synchronized mouth motion.'

"$CLI" generate --avc \
  --models-dir "$MODELS_DIR" \
  --model longcat_video_avatar_1.5_dmd_i8x.ckpt \
  --image "$IMAGE" \
  --audio "$AUDIO" \
  --prompt "$PROMPT" \
  --steps 8 --cfg 1 \
  --segment-frames 93 --cond-frames 13 \
  --width 448 --height 320 \
  --seed 42 \
  --config-json '{"shift":7}' \
  --no-download-missing \
  --disable-preview \
  --video-format h264 \
  --output "$OUTPUT"

With 93/13, each later segment contributes 80 new frames. For an audio target of T frames:

stride = 93 - 13 = 80
segments = T <= 93 ? 1 : ceil((T - 93) / 80) + 1
sampling steps = segments * 8

The CLI computes Whisper features for the generated span, reuses the last 13 decoded frames as the next segment's clean condition, drops overlap frames, trims to the audio target, and muxes the input speech into the output container.

For the full approximately 82-second reference workload, keep 93/13 and change only the audio, resolution, and output:

"$CLI" generate --avc \
  --models-dir "$MODELS_DIR" \
  --model longcat_video_avatar_1.5_dmd_i8x.ckpt \
  --image "$IMAGE" --audio /path/to/man.mp3 \
  --prompt "$PROMPT" \
  --steps 8 --cfg 1 \
  --segment-frames 93 --cond-frames 13 \
  --width 832 --height 512 \
  --seed 42 --config-json '{"shift":7}' \
  --no-download-missing --disable-preview \
  --video-format h264 \
  --output /path/to/man_832x512_i8x.mp4

Generate One 93-Frame Clip Without AVC

Omit --avc, --segment-frames, and --cond-frames. Set --frames 93 explicitly:

"$CLI" generate \
  --models-dir "$MODELS_DIR" \
  --model longcat_video_avatar_1.5_dmd_i8x.ckpt \
  --image "$IMAGE" \
  --audio "$AUDIO" \
  --prompt "$PROMPT" \
  --steps 8 --cfg 1 \
  --frames 93 \
  --width 448 --height 320 \
  --seed 42 \
  --config-json '{"shift":7}' \
  --no-download-missing \
  --disable-preview \
  --video-format h264 \
  --output /path/to/longcat_93f_i8x.mp4

At 25 fps, 93 frames produce 3.72 seconds. The exported audio is trimmed or padded to the same duration.

Compare i8x And q8p

Run comparisons serially on the same machine. Keep all of these identical:

• CLI binary and build mode
• image, audio, and prompt
• width, height, and frame or AVC settings
• steps, CFG, shift, and seed
• preview and video-format flags
• system power mode and competing workload

Change only:

--model longcat_video_avatar_1.5_dmd_i8x.ckpt
--model longcat_video_avatar_1.5_dmd_q8p.ckpt

Wrap each command with /usr/bin/time -p. Record both the CLI's generation/sampling summary and the external real time:

/usr/bin/time -p "$CLI" generate ...

Do not call model-loading, sampling-step, generation, and wall-clock speedups interchangeable. Report each metric by name.

Measured M5 Max Baseline

The following results were measured on 2026-07-13 on an Apple M5 Max with 48 GB memory. Both models used 448x320, 8 steps, CFG 1, shift 7, seed 42, H.264, disabled preview, and the same image, prompt, and valid 7.988-second audio. Use these as reference data, not a universal performance guarantee.

Workload	Model	Sampling steps	Generation	Median / step	Wall time
AVC 93/13, 200 output frames	i8x	24	368.45 s	10.69 s	421.77 s
AVC 93/13, 200 output frames	q8p	24	497.04 s	14.92 s	550.07 s
No AVC, 93 output frames	i8x	8	86.59 s	7.98 s	134.70 s
No AVC, 93 output frames	q8p	8	136.40 s	13.23 s	184.26 s

Observed speedups from that matched run:

• AVC wall time: i8x was 1.30x faster; median sampling step was 1.40x faster.
• Non-AVC wall time: i8x was 1.37x faster; median sampling step was 1.66x faster.

An earlier 832x512, approximately 82-second, AVC 93/13 workload was roughly 9 hours with q8p and 3 hours or more with i8x. Treat that as a historical high-resolution observation, not a canonical 3x claim, because it was not captured with the same benchmark ledger as the table above. Re-run both checkpoints with matched commands before publishing a 3x result.

Verify Outputs

Check existence, size, codecs, dimensions, and duration:

ls -lh "$OUTPUT"
mdls \
  -name kMDItemCodecs \
  -name kMDItemDurationSeconds \
  -name kMDItemPixelWidth \
  -name kMDItemPixelHeight \
  "$OUTPUT"

On macOS, count actual video samples with AVFoundation when ffprobe is unavailable:

VIDEO="$OUTPUT" xcrun swift -e '
import AVFoundation
import Foundation
let asset = AVURLAsset(url: URL(fileURLWithPath: ProcessInfo.processInfo.environment["VIDEO"]!))
let track = asset.tracks(withMediaType: .video).first!
let reader = try AVAssetReader(asset: asset)
let output = AVAssetReaderTrackOutput(track: track, outputSettings: nil)
reader.add(output)
reader.startReading()
var samples = 0
while let buffer = output.copyNextSampleBuffer() {
  samples += CMSampleBufferGetNumSamples(buffer)
}
print("video_samples=\(samples) status=\(reader.status.rawValue)")
'

Expect 93 samples for --frames 93. For AVC, expect the audio-derived target; a valid 7.988-second input at 25 fps produces 200 samples and an 8.00-second MP4.

Inspect visual continuity around each AVC boundary. With 93/13, the first output boundary is near frame 93, and later boundaries advance by 80 frames. Compare i8x and q8p boundary frames before making a quality claim.

Troubleshoot

• --frames must be 4k + 1: use 93 for the canonical single-shot run.
• --segment-frames or --cond-frames validation fails: use 93/13; both values must be 4k + 1, and segment frames must exceed condition frames.
• --frames cannot be used with --avc: remove --frames; AVC duration follows audio.
• --avc currently supports only local generation: remove --remote and --cloud-compute.
• Missing Whisper error: place the selected audio encoder in MODELS_DIR or pass --audio-encoder-file with its filename.
• Audio reports zero duration or zero packets: replace or properly transcode the source file before generation.
• High-resolution run exhausts memory: validate first at 448x320, close competing GPU workloads, then retry 832x512. Do not change segment size during an i8x/q8p comparison.
• Unexpectedly weak speedup: compare sampling metrics separately from Whisper, model loading, VAE, and video encoding. Fixed costs dominate short low-resolution clips.
• Segment boundary jump: confirm both runs use the same 93/13, seed, audio, and reference image; inspect frames around each 80-frame stride before changing the continuation policy.
• --zero-audio-features is a hidden pipeline diagnostic. Never use it for a quality result.

Keep the current stable LongCat behavior unless debugging model internals. In particular, do not change masked-reference attention or continuation policy merely to improve a benchmark number.