Run the direct Material Agent CLI for config-driven VLM material assignment pipelines. Use when the user wants to run material-agent commands directly, edit or execute a material-agent config YAML, resume a fixed pipeline, benchmark material predictions, build datasets from USD files, apply predicted materials, configure a material run, or try a SimReady demo.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Run the direct Material Agent CLI for config-driven VLM material assignment pipelines. Use when the user wants to run material-agent commands directly, edit or execute a material-agent config YAML, resume a fixed pipeline, benchmark material predictions, build datasets from USD files, apply predicted materials, configure a material run, or try a SimReady demo.
Requires the material-agent CLI, a repo Python environment, provider credentials for the selected VLM/LLM/image-generation backends, a render endpoint for remote rendering configs, and a materials manifest with USD material bindings.
Material Agent CLI
The Material Agent assigns materials to 3D object parts by rendering USD prims,
asking a VLM to choose from a constrained material library, and optionally
applying the predictions back to a USD layer.
When to Use
Use when the user asks to run material-agent directly from the command
line.
Use when the user wants a config-driven Material Agent run, to resume a failed
material pipeline, or to apply existing predictions to USD.
Use when the user wants to iteratively refine material assignments with a
predict/apply/render/judge loop.
Use when the user wants to benchmark material predictions or build VLM-ready
datasets from USD renders.
Use when the user wants a public SimReady demo asset with minimal local data
setup.
For an interactive Content Workbench asset workflow with a Codex or Claude
runner, start the agent from agentic/ and use its content-workflow-cli
skill.
Use service or Docker deploy skills instead when the user wants to operate
the REST service rather than the local CLI.
Limitations
Keep secrets out of chat and commits. Tell the user to set provider keys in
their local environment or repo-root .env; never ask them to paste keys.
The CLI needs a valid config YAML, a readable USD input, and a materials
manifest containing material names, descriptions, and USD bindings.
Config paths such as input.usd_path, input.reference_images, and
materials.path resolve relative to the config file, not the current shell
directory.
Step configs must not contain path keys such as usd_path, output_dir,
dataset, or predictions_path; the executor wires paths from the project
and input sections.
Remote rendering or optimization configs need deployed services. For a local
OVRTX Docker sidecar, use RENDER_ENDPOINT=http://localhost:8001 and keep
render concurrency conservative unless the endpoint fronts multiple service
instances.
Generated reference images are optional and need a configured image
generation backend plus its required key or endpoint.
Prerequisites
Locate the CLI and checkout with the bounded discovery sequence below. If a
checkout is used, activate its Python environment before running commands.
Set the VLM/LLM provider key required by the selected backend. Public
defaults usually use NVIDIA_API_KEY; other supported backends can use
OPENAI_API_KEY, ANTHROPIC_API_KEY, or GOOGLE_API_KEY.
Set RENDER_ENDPOINT and OPTIMIZER_ENDPOINT only when the config uses
remote rendering or optimization services.
Set WU_S3_BUCKET, WU_S3_PROFILE, WU_S3_REGION, and standard AWS
credentials only when the run uploads assets to S3.
Prepare a materials manifest YAML. The default library lives under
apps/material_agent/data/materials/material_libs_default/.
Locate the CLI and Checkout
Use only known candidates; do not crawl the filesystem or assume a host-specific
checkout path.
Run command -v material-agent (or Get-Command material-agent in a native
Windows convenience shell). If it succeeds, record the executable path.
Ask Git for the checkout containing the current directory with
git rev-parse --show-toplevel. Accept it only when
apps/material_agent/pyproject.toml exists below that root.
If the agent host exposes workspace roots, or the user supplied a checkout
path, validate only those finite candidates with
git -C <candidate> rev-parse --show-toplevel and the same repository marker.
If neither the CLI nor a valid checkout is found, stop and report that PATH,
the current Git root, and the available workspace/user candidates were
checked. Ask for the checkout path or installation instructions. Do not run
an unbounded search such as find /.
When a valid checkout is selected, it takes precedence over the executable
recorded from PATH. Change to the checkout root and activate that checkout's
.venv with the current shell's activation command (source .venv/bin/activate on POSIX shells, .\.venv\Scripts\Activate.ps1 in
PowerShell, or .venv\Scripts\activate.bat in cmd.exe). Stop and report
the failure if activation does not succeed. Then re-run command -v material-agent or Get-Command material-agent, normalize the result, and
require it to resolve below the selected checkout's .venv executable
directory. Stop and report a missing or mismatched executable; do not fall back
to the previously recorded installed CLI after selecting a checkout.
If no checkout is selected, retain the installed CLI recorded from PATH and
use user-provided config and asset paths instead of assuming the repository
examples are available.
Instructions
Locate the CLI and checkout as described above. When using a checkout, start
from its root, activate .venv, re-resolve material-agent, and verify that
its normalized path is below that checkout's .venv executable directory.
Stop and report any activation, missing-executable, or path-mismatch failure.
Choose a config. For a first local run, use
apps/material_agent/configs/unified_example.yaml. For a user asset, copy
that config or use material-agent configure.
Verify the config points at the input USD, optional reference images, and
materials manifest. Keep relative paths relative to the config file.
Run a dry run before a new or heavily edited config.
Run the full pipeline, or use --only, --skip, and --resume to control
execution.
Inspect the working directory and report the key artifacts from the output
format. When predictions are present, also surface every record where
materials.evidence_reconciliation.review_required is true; do not treat
a successful pipeline exit as approval of those material assignments.
source .venv/bin/activate
material-agent run apps/material_agent/configs/unified_example.yaml --dry-run
material-agent run apps/material_agent/configs/unified_example.yaml
Primary Command
material-agent run <config.yaml> [OPTIONS]
Option
Description
--skip <steps>
Comma-separated steps to skip.
--only <steps>
Comma-separated steps to run exclusively.
--session-id <id>
Reuse or override the session ID.
--resume
Continue from the last successful checkpoint.
--dry-run
Show the pipeline plan without executing.
--clean
Delete the working directory before starting.
--verbose, -v
Enable debug logging.
--log-file <path>
Write logs to a file.
--log-level <level>
Override the default INFO log level.
Other Commands
Command
Description
material-agent configure <output.yaml>
Interactive config creation wizard.
material-agent predict <config.yaml>
Run VLM prediction only.
material-agent apply <config.yaml>
Apply predictions to USD only.
material-agent refine <config.yaml>
Iteratively refine materials with predict/apply/render/judge.
build_dataset_pdf_vectorstore - index optional PDF specification evidence.
build_dataset_prepare_dataset - assemble rendered images and visual
prompts while retaining optional specification evidence separately.
cluster_prims - group visually similar prims before prediction.
predict - run VLM material assignment.
expand_cluster_predictions - expand cluster-level predictions back to
member prims.
benchmark - run prediction plus LLM-judge evaluation; mutually exclusive
with predict in one run.
validate_predictions - validate or repair predicted material names.
harmonize_predictions - resolve conflicts for instanced or repeated
parts.
restore_usd - remap predictions from optimized paths back to original
paths before application or refinement.
apply - apply predictions to USD.
evaluate - score existing predictions against ground truth with an LLM
judge.
refine - run the iterative predict/apply/render/judge loop; mutually
exclusive with apply in one run.
validate_output - compare the materialized output against the input
baseline.
render - render final output images when enabled.
See references/pipeline-steps.md for configuration details, outputs, and
step-specific caveats.
Common Workflows
# Run the configured pipeline.
material-agent run apps/material_agent/configs/unified_example.yaml
# Run only prediction, application, and final render.
material-agent run apps/material_agent/configs/unified_example.yaml --only predict,apply,render
# Skip optimization when the USD is already prepared.
material-agent run apps/material_agent/configs/unified_example.yaml --skip optimize_usd
# Preview what the pipeline will do.
material-agent run apps/material_agent/configs/unified_example.yaml --dry-run
# Resume after a failed step.
material-agent run apps/material_agent/configs/unified_example.yaml --resume
# Create a config with a materials manifest and reference image.
material-agent configure my_pipeline.yaml -m materials/my_materials.yaml -r reference.jpg
# Benchmark a configured dataset.
material-agent benchmark configs/benchmark.yaml -d dataset.jsonl -o results/
Generated Reference Images
Enable render_preview and generate_reference_image when the user has no
reference photos and wants a text-described target appearance. The generated
image is injected into the dataset for the prediction step.
steps:render_preview:enabled:truecameras: ["+x+y+z"]
generate_reference_image:enabled:trueprompt:"aluminum frame with a blue plastic tray"num_images:1
SimReady Demo
When the user asks to "try material agent" or run a public demo asset, follow
references/simready-quickstart.md. It covers downloading curated SimReady
assets, writing a config that uses the shipped default material library, and
running the pipeline end to end. For UR10 assets, keep
prim_filters.skip_instances: false so the agent sees meshes.
Config Authoring
Prefer copying apps/material_agent/configs/unified_example.yaml for new
configs. Adapt only the user-specific fields:
project.name and project.session_id
input.usd_path
input.reference_images when reference photos are available
materials.path
steps.predict.vlm.model
steps.render.enabled
Use references/config-template.yaml for a complete ready-to-adapt template
and references/config-reference.md for the full schema. Keep the prompts,
renderer settings, and prediction settings unless the user explicitly asks to
change them.
Local OpenAI-Compatible VLM Limits
For a local OpenAI-compatible endpoint, set steps.predict.vlm.backend to
openai, configure its model and base URL, and use max_tokens as the Material
Agent config field. The OpenAI backend normalizes that value to the request
field required by the selected model family; do not copy a provider-specific
max_completion_tokens field into a generic local-model config or set both
fields.
Before running, inspect the local server's model metadata, launch arguments, or
operator documentation for both its context-window and maximum-output limits.
Not every /v1/models response advertises those limits. Set max_tokens no
higher than the server's advertised output cap and leave enough context for the
prompt, material list, and images. Never assume a fixed cap from an example
config. Keep any endpoint credential in the local environment; never ask the
user to paste it.
Finding Materials
A materials file is a YAML manifest with a library_path pointing at the USD
material library and entries listing available materials:
library_path:"path/to/material_libs.usd"entries:-name:"Aluminum Polished"description:"A polished aluminum for structural parts"binding:"/World/metal_library/Looks/Aluminum_Polished"
Use the default library first, ask for a user-provided manifest second, and
create a new manifest only when the user has a material USD and wants that
library enumerated.
Output Format
Report these items after a run or handoff:
Command executed and whether it was full pipeline, --only, --skip,
--resume, or --dry-run.
Config path and session ID.
Working directory, usually .<session_id>/ next to the config unless
project.working_dir overrides it.
Key artifacts when present:
validation/input/ for pre-run USD validation reports.
dataset/usd/ for rendered prim images and manifests.
dataset/dataset.jsonl for VLM-ready records.
clusters/ for clustering reports and representative mappings.
generated_refs/ for generated reference images.
predictions/predictions.jsonl and report.html for VLM output.
Report the prim ID, visual material, and conflicting specification claims
for every prediction whose
materials.evidence_reconciliation.review_required value is true.
evaluation/ for LLM-judge scoring outputs.
iterations/ for iterative refinement artifacts.
restored/restored_predictions.jsonl when restore/remap ran.
output/output.usd or the configured output.usd_path.
validation/output/ for post-run USD validation reports.
renders/ for final render images.
Any missing credentials, service endpoints, invalid material bindings, or
config-path issues.
Troubleshooting
Symptom
Cause
Fix
material-agent or the checkout cannot be found
The CLI is not on PATH and the current/known workspace roots are not a valid checkout.
Follow the bounded discovery sequence, report the candidates checked, and ask for a checkout path or installation instructions; do not search the whole filesystem.
API key required
The selected VLM, LLM, or image-generation backend has no credential.
Set the required key locally or in .env; do not paste it into chat.
Local OpenAI-compatible server rejects the token limit
The copied config requests more output than the selected server/model allows, or uses a provider-specific token field.
Inspect the server's advertised model/output limits, use steps.predict.vlm.max_tokens, and lower it below the server cap while preserving room for prompt and image context.
Pipeline fails midway
A step failed after writing partial artifacts.
Re-run with --resume; use --clean only when the user wants to discard prior artifacts.
Forbidden path key in step config
Step configs contain path keys that the executor owns.
Remove usd_path, output_dir, dataset, or predictions_path from step configs.
Relative paths resolve unexpectedly
Config paths resolve from the config file directory.
Rewrite paths relative to the config file or make them absolute.
No meshes found for a SimReady UR10 asset
Instance filtering hid the geometry.
Set prim_filters.skip_instances: false.
Remote rendering fails or stalls
RENDER_ENDPOINT is missing, unhealthy, or over-concurrent.
Check endpoint health and keep local OVRTX worker/request concurrency at 1.
Material names do not match the library
Predictions chose names outside the manifest.
Keep validate_predictions enabled and verify the manifest entries are descriptive.
Prediction requires evidence review
Untrusted specification claims conflict with the fixed visual material label.
Report the prim ID, visual material, and conflicting_spec_materials; keep the visual label unchanged until a human resolves the conflict.
