lc-cli

name	lc-cli
description	Reference for `lc` CLI execution: commands (init/run/status/verify/build/export), the Spec-Code Invariant (`astra.yaml` and code never diverge), status interpretation (ok/stale/missing/alias), failure diagnosis, multiverse runs, scratch overrides for HPC, sub-analysis scaffolding, publishing via WRROC. Invoke whenever running, debugging, or diagnosing `lc` workflows; whenever interpreting `lc status` / `lc verify` output; or whenever the user asks about the development workflow surrounding `astra.yaml`.
allowed-tools	Read, Glob, Grep, Bash(lc:*), Bash(astra:*)

lightcone-cli Reference

Reference for lightcone-cli execution: CLI commands, development workflow, status interpretation, and failure diagnosis. For astra.yaml spec syntax, invoke /astra.

CLI Reference

lc init [DIR] [--permissions yolo|recommended|minimal] [--scratch PATH]  # Scaffold a new ASTRA project
lc run [OUTPUTS...] [--universe NAME] [--force] [--verbose] [--rerun-triggers TRIGGERS]  # Materialize outputs
lc build [--force] [--runtime docker]                             # Build container images from specs
lc status [--universe NAME] [--json]                              # Materialization status (text or JSON)
lc verify [--universe NAME]                                       # Recompute hashes and walk the provenance chain
lc export wrroc [--output PATH] [--universe NAME] [--zip] [--metadata-only] [--author "NAME <EMAIL>"]  # Export Workflow Run RO-Crate bundle

lc run is quiet by default — pass --verbose to see worker output. --scratch is only relevant on HPC sites where $HOME doesn't honor flock (NERSC etc.); it redirects Snakemake state and Dask spill onto the named filesystem.

The first lc invocation auto-creates ~/.lightcone/config.yaml:

container:
  runtime: auto    # or: docker | podman | podman-hpc | none

Always run via lc. Recipes must execute through lc run so that container builds, option resolution, resource limits, and result paths are applied. Treat the underlying execution engine as a black box — never invoke schedulers or container runtimes directly, that will bypass reproducibility guarantees.

Creating Sub-Analyses

Sub-analyses are scaffolded by hand, since each one is just another astra.yaml nested in a directory. To add one:

1. Create analyses/<name>/ with its own astra.yaml (and optionally src/, universes/baseline.yaml, results/).
2. Add a path: entry to the parent astra.yaml under analyses: (e.g. analyses: { my_sub: { path: ./analyses/my_sub } }).
3. Add a <name>: { universe: baseline } entry to each existing parent universe file.

Populate the sub-analysis's astra.yaml with inputs, outputs, and decisions. Use from: references to wire inputs and decisions to the parent or siblings — invoke /astra and see "Composition Mechanics" for the grammar.

Development Workflow

Three overlapping phases:

1. Write & Debug — Run scripts directly (python src/compute.py) to iterate. Write them recipe-ready from the start: parameterize decisions, write to convention paths, one script per output.
2. Integrate — Add recipe: blocks to outputs in astra.yaml. Track with lc status (alias / missing / stale / ok). Set container: at analysis level or per-recipe — pass an image name (e.g., python:3.12-slim) or a path to a Containerfile (e.g., Containerfile).
3. Materialize — lc run executes recipes inside their declared containers and writes a content-addressed manifest next to each output. Done when lc status shows all ok.

Bare lc run materializes every output across every universe in universes/*.yaml; pass OUTPUT_ID... to scope to specific outputs and --universe NAME to scope to one universe. Build iteratively — name one upstream output at a time (lc run <output_id> --universe <name>) so you can inspect each intermediate before chaining further downstream, rather than running the whole DAG and debugging from the bottom of a long failure trace. lc run auto-builds container images on demand, so lc build is only needed for pre-warming or forcing a rebuild with --force.

Outputs land at results/<universe>/<output_id>/, with the per-output manifest at <output_dir>/.lightcone-manifest.json. Path-rooted sub-analyses prefix the sub's path: <sub_path>/results/<universe>/<output_id>/.

An output is not done until lc run produces it. Running scripts directly is for debugging only — final results must always come from lc run so they are reproducible.

Spec-Code Invariant

astra.yaml must always reflect the code and vice versa. When you change one, update the other immediately:

• Add a decision to code? Add it to astra.yaml and all universe files.
• Add an output or change a script? Update the recipe: block in astra.yaml.
• Remove or rename something? Update both sides and run astra validate astra.yaml.

Status Interpretation

lc status shows each declared output's materialization state per universe. Pass --json for machine-readable output.

• ok — Recipe exists, results on disk, manifest matches the current spec. Done.
• stale — Recipe or decisions changed since the last run. Re-run lc run.
• missing — Recipe exists but no manifest (never run, or output deleted). Run lc run.
• alias — Output has no recipe of its own; produced as a side effect of an upstream output (or a from: reference into a sub-analysis). Not independently materializable.

Failure Diagnosis

• Script arg not recognized — The recipe's command template controls how decisions reach the script. Make sure each {decisions.<id>} is paired with a flag the script's argparse defines (e.g. --<id> {decisions.<id>} ↔ parser.add_argument('--<id>')).
• Recipe input not found — Materialize upstream outputs first.
• Undeclared placeholder error — A {decisions.<id>} or {inputs.<id>} in the recipe references something not listed in Output.decisions / Output.inputs. Add it to the Output's declaration, or remove the placeholder.
• lc verify failure — missing_manifest (output dir exists with no .lightcone-manifest.json), tampered_data (bytes on disk no longer hash to the recorded data_version), or broken_chain (an upstream's data_version drifted from what this output's manifest recorded). Re-run the affected output with lc run to repair.

After failure: fix, then lc run <output_id> --universe <name>.

Publishing Analyses

lc export wrroc bundles the project's manifests, workflow definition, decisions, and (optionally) data files into a Workflow Run RO-Crate — a JSON-LD package readable by RO-Crate-aware archives (WorkflowHub, Zenodo's RO-Crate plugin, etc.). The lightcone manifest format on disk is unchanged; the bundle is generated on demand.

lc export wrroc                                # ./wrroc/ directory
lc export wrroc -o run.zip --zip               # zip bundle
lc export wrroc --metadata-only                # provenance graph + manifests only (no data files)
lc export wrroc -u baseline -u alt_method      # restrict to specific universes
lc export wrroc --author "Name <email@host>"   # override git config

The bundle's @graph contains a ComputationalWorkflow (the astra.yaml), one CreateAction per materialized output (with object referencing upstream datasets and external inputs, result referencing the produced dataset, and instrument referencing the recipe SoftwareApplication), PropertyValue entries for decisions and provenance metadata (code_version, data_version), and a Person for the author.