mattergen-evaluation

name	mattergen-evaluation
description	Evaluate MatterGen-generated crystal structures for novelty, uniqueness, stability, and S.U.N. criteria, and produce filtered relaxed structures and evaluation reports.
tags	["MatterGen","evaluation"]
tools	["run_bash"]
dependent_skills	[]

MatterGen Skill

Use this skill to evaluate structures generated by a MatterGen model, compute quality metrics such as novelty, uniqueness, stability, and S.U.N.-related results, and filter the relaxed structures that satisfy the S.U.N. criteria when detailed evaluation outputs are available.

Mattergen Evaluation

The official MatterGen README also documents mattergen-evaluate, which can:

• relax generated structures by MatterSim,
• compute metrics such as novelty, uniqueness, stability, and S.U.N.-related benchmark metrics,
• filter structures

Reference:

• Official README evaluation section: https://github.com/microsoft/mattergen/blob/main/README.md

Evaluation With MatterSim on local

Important:

• The commands below are templates. Replace all placeholder paths with real absolute paths before execution.

Default MatterSim evaluation

First download the reference dataset, then evaluate the generated structures:

git lfs pull -I data-release/alex-mp/reference_MP2020correction.gz --exclude=""

ts=$(date +"%Y%m%d%H%M%S")
workdir="matg/${ts}.mattergen_evaluation"
mkdir -p "$workdir"
cd "$workdir"

"${MATTERGEN_ENV}/bin/mattergen-evaluate" \
  --structures_path=/abs/path/to/generated_results \ 
  --save_as=/abs/path/to/generated_results/metrics.json \
  --save_detailed_as=/abs/path/to/generated_results/detailed_metrics.json \
  --structures_output_path=/abs/path/to/generated_results/relaxed_structures.extxyz

Evaluation With MatterSim on Bohrium

When submitting MatterGen jobs to Bohrium through the bohrium skill, Bohrium-specific submission settings can be read from environment variables such as BOHRIUM_MAT_IMAGE and BOHRIUM_MAT_MACHINE.

Prepare a writable local job directory first:

ts=$(date +"%Y%m%d%H%M%S")
workdir="matg/${ts}.mattergen_evaluation"
mkdir -p "$workdir"
cd "$workdir"

Then copy the reference dataset into the workdir, using the local paths from the environment variables eval_reference, unless the user explicitly provides different paths.

Then copy the structures to be evaluated into the workdir, preferably in extxyz format.

In submission.json:

• set command to a generation command such as:

mkdir -p results && mattergen-evaluate --structures_path=generated_results --reference_dataset_path=dataset.gz --save_as=results/metrics.json --save_detailed_as=results/detailed_metrics.json --structures_output_path=results/relaxed_structures.extxyz

• structures_path and reference_dataset_path must point to the actual input files you are using, with their real filenames, for example generated.extxyz and MP_2020.gz.

• forward_files should include the reference dataset, structures, such as generated.extxyz, MP_2020.gz.

• backward_files can be results.

Filter structures that satisfy the S.U.N. criteria

After evaluation, when both relaxed_structures.extxyz and detailed_metrics.json are available, filter the relaxed structures using the novel_unique_stable list in detailed_metrics.json.

Filtering procedure:

• Read the boolean list novel_unique_stable from detailed_metrics.json.
• Treat each true entry in novel_unique_stable as one structure that satisfies the S.U.N. criteria.
• Match the list index in novel_unique_stable to the frame index in relaxed_structures.extxyz.
• Keep only the frames whose corresponding novel_unique_stable value is true.
• Write the retained structures to a new file such as /abs/path/to/generated_results/sun_structures.extxyz.
• Return the absolute path to the generated S.U.N. extxyz file and the number of retained structures.

What to report to the user

After evaluation, report at minimum:

• the absolute path to detailed_metrics.json, if generated,
• the absolute path to relaxed_structures.extxyz, if generated,
• the absolute path to sun_structures.extxyz, if generated,
• the number of structures retained after S.U.N. filtering, if filtering was performed,
• the key aggregate metrics from metrics.json, especially stability, uniqueness, novelty, and S.U.N.-related results when present.