| name | 3dmem-bench-eval |
| description | Evaluates an embodied 3D agent's ability to manage long-term spatial-temporal memory and execute complex, multi-room tasks. It probes the model's capacity for in-domain generalization, in-the-wild robustness, and long-horizon reasoning across navigation, question answering, and scene captioning. Use when the user wants to benchmark on 3DMem-Bench, or asks about evaluating this task. Reports success rate (SR). |
| metadata | {"skill_kind":"dataset_eval","source_arxiv":2505.22657,"bibtex_key":"hu20253dllmmem","confidence":"high"} |
3dmem-bench-eval
3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model — Wenbo Hu et al. (2025) (arXiv:2505.22657, 2025)
What this evaluates
Evaluates an embodied 3D agent's ability to manage long-term spatial-temporal memory and execute complex, multi-room tasks. It probes the model's capacity for in-domain generalization, in-the-wild robustness, and long-horizon reasoning across navigation, question answering, and scene captioning.
Datasets
- 3DMem-Bench — total 26000; splits: train (-1), test (-1)
Metrics
success rate (SR) (primary) — range: percent
- Percentage of tasks where the agent successfully completes the specified objective. Calculated as (number of successful executions / total number of tasks) * 100.
sub-success rate (Sub-SR) — range: percent
- Percentage of tasks where the agent successfully completes a sub-goal or intermediate step within a multi-step task.
accuracy — range: percent
- Proportion of correctly answered open-ended EQA questions, evaluated using an LLM-as-judge protocol with Gemini.
BLEU-1 / BLEU-4 / METEOR — range: [0, 100]
- Standard n-gram overlap and METEOR scores between generated captions and ground-truth references.
Input / output format
Input: Task instructions, current working memory state, and fused spatial-temporal features from dense 3D episodic memory, formatted as token sequences up to an 8192-token context window.
Output: Action execution trajectories for embodied tasks, or natural language responses for EQA and captioning tasks.
Scoring recipe
def compute_sr(predictions, golds):
correct = sum(1 for p, g in zip(predictions, golds) if p == g)
return (correct / len(golds)) * 100
def compute_eqa_accuracy(predictions, golds):
scores = [judge_score(p, g) for p, g in zip(predictions, golds)]
return (sum(scores) / len(scores)) * 100
def compute_captioning_metrics(predictions, golds):
return compute_bleu_meteor(predictions, golds)
Common pitfalls
- Confusing 'in-domain' vs 'in-the-wild' test splits, as baseline performance drops sharply in the latter due to distribution shift.
- Assuming 'Everything in Context' is a practical baseline; the 8192-token limit makes it infeasible for long-horizon, multi-room scenarios.
- LLM-as-judge evaluation for EQA relies on specific Gemini prompts, which may introduce bias or variance compared to exact-match metrics.
Evidence (verbatim from paper)
As shown in Table[2(a)], 3DLLM-Mem significantly outperforms all existing approaches on both in-domain and in-the-wild embodied tasks. Notably, while the performance of other methods drops sharply in the in-the-wild setting, our method demonstrates strong generalization capabilities with a average success rate of 32.1%.
Citation
@misc{hu20253dllmmem,
title={3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model},
author={Wenbo Hu et al. (2025)},
year={2025},
note={arXiv:2505.22657}
}
