| name | era-embodied-agents |
| title | ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online RL |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2510.12693 |
| keywords | ["embodied-agents","vla","trajectory-augmentation","online-rl","self-summarization"] |
| description | Transform vision-language models into embodied agents through two stages: learning embodied priors from trajectory-augmented data with LLM reasoning, then online RL with self-summarization and dense rewards for long-horizon tasks. |
ERA: Two-Stage Framework for VLM-to-Agent Transformation
Vision-language models excel at understanding but lack embodied action grounding. ERA transforms VLMs into effective embodied agents through a two-stage approach: first distilling embodied knowledge, then refining with online RL.
Core insight: VLMs can learn embodied reasoning by combining trajectory demonstrations with LLM-generated reasoning, then adapting through online RL with proper credit assignment and context management. Small models (3B) can match or exceed larger models (GPT-4o) on embodied tasks.
Core Concept
Embodied Prior Learning: Augment trajectory data with structured reasoning from stronger models, grounding language understanding in action and environment constraints.
Online RL Refinement: Adapt learned priors through RL with three mechanisms: self-summarization for context, dense reward shaping, and turn-level policy optimization.
Architecture Overview
- Trajectory Augmentation Engine: Enriches raw trajectory data with reasoning steps
- Prior Learning Module: Fine-tunes VLM on augmented trajectories
- Online RL Agent: Explores environment with shaped rewards
- Context Manager: Self-summarization for long-horizon state tracking
Implementation Steps
Stage 1: Trajectory Augmentation with LLM Reasoning
Enrich trajectories with structured reasoning from stronger models:
from transformers import AutoTokenizer, AutoModelForCausalLM
class TrajectoryAugmenter:
def __init__(self, reasoning_model='claude-opus'):
self.reasoning_model = reasoning_model
self.tokenizer = AutoTokenizer.from_pretrained('llama-7b')
def augment_trajectory(self, trajectory):
"""
Add reasoning steps to trajectory.
trajectory: list of (observation, action) pairs
"""
augmented = []
for step_idx, (obs, action) in (trajectory):
reasoning = ._generate_reasoning(
trajectory[:step_idx],
obs,
action
)
augmented_step = {
: obs,
: action,
: reasoning,
: step_idx,
: trajectory[:step_idx]
}
augmented.append(augmented_step)
augmented
():
prompt =
reasoning = .reasoning_model.generate(
prompt,
max_tokens=,
temperature=
)
reasoning
(nn.Module):
():
().__init__()
.vla = AutoModelForCausalLM.from_pretrained(vla_model_name)
.tokenizer = AutoTokenizer.from_pretrained(vla_model_name)
():
obs_tokens = .tokenizer.encode()
reasoning_tokens = .tokenizer.encode(
)
action_tokens = .tokenizer.encode()
input_ids = torch.tensor(
obs_tokens + reasoning_tokens + action_tokens[:-]
)
logits = .vla(input_ids).logits
target = torch.tensor(action_tokens[:])
loss = torch.nn.functional.cross_entropy(
logits[-(target):],
target
)
loss
():
optimizer = torch.optim.AdamW(vla.parameters(), lr=lr)
epoch (num_epochs):
total_loss =
trajectory augmented_trajectories:
step trajectory:
loss = vla(
step[],
step[],
step[]
)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
()
vla
