원클릭으로 Manus에서 모든 스킬 실행

a3-eval

스타1

포크0

업데이트2026년 5월 11일 07:07

Evaluates mobile GUI agents on completing multi-step tasks across 20 real-world Android applications. It probes both final task completion capability and the agent's ability to navigate intermediate essential states without getting stuck or making terminal errors. Use when the user wants to benchmark on A3, or asks about evaluating this task. Reports Task Success Rate (SR).

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

qhjqhj00

qhjqhj00/research-skills-pool

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

a3-eval

A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation — Chai et al. (2025) (arXiv:2501.01149, 2025)

What this evaluates

Datasets

A3 — total 201; splits: test (201)

Metrics

Task Success Rate (SR) (primary) — range: percent
- Percentage of tasks where the agent successfully completes the entire user instruction. Calculated as (successful tasks / total tasks) * 100%.
Essential State Achieved Rate (ESAR) — range: percent
- Percentage of predefined essential intermediate states successfully reached during task execution. Calculated as (essential states achieved / total essential states) * 100%.

Input / output format

Input: Task instruction, initial app state/screenshots, and dynamic UI context for Android mobile applications.

Output: Sequential GUI actions (e.g., tap, swipe, type) or final completion signal per task.

Scoring recipe

def score(predictions, gold):
    total_tasks = len(predictions)
    successful_tasks = sum(1 for p in predictions if p['task_success'])
    sr = (successful_tasks / total_tasks) * 100
    
    total_essential_states = len(gold['essential_states'])
    achieved_states = sum(1 for p in predictions if p['essential_state_achieved'])
    esar = (achieved_states / total_essential_states) * 100
    
    return {'SR': sr, 'ESAR': esar}

Common pitfalls

SR and ESAR measure different failure modes; ESAR is typically much higher than SR because agents often start tasks correctly but fail on long-horizon navigation or progress awareness.
The automated evaluation relies on the A3RM reward model (or Gemini), which may hallucinate success for ambiguous states despite high precision, so manual verification is recommended for edge cases.

Evidence (verbatim from paper)

We report Task Success Rate (SR) and Essential State Achieved Rate (ESAR) across task categories and difficulties.

Citation

@misc{chai2025a3,
  title={A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation},
  author={Chai et al. (2025)},
  year={2025},
  note={arXiv:2501.01149}
}

arXiv: 2501.01149

이 저장소의 다른 Skills

같은 저장소

ether0-chemistry-rewards

qhjqhj00/research-skills-pool

Open-weights chemistry reasoning model + verifiable reward functions from FutureHouse's ether0 (arXiv 2506.17238). Use to score model-generated chemistry outputs (SMILES validity, molecular completion, synthesis reasoning) against ground truth, or to run the open-weights ether0 model itself for chemistry reasoning. Also useful for visualizing molecules and reactions from SMILES.

2026-05-111

2m-belebele-eval

qhjqhj00/research-skills-pool

Multilingual reading comprehension across text and speech modalities. It probes a model's ability to understand spoken or written passages in 39 languages and answer multiple-choice questions based on them. Use when the user wants to benchmark on 2M-Belebele, or asks about evaluating this task. Reports accuracy.

2026-05-111

360roam-eval

qhjqhj00/research-skills-pool

Evaluates the capability of neural radiance field models to perform real-time, high-fidelity novel view synthesis on large-scale indoor scenes using 360° panoramic imagery. It probes the trade-off between rendering quality, computational efficiency, and geometric awareness in complex, unbounded indoor environments. Use when the user wants to benchmark on 360Roam Dataset, or asks about evaluating this task. Reports PSNR.

2026-05-111

3d-2d-vl-grounding-eval

qhjqhj00/research-skills-pool

Evaluates a unified vision-language model's ability to ground natural language instructions to 3D objects and 2D regions, as well as answer 3D visual questions. It probes spatial reasoning, cross-modal alignment, and robustness to different 3D input representations (mesh-sampled vs. sensor RGB-D point clouds). Use when the user wants to benchmark on SR3D, NR3D, ScanRefer, RefCOCO, RefCOCO+, RefCOCOg, ScanQA, SQA3D, or asks about evaluating this task. Reports top-1 accuracy (Acc@25/50/75).

2026-05-111

3d-affordance-net-eval

qhjqhj00/research-skills-pool

Evaluates 3D point cloud networks on visual object affordance understanding. It probes the model's ability to predict point-wise probabilistic scores for 18 affordance classes given full, partial, or rotated 3D shapes. Use when the user wants to benchmark on 3D AffordanceNet, or asks about evaluating this task. Reports mAP.

2026-05-111

3d-face-recon-eval

qhjqhj00/research-skills-pool

Evaluates the accuracy and realism of 3D face reconstruction and generation from single 2D images. Probes shape reconstruction fidelity against ground truth meshes, texture identity preservation across novel poses, and the diversity of synthesized 3D faces. Use when the user wants to benchmark on NoW Benchmark, REALY 3D Benchmark, or asks about evaluating this task. Reports Per-vertex error (mm).

2026-05-111

name	a3-eval
description	Evaluates mobile GUI agents on completing multi-step tasks across 20 real-world Android applications. It probes both final task completion capability and the agent's ability to navigate intermediate essential states without getting stuck or making terminal errors. Use when the user wants to benchmark on A3, or asks about evaluating this task. Reports Task Success Rate (SR).
metadata	{"skill_kind":"dataset_eval","source_arxiv":2501.01149,"bibtex_key":"chai2025a3","confidence":"high"}

a3-eval

A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation — Chai et al. (2025) (arXiv:2501.01149, 2025)

What this evaluates

Datasets

A3 — total 201; splits: test (201)

Metrics

Task Success Rate (SR) (primary) — range: percent
- Percentage of tasks where the agent successfully completes the entire user instruction. Calculated as (successful tasks / total tasks) * 100%.
Essential State Achieved Rate (ESAR) — range: percent
- Percentage of predefined essential intermediate states successfully reached during task execution. Calculated as (essential states achieved / total essential states) * 100%.

Input / output format

Input: Task instruction, initial app state/screenshots, and dynamic UI context for Android mobile applications.

Output: Sequential GUI actions (e.g., tap, swipe, type) or final completion signal per task.

Scoring recipe

def score(predictions, gold):
    total_tasks = len(predictions)
    successful_tasks = sum(1 for p in predictions if p['task_success'])
    sr = (successful_tasks / total_tasks) * 100
    
    total_essential_states = len(gold['essential_states'])
    achieved_states = sum(1 for p in predictions if p['essential_state_achieved'])
    esar = (achieved_states / total_essential_states) * 100
    
    return {'SR': sr, 'ESAR': esar}

Common pitfalls

SR and ESAR measure different failure modes; ESAR is typically much higher than SR because agents often start tasks correctly but fail on long-horizon navigation or progress awareness.
The automated evaluation relies on the A3RM reward model (or Gemini), which may hallucinate success for ambiguous states despite high precision, so manual verification is recommended for edge cases.

Evidence (verbatim from paper)

We report Task Success Rate (SR) and Essential State Achieved Rate (ESAR) across task categories and difficulties.

Citation

@misc{chai2025a3,
  title={A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation},
  author={Chai et al. (2025)},
  year={2025},
  note={arXiv:2501.01149}
}

arXiv: 2501.01149