<div align="center">
<img src="assets/ClawGUI-Logo.png" height="140" alt="ClawGUI Logo">
<h1>ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents</h1>
[](https://www.python.org/downloads/release/python-3120/)
[](https://opensource.org/licenses/Apache-2.0)
[](https://github.com/ZJU-REAL/ClawGUI/stargazers)
[](https://arxiv.org/abs/2604.11784)
[](https://huggingface.co/papers/2604.11784)
[](https://huggingface.co/SugarVapeur/OpenGUI-2B)
[](https://www.modelscope.cn/models/SugarFree/OpenGUI-2B)
[](https://zju-real.github.io/ClawGUI-Page/)
[English](README.md) | [ไธญๆ](README_zh.md)
</div>
<div align="center">
<b>A full-stack framework for GUI agents, covering online RL training, standardized evaluation, and deployment.</b>
<table>
<tr>
<td align="center">
<video src="https://github.com/user-attachments/assets/38471771-c614-4520-a4a1-0b985546e023" controls width="420"></video>
<br><b>ClawGUI-Agent controls a real phone<br>via natural language</b>
</td>
<td align="center">
<video src="https://github.com/user-attachments/assets/a72f9229-15e4-439e-aeb1-82a05b843fbe" controls width="420"></video>
<br><b>ClawGUI-RL trains a GUI agent with online<br>reinforcement learning</b>
</td>
</tr>
</table>
</div>
## News
+ ๐ **[2026/4/14]** Our paper is available on arXiv: [ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents](https://arxiv.org/abs/2604.11784).
+ ๐ฅ **[2026/4/13]** ClawGUI is released โ train with ClawGUI-RL (GiGPO), evaluate with ClawGUI-Eval, deploy with ClawGUI-Agent. ClawGUI-2B, a 2B agent trained end-to-end with this pipeline, hits **17.1** MobileWorld SR vs. the **11.1** baseline. See [Quick Start](#-quick-start).
## Table of Contents
- [Overview](#-overview)
- [Architecture](#๏ธ-architecture)
- [Quick Start](#-quick-start)
- [ClawGUI-RL โ Build](#-clawgui-rl--build)
- [ClawGUI-Eval โ Evaluate](#-clawgui-eval--evaluate)
- [ClawGUI-Agent โ Deploy](#-clawgui-agent--deploy)
- [ClawGUI-Skills โ Self-Evolving Skills](#-clawgui-skills--self-evolving-skills)
- [ClawGUI-APP โ On-Device Deploy](#-clawgui-app--on-device-deploy)
- [Roadmap](#๏ธ-roadmap)
- [Acknowledgements](#-acknowledgements)
- [License](#-license)
## ๐ก Overview
**ClawGUI** is a research framework for GUI agents, covering the complete lifecycle from **online RL training** and **standardized evaluation** to **real-device deployment**.
Building a capable GUI agent involves three tightly coupled problems that are rarely solved together: you need an environment to train the agent online, rigorous benchmarks to measure what it has learned, and a production system to deploy it on real devices. ClawGUI addresses all three.
| Module | Role |
|--------|------|
| ๐ **[ClawGUI-RL](clawgui-rl/)** | **Build** โ Train GUI agents online with scalable RL: parallel Docker environments, real Android devices, and GiGPO+PRM for fine-grained step-level rewards |
| ๐ **[ClawGUI-Eval](clawgui-eval/)** | **Evaluate** โ Measure what the agent has learned: 6 benchmarks, 11+ models, 95.8% faithful reproduction of official results |
| ๐ค **[ClawGUI-Agent](clawgui-agent/)** | **Deploy** โ Use GUI agents in the real world: control mobile devices via natural language through 12+ chat platforms, with one-command evaluation built in |
| ๐งฉ **[ClawGUI-Skills](clawgui-skills/)** | **Self-evolving skills** โ Training-free skill evolution proposed and validated in our paper: structured packages, retrieval, failure diagnosis, restricted revision, and reuse |
| ๐ฑ **[ClawGUI-APP](clawgui-app/)** | **On-Device Deploy** โ Run the full brain + GUI agent stack directly on one Android phone, no desktop coordinator needed, powered by Shizuku |
| ๐ **ClawGUI-2B** | End-to-end validation: trained entirely with ClawGUI-RL and GiGPO, achieving **17.1** MobileWorld SR vs. the **11.1** baseline |
## ๐๏ธ Architecture
<div align="center">
<img src="assets/clawgui-framework.png" width="95%" alt="ClawGUI System Architecture">
</div>
## ๐ Quick Start
```bash
git clone https://github.com/ZJU-REAL/ClawGUI.git
cd ClawGUI
```
Each module is independent with its own environment. Click into each one for full installation and usage instructions.
### ๐ ClawGUI-RL โ Build
> ๐ [`clawgui-rl/`](clawgui-rl/) ยท ๐ [Full Documentation](clawgui-rl/README.md)
ClawGUI-RL trains GUI agents with online reinforcement learning. It runs dozens of Docker-based Android emulators in parallel or trains directly on physical devices โ and replaces standard GRPO with GiGPO+PRM for fine-grained step-level rewards that drive stronger policy learning.
- **Parallel multi-environment** โ Dozens of Docker-based virtual Android environments simultaneously
- **Real-device training** โ Physical or cloud Android phones with the same API
- **GiGPO + PRM** โ Fine-grained step-level reward for better policy optimization than standard GRPO
- **Spare server rotation** โ Automatic failover keeps training running without interruption
- **Episode visualization** โ Record and replay any training trajectory
<div align="center">
<img src="clawgui-rl/assets/clawgui-rl-framework.png" width="95%" alt="ClawGUI-RL Architecture">
</div>
โ **[Get started with ClawGUI-RL](clawgui-rl/README.md)**
### ๐ ClawGUI-Eval โ Evaluate
> ๐ [`clawgui-eval/`](clawgui-eval/) ยท ๐ [Full Documentation](clawgui-eval/README.md) ยท [๐ค Dataset](https://huggingface.co/datasets/johnzqlu/clawgui-eval) ยท [๐ค ModelScope](https://modelscope.cn/datasets/Matrix0602/clawgui-eval)
ClawGUI-Eval gives GUI grounding research a reliable measurement baseline. Its three-stage **Infer โ Judge โ Metric** pipeline covers 6 benchmarks and 11+ models, with a **95.8%** reproduction rate against official results โ so numbers across papers are actually comparable.
- **6 benchmarks** โ ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G, AndroidControl
- **11+ models** โ Qwen3-VL, Qwen2.5-VL, UI-TARS, MAI-UI, GUI-G2, UI-Venus, Gemini, Seed 1.8, and more
- **Dual backend** โ Local GPU (`transformers`) or remote API (OpenAI-compatible)
- **Multi-GPU & multi-thread** โ Parallel inference with automatic resume
- **ClawGUI-Agent integration** โ Pair with ClawGUI-Agent to run the full pipeline via natural language
<div align="center">
<img src="clawgui-eval/assets/clawgui-eval-arch.png" width="95%" alt="ClawGUI-Eval Architecture">
</div>
โ **[Get started with ClawGUI-Eval](clawgui-eval/README.md)**
### ๐ค ClawGUI-Agent โ Deploy
> ๐ [`clawgui-agent/`](clawgui-agent/) ยท ๐ [Full Documentation](clawgui-agent/README.md) ยท [ไธญๆ](clawgui-agent/README_CN.md)
ClawGUI-Agent closes the loop from training to production. Built on OpenClaw and powered by nanobot, it lets you control Android, HarmonyOS, or iOS devices with natural language from 12+ chat platforms โ and trigger the full ClawGUI-Eval benchmark pipeline with a single sentence, no scripts required.
- **Cross-platform** โ Android (ADB), HarmonyOS (HDC), iOS (XCTest)
- **Multi-model** โ AutoGLM, MAI-UI, GUI-Owl, Qwen-VL, UI-TARS via OpenAI-compatible API
- **One-command evaluation** โ Say "benchmark qwen3vl on screenspot-pro" and it handles env check โ multi-GPU inference โ judging โ metrics โ result comparison
- **Personalized memory** โ Automatically learns user preferences and injects context across tasks
- **Episode recording** โ Every task saved as structured episodes for replay and dataset building
- **Web UI** โ Gradio interface for device management, task execution, and memory inspection
<div align="center">
<img src="clawgui-agent/assets/clawgui-agent-logo.png" width="90%" alt="ClawGUI-Agent">
</div>
โ **[Get started with ClawGUI-Agent](clawgui-agent/README.md)**
### ๐งฉ ClawGUI-Skills โ Self-Evolving Skills
> ๐ [`clawgui-skills/`](clawgui-skills/) ยท ๐ [Full Documentation](clawgui-skills/README.md) ยท [ไธญๆ](clawgui-skills/README_zh.md)
ClawGUI-Skills implements the training-free self-evolving GUI skill architecture proposed and validated in our paper **โReflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents.โ** It stores procedural task knowledge as structured skill packages and lets PhoneAgent retrieve, inject, diagnose, and revise them on demand.
- **Four modes** โ `off`, `trace`, `reuse`, and `evolve`; disabled by default to avoid extra context cost
- **Structured packages** โ `meta_info.json`, `plan.md`, `backup.md`, `recover.md`, and `failure_examples/`
- **Instant revision** โ failed runs are diagnosed by an isolated verifier and mapped to targeted skill-file edits
- **Visual inspection** โ the Web UI shows matched skill name, `skill_id`, injected context, revisions, and failure examples
โ **[Get started with ClawGUI-Skills](clawgui-skills/README.md)**
### ๐ฑ ClawGUI-APP โ On-Device Deploy
> ๐ [`clawgui-app/`](clawgui-app/) ยท ๐ [Setup Guide](clawgui-app/SETUP.md)
ClawGUI-APP runs the full ClawGUI "brain + GUI agent" stack directly on one Android phone, removing the old split architecture where a desktop host orchestrates tasks and the phone only executes them. Built on Shizuku for high-privilege, non-root device control.
- **Phone-only workflow** โ No desktop coordinator required; a device with Shizuku is enough
- **Two-agent design** โ Brain LLM handles planning and tool orchestration, phone agent handles screen understanding and actions
- **Multi-model support** โ AutoGLM, MAI-UI, GUI-Owl, Qwen-VL, UI-TARS and more via OpenAI-compatible API
- **Voice input (STT)** โ Tap-to-record microphone with OpenAI-compatible speech-to-text transcription (SiliconFlow, Groq Whisper, etc.)
- **Conversation + automation** โ Sessions, long-term memory, external channels (Feishu), and trace replay
- **Built for real usage** โ Floating overlay status, built-in IME, session persistence, and diagnostics
โ **[Build ClawGUI-APP](clawgui-app/SETUP.md)**
## ๐ฏ Roadmap
- [x] **ClawGUI-Agent** โ GUI agent framework for phone control and evaluation via natural language
- [x] **ClawGUI-RL** โ Scalable mobile online RL training infrastructure with GiGPO + PRM
- [x] **ClawGUI-Eval** โ Standardized GUI grounding evaluation suite with 6 benchmarks and 95%+ reproduction rate
- [x] **ClawGUI-2B** โ 2B GUI agent trained with GiGPO, achieving 17.1 MobileWorld SR (vs. 11.1 baseline)
- [x] **On-device ClawGUI-Agent ([ClawGUI-APP](clawgui-app/))** โ Deploy ClawGUI-Agent directly on real phones โ no desktop coordinator, paving the way for fully on-device inference (brain/VLM still served via cloud API today)
- [ ] **Desktop Online RL** โ Extend ClawGUI-RL to desktop environments for online reinforcement learning
- [ ] **Web Online RL** โ Extend ClawGUI-RL to web environments for online reinforcement learning
- [ ] **More Skills for ClawGUI-Agent** โ Add more pluggable skills to expand ClawGUI-Agent's capabilities
- [ ] **Hybrid CLI & GUI Mechanism** โ Explore hybrid interaction combining command-line and GUI operations
- [ ] **Real-time RL** โ Integrate real-time reinforcement learning based on the OPD algorithm for ClawGUI-RL and ClawGUI-Agent
## ๐ค Contributing
We welcome contributions of all kinds โ new model support, new RL environments, bug fixes, and documentation improvements. See [CONTRIBUTING.md](CONTRIBUTING.md) for how to get started, module-specific guidelines, and PR requirements.
<a href="https://github.com/ZJU-REAL/ClawGUI/graphs/contributors">
<img src="https://contrib.rocks/image?repo=ZJU-REAL/ClawGUI" />
</a>
## ๐ Acknowledgements
ClawGUI is built upon the following excellent open-source projects. We sincerely thank their contributors:
- [**verl-agent**](https://github.com/langfengq/verl-agent)
- [**MAI-UI**](https://github.com/Tongyi-MAI/MAI-UI)
- [**MobileWorld**](https://github.com/Tongyi-MAI/MobileWorld)
- [**Mobile-Agent**](https://github.com/x-plug/mobileagent)
- [**nanobot**](https://github.com/HKUDS/nanobot)
- [**Open-AutoGLM**](https://github.com/zai-org/Open-AutoGLM)
## License
This project is licensed under the [Apache License 2.0](LICENSE).
## ๐ Citation
If you find ClawGUI useful in your research, please consider citing our paper:
```bibtex
@article{tang2026clawgui,
title={ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents},
author={Tang, Fei and Lu, Zhiqiong and Zhang, Boxuan and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and Shen, Yongliang},
journal={arXiv preprint arXiv:2604.11784},
year={2026}
}
```
## Star History
<a href="https://www.star-history.com/?repos=ZJU-REAL%2FClawGUI&type=date&legend=top-left">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=ZJU-REAL/ClawGUI&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=ZJU-REAL/ClawGUI&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=ZJU-REAL/ClawGUI&type=date&legend=top-left" />
</picture>
</a>