| name | gui_agent |
| display_name | GUI Agent Specialist |
| description | Natural language web UI control — element detection, targeted interaction, and automated form filling |
| version | 0.1.0 |
| source_repo | alibaba/page-agent |
| license | Apache-2.0 |
| tier | core |
| capabilities | ["gui_automation","web_interaction","element_detection","form_filling"] |
| allowed_tools | ["detect_elements","interact_element","fill_form"] |
| output_formats | ["python_api","cli","mcp_server","agent_skill","rest_api"] |
GUI Agent Specialist
Overview
The GUI Agent specialist wraps the alibaba/page-agent
natural-language web UI control framework into a composable OSS Agent Lab specialist. Given a
plain-English instruction and a target URL, it:
- Detects elements — discovers interactive UI elements on the page, optionally narrowed by
a semantic description of the target.
- Interacts — performs the requested action (click, type, hover, focus, or clear) on the
matched element.
- Fills forms — when form data is provided, populates every field, validates values, and
reports whether the form is ready to submit.
Each stage is independently callable as a tool, making the specialist suitable for targeted
automation steps as well as full end-to-end web-interaction pipelines.
Capabilities
- gui_automation: End-to-end pipeline that detects elements and performs actions from a
natural-language description.
- web_interaction: Single-element actions (click, type, hover, focus, clear) driven by
element ID.
- element_detection: Structured discovery of interactive elements on any URL, with optional
semantic filtering.
- form_filling: Batch field population with built-in validation and submit-readiness reporting.
Tools
| Tool | Description | Parameters | Side Effects |
|---|
detect_elements | Detect interactive UI elements on a page | url, description | None (read-only) |
interact_element | Perform an action on a specific element | element_id, action, value | Modifies page state |
fill_form | Fill a web form with field-value pairs | url, form_data | Modifies page state |
Tool Parameter Reference
detect_elements
url: str — fully-qualified URL of the target page (required)description: str | None — semantic hint to filter detected elements (default: )
