fine-grained-widget-grounding
Use this skill when the user wants tiny, crowded, or easy-to-misclick GUI targets. Trigger it for requests like “make the answer depend on a small icon,” “use repeated buttons so the agent has to pick the right one,” “test tiny checkboxes or reply icons,” or “make it fail unless it clicks the exact small widget.” This skill is for GUI tasks where precision, counting, and local disambiguation matter more than broad page understanding.
Informations de source
- Dépôt
- Dingxingdi/paper_fast_search_backup
- Dernière activité de la source
- 8 avril 2026 à 15:14
- Langue détectée de SKILL.md
- anglais
- Étoiles
- 0
- Forks
- 0
Options d'installation
Le prompt qui vérifie d'abord la source est sélectionné par défaut. Vous pouvez passer à une commande directe ou télécharger une copie locale.
Vérifiez les fichiers source
Lisez SKILL.md et les fichiers associés affichés par SkillsMP avant de décider de l'installer.
Explorateur de fichiers
4 fichiersAffichage de SKILL.md
SKILL.md
Instructions source · Aperçu en lecture seule- name
- fine-grained-widget-grounding
- description
- Use this skill when the user wants tiny, crowded, or easy-to-misclick GUI targets. Trigger it for requests like “make the answer depend on a small icon,” “use repeated buttons so the agent has to pick the right one,” “test tiny checkboxes or reply icons,” or “make it fail unless it clicks the exact small widget.” This skill is for GUI tasks where precision, counting, and local disambiguation matter more than broad page understanding.
# Skill: fine-grained-widget-grounding
## 1. Capability Definition & Real Case
* **Professional Definition**: The capability to localize and act on a precise GUI widget in dense, repetitive, or small-scale layouts, especially when the target belongs to a long-tail element type such as icons, toggles, radio buttons, or small action affordances. The agent must separate the true target from visually similar distractors and execute a pixel-accurate action on the right element.
* **Dimension Hierarchy**: GUI Perception and Environment Modeling->Element Grounding->fine-grained-widget-grounding
### Real Case
**[Case 1]**
* **Initial Environment**: An email client shows an inbox with several messages arranged in a vertical list. Each row contains multiple compact icons, and the target action is the small reply icon associated specifically with the second email rather than the first or third.
* **Real Question**: Click the reply icon near the second email.
* **Real Trajectory**: Count the email rows from top to bottom, inspect the action cluster attached to the second row, distinguish the reply icon from adjacent icons such as archive or more-actions, and click that precise icon.
* **Real Answer**: The reply composer for the second email is opened.
* **Why this demonstrates the capability**: This case stresses fine-grained localization because multiple nearly identical icons are present and the instruction depends on ordinal position. The agent must jointly solve row counting, icon classification, and precise pointing under a dense layout. That makes the task a direct probe of high-resolution widget grounding rather than generic page comprehension.
## Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially:
1. **Phase 1: Environment Exploration**
Read the exploration guidelines to discover raw knowledge seeds:
`references/EXPLORATION.md`
2. **Phase 2: Trajectory Selection**
Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
`references/SELECTION.md`
3. **Phase 3: Data Synthesis**
Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
`references/SYNTHESIS.md`
Voir sur GitHub