ui-navigation-trajectory-retrieval

name

ui-navigation-trajectory-retrieval

description

Use this skill when the user asks questions about how to find specific features or perform multi-step tasks within a software application. Trigger it for requests like 'where is the settings menu?', 'guide me to the logout button', 'how do I view the privacy policy in this app?', or 'show me the path to buy a stock'. This skill is specifically for retrieving structured navigation sequences from specialized knowledge bases that map user intents to application UI transition graphs.

Skill: ui-navigation-trajectory-retrieval

1. Capability Definition & Real Case

• Professional Definition: The ability to retrieve and synthesize actionable multi-step navigation trajectories from a structured vector database derived from UI Transition Graphs (UTGs). This involves mapping natural language user intents to pre-computed sequences of UI actions (clicks, swipes, inputs) within a bounded application environment, enabling the agent to provide precise navigational guidance even for deeply nested or unconventional interface structures.
• Dimension Hierarchy: Specialized Knowledge Navigation->Corpus-Specific Retrieval & Reasoning->ui-navigation-trajectory-retrieval

Real Case

[Case 1]

• Initial Environment: A RAG-ready structured knowledge base containing the UI Transition Graph (UTG) for a reading application. The graph contains nodes representing screens (Home, Me, Settings, About) and edges representing click actions on specific widgets.
• Real Question: View Pomodoro Reading App’s Privacy Policy.
• Real Trajectory: [{'step': 1, 'observation': 'Home Screen with various icons.', 'action': "Click the 'Me' button to enter the user profile section."}, {'step': 2, 'observation': 'Profile Screen with account options.', 'action': "Click the 'Settings' gear icon."}, {'step': 3, 'observation': 'Settings List.', 'action': "Click 'About Pomodoro Reading App'."}, {'step': 4, 'observation': 'About page with legal links.', 'action': "Click 'Privacy Policy' to fulfill the request."}]
• Real Answer: To view the Privacy Policy, first click on the 'Me' icon, then go to 'Settings', select 'About Pomodoro Reading App', and finally click on 'Privacy Policy'.
• Why this demonstrates the capability: The question requires navigating a deeply nested UI structure where the target ('Privacy Policy') is not visible from the start. The agent demonstrates the capability by retrieving a specific four-step trajectory from the structured knowledge base that successfully links the initial 'Home' state to the final 'Privacy Policy' state through intermediate milestones.

---

[Case 2]

• Initial Environment: A vector database representing a financial application's navigational flow. The database indexes high-level intents like 'Invest', 'Support', and 'Account Management' alongside their required interaction sequences.
• Real Question: Find online financial customer support.
• Real Trajectory: [{'step': 1, 'observation': 'App Landing Page.', 'action': "Click the 'Financial Management' menu button."}, {'step': 2, 'observation': 'Financial main interface.', 'action': 'Click the search bar at the top.'}, {'step': 3, 'observation': 'Search input activated.', 'action': "Input 'online financial customer support'."}, {'step': 4, 'observation': 'Search results list.', 'action': "Click the 'Online Support' button to enter the chat."}]
• Real Answer: You can find support by clicking 'Financial Management', using the search bar to type 'online financial customer support', and then selecting the 'Online Support' button.
• Why this demonstrates the capability: This case shows intent-guided retrieval where a complex search-and-click sequence is recovered from its vector embedding. Success depends on the RAG agent's ability to match the user's abstract intent to a concrete, multi-modal action sequence (text input + buttons) found in the specialized application corpus.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

1. Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md

2. Phase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md

3. Phase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md