ui-navigation-trajectory-retrieval

# Skill: ui-navigation-trajectory-retrieval ## 1. Capability Definition & Real Case * **Professional Definition**: The ability to retrieve and synthesize actionable multi-step navigation trajectories from a structured vector database derived from UI Transition Graphs (UTGs). This involves mapping natural language user intents to pre-computed sequences of UI actions (clicks, swipes, inputs) within a bounded application environment, enabling the agent to provide precise navigational guidance even for deeply nested or unconventional interface structures. * **Dimension Hierarchy**: Specialized Knowledge Navigation->Corpus-Specific Retrieval & Reasoning->ui-navigation-trajectory-retrieval ### Real Case **[Case 1]** * **Initial Environment**: A RAG-ready structured knowledge base containing the UI Transition Graph (UTG) for a reading application. The graph contains nodes representing screens (Home, Me, Settings, About) and edges representing click actions on specific widgets. * **Real Question**: View Pomodoro Reading App’s Privacy Policy. * **Real Trajectory**: [{'step': 1, 'observation': 'Home Screen with various icons.', 'action': "Click the 'Me' button to enter the user profile section."}, {'step': 2, 'observation': 'Profile Screen with account options.', 'action': "Click the 'Settings' gear icon."}, {'step': 3, 'observation': 'Settings List.', 'action': "Click 'About Pomodoro Reading App'."}, {'step': 4, 'observation': 'About page with legal links.', 'action': "Click 'Privacy Policy' to fulfill the request."}] * **Real Answer**: To view the Privacy Policy, first click on the 'Me' icon, then go to 'Settings', select 'About Pomodoro Reading App', and finally click on 'Privacy Policy'. * **Why this demonstrates the capability**: The question requires navigating a deeply nested UI structure where the target ('Privacy Policy') is not visible from the start. The agent demonstrates the capability by retrieving a specific four-step trajectory from the structured knowledge base that successfully links the initial 'Home' state to the final 'Privacy Policy' state through intermediate milestones. --- **[Case 2]** * **Initial Environment**: A vector database representing a financial application's navigational flow. The database indexes high-level intents like 'Invest', 'Support', and 'Account Management' alongside their required interaction sequences. * **Real Question**: Find online financial customer support. * **Real Trajectory**: [{'step': 1, 'observation': 'App Landing Page.', 'action': "Click the 'Financial Management' menu button."}, {'step': 2, 'observation': 'Financial main interface.', 'action': 'Click the search bar at the top.'}, {'step': 3, 'observation': 'Search input activated.', 'action': "Input 'online financial customer support'."}, {'step': 4, 'observation': 'Search results list.', 'action': "Click the 'Online Support' button to enter the chat."}] * **Real Answer**: You can find support by clicking 'Financial Management', using the search bar to type 'online financial customer support', and then selecting the 'Online Support' button. * **Why this demonstrates the capability**: This case shows intent-guided retrieval where a complex search-and-click sequence is recovered from its vector embedding. Success depends on the RAG agent's ability to match the user's abstract intent to a concrete, multi-modal action sequence (text input + buttons) found in the specialized application corpus. ## Pipeline Execution Instructions To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially: 1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md` 2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md` 3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`