dingxingdi Skill: ui-navigation-trajectory-retrieval
1. Capability Definition & Real Case
- Professional Definition: The ability to retrieve and synthesize actionable multi-step navigation trajectories from a structured vector database derived from UI Transition Graphs (UTGs). This involves mapping natural language user intents to pre-computed sequences of UI actions (clicks, swipes, inputs) within a bounded application environment, enabling the agent to provide precise navigational guidance even for deeply nested or unconventional interface structures.
- Dimension Hierarchy: Specialized Knowledge Navigation->Corpus-Specific Retrieval & Reasoning->ui-navigation-trajectory-retrieval
Real Case
[Case 1]
- Initial Environment: A RAG-ready structured knowledge base containing the UI Transition Graph (UTG) for a reading application. The graph contains nodes representing screens (Home, Me, Settings, About) and edges representing click actions on specific widgets.
- Real Question: View Pomodoro Reading App’s Privacy Policy.
- Real Trajectory: [{'step': 1, 'observation': 'Home Screen with various icons.', 'action': "Click the 'Me' button to enter the user profile section."}, {'step': 2, 'observation': 'Profile Screen with account options.', 'action': "Click the 'Settings' gear icon."}, {'step': 3, 'observation': 'Settings List.', 'action': "Click 'About Pomodoro Reading App'."}, {'step': 4, 'observation': 'About page with legal links.', 'action': "Click 'Privacy Policy' to fulfill the request."}]
- Real Answer: To view the Privacy Policy, first click on the 'Me' icon, then go to 'Settings', select 'About Pomodoro Reading App', and finally click on 'Privacy Policy'.
- Why this demonstrates the capability: The question requires navigating a deeply nested UI structure where the target ('Privacy Policy') is not visible from the start. The agent demonstrates the capability by retrieving a specific four-step trajectory from the structured knowledge base that successfully links the initial 'Home' state to the final 'Privacy Policy' state through intermediate milestones.
[Case 2]
- Initial Environment: A vector database representing a financial application's navigational flow. The database indexes high-level intents like 'Invest', 'Support', and 'Account Management' alongside their required interaction sequences.
- Real Question: Find online financial customer support.
- Real Trajectory: [{'step': 1, 'observation': 'App Landing Page.', 'action': "Click the 'Financial Management' menu button."}, {'step': 2, 'observation': 'Financial main interface.', 'action': 'Click the search bar at the top.'}, {'step': 3, 'observation': 'Search input activated.', 'action': "Input 'online financial customer support'."}, {'step': 4, 'observation': 'Search results list.', 'action': "Click the 'Online Support' button to enter the chat."}]
- Real Answer: You can find support by clicking 'Financial Management', using the search bar to type 'online financial customer support', and then selecting the 'Online Support' button.
- Why this demonstrates the capability: This case shows intent-guided retrieval where a complex search-and-click sequence is recovered from its vector embedding. Success depends on the RAG agent's ability to match the user's abstract intent to a concrete, multi-modal action sequence (text input + buttons) found in the specialized application corpus.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.mdPhase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.mdPhase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md