metadata-augmented-analytical-table-identification

name

metadata-augmented analytical table identification

description

Use this skill when the user wants to find information trapped inside data tables, spreadsheets, or databases but doesn't know which file to open. It is perfect for requests like 'find the table with school scores in Alameda', 'which database has the sports results for 2009', 'check if this fact about racing is true', or 'write a query for the city furthest from Boston'. Use this specifically when table titles are missing, uninformative, or when the answer requires pulling from multiple related databases.

Skill: metadata-augmented analytical table identification

1. Capability Definition & Real Case

• Professional Definition: The ability to perform open-domain table retrieval by mapping natural language analytical or lookup queries to relevant structured data containers within a large, heterogeneous corpus. This capability involves utilizing dense semantic embeddings of table schemas, metadata (such as generated summaries or titles), and sampled row contents to overcome the lexical gap between unstructured queries and structured storage, particularly in scenarios involving missing metadata or 'Text-to-SQL' multi-table dependencies.
• Dimension Hierarchy: Specialized Knowledge Navigation->Corpus-Specific Retrieval & Reasoning->metadata-augmented analytical table identification

Real Case

[Case 1]

• Initial Environment: A large-scale RAG corpus containing thousands of unlabeled education spreadsheets and CSV files from California school districts.
• Real Question: What is the highest eligible free rate for K-12 students in the schools in Alameda County?
• Real Trajectory: 1. The agent analyzes the query to identify key entities ('Alameda County') and metrics ('eligible free rate', 'K-12 students'). 2. It scans the metadata-augmented index, searching for tables where generated summaries mention 'education', 'enrollment', or 'school nutrition'. 3. It identifies a table with a generic title like '2-1570-4.csv' but whose columns include 'COUNTY', 'SCHOOL_NAME', and 'FREE_MEAL_RATE'. 4. The agent extracts the specific rows for schools in Alameda County to determine the maximum value among K-12 institutions.
• Real Answer: 0.98 (or the specific calculated maximum found in the retrieved record).
• Why this demonstrates the capability: This case demonstrates the ability to perform 'analytical retrieval' where the question mimics a data analysis task. The agent must successfully navigate a corpus of tables with uninformative titles by using semantic schema and content matching to locate the correct structured container for a calculation.

---

[Case 2]

• Initial Environment: A multi-database environment containing historical sports archives, including the 2009 World Games participation data and athlete performance logs.
• Real Question: How's Huang Yu-ting doing in the 2009 World Games?
• Real Trajectory: 1. The agent identifies 'Huang Yu-ting' as the primary entity and '2009 World Games' as the temporal and event context. 2. It retrieves the table whose generated summary explicitly mentions 'World Games 2009 Results' or contains the name in the sampled row content. 3. It locates the specific row corresponding to the athlete and synthesizes a natural language summary of her performance/rank.
• Real Answer: Huang Yu-ting won three gold medals in speed skating at the 2009 World Games.
• Why this demonstrates the capability: This is a 'Lookup Task' in an open-domain setting. It tests whether the retriever can distinguish between multiple tables with similar domain labels (e.g., World Games 2008 vs 2009) to find the specific record for an entity.

---

[Case 3]

• Initial Environment: A relational database corpus for a Text-to-SQL agent, featuring complex schemas across multiple interconnected databases like 'Spider' or 'BIRD'.
• Real Question: What is the name of the city furthest to Boston?
• Real Trajectory: 1. The agent identifies the need for a 'city' entity and a spatial metric (distance from Boston). 2. It performs 'Capped Recall' exploration, identifying that it must retrieve both a 'Cities' table and a 'Coordinates' or 'Geography' table if the information is split. 3. It retrieves the schema for the relevant database and generates the necessary SQL logic to calculate distances.
• Real Answer: Honolulu (or the specific city returned by the SQL execution).
• Why this demonstrates the capability: This demonstrates 'Multi-Table Dependency Identification'. For Text-to-SQL tasks, the agent must identify all tables required to form a valid join, even if the user query only explicitly mentions one entity type.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

1. Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md

2. Phase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md

3. Phase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md