harnessing-precision-querying-retrieval-augmented

name

harnessing-precision-querying-retrieval-augmented

description

LLM-driven precision querying of structured tabular data via Python/Pandas code generation and retrieval-augmented extraction from unstructured clinical text. Use when: 'query this table in natural language', 'extract information from clinical notes', 'build a RAG pipeline for medical records', 'generate Pandas code from a question', 'answer questions about EHR data', 'set up evaluation for Q&A over datasets'.

Precision Querying and Retrieval-Augmented Knowledge Extraction

This skill enables Claude to perform two complementary data science tasks on tabular and unstructured text data: (1) converting natural language questions into executable Python/Pandas code that queries structured datasets, and (2) building Retrieval-Augmented Generation (RAG) pipelines that extract precise answers from unstructured text documents. The approach follows the dual-pipeline framework from Rubio Jan et al. (2026), which demonstrated that LLMs can reliably interact with large structured datasets for analytics and extract semantically correct information from free-text records when supported by retrieval augmentation and a synthetic evaluation harness.

When to Use

• When the user has a CSV, Parquet, or database table and wants to ask natural language questions about it (e.g., "What is the average length of stay for patients over 65?")
• When the user needs to extract structured facts from unstructured clinical notes, discharge summaries, or other free-text medical documents
• When building a RAG pipeline over a corpus of clinical or domain-specific text
• When the user wants to auto-generate synthetic question-answer pairs to evaluate a querying or extraction system
• When converting analyst questions into Pandas code that runs against a DataFrame
• When the user needs to combine structured table querying with unstructured text retrieval in a single analytical workflow
• When evaluating LLM-generated answers against ground truth using exact match, semantic similarity, or human-judgment protocols

Key Technique

Task 1 -- Structured Data Querying (NL-to-Pandas). The user poses a natural language question about a tabular dataset. The LLM receives a schema description (column names, data types, sample rows, value distributions) and generates executable Python/Pandas code that answers the question. The generated code is executed in a sandboxed environment, and the result is returned as the answer. The critical insight is that schema-aware prompting -- providing the LLM with rich metadata about the DataFrame rather than raw data -- dramatically improves code correctness. Column descriptions, categorical value lists, and data type annotations reduce hallucinated column names and type errors.

Task 2 -- RAG over Unstructured Text. Clinical notes or other free-text documents are chunked, embedded into a vector store, and retrieved at query time. The retrieved chunks are injected into the LLM prompt as context, and the LLM generates an answer grounded in those chunks. The key design choices are: chunk size (typically 256-512 tokens for clinical notes to preserve paragraph-level context), overlap (10-20% to avoid splitting key phrases), embedding model selection (domain-tuned embeddings outperform general-purpose ones for medical text), and a generation prompt that instructs the LLM to answer strictly from the provided context with citations to source chunks.

Synthetic Evaluation Framework. Rather than manually creating test sets, the framework auto-generates question-answer pairs tailored to each dataset. For structured data, questions are generated by programmatically composing query templates (aggregations, filters, joins) over the schema and executing them to get ground-truth answers. For unstructured text, an LLM generates questions about key entities and facts in each document chunk, with answers extracted directly from the text. This enables rapid, repeatable evaluation across datasets without manual annotation.

Step-by-Step Workflow

Pipeline A: Natural Language to Pandas Code

1. Load and profile the dataset. Read the CSV/Parquet into a DataFrame. Extract schema metadata: column names, dtypes, null counts, unique value counts, and 3-5 sample rows. For categorical columns with fewer than 20 unique values, list all values. For numeric columns, compute min/max/mean.

2. Build the schema prompt. Construct a system prompt that includes the DataFrame variable name, full schema metadata, and an instruction like: "Generate a single Python expression using Pandas that answers the user's question. Use only the columns listed above. Return only executable code, no explanation. Assign the final result to a variable called answer."

3. Generate the Pandas code. Pass the user's natural language question along with the schema prompt to the LLM. The LLM outputs Python/Pandas code.

4. Validate before execution. Check the generated code for: (a) references to columns not in the schema, (b) dangerous operations (file I/O, imports beyond pandas/numpy, exec/eval calls), (c) syntactic correctness via ast.parse(). Reject and re-prompt if validation fails.

5. Execute in a sandboxed environment. Run the validated code against the actual DataFrame. Capture the value of answer. If execution raises an exception, feed the error traceback back to the LLM for a corrective attempt (up to 2 retries).

6. Format and return the result. Present the answer in a human-readable format. If the result is a DataFrame, render it as a markdown table. If scalar, present it with appropriate units or context.

Pipeline B: RAG over Unstructured Text

7. Chunk the documents. Split clinical notes or text documents into chunks of 256-512 tokens with 10-20% overlap. Preserve paragraph and section boundaries where possible. Attach metadata to each chunk (source document ID, section heading, position).

8. Embed and index. Embed each chunk using a sentence-transformer or domain-specific embedding model (e.g., all-MiniLM-L6-v2 for general use, or a biomedical model like BioLord for clinical text). Store embeddings in a vector database (FAISS, ChromaDB, or similar).

9. Retrieve relevant chunks. At query time, embed the user's question with the same model and retrieve the top-k most similar chunks (k=3-5). Optionally apply a reranking step using a cross-encoder to improve precision.

10. Generate a grounded answer. Construct a prompt with the retrieved chunks as context and an instruction: "Answer the question using ONLY the information in the provided context. If the answer is not in the context, say so. Cite the source chunk number for each claim." Pass this to the LLM and return the response.

Evaluation (Optional)

11. Generate synthetic Q&A pairs. For structured data: programmatically compose queries (single-column aggregations, multi-column filters, joins) and execute them to get ground-truth answers. For unstructured text: prompt an LLM to generate fact-based questions from each chunk, with answers copied verbatim from the text.

12. Evaluate with multiple metrics. Compare generated answers to ground truth using: exact string match (after normalization), semantic similarity (cosine similarity of embeddings), and optionally flag ambiguous cases for human review.

Concrete Examples

Example 1: Natural Language Query over a Patient Admissions Table

User: "I have a CSV of hospital admissions. What's the most common admission type for patients who stayed longer than 7 days?"

Approach:

1. Load admissions.csv into a DataFrame and profile it.
2. Identify relevant columns: ADMISSION_TYPE, LOS (length of stay), dtypes.
3. Generate schema prompt with column metadata.
4. LLM generates:

answer = df[df['LOS'] > 7]['ADMISSION_TYPE'].value_counts().idxmax()

5. Validate: columns exist, no dangerous ops. Execute.
6. Return: "The most common admission type for stays longer than 7 days is EMERGENCY."

Example 2: RAG Extraction from Discharge Summaries

User: "Search these discharge notes and tell me what medications were prescribed for patient 12345 at discharge."

Approach:

1. Chunk the discharge summary corpus (one note per patient, split into ~400-token chunks).
2. Embed and index with a biomedical sentence-transformer.
3. Retrieve top-5 chunks matching "medications prescribed discharge patient 12345".
4. Construct grounded generation prompt with retrieved chunks.
5. LLM responds:

Based on the discharge summary (Chunk 3, Document ID: note_12345):
- Metoprolol 50mg twice daily
- Lisinopril 10mg once daily
- Aspirin 81mg once daily

Source: Discharge note dated 2024-03-15, "Discharge Medications" section.

Example 3: Synthetic Q&A Generation for Evaluation

User: "Generate test questions for evaluating my querying system against this lab results table."

Approach:

1. Profile the labevents table: columns include SUBJECT_ID, ITEMID, VALUENUM, VALUEUOM, CHARTTIME.
2. Programmatically generate question templates:
   • Aggregation: "What is the average {VALUENUM} for lab item {ITEMID}?"
   • Filter: "How many lab results for patient {SUBJECT_ID} had values above {threshold}?"
   • Temporal: "What was the latest lab result for item {ITEMID} for patient {SUBJECT_ID}?"
3. Execute each template's corresponding Pandas query to get ground-truth answers.
4. Output Q&A pairs as JSON:

[
  {
    "question": "What is the average value for lab item 50862?",
    "ground_truth": "12.4",
    "query": "df[df['ITEMID']==50862]['VALUENUM'].mean()",
    "category": "aggregation"
  },
  {
    "question": "How many lab results for patient 109 had values above 100?",
    "ground_truth": "7",
    "query": "len(df[(df['SUBJECT_ID']==109) & (df['VALUENUM']>100)])",
    "category": "filter"
  }
]

Best Practices

• Do: Include full column metadata (names, types, sample values, distributions) in the schema prompt. The more the LLM knows about the data, the fewer hallucinated column references it produces.
• Do: Always validate generated Pandas code with ast.parse() and a column-name allowlist before execution. Never execute blindly.
• Do: Use chunk overlap (10-20%) when splitting documents to avoid losing information at chunk boundaries, especially for clinical text where key findings span paragraph breaks.
• Do: Instruct the RAG generation prompt to cite specific chunk numbers. This makes answers verifiable and builds user trust.
• Do: Normalize answers before exact-match comparison (lowercase, strip whitespace, standardize date formats, round numeric values to consistent precision).
• Avoid: Passing entire raw datasets into the LLM context. Use schema metadata for structured querying, not the full table.
• Avoid: Using generic embedding models for highly specialized domains without testing. Biomedical embeddings significantly outperform general-purpose ones on clinical text.
• Avoid: Setting chunk sizes too small (<128 tokens) for clinical notes -- clinical context often requires paragraph-level coherence to be interpretable.
• Avoid: Relying solely on exact-match evaluation. Semantically equivalent answers ("3 days" vs "72 hours") will fail exact match but are correct. Always pair with semantic similarity.

Error Handling

Error	Cause	Resolution
KeyError on column name	LLM hallucinated a column that doesn't exist	Re-prompt with explicit column list and the error message; limit retries to 2
Empty retrieval results	Query embedding doesn't match any chunks above similarity threshold	Lower the similarity threshold, or rephrase the query; consider hybrid search (keyword + semantic)
Generated code produces wrong dtype	LLM treated a string column as numeric or vice versa	Include explicit dtype information in schema prompt; add type-casting hints
Execution timeout	Generated code contains an expensive operation (e.g., cartesian join)	Set a timeout on code execution (5-10 seconds); reject and re-prompt with a complexity constraint
RAG answer contradicts source	LLM generated plausible-sounding but unsupported claims	Strengthen the grounding instruction; add "If uncertain, quote the exact text from context"
Semantic similarity score is low despite correct answer	Different phrasing or units between generated and ground-truth answer	Apply answer normalization (unit conversion, synonym mapping) before comparison

Limitations

• Schema complexity ceiling. NL-to-Pandas generation degrades with highly normalized schemas requiring multiple joins across many tables. For queries spanning more than 3-4 tables, consider pre-joining or creating views.
• Clinical terminology ambiguity. Medical abbreviations and jargon can cause retrieval failures if the embedding model wasn't trained on clinical text. Domain-specific embeddings help but don't eliminate this.
• Code generation is not deterministic. The same question may produce different (but equivalent) Pandas code across runs. Evaluation must account for functional equivalence, not syntactic identity.
• Privacy constraints. Real clinical data (e.g., MIMIC-III) requires data use agreements. The synthetic Q&A framework helps test pipelines without exposing protected health information, but production deployment needs careful de-identification.
• Context window limits. Very large documents or tables with many columns may exceed the LLM's context window even with schema summarization. Prioritize the most relevant columns or chunks.
• Single-turn only. The framework as described handles single questions. Multi-turn conversational querying (follow-up questions, pronoun resolution) requires additional session state management.

Reference

Rubio Jan, J.J., Wu, J., & Ive, J. (2026). Harnessing Large Language Models for Precision Querying and Retrieval-Augmented Knowledge Extraction in Clinical Data Science. arXiv:2601.20674v1. https://arxiv.org/abs/2601.20674v1

Key takeaway: The paper's dual-pipeline approach (schema-aware NL-to-Pandas + chunk-and-retrieve RAG) combined with automatic synthetic Q&A evaluation provides a reproducible framework for building and validating LLM-powered data access over both structured tables and unstructured text in clinical and other domain-specific settings.