langextract-extraction

name	langextract-extraction
description	Extract structured entities from unstructured text using few-shot prompting with LLMs. Use when you need to extract people, places, events, relationships, emotions, or any structured information from text documents. Supports custom entity types, attributes, and relationships. Works with any text: articles, documents, conversations, social media, or domain-specific content.

LangExtract-Style Entity Extraction

Extract structured, grounded entities from unstructured text using few-shot prompting.

Core Concept

LangExtract's approach: Show, don't tell. Instead of complex instructions, provide 1-2 examples of the exact extraction format you want. The LLM learns the pattern and applies it to new text.

Quick Start

Basic Extraction

User request: "Extract people and events from this meeting transcript"

Your workflow:

1. Define extraction format with one example:

Examples
Q: Today I met with Sarah Johnson at the downtown office. We discussed the Q4 launch plan.
A: {
  "extractions": [
    {
      "人物": "Sarah Johnson",
      "人物_attributes": {}
    },
    {
      "地点": "downtown office",
      "地点_attributes": {}
    },
    {
      "事件": "Q4 launch plan discussion",
      "事件_attributes": {}
    }
  ]
}

Q: [USER'S TEXT HERE]
A:

2. Send to LLM and parse JSON response

3. Align extractions to source text (find character positions)

That's it. No complex prompts needed.

Extraction Format

Standard Structure

{
  "extractions": [
    {
      "类别": "提取的文本",
      "类别_attributes": {
        "key": "value"
      }
    }
  ]
}

Entity Types

Common entity types (customize as needed):

• 人物 (Person): Names, roles, descriptions
• 地点 (Location): Physical/virtual locations, addresses
• 事件 (Event): Actions, occurrences, milestones
• 时间 (Time): Dates, time periods, durations
• 组织 (Organization): Companies, teams, groups
• 情感 (Emotion): Feelings, sentiments, reactions
• 关系 (Relationship): Connections between entities

Adding Attributes

Attributes provide structured metadata:

{
  "人物": "张三",
  "人物_attributes": {
    "role": "工程师",
    "department": "研发部"
  }
}

Relationships

Model relationships between entities:

{
  "关系": "张三是李四的经理",
  "关系_attributes": {
    "type": "管理关系",
    "source": "张三",
    "target": "李四",
    "direction": "manages"
  }
}

Workflow

Step 1: Understand Requirements

Ask clarifying questions:

• What entity types to extract?
• What attributes are important?
• Any relationships to capture?
• Domain-specific requirements?

Step 2: Create Example

Build ONE high-quality example showing:

• All entity types
• Attribute structure
• Relationship format (if needed)

Example text should be similar to target text (same domain, style, length).

Step 3: Build Prompt

[Optional: Brief instruction]

Examples
Q: [Example text]
A: [Example JSON with extractions]

Q: [Target text - can be chunked if long]
A:

Key principles:

• Keep instruction minimal or omit entirely
• One example is usually enough
• Example quality > quantity
• Match example domain to target domain

Step 4: Process Response

1. Parse JSON: Extract the extractions array
2. Align to source: Find each extraction in original text
3. Record positions: Store character offsets (start, end)
4. Handle chunks: If text was split, merge results

Step 5: Present Results

Format options:

• Grouped by type: List all people, then all events, etc.
• With attributes: Show entity + its metadata
• With positions: Link back to source text
• Visualization: Generate HTML with highlights

Advanced Patterns

Long Documents

For text > 3000 chars, split into chunks:

1. Chunk with overlap: Keep 200-300 chars overlap between chunks
2. Add context: Include previous chunk tail in prompt
3. Process in parallel: Send multiple chunks simultaneously
4. Merge results: Deduplicate overlapping extractions

Context injection:

[Previous text]: ...end of previous chunk...

Examples
Q: [Example]
A: [Example JSON]

Q: [Current chunk]
A:

Multi-pass Extraction

For higher recall, run extraction multiple times:

1. First pass: Extract obvious entities
2. Second pass: Extract with different temperature/prompt
3. Merge: Keep non-overlapping extractions from both passes

Domain-Specific Extraction

Medical records:

{
  "症状": "头痛",
  "症状_attributes": {
    "severity": "中度",
    "duration": "3天",
    "frequency": "间歇性"
  }
}

Legal documents:

{
  "条款": "违约责任",
  "条款_attributes": {
    "section": "第5条",
    "type": "义务性条款",
    "parties": ["甲方", "乙方"]
  }
}

Social media:

{
  "话题": "#AI技术",
  "话题_attributes": {
    "sentiment": "正面",
    "engagement": "高"
  }
}

Alignment Algorithm

After extraction, map each entity back to source text:

Exact Match

# Find exact text in source
position = source_text.find(extraction_text)
if position >= 0:
    char_interval = (position, position + len(extraction_text))

Fuzzy Match

When exact match fails:

1. Tokenize both texts
2. Find longest common subsequence
3. Calculate overlap ratio
4. Accept if ratio > 0.75

Handle Failures

• Mark as "not aligned" if no match found
• Still include in results (extraction is valid even without position)

Common Patterns

Pattern 1: Simple Entity List

Use case: Extract names, places, dates

Example:

{
  "extractions": [
    {"人物": "Alice", "人物_attributes": {}},
    {"地点": "Paris", "地点_attributes": {}},
    {"时间": "2024-01-15", "时间_attributes": {}}
  ]
}

Pattern 2: Rich Attributes

Use case: Detailed entity metadata

Example:

{
  "extractions": [
    {
      "产品": "iPhone 15",
      "产品_attributes": {
        "category": "手机",
        "price": "5999",
        "features": ["A17芯片", "钛金属边框"]
      }
    }
  ]
}

Pattern 3: Knowledge Graph

Use case: Build entity relationships

Example:

{
  "extractions": [
    {"人物": "张三", "人物_attributes": {"role": "CEO"}},
    {"人物": "李四", "人物_attributes": {"role": "CTO"}},
    {
      "关系": "张三管理李四",
      "关系_attributes": {
        "type": "reports_to",
        "source": "李四",
        "target": "张三"
      }
    }
  ]
}

Tips

Prompt Design

✅ Do:

• Use one clear example
• Match example to target domain
• Keep instructions minimal
• Show exact JSON format

❌ Don't:

• Write long instructions
• Use multiple conflicting examples
• Over-explain the task
• Add unnecessary fields

Example Quality

Good example:

• Realistic text from target domain
• Shows all entity types
• Demonstrates attribute structure
• Similar length to target

Bad example:

• Generic/artificial text
• Missing entity types
• Inconsistent format
• Wrong domain

Attribute Design

Structured attributes (preferred):

"人物_attributes": {
  "age": "30",
  "occupation": "engineer"
}

Unstructured attributes (avoid):

"人物_attributes": {
  "info": "30 years old engineer"
}

Error Handling

• Invalid JSON: Retry with clearer example
• Missing entities: Try multi-pass extraction
• Wrong format: Check example matches desired output
• Hallucinations: Verify extractions exist in source text

Output Formats

Console Display

提取结果:
============================================================

【人物】(3 个)
------------------------------------------------------------
  • 张三
    描述: 项目经理
    属性: {"department": "研发部"}

【事件】(2 个)
------------------------------------------------------------
  • 项目启动会
    描述: 讨论Q1目标

JSON Export

{
  "document_id": "doc_001",
  "text": "原始文本...",
  "extractions": [
    {
      "extraction_class": "人物",
      "extraction_text": "张三",
      "char_interval": {"start": 10, "end": 12},
      "description": "项目经理",
      "attributes": {"department": "研发部"}
    }
  ]
}

HTML Visualization

Generate interactive HTML with:

• Highlighted entities (color-coded by type)
• Hover tooltips showing attributes
• Click to see details
• Filter by entity type

Reference

See EXAMPLES.md for complete working examples across different domains.