| name | knowledge-extraction |
| description | Extract structured knowledge (summary, compiled truth, tags, temporal range) from text chunks during ingest Step 2. Uses domain-specific references that customize the output schema and prompt. The prompt is composed from the reference's system prompt plus the tag hierarchy injected into the user message. |
| license | MIT |
| compatibility | Requires the TypeScript ingest extraction gateway (`packages/ingest-ai`) with `@earendil-works/pi-ai` tool-call support. |
| metadata | {"author":"Hello-Wiki","version":"0.2.0","stage":"mvp"} |
Knowledge Extraction Skill
Extract structured knowledge from a single text chunk. Invoked once per chunk during the ingest pipeline (Text → Structured JSON). The TypeScript gateway builds pi-ai context and requires the model to call emit_extracted_knowledge. The output fills schema.sql columns that require LLM population: raw_chunks.summary, pages.title, pages.compiled_truth, raw_chunks.effective_range / pages.effective_range, and tags (via suggested_tags → page_tags). raw_chunks.extra_metadata is populated by the parser/persistence layer with file metadata — the LLM does not touch it.
Prompt Composition
The LLM call uses a two-message structure:
┌────────────────────────────────────────────────┐
│ SYSTEM PROMPT (reference/prompt.md) │
│ │
│ • Domain knowledge & extraction rules │
│ • Output format specification (JSON example │
│ matching the template.json schema) │
│ • Guardrails │
│ • Context placeholders filled at runtime: │
│ {source_document}, {source_page}, │
│ {chunk_index}, {total_chunks} │
├────────────────────────────────────────────────┤
│ USER PROMPT (built at runtime) │
│ │
│ AVAILABLE TAGS │
│ ────────────── │
│ {indented tag tree from serialize_tag_tree()} │
│ │
│ TEXT TO ANALYZE │
│ ──────────────── │
│ {chunk_text} │
└────────────────────────────────────────────────┘
System prompt: the preset rules from the selected reference's prompt.md. Contains extraction rules and an output format example. Context variables ({source_document}, {source_page}, {chunk_index}, {total_chunks}) are injected at runtime.
User prompt: the tag tree (queried from SELECT name, label, level, is_leaf, path FROM tags ORDER BY path and serialized with serialize_tag_tree()) + the chunk text.
Structured output: packages/ingest-ai exposes a TypeBox schema as the emit_extracted_knowledge tool and validates the model tool call before returning JSON to Python. The model MUST call that tool with valid arguments matching the schema.
Domain Reference Selection
The skill resolves the reference by matching domain against references/index.yaml:
| Reference | When |
|---|
general | Mixed-content or unspecified domain (default fallback) |
university_policy | Higher education administration documents |
Selection logic: match domain → reference id. No match → fallback to default: true.
Each reference provides:
template.json — JSON Schema documentation for the extraction output contract. The runtime TypeScript schema in packages/ingest-ai enforces the shared fields used by Python persistence.prompt.md — System prompt with domain-specific extraction rules.
Output Fields
| Field | Required | Schema.sql Target |
|---|
chunk_summary | ✅ | raw_chunks.summary — also shared to page_timeline.source_description |
page_title | ✅ | pages.title — trigram fuzzy search index |
compiled_truth | ✅ | pages.compiled_truth — primary vector search + QA |
suggested_tags[] | ✅ (2–6) | tags table + page_tags junction |
effective_range{} | ✅ | raw_chunks.effective_range + pages.effective_range |
Mapping to Schema.sql Tables
chunk_summary ──→ raw_chunks.summary ──→ page_timeline.source_description (copy)
page_title ──→ pages.title
compiled_truth ──→ pages.compiled_truth ──→ page_versions.compiled_truth (snapshot)
suggested_tags ──→ tags table ──→ page_tags.page_id + page_tags.tag_id
effective_range──→ raw_chunks.effective_range (tstzrange)
──→ pages.effective_range (tstzrange)
Guardrails
- No fabrication: all 5 fields are REQUIRED. Use
null for unknown dates in effective_range, but always provide a description and stale_risk assessment. Select at least 2 tags even for ambiguous content — prefer unknown/general tags over fabrication
- No evaluation: extract and synthesize facts as written — do not judge correctness
- Single-chunk scope: only extract from THIS chunk
- Language follows source: output in the same language as the input
- Tag tree is authoritative: use tag names from the injected hierarchy. Propose new leaves only when genuinely needed
