Expert guidance on LLM prompting patterns used in this project. Covers versioned prompts, schema enforcement, category calibration, multi-provider fallback, and narrative generation. Activates when working on prompts, LLM integration, or evaluation logic.
Expert guidance on LLM prompting patterns used in this project. Covers versioned prompts, schema enforcement, category calibration, multi-provider fallback, and narrative generation. Activates when working on prompts, LLM integration, or evaluation logic.
LLM Prompting Expert
Deep expertise on the prompting infrastructure, patterns, and conventions used in this codebase.
When This Skill Activates
Writing or modifying prompts in data-pipeline/src/llm/prompts/
Working on narrative generation (baseline.py, rich_phase.py, src/services/rich_narrative_generator.py) or the judges in src/judges/
Debugging LLM outputs or schema validation
Adding new LLM-powered features
Optimizing prompt performance or cost
Prompt Infrastructure
Versioned Prompt System
Location: data-pipeline/src/llm/prompts/
Format: All prompts use frontmatter with version tracking:
Rule: Content changes without version bumps trigger warnings/errors.
Prompt Files
File
Version
Purpose
baseline_narrative.txt
v2.0.0
Baseline narrative with inline citations
rich_narrative_v2.txt
v3.1.0
Investment-memo rich narrative
rich_strategic_narrative.txt
v1.0.0
Deep strategic analysis narrative
charity_navigator_financials.txt
v1.0.0
CN financial extraction
categories/*.txt
varies
16 category calibrations
Versions drift — trust the file frontmatter (head -4 <file>), not this table.
LLM Client Architecture
File: data-pipeline/src/llm/llm_client.py
Multi-provider via LiteLLM. Models, prices, and fallback chains are
defined in MODEL_REGISTRY and TASK_MODELS in llm_client.py — check
there; doc snapshots of model names go stale. Task selection is via the
LLMTask enum (NARRATIVE_GENERATION, WEBSITE_EXTRACTION,
PDF_EXTRACTION, PREMIUM_NARRATIVE, PREMIUM_PDF_EXTRACTION,
EVALUATION_SCORING, RICH_STRATEGIC_NARRATIVE, LLM_JUDGE).
Fallback logic:
Transient errors (429, 503, rate limits): Try fallback model
## WORLD-CLASS BENCHMARKS
- MSF (Doctors Without Borders)
- ICRC (International Committee of the Red Cross)
- UNHCR
- World Food Programme
- Partners in Health
## SCORING CALIBRATION FOR HUMANITARIAN ORGS
### Systemic Leverage (0-30)
26-30: International treaty influence, paradigm-shifting models
20-25: Operating at scale ($100M+), policy influence
...
How it works:
category_classifier.py maps EIN → category via config/charity_categories.yaml
NarrativeEvaluator inserts category prompt before charity data
LLM calibrates scores against domain-specific benchmarks
Pattern: Trust what the charity claims, don't make independent rulings.
ZAKAT_ELIGIBLE: Charity explicitly claims zakat eligibility on website
SADAQAH_STRATEGIC: No zakat claim BUT tier_1_strategic_fit.subtotal > 35
SADAQAH_ONLY: No zakat claim AND tier_1_strategic_fit.subtotal <= 35
8 Asnaf embedded in prompt:
Al-Fuqara (the poor)
Al-Masakin (the destitute)
Al-Amileen (zakat administrators)
Al-Muallafatul Quloob (hearts to be reconciled)
Fi Al-Riqab (freeing from bondage)
Al-Gharimeen (those in debt)
Fi Sabilillah (in Allah's cause)
Ibn Al-Sabil (stranded travelers)
Required disclaimer: "This analysis is informational only and does NOT constitute a fatwa."
Important: Judge provides quality metrics only, not routing decisions.
Evidence Citation Pattern
Every claim requires structured evidence:
classEvidence(BaseModel):
claim: str# The claim being made
source: str# "Form 990", "Charity Navigator", "Website"
source_year: int# Year of data
field: str# Specific field from source
value: str# Actual value
confidence: str# "HIGH", "MEDIUM", "LOW"
Prompt principle: "Every claim MUST cite source and year."
Retry & Validation
Generation retry loop (max 3 attempts):
for attempt inrange(MAX_GENERATION_RETRIES):
response = llm_client.generate(prompt, json_schema=schema)
# Validation checks:
narrative = BaselineNarrative.model_validate_json(response)
validate_sub_score_sums(narrative)
density = calculate_information_density(narrative)
if density >= 0.80:
return narrative
# Retry with feedback
prompt = add_density_feedback(prompt, density)
Information density check:
Counts populated fields vs total schema fields
Threshold: 0.80 (80% of fields must be populated)
Below threshold → human review or retry
Cost Tracking
Every LLM call logs:
classLLMResponse:
model_version: str
prompt_version: str
prompt_hash: str
db_snapshot_version: str
timestamp: datetime
prompt_tokens: int
completion_tokens: int
cost_usd: float
Model registry in llm_client.py includes pricing for 30+ models.
Prompt Engineering Patterns
Pattern
How It's Used
Chain-of-thought
Each dimension has narrative field explaining reasoning
Scoring rubrics
Detailed 5-7 level tables embedded in prompts
Few-shot examples
Real org comparisons (MSF, Water.org, ACLU)
Low temperature
temperature=0.3 for consistent output
Schema enforcement
JSON mode ensures exact structure
Category calibration
Domain-specific benchmarks injected
Self-assertion
Zakat based on charity's claim, not judgment
Deterministic fallback
Critical scores calculated outside LLM
Key Files Reference
File
Purpose
prompt_loader.py
Load/validate/version prompts
llm_client.py
Unified LLM interface + model registry
src/services/rich_narrative_generator.py
Narrative generation with schema enforcement
src/judges/
LLM-as-judge quality gating (judge_phase.py)
baseline_narrative.txt
Main prompt (v2.0.0)
categories/*.txt
16 category calibrations
schemas/baseline.py
BaselineNarrative Pydantic model
schemas/judge.py
JudgeResult model
Anti-Patterns
Don't:
Change prompt content without bumping version
Let LLM determine scores that should be deterministic
Skip schema validation
Use high temperature for scoring tasks
Embed specific charity data in prompt templates
Make fatwa-like zakat rulings (use self-assertion)
