verse-topic-tagger

name

verse-topic-tagger

description

Analyse a Bible verse's topic areas and link verses to topics in Raamattu Nyt (bible_schema.topical_references — the jae↔aihe link table). Use for: (1) tagging a single verse/passage — given a reference, find/assign its topics and insert topical_references; (2) per-book/chapter COVERAGE scans for under-tagged verses; (3) junk/QA CLEANUP of bad references (over-wide range refs, cluster bleed, wrong scope); (4) SOURCE-based rematch from easton_articles / smiths_articles to topical_references. Triggers: "tägää jae", "liitä jae aiheeseen", "jakeen aihealueet", "analysoi jakeen aiheet", "aihekattavuus", "topical_references", "roskaviite", "junk ref", "cluster bleed", "coverage gap", "Easton/Smith rematch", "verse topic tagging". Also owns (5) the VERSE SCHEMA (jae-skeema) — Matthew Henry turned into a structured Finnish exposition per outline point, and the verification of its claims against the bounded English source; triggers: "jae-skeema", "jakeiden skeema", "Matthew Henry", "rakenteinen suomennos", "structured_adaptation", "commentary adaptation", "outline point", "commentary_sections", "claim_source_checks", "väitteen vahvistus", "claim verification", "verify-claim-source". NOT for topic CRUD / taxonomy / translations / aliases / merges → use `topic-manager`.

Verse Topic Tagger

Analyse a verse's topic areas and manage the jae↔aihe links in bible_schema.topical_references. This skill is verse-first (given a verse → its topics). For topic-first work (create/edit topics, translations, aliases, taxonomy, merges) use topic-manager — it owns topical_topics and its verse-attach INSERT patterns; this skill reuses that convention and adds the verse-side analysis, coverage, cleanup and source-rematch workflows.

All DB work goes through the Supabase MCP (mcp__plugin_supabase_supabase__execute_sql for reads/DML, apply_migration only for DDL). Project id: iryqgmjauybluwnqhxbg.

Read first

• Docs/context/topic-domains.md — 17 category roots + what each root "takes" (decide which root a candidate topic belongs to).
• references/learnings.md (this skill) — load-bearing gotchas. Read before any INSERT/DELETE.

Data model (the link table)

bible_schema.topical_references — one row = "topic T applies to verse/range R":

col	meaning
topic_id uuid	FK → topical_topics.id
osis_start text	start verse, canonical OSIS e.g. Rom.8.28 (single verse = start only, osis_end NULL)
osis_end text	end verse for a range (NULL = single verse)
relevance_score int	1–5; 5 = core proof verse, 1 = weak/tangential. Drives ordering + display cap.
source text	provenance tag (see below). Legacy bulk rows are NULL.

Reads / analysis:

• get_topics_for_verse(p_osis, p_limit, p_near_window, p_version_code) → a verse's topics with scope (own/near/passage), is_core, relevance. Use to SEE what a verse already has before adding.
• get_topics_for_osis_refs(p_osis_refs text[]) → {osis_ref, topic_id, topic_name, topic_slug, topic_slug_fi} for a batch (matches osis_start = ANY).
• Backup of curated refs: bible_schema._topical_refs_curation_backup.

Source tag convention (set source on every row you insert — makes work auditable + reversible):

• Coverage scans: coverage-gap-scan/YYYY-MM/<OsisBook><Chapter> (e.g. coverage-gap-scan/2026-07/2Sam14).
• One-off manual tagging: manual.
• Source rematch: easton-rematch/YYYY-MM / smiths-rematch/YYYY-MM.

Non-negotiable invariants (details in references/learnings.md)

1. NEVER guess OSIS. Verify every ref exists before inserting — bad OSIS is silently never displayed (resolution is query-time). Book codes match verse_keys (John, 2Sam, 1Cor, Titus, Ps…), NOT Finnish/URL names.

   SELECT osis FROM bible_schema.verse_keys WHERE osis IN ('Rom.8.28','Rom.8.29');
   

2. Dedup / delete key = the TRIPLE (topic_id, osis_start, COALESCE(osis_end, osis_start)) — matches the unique index. Deleting/checking on osis_start alone is wrong.
3. Never compare an OSIS range as text. 'John.3.2' < 'John.3.21' is a string lie. Compare numerically (book, chapter, verse) when deciding containment.
4. qa_status='ok' on the topic does NOT mean its refs are good. Ref quality is independent — judge each ref on the verse text.
5. Set relevance_score deliberately (5 core → 1 tangential); don't flat-score — and don't score in isolation either. Read the topic's existing top refs first, and reserve 5 for single-verse refs (range ≤ 4, 10+ verses ≤ 3). See "Scoring is a TOPIC-side judgement".
6. Single verse → osis_start only, osis_end NULL. Range → both. Prefer the tightest range that the verse text actually supports.

Workflow selection

Task	Section
Tag one verse/passage	§1 below (happy path)
Scan a book/chapter for under-tagged verses	references/coverage-scan.md
Clean junk refs (wide ranges, cluster bleed, wrong scope)	references/junk-cleanup.md
Rematch refs from Easton/Smith articles	references/source-rematch.md
Verse schema — Matthew Henry as a structured FI exposition, its claims + verification	references/verse-schema.md

§1 — Tag a single verse / passage (happy path)

1. Resolve + verify the reference to OSIS (verse_keys; §invariant 1). Read the verse text so tagging is grounded, not guessed:

   SELECT vk.osis, v.text FROM bible_schema.verse_keys vk
   JOIN bible_schema.verses v ON v.verse_key_id = vk.id
   WHERE vk.osis = 'Rom.8.28';
   

   (If the join columns differ in your schema snapshot, confirm with Docs/context/db-schema-short.md; the point is: read the text first.)

2. See what it already has — avoid duplicates and understand gaps:

   SELECT * FROM bible_schema.get_topics_for_verse('Rom.8.28', 30, 4, 'finstlk201');
   

   scope='own' = tagged directly; near/passage = inherited from a nearby / range ref, NOT a direct tag on this verse.

3. Read the FI verse schema for the passage — and say so if it is missing.

   SELECT title_fi, summary_fi, key_lessons_fi, editorial_status
   FROM bible_schema.get_commentary_adaptation_for_verse('John.6.35', 'finstlk201', 'mhc', 'fi');
   

   • Present: treat key_lessons_fi as a CHECKLIST, not inspiration. Walk it line by line; each lesson must end up either (a) tagged to a topic or (b) written down as a taxonomy gap. Measured on John 6–7: 11 lessons, 8 tagged, 3 missed whose topics existed already (Yhteys Kristuksen kanssa, kirkastaminen, jaot) — the miss came from reading the schema as inspiration.
   • Missing (the common case — 108 adaptations exist out of 5 360 sections): in the answer, e.g. Never let its absence pass silently: the user cannot tell from the result whether the schema was consulted or did not exist. Offer to queue generation (, RPC ), and continue the tagging without it.

Scope & relevance guidance

• own vs range spill: a broad range ref (osis_start..osis_end) makes the topic appear as near/passage on every inner verse. Don't add a whole-chapter range to force one verse's topic — tag the specific verse (own) instead. Wide ranges are the main junk source (references/junk-cleanup.md).
• relevance_score: 5 = the verse is a primary proof-text of the topic; 3 = clearly relevant; 1 = mentioned/tangential. Core topics with high scores surface first in the app.

Scoring is a TOPIC-side judgement — anchor it, don't invent it

The score answers "how central is this verse to the topic", and it is read in the topic → verses direction: get_topic_preview orders refs by relevance_score DESC and then truncates. A score set while looking only at one verse therefore decides what the topic page shows, sight unseen.

Scoring verse-at-a-time inflates the scale. Measured 2026-08-06 across the whole link table:

Source	n	avg	share ≥4
legacy (source NULL)	100 595	1.74	8.1 %
coverage-gap-scan/*	899	3.38	37.3 %
nave-figurative/*	748	3.75	78 %
manual	74	3.95	65 %

Every freshly tagged verse thus heads its topic's list ahead of essentially the entire legacy corpus — not on merit, but because nobody looked at the neighbours.

Rule 1 — read the topic's current top refs before you score. One query per topic, then place the new ref relative to what is already there:

SELECT tr.osis_start, tr.osis_end, tr.relevance_score, tr.source
FROM bible_schema.topical_references tr
JOIN bible_schema.topical_topics t ON t.id = tr.topic_id
WHERE t.slug = 'providence'
ORDER BY tr.relevance_score DESC NULLS LAST
LIMIT 10;

If the topic's existing 5s are stronger proof-texts than your verse, your verse is not a 5 — whatever it looked like in isolation. If they are weaker, that is a legacy-calibration finding: note it, don't fix it here (whole-corpus rescoring is class-2 bulk work for topic-bulk, not a per-row edit).

Rule 2 — 5 is reserved for single-verse refs. A multi-verse range caps at 4, and 10+ verses at 3. Exception: when the topic IS the passage — a named discourse or narrative whose extent is the topic (vuorisaarna = Matt 5–7, spitaali = Lev 13–14, Simson = Judg 13–16) — the range is the topic's definition, not a proof text, and the cap does not apply. Test: would the topic still mean the same thing if you replaced the range with one verse? If no, keep the score. Found 2026-08-07 while sweeping cross-chapter ranges; applying the cap blindly would have demoted the Sermon on the Mount on a technicality. The rubric says the verse is a primary proof-text; a whole passage cannot be that at the same resolution, and a range already spills onto every inner verse as near/passage. Measured 2026-08-06: 15 skill-written refs spanning 10+ verses carry a 5 — the pattern this rule stops.

Rule 3 — on a large topic, score the TOP, not everything. get_topic_preview truncates at 20, so on a 200-ref topic the only question the score answers is who is in the top 20. Lifting the 30–40 strongest verses to 3/4/5 and leaving the tail at 1 delivers the full reader-visible benefit for a fraction of the work. Measured 2026-08-07: kiusaus 211 refs → 11×5, 21×4, 12×3, 167×1; sävyisyys 119 → 9/16/11. Find the top by knowing the topic's classic proof-texts, not by reading 200 verses (kiusaus: Jas 1:13–14, 1Cor 10:13, Matt 6:13, Matt 26:41, Heb 4:15).

⚠️ Do NOT stamp the untouched tail with a source tag to mark it "reviewed". source IS NULL is the legacy marker that the calibration measurement depends on (99.3 % of the flat-topic queue is legacy — that number is how the work is scoped). Overwriting it on tail rows would destroy the signal to record something the level-count detector already handles: once a topic has 3+ distinct scores it drops out of the flat list by itself. Record completed topics in the task note instead.

Where the score does NOT decide much: the verse page. get_topics_for_verse orders scope → is_core → is_biblical → combined_score, so relevance is only a late tiebreaker there (it still truncates via p_limit). Don't reason about scoring from what you see on the verse page.

Verse schema (jae-skeema) — the second half of this domain

Beyond "which topics does this verse carry", this skill owns the verse schema: Matthew Henry's commentary turned into a structured, edited Finnish exposition (structured_adaptation, NOT a translation) written per outline point, so every Finnish sentence traces back to a bounded slice of the English source. It is the verse-side twin of the topic schema (topical_topics.structured_schema, owned by topic-manager) and it reuses that system's verification machinery: claims are judged against Henry with the same verdict vocabulary and land in the same admin inbox /ohjaamo/topics/claims; drafts are generated/reviewed the way /ohjaamo/topics/generation does it.

Four things to know before touching it — full detail in references/verse-schema.md:

1. The adaptation layer SHIPPED (this bullet used to say it was unbuilt). Live: commentary_adaptations (+_points, _applications, _checks, _jobs), edge functions adapt-commentary / verify-adaptation, reader RPC get_commentary_adaptation_for_verse, ohjaamo /ohjaamo/commentary. Published in two curation levels (ai = AI-structured, reader sees a warning; curated = an editor checked it) — see references/verse-schema.md §8b. Check the DB before quoting numbers.
2. The unit is the section + outline point, never the verse (981 NT sections = 981 model calls; verse-at-a-time is 10–15× the cost).
3. Grouping rule: an outline row without ordinal_text merges into the previous numbered point (25.1 % of rows) — 1:1 generation produces duplicate points (Ps. 23 is the canonical trap).
4. Never write *_fi into commentary_sections / commentary_outline_points — they are re-runnable source data; a reseed would wipe edited text.

After significant tagging/cleanup

• Invoke /opi to capture any new gotcha (canonical home + fan-out).
• If a batch is large or a new convention emerges, tell docs-updater to note it (manifest topics-domain invariants live there).