subtext-annotate

name

subtext-annotate

description

Convert raw text (emails, meeting notes, markdown documents, pasted prose) into a subtext JSON document — a sentence-level document tree where every sentence has a stable ID and metadata — then interview the user with clarifying questions about anything a reader might find unclear, attaching the answers to the exact sentences they clarify. Use when the user wants to "subtext" a document, structure text at the sentence level, or capture clarifications and missing context alongside a document.

subtext annotate

Turn a chunk of text into a subtext document, then enrich it by asking the user clarifying questions — things in the document that would be unclear to a reader — and recording the answers as sentence-level metadata. The JSON schema, parsing rules, and a worked example live in references/format.md — read that file before producing any JSON. The final artifact is delivered in a columnar encoding described in references/columnar.md, read at Step 4.

Workflow

Step 1 — Normalize to CommonMark

Convert the input to clean CommonMark markdown before structuring it:

• Preserve existing markdown structure (headings, lists, tables, code fences) as-is.
• For unstructured text (emails, plain prose), infer light structure: a # title, ## headings for clear topic shifts, lists where the text enumerates. Do not invent structure that isn't implied by the content.
• Convert inline links and bracketed references to [[ref-id]] citation markers and collect each URL into the sources map (see format.md).
• Normalize curly quotes to straight quotes and non-breaking spaces to regular spaces.
• If display math ($$ … $$) appears, emit it as a code block with language latex. Math handling is otherwise out of scope.

Show the normalized markdown to the user only if the input required significant restructuring; otherwise proceed.

Step 2 — Build the document tree

Convert the normalized markdown into subtext JSON following the rules in references/format.md. Key points:

• First # heading becomes metadata.title; remaining headings become nested sections.
• Split paragraphs and list items into individual sentences with sequential IDs (s-0001, s-0002, …) in reading order across the whole document.
• Strip citation markers out of sentence text into each sentence's citations list.
• Leave provenance fields (authored_by, authored_at, …) absent. The document already exists; its authorship history is usually unknown. Only stamp provenance if the user volunteers it.

Step 3 — Clarification interview

Read the document as a skeptical first-time reader with no shared context, and find the places where that reader would stumble. Look for:

• Undefined acronyms, jargon, or codenames — "the WER regression", "Project Halcyon"
• Ambiguous references — "he approved it", "the previous approach", "as discussed"
• Relative or vague time — "next quarter", "soon", "by EOW" (relative to when?)
• Unidentified people or teams — who is "Sam"? which team owns this?
• Vague quantities or claims — "significantly faster", "most customers", numbers with no baseline or source
• Unstated assumptions or missing context — decisions referenced but not explained, steps that assume knowledge the reader lacks
• Unclear ownership or next steps — actions with no actor or deadline

Then interview the user. Prioritize questions by how much each answer would help a reader, and ask in batches — never one question per message.

If a structured user-input tool is available (e.g. AskUserQuestion), prefer it over asking in prose:

• Batch up to 4 questions per call; run a second call rather than overflowing one.
• Quote the sentence in each question text, e.g. "Sam approved the rollout." — Who is Sam?
• Where plausible interpretations exist, offer them as 2–4 options (e.g. for "next quarter": Q3 2026, Q4 2026); the built-in "Other" covers free-text answers. For questions with no guessable options, offer Skip — leave as open question plus a best-guess option, and let "Other" carry the real answer.
• Treat a skipped/declined question as unanswered (it becomes an open question, below).

Otherwise, interview in chat:

• Group questions by section; cap a batch at 5–8 and offer a second round only if the document warrants it.
• Anchor every question by quoting the sentence text (and naming the section when it helps).
• Number the questions so the user can answer any subset tersely ("1: Sam Okafor, VP Eng; 3: skip").
• Accept "looks clear" / "skip the rest" at any point and move on.

Either way: if a question applies to the whole document (intended audience, purpose, the date "today" refers to), ask it once as a document-level question, not per sentence. Never show internal sentence IDs (s-NNNN) to the user — they are bookkeeping for the JSON, not something the user needs to see; track the sentence each question targets yourself.

Record the answers:

• Sentence-level answers → append to that sentence's meta.clarifications as {"question": "...", "answer": "..."}.
• Document-level answers → keys in metadata (e.g. audience, purpose, context_date).
• Questions the user skipped or couldn't answer → collect as {"target_id": "s-NNNN", "question": "..."} entries in metadata.open_questions, so future readers see what remains unresolved. Drop questions the user dismissed as already clear.
• Do NOT rewrite sentence text with the answers — the text stays verbatim; clarifications live alongside it. If the user explicitly asks for the clarifications to be worked into the prose, do that as a separate revision after delivering the annotated document.

Step 4 — Deliver

1. Validate the nested tree: IDs are sequential with no gaps, absent optional fields are omitted (not null), empty meta dicts are omitted.
2. Read references/columnar.md and re-encode the tree into the contextdoc/columnar-v1 format — the delivery format. Its embedded $legend teaches any agent that later consumes the file how to read it with no other context. Run that file's validation checklist.
3. Output the columnar JSON (2-space indent) as a downloadable artifact named <kebab-case-title>.subtext.json. If the user asks for the plain nested tree instead, deliver that per format.md.
4. Summarize in one or two sentences: section count, sentence count, how many clarifications were captured, and how many open questions remain (e.g. "31 sentences, 6 clarifications attached, 2 open questions").