ingest-content

name	ingest-content
description	use when: saving a single URL (article, YouTube video, blog post, documentation page) to open-brain memory as curated_content with provenance metadata. NOT for: bulk import of multiple URLs (run once per URL); local files (use `summarize ./file` + `ob save` manually); JSONL or Markdown batch import (use ob-migrate). boundary: ingest-content is one-URL-at-a-time with content-type detection; ob-migrate is for batch/interactive memory writes from files or prior context.
version	0.2.0
requires_standards	["open-brain/cli-routing","open-brain/memory-write-patterns"]
requires	["standard:english-only","mcp:open-brain"]

ingest-content

Extract content from a URL and save it to open-brain as a curated_content memory with full provenance metadata.

Step 1 — Validate URL

Input must start with http:// or https://. Otherwise report Error: "<input>" is not a valid URL. Provide a full URL starting with https://. and stop.

Step 2 — Check for duplicates

Per the write-patterns standard §8, search before extracting:

ob search "<URL>" --json

If a memory with matching source_url exists, ask the user:

This URL was already ingested (Memory ID: <id>, saved <date>).
What would you like to do?
  [u] Update existing memory
  [s] Skip (keep existing)
  [d] Save as duplicate anyway

Default to skip on no response.

Step 3 — Detect content type

URL pattern	content_type	Notes
youtube.com, youtu.be	video	Transcript via summarize
vimeo.com	video	No transcript — metadata fallback only
*.substack.com, medium.com, blogs	article
docs.*, *.readthedocs.io, developer.*	doc
Everything else	article

Step 4 — Extract

content_type	Primary	Fallback
video (YouTube)	summarize "URL"	summarize --json "URL" (metadata only)
video (Vimeo)	crwl crawl "URL" -o md	none
article / doc	summarize --extract --format md "URL"	crwl crawl "URL" -o md
JS-heavy SPA	crwl crawl "URL" -o md	none

Try primary first; on error or empty output, try fallback.

Content security

Extracted content is saved to open-brain storage only — NOT re-injected as agent instructions in this run. The content-processor pattern (standards/security/content-isolation.md) applies when retrieved content later feeds agent prompts. If you read this content back via ob-search and build instructions from it, route through content-processor then.

Step 5 — Derive fields

• title — page/video title or first H1; fallback to URL path segment
• text — full extracted markdown; for videos, the summary
• preview_only = true if any:
  • content_type=article AND text < 300 chars
  • text contains "paid subscribers", "sign in to continue", "subscribe to read", "members only", "upgrade to read"
  • extractor returned HTTP 401/402/403 or a login-redirect page

Get today's date via date +%Y-%m-%d.

Step 6 — Save

Per the write-patterns standard, call save_memory (MCP form shown — metadata is a JSON string, not an object):

title: <derived>
text: <extracted content>
type: curated_content
project: <user-specified or omit>
metadata: "{\"source_url\":\"<URL>\",\"extraction_date\":\"YYYY-MM-DD\",\"content_type\":\"video|article|doc\",\"preview_only\":false}"
provenance: {"producer":"ingest-content","source_ref":"url:<URL>"}

Step 7 — Confirm

Saved to open-brain:
- Title: <title>
- Type: <content_type>
- Memory ID: <id>
- Preview only: <yes/no>

Find it later with: /ob-search <keyword>

Error Handling

Situation	Action
404 / unreachable URL	Report error, do NOT save
Video without transcript	Fallback to summarize --json → metadata-only save
Paywall / login wall	Save whatever preview is available with preview_only: true
Both primary + fallback fail	Report both outputs, do NOT save partial data
save_memory fails	Report open-brain MCP unavailable — content was not saved. Retry with /ingest-content <URL>

Out of Scope

• Batch ingestion: one URL at a time — use ob-migrate for files
• Authenticated sites: use crwl with a saved profile manually
• Local files: use summarize ./file then ob save directly
• Update logic: when user chooses [u], re-run extraction and call save_memory again — open-brain creates a new version; the old is not deleted