| name | content-ingestion |
| description | Ingest URLs, YouTube videos, and local files (PDF/DOCX/etc) into the Daemion knowledge wiki. Triggers on "ingest", "add this", "remember this article/video". CRITICAL — extraction writes to raw/ only; you are NOT done until you READ COLD, FLAG CONTRADICTIONS with the existing wiki, propose a DECIDED compile plan, and compile with the user via knowledge-wiki skill. Silently overwriting existing claims or reporting "ingested successfully" after extraction alone is a skill violation. |
Content Ingestion
You are the wiki's librarian. You decide structure. The user decides signal.
This skill is the INPUT PIPE. It handles extraction — turning URLs, YouTube videos, PDFs, and local files into raw markdown sources under ~/.daemion/knowledge/raw/. It does NOT compile those raw sources into the wiki.
For the compile step (raw → wiki pages with cross-references, provenance, contradiction handling), see the knowledge-wiki skill at ~/.daemion/skills/knowledge-wiki/SKILL.md.
For the operational loop that wraps both (wiki-first research, autonomous maintenance, end-of-response summary, the Linus principle), see the librarian skill at ~/.daemion/skills/librarian/SKILL.md.
The three skills work together: content-ingestion (input pipe) → knowledge-wiki (substrate) → librarian (workflow). All three are installed on every Daemion agent. When an agent says "ingest this URL," the ingest flow goes content-ingestion → librarian decides quick-vs-deep mode → knowledge-wiki primitives write the compile result → lint runs → end-of-response summary.
⚠️ Extraction is step 1 of 2 — READ THIS FIRST
The biggest failure mode of this skill is an agent writing a file to raw/ and reporting success. That is a lie. Content in raw/ is not in the wiki. It's in a staging area waiting for compile. If you stop at extraction and say "ingested successfully," you've broken the user's trust and the wiki's integrity.
After extraction, your task is NOT done. You MUST:
-
READ COLD. Read the extracted file with NO reference to existing wiki pages. Extract 8–15 distinct idea clusters per substantive source (>5K words). Do NOT start from "what does this connect to in the wiki?" — that filters out everything novel. Read the source on its own terms first. See read-cold-map-warm below.
-
MAP WARM. Only after the flat list is written, compare against existing wiki pages. Classify each cluster as: extends-existing, new-page, contradicts-existing, or archive-only. Contradictions are first-class output — see Flagging Contradictions below. If you find one, you MUST surface it in the compile plan, not quietly pick a side.
-
DECIDE placement yourself. Do not ask the user "where should this go?" or "new folder or existing?" You are the librarian. Those are your calls. The user decides SIGNAL (what's worth keeping). You decide STRUCTURE (where it lives). See decide-vs-ask below.
-
DEFAULT to extending existing pages. A 2-hour podcast should produce AT MOST 3 new pages. Most clusters fold into existing pages as new sections. See the new-page-vs-new-section heuristic below.
-
PROPOSE a decided plan. "Here's what I'll write: [list of pages + updates]. Stop me if wrong." Not "should I?" Not "where should X go?" A decided plan with one or two genuine tiebreakers at most.
-
WRITE the pages via the knowledge-wiki skill's compile workflow. Inline attribution to named sources. Create a people/<name>.md entity page for EVERY non-trivial human voice in the source — including hosts/interviewers, not just the primary speaker. A podcast interview produces TWO entity pages minimum (host + guest). Do NOT create pages for podcasts, newsletters, orgs, or publications prospectively — attribute to humans, not shows. See Source-entity rule below. Proper provenance. Dense wikilinks (5–15 per page).
-
LINT after every write.
-
Report the final wiki state — which pages exist now that didn't before, which existing pages got new sections. Acceptable phrasing examples:
- ✅ "Extracted to raw/ (23K words). Proposed compile plan: 2 new pages + 4 section updates. Stop me if wrong."
- ✅ "Compiled into [[page-a]], [[page-b]], and sections added to [[c]], [[d]], [[e]]. Lint clean."
- ❌ "Ingested successfully." (ambiguous — did you read it? did you compile?)
- ❌ "Added the video to the wiki." (only true after compile)
- ❌ "Done." (lazy)
If the skill body ever gets truncated below this point, the above is the minimum behavior. The frontmatter description repeats the core rule. The ingest.ts script also prints this rule in its tool output. Three layers of protection exist because this is the failure that breaks the system.
Flagging Contradictions
Contradiction detection is a required output of the MAP WARM step — not an optional side note. When a new source disagrees with what the wiki already says, that disagreement is itself high-value knowledge. Silently picking one side erases the learning. This is the single fastest way to corrupt the wiki.
The rule: if any cluster from your cold read contradicts an existing wiki page, the compile plan you propose to the user MUST list it as a contradiction, with both claims attributed and dated. Do not quietly "update" the old page. Do not quietly drop the new claim. Surface it.
What counts as a contradiction
- Direct factual disagreement — old page says "gateway uses SQLite," new source says "gateway uses Postgres"
- Reversal of a decision — old page says "we chose cron chaining," new source says "cron chaining is deprecated, use self-scheduling"
- Updated understanding — old page asserts X with confidence, new source reveals X was based on a misreading of the primary source
- Scope mismatch surfacing as conflict — old page says "always do X," new source says "never do X" (often resolvable as both-are-true under different scopes)
- Speaker-internal reversal — within the SAME source, the speaker changes their mind (early in interview said X, later said Y). Both matter.
How to flag during MAP WARM
For each cluster you classify as contradicts-existing, record:
CONTRADICTION:
new claim: <what the source says> — per [[source-slug]] ([date/timestamp])
old claim: <what the wiki currently says> — per [[existing-page]] ([capture date])
stakes: low | medium | high
suggested: ambiguous | supersede | both-scoped | escalate
reasoning: <why you're suggesting that resolution — one sentence>
- low stakes — terminology, historical minutiae, non-load-bearing fact. Safe to auto-resolve with
^[ambiguous] or a supersession note.
- medium stakes — affects a project page but not user preferences or architecture. Propose a resolution, let the user accept or override in one click.
- high stakes — architecture decisions, user preferences, project priorities, cross-agent claims. MUST escalate — do not pick a side yourself. Present both claims and ask.
How to surface in the compile plan
Contradictions appear as their OWN bullet in the decided plan, not buried inside a section update:
"Proposed compile plan:
- NEW:
topics/strategy/exponential-bet-sizing.md
- EXTEND: [[vision]] — section on "growth as product"
- ⚠️ CONTRADICTION (high stakes): Amol claims you should hire PMs early. [[project-structure]] currently says "no PMs until Series B" per your 2026-01 note. Which stands? Both? Escalating to you.
- ⚠️ CONTRADICTION (low stakes): Amol says Claude Code launched in Feb 2025; [[claude-code-timeline]] says March 2025. Suggesting
^[ambiguous] with both citations. OK?
- ARCHIVE ONLY: 2 clusters"
The user can then accept, override, or ask follow-ups — but the contradiction is visible, not hidden.
What NOT to do
- Do NOT edit the old page to match the new source without recording the old claim somewhere. Silent overwrite = confident lie.
- Do NOT drop the new claim because it's "inconvenient" or contradicts established knowledge. The disagreement IS the signal.
- Do NOT auto-resolve high-stakes contradictions. Escalate.
- Do NOT bury contradictions inside section-update bullets. They get their own line in the plan so the user can't miss them.
- Do NOT classify a cluster as
extends-existing when it actually reverses the existing claim. That's a contradiction, not an extension.
Handoff to knowledge-wiki skill
You detect and flag here. The actual resolution writing (ambiguity markers, supersession notes, cross-scope handling) happens when you WRITE the pages via the knowledge-wiki skill. That skill's "Contradiction Handling — When Sources Disagree" section defines the four resolution patterns (both-stand / supersede / both-scoped / escalate) and how they get written into the wiki. Read it before you write.
The test: if the user reads your compile plan and can't tell from it alone "is anything in this source disagreeing with what we already believe?", you failed the MAP WARM step. Go back.
Source-entity rule
Every non-trivial human voice in a source gets their own people/<name>.md entity page — not just the primary speaker. This is load-bearing for attribution quality over time.
Who counts as a "non-trivial human voice"
- The author, guest, speaker, or author — obviously
- The interviewer, host, or moderator — YES. Their framing questions are their own knowledge. "What does this host think about X across 500 interviews?" is only answerable if the host has been a first-class entity from day one.
- Named co-authors of papers or talks — each gets a page
- Anyone cited 2+ times across your existing wiki pages — if they come up repeatedly, create an entity page even if they're not in the current source
How many entity pages per compile
| Source type | Entity pages you should create |
|---|
| Podcast interview (1 host + 1 guest) | 2 — both get pages |
| Panel discussion (1 host + 3 guests) | 4 — all participants |
| Multi-author paper | one per named author |
| Blog post by single author | 1 |
| Conference talk, single speaker | 1 |
| Company blog post with no named author | 0 (attribute to the org via prose, or skip attribution) |
Common failure: compiling a podcast and creating only the guest's page because "the host was just asking questions." This loses the host's framing knowledge forever. Always create both.
What NOT to create pages for
Do NOT create pages for publications, newsletters, podcasts, shows, orgs, or companies on first ingestion. You attribute to the humans who speak for them:
- "per Amol on [[lennys-podcast-amol]]" — the podcast itself is just the raw file slug
- "per Lenny's framing question" — attribute to
[[lenny-rachitsky]], not to [[lennys-podcast]]
- "Anthropic announced X" in prose, without a wikilink — or attribute to the specific spokesperson
The three-instance rule: you may create a non-human entity page ONLY when BOTH of these are true:
- The same non-human entity is cited on 3+ wiki pages
- You find yourself wanting to write a page about the entity itself (not just link to it)
When the trigger fires: put the page in people/ (not a new folder), add entity_type: publication | org | show | newsletter to the frontmatter, scope public. One folder for all entities — don't create parallel sources/ or orgs/ taxonomies.
Integration with the decided plan
Your decided compile plan MUST include the entity pages explicitly — not hide them inside "and I'll attribute to the speaker." They are first-class deliverables:
"Proposed plan:
- ENTITY:
people/amol-avasare.md (public — Head of Growth, Anthropic)
- ENTITY:
people/lenny-rachitsky.md (public — podcast host)
- NEW:
topics/strategy/exponential-bet-sizing.md
- ..."
This makes the count visible to the user — they can challenge "why is this person getting a page?" or "why aren't you creating one for the co-host?" before you write.
Speaker Attribution — Confidence Tiers
For multi-speaker sources (interviews, podcasts, panels), you are inferring attribution from context — not reading ground truth. YouTube auto-captions, yt-dlp transcripts, and many quoted article formats don't cleanly separate speakers. You're reading context cues to guess who said what. That guess has a confidence level — mark it.
Three tiers (enforced by the knowledge-wiki skill when you write pages):
| Tier | Attribution format | When it applies |
|---|
| High | per [[speaker-name]] | Explicit speaker markers in the raw file (>>, labeled turns) AND clear content ownership cues |
| Medium | per [[speaker-name]] ^[inferred] | No explicit marker but topic/self-reference/voice makes it obvious (e.g., "my team at Anthropic," "when I joined Mercury") |
| Low | per [[raw-file-slug]] (attribute to the raw file, not a person) | Could plausibly be either speaker; rapid back-and-forth; cross-quoting; reactive fragments ("Yeah." "Right.") |
Core rule: when in doubt, drop down a tier. A wrong attribution is worse than a vague one.
What this means during MAP WARM:
- For each cluster you extract, note the confidence level of the attribution before writing it into the compile plan
- If an interview has 90% high-confidence and 10% ambiguous passages, you can still high-confidence the 90% — you don't have to drop everything to low. Tier per-claim, not per-source.
- If you find yourself writing "Amol said X but actually it might have been Lenny," STOP — that's a low-tier claim. Attribute to the raw file.
- If the source is a panel with 3+ speakers and no explicit labels, default EVERYTHING to medium at best, and most to low. Panels are attribution hazards.
Default by source type:
- Long-form 1-on-1 Q&A interview (Lenny's Podcast style) → high default
- Panel / roundtable / 3+ speakers → medium at best, mostly low
- Solo podcast, blog post, talk → doesn't apply (single author)
- Quoted fragments in articles without clear attribution → low
See the knowledge-wiki skill's "Speaker Attribution Confidence Tiers" section for the full writing rules. This skill just needs to make sure you classify each claim's confidence during MAP WARM so the writing step has accurate tier info to work with.
Purpose — Extraction Only
This skill does one thing: turn a URL or file into a plain markdown file with frontmatter, written to ~/.daemion/knowledge/raw/<category>/<date>-<slug>.md.
It does NOT:
- Compile raw into wiki pages → use
knowledge-wiki skill's compile step
- Query knowledge → use
knowledge-wiki skill's search
- Extract entities or build knowledge graphs → removed (the old Engram approach is deprecated)
- Summarize content → if the user wants a summary, fetch and summarize directly without using this skill
Think of this skill as the input pipe to the wiki, nothing more. Keep it simple. Everything downstream is another skill's job — BUT your responsibility does not end at raw/. You must complete the handoff to compile before the task is done (see above).
When To Use
Use this skill when:
- User says "ingest this", "add this to the wiki", "remember this article/video", "store this", "save this link"
- User shares multiple URLs they want retained for later
- User drops a local file path (PDF, DOCX, markdown, etc.) they want added
- User asks you to watch a YouTube video and retain it
- Batch-loading reference material for a research session
Do NOT use this skill when:
- User wants a quick summary — just fetch and summarize directly
- User wants to edit or query existing wiki knowledge — use
knowledge-wiki skill
- URL points to a binary download with no readable content (installer, zip, image-only)
- User is actively discussing an article — read it directly into context, don't ingest yet
- User wants compilation from raw to wiki pages — use
knowledge-wiki skill's compile step
The Flow
User input (URL or file path)
↓
[extract-*.sh] ← source-type detection, extract to JSON
↓
[ingest.ts] ← JSON → markdown with frontmatter + source_hash
↓
~/.daemion/knowledge/raw/<category>/<date>-<slug>.md
↓
(later, separate skill)
↓
[knowledge-wiki compile] → wiki/<scope>/<folder>/<slug>.md
This skill owns the first three steps. The fourth is explicitly another skill's responsibility.
Scripts
All scripts live in scripts/. Read references/extractors.md for when to use which.
| Script | Purpose |
|---|
scripts/ingest.ts | Main entry point. Single URL, single file, or --file batch.txt. Handles idempotency, frontmatter, write. |
scripts/extract-url.sh | Web extraction. Uses lightpanda if available, falls back to curl + HTML strip. |
scripts/extract-youtube.sh | YouTube transcript via yt-dlp. Captures metadata: title, channel, duration, chapters. |
scripts/extract-file.sh | Local file via liteparse (PDF, DOCX, XLSX, PPTX, images). Falls back to pdftotext for PDFs. |
Single Source Workflow
npx tsx ~/.daemion/skills/content-ingestion/scripts/ingest.ts "https://example.com/article"
npx tsx ~/.daemion/skills/content-ingestion/scripts/ingest.ts "https://youtube.com/watch?v=abc123"
npx tsx ~/.daemion/skills/content-ingestion/scripts/ingest.ts "~/Downloads/paper.pdf"
npx tsx ~/.daemion/skills/content-ingestion/scripts/ingest.ts --dry-run "https://example.com"
npx tsx ~/.daemion/skills/content-ingestion/scripts/ingest.ts --category videos "https://example.com/talk"
npx tsx ~/.daemion/skills/content-ingestion/scripts/ingest.ts --confidence high "https://primary-source.com"
Batch Workflow
npx tsx ~/.daemion/skills/content-ingestion/scripts/ingest.ts --file urls.txt
- Failures are per-URL — one bad URL doesn't kill the batch
- Already-ingested sources are skipped via
source_hash deduplication
- Summary reported at end: written, skipped, failed
See templates/batch-urls.txt for the expected input format.
Decision Tree — What Extractor Runs
Input
├── exists on disk OR file:// prefix → extract-file.sh (liteparse)
├── contains youtube.com OR youtu.be → extract-youtube.sh (yt-dlp)
└── any other URL → extract-url.sh (lightpanda or curl fallback)
The ingest.ts script handles this detection automatically. You only need to call the individual extractors directly when debugging or when auto-detection fails.
Category Selection
Extracted sources go into subdirectories under ~/.daemion/knowledge/raw/. See references/category-rules.md for full rules. Quick version:
| Category | For |
|---|
links/ | Web articles, blog posts, papers, documentation |
videos/ | YouTube transcripts, podcasts, video talks |
files/ | Local documents, PDFs, slide decks, images |
sessions/claude-code/ | Claude Code session transcripts (auto-written by session-end hook) |
sessions/daemion/ | Daemion thread summaries |
sessions/pi/ | Pi session extracts |
Override with --category <name> if auto-detection is wrong.
Provenance and Confidence
Every raw source carries a confidence level used downstream by knowledge-wiki lint:
high — peer-reviewed paper, official docs, primary source. Lint trusts claims.medium — default. Blog post, conference talk, well-known author.low — podcast, casual video, forum post, unverified claim. Lint flags derived claims as ^[inferred] more aggressively.
Set with --confidence high|medium|low. Without this flag, the default is medium.
Provenance frontmatter details are in references/frontmatter-schema.md.
Idempotency
Every extracted source is hashed (source_hash = SHA-256 of content, first 8 chars). Before writing, ingest.ts checks if any existing file in raw/ has the same hash. If yes → skip with a [skipped: already ingested] message.
This means:
- Re-running ingestion on the same URL is safe and cheap
- Updated content gets a new hash and becomes a new entry (both kept — let the wiki compile decide what to do)
- Batch mode naturally deduplicates across runs
Dependencies
| Dependency | Required | Purpose | Install |
|---|
node / npx tsx | ✅ | Runs ingest.ts | Bundled with Node.js |
curl | ✅ | HTTP fallback for URL extraction | System default |
yt-dlp | Recommended | YouTube transcript extraction | brew install yt-dlp |
lightpanda | Recommended | Clean web → markdown extraction | See lightpanda docs |
liteparse | Recommended | Multi-format local file extraction | pip install liteparse |
pdftotext | Optional | PDF fallback if liteparse missing | brew install poppler |
jq | Optional | JSON processing in shell scripts | brew install jq |
Degraded modes:
- No
lightpanda → falls back to curl + naive HTML stripping (works, less clean)
- No
yt-dlp → YouTube extraction fails with clear error
- No
liteparse → PDF uses pdftotext fallback; DOCX/XLSX/PPTX fail with clear error
Check references/failure-modes.md for full failure recovery.
Failure Modes
See references/failure-modes.md for the full catalog. Quick reference:
| Failure | What happens | How to recover |
|---|
| 404 / network timeout | Per-source failure logged, batch continues | Retry later, or skip |
| Paywall | Extraction returns minimal content | Flag with --confidence low or skip |
| Bot/challenge page captured as content | Raw file title may be Untitled and body says browser unsupported, verification required, Cloudflare, or security challenge; hash dedupe may then falsely mark different protected URLs as “already ingested” because they all captured the same challenge page | Treat it as a failed extraction, not a valid raw source. Use an alternate reader such as https://r.jina.ai/http://https://example.com/path, RSS/sitemap metadata, or direct official mirrors/PDFs; then write a curated raw file with the real source URL, extraction note, and distinct slug/hash before compiling. |
| DRM-protected YouTube | yt-dlp fails | Cannot ingest; use manual notes |
| Missing extractor | Clear error with install instructions | Install the tool, re-run |
| Rate limit | Extraction delays/fails | Wait, retry with smaller batch |
| Malformed HTML | Fallback to raw text | Usable but messier — lint will flag |
Reference Docs
Loaded as needed, not by default:
| File | When to read |
|---|
references/frontmatter-schema.md | Before hand-editing raw files, or when debugging frontmatter parse errors |
references/category-rules.md | When deciding which subdirectory a source belongs in, or when --category override is needed |
references/extractors.md | When source extraction fails or quality is poor — compares extractors and fallbacks |
references/failure-modes.md | When ingestion fails unexpectedly — catalog of known failures and recovery strategies |
Examples
See examples/ for concrete walkthroughs:
examples/youtube-ingestion.md — full flow for a YouTube videoexamples/batch-articles.md — multi-URL batchexamples/local-pdf.md — local document ingestion
What Changed From The Old Skill
The previous content-ingestion skill (at ~/.claude/skills/content-ingestion/) is deprecated. It targeted the Engram memory graph (Neo4j + embeddings + Haiku triplet extraction). This skill replaces it with a much simpler flow that targets the Daemion knowledge wiki.
Kept: the extraction layer (yt-dlp, lightpanda, liteparse) — those were storage-agnostic.
Dropped: Haiku triplet extraction, Neo4j storage, Ollama embeddings, entity deduplication, cross-source synthesis. The wiki compile pass handles the equivalent concerns differently (and better, per the Karpathy LLM Wiki pattern).
Do not use the old skill. It's marked deprecated with a pointer to this one.
For the migration rationale, see ~/.daemion/skills/knowledge-wiki/SKILL.md — the whole knowledge substrate moved from Engram to a compiled wiki.
