knowledge-ingestion

# Knowledge Ingestion ## When to Use Use this skill when you need to turn a trusted source into structured ancestral knowledge. Typical sources: - YouTube videos - blog posts - podcasts with transcripts - articles and news - any page Firecrawl can read ## Procedure ### Single Source 1. Use Firecrawl to search, scrape, or interact until the source text is in context. 2. If the source is YouTube, follow this acquisition ladder in order: - Start with `firecrawl scrape "<watch-url>"`. - If the scrape only returns metadata, description, comments, or related videos, run a Firecrawl web search for the exact video title plus the channel name with `--scrape`. - If the title-based search still looks weak, retry with the video ID plus `YouTube` and the channel name because that sometimes surfaces a transcript-bearing result when the exact-title query does not. - Treat either a `## Transcript` section or a `Search transcript` block as valid spoken-source text. - If transcript text is still missing, try `firecrawl interact` on the video page to open `Show transcript` or the closest transcript UI. - If transcript or captions are still unavailable, do not invent lessons from the title alone. Skip storage or queue the source for later review. 3. For YouTube specifically, treat comments, recommendations, and autogenerated metadata as supporting context only. The spoken transcript is the primary evidence. 4. Read `seed/knowledge-categories.md` before choosing categories. 5. Clean up the source mentally before storing anything: - remove transcript noise, - ignore intros, outros, and filler, - preserve the practitioner's actual point. 6. Distill the source into small, self-contained lessons. 7. For each lesson, decide: - `entry_type`: `principle`, `technique`, `signal`, or `mistake` - `category`: a simple reusable category from the reference file - `tags`: specific breeds, products, species, seasons, or management terms 8. Call `find_similar_lessons` with the lesson content and author. 9. If a close same-author match exists, consolidate by calling `update_lesson`. 10. If no close match exists, create a new lesson by calling `store_lesson`. ### Batch Ingestion: Interactive Setup Use this when the user says things like "process all of Greg Judy tonight." 1. Use Firecrawl to discover source URLs. 2. Call `create_ingestion_batch` with a batch name, author, and the discovered source list. 3. The batch tool will queue sources and mark already-ingested URLs as skipped. 4. Create one-shot cron jobs or delegated tasks that: - call `claim_ingestion_batch_item`, - stop if no item is returned, - ingest the claimed source with this skill, - call `record_ingestion_batch_result` when finished. 5. Use `get_ingestion_batch_status` to monitor progress and totals. ### Batch Ingestion: Queue + Drainer Use this when sources should be processed gradually and safely. 1. Discover URLs with Firecrawl. 2. Call `create_ingestion_batch` for the author or source campaign. 3. A separate recurring cron should: - call `claim_ingestion_batch_item`, - respond `[SILENT]` if no item is returned, - otherwise process exactly one returned source, - call `record_ingestion_batch_result`, - then exit. 4. Use `get_ingestion_batch_status` to inspect remaining queue depth, failures, and lesson totals. ## Pitfalls - Do not store generic fluff. - Do not lose attribution. - Prefer small, specific lessons over long summaries. - Do not pretend metadata-only YouTube scrapes are full transcripts. - Do not infer a durable lesson from a title if the spoken content is unavailable. - Do not merge different authors into the same lesson. - Do not invent new categories when an existing one already fits. - When consolidating, keep the lesson richer, not blander.