knowledge-ingestion

name	knowledge-ingestion
description	Turn trusted sources into structured ancestral knowledge with Firecrawl and openPasture tools.
version	1.0.0

Knowledge Ingestion

When to Use

Use this skill when you need to turn a trusted source into structured ancestral knowledge.

Typical sources:

• YouTube videos
• blog posts
• podcasts with transcripts
• articles and news
• any page Firecrawl can read

Procedure

Single Source

1. Use Firecrawl to search, scrape, or interact until the source text is in context.
2. If the source is YouTube, follow this acquisition ladder in order:
   • Start with firecrawl scrape "<watch-url>".
   • If the scrape only returns metadata, description, comments, or related videos, run a Firecrawl web search for the exact video title plus the channel name with --scrape.
   • If the title-based search still looks weak, retry with the video ID plus YouTube and the channel name because that sometimes surfaces a transcript-bearing result when the exact-title query does not.
   • Treat either a ## Transcript section or a Search transcript block as valid spoken-source text.
   • If transcript text is still missing, try firecrawl interact on the video page to open Show transcript or the closest transcript UI.
   • If transcript or captions are still unavailable, do not invent lessons from the title alone. Skip storage or queue the source for later review.
3. For YouTube specifically, treat comments, recommendations, and autogenerated metadata as supporting context only. The spoken transcript is the primary evidence.
4. Read seed/knowledge-categories.md before choosing categories.
5. Clean up the source mentally before storing anything:
   • remove transcript noise,
   • ignore intros, outros, and filler,
   • preserve the practitioner's actual point.
6. Distill the source into small, self-contained lessons.
7. For each lesson, decide:
   • entry_type: principle, technique, signal, or mistake
   • category: a simple reusable category from the reference file
   • tags: specific breeds, products, species, seasons, or management terms
8. Call find_similar_lessons with the lesson content and author.
9. If a close same-author match exists, consolidate by calling update_lesson.
10. If no close match exists, create a new lesson by calling store_lesson.

Batch Ingestion: Interactive Setup

Use this when the user says things like "process all of Greg Judy tonight."

1. Use Firecrawl to discover source URLs.
2. Call create_ingestion_batch with a batch name, author, and the discovered source list.
3. The batch tool will queue sources and mark already-ingested URLs as skipped.
4. Create one-shot cron jobs or delegated tasks that:
   • call claim_ingestion_batch_item,
   • stop if no item is returned,
   • ingest the claimed source with this skill,
   • call record_ingestion_batch_result when finished.
5. Use get_ingestion_batch_status to monitor progress and totals.

Batch Ingestion: Queue + Drainer

Use this when sources should be processed gradually and safely.

1. Discover URLs with Firecrawl.
2. Call create_ingestion_batch for the author or source campaign.
3. A separate recurring cron should:
   • call claim_ingestion_batch_item,
   • respond [SILENT] if no item is returned,
   • otherwise process exactly one returned source,
   • call record_ingestion_batch_result,
   • then exit.
4. Use get_ingestion_batch_status to inspect remaining queue depth, failures, and lesson totals.

Pitfalls

• Do not store generic fluff.
• Do not lose attribution.
• Prefer small, specific lessons over long summaries.
• Do not pretend metadata-only YouTube scrapes are full transcripts.
• Do not infer a durable lesson from a title if the spoken content is unavailable.
• Do not merge different authors into the same lesson.
• Do not invent new categories when an existing one already fits.
• When consolidating, keep the lesson richer, not blander.