ai-engineer-learning

name	ai-engineer-learning
description	YouTube transcript ingestion pipeline and RAG query interface for the AI Engineer persona. All 763 @aiDotEngineer channel videos indexed in Qdrant. Run weekly to stay current.
compatibility	Created for Zo Computer
metadata	{"author":"marlandoj.zo.computer"}

AI Engineer Learning Skill

Consumes, indexes, and queries the entire @aiDotEngineer YouTube channel — a leading source of AI engineering talks from Google DeepMind, Anthropic, OpenAI, Hugging Face, and more.

Current Status (2026-06-15)

• 822 Qdrant points across 763 unique videos
• 44 videos — full transcript-backed (deep chunk embeddings)
• 719 videos — metadata-indexed (title + duration + URL, still semantically searchable by topic)
• Collection: ai-engineer-videos @ 127.0.0.1:6333

When to Use

• The AI Engineer persona is working on Zouroboros and needs current best practices
• Building agents, RAG systems, evals, MCP servers, or AI infrastructure
• Need to reference talks on: agent architecture, context management, evaluation, MCP, RAG, model deployment, voice AI, multi-agent systems, observability
• Weekly: run the update pipeline to ingest new videos

Query the Knowledge Base

# Semantic search (primary interface)
bun Skills/ai-engineer-learning/scripts/query.ts "how to build durable agents"
bun Skills/ai-engineer-learning/scripts/query.ts "context management" --top 10
bun Skills/ai-engineer-learning/scripts/query.ts "evaluation frameworks"

# Within AI Engineer persona, this is the first thing to call before suggesting approaches

Ingestion Pipeline

Add New Videos (weekly sync)

# 1. Update channel-videos.txt with new videos
cd /home/workspace/Skills/ai-engineer-learning/scripts
bun update-channel.ts   # or manually append to channel-videos.txt

# 2. Index new videos (metadata-only first — always works, no YouTube access needed)
python3 ingest-metadata.py

# 3. Upgrade to transcript-backed if transcripts are available via save_webpage
#    (saves markdown to /home/workspace/Articles/, then process with:)
bun process-transcripts.ts

Upgrade Metadata → Transcript (when articles are available)

# Videos saved via Zo's save_webpage land in /home/workspace/Articles/
# as "* :: www.youtube.com.md" files. Run this to embed them:
bun Skills/ai-engineer-learning/scripts/process-transcripts.ts

NotebookLM Path (richer knowledge, needs Google session)

Use this for natural language deep-dives on topics. Requires valid NotebookLM cookies.

# Check session status
cd /home/workspace/Skills/notebooklm-skill/scripts
python3.12 nlm.py status

# If expired — re-export cookies (90-second process):
#   1. Open notebooklm.google.com in Chrome (must be signed in)
#   2. Install Cookie-Editor extension, click Export, copy JSON
#   3. Paste to /tmp/cookies.json on Zo
#   4. python3.12 nlm.py import-cookies /tmp/cookies.json

# Bulk-add remaining 535 videos to 2 NotebookLM notebooks (one-shot after auth):
python3.12 /home/workspace/Skills/ai-engineer-learning/scripts/load-to-notebooklm.py

# Query NotebookLM knowledge base:
python3.12 nlm.py ask --notebook <nb_id> --question "best practices for agent memory"
# (Notebook IDs saved to Projects/ai-engineer-learning/notebooklm-notebooks.json after load)

Files

• scripts/ingest-metadata.py — Metadata-only Qdrant indexer (no YouTube access needed)
• scripts/ingest-batch.py — Full transcript Qdrant indexer (requires residential IP / save_webpage)
• scripts/process-transcripts.ts — Embeds Articles/ markdown saves into Qdrant
• scripts/load-to-notebooklm.py — Bulk-add YouTube URLs to NotebookLM notebooks
• scripts/query.ts — Semantic search + research notebook generator
• Projects/ai-engineer-learning/channel-videos.txt — Video metadata catalog (763 videos)
• Projects/ai-engineer-learning/processed.json — State file (tracks ingested videos)
• Projects/ai-engineer-learning/notebooklm-notebooks.json — NotebookLM notebook IDs (after load)

Scheduled Agent

Weekly agent [ZBR] Ingest AI Engineer Videos runs every Monday at 0200 UTC:

source /root/.zo_secrets && python3 /home/workspace/Skills/ai-engineer-learning/scripts/ingest-metadata.py

This indexes new videos from channel-videos.txt and embeds them into Qdrant.