chromadb-integration-skills

# ChromaDB Integration Skills **Purpose**: This skill teaches agents how to integrate ChromaDB for semantic search, persistent storage, and pattern matching across ANY domain - research, code, trading, legal, documentation, and more. **Critical Use Case**: When agents need to work with large datasets (1000+ items), perform semantic search, maintain persistent knowledge, or learn from historical patterns, ChromaDB eliminates token limits and enables powerful vector-based retrieval. **Used By**: All agent types - researchers, developers, traders, legal analysts, documentation writers, QA testers, etc. --- ## When to Use ChromaDB Integration Use ChromaDB when: - **Large Datasets**: Working with 1000+ items (documents, code files, bugs, trades, contracts, etc.) - **Semantic Search**: Finding items by meaning, not just keywords - **Persistent Memory**: Knowledge needs to survive across sessions, days, months - **Pattern Matching**: Identifying similar historical cases/patterns for decision-making - **Cross-Session Learning**: Building institutional knowledge over time - **Token Limits**: Data too large to fit in context window (100K+ tokens) - **Aggregation**: Combining results from multiple queries/sources --- ## Core ChromaDB Concepts ### Collections **Definition**: Named vector databases storing documents with embeddings and metadata **Naming Strategy**: - **Domain-based**: `{domain}_{purpose}_{identifier}` - **Examples**: - Research: `research_prior_art_blockchain_2024`, `research_literature_ml_transformers` - Code: `codebase_api_endpoints`, `codebase_bug_patterns_auth` - Trading: `backtest_results_sma_strategy`, `market_conditions_spy_2024` - Legal: `case_law_patent_eligibility`, `contracts_saas_clauses` - Documentation: `api_docs_v2`, `architecture_decisions_2024` ### Documents **Definition**: Text content to be searched semantically **Best Practices**: - **Chunk Size**: 200-500 words optimal (too small = context loss, too large = poor granularity) - **Content Format**: Title + summary + key details (e.g., `"Patent US10123456 - Blockchain Authentication. Abstract: A method for..."`)) - **Deduplication**: Use unique IDs to prevent duplicate storage ### Metadata **Definition**: Structured data for filtering, not semantic search **Strategy**: ```javascript { // Temporal filters "date": "2024-11-14", "year": 2024, "month": 11, // Categorical filters "type": "bug_report", "category": "authentication", "severity": "high", // Numeric filters "citations": 42, "price": 150.25, "performance_score": 0.87, // Source tracking "source": "github_issue", "author": "kim-asplund", "url": "https://..." } ``` ### Embeddings **Definition**: Vector representations enabling semantic similarity **How It Works**: - ChromaDB automatically generates embeddings from document text - Similar meanings → similar vectors → close in vector space - Distance metrics (cosine, euclidean) measure similarity --- ## Universal ChromaDB Workflow ### Phase 1: Collection Design ```javascript // Step 1: Design collection strategy based on agent type const collectionStrategy = { research_agent: "One collection per research topic/question", code_agent: "Collections by codebase module/feature", trading_agent: "Collections by strategy/timeframe/symbol", legal_agent: "Collections by practice area/jurisdiction", documentation_agent: "Collections by project/version" }; // Step 2: Create collection with descriptive metadata mcp__chroma__create_collection({ collection_name: "{domain}_{purpose}_{identifier}", embedding_function_name: "default", // Uses sentence transformers metadata: { created_date: "2024-11-14", domain: "research|code|trading|legal|docs", purpose: "Descriptive purpose", total_items: 0, // Will update last_updated: "2024-11-14" } }); ``` ### Phase 2: Data Ingestion ```javascript // Step 1: Batch data collection (minimize API calls) const items = collectAllItems(); // From API, files, database, etc. // Step 2: Transform to ChromaDB format const documents = items.map(item => formatDocument(item)); const ids = items.map(item => item.id || generateUniqueId()); const metadatas = items.map(item => extractMetadata(item)); // Step 3: Batch insert (ChromaDB handles chunking automatically) mcp__chroma__add_documents({ collection_name: collectionName, documents: documents, ids: ids, metadatas: metadatas }); // Step 4: Update collection metadata mcp__chroma__modify_collection({ collection_name: collectionName, new_metadata: { ...existingMetadata, total_items: items.length, last_updated: new Date().toISOString() } }); ``` ### Phase 3: Semantic Search ```javascript // Step 1: Formulate semantic query (natural language works!) const query = "authentication failures in production environment"; // Step 2: Execute semantic search with filters const results = mcp__chroma__query_documents({ collection_name: collectionName, query_texts: [query], n_results: 20, where: { "$and": [ { "environment": "production" }, { "severity": { "$in": ["high", "critical"] } }, { "date": { "$gte": "2024-01-01" } } ] }, include: ["documents", "metadatas", "distances"] }); // Step 3: Filter by semantic similarity (distance threshold) const highlyRelevant = results.ids[0].filter((id, idx) => results.distances[0][idx] < 0.3 // Adjust threshold based on use case ); // Step 4: Retrieve full details if needed const fullDetails = mcp__chroma__get_documents({ collection_name: collectionName, ids: highlyRelevant, include: ["documents", "metadatas"] }); ``` ### Phase 4: Pattern Matching ```javascript // Cross-collection pattern detection const allCollections = mcp__chroma__list_collections(); const relevantCollections = allCollections.filter(c => c.startsWith(collectionPrefix) ); const patterns = []; for (const collection of relevantCollections) { const matches = mcp__chroma__query_documents({ collection_name: collection, query_texts: [patternQuery], n_results: 10, where: { "outcome": "success" } // Only successful cases }); if (matches.ids[0].length > 0) { patterns.push({ collection: collection, matches: matches, success_rate: calculateSuccessRate(matches) }); } } // Identify best pattern const bestPattern = patterns.sort((a, b) => b.success_rate - a.success_rate )[0]; ``` --- ## Use Case Templates ### Template 1: Research Agent - Literature Review **Problem**: Store 1000+ research papers, find semantically similar work ```javascript // Collection: research_literature_{topic} const papers = fetchPapersFromAPI("machine learning transformers"); mcp__chroma__create_collection({ collection_name: "research_literature_ml_transformers", metadata: { topic: "ML Transformers", papers_count: 0 } }); // Store papers with rich metadata papers.forEach(paper => { mcp__chroma__add_documents({ collection_name: "research_literature_ml_transformers", documents: [`${paper.title}. ${paper.abstract}`], ids: [paper.doi || paper.id], metadatas: [{ title: paper.title, authors: paper.authors.join(", "), year: paper.year, citations: paper.citation_count, venue: paper.venue, url: paper.url }] }); }); // Semantic search: "Find papers about attention mechanisms for vision" const relevant = mcp__chroma__query_documents({ collection_name: "research_literature_ml_transformers", query_texts: ["attention mechanisms computer vision"], n_results: 20, where: { "year": { "$gte": 2020 }, "citations": { "$gte": 50 } } }); ``` **Benefits**: No token limits, semantic discovery, citation filtering, persistent library --- ### Template 2: Code Agent - Bug Pattern Recognition **Problem**: Store bug reports, identify similar issues, suggest solutions ```javascript // Collection: codebase_bug_patterns_{module} const bugs = fetchAllGitHubIssues("is:issue label:bug"); mcp__chroma__create_collection({ collection_name: "codebase_bug_patterns_auth", metadata: { module: "authentication", total_bugs: 0 } }); // Store bugs with solutions bugs.forEach(bug => { mcp__chroma__add_documents({ collection_name: "codebase_bug_patterns_auth", documents: [`Bug #${bug.number}: ${bug.title}. ${bug.body}`], ids: [`bug_${bug.number}`], metadatas: [{ number: bug.number, title: bug.title, severity: bug.labels.find(l => l.startsWith("severity:"))?.split(":")[1], status: bug.state, solution: bug.resolution || "No solution yet", created_at: bug.created_at, resolved_at: bug.closed_at, url: bug.html_url }] }); }); // New bug arrives - find similar historical bugs const newBugDescription = "User login fails with 401 error after password reset"; const similarBugs = mcp__chroma__query_documents({ collection_name: "codebase_bug_patterns_auth", query_texts: [newBugDescription], n_results: 10, where: { "status": "closed", "solution": { "$ne": "No solution yet" } } }); // Extract solution from most similar resolved bug const suggestedSolution = similarBugs.metadatas[0][0].solution; ``` **Benefits**: Instant bug pattern matching, solution reuse, similar issue detection --- ### Template 3: Trading Agent - Backtest Results Database **Problem**: Store 10,000+ backtest results, identify optimal parameter patterns ```javascript // Collection: backtest_results_{strategy_name} const backtests = runParameterSweep(strategyCode, parameterRanges); mcp__chroma__create_collection({ collection_name: "backtest_results_sma_crossover", metadata: { strategy: "SMA Crossover", total_backtests: 0 } }); // Store each backtest with parameters + results backtests.forEach(backtest => { const description = ` SMA Crossover strategy with fast=${backtest.params.fast_period}, slow=${backtest.params.slow_period}, stop_loss=${backtest.params.stop_loss}. Market conditions: ${backtest.market_regime}, volatility=${backtest.avg_volatility}. `; mcp__chroma__add_documents({ collection_name: "backtest_results_sma_crossover", documents: [description], ids: [`backtest_${backtest.id}`], metadatas: [{ fast_period: backtest.params.fast_period, slow_period: backtest.params.slow_period, stop_loss: backtest.params.stop_loss, sharpe_ratio: backtest.sharpe_ratio, max_drawdown: backtest.max_drawdown, win_rate: backtest.win_rate, total_return: backtest.total_return, market_regime: backtest.market_regime, symbol: backtest.symbol, timeframe: backtest.timeframe, start_date: backtest.start_date, end_date: backtest.end_date }] }); }); // Find optimal parameters for current market conditions const currentMarket = analyzeCurrentMarket(); const marketDescription = ` Market regime: ${currentMarket.regime}, volatility: ${currentMarket.volatility}, trend strength: ${currentMarket.trend_strength} `; const optimalBacktests = mcp__chroma__query_documents({ collection_name: "backtest_results_sma_crossover", query_texts: [marketDescription], n_results: 20, where: { "$and": [