chromadb-integration-skills

name	chromadb-integration-skills
description	Universal ChromaDB integration patterns for semantic search, persistent storage, and pattern matching across all agent types. Use when agents need to store/search large datasets, build knowledge bases, perform semantic analysis, or maintain persistent memory across sessions.
license	MIT

ChromaDB Integration Skills

Purpose: This skill teaches agents how to integrate ChromaDB for semantic search, persistent storage, and pattern matching across ANY domain - research, code, trading, legal, documentation, and more.

Critical Use Case: When agents need to work with large datasets (1000+ items), perform semantic search, maintain persistent knowledge, or learn from historical patterns, ChromaDB eliminates token limits and enables powerful vector-based retrieval.

Used By: All agent types - researchers, developers, traders, legal analysts, documentation writers, QA testers, etc.

---

When to Use ChromaDB Integration

Use ChromaDB when:

• Large Datasets: Working with 1000+ items (documents, code files, bugs, trades, contracts, etc.)
• Semantic Search: Finding items by meaning, not just keywords
• Persistent Memory: Knowledge needs to survive across sessions, days, months
• Pattern Matching: Identifying similar historical cases/patterns for decision-making
• Cross-Session Learning: Building institutional knowledge over time
• Token Limits: Data too large to fit in context window (100K+ tokens)
• Aggregation: Combining results from multiple queries/sources

---

Core ChromaDB Concepts

Collections

Definition: Named vector databases storing documents with embeddings and metadata

Naming Strategy:

• Domain-based: {domain}_{purpose}_{identifier}
• Examples:
  • Research: research_prior_art_blockchain_2024, research_literature_ml_transformers
  • Code: codebase_api_endpoints, codebase_bug_patterns_auth
  • Trading: backtest_results_sma_strategy, market_conditions_spy_2024
  • Legal: case_law_patent_eligibility, contracts_saas_clauses
  • Documentation: api_docs_v2, architecture_decisions_2024

Documents

Definition: Text content to be searched semantically

Best Practices:

• Chunk Size: 200-500 words optimal (too small = context loss, too large = poor granularity)
• Content Format: Title + summary + key details (e.g., "Patent US10123456 - Blockchain Authentication. Abstract: A method for..."))
• Deduplication: Use unique IDs to prevent duplicate storage

Metadata

Definition: Structured data for filtering, not semantic search

Strategy:

{
  // Temporal filters
  "date": "2024-11-14",
  "year": 2024,
  "month": 11,

  // Categorical filters
  "type": "bug_report",
  "category": "authentication",
  "severity": "high",

  // Numeric filters
  "citations": 42,
  "price": 150.25,
  "performance_score": 0.87,

  // Source tracking
  "source": "github_issue",
  "author": "kim-asplund",
  "url": "https://..."
}

Embeddings

Definition: Vector representations enabling semantic similarity

How It Works:

• ChromaDB automatically generates embeddings from document text
• Similar meanings → similar vectors → close in vector space
• Distance metrics (cosine, euclidean) measure similarity

---

Universal ChromaDB Workflow

Phase 1: Collection Design

// Step 1: Design collection strategy based on agent type
const collectionStrategy = {
  research_agent: "One collection per research topic/question",
  code_agent: "Collections by codebase module/feature",
  trading_agent: "Collections by strategy/timeframe/symbol",
  legal_agent: "Collections by practice area/jurisdiction",
  documentation_agent: "Collections by project/version"
};

// Step 2: Create collection with descriptive metadata
mcp__chroma__create_collection({
  collection_name: "{domain}_{purpose}_{identifier}",
  embedding_function_name: "default",  // Uses sentence transformers
  metadata: {
    created_date: "2024-11-14",
    domain: "research|code|trading|legal|docs",
    purpose: "Descriptive purpose",
    total_items: 0,  // Will update
    last_updated: "2024-11-14"
  }
});

Phase 2: Data Ingestion

// Step 1: Batch data collection (minimize API calls)
const items = collectAllItems();  // From API, files, database, etc.

// Step 2: Transform to ChromaDB format
const documents = items.map(item => formatDocument(item));
const ids = items.map(item => item.id || generateUniqueId());
const metadatas = items.map(item => extractMetadata(item));

// Step 3: Batch insert (ChromaDB handles chunking automatically)
mcp__chroma__add_documents({
  collection_name: collectionName,
  documents: documents,
  ids: ids,
  metadatas: metadatas
});

// Step 4: Update collection metadata
mcp__chroma__modify_collection({
  collection_name: collectionName,
  new_metadata: {
    ...existingMetadata,
    total_items: items.length,
    last_updated: new Date().toISOString()
  }
});

Phase 3: Semantic Search

// Step 1: Formulate semantic query (natural language works!)
const query = "authentication failures in production environment";

// Step 2: Execute semantic search with filters
const results = mcp__chroma__query_documents({
  collection_name: collectionName,
  query_texts: [query],
  n_results: 20,
  where: {
    "$and": [
      { "environment": "production" },
      { "severity": { "$in": ["high", "critical"] } },
      { "date": { "$gte": "2024-01-01" } }
    ]
  },
  include: ["documents", "metadatas", "distances"]
});

// Step 3: Filter by semantic similarity (distance threshold)
const highlyRelevant = results.ids[0].filter((id, idx) =>
  results.distances[0][idx] < 0.3  // Adjust threshold based on use case
);

// Step 4: Retrieve full details if needed
const fullDetails = mcp__chroma__get_documents({
  collection_name: collectionName,
  ids: highlyRelevant,
  include: ["documents", "metadatas"]
});

Phase 4: Pattern Matching

// Cross-collection pattern detection
const allCollections = mcp__chroma__list_collections();
const relevantCollections = allCollections.filter(c =>
  c.startsWith(collectionPrefix)
);

const patterns = [];
for (const collection of relevantCollections) {
  const matches = mcp__chroma__query_documents({
    collection_name: collection,
    query_texts: [patternQuery],
    n_results: 10,
    where: { "outcome": "success" }  // Only successful cases
  });

  if (matches.ids[0].length > 0) {
    patterns.push({
      collection: collection,
      matches: matches,
      success_rate: calculateSuccessRate(matches)
    });
  }
}

// Identify best pattern
const bestPattern = patterns.sort((a, b) =>
  b.success_rate - a.success_rate
)[0];

---

Use Case Templates

Template 1: Research Agent - Literature Review

Problem: Store 1000+ research papers, find semantically similar work

// Collection: research_literature_{topic}
const papers = fetchPapersFromAPI("machine learning transformers");

mcp__chroma__create_collection({
  collection_name: "research_literature_ml_transformers",
  metadata: { topic: "ML Transformers", papers_count: 0 }
});

// Store papers with rich metadata
papers.forEach(paper => {
  mcp__chroma__add_documents({
    collection_name: "research_literature_ml_transformers",
    documents: [`${paper.title}. ${paper.abstract}`],
    ids: [paper.doi || paper.id],
    metadatas: [{
      title: paper.title,
      authors: paper.authors.join(", "),
      year: paper.year,
      citations: paper.citation_count,
      venue: paper.venue,
      url: paper.url
    }]
  });
});

// Semantic search: "Find papers about attention mechanisms for vision"
const relevant = mcp__chroma__query_documents({
  collection_name: "research_literature_ml_transformers",
  query_texts: ["attention mechanisms computer vision"],
  n_results: 20,
  where: { "year": { "$gte": 2020 }, "citations": { "$gte": 50 } }
});

Benefits: No token limits, semantic discovery, citation filtering, persistent library

---

Template 2: Code Agent - Bug Pattern Recognition

Problem: Store bug reports, identify similar issues, suggest solutions

// Collection: codebase_bug_patterns_{module}
const bugs = fetchAllGitHubIssues("is:issue label:bug");

mcp__chroma__create_collection({
  collection_name: "codebase_bug_patterns_auth",
  metadata: { module: "authentication", total_bugs: 0 }
});

// Store bugs with solutions
bugs.forEach(bug => {
  mcp__chroma__add_documents({
    collection_name: "codebase_bug_patterns_auth",
    documents: [`Bug #${bug.number}: ${bug.title}. ${bug.body}`],
    ids: [`bug_${bug.number}`],
    metadatas: [{
      number: bug.number,
      title: bug.title,
      severity: bug.labels.find(l => l.startsWith("severity:"))?.split(":")[1],
      status: bug.state,
      solution: bug.resolution || "No solution yet",
      created_at: bug.created_at,
      resolved_at: bug.closed_at,
      url: bug.html_url
    }]
  });
});

// New bug arrives - find similar historical bugs
const newBugDescription = "User login fails with 401 error after password reset";

const similarBugs = mcp__chroma__query_documents({
  collection_name: "codebase_bug_patterns_auth",
  query_texts: [newBugDescription],
  n_results: 10,
  where: { "status": "closed", "solution": { "$ne": "No solution yet" } }
});

// Extract solution from most similar resolved bug
const suggestedSolution = similarBugs.metadatas[0][0].solution;

Benefits: Instant bug pattern matching, solution reuse, similar issue detection

---

Template 3: Trading Agent - Backtest Results Database

Problem: Store 10,000+ backtest results, identify optimal parameter patterns

// Collection: backtest_results_{strategy_name}
const backtests = runParameterSweep(strategyCode, parameterRanges);

mcp__chroma__create_collection({
  collection_name: "backtest_results_sma_crossover",
  metadata: { strategy: "SMA Crossover", total_backtests: 0 }
});

// Store each backtest with parameters + results
backtests.forEach(backtest => {
  const description = `
    SMA Crossover strategy with fast=${backtest.params.fast_period},
    slow=${backtest.params.slow_period}, stop_loss=${backtest.params.stop_loss}.
    Market conditions: ${backtest.market_regime}, volatility=${backtest.avg_volatility}.
  `;

  mcp__chroma__add_documents({
    collection_name: "backtest_results_sma_crossover",
    documents: [description],
    ids: [`backtest_${backtest.id}`],
    metadatas: [{
      fast_period: backtest.params.fast_period,
      slow_period: backtest.params.slow_period,
      stop_loss: backtest.params.stop_loss,
      sharpe_ratio: backtest.sharpe_ratio,
      max_drawdown: backtest.max_drawdown,
      win_rate: backtest.win_rate,
      total_return: backtest.total_return,
      market_regime: backtest.market_regime,
      symbol: backtest.symbol,
      timeframe: backtest.timeframe,
      start_date: backtest.start_date,
      end_date: backtest.end_date
    }]
  });
});

// Find optimal parameters for current market conditions
const currentMarket = analyzeCurrentMarket();
const marketDescription = `
  Market regime: ${currentMarket.regime}, volatility: ${currentMarket.volatility},
  trend strength: ${currentMarket.trend_strength}
`;

const optimalBacktests = mcp__chroma__query_documents({
  collection_name: "backtest_results_sma_crossover",
  query_texts: [marketDescription],
  n_results: 20,
  where: {
    "$and": [