関連職種SOC
SOC 職業分類に基づく
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/CleanExpo/NodeJS-Starter-V1 --skill vector-searchコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Route complex requests to the right specialist agent or chain of agents. This skill acts as the central brain of an agent swarm — it analyses what the user needs, determines which specialist domain(s) are required, and coordinates parallel or sequential agent execution. Use this skill when a request spans multiple domains (e.g., "research competitors and create a pitch deck"), when you need to decide which specialist should handle an ambiguous request, or when a task requires a multi-step pipeline across different skills. Triggers on: multi-step requests, cross-domain tasks, "coordinate", "plan this out", "I need help with multiple things", or any complex request that touches more than one specialist area. Also triggers when the user seems unsure which tool or approach to use.
>-
Hybrid DAG execution primitive combining deterministic and agentic nodes with hard iteration caps
SKILL.md を表示中
SOC 職業分類に基づく
| id | vector-search |
| name | vector-search |
| type | skill |
| version | 1.0.0 |
| created | 20/03/2026 |
| modified | 20/03/2026 |
| status | active |
| metadata | {"author":"NodeJS-Starter-V1","version":"1.0.0","locale":"en-AU"} |
| description | >- |
| context | fork |
Codifies the project's dual vector search systems (Memory Store for agent domain knowledge, RAG Pipeline for document retrieval), the multi-provider embedding abstraction, pgvector indexing, hybrid search scoring, and chunking strategies. All patterns are built on Supabase/PostgreSQL with pgvector.
Codifies pgvector embedding queries, similarity search, hybrid search, and multi-provider embedding generation for NodeJS-Starter-V1's Supabase/PostgreSQL stack, covering the Memory Store and RAG Pipeline vector infrastructure, indexing strategies, and chunking patterns.
dashboard-patterns instead)tsvector directly)metrics-collector instead)structured-logging instead)EmbeddingProvider abstraction. Never call OpenAI/Ollama directly.| System | Location | Purpose | Table |
|---|---|---|---|
| Memory Store | src/memory/store.py | Agent domain knowledge (patterns, preferences, debugging) | domain_memories |
| RAG Pipeline | src/rag/storage.py | Document retrieval (uploaded docs, chunked content) | document_chunks |
Both share the same EmbeddingProvider abstraction from src/memory/embeddings.py.
| Provider | Model | Dimensions | Use Case |
|---|---|---|---|
| OpenAI | text-embedding-3-small | 1536 | Production (preferred) |
| Ollama | nomic-embed-text | 768 | Local development (free) |
| Simple | Hash-based | 1536 | Testing only (deterministic) |
Selection via get_embedding_provider() — checks OPENAI_API_KEY, then ANTHROPIC_API_KEY, then falls back to SimpleEmbeddingProvider.
| Route | Method | Search Type |
|---|---|---|
/rag/search | POST | Vector, hybrid, or keyword |
/rag/upload | POST | Document ingestion + embedding |
/api/search | POST | Full-text search (tsvector only) |
| Table | Vector Column | Index Type | Distance Function |
|---|---|---|---|
documents | VECTOR(1536) | IVFFlat | vector_cosine_ops |
domain_memories | embedding | — | Cosine (via RPC) |
document_chunks | embedding | — | Cosine (via RPC) |
The EmbeddingProvider abstract base class defines a single method:
class EmbeddingProvider(ABC):
@abstractmethod
async def get_embedding(self, text: str) -> list[float]:
"""Generate embedding vector for text."""
pass
Three implementations: OpenAIEmbeddingProvider (calls /v1/embeddings via httpx), OllamaEmbeddingProvider (local /api/embeddings), SimpleEmbeddingProvider (hash-based, testing only).
EmbeddingProviderget_embedding() returning a fixed-dimension vectorget_embedding_provider()All vectors in a table MUST share the same dimension. If mixing providers with different dimensions (e.g., OpenAI 1536 vs Ollama 768), either:
The project currently standardises on 1536 dimensions (OpenAI).
MemoryStore.find_similar() generates a query embedding and calls the find_similar_memories PostgreSQL RPC:
async def find_similar(self, query_text: str, domain: MemoryDomain | None = None,
user_id: str | None = None, similarity_threshold: float = 0.7, limit: int = 10,
) -> list[dict[str, Any]]:
query_embedding = await self.embedding_provider.get_embedding(query_text)
result = self.client.rpc("find_similar_memories", {
"query_embedding": json.dumps(query_embedding),
"match_threshold": similarity_threshold,
"match_count": limit,
"filter_domain": domain.value if domain else None,
"filter_user_id": user_id,
}).execute()
return result.data or []
Key parameters: match_threshold (0.0–1.0, cosine similarity minimum), match_count (max results). Domain and user filters are applied server-side in the RPC function.
RAGStore.hybrid_search() combines vector similarity with keyword matching using configurable weights:
async def hybrid_search(self, query: str, project_id: str,
vector_weight: float = 0.6, keyword_weight: float = 0.4,
limit: int = 10, threshold: float = 0.5,
) -> list[dict[str, Any]]:
query_embedding = await self.embedding_provider.get_embedding(query)
result = self.client.rpc("hybrid_search", {
"query_text": query,
"query_embedding": query_embedding,
"project_id_filter": project_id,
"vector_weight": vector_weight,
"keyword_weight": keyword_weight,
"match_threshold": threshold,
"match_count": limit,
}).execute()
return result.data or []
Default weights: 60% vector + 40% keyword. Adjust for domain:
The /api/search route uses native PostgreSQL full-text search with ts_rank:
func.ts_rank(
func.to_tsvector("english", Document.title + " " + Document.content),
func.plainto_tsquery("english", query_text),
32, # RANK_CD normalisation flag
).label("relevance")
This is independent of vector search and uses the documents table directly via SQLAlchemy.
The project uses IVFFlat for approximate nearest-neighbour search:
CREATE INDEX idx_documents_embedding
ON documents USING ivfflat (embedding vector_cosine_ops);
IVFFlat partitions vectors into lists (clusters). Query searches only the nearest cluster(s), trading recall for speed.
Tuning parameters:
lists (build-time): Number of clusters. Rule of thumb: sqrt(row_count) for < 1M rowsprobes (query-time): Number of clusters to search. Higher = better recall, slower. Default: 1-- Set probes for a session (higher = more accurate, slower)
SET ivfflat.probes = 10;
For datasets > 10K rows, prefer HNSW (Hierarchical Navigable Small World):
CREATE INDEX idx_documents_embedding_hnsw
ON documents USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
HNSW provides better recall than IVFFlat without manual tuning. Higher m and ef_construction improve quality at the cost of build time and memory.
| Function | Operator | Index Ops | Use When |
|---|---|---|---|
| Cosine similarity | <=> | vector_cosine_ops | Normalised embeddings (most common) |
| L2 distance | <-> | vector_l2_ops | Raw distance comparison |
| Inner product | <#> | vector_ip_ops | Pre-normalised, performance-critical |
The project uses cosine similarity (vector_cosine_ops) throughout.
The RAG pipeline supports five chunking strategies via ChunkingStrategy enum:
| Strategy | When to Use | Config |
|---|---|---|
FIXED_SIZE | Uniform chunks, simple content | chunk_size=512, chunk_overlap=50 |
SEMANTIC | Respects paragraph/section boundaries | Same + boundary detection |
RECURSIVE | Nested structure (Markdown, HTML) | Splits by headers, then paragraphs, then sentences |
PARENT_CHILD | Best recall with context | parent_chunk_size=2048, child chunk_size=512 |
CODE_AWARE | Source code files | Splits by functions/classes |
Default: PARENT_CHILD with 512-token children and 2048-token parents. Search matches children; context retrieval includes the parent chunk.
PipelineConfig(
chunking_strategy=ChunkingStrategy.PARENT_CHILD,
chunk_size=512,
chunk_overlap=50,
parent_chunk_size=2048,
generate_embeddings=True,
generate_keywords=True,
)
| Threshold | Meaning | Use Case |
|---|---|---|
| 0.9+ | Near-exact semantic match | Deduplication |
| 0.7–0.9 | Strong relevance | Default search |
| 0.5–0.7 | Moderate relevance | Exploratory search |
| < 0.5 | Weak match | Usually noise |
The Memory Store defaults to similarity_threshold=0.7. The RAG Pipeline defaults to min_score=0.5.
MemoryStore.update_relevance() adjusts memory relevance based on feedback:
decay_rate, default 0.1, floored at 0.0)MemoryStore.prune_stale() removes memories below min_relevance=0.3 or older than max_age_days=90 via the prune_stale_memories RPC.
| Model | Fields | Purpose |
|---|---|---|
MemoryEntry | domain, category, key, value, embedding, relevance_score, access_count | Core memory unit |
MemoryQuery | domain, category, query_text, similarity_threshold, tags, limit, offset | Query specification |
MemoryResult | entries, total_count, query | Paginated result |
MemoryDomain | KNOWLEDGE, PREFERENCE, TESTING, DEBUGGING | Domain enum |
| Model | Fields | Purpose |
|---|---|---|
DocumentChunk | source_id, content, embedding, chunk_level, heading_hierarchy, keywords | Chunk record |
DocumentSource | source_type, source_uri, status, metadata | Source tracking |
SearchRequest | query, project_id, search_type, vector_weight, keyword_weight, min_score | Search input |
SearchResult | chunk_id, content, vector_score, keyword_score, combined_score | Result item |
SearchResponse | results, total_count, search_type, execution_time_ms | Search output |
CREATE TABLE documents (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
title VARCHAR(500) NOT NULL,
content TEXT NOT NULL,
embedding VECTOR(1536),
-- ... other columns
);
CREATE INDEX idx_documents_embedding ON documents USING ivfflat (embedding vector_cosine_ops);
Stores agent memories with embeddings for semantic retrieval. Accessed via MemoryStore class.
Stores RAG pipeline chunks with embeddings. Accessed via RAGStore class. Includes heading_hierarchy, summary, entities, keywords, and classification_tags for enriched retrieval.
| Function | Purpose |
|---|---|
find_similar_memories | Cosine similarity search on domain_memories with domain/user filters |
hybrid_search | Combined vector + keyword search on document_chunks |
prune_stale_memories | Delete low-relevance or expired memories |
increment_memory_access | Increment access count on retrieval |
| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
| Client-side similarity computation | Downloads all vectors, O(n) per query, no index usage | PostgreSQL RPC with pgvector index |
| Mixing embedding dimensions in one column | VECTOR(1536) rejects 768-dim vectors | Standardise dimension or use separate columns |
| No similarity threshold | Returns noise matches below 0.3 | Always set match_threshold (0.5–0.7) |
| Embedding at query time without caching | Re-embeds identical queries | Cache query embeddings for repeated searches |
IVFFlat with probes=1 on large datasets | Poor recall (misses relevant results) | Increase probes or migrate to HNSW |
| Storing embeddings without indexing | Sequential scan on every query | Create IVFFlat or HNSW index |
| Hardcoding OpenAI API calls | Breaks local development, vendor lock-in | Use EmbeddingProvider abstraction |
| Chunking without overlap | Loses context at chunk boundaries | Set chunk_overlap=50 minimum |
EmbeddingProvider abstraction (never direct API calls)metrics-collectorerror-taxonomy codesstructured-logging[AGENT_ACTIVATED]: Vector Search
[PHASE]: {Design | Implementation | Review}
[STATUS]: {in_progress | complete}
{vector search analysis or implementation guidance}
[NEXT_ACTION]: {what to do next}
search_query_duration_ms histogram for search latencysearch_result_count gauge for average results per queryembedding_generation_duration_ms histogram for provider latencyDATA_VECTOR_PROVIDER_UNAVAILABLE (503) — embedding provider downDATA_VECTOR_DIMENSION_MISMATCH (422) — wrong embedding dimensionDATA_VECTOR_THRESHOLD_INVALID (422) — threshold out of [0, 1] rangeSearchRequest validated via Pydantic (query non-empty, threshold in range, limit bounded)PipelineConfig validates chunk sizes and strategy enumDataStrip for aggregate metricsdocument_chunks table