obsidian-brain-eval

name	obsidian-brain-eval
description	Test your Obsidian RAG, measure your second brain's retrieval quality, and score your vault on Recall@10 against a gold-set generated from your own notes. Pluggable retrieval backends (BM25 zero-infra, LanceDB hybrid FTS+vector). Pure Python CLI, read-only. The skill is the only tool that publishes a methodology and a default 0.85 pass/fail threshold for "is your vault chat actually working". Use when the user asks any of: test my obsidian rag, measure my second brain, is my pkm answering well, obsidian retrieval quality, test my obsidian search, recall at 10 my vault, evaluate my obsidian rag, obsidian rag benchmark, score my obsidian retrieval, is my second brain working, obsidian search quality, pkm retrieval eval, obsidian eval framework, gold set obsidian, smart connections eval, obsidian copilot eval, rag over my vault, does my vault chat work, vault retrieval test. Works on any markdown vault using wikilinks (Obsidian, Logseq, Foam, Quartz). Complements obsidian-graph-auditor (measures graph topology); brain-eval measures retrieval quality.
license	MIT
author	Dhiraj Singh Pawar (build-with-dhiraj)
homepage	https://github.com/build-with-dhiraj/obsidian-brain-eval
version	0.1.0

Obsidian Brain Eval

Measure whether your Obsidian RAG actually works. Generate a gold-set from your vault, score any retrieval backend on Recall@10, and get a PASS or BELOW TARGET grade against a 0.85 default threshold.

When to use this skill

Trigger this skill whenever the user wants to:

1. Test their vault's RAG. "test my obsidian rag", "is my second brain answering well", "does my vault chat work", "rag over my vault test".
2. Measure retrieval quality. "obsidian retrieval quality", "score my obsidian retrieval", "recall at 10 my vault", "obsidian search quality".
3. Benchmark a retrieval system. "obsidian rag benchmark", "compare BM25 vs hybrid in my vault", "evaluate Smart Connections", "Obsidian Copilot eval", "vault retrieval test".
4. Generate a gold-set. "make a gold-set for my vault", "create eval questions from my notes", "gold set obsidian".
5. Establish a CI gate. Run score --backend ... in CI to block retrieval regressions.
6. Decide what to improve next. When Recall@10 < 0.85, the misses tell you where the retrieval layer fails (paraphrase, rare vocab, chunking).

What it does

• generate — Sample N representative notes from your vault, ask GPT-4.1 to write ONE natural question per note that the note answers, and write a JSONL gold-set. Each record pins the source note plus any notes it strongly links via frontmatter (entities, topics, related). Costs roughly $1-2 of OpenAI for 40 questions. Idempotent: skips notes already in the gold-set unless --force.
• score — Run each gold question through a pluggable retrieval backend and check whether a known-relevant note appears in the top-k. Reports Recall@k, hit/miss per question, and a rank distribution. No GPT cost.

How to run

After installation (see ## Install below):

# Step 1 (optional): generate a 40-question gold-set with GPT-4.1.
# Skip this and use the shipped sample if you don't want to spend GPT.
export OPENAI_API_KEY=...
obsidian-brain-eval generate --vault ~/Documents/MyVault --out gold.jsonl --n 40

# Step 2: score with the zero-infra BM25 backend.
obsidian-brain-eval score --vault ~/Documents/MyVault --gold gold.jsonl --backend naive


obsidian-brain-eval score --vault ~/Documents/MyVault --gold gold.jsonl \
    --backend lancedb --db ~/Documents/MyVault/.lancedb


obsidian-brain-eval score --vault ~/Documents/MyVault --gold gold.jsonl \
    --backend naive --quiet --json-out result.json


obsidian-brain-eval score --vault examples/demo-vault \
    --gold examples/sample-gold-set.jsonl --backend naive