| name | assess-ai-search |
| description | Measure precision / recall / MRR of Kalinka's CLAP-backed `ai_search` against the local `localfiles.db`. Use when the user asks to "evaluate the AI search", "benchmark CLAP", "check semantic-search quality", "measure retrieval precision/recall", or wants improvement suggestions for the embedding-based search. Generates a ~100-query benchmark, builds ground truth from artist heuristics + predicted tags + album metadata, runs the queries against both the audio-side (`vec_tracks_clap`) and text-side (`vec_tracks_clap_text`) vector indexes, and writes a Markdown report under `tmp/ai_search_eval/`. |
assess-ai-search
This skill has two benchmarks — pick by what you're asking:
| You want to know… | Use | Ground truth | Where |
|---|
| Is the CLAP retrieval model good? (stable yardstick for model/encoder/ranking changes) | MTG-Jamendo canonical benchmark | official human-curated tags — reliable, library-independent | mtg_jamendo/ |
| How good is search on this user's library? (coverage + enrichment included) | per-library harness (below) | local tagger + album_genre + curated artist_labels — only reliable on the dev library | queries.json + tools/ |
The per-library numbers are a conservative floor on an arbitrary library (sparse tags,
artist_labels curated for one collection); the MTG-Jamendo benchmark is the
trustworthy absolute measure. Both encode with the production ONNX encoder.
Reproducible quality benchmark for the AI search in kalinka-plugin-localfiles. The skill assumes:
- DB at
~/kalinka/var/lib/kalinka/localfiles.db (the dev-run active DB; a stale float-vector copy may linger at ~/kalinka/localfiles.db) (override with KALINKA_DB)
- Embeddings already computed (
embedding_clap_audio, embedding_clap_text, tags_predicted are populated)
- The ONNX CLAP wrapper (
packages/kalinka-plugin-localfiles/src/kalinka_plugin_localfiles/embedder/clap_onnx.py) is loadable and its model files (clap_text_encoder.onnx, clap_tokenizer.json) are present in KALINKA_MODEL_DIR (default ~/kalinka/models). This is mandatory — the stored vectors are ONNX-encoded; see the encoder note in "Reporting caveats".
The two vector indexes have different semantics and the skill measures both:
| Index | Side | What is stored | Used by |
|---|
vec_tracks_clap | audio | CLAP audio embedding of the first 10s | test_ai_search.py (canonical CLAP retrieval) |
vec_tracks_clap_text | text | CLAP text embedding of the track's title/artist/album string | comparison only (FTS handles metadata lookups) |
Production searcher._knn_leg queries the audio index (knn_search_audio → vec_tracks_clap): CLAP is contrastively trained text↔audio, so text-query → audio-embedding is the canonical retrieval direction.
A text query is encoded with the CLAP text encoder and KNN'd against both. Reporting both makes the audio↔text vs text↔text gap visible — that single number drives most of the "should we switch indexes?" decision.
Procedure
Run all steps in order. Do not skip step 1 or 2 — the report is only meaningful if ground truth was rebuilt against the current DB.
1. Sanity-check the DB
sqlite3 "${KALINKA_DB:-$HOME/kalinka/var/lib/kalinka/localfiles.db}" "
SELECT
(SELECT COUNT(*) FROM tracks) AS tracks,
(SELECT COUNT(*) FROM tracks WHERE embedding_clap_audio IS NOT NULL) AS audio_embedded,
(SELECT COUNT(*) FROM tracks WHERE embedding_clap_text IS NOT NULL) AS text_embedded,
(SELECT COUNT(*) FROM tracks WHERE tags_predicted IS NOT NULL AND tags_predicted != '') AS tagged,
(SELECT COUNT(*) FROM vec_tracks_clap) AS vec_audio_rows,
(SELECT COUNT(*) FROM vec_tracks_clap_text) AS vec_text_rows;
"
If audio/text coverage is <90%, flag it in the report — low coverage caps recall mechanically and is the first thing to fix before tuning weights.
2. Build ground truth
Run tools/build_ground_truth.py (bundled with this skill). It joins tracks, albums, artists and writes tmp/ai_search_eval/ground_truth.jsonl with one record per track:
{
"track_id": "...", "title": "...", "artist": "...", "album": "...",
"album_genre": "rock",
"year": 1985,
"predicted_top_genres": ["rock", "pop"],
"predicted_subgenres": ["prog rock"],
"mood_cluster": 2,
"danceability": 0.21,
"artist_labels": ["rock","prog rock","psychedelic","classic rock"]
}
Artist labels come from queries.json → artist_labels — a curated map of well-known artists in this specific library (Pink Floyd, Vangelis, Daft Punk, VNV Nation, Кино, etc.). These exist because predicted tags are sparse (~30% of tracks have non-empty genres) and albums.genre is dirty. The map is intentionally narrow: only include an artist when their canonical genre is unambiguous, otherwise leave them to fall back on predicted tags.
3. Run the benchmark
python .claude/skills/assess-ai-search/tools/evaluate.py \
--db "${KALINKA_DB:-$HOME/kalinka/var/lib/kalinka/localfiles.db}" \
--queries .claude/skills/assess-ai-search/queries.json \
--truth tmp/ai_search_eval/ground_truth.jsonl \
--k 10 \
--indexes audio,text \
--out tmp/ai_search_eval/results.json
Per query and per index the script reports:
- Precision@K — fraction of top-K results that are relevant per the query's rule
- Recall@K —
relevant_in_topK / min(K, total_relevant_in_library) (capped recall — see "Reporting caveats")
- MRR — mean reciprocal rank of the first relevant hit (0 if none in top K)
- Hit-Rate@K — fraction of queries with ≥1 relevant in top K (more informative than recall when relevant set is tiny)
- Distance distribution — min/median/max KNN distance of relevant vs non-relevant hits, used to suggest a similarity threshold
4. Compose the report
Read results.json and write tmp/ai_search_eval/REPORT.md with the sections below. Each table cell that's worse than the "good" threshold should be tagged (low) and link to the per-query JSON so the reader can inspect failures.
| Section | Content | Good if |
|---|
| Coverage | embedding + tag coverage from step 1 | ≥95% |
| Headline | mean P@10, R@10, MRR, Hit@10 — audio vs text | P@10 ≥ 0.40, Hit@10 ≥ 0.70 |
| By category | one row per category in queries.json (genre, subgenre, mood, energy, instrument, activity, era, abstract) | |
| Top failures | 10 queries with lowest P@10 (audio + text), with their top-3 wrong hits | |
| Top wins | 10 queries with highest P@10 | |
| Audio vs text gap | queries where the two indexes disagree the most | |
| Improvement suggestions | filled-in checklist from "Improvement suggestions" below | |
5. Suggest improvements
After looking at the numbers, walk the checklist below and write findings under the corresponding heading. Only include items the data supports — empty headings are fine, speculative ones are not.
A. Pipeline / coverage
B. Index choice (audio vs text)
CLAP is contrastively trained text↔audio, so a text query naturally retrieves over the audio index — which is what production already queries (searcher._knn_leg → knn_search_audio). The benchmark confirms audio > text on mood / activity / abstract / instrument (the timbre-driven categories); text only edges ahead on subgenre / era (metadata-word driven, which FTS already covers). So keep the audio leg; do not route NL queries through the text vector index — it's lexical and degenerates on sparse metadata (many tracks with empty/"Untitled" text collapse to one near-identical embedding).
The text index is plausibly better for queries that name an artist / album / language explicitly — those should be handled by FTS anyway. Confirm with the per-category gap before recommending the flip.
C. Re-ranker weights (SearcherConfig.weight_fts / weight_knn / weight_genre / weight_mood / weight_danceability, currently 0.35 / 0.30 / 0.20 / 0.10 / 0.05)
D. Query understanding
E. Model upgrades
F. Evaluation methodology
Reporting caveats
Corpus skew distorts raw P@10 — always report lift-over-random
When the library is dominated by a single class (e.g., 60% electronic, or 90% music with mood_cluster=2), queries whose relevance rule overlaps that class get high P@10 for free. A random retriever scores total_relevant / library_size per query in expectation.
Always compute and report:
random_baseline_P@10 = total_relevant / N (per query)lift = observed_P@10 − random_baseline_P@10- A query with
P@10 = 0.90 and random_baseline = 0.45 has only +0.45 lift — the model captured half the apparent headroom. A query with P@10 = 0.70 and random_baseline = 0.04 has +0.66 lift and is a genuine retrieval win.
Queries with negative lift are the diagnostic gold — they mean the retriever is doing worse than throwing darts. Typically: highly-prevalent relevance classes where the retriever's top-10 happens to cluster in the small non-relevant minority. Surface those in the report.
Use tools/analyze.py (bundled) to compute lift on an existing results.json / results.endpoint.json without re-running the encoder.
Recall@K is structurally capped — report what's actually achievable
At K=10 with N=519 tracks and an average class size of ~300 (typical for mood/era queries), the mathematical maximum uncapped R@10 is K / total_relevant ≈ 3%. Reporting "R@10 = 0.05" reads like a failure but is within touching distance of the theoretical ceiling — the metric is broken, not the model.
At small library scale, do all three:
- Capped R@10 =
n_rel_in_topK / min(K, total_relevant) — what evaluate.py reports. Equivalent to P@10 when total_relevant ≥ K (almost always true in this library), so largely redundant.
- Uncapped R@10 =
n_rel_in_topK / total_relevant — honest but mechanically tiny for prevalent classes.
- Recall-at-ceiling =
uncapped_R@K × total_relevant / K — fraction of the achievable recall headroom captured. Bounded in [0, 1] regardless of class size. This is the metric to compare across queries with different class sizes.
For meaningful aggregate recall, raise K to min(50, total_relevant) or report R@50 separately. At K=50 the structural cap on uncapped recall is loose enough that the number reflects model quality, not class size.
Other gotchas
- Ground truth is heuristic, not curated. A "wrong" hit might actually be relevant — sample 5-10 disagreements by ear when the numbers look surprising. Do not retune weights against a P@10 delta smaller than 0.05 — it's within the noise of the labelling heuristic.
- Encode with ONNX, not
laion_clap. The stored vec_tracks_clap* vectors are generated by the ONNX encoder (ClapOnnxModel). laion_clap lives in a different embedding space — querying ONNX-audio vectors with laion_clap-text vectors is near-random (cross-space KNN, distances ~1.31, ~½ the precision/MRR/lift). This is not a 4th-decimal difference. evaluate.py defaults to the ONNX encoder; KALINKA_FORCE_LAION=1 switches to laion_clap only for an explicit, loudly-warned comparison. Older results.*.json / baseline reports in tmp/ai_search_eval/ predate this fix and were produced with laion_clap — treat them as invalid.
- 6 queries in
queries.json may have total_relevant == 0 if the library lacks those genres entirely. These queries always score 0 and drag the mean P@10 down by ~6 pp. Exclude them from the headline (filter total_relevant > 0) or annotate them with "corpus_ood": true in queries.json.
Files in this skill
SKILL.md — this filequeries.json — 100 benchmark queries grouped by category, with relevance rules and the artist_labels lookuptools/build_ground_truth.py — joins the DB into ground_truth.jsonltools/evaluate.py — encodes queries, runs KNN against both vec indexes directly, computes raw metricstools/evaluate_endpoint.py — same metrics but hits the live /ai_search endpoint (full FTS + KNN + re-rank pipeline)tools/analyze.py — post-hoc analysis: lift-over-random, recall-at-ceiling, real-wins vs corpus-skew artifactsmtg_jamendo/ — canonical library-independent benchmark (MTG-Jamendo, official tags as ground truth). See mtg_jamendo/README.md; pipeline is select_subset.py → fetch_audio.py → embed.py → gen_queries.py → run_eval.py, artifacts default to tmp/mtg_jamendo_eval/.
If a script is missing or out of date relative to the DB schema, regenerate it from this SKILL.md rather than hand-patching — the skill is the source of truth.
