| name | andenken-embed |
| description | andenken embedding-maintenance workbench — the fixed flow the repo steward runs every time: status → sync(sessions+md) → verify → compact → oracle push. Re-embed/incremental for session + garden(md) indexes, integrity checks, defrag (pinned to 4 cores), oracle replication. Triggers: 'reindex', 'sync sessions', 'sync md', 'compact', 'oracle push', 'verify index', '임베딩 다시 하자', '세션/가든 임베딩', 'andenken 인덱스 정리'. |
| user_invocable | true |
andenken-embed — embedding maintenance workbench
Fixes the index-maintenance loop the steward repeats, so the commands aren't
rediscovered each time (saves tokens) and any agent/pi can pick it up in this repo.
Run everything from the repo root (~/repos/gh/andenken) via ./run.sh. run.sh
sources ~/.env.local for provider/keys and enforces the provider/dim guards.
- Just need a light, live session-only bump? → use the
memory-sync skill instead.
This skill is the full workbench: sessions + md maintenance + compact + oracle replication.
- Search (search-sessions / search-md) is not here — that's
semantic-memory.
Two tracks, one discipline
| Track | provider | dim | index | incremental command |
|---|
| sessions | OpenRouter qwen/qwen3-embedding-8b | 4096d | data/sessions.lance | ./run.sh sync:sessions [--push] |
| md (garden) | same 8B | 4096d | data/md.lance | ./run.sh sync:md + ./run.sh sync:md:oracle |
| org | (768d mismatch) | — | data/org.lance | production-disabled — do not touch |
org is service-disabled by a dim mismatch (provider 768 vs DB 2560/config). Its
indexing/search paths are refused; it is not a maintenance target.
Cost: paid remote (OpenRouter, $0.01/M tokens). Small but not zero. An
incremental run over tens of sessions / hundreds of md files is ~$0.01. Full
rebuild is a separate gate (below).
Normal flow (incremental re-embed)
cd ~/repos/gh/andenken
./run.sh status
./run.sh estimate:sessions
./run.sh estimate:md
./run.sh sync:sessions
./run.sh sync:md
./run.sh verify sessions
./run.sh verify md
./run.sh compact sessions
./run.sh compact md
./run.sh sync:sessions --push
./run.sh sync:md:oracle
Run long jobs (tens of sessions / hundreds of md) in the background and wait for
the completion signal. No short polling sleeps — re-calling the same sync is
a single-writer race.
Whole flow at once (background recommended)
./run.sh sync:sessions && ./run.sh sync:md \
&& ./run.sh verify sessions && ./run.sh verify md
Safety guards (already enforced by the scripts)
- dim 4096 preflight: each incremental confirms the provider dim with 1 call
before embedding. On mismatch with the DB dim → API-0 abort — never embed at
the wrong dim. That case needs a full rebuild (
scripts/rebuild-sessions-full.sh) first.
- to_index=0 → API-0 exit: nothing to embed → no probe. With
--push, it
still rsyncs the local DB and manifest (for example after compact), also at
zero API cost.
- org isolation: the sessions script never reads or writes
ANDENKEN_ORG_*,
ANDENKEN_VLLM_*, or org.lance.
compact — always core-limited
compact/cleanup call LanceDB table.optimize(), whose Rust rayon/tokio pools
pin all 16 cores to 100%. run.sh binds CPU affinity with taskset -c — rayon/tokio
size their pools from available_parallelism() (sched_getaffinity), so capping the
core set to 4 caps the threads to 4.
./run.sh compact md
ANDENKEN_COMPACT_CPUS=0-7 ./run.sh compact md
- Default
0-3. Override with ANDENKEN_COMPACT_CPUS in taskset -c syntax (0-3, 0,2,4,6).
- compact only when fragments grew a lot — not every increment. md was once
compacted 162 → 1 fragment.
verify reports the fragment count.
Oracle replication — DB and manifest travel together
thinkpad is the source of truth, oracle is a query replica. Rules:
- §6.6 (INVARIANT.md): don't ship only the
.lance DB. Sessions must rsync
session-manifest.json, md must rsync md-manifest.json alongside — otherwise
the replica is inconsistent. Both sync commands already carry the manifest.
- §7.1: oracle is not an indexing node. Running the indexer on oracle diverges
it from the source. Replicate only.
- push is outward-facing — run after GLG confirms. Embed + verify must pass first.
- See
INVARIANT.md §6.4–§6.6, §7–§7.1.
Full rebuild — human gate (no agent automation)
Only when a whole rebuild (not incremental) is required:
./run.sh rebuild:sessions:dry
./run.sh rebuild:sessions
- ₩100K-incident residual safety: agents do not automate full-sync / cost gates /
destructive rebuilds. A human decides after reading the estimate.
- md full rebuild only after reviewing
ANDENKEN_ALLOW_PAID_FULL_REBUILD=1.
single-writer
- One writer per track at a time.
sync:sessions takes a non-blocking flock
(data/.sync-sessions.lock), so a second sessions sync — cron or manual — backs
off cleanly ("already running") instead of racing. compact/cleanup and md are
not yet lock-guarded, so still avoid running two of those on the same track at once.
- Check by hand with a self-match-safe pattern:
pgrep -af '[s]ync-sessions'
(a plain pgrep -af sync-sessions also matches its own command line).
- Don't re-launch a background sync out of impatience. Wait for the completion signal.
Quick reference
| Want to | Command |
|---|
| Current state | ./run.sh status (status:json for machines) |
| Sessions incremental | ./run.sh sync:sessions |
| Garden incremental | ./run.sh sync:md |
| Verify | ./run.sh verify sessions|md|all |
| Defrag (4 cores) | ./run.sh compact sessions|md |
| dedup+orphan+manifest repair | ./run.sh cleanup sessions|md (includes compact, pinned) |
| Operator triage | ./run.sh doctor --sessions|--md [--json] |
| Oracle replicate | ./run.sh sync:sessions --push / ./run.sh sync:md:oracle |
| Cost estimate (API-0) | ./run.sh estimate:sessions|md [--full] |
SSOT is run.sh + scripts/ + INVARIANT.md. This skill is a signpost for that
flow — if behavior and docs disagree, run.sh/INVARIANT.md win.
