build-latentspace

name	build-latentspace
description	Rebuild the D3.js post map by embedding all blog posts with OpenAI and reducing to 2D with deterministic PCA. Run when posts are added, removed, or labels need refreshing.
user-invocable	true
allowed-tools	Bash Read

Build Latent Space

Rebuild data/latentspace.json — the single data source for the D3.js latent space visualization on the landing page.

What it does

1. Collects all eligible posts from note/_posts and testbed/_posts (excludes _ prefixed files and empty posts)
2. Embeds each post using OpenAI text-embedding-3-small
3. Reduces embeddings to 2D with deterministic PCA
4. Assigns a label based on emoji frontmatter
5. Extracts connected links from related frontmatter
6. Writes the result to data/latentspace.json

Incremental mode: cached embeddings in data/embeddings.json avoid repeat OpenAI calls. Coordinates are recomputed from the full embedding matrix with deterministic PCA.

Prerequisites

• OPENAI_API_KEY environment variable must be set
• Python dependencies: openai, beautifulsoup4, lxml, numpy, scikit-learn, tiktoken

Run

SKILL_DIR="${CLAUDE_SKILL_DIR:-$(git rev-parse --show-toplevel)/.agents/shared/skills/build-latentspace}"
python3 "${SKILL_DIR}/scripts/build_post_map.py"

Verify the output looks correct:

python3 -c "import json; data=json.load(open('data/latentspace.json')); print(f'{len(data)} posts'); print({d['label'] for d in data}); print(f'{sum(len(d[\"connected\"]) for d in data)} connections')"

Output schema

{
  "title": "GAN",
  "slug": "gan",
  "url": "/gan/",
  "category": "note",
  "label": "Brain",
  "connected": ["conditional-gan", "building-gan"],
  "x": 0.3421,
  "y": 0.7812
}

Label mapping

Labels are derived from the emoji frontmatter field of each post:

Emoji file	Label
brain.png	Brain
books.png	Books
pin.png	Pin
eyes.png	Eyes
wordballoon-with-dots.png	Wordballoon
robot.png	Robot
storm.png	Storm
(testbed posts)	Testbed
(no emoji)	Note