graphify

# /graphify Turn any folder of files into a navigable knowledge graph with community detection, an honest audit trail, and three outputs: interactive HTML, GraphRAG-ready JSON, and a plain-language GRAPH_REPORT.md. ## Usage ``` /graphify # full pipeline on current directory → Obsidian vault /graphify <path> # full pipeline on specific path /graphify https://github.com/<owner>/<repo> # clone repo then run full pipeline on it /graphify https://github.com/<owner>/<repo> --branch <branch> # clone a specific branch /graphify <url1> <url2> ... # clone multiple repos, build each, merge into one cross-repo graph /graphify <path> --mode deep # thorough extraction, richer INFERRED edges /graphify <path> --update # incremental - re-extract only new/changed files /graphify <path> --directed # build directed graph (preserves edge direction: source→target) /graphify <path> --whisper-model medium # use a larger Whisper model for better transcription accuracy /graphify <path> --cluster-only # rerun clustering on existing graph /graphify <path> --no-viz # skip visualization, just report + JSON /graphify <path> --html # (HTML is generated by default - this flag is a no-op) /graphify <path> --svg # also export graph.svg (embeds in Notion, GitHub) /graphify <path> --graphml # export graph.graphml (Gephi, yEd) /graphify <path> --neo4j # generate graphify-out/cypher.txt for Neo4j /graphify <path> --neo4j-push bolt://localhost:7687 # push directly to Neo4j /graphify <path> --mcp # start MCP stdio server for agent access /graphify <path> --watch # watch folder, auto-rebuild on code changes (no LLM needed) /graphify <path> --wiki # build agent-crawlable wiki (index.md + one article per community) /graphify <path> --obsidian --obsidian-dir ~/vaults/my-project # write vault to custom path (e.g. existing vault) /graphify add <url> # fetch URL, save to ./raw, update graph /graphify add <url> --author "Name" # tag who wrote it /graphify add <url> --contributor "Name" # tag who added it to the corpus /graphify query "<question>" # BFS traversal - broad context /graphify query "<question>" --dfs # DFS - trace a specific path /graphify query "<question>" --budget 1500 # cap answer at N tokens /graphify path "AuthModule" "Database" # shortest path between two concepts /graphify explain "SwinTransformer" # plain-language explanation of a node ``` ## What graphify is for Drop any folder of code, docs, papers, images, or video into graphify and get a queryable knowledge graph. Persistent across sessions, honest audit trail (EXTRACTED/INFERRED/AMBIGUOUS), community detection surfaces cross-document connections you wouldn't think to ask about. ## What You Must Do When Invoked If the user invoked `/graphify --help` or `/graphify -h` (with no other arguments), print the contents of the `## Usage` section above verbatim and stop. Do not run any commands, do not detect files, do not default the path to `.`. Just print the Usage block and return. **Fast path — existing graph:** Before doing anything else, check whether `graphify-out/graph.json` exists. The expected location is `graphify-out/graph.json` relative to the **current working directory** (i.e. the project root where you are running commands). If it exists AND the user's request is a natural-language question about the codebase (e.g. "How does X work?", "What calls Y?", "Trace the data flow through Z") and NOT an explicit rebuild command (`--update`, `--cluster-only`, or a bare path/URL that implies fresh extraction): **skip Steps 1–5 entirely and jump straight to `## For /graphify query`.** Run `graphify query "<question>"` immediately. Do not run detect. Do not check corpus size. Do not ask the user to narrow. The graph is already built — use it. If no path was given, use `.` (current directory). Do not ask the user for a path. If the path argument starts with `https://github.com/` or `http://github.com/`, treat it as a GitHub URL - run Step 0 before anything else, then continue with the resolved local path. Follow these steps in order. Do not skip steps. ### Step 0 - GitHub repos and multi-path merge (only if a URL or several paths) Only when the path is one or more `https://github.com/...` URLs, or several local subfolders to merge. See `references/github-and-merge.md` for the clone, cross-repo merge, and monorepo flow, then continue with the resolved local path. A plain local path skips this step. ### Step 1 - Ensure graphify is installed ```bash # Detect the correct Python interpreter (handles uv tool, pipx, venv, system installs) PYTHON="" GRAPHIFY_BIN=$(which graphify 2>/dev/null) # 1. uv tool installs — most reliable on modern Mac/Linux if [ -z "$PYTHON" ] && command -v uv >/dev/null 2>&1; then _UV_PY=$(uv tool run graphifyy python -c "import sys; print(sys.executable)" 2>/dev/null) if [ -n "$_UV_PY" ]; then PYTHON="$_UV_PY"; fi fi # 2. Read shebang from graphify binary (pipx and direct pip installs) if [ -z "$PYTHON" ] && [ -n "$GRAPHIFY_BIN" ]; then _SHEBANG=$(head -1 "$GRAPHIFY_BIN" | tr -d '#!') case "$_SHEBANG" in *[!a-zA-Z0-9/_.-]*) ;; *) "$_SHEBANG" -c "import graphify" 2>/dev/null && PYTHON="$_SHEBANG" ;; esac fi # 3. Fall back to python3 if [ -z "$PYTHON" ]; then PYTHON="python3"; fi if ! "$PYTHON" -c "import graphify" 2>/dev/null; then if command -v uv >/dev/null 2>&1; then uv tool install --upgrade graphifyy -q 2>&1 | tail -3 _UV_PY=$(uv tool run graphifyy python -c "import sys; print(sys.executable)" 2>/dev/null) if [ -n "$_UV_PY" ]; then PYTHON="$_UV_PY"; fi else "$PYTHON" -m pip install graphifyy -q 2>/dev/null \ || "$PYTHON" -m pip install graphifyy -q --break-system-packages 2>&1 | tail -3 fi fi # Write interpreter path for all subsequent steps (persists across invocations) mkdir -p graphify-out "$PYTHON" -c "import sys; open('graphify-out/.graphify_python', 'w', encoding='utf-8').write(sys.executable)" # Save scan root so `graphify update` (no args) knows where to look next time echo "$(cd INPUT_PATH && pwd)" > graphify-out/.graphify_root ``` If the import succeeds, print nothing and move straight to Step 2. **In every subsequent bash block, replace `python3` with `$(cat graphify-out/.graphify_python)` to use the correct interpreter.** ### Step 2 - Detect files ```bash $(cat graphify-out/.graphify_python) -c " import json from graphify.detect import detect from pathlib import Path result = detect(Path('INPUT_PATH')) print(json.dumps(result, ensure_ascii=False)) " > graphify-out/.graphify_detect.json ``` Replace INPUT_PATH with the actual path the user provided. Do NOT cat or print the JSON - read it silently and present a clean summary instead: ``` Corpus: X files · ~Y words code: N files (.py .ts .go ...) docs: N files (.md .txt ...) papers: N files (.pdf ...) images: N files video: N files (.mp4 .mp3 ...) ``` Omit any category with 0 files from the summary. Then act on it: - If `total_files` is 0: stop with "No supported files found in [path]." - If `skipped_sensitive` is non-empty: mention file count skipped, not the file names. - If `total_words` > 2,000,000 OR `total_files` > 500: show the warning. Then compute the top 5 first-level subdirectories by file count: - Read `scan_root` from the detect JSON (always an absolute path to the resolved INPUT_PATH). - Concatenate all file lists across all types (`code`, `document`, `paper`, `image`, `video`). - Filter out any path that starts with `scan_root + "/graphify-out/"` to exclude converted sidecars. - For each file, strip the `scan_root` prefix and take the first path component. Files directly in `scan_root` with no subdirectory count as `(root)`. - If all files are in `(root)` with no subdirectories, do not ask to narrow — no subfolders exist. Instead suggest `--no-cluster` to skip the expensive clustering step and proceed. - Otherwise rank by count, show the top 5 with file counts, then ask which subfolder to run on. Wait for the user's answer before proceeding. - Otherwise: proceed directly to Step 2.5 if video files were detected, or Step 3 if not. ### Step 2.5 - Video and audio (only if video files detected) Skip this step entirely if `detect` returned zero `video` files. When the corpus has video or audio, see `references/transcribe.md` to transcribe them to text first, then treat the transcripts as doc files in Step 3. ### Step 3 - Extract entities and relationships **Before starting:** note whether `--mode deep` was given. You must pass `DEEP_MODE=true` to every subagent in Step B2 if it was. Track this from the original invocation - do not lose it. This step has two parts: **structural extraction** (deterministic, free) and **semantic extraction** (LLM, costs tokens). **Before dispatching subagents:** check whether `GEMINI_API_KEY` or `GOOGLE_API_KEY` is set. If neither is set, print this one-liner to the user: > Tip: set `GEMINI_API_KEY` or `GOOGLE_API_KEY` to use Gemini for semantic extraction (`pip install 'graphifyy[gemini]'`). Print it once, then continue. If `GEMINI_API_KEY` or `GOOGLE_API_KEY` IS set, use `graphify.llm.extract_corpus_parallel(files, backend="gemini")` for semantic extraction instead of dispatching Claude subagents. The default Gemini model is `gemini-3-flash-preview`; set `GRAPHIFY_GEMINI_MODEL` or pass `--model` in headless CLI flows to override it. > **No other API keys are read.** If `GEMINI_API_KEY`/`GOOGLE_API_KEY` are unset, fall straight through to Claude Code subagent dispatch (Part B below) — the host session itself is the LLM. graphify does **not** read `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or any other provider key from the environment. If a host agent prompts the user for `ANTHROPIC_API_KEY` to run extraction, that prompt is a misread of this skill — ignore it and dispatch subagents as written. **Run Part A (AST) and Part B (semantic) in parallel. Dispatch all semantic subagents AND start AST extraction in the same message. Both can run simultaneously since they operate on different file types. Merge results in Part C as before.** Note: Parallelizing AST + semantic saves 5-15s on large corpora. AST is deterministic and fast; start it while subagents are processing docs/papers. #### Part A - Structural extraction for code files For any code files detected, run AST extraction in parallel with Part B subagents: ```bash $(cat graphify-out/.graphify_python) -c " import sys, json from graphify.extract import collect_files, extract from pathlib import Path import json code_files = [] detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\"utf-8\")) for f in detect.get('files', {}).get('code', []): code_files.extend(collect_files(Path(f)) if Path(f).is_dir() else [Path(f)]) if code_files: result = extract(code_files, cache_root=Path('.')) Path('graphify-out/.graphify_ast.json').write_text(json.dumps(result, indent=2, ensure_ascii=False), encoding=\"utf-8\") print(f'AST: {len(result[\"nodes\"])} nodes, {len(result[\"edges\"])} edges') else: Path('graphify-out/.graphify_ast.json').write_text(json.dumps({'nodes':[],'edges':[],'input_tokens':0,'output_tokens':0}, ensure_ascii=False), encoding=\"utf-8\") print('No code files - skipping AST extraction') " ``` #### Part B - Semantic extraction (parallel subagents) **Fast path:** If detection found zero docs, papers, and images (code-only corpus), skip Part B entirely and go straight to Part C. AST handles code - there is nothing for semantic subagents to do. **MANDATORY: You MUST use the Agent tool here. Reading files yourself one-by-one is forbidden - it is 5-10x slower. If you do not use the Agent tool you are doing this wrong.** Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` - Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" **Step B0 - Check extraction cache first** Before dispatching any subagents, check which files already have cached extraction results: ```bash $(cat graphify-out/.graphify_python) -c " import json from graphify.cache import check_semantic_cache from pathlib import Path detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\"utf-8\")) all_files = [f for files in detect['files'].values() for f in files] cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files) if cached_nodes or cached_edges or cached_hyperedges: Path('graphify-out/.graphify_cached.json').write_text(json.dumps({'nodes': cached_nodes, 'edges': cached_edges, 'hyperedges': cached_hyperedges}, ensure_ascii=False), encoding=\"utf-8\") Path('graphify-out/.graphify_uncached.txt').write_text('\n'.join(uncached), encoding=\"utf-8\") print(f'Cache: {len(all_files)-len(uncached)} files hit, {len(uncached)} files need extraction') " ``` Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt`. If all files are cached, skip to Part C directly. **Step B1 - Split into chunks** Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. **Step B2 - Dispatch ALL subagents in a single message** Call the Agent tool multiple times IN THE SAME RESPONSE - one call per chunk. This is the only way they run in parallel. If you make one Agent call, wait, then make another, you are doing it sequentially and defeating the purpose. **IMPORTANT - subagent type:** Always use `subagent_type="general-purpose"`. Do NOT use `Explore` - it is read-only and cannot write chunk files to disk, which silently drops extraction results. General-purpose has Write and Bash access which the subagent needs. Concrete example for 3 chunks: ``` [Agent tool call 1: files 1-15, subagent_type="general-purpose"] [Agent tool call 2: files 16-30, subagent_type="general-purpose"] [Agent tool call 3: files 31-45, subagent_type="general-purpose"] ``` All three in one message. Not three separate messages. Each subagent receives this exact prompt (substitute FILE_LIST, CHUNK_NUM, TOTAL_CHUNKS, DEEP_MODE, and CHUNK_PATH). CHUNK_PATH must be an **absolute** path — derive it before dispatching: ```bash PROJECT_ROOT=$(cat graphify-out/.graphify_root) # Then for chunk N: CHUNK_PATH="${PROJECT_ROOT}/graphify-out/.graphify_chunk_0N.json" ```