skill-literature

# Literature Skill (Direct Execution) Direct execution skill for managing `specs/literature/` directories. Handles PDF/DJVU-to-markdown conversion, index.json maintenance, and filesystem validation. Runs inline using AskUserQuestion for interactivity. **Key behavior**: Users see scan results and proposed keywords/summaries BEFORE any files are written. Users confirm chunk boundaries and metadata before conversion completes. ## Context References Reference (do not load eagerly): - Path: `@specs/literature/index.json` - Current literature index - Path: `@specs/702_create_literature_command/reports/01_lit-command.md` - Research findings --- ## Execution ### Step 1: Parse Arguments Extract mode, optional file, and optional query from skill args: ```bash # Parse from skill args: "mode={mode} file={file}" or "mode=search query={query text}" mode=$(echo "$ARGUMENTS" | grep -oP 'mode=\K\S+' | head -1) file=$(echo "$ARGUMENTS" | grep -oP 'file=\K\S+' | head -1) # Extract query: everything after "query=" (supports spaces in query text) query=$(echo "$ARGUMENTS" | sed 's/.*query=//' | sed 's/^[[:space:]]*//') # Default to status mode if not specified if [ -z "$mode" ]; then mode="status" fi # Resolve file path (may be relative or absolute) if [ -n "$file" ]; then if [[ "$file" != /* ]]; then file="specs/literature/$file" fi fi ``` ### Step 2: Generate Session ID ```bash source .claude/scripts/lib/common.sh session_id="$(common_session_id)" # Two-tier fallback: use LITERATURE_DIR if set and exists, otherwise use per-project specs/literature/ if [ -n "${LITERATURE_DIR:-}" ] && [ -d "$LITERATURE_DIR" ]; then lit_dir="$LITERATURE_DIR" else lit_dir="specs/literature" fi index_file="$lit_dir/index.json" # Determine sources/ prefix for centralized repo if [ -n "${LITERATURE_DIR:-}" ] && [ "$lit_dir" = "$LITERATURE_DIR" ]; then sources_prefix="sources/" else sources_prefix="" fi ``` ### Step 3: Check Tool Availability Detect available conversion tools: ```bash has_pdftotext=$(which pdftotext 2>/dev/null && echo "yes" || echo "no") has_pdfinfo=$(which pdfinfo 2>/dev/null && echo "yes" || echo "no") has_djvutxt=$(which djvutxt 2>/dev/null && echo "yes" || echo "no") ``` ### Step 4: Dispatch to Mode Handler Route to the appropriate mode: ```bash case "$mode" in status) handle_status ;; scan) handle_scan ;; convert) handle_convert ;; validate) handle_validate ;; index) handle_index ;; search) handle_search ;; ingest) handle_ingest ;; rebuild) handle_rebuild ;; *) echo "Error: Unknown mode '$mode'. Available: status, scan, convert, validate, index, search, ingest, rebuild" exit 1 ;; esac ``` --- ## Mode: Ingest Full pipeline ingestion: convert PDF/DJVU to markdown, chunk hierarchically, index in global SQLite FTS5 database, and optionally load into local specs/literature/. ### Ingest Step 1: Resolve Source Path ```bash if [ -z "$file" ]; then echo "Error: --ingest requires a path or --zotero key." echo "Usage: /literature --ingest <path> | /literature --ingest --zotero <key>" exit 1 fi ``` ### Ingest Step 2: Invoke literature-ingest.sh Find the ingest script relative to the skill's script directory: ```bash SCRIPT_DIR="$(dirname "$0")/../../scripts" INGEST_SCRIPT="$SCRIPT_DIR/literature-ingest.sh" if [ ! -x "$INGEST_SCRIPT" ]; then echo "Error: literature-ingest.sh not found at: $INGEST_SCRIPT" exit 1 fi # Route to ingest script with appropriate flags if [ -n "$zotero_key" ]; then "$INGEST_SCRIPT" --zotero "$zotero_key" "$@" else "$INGEST_SCRIPT" "$file" "$@" fi ``` Where: - `$file` is the source path (PDF, DJVU, or directory) - `$zotero_key` is the Zotero citation key (if using `--zotero`) - Remaining `$@` may include `--no-local` or `--local` flags ### Ingest Step 3: Display Result The `literature-ingest.sh` script outputs a summary to stdout on completion. Relay this output to the user verbatim, then add: ``` To search the ingested literature: /literature --search "query" Or use --lit flag in research/plan/implement commands to enable agent search. ``` ### Ingest Examples ```bash # Ingest a single PDF /literature --ingest ~/Papers/modal-logic.pdf # Ingest all PDFs in a directory /literature --ingest ~/Papers/modal-logic/ # Ingest from Zotero (requires zotero-library.json) /literature --ingest --zotero "BlackburnDeRijkeVenema2001" # Ingest and skip local loading prompt /literature --ingest ~/Papers/modal-logic.pdf --no-local # Ingest and automatically load into specs/literature/ /literature --ingest ~/Papers/modal-logic.pdf --local ``` --- ## Mode: Status (Default) Show health report: processed vs unprocessed files and index.json state. ### Status Step 1: Check Directory ```bash if [ ! -d "$lit_dir" ]; then echo "## Literature Status" echo "" echo "No specs/literature/ directory found." echo "Create it and add PDF/DJVU files to get started." echo "" echo "**Tool Availability**:" echo "- pdftotext: $has_pdftotext" echo "- djvutxt: $has_djvutxt ($([ "$has_djvutxt" = "no" ] && echo 'install: nix-env -iA nixpkgs.djvulibre' || echo 'available'))" exit 0 fi ``` ### Status Step 2: Scan for Files ```bash # Find all PDF and DJVU source files pdf_files=$(find "$lit_dir" -name "*.pdf" 2>/dev/null | sort) djvu_files=$(find "$lit_dir" -name "*.djvu" 2>/dev/null | sort) all_source_files="$pdf_files $djvu_files" # Find all markdown files (excluding any in subdirectory source_files/) md_files=$(find "$lit_dir" -name "*.md" -not -path "*/source_files/*" 2>/dev/null | sort) ``` ### Status Step 3: Read Index ```bash if [ -f "$index_file" ]; then entry_count=$(jq '.entries | length' "$index_file" 2>/dev/null || echo "0") indexed_paths=$(jq -r '.entries[].path' "$index_file" 2>/dev/null || echo "") else entry_count=0 indexed_paths="" fi ``` ### Status Step 4: Compute Counts ```bash # Count source files pdf_count=$(echo "$pdf_files" | grep -c "\.pdf$" 2>/dev/null || echo 0) djvu_count=$(echo "$djvu_files" | grep -c "\.djvu$" 2>/dev/null || echo 0) md_count=$(echo "$md_files" | grep -c "\.md$" 2>/dev/null || echo 0) # Identify unprocessed source files (PDFs/DJVUs without corresponding .md) unprocessed=() for src in $pdf_files $djvu_files; do basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//') # Check if any .md file starts with this basename if ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; then unprocessed+=("$src") fi done unprocessed_count=${#unprocessed[@]} processed_count=$(( pdf_count + djvu_count - unprocessed_count )) ``` ### Status Step 5: Display Report ``` ## Literature Status **Directory**: specs/literature/ **Source Files**: {pdf_count} PDFs, {djvu_count} DJVUs **Converted**: {processed_count} processed, {unprocessed_count} unprocessed **Markdown Files**: {md_count} **Index Entries**: {entry_count} **Tool Availability**: - pdftotext: {has_pdftotext} - djvutxt: {has_djvutxt} {install hint if no} {if unprocessed_count > 0} **Unprocessed Files** ({unprocessed_count}): - {file1} - {file2} ... Run `/literature --convert` to convert all, or `/literature --scan` to see details. {end if} {if entry_count > 0 and md_count != entry_count} **Index Health**: {entry_count} indexed entries, {md_count} markdown files — run `/literature --validate` to check consistency. {end if} ``` --- ## Mode: Scan Find PDF/DJVU files lacking corresponding markdown conversions. ### Scan Step 1: Check Directory Same as Status Step 1 — exit gracefully if directory missing. ### Scan Step 2: Find Unprocessed Files ```bash unprocessed=() for src in $(find "$lit_dir" -name "*.pdf" -o -name "*.djvu" 2>/dev/null | sort); do basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//') if ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; then unprocessed+=("$src") fi done ``` ### Scan Step 3: Get Page Counts For each unprocessed file, get page count via pdfinfo: ```bash for src in "${unprocessed[@]}"; do ext="${src##*.}" if [ "$ext" = "pdf" ]; then if [ "$has_pdfinfo" = "yes" ]; then pages=$(pdfinfo "$src" 2>/dev/null | grep "^Pages:" | awk '{print $2}') else pages="unknown" fi elif [ "$ext" = "djvu" ]; then if [ "$has_djvutxt" = "yes" ]; then # djvused can get page count: djvused -e n file.djvu pages=$(djvused -e n "$src" 2>/dev/null || echo "unknown") else pages="unknown (djvutxt not installed)" fi fi echo "- $src ($pages pages)" done ``` ### Scan Step 4: Display Results ``` ## Literature Scan Results **Unprocessed Files** ({count}): - {file1} ({N} pages) - {file2} ({N} pages) ... **Tool Status**: - pdftotext: {status} - djvutxt: {status} {install hint if unavailable} **Next Steps**: - Convert all: `/literature --convert` - Convert one: `/literature --convert path/to/file.pdf` ``` If no unprocessed files found: ``` ## Literature Scan Results All source files have been converted. No unprocessed PDFs or DJVUs found. **Files**: {N} PDFs, {M} DJVUs — all converted **Index**: {entry_count} entries in index.json Run `/literature --validate` to check index.json consistency. ``` --- ## Mode: Validate Check index.json against the filesystem for stale entries, missing files, and token count drift. ### Validate Step 1: Load Index ```bash if [ ! -f "$index_file" ]; then echo "## Literature Validation" echo "" echo "No index.json found at $index_file."