image-mining

# Image Mining > *"I mine pixels for atoms. Reality is just compressed resources."* > > *"Every image is a lode. Every pixel, potential ore."* **Image Mining** extends the Kitchen Counter's DECOMPOSE action to **images**. Your camera isn't just a recorder — it's a **PICKAXE FOR VISUAL REALITY**. --- ## 📑 Index **Quick Start** - [The Core Insight](#the-core-insight) - [Preferred Mode: Native LLM Vision](#preferred-mode-native-llm-vision) **Operation Modes** - [When to Use Remote API](#when-to-use-remote-api) - [What Can Be Mined](#what-can-be-mined) **Extensibility** - [Extensible Analyzer Pipeline](#extensible-analyzer-pipeline) - [Leela Customer Models](#leela-customer-models) - [Adding Your Own Analyzer](#adding-your-own-analyzer) **Protocols** - [YAML Jazz Output Style](#yaml-jazz-output-style) - [How Mining Works](#how-mining-works) - [Character Recognition](#character-recognition) - [Multi-Look Mining](#multi-look-mining) **Reference** - [Depth Levels](#depth-levels) - [Resource Categories](#resource-categories) - [Example Outputs](#example-outputs) --- ## The Core Insight ``` 📷 Camera Shot → 🖼️ Image → ⛏️ MINE → 💎 Resources ``` Just like the Kitchen Counter breaks down: - `sandwich` → `bread + cheese + lettuce` - `lamp` → `brass + glass + wick + oil` - `water` → `hydrogen + oxygen` **Images** can be broken down into: - `ore_vein.png` → `iron-ore × 12` + `stone × 8` - `forest.png` → `wood × 5` + `leaves × 20` + `seeds × 3` - `treasure_pile.png` → `gold × 100` + `gems × 15` - `sunset.png` → `orange_hue × 1` + `warmth × 1` + `nostalgia × 1` --- ## Preferred Mode: Native LLM Vision > *"The LLM IS the context assembler. Don't script what it does naturally."* When mining images, **prefer native LLM vision** (Cursor/Claude reading images directly): ``` ┌─────────────────────────────────────────────────────────────────┐ │ NATIVE MODE (PREFERRED) │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Cursor/Claude already has: │ │ ✓ The room YAML (spatial context) │ │ ✓ Character files (who might appear) │ │ ✓ Previous mining passes (what's been noticed) │ │ ✓ The prompt.yml (what was intended) │ │ ✓ The whole codebase (cultural references) │ │ │ │ Just READ the image. The context is already there. │ │ No bash commands. No sister scripts. Just LOOK. │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` ### Why Native Beats Remote API | Aspect | Native (Cursor/Claude) | Remote API (mine.py) | |--------|------------------------|----------------------| | Context | Already loaded | Must be assembled | | Prior mining | Visible in chat | Passed via stdin | | Room context | Just read the file | Python parses YAML | | Synthesis | LLM does it naturally | Script concatenates | | Iteration | Conversational | Re-run command | ### When to Use Remote API Use `mine.py` or remote API calls when: - **Multi-perspective mining** — different models see different things! - **Batch processing** — mining 100 images overnight - **CI/CD** — automated pipelines with no LLM orchestrator - **Rate limiting** — your LLM can't do vision but can call one that does **Multi-perspective is the killer use case:** Claude sees narrative, GPT-4V sees objects, Gemini sees spatial relationships. Layer them all for rich interpretation. Even then, have the **orchestrating LLM assemble the context**: ``` ┌─────────────────────────────────────────────────────────────────┐ │ REMOTE API WITH LLM ASSEMBLY │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ 1. LLM reads context files (room, characters, prior mining) │ │ 2. LLM synthesizes: "What to look for in this image" │ │ 3. LLM calls remote vision API with image + synthesized prompt│ │ 4. LLM post-processes response into YAML Jazz │ │ │ │ The SMART WORK happens in the orchestrating LLM. │ │ Remote API just does vision with good instructions. │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` ### Native Mode Workflow ```bash # DON'T do this: python mine.py image.png --context room.yml --characters chars/ --prior mined.yml # DO this (in Cursor/Claude): # 1. Read the image # 2. Read room.yml, character files, prior -mined.yml # 3. Look at the image with all that context # 4. Write YAML Jazz output ``` The LLM context window IS the context assembly mechanism. Use it. --- ## What Can Be Mined Image mining works on **ANY visual content**, not just AI-generated images: ``` ┌─────────────────────────────────────────────────────────────────┐ │ MINEABLE SOURCES │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ 🎨 AI-Generated Images │ │ - DALL-E, Midjourney, Stable Diffusion outputs │ │ - Has prompt.yml sidecar with generation context │ │ │ │ 📸 Real Photos │ │ - Phone camera, DSLR, scanned prints │ │ - No prompt — mine what you see │ │ │ │ 📊 Graphs and Charts │ │ - Data visualizations, dashboards │ │ - Extract trends, outliers, relationships │ │ │ │ 🖥️ Screenshots │ │ - UI states, error messages, configurations │ │ - Mine the interface, not just pixels │ │ │ │ 📝 Text Images │ │ - Scanned documents, handwritten notes, signs │ │ - OCR + semantic extraction │ │ │ │ 📄 PDFs │ │ - Documents, papers, invoices │ │ - Cursor may already support — try it! │ │ │ │ 🗺️ Maps and Diagrams │ │ - Architecture diagrams, floor plans, mind maps │ │ - Extract spatial relationships │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` ### Source Examples **Generated Image (has context):** ```yaml postal: type: text to: "visualizer" body: "Take a photo of that ore vein on the wall" attachments: - type: image action: generate prompt: "Rich iron ore vein in cavern wall, glittering..." ``` **Real Photo (mine what you see):** ```yaml postal: type: text to: "miner" body: "Here's a photo of the treasure room" attachments: - type: image action: upload source: "camera_roll" file: "treasure-room.jpg" ``` **Screenshot (extract UI state):** ```yaml # Mine the error dialog resources: error-type: "permission-denied" affected-file: "/etc/passwd" suggested-action: "run as sudo" stack-depth: 3 ``` **Graph (extract data relationships):** ```yaml # Mine the sales chart resources: trend: "upward" peak-month: "december" anomaly: "march-dip" yoy-growth: "23%" ``` **All become mineable resources!** --- ## Extensible Analyzer Pipeline > *"Different images need different tools. The CLI is a pipeline, not a monolith."* The `mine.py` CLI supports pluggable analyzers that run before, during, or after LLM vision: ``` ┌─────────────────────────────────────────────────────────────────┐ │ ANALYZER PIPELINE │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ 1. PRE-PROCESSORS │ │ resize, normalize, enhance, format conversion │ │ │ │ 2. CUSTOM ANALYZERS (parallel or sequential) │ │ ├── pose-detection (MediaPipe, OpenPose) │ │ ├── object-detection (YOLO, Detectron2) │ │ ├── ocr-extraction (Tesseract, PaddleOCR) │ │ ├── face-analysis (expression, demographics) │ │ └── leela-customer-models (your trained models!) │ │ │ │ 3. LLM VISION │ │ Receives ALL prior results as context │ │ Synthesizes semantic interpretation │ │ │ │ 4. POST-PROCESSORS │ │ format, validate, merge into final YAML Jazz │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` ### Example: Multi-Analyzer Pipeline ```bash mine.py fashion-shoot.jpg \ --analyzer pose-detection \ --analyzer face-analysis \ --analyzer leela://acme/gesture-classifier \ --depth philosophical ``` This runs: 1. **pose-detection** — Extracts body keypoints, gesture classification 2. **face-analysis** — Detects expressions, demographics 3. **leela://acme/gesture-classifier** — Customer's trained model from Leela registry 4. **LLM vision** — Gets ALL the above as context, synthesizes final interpretation ### Leela Customer Models Pull customer-specific models trained on the Leela platform: ```bash # From Leela model registry mine.py widget-photo.jpg --analyzer leela://customer-id/defect-detector-v3 # Local model file mine.py widget-photo.jpg --analyzer ./models/my-classifier.pt ``` Output merges into the mining YAML: ```yaml leela_analysis: model: "acme-widget-defect-v3" customer: "acme-corp" detections: - class: "hairline_crack" confidence: 0.91 severity: "minor" location: "top_left_quadrant" ``` ### Adding Your Own Analyzer ```python # analyzers/my_analyzer.py def analyze(image_path: str, config: dict) -> dict: """Run analysis, return structured data for YAML output.""" # Your model inference here return { "my_analysis": { "detected": ["thing1", "thing2"], "confidence": 0.95 } } def can_handle(image_path: str, context: dict) -> bool: """Return True if this analyzer should run on this image.""" # Auto-detect logic, or return False for explicit-only return "manufacturing" in context.get("tags", []) ``` Register in `analyzers/registry.yml`: ```yaml analyzers: my-analyzer: module: "analyzers.my_analyzer" auto-detect: true requires: ["torch", "my-model-package"] ``` ### Why Pipeline Beats Monolith | Approach | Pros | Cons | |----------|------|------| | **Monolith** | Simple | Can't add domain models | | **Pipeline** | Extensible, composable | Slightly more complex | The LLM is great at semantic synthesis, but it can't run your custom pose detection model. The pipeline lets each tool do what it's best at: - **Custom models** → Precise detection, trained on your data - **LLM vision** → Semantic interpretation, narrative synthesis - **Together** → The best of both worlds --- ## YAML Jazz Output Style > *"Comments are SEMANTIC DATA, not just documentation!"* YAML Jazz is the output format for mining results. Structure provides the backbone; comments provide the insight. ### The Rules 1. **COMMENT LIBERALLY** — Every insight deserves a note 2. **Inline comments** for quick observations 3. **`notes:` fields** for longer thoughts 4. **Capture confidence, hunches, metaphors** 5. **Think out loud** — the reader benefits from your reasoning ### Example Output ```yaml # Mining results for treasure-room.jpg # Depth: full | Provider: openai/gpt-4o resources: gold: quantity: 150 # Piled in mounds — not scattered, PLACED confidence: 0.85 # Torchlight glints clearly off the metal notes: | Mix of Roman denarii and medieval florins. Centuries of accumulation. This isn't a king's orderly treasury — this is a thieves' hoard. Generations of stolen wealth, piled and forgotten. The dust layer says nobody's touched it in ages. danger: intensity: 0.7 # Not immediate, but PRESENT confidence: 0.75 # Hard to see into the corners sources: - "Skeleton in corner — previous seeker, didn't make it" - "Shadows too dark for natural torchlight — something absorbs" - "Dust undisturbed except ONE trail — something still comes here" notes: "This hoard is guarded. Or cursed. Probably both." nostalgia: intensity: 0.4 # Whisper of lost civilizations confidence: 0.6 # Subjective, but the coins evoke it notes: "Who were they? Where did this come from? All gone now." dominant_colors: - name: "treasure-gold" hex: "#FFD700" coverage: 0.4 # Catches the eye first — that's the point - name: "shadow-purple" hex: "#2D1B4E" coverage: 0.3 # Where the danger lives implied_smells: - dust # Centuries of it - old metal # Copper, bronze, the tang of coins - something rotting # Not recent, but not ancient either exhausted: false mining_notes: | Rich lode for material and philosophical mining. The image is ABOUT greed and its costs. The skeleton says everything. # Meta-observation: This image wants to be a warning. # "Here lies what you seek — and what happens when you find it." ``` ### Why Comments Matter An uncommented extraction is like a song without soul. The best mining results read like **poetry annotated by a geologist**. When you mine, capture: - Why you estimated that quantity - What visual cues led to this inference - What's uncertain, what surprised you - Metaphors that capture the essence --- ## How Mining Works ### Step 1: ANALYZE (LLM scans for resources) The LLM looks at the image AND checks what resources are **currently requested** by the logistics network: ```yaml analyze: image: "treasure-room.jpg" # LLM knows what's NEEDED from logistics requesters logistics_context: active_requests: - { item: "gold", requester: "forge/", needed: 100 } - { item: "gems", requester: "jewelry-shop/", needed: 50 } - { item: "iron-ore", requester: "smelter/", needed: 200 } # LLM identifies what CAN BE MINED that matches requests analysis_prompt: | Look at this image. What resources can you identify? Prioritize resources that match these requests: {requests} For each resource, estimate quantity available. ``` ### Step 2: INSTANTIATE (Resource map attached to image) The LLM returns a resource mapping that gets stored ON the image: ```yaml image: id: "treasure-room-photo" file: "treasure-room.jpg" type: mineable-image # RESOURCE MAP (instantiated by LLM analysis) resources: gold: total: 150 # Total available remaining: 150 # Not yet mined per_turn: 10 # Can extract 10 per turn gems: total: 45 remaining: 45 per_turn: 5 ancient-coins: total: 30 remaining: 30 per_turn: 3 rare: true # Bonus find! dust: total: 500 remaining: 500 per_turn: 50 value: low # Metadata analyzed_at: "2026-01-10T14:30:00Z" exhausted: false ``` ### Step 3: MINE (Progressive extraction, N per turn) Each turn, you can mine resources from the image: ```yaml action: MINE target: "treasure-room-photo" # This turn's extraction (limited by per_turn rates) result: extracted: - item: gold quantity: 10 # per_turn limit destination: "forge/" - item: gems quantity: 5 destination: "jewelry-shop/" # Image state updated image_state: resources: