doc-convert

name	doc-convert
description	Convert project documents to readable formats and manage knowledge base. Use when: user needs to convert PDF/Excel/CMSIS-Pack files, build knowledge base indexes, or perform semantic search on technical documents. Supports batch conversion, caching, and RAG retrieval.

DOC-CONVERT - Document Conversion & Knowledge Base Skill

Automatically convert project documents and build searchable knowledge base with vector + keyword hybrid retrieval.

When to Use

✅ USE this skill when:

• "Convert these PDFs to markdown"
• "Build knowledge base from technical documents"
• "Search for register configuration in datasheets"
• "Extract text from Excel requirements"
• "Extract CMSIS pack files"
• "Setup RAG retrieval for embedded projects"
• "Query knowledge base with semantic search"

When NOT to Use

❌ DON'T use this skill when:

• Already have text/markdown files → skip conversion
• Need to generate code → use fae skill instead
• Only need to read a single file → use read tool directly
• Simple keyword search → use grep or text search

Input/Output Paths

Fixed Paths:

# Input: Source documents
INPUT_DIR="/Users/sinomcu/.openclaw/workspace/fae_input/sources/"

# Output: Converted files and indexes
CACHE_DIR="/Users/sinomcu/.openclaw/workspace/fae_input/cache/"
├── text/          # Converted text files (.md, .csv)
└── extracted/     # Extracted archives (.pack contents)

INDEX_DIR="/Users/sinomcu/.openclaw/workspace/fae_input/indexes/"
├── vector/        # Vector embeddings (ChromaDB)
└── keyword/       # Keyword index (Whoosh)

Supported Formats

Format	Extension	Converter	Output
PDF	.pdf	converter.py (内置 pdf_to_md.py)	.md (Markdown)
Excel	.xlsx, .xls	converter.py (内置 xlsx2csv.py)	.csv (per sheet)
CMSIS Pack	.pack	converter.py (内置 pack_extractor.py)	Extracted folder
Netlist	.net	手动读取	不自动转换，由助手直接读取原始文件

⚠️ Note on Schematic PDFs: If the PDF is a scanned image (non-vector), text extraction will fail (0 bytes). In this case, provide pin assignment information manually (Excel, CSV, or markdown table) rather than relying on netlist auto-conversion.

Usage Examples

Example 1: Convert Single PDF

User: "Convert the device datasheet PDF to markdown"

Assistant: (uses doc-convert skill)

python3 /Users/sinomcu/.openclaw/workspace/skills/doc-convert/scripts/pdf_to_md.py \
  /Users/sinomcu/.openclaw/workspace/fae_input/sources/MC60F3136_数据手册.pdf \
  /Users/sinomcu/.openclaw/workspace/fae_input/cache/text/

Output: cache/text/MC60F3136_数据手册.md

---

Example 2: Batch Convert All Documents

User: "Convert all documents in the sources folder"

Assistant: (uses doc-convert skill)

python3 /Users/sinomcu/.openclaw/workspace/skills/doc-convert/converter.py \
  --input /Users/sinomcu/.openclaw/workspace/fae_input/sources/ \
  --output /Users/sinomcu/.openclaw/workspace/fae_input/cache/ \
  --recursive

Output: All documents converted to appropriate formats

---

Example 3: Build Knowledge Base

User: "Build knowledge base from the converted documents"

Assistant: (uses doc-convert skill)

python3 /Users/sinomcu/.openclaw/workspace/skills/doc-convert/kb_manager.py build

Output: Vector + keyword indexes created in fae_input/indexes/

💡 Tip: kb_manager.py now uses absolute paths internally, so it can be run from any directory without needing cd.

---

Example 4: Query Knowledge Base

User: "How to configure TIM6 prescaler?"

Assistant: (uses doc-convert skill)

python3 /Users/sinomcu/.openclaw/workspace/skills/doc-convert/kb_manager.py query "How to configure TIM6 prescaler?"

Output: Relevant chunks from datasheets with semantic search

---

Example 5: Check Knowledge Base Status

User: "Show knowledge base status"

Assistant: (uses doc-convert skill)

python3 /Users/sinomcu/.openclaw/workspace/skills/doc-convert/kb_manager.py status

Output:

📊 知识库状态

向量索引：156 个分块
关键词索引：156 个分块
文档缓存：8 个文档
源文件：12 个

---

Directory Structure

skills/doc-convert/
├── SKILL.md                 # This file
├── converter.py             # Batch conversion tool
├── kb_manager.py            # Knowledge base manager
├── requirements.txt         # Python dependencies
├── scripts/                 # Conversion scripts (used by converter.py)
│   ├── pdf_to_md.py        # PDF → Markdown
│   ├── xlsx2csv.py         # Excel → CSV
│   ├── pack_extractor.py   # CMSIS Pack extraction
│   └── netlist_to_md.py    # .net → Markdown (已停用，保留供后续修复)
├── retriever/               # Retrieval engine
│   ├── __init__.py
│   ├── embedder.py         # Embedding
│   ├── vector_store.py     # ChromaDB vector store
│   ├── keyword_index.py    # Whoosh keyword index
│   └── hybrid_search.py    # Hybrid retrieval
└── kb/                      # Knowledge base config
    └── config.json          # Configuration

Workflow

Phase 1: Document Conversion

1. Check cache - Skip if already converted
2. Convert documents - PDF→MD, Excel→CSV, Pack→Extracted
3. Store in cache - Save to fae_input/cache/text/

Phase 2: Knowledge Base Build

1. Load converted files - Read from cache
2. Chunk documents - Split into semantic chunks (800 chars)
3. Generate metadata - Auto-extract from filename:
   • mcu_model - e.g., "MC30P6060", "MS8046" (regex: [A-Z]{2}\d+[A-Z0-9]*)
   • mcu_family - e.g., "MC30", "MS80" (prefix of model)
   • doc_type - e.g., "数据手册", "用户手册", "原理图", "需求规格", "库文件"
   • doc_id - MD5 hash of file path (16 chars)
   • filename - Original filename
   • chunk_idx - Sequential index within document
4. Generate embeddings - Use SentenceTransformer paraphrase-multilingual-MiniLM-L12-v2
5. Build indexes - Vector (ChromaDB with metadata) + Keyword (Whoosh with metadata fields)
6. Save indexes - Store in fae_input/indexes/

Phase 3: Query & Retrieval

1. Auto-detect MCU model - Extract model name from query (e.g., "MC30P6060" → metadata filter)
2. Embed query - Convert question to embedding
3. Metadata filtering - Narrow search scope before vector comparison
4. Hybrid search - Vector + Keyword retrieval (configurable weights: BM25=0.8, Vector=0.2)
5. Reciprocal Rank Fusion - Merge results with weighted scoring
6. Return context - Provide top-k chunks to LLM for RAG

Integration with Other Skills

FAE Skill

FAE → doc-convert (convert) → cache/ → FAE (read) → generate code

Knowledge Base (RAG)

KB → doc-convert (convert+index) → indexes/ → KB (query) → RAG context

Commands Reference

Conversion

# Batch convert
python3 converter.py -i fae_input/sources/ -o fae_input/cache/ -r

# Single file
python3 converter.py -i device.pdf -o cache/text/

Knowledge Base

# Build index
python3 kb_manager.py build

# Query
python3 kb_manager.py query "your question"

# Status
python3 kb_manager.py status

# Clear
python3 kb_manager.py clear

Dependencies

Install required packages:

cd /Users/sinomcu/.openclaw/workspace/skills/doc-convert
pip3 install -r requirements.txt

If the model is missing and the user tries to build/query:

1. 检查 paraphrase-multilingual-MiniLM-L12-v2/ 是否存在
2. 如果不存在，告知用户需要下载 (~400MB)
3. 运行 python3 scripts/download_model.py
4. 下载完成后继续原来的 build/query 命令

这样 OpenClaw 读取 SKILL.md 时就会按照这个流程自动处理。同时也加了 embedder.py 里的降级逻辑——即使 SKILL.md 的指令没触发，代码层面也会自动从 HuggingFace 下载到 ~/.cache/huggingface/。

Key dependencies:

• pymupdf4llm - PDF to Markdown conversion
• pandas - Excel processing
• chromadb - Vector database
• whoosh - Keyword search engine
• sentence-transformers - Embedding models

Performance Tips

🚀 Performance Optimizations (Implemented)

• ✅ Parallel Processing - Multi-core CPU utilization for document conversion

  • Default: 4 workers, adjustable with --workers flag
  • Speedup: 2-4x for multiple files

• ✅ Batch Embedding - Process chunks in batches during KB build

  • Default batch size: 64, adjustable with --batch-size
  • Speedup: 3-5x for embedding generation

• ✅ Incremental Build - Skip unchanged documents using content hash

  • Automatic hash validation for each file
  • Speedup: 80-90% for subsequent builds
  • Use --full-rebuild to force complete rebuild

• ✅ Smart Caching - Content-based cache validation

  • MD5 hash tracking in .cache_meta.json
  • Automatic cache invalidation on file changes

📋 Usage Examples

# Parallel conversion with 8 workers
python3 converter.py -i sources/ -o cache/ -r -p -w 8

# Batch embedding with custom size
python3 kb_manager.py build --batch-size 128

# Force full rebuild (ignore incremental)
python3 kb_manager.py build --full-rebuild

# Sequential mode (disable parallel)
python3 converter.py -i sources/ -o cache/ -r -s

⚙️ Configuration

Add to kb/config.json:

{
  "batch_size": 64,
  "parallel_workers": 4,
  "use_incremental_build": true,
  "bm25_weight": 0.8,
  "vector_weight": 0.2
}

Metadata Filtering

Auto-Detection

When querying, the system automatically detects MCU model names and applies metadata filtering:

# Query contains "MC30P6060" → auto-filters to mcu_model=MC30P6060
python3 kb_manager.py query "MC30P6060 PWM configuration"

This narrows the search from 18,935 chunks across 132 documents down to ~110 chunks from MC30P6060 only, dramatically improving relevance.

Metadata Schema

Each chunk carries these metadata fields:

Field	Example	Source
mcu_model	"MC30P6060"	Extracted from filename via regex
mcu_family	"MC30"	Prefix of model name
doc_type	"用户手册"	Keyword matching on filename
doc_id	"857e26c43b180ab5"	MD5 hash of file path
filename	"MC30P6060用户手册_V1.9.pdf"	Original filename
chunk_idx	0, 1, 2...	Sequential index

Configurable Weights

Add to kb/config.json:

{
  "bm25_weight": 0.8,
  "vector_weight": 0.2
}

• bm25_weight=0.8 (default): Prioritize exact keyword matches — better for technical queries with model numbers, register names, hex values
• vector_weight=0.2: Semantic similarity as fallback — catches paraphrased questions like "how to set up timer" matching "定时器配置"

Query Examples

# Auto-filters to MC30P6060 (model detected)
python3 kb_manager.py query "MC30P6060 timer configuration"

# Auto-filters to MS8046
python3 kb_manager.py query "MS8046 ADC resolution"

# No model detected → searches all documents
python3 kb_manager.py query "how to configure PWM dead time"

---

Error Handling

• ❌ File not found → Ask user to check path
• ❌ Unsupported format → List supported formats
• ❌ Conversion failed → Show error and suggest alternatives
• ❌ Index build failed → Check dependencies and disk space
• ❌ Query failed → Verify index exists and is not corrupted

Notes

• All conversions are lossless for text content
• Images in PDFs are not extracted by default (text-only mode)
• Schematic PDFs that are scanned images will produce 0-byte output → use .net netlist files instead
• Excel formulas are evaluated (values only, not formulas)
• CMSIS Pack extraction preserves directory structure
• Vector embeddings require Ollama running locally
• First build downloads embedding model (~274 MB)

🔌 Handling Schematic Netlist Files (.net)

⚠️ 注意： .net 网表文件不会被 converter.py 自动转换。原因：

1. KiCad 格式的 .net 文件是嵌套 S-表达式结构，netlist_to_md.py 的正则解析存在 bug（无法提取 node 连接，导致所有 network connections = 0）
2. 转换后的错误数据进入知识库会导致 FAE 生成代码时引脚信息错误
3. converter.py 的 convert_all() 仅支持 .pdf、.xlsx、.xls、.pack 四种格式

推荐做法：

• 手动提供引脚分配表（Excel/CSV/Markdown 表格）放入 fae_input/sources/，由 doc-convert 正常转换进知识库
• 或者将网表中的关键引脚信息以结构化格式单独存放（如 fae_input/pinout/）
• netlist_to_md.py 文件保留在 scripts/ 目录，待修复后可重新启用

---

This skill combines document conversion and knowledge base management for efficient RAG-based retrieval in embedded projects.