research-literature-kg

name	research-literature-kg
description	Build and analyze knowledge graphs from research literature. Automated pipeline: arxiv search → entity extraction → KG construction → vector embeddings → semantic search → skill pattern extraction. Use when user asks to analyze papers, build research knowledge bases, find related work, or extract reusable patterns from academic literature.

Research Literature Knowledge Graph

Description

Automated pipeline for building and analyzing knowledge graphs from academic research literature. Integrates arxiv search, entity extraction, vector embeddings, and graph algorithms to discover patterns and extract reusable skill patterns.

Activation Keywords

• research literature KG
• build knowledge graph from papers
• paper analysis pipeline
• arxiv to KG
• 文献知识图谱
• 科研论文分析
• 论文知识库
• extract skills from papers

Tools Used

• exec: Run Python scripts, kg_tool CLI, arxiv API queries
• web_search: Search for related research
• web_fetch: Fetch paper content from arxiv
• read: Read existing skills, KG schema
• write: Store results, update memory
• feishu_bitable_app: Store structured paper metadata (optional)

Workflow

Phase 1: Literature Collection

1. Define research scope:

   • Primary topic (daily focus)
   • Secondary topic (weekly theme)
   • Keywords for search

2. Search arxiv (HTTPS required, rate limit ~3 req/sec):

   query = f'cat:{category}+AND+all:{keywords}'
   url = f'https://export.arxiv.org/api/query?search_query={query}&max_results=10&sortBy=submittedDate&sortOrder=descending'
   # Wait 3-5s between requests. On 429, wait 10s+ and retry.
   # Check cached JSON files first (e.g. scripts/*_papers.json) before hitting API.
   

3. Parse results: Extract title, authors, abstract, arxiv_id, category, published_date

Phase 2: KG Construction

1. Database: kg.db at /Users/hiyenwong/.openclaw/workspace/kg.db
2. Schema: See references/kg-db-schema.md for verified column names and types
3. Insert entities: Use execute_code with sqlite3 — columns are title, url, content, authors, published_date, category, source
4. No description column — use content for abstracts

Phase 3: Vector Embeddings

1. Generate embeddings — 1024-byte BLOBs (256 float32 values) stored as vector_data
2. Fallback: hash-based vectors when embedding model unavailable:

   import struct, hashlib
   values = [struct.unpack('f', hashlib.md5(f"{text}{i}".encode()).digest()[:4])[0] for i in range(256)]
   vector_bytes = struct.pack('256f', *values)
   

Phase 4: Graph Analysis

1. PageRank: Find important papers
2. Louvain: Detect research clusters
3. Semantic search: Find related papers

Phase 5: Pattern Extraction

1. Identify patterns: Look for recurring methods, frameworks, approaches
2. Extract skills: Use skill-extractor skill
3. Create new skill: Use skill-creator skill

Database Schema (Actual — verified 2026-06-14)

CRITICAL: The schema evolved from earlier versions. The kg_entities table is now a generic entity table, NOT a flat paper table. There is also a separate arxiv_papers table.

-- kg_entities: generic entity table
CREATE TABLE kg_entities (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    type TEXT NOT NULL,
    description TEXT,
    metadata TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    source TEXT DEFAULT '',
    created_date TEXT DEFAULT '',
    UNIQUE(name, type)
);

-- arxiv_papers: dedicated paper table
CREATE TABLE arxiv_papers (
    id TEXT PRIMARY KEY,
    title TEXT, url TEXT, abstract TEXT, authors TEXT,
    published TEXT, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- kg_vectors: embedding + text
CREATE TABLE kg_vectors (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    entity_id INTEGER,
    embedding BLOB,
    text TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (entity_id) REFERENCES kg_entities(id)
);

-- kg_edges: edge list
CREATE TABLE kg_edges (
    source TEXT, target TEXT, relation TEXT,
    weight REAL DEFAULT 1.0,
    PRIMARY KEY (source, target, relation)
);

⚠️ No title, content, url columns in kg_entities. Use name, type, description, metadata. ⚠️ No vector_data column — kg_vectors uses embedding BLOB + text TEXT. ⚠️ kg_edges uses TEXT source/target, not INTEGER foreign keys.

Correct Insert Patterns

# Into arxiv_papers (preferred for papers)
cursor.execute('INSERT OR IGNORE INTO arxiv_papers (id, title, url, abstract, published) VALUES (?,?,?,?,?)',
    (arxiv_id, title, url, abstract, published))

# Into kg_entities (generic)
cursor.execute('INSERT OR IGNORE INTO kg_entities (name, type, description, metadata, source, created_date) VALUES (?,?,?,?,?,?)',
    (title[:200], 'arxiv_paper', summary[:300],
     json.dumps({'category': cat, 'arxiv_id': arxiv_id}),
     'arxiv-cron', published[:10]))

Also see references/kg-db-schema.md for complete schema.

Error Handling

httpx Proxy Syntax

When using httpx with a proxy, use proxy="http://..." (singular, string). Do NOT use proxies={} dict syntax — it causes Client.__init__() got an unexpected keyword argument 'proxies'.

with httpx.Client(timeout=30, proxy="http://127.0.0.1:7890") as client:
    resp = client.get(url)

kg.db Schema Mismatch

The kg_entities table schema changed from a flat paper table to a generic entity table (name, type, description, metadata columns). Always sqlite3 kg.db ".schema" before inserting. There is now a separate arxiv_papers table for paper data. Insert into both for full integration.

arxiv API Rate Limit (429)

• Always use HTTPS (HTTP may be blocked)
• Wait 3-5s between requests
• On 429: wait 10+ seconds before retry
• Fallback: Check for pre-cached JSON files in scripts/ directory (cron jobs save results there)

Embedding Generation Failure

• Fall back to hash-based 256-float vectors (see Phase 3)
• No sentence-transformers required for fallback

Vector Dimension Mismatch

• kg.db vectors are 1024 bytes (256 float32)
• Ensure struct.pack('256f', *values) format matches

Related Skills

• arxiv-search: Paper search details
• skill-extractor: Pattern extraction
• skill-creator: Skill creation
• feishu-bitable: Alternative storage

Notes

• KG persists across sessions via SQLite
• Vectors enable semantic search
• Weekly topics rotate through domains
• Daily quantum mechanics focus