| name | hfpclawer-academic-integrity |
| description | Feed a paper draft to automatically extract citations, verify each one against local + Semantic Scholar + OpenAlex, detect fabricated references, and generate a structured integrity report. Designed for researchers, reviewers, and literature-survey authors.
|
| category | research |
| author | Li Shen |
| version | 1.0.0 |
| metadata | {"hermes":{"tags":["academic-integrity","citation","verification","hallucination","research","audit"],"related_skills":["hfpclawer-paper-search","hfpclawer-citation-audit"],"requires":[{"hermes_tool":"web_extract"},{"mcp_server":"arxiv-search (for citation_graph)"}]}} |
hfpclawer Academic Integrity Audit
Take a paper draft (text, URL, or PDF) and verify every citation against
local FTS5 → Semantic Scholar → OpenAlex. Spot fabricated references,
misattributed authors, and wrong publication years.
Who this is for: Researchers who need to fact-check citations in
LLM-generated papers, peer review submissions, or their own drafts
before submission.
Overview
User: "Check this paper draft for citation integrity"
│
┌────────────┴────────────┐
▼ ▼
Extract citations Load paper (text/URL/MEDIA pdf)
│ │
└────────────┬────────────┘
▼
┌─────────────────────────┐
│ For each citation: │
│ L1: Local FTS5 (1ms) │ ← hfpclawer audit
│ L2: Semantic Scholar │ ← hfpclawer citation_audit_s2
│ L3: OpenAlex │ ← hfpclawer citation_audit_oa
│ L4: Citation Graph │ ← MCP arxiv-search citation_graph
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ Integrity Report │
│ [OK] VERIFIED 12 │
│ [?] SUSPECTED 2 │
│ [NF] NOT_FOUND 0 │
│ [ERR] ERROR 1 │
│ ⚠ FABRICATED 1 │ ← title+author mismatch => hallucination
└─────────────────────────┘
Prerequisites
pip install hfpclawer>=0.5.0
Optional but recommended:
Quick Start
1. Verify a Draft Paper (most common)
Paste the paper text (including its references section) and ask:
> Run academic integrity audit on this draft.
[paste paper text with references]
# Output:
# ┌─────────────────────────────────┐
# │ Academic Integrity Audit Report │
# ├─────────────────────────────────┤
# │ [OK] VERIFIED "Fourier Neural Operator" (Li, 2020) │
# │ [OK] VERIFIED "Physics-Informed Neural Networks" (2019) │
# │ [?] SUSPECTED "Deep Learning for PDEs" │ ← no author/date
# │ [NF] NOT_FOUND "Novel Superconductor at 300K" (Smith 2025)│ ← possible hallucination
# │ [NF] NOT_FOUND "Quantum PDE Solver" (Chen 2024) │ ← possible hallucination
# └─────────────────────────────────┘
2. Audit a Paper from a URL (arXiv / blog / preprint)
> Check this paper for citation integrity:
> https://arxiv.org/abs/2010.08895
The skill will:
- web_extract the paper text
- Parse the references section
- Run each citation through L1→L2→L3→L4
- Return a formatted integrity report
3. Audit a Local PDF or Markdown File
Provide the file via Hermes Agent:
> Read and audit citations in MEDIA:/path/to/paper.pdf
The skill reads the text, extracts citations, and reports.
How It Works
Citation Extraction
The extractor finds these patterns:
"Title" (Author, Year) — author-year-title formatAuthor (Year) "Title" — title-quoted formatarXiv:XXXX.XXXXX — arxiv IDs[1] Author. "Title". Journal. Year. — numbered bibliography (basic)
Verification Cascade
| Level | Source | Speed | What It Confirms |
|---|
| L1 | Local FTS5 (arxiv_meta.db) | ~1ms | arXiv ID + title match |
| L2 | Semantic Scholar API | ~200ms | Title + authors + year |
| L3 | OpenAlex API | ~200ms | Title + authors + venue + DOI |
| L4 | Citation Graph (MCP) | ~500ms | Citing/cited-by validation |
Status Definitions
| Status | Icon | Meaning |
|---|
| VERIFIED | ✅ | Found in ≥1 source with title similarity ≥0.70 |
| SUSPECTED | ⚠️ | Possible match (0.40-0.69) — may be real but uncertain |
| NOT_FOUND | ❌ | No match in any source — potential hallucination |
| FABRICATED | 🚨 | NOT_FOUND + no author match + unlikely title — likely hallucination |
| ERROR | 🔧 | Source unavailable (DB not found, rate limited) |
Integrity Report Example
═══ Academic Integrity Audit ═══
Paper: "A Survey of Neural PDE Solvers"
Total citations scanned: 22
Time: 2026-05-20T14:30:00Z
─────────────────────────────────
✅ VERIFIED 15/22 (68%)
⚠️ SUSPECTED 4/22 (18%)
❌ NOT_FOUND 2/22 (9%) ← review these
🚨 FABRICATED 1/22 (5%) ← "Quantum Fourier Neural Net" (Li 2025)
🔧 ERROR 0/22 (0%)
─────────────────────────────────
Verification breakdown by source:
L1 Local FTS5: 11 verified
L2 Semantic Scholar: +6 verified (cascaded)
L3 OpenAlex: +2 verified (cascaded)
─────────────────────────────────
FABRICATED citations:
🚨 "Quantum Fourier Neural Net" (Li 2025)
No arXiv match, no S2 match, no OpenAlex match
Title sounds AI-generated (no such paper exists)
─────────────────────────────────
Recommendations:
- Remove 1 fabricated citation
- Check 4 suspected citations manually (2 may be preprints w/o arXiv ID)
- Add DOI/arXiv IDs to 3 citations for future verifiability
Common Pitfalls
-
arXiv-only papers may not appear in S2 or OpenAlex for 1-2 weeks after posting.
If a very recent paper shows NOT_FOUND, try L4 (citation graph) or check manually.
-
Title substring matching can produce false VERIFIED for short titles.
e.g. "Neural Operator" → matches "Fourier Neural Operator" (accurate)
but "Deep Learning" → too generic, will NOT_FOUND or false-match.
-
S2 rate limits without API key: ~1 req/s, ~100 req/day. Set S2_API_KEY.
-
Local FTS5 requires arxiv-metadata-service clone.
Without it, the cascade starts at L2 (slower but works).
-
Citation extractor is regex-based, not ML-based. Some citation styles
(especially Chicago notes-bibliography) may not be parsed correctly.
For best results, provide the references section in plain author-title-year format.
Verification Checklist
