| name | update-references |
| description | Find, download, read, and summarize research papers related to the work in the current directory. Searches arXiv, OpenAlex, Semantic Scholar, and Google Scholar, with a hierarchical mode (hubs, surveys, citation snowball). Maintains a CSL-YAML reference database (references.yaml) compatible with pandoc's --bibliography flag, so the same file serves both the skill and document builds. Triggers: update references, refresh the bibliography, find papers, related work, literature review, what's been published on, recent papers, cite sources for, background reading, summarize this paper, search scholar. |
Update references (research paper search)
This skill turns "find me what's been written about this" into a small,
repeatable pipeline: read what the user is working on, search arXiv (and, for
off-arXiv work, Semantic Scholar or Google Scholar) for related papers,
download the relevant ones, read
them, write a summary per paper, and keep a Citation Style Language (CSL) YAML
database so work is never
repeated and versions stay current.
The standing interests are LLMs, AI agents, finite state machines, and
declarative agent patterns — the territory of spindle
(github.com/petar-djukic/spindle), a Go state-machine engine for agentic loops.
Lean toward those topics, but always sharpen the search using whatever the
current working directory is actually about.
Where things live
This skill is context-aware. It runs from the directory the user is working in,
and it stores its outputs there — not in a fixed global location. Resolve paths
like this:
- Database:
references.yaml under the current working directory, unless
an existing one is already nearby (look for a references.yaml at or above
the working directory and reuse it). Pass the chosen path as --db to every script call so
search, fetch, and record all agree. If only a legacy arxiv/papers.yaml
exists, the scripts read it transparently and convert on next write.
- PDFs:
<db-dir>/pdfs/ (transient — they exist to be read; they don't need
to be committed). The script resolves this relative to the database, not
the current directory, so PDFs land next to the db even when run from
elsewhere.
- Papers (markdown):
<db-dir>/papers/ — fetch converts each PDF to
markdown automatically (best-effort via pymupdf4llm), preserving headings,
tables, and math. Papers are readable in any editor or tool.
- Summaries:
<db-dir>/summaries/, one file per paper.
The PDF, markdown, and summary of a paper share one human-friendly stem:
<Family>-<Year>-<title-slug>-<source>-<id> (for example
Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052). The
<source>-<id> tail is arxiv-<id>vN, doi-<doi>, or scholar-<key>, so any
file traces back to its source at a glance. This differs from the pandoc
citation key, which stays short — just <family>-<year> (see the database
format). repair renames files created under the older naming scheme.
- Manual-download list:
<db-dir>/downloads-needed.md — regenerated by
scholar.py pending, a checklist of papers that could not be downloaded
(paywalled or behind a login) with a landing URL for each. Derived from the
database, so it is disposable, not hand-maintained. An entry carrying
download: deferred is kept off the checklist (it lists compactly at the
bottom instead) — set it on metadata-only entries the current work does not
cite, and remove it when a citation pass starts citing the entry.
If the user names a directory or an existing database, use that instead. Because
every output path is derived from --db, passing an absolute --db keeps all
artifacts together regardless of the working directory.
The database format
The database is CSL (Citation Style Language) YAML — by default a bare list with
no root key. Each entry has
standard CSL fields that pandoc understands (id, type, title, author,
container-title, URL, issued) plus skill-internal fields (status,
version, pdf_path, arxiv_id, etc.) that pandoc ignores. This means the
file is directly usable as pandoc --bibliography references.yaml with no
conversion step.
A references: root key is also accepted, because that is the form pandoc's own
documentation shows and what an existing pandoc bibliography usually looks like;
so is the skill's earlier papers: key. Whichever shape a file arrives in is
preserved when the skill writes it back, so pointing the skill at a bibliography
you already maintain does not reformat it. Any other shape is refused rather
than read as an empty database, and no command will write an empty database over
a file that had entries.
Every write goes out in yq's normal form: the skill runs yq -i '.' over the
file after serializing it. PyYAML and yq indent nested sequences differently,
and on a bibliography of a few hundred entries that disagreement turns any
hand-edit made with yq into a diff the length of the file. One tool decides the
format, and it is the one the operator edits with. Without yq on PATH the
database is still written and still valid, in PyYAML's style, and the skill says
so on stderr — the next yq edit will reformat it whole.
An entry looks like:
- id: lee-2026
type: article
title: "Meta-Harness: End-to-End Optimization of Model Harnesses"
author:
- family: Lee
given: Yoonho
container-title: arXiv preprint arXiv:2603.28052
URL: https://arxiv.org/abs/2603.28052
issued:
year: 2026
arxiv_id: "2603.28052"
version: 1
status: downloaded
pdf_path: pdfs/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.pdf
md_path: papers/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.md
The id field is a pandoc citation key (used as @lee-2026 in markdown),
generated from the first author's family name and the year — just
<family>-<year>, with a letter suffix on collision (lee-2026a). It is
deliberately short; the richer provenance lives in the file names. The
arxiv_id field is the base arXiv identifier used for deduplication and
version tracking.
Entries fetched through OpenAlex additionally carry two marker blocks that
later steps use (pandoc ignores them):
openalex_id: W2158864412
ranking:
cited_by: 5671
references_count: 52
fwci: 102.6
citation_percentile: 100.0
venue: IEEE Transactions on Software Engineering
institutions: [Massachusetts Institute of Technology]
institution_types: [education]
retrieved: 2026-07-15
discovery:
method: survey-references
via: "Wang et al. 2024 autonomous-agents survey"
ranking answers "how important, by whose measure, as of when"; discovery
answers "how did this paper enter the corpus". Both persist so the reading
step can prioritize and the summaries can cite them.
Running the scripts
The Python scripts run in a pixi-managed environment that ships beside the
skill: pixi.toml and pixi.lock sit at the root of the agent directory
(the parent of this skill's skills/ folder). Provision it once per machine —
the agent does this on repo open — with the preflight:
<agent-dir>/scripts/ensure-env.sh
Then run every script through pixi. For brevity the commands below write
$RUN for the wrapper:
RUN="pixi run --manifest-path <skill>/../../pixi.toml python"
where <skill>/../../ resolves to the agent directory holding pixi.toml.
This replaces the older pip install --user path — no packages are installed
into the ambient interpreter.
The workflow
0. Reconcile the database
Run this first, every time the skill is invoked. The goal is a complete
database in the correct naming convention before anything else happens:
$RUN <skill>/scripts/arxiv.py --db <db-path> reconcile
reconcile (the same command as repair) walks the whole database and the
pdfs/ directory and makes disk and db agree:
- an entry missing its markdown is converted;
- pdf/markdown/summary files on the older naming scheme are renamed to the
human-friendly stem (paths updated in the db; citation ids left intact);
- a PDF in
pdfs/ that no entry references is imported — its metadata is
recovered from an arXiv id in the filename (fetched from arXiv), or, failing
that, from the PDF's embedded title/author (recorded status: needs-review
with year nd). Anything unrecoverable is listed in
<db-dir>/unregistered-pdfs.md with a ready-to-run ingest command.
It is idempotent and prints counts (converted, renamed, collisions,
imported, needs_review, unregistered). A non-zero collisions means two
entries resolved to the same filename — usually a duplicated citation id — so
the rename was refused and those files were left on their old names rather than
one overwriting the other. Fix the duplicate and re-run.
The same count covers a refused import: an orphan that identifies as a paper
the database already holds is a duplicate copy, not a new paper, so it is left
in pdfs/ under its own name instead of being renamed over the copy already
there. Delete it, or correct the entry it duplicates. To register an
unidentified PDF by hand, give ingest the metadata and it creates the entry:
$RUN <skill>/scripts/scholar.py --db <db-path> ingest --file pdfs/<file>.pdf \
--title "Exact Title" --authors "Given Family" --year 2024
After reconciling, refresh the Obsidian tags and source links on every summary
(safe to run each time — it only changes what is stale):
$RUN <skill>/scripts/keywords.py --db <db-path> tag
1. Understand the current work first
Before searching, read what's in the working directory — a draft paper, notes,
an outline, existing summaries. The search is only as good as its query, and the
query should come from the actual problem the user is working on, not just the
standing topic list. If the directory is empty or the intent is unclear, ask the
user what angle they care about (one question, then proceed).
2-3. Search, dedupe, fetch
Per-backend mechanics — source order, dedupe rules, and how each fetch path
behaves — are in search-and-fetch.md.
In short: search the hierarchical backends, dedupe against references.yaml
before fetching anything, fetch the PDF where a direct link exists, and record
a metadata-only entry where it does not.
4. Read and summarize
When several papers await reading, use the ranking markers to order them:
pivotal first (high fwci/cited_by), and read a survey before the papers
discovered through it (discovery.via tells you which those are).
Read the markdown conversion that fetch produced — its path is in the
md_path field of the fetch output and the db entry (under
<db-dir>/papers/). If the markdown file is missing (e.g. an older fetch
before markdown conversion was available), run repair first to regenerate
it. Then write a summary file following
references/summary-template.md exactly. The summary's job is to let the user
decide, in two minutes, whether to cite the paper — so the "Relevance to the
current work" section carries the weight. Tie findings back to the current draft
and, where it fits, to spindle's state-machine / declarative-agent view. Prefer
the paper's own numbers over adjectives. An honest "low relevance" beats a
stretch.
Name the summary with the same stem as the paper's PDF and markdown so its
source is obvious:
<db-dir>/summaries/<Family>-<Year>-<title-slug>-<source>-<id>.md (the stem is
the basename of the entry's md_path, with the summaries/ directory). If you
are unsure of the exact stem, run repair afterward — it renames any summary
to match.
5. Record it
Close the loop so the database reflects reality:
$RUN <skill>/scripts/arxiv.py --db <db-path> record --id 2310.12345 \
--summary-file summaries/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.md \
--topics llm agents fsm declarative-agents \
--relevance "One line on why it matters to this work."
This flips the entry to status: summarized. The database is now the
single source of truth: re-running a search later will mark these papers
known and skip them.
Then apply Obsidian tags and source links to the new summaries:
$RUN <skill>/scripts/keywords.py --db <db-path> tag
This walks the db and, for every summary, writes a normalized tags: block
into the frontmatter — merging the paper's declared keywords (or a frequency
fallback when it declares none) with the entry's topics, plus a paper root
tag — and adds a **Source:** line linking the local PDF, the converted full
text, and the source URL. It also drops a one-time PDF back-link at the top of
each converted paper. Tags a human added are preserved; --dry-run previews
without writing; a second run is a no-op. Because it walks the whole db, it
also tags any older summary that predates this step, so a corpus converges.
6. Report back
Summarize what was found: how many candidates, how many were new vs. already
known, which were summarized, and a one-line takeaway per paper with a link to
its summary file. Point out the two or three most relevant to the current work.
If any papers came back metadata-only, run scholar.py pending and hand the
user downloads-needed.md so they can fetch those PDFs and return them via
ingest.
Repair / reconcile
repair and reconcile are the same command — the database reconcile run at
step 0. Run it any time to bring disk and db back into agreement:
$RUN <skill>/scripts/arxiv.py --db <db-path> reconcile
It walks every entry, re-converts any missing md_path from the PDF on disk,
migrates legacy text_path entries, and renames any pdf/markdown/summary still
on the older naming scheme to the human-friendly stem (updating pdf_path,
md_path, summary_file; citation id values left unchanged). It then scans
pdfs/ for files no entry references and imports them (arXiv-id recovery, then
embedded-metadata needs-review, then the unregistered-pdfs.md list). It is
idempotent — correctly-named, tracked files are left alone — and no step ever
moves a file onto one that already exists: a rename or an import that would
replace another paper is refused, counted as a collision, and explained on
stderr. Prints a JSON summary (checked, converted, renamed, collisions,
imported, needs_review, unregistered, skipped).
Dependencies
The pixi environment (see "Running the scripts") supplies PyYAML (required),
pymupdf4llm (PDF-to-markdown conversion preserving headings, tables, and
math), and pypdf (the plain-text fallback). It is provisioned by
ensure-env.sh; no pip install is needed. If the conversion libraries were
somehow absent, fetch still downloads the PDF and just skips the conversion.
yq (the Go implementation, brew install yq) normalizes the database on
write; it is optional, and its absence costs formatting, not correctness.
Everything else is Python stdlib plus the arXiv public API — no key needed. Be
a good citizen: the script already retries with backoff; don't hammer the API
with huge --max values in a tight loop.
OpenAlex needs no key — set OPENALEX_MAILTO (any email) for the polite
pool's higher rate limits. Google Scholar search requires a SerpAPI key (same key as the idea-factory
job-search skill). Semantic Scholar needs no key — the public Graph API is
open, though shared and rate-limited; an optional SEMANTIC_SCHOLAR_API_KEY
(or --api-key) raises the limit.
