arxiv-digest

name	arxiv-digest
description	Fetch, rank, and summarize new arXiv papers as a personalized digest with preference and collaborator signals. Use for requests such as "what's new on arxiv", "papers today", weekly or monthly preprint updates, arbitrary arXiv date ranges, paper recommendations, abstract collection, PDF archiving, and follow-up reading.

arXiv Digest

Produce a complete, auditable candidate set first. Rank and select it with bundled scripts. Use the model only to summarize the selected papers.

Core workflow

1. Resolve the installed skill directory and run scripts from its scripts/ directory.
2. Load preferences and the researcher profile from the storage root.
3. Fetch once and require a complete manifest.
4. Rank and select with arxiv_rank.py.
5. Read abstracts only for selected papers and write a concise digest.
6. Save ranking/selection artifacts and mark delivered dates read when appropriate.

Never rank an unsorted pool manually. Never present a partial fetch as complete.

Storage

Resolve storage in this order:

1. --storage-dir
2. ARXIV_DIGEST_HOME
3. XDG_DATA_HOME/arxiv-digest
4. ~/.local/share/arxiv-digest

Use researcher_profile.json and arxiv_preferences.json. Do not store durable user data inside the skill installation.

If either file is missing, read the relevant schema before creating it:

• preferences: references/preference-schema.md
• researcher/network profile: references/profile-schema.md
• storage migration or read-state work: references/storage-guide.md

Fetch a personalized candidate set

Use arxiv_fetch.py for today, rolling windows, months, and personalized date ranges:

python3 scripts/arxiv_fetch.py \
  --prefs ~/.local/share/arxiv-digest/arxiv_preferences.json \
  --period 30d \
  --output fetch_30d.json \
  --manifest fetch_30d.manifest.json

Accepted periods: today, 7d, 30d, Nd, recent, YYYY-MM-DD, YYYY-MM, and YYYY-MM-DD:YYYY-MM-DD. Rolling Nd windows contain exactly N inclusive calendar dates.

Default digest candidates are canonical new submissions:

• exclude replacements;
• exclude RSS/HTML cross-list announcements;
• for Atom date-range queries, exclude papers whose primary category is outside the requested category set;
• deduplicate by versionless arXiv ID.

The fetcher refuses failed date/category chunks by default. Use --allow-partial only when the user explicitly accepts an incomplete result, and label the digest partial.

Use --include-cross-lists or --include-replacements only when requested. Use web search only after the script fails; verify every fallback record and disclose incomplete coverage.

Fetch an exhaustive explicit range

Use arxiv_fetch_range.py for archival abstract exports or PDF downloads:

python3 scripts/arxiv_fetch_range.py 2026-07-28 2026-08-10 \
  --category astro-ph.CO \
  --output arxiv_2026-07-28_to_2026-08-10.jsonl

START and END are inclusive UTC first-submission dates. Repeat --category for OR semantics. PDF download is opt-in through --pdf-dir and guarded by --max-pdfs. Preserve its cache, retry, PDF validation, and atomic-output behavior.

Rank and select

Run the deterministic ranker after a successful fetch:

python3 scripts/arxiv_rank.py \
  --papers fetch_30d.json \
  --manifest fetch_30d.manifest.json \
  --prefs ~/.local/share/arxiv-digest/arxiv_preferences.json \
  --profile ~/.local/share/arxiv-digest/researcher_profile.json \
  --period 30d \
  --output ranked_30d.json \
  --selection-output selected_30d.json

Treat its scores, reasons, exclusions, tier membership, and preflight result as authoritative. Do not substitute substring matching or model-selected rankings. Read references/scoring-guide.md only when changing scoring behavior or explaining a score.

Default output caps are:

Window	Top	Solid	Boundary	Total cap
today	5	5	2	12
7 days	8	7	3	18
30 days	10	7	3	20

These are caps, not quotas. Show fewer when evidence is weak. For large windows, provide the shortlist in chat and link the full ranked artifact. Expand only when the user asks.

Write the digest

Use three sections: Top Picks, Solid Matches, and Boundary Expanders. Preserve ranked order within each section.

For each paper include:

#01 Full Paper Title (2601.12345) -- Author A, Author B, Author C
One or two abstract-grounded sentences describing the problem, method, and main result.
Why it matches: concrete ranking reasons.
Link: arXiv

Use the original complete title. Do not infer claims absent from the abstract. Mark recognized co-authors. Boundary papers may add Extends toward:. Avoid generic praise.

Begin with a compact provenance line containing the inclusive window, retrieval timestamp, categories, fetched/candidate counts, exclusions, and manifest completeness. If partial, put PARTIAL DIGEST at the top.

Feedback and read state

Ask which paper IDs were useful only when feedback would improve the next digest. Apply explicit likes/dislikes to preferences; do not infer durable preferences from silence.

After delivery:

python3 scripts/storage_manager.py mark-read --date YYYY-MM-DD

Use storage_manager.py unread-range for unread-window requests. storage_manager.py status must remain read-only.

Full-paper follow-up

When the user selects a paper, resolve its digest index to arXiv ID and fetch the full PDF. Use the pdf skill for claim, equation, figure, or section-level analysis. Use jupyter-notebook only for requested derivations or experiments. Clearly distinguish paper claims from interpretation.

Failure rules

• Empty complete window: report that it is empty; do not silently broaden it.
• Any failed manifest chunk: stop unless partial output was explicitly accepted.
• Invalid/missing ID or abstract: exclude it and report the exclusion count.
• Network and web fallback both fail: report failure; never fabricate papers.
• Large pool: keep ranking on disk and load only selected paper records into model context.