paper-ingestion

name

paper-ingestion

description

How the arXiv/paper-to-repo ingestion seam and the reproduction job/manifest cache work in DaxAlgo Terminal. Covers IPaperIngestClient (Null/Http, subprocess + HTTP/JSON, 127.0.0.1 only) resolving a paper URL to PaperRef + candidate RepoRef[], and the SQLite ReproJobStore cloned from ArchiveManifestStore (cache keyed by arXiv id + repo commit + config hash, sha256 artifact refs, retention). Use when touching IPaperIngestClient, ReproJobStore, HttpPaperIngestClient, the repro sidecar endpoints, or wiring AddPaperResearch().

Paper Ingestion + Job Cache

The front of the Paper Lab pipeline: resolve a paper URL to something runnable, and persist the reproduction jobs/results so the same paper is never re-run needlessly. Both pieces are deliberately boring — they clone existing, proven patterns.

Ingestion seam — IPaperIngestClient

ResolveAsync(string url, ct) -> (PaperRef paper, IReadOnlyList<RepoRef> repos).

• Clone the shape of IAiAnalystClient: IsAvailable; methods never throw; failure surfaces as an empty/Failed result, not an exception. Hot-swap Null↔Http via IOptionsMonitor.
• NullPaperIngestClient (default) lets the app build/run with no sidecar (degrade-gracefully, same as the AI analyst). HttpPaperIngestClient mirrors HttpAiAnalystClient: named HttpClient, snake_case JSON, fold-all-failures, sidecar bound to 127.0.0.1 only.
• Resolution logic (arXiv metadata + PDF + discovering the code repo via Papers-with-Code / README / arXiv links) lives in the Python sidecar (tools/python-ml/) — no native deps in the C# build.
• PaperRef carries arXiv id + title + the paper URL (becomes ResearchPaperUrl on the saved strategy). RepoRef carries the git URL + a pinned commit (the cache key depends on it).

Job / manifest store — ReproJobStore

Persisted in SQLite, cloned from ArchiveManifestStore (namespace TradingTerminal.Infrastructure.MarketData.Archive, lives in Infrastructure):

• internal sealed, private write connection, EnsureSchema, micros timestamps, JSON-serialized blob/artifact refs with sha256, retention / soft-delete (MarkLocalDeleted equivalent).
• Cache key = (PaperRef.ArxivId, RepoRef.Commit, ConfigHash) → a FindCovering-style lookup before submit returns the existing ReproJob/ReproResult instead of spawning a new container.
• On startup, LocalReproOrchestrator reads back unfinished (Queued/Running) jobs and requeues them — the store is what makes jobs survive an app restart.

Wiring

• AddPaperResearch(config) (in ResearchReproServiceCollectionExtensions) registers the store + ingest client (Null by default, Http when ResearchReproOptions points at a sidecar) + the orchestrator + the runner (by SandboxOptions.Kind).
• Called from App.xaml.cs alongside AddPaperLab().

Layer placement

• IPaperIngestClient, PaperRef, RepoRef, ReproJob, ReproSpec, ReproResult → Core/Research/.
• HttpPaperIngestClient, NullPaperIngestClient, ReproJobStore → Infrastructure/Research/.

What NOT to do

• Don't throw across IPaperIngestClient — fold failures (no sidecar, bad URL, no repo found) into an empty/Failed result.
• Don't bind the sidecar to 0.0.0.0 — 127.0.0.1 only.
• Don't skip the commit pin — without it the cache key is meaningless and reproductions aren't deterministic.
• Don't put resolution/scraping logic in the C# build — it belongs in the sidecar.