research-repo

name

research-repo

description

Scaffold or audit an entire research project repository organized around its source library. Use when starting, structuring, organizing, or reviewing a research repo. Build sources/{og,md,unprocessed}, references.bib, a PDF-to-Markdown converter, a repo-local process-source skill, AGENTS.md, .gitignore, .venv, and the relevant analysis/manuscript/review folders; or audit an existing layout. Not for one-PDF intake or publication replication packages.

Research Repository Scaffold & Audit

Scope and organizing principle

This skill sets up a new research repository, or audits an existing one, around a single organizing principle: the source library is the spine of the project, and everything else grows from it.

sources/og/ holds the original PDFs and documents — archival, gitignored (size + copyright), rarely read directly. sources/md/ holds their Markdown conversions, tracked in git: this is the LLM-readable knowledge base, and sources are always read from here, not from the PDFs. sources/unprocessed/ is the drop zone where new PDFs land until they are named, converted, and filed. sources/references.bib keys the corpus to the manuscript — one entry per source, author+year resolvable to its md/ file, so a \cite key maps to a document actually read. A conversion script (scripts/convert-sources.sh, OpenDataLoader PDF) and a process-source intake skill keep the three in sync. Get that spine wrong — PDFs scattered, no Markdown, a bib that does not match what was read — and every downstream skill (literature-review, citation-check, fact-check, paper-tex) is working on sand.

Everything else — data/, analysis scripts/, manuscript/ or paper/, review/ + codebook/, figures/, tables/, replication/ — grows outward from that spine, conditioned on what kind of project this is.

Instructions

Step 1. Resolve the target and decide the mode

Use the path supplied with the invocation; otherwise default to the current working directory and confirm once before writing into it. Then decide the mode from what is there:

• Scaffold mode — the directory is empty, is not yet a git repo, or has no sources/ and no manuscript. You will build the spine and the outward folders.
• Audit mode — the directory already has research content (a sources/ tree, a manuscript, analysis scripts, a bib). You will read what exists, compare it against the convention, and report present / partial / missing. Never overwrite an existing file without explicit confirmation; offer to create only what is missing.

If a repo is half-built (e.g. PDFs exist but no md/, or an og/ with no convert script), that is audit mode with scaffolding gaps — report the gaps and offer to fill them.

Step 2. Identify the project archetype

The bibliography is the spine of every project; the outward folders — and whether a PDF corpus is even appropriate — depend on what the project is. Detect the archetype from concrete signals in the directory, or ask one question if it is genuinely ambiguous.

Archetype	Concrete signals	Outward folders to scaffold
Literature / systematic review	many sources/md/ files; a coding protocol; an inventory CSV	review/ (inventory CSV + per-source annotations), codebook/ (evaluation protocol), data/ (open datasets collected)
Empirical analysis / paper	data files (*.csv, *.dta) beside estimation scripts (*.R, *.do, *.py); a manuscript heading to a journal	data/, scripts/ (analysis), paper/ or manuscript/, figures/, tables/, replication/
Corpus / digitization	scanned images / OCR output; a large derived corpus	data/ (raw + derived corpus), scripts/ (OCR/cleanup pipeline), plus sources/ for the methods literature
Lightweight / theory paper	a manuscript and a .bib but no archived PDFs and no large dataset	a paper/ or manuscript/ folder and its .bib; no sources/ PDF corpus unless the user asks for one
Mixed	several of the above	scaffold the union; keep one shared bibliography

Most projects have a literature corpus and head to a paper; when in doubt, scaffold the spine plus data/, scripts/, and the manuscript folder, and let the project grow the rest. But some repos legitimately keep no PDF corpus — a theory paper, or a short empirical paper with a hand-maintained .bib. There the spine is the bibliography alone: audit the .bib wherever it lives (often paper/references.bib) and offer — never impose — a sources/og/sources/md tree.

Step 3. Scaffold the sources spine

For any project that reads and archives a literature, this is the core. (For a corpus-free repo — the lightweight/theory archetype in Step 2 — scaffold only the bibliography and the manuscript folder, then skip to Step 4.) Create:

<repo>/
├── sources/
│   ├── og/                 # original PDFs/docs — gitignored
│   ├── md/                 # Markdown conversions — tracked
│   ├── unprocessed/        # drop zone for new PDFs
│   ├── references.bib      # bibliography (one entry per source)
│   └── README.md           # the convention, written down
├── scripts/
│   └── convert-sources.sh  # PDF/docx → Markdown (OpenDataLoader PDF + pandoc)
├── .agents/
│   └── skills/
│       └── process-source/
│           └── SKILL.md    # repo-local per-PDF intake skill
├── AGENTS.md               # canonical project conventions for Codex
├── CLAUDE.md -> AGENTS.md  # optional cross-platform compatibility symlink
├── .gitignore
└── .venv/                  # python env for opendataloader-pdf (created in setup)

First, make sure the repo is under version control — the whole tracked/gitignored split (og/ ignored, md/ tracked) only takes effect once git exists:

git rev-parse --git-dir >/dev/null 2>&1 || git init

Write the templates from the Templates section below. A newly written file is mode 0644, so make the conversion script executable immediately after writing it — otherwise ./scripts/convert-sources.sh fails with "permission denied", both in the smoke test below and in the two places the generated scaffold tells the user to run it (sources/README.md step 2 and the repo-local process-source skill's step 3):

chmod +x scripts/convert-sources.sh

Then set up the conversion environment — but verify the toolchain first, because convert-sources.sh runs under set -euo pipefail and will abort opaquely if the Java backend is missing:

command -v python3 && python3 --version
command -v java && java -version          # OpenDataLoader PDF needs Java 11+; if absent, stop and tell the user to install it
python3 -m venv .venv
.venv/bin/pip install --upgrade pip opendataloader-pdf

Optionally link CLAUDE.md to AGENTS.md so Claude Code reads the same canonical instructions:

test -e CLAUDE.md || ln -s AGENTS.md CLAUDE.md

Finally, smoke-test the pipeline — on an empty og/ it should print Nothing new to convert.:

./scripts/convert-sources.sh

Step 4. Build outward (archetype-conditioned)

Create only the outward folders the archetype calls for (Step 2). Leave them empty with a one-line purpose in the README — the project fills them. Common folders and what they hold:

• data/ — datasets. Raw inputs and derived analysis-ready files. Large/binary data is gitignored (see template); document restricted data rather than committing it.
• scripts/ — analysis and utility code (the convert script already lives here).
• review/ + codebook/ (review projects) — the inventory CSV (one row per source), per-source prose annotations, and the coding protocol that governs them. This is where the sources/md/ corpus gets turned into structured evidence.
• manuscript/ or paper/ — the draft. Cites sources/references.bib. Hand this to paper-tex to typeset.
• figures/, tables/ — generated outputs. Pair with the figures / tables skills.
• replication/ — the public reproducibility package. Do not hand-build this here — when the paper is ready, call replication-package to scaffold it properly.
• logs/, meetings/ — session logs and notes; usually local-only (gitignored).

Step 5. Document the intake pipeline

The spine only stays trustworthy if every new source flows through the same pipeline. Write it into sources/README.md and the repo-local process-source skill, and state it to the user:

drop in sources/unprocessed/  →  identify (title/authors/year/venue)
  →  rename to author-year-slug, move to sources/og/
  →  ./scripts/convert-sources.sh  (→ sources/md/<name>.md)
  →  add a BibTeX entry to sources/references.bib
  →  (review projects: add inventory row + annotation)

The per-PDF mechanics belong to process-source, which the scaffolded .agents/skills/process-source/SKILL.md points at. This skill guarantees the pipeline exists and is wired; process-source runs it.

Step 6. The BibTeX contract

The bibliography — sources/references.bib, or wherever the project keeps it (e.g. paper/references.bib) — is the contract between the manuscript and the knowledge base: a \cite key is only trustworthy if it resolves to a source you have actually read and filed in sources/md/. Keep it honest:

• One entry per source, added at intake (Step 5), never in a batch at the end.
• Citekey follows the project's own key style (e.g. hainmueller_hopkins_yamamoto_2014 or hainmueller-etal-2014-conjoint). Do not impose a scheme on an existing project — but keep every key author+year resolvable to its md/ filename, because citation-check and fact-check map keys to source files by author and year. The filename uses author-year-slug; the bib key can differ in punctuation but must point at the same work.
• sources/missing.bib (optional, recommended) — a second bib for works that are cited but have no PDF in og/: paywalled articles with no preprint mirror, books, dissertations, authoritative web resources (a standard, a DOI registry). Record why each is missing and how to acquire it, so "cited but unfiled" stays visible instead of silently absent. A needs_updates.md-style flag file serves the same purpose if the project already keeps one.

Step 7. Audit checklist (audit mode)

Read the existing repo and report each item as present / partial / missing. If the repo keeps no PDF corpus by design (the lightweight/theory archetype), mark the PDF-corpus items n/a for this archetype, not missing, and audit the bibliography wherever it actually lives — discover it with find . -name '*.bib' -not -path './.venv/*' -not -path './.git/*'.

Spine

• sources/og/, sources/md/, sources/unprocessed/ all exist.
• sources/og/ is gitignored; sources/md/ is tracked (the LLM-readable corpus must be in git).
• Every og/*.pdf has a matching md/*.md (no unconverted sources). List the orphans.
• Every md/*.md has a references.bib entry, and every bib entry resolves to a source (flag bib entries with no file and files with no bib — drift in either direction).
• Filenames follow author-year-slug (lowercase, hyphens, ≤3 authors then firstauthor-etal).
• sources/references.bib exists; keys are author+year resolvable to filenames.
• sources/README.md documents the convention.

Detection recipes (run from the repo root; approximate starting points, not gospel):

# Orphan PDFs — in og/ but never converted to md/
comm -23 <(cd sources/og && ls *.pdf 2>/dev/null | sed 's/\.pdf$//' | sort) \
         <(cd sources/md && ls *.md  2>/dev/null | sed 's/\.md$//'  | sort)

# Bib keys — compare against md/ stems by author+year to catch drift in either
# direction (a source with no entry, an entry with no source). Keys may use
# underscores where filenames use hyphens.
grep -oE '^@[a-zA-Z]+\{[^,]+' sources/references.bib | sed 's/^@[a-zA-Z]*{//' | sort

# Filenames that violate author-year-slug (lowercase-hyphen, four-digit year, slug)
ls sources/og | grep -vE '^[a-z0-9]+(-[a-z0-9]+)*-(19|20)[0-9]{2}-[a-z0-9-]+\.(pdf|docx)$'

Pipeline

• scripts/convert-sources.sh exists and points at this repo's sources/og → sources/md.
• .venv/ with opendataloader-pdf is set up (or setup is documented); Java 11+ available.
• A process-source command (or the global skill) is available for intake.

Repo conventions

• AGENTS.md documents the project and its conventions; any CLAUDE.md compatibility file agrees with it.
• .gitignore excludes sources/og/, .venv/, large data, secrets, and OS/editor cruft.
• Outward folders match the archetype and have a stated purpose (README).
• No secrets, answer keys, or restricted data in the tracked path.

Step 8. Report

Output a short report:

1. Mode (scaffold / audit) and detected archetype.
2. The tree created (scaffold) or the present/partial/missing diff (audit).
3. For audits: orphan PDFs, bib drift, naming violations, and pipeline gaps, each with a one-line fix.
4. The next three actions (typically: set up .venv and run convert-sources.sh; drop the first PDFs in unprocessed/ and run $process-source; fill in AGENTS.md placeholders).

Do not commit unless asked.

Templates

sources/README.md

# Sources

The source library is the spine of this project. Read sources from `md/`, never the PDFs.

## Structure

- `og/` — original PDFs and documents (gitignored; not pushed)
- `md/` — LLM-readable Markdown conversions (tracked in git)
  - `<name>_images/` folders hold figures/tables extracted from the PDF
- `unprocessed/` — drop zone for new PDFs awaiting intake
- `references.bib` — bibliography (one entry per source)
- `missing.bib` — cited works with no PDF available (paywalled, books, web resources)

## Naming convention

`author-year-slug`, lowercase with hyphens:

- Up to three authors → use all three; more → `firstauthor-etal`
- Then year, then a 2–4 word content slug
- Example: `hainmueller-hopkins-yamamoto-2014-causal-inference-conjoint`

The PDF in `og/` carries this name through to `md/`.

## Adding a source

1. Drop the PDF (or `.docx`) into `sources/unprocessed/`
2. Run the `process-source` command (or do it by hand): rename to `author-year-slug`,
   move to `og/`, run `./scripts/convert-sources.sh`, add a `references.bib` entry
3. The convert script only touches files without a matching `.md` — re-running is safe
4. Commit the new `md/<name>.md` (and its `_images/`); `og/` stays local

## Requirements

- Python venv at  with  installed
  (for ) · Java 11+ (OpenDataLoader PDF backend)





PDFs are awkward for LLMs. The Markdown versions preserve text, headings, and
references in a form agents can read directly, enabling source-grounded analysis
and writing. OpenDataLoader PDF: https://github.com/opendataloader-project/opendataloader-pdf

scripts/convert-sources.sh

#!/usr/bin/env bash
# Convert new PDFs/docx in sources/og/ to Markdown in sources/md/.
# Skips files that already have a corresponding .md.
# Usage: ./scripts/convert-sources.sh          # incremental (new files only)
#        ./scripts/convert-sources.sh --all     # reconvert everything
set -euo pipefail

PROJECT_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
SOURCES="$PROJECT_ROOT/sources/og"
OUTPUT="$PROJECT_ROOT/sources/md"
VENV="$PROJECT_ROOT/.venv"

FORCE=false
[[ "${1:-}" == "--all" ]] && FORCE=true

mkdir -p "$OUTPUT"
source "$VENV/bin/activate"

converted=0

# PDFs via OpenDataLoader PDF
for pdf in "$SOURCES"/*.pdf; do
    [ -f "$pdf" ] || continue
    base="$(basename "$pdf" .pdf)"
    if [[ "$FORCE" == false && -f "$OUTPUT/$base.md" ]]; then
        continue
    
     
    opendataloader-pdf  --format markdown --output-dir 
    converted=$((converted + ))



 docx  /*.docx; 
    [ -f  ] || 
    base=
     [[  ==  && -f  ]]; 
        
    
     
    pandoc  -t markdown -o 
    converted=$((converted + ))


 [[  -eq 0 ]]; 
     

     

For large corpora, a parallel variant (xargs -P N over opendataloader-pdf) speeds bulk conversion; the incremental script above is the default and is safe to re-run.

.agents/skills/process-source/SKILL.md

---
name: process-source
description: Process new papers from sources/unprocessed through the repository's complete source-intake pipeline. Use when adding PDFs or documents to this research project.
---

# Process unprocessed sources

Check `sources/unprocessed/` for new PDFs/docs. For each one:

1. **Identify** — read it; determine title, authors, year, venue.
2. **Rename & move** — to `sources/og/` using `author-year-slug` (see `sources/README.md`).
3. **Convert** — run `./scripts/convert-sources.sh` to generate `sources/md/<name>.md`.
4. **BibTeX** — add an entry to `sources/references.bib` (author+year resolvable to the filename).
5. **(Review projects)** — classify per `codebook/` and add an inventory row + annotation.
6. **Clean up** — remove the file from `sources/unprocessed/`.

Flag anything needing human review (paywalled, poor OCR, data-integrity concerns).
If a cited work has no obtainable PDF, record it in `sources/missing.bib` instead.

Prefer this project's conventions; defer to the global `process-source` skill for the generic mechanics.

AGENTS.md

# <Project name>

<One paragraph: what this project is, who it is for, and its current phase.>

## Repository structure

The source library is the spine. See `sources/README.md` for the intake convention.

- `sources/md/` — LLM-readable Markdown corpus. **Always read sources from here, not the PDFs.**
- `sources/unprocessed/` — drop zone for new PDFs.
- `sources/references.bib` — bibliography; the citation contract for the manuscript.
- `scripts/` — conversion and analysis code.
- <archetype folders: review/ + codebook/  |  data/ + paper/ + figures/ + tables/ + replication/>

## Conventions

- Source naming: `author-year-slug` (lowercase, hyphens).
- New sources flow through the intake pipeline (`$process-source`), never dropped straight into `md/`.
- Read from `sources/md/`; cite from `sources/references.bib`.
 Large data, original PDFs, , and secrets are gitignored.

.gitignore

# Original PDFs and large binaries — keep local, don't push
sources/og/
*.pdf
*.xlsx
*.dta
*.7z
*.zip

# Large data files (keep code, not data)
data/**/*.csv
data/**/*.tsv
data/**/*.png

# Python environment
.venv/

# Sensitive — never commit
.env
.env.*
*.pem
*.key
*ANSWER_KEY*
*do_not_share*

# Local-only working areas
logs/

# macOS / R / editors
.DS_Store
.Rhistory
.vscode/
.idea/

Adjust the data rules to the project: if a dataset is public and small, track it; if it is large or restricted, gitignore it and document it in the README. The default keeps original PDFs out of git (copyright + size) while keeping the md/ corpus tracked.

When to reach for this skill vs. siblings

• research-repo (this) — create or audit the working repository's structure, anchored on sources/. Use at project start, or when a repo has grown messy.
• process-source — run the per-PDF intake into the structure this skill creates: the repo-local skill scaffolded in Step 3, or a global equivalent. Use every time a new paper arrives.
• replication-package — scaffold or audit the public reproducibility package built from the finished paper, near submission. This skill creates the working repo; that one creates the archive.
• literature-review / citation-check / fact-check — consumers of the sources/md/ + references.bib knowledge base this skill establishes.