ai-research-workflow

# AI research workflow rules Rules for AI-assisted data-analysis projects. They apply when documents share numbers, the output is external-facing, or the pipeline will be re-run. One-off exploratory scripts are exempt. The rules are not independent: "numbers flow through code" only holds because there is a single source of truth, and testing the outputs is what makes both enforceable. Read all of it before starting. **"Deliver in progress" applies only when the project renders a document.** Everything else applies to any analysis pipeline, including one whose output is a dataset, a model, or a set of figures. ## Single source of truth - Every value, threshold, vintage, path, and definition has exactly one defining location: helper modules, a paths module, one parser per external benchmark, one analysis-ready file. - A magic number, duplicated path literal, re-implemented rule, or second derivation is a bug. Use the canonical source or ask. ## Numbers flow through code, never prose - Numbers derived from in-project work are never typed into prose. Compute them in a pipeline step, persist them, and inline them via templating (`r scalars$x`). - Numbers from an external source are the exception; a citation is required and sufficient. ## Text is a cost - Code and text compound: each adds a place for error and drift, and makes an existing error harder to find — particularly for human readers. New work can be worth the cost, but the cost is never zero. ## Preserve names end-to-end - The source-data name is the name everywhere downstream. A mean of `value` is `value_mean`, not `cityDollars`. - Rename only for figure/table labels, in one labels lookup. - One term per concept, in prose as in code, reused exactly. Never let a concept drift into a synonym. Shorthand and abbreviations are fine if defined once, in a definitions section, and used everywhere after. - Say Extremely Low Income (ELI) once and it is "ELI" in **every** later instance and in every related document — **never** an alias like "poor." ## Test the outputs, not just the code - Four layers: external benchmark agreement (within tolerance bands), internal consistency (identities, monotonicity, no-NA, row counts), document-vs-source (published number == persisted artifact), code invariants (scalar references exist, no orphan figures, no leftover placeholders). - A failing test means a pipeline problem. Never edit the test or loosen a tolerance. - Re-run the affected step plus tests after any code change. ## Validate by reproduction - Pick an external target the pipeline should reproduce (published report, public dataset, hand calculation). Pick it at project start, not after. ## Pipeline structure - Numbered scripts, one job and one canonical artifact each. Downstream reads the artifact, never re-derives it. The document is a downstream consumer. - Add a shared helper at the third call site, not the second. Exception: tiny scalar-lookup functions shared across documents. - When a script reads a column another document depends on, say so in a comment: `# eli_worker_core from 11_classify_workers.R; tech report reads it.` ## Charts and tables - Self-contained: a reader should not need the surrounding context to understand one. - Carry *brief* substantive data information toward that end — at least one line, no more than three without approval. - The caption's last line is "Source: [other sources if any, and] AEI Housing Center, aei.org/housing." ## Alternates are siblings, never substitutions - Comparing vintages, geographies, thresholds, or definitions: build named siblings of the canonical artifact (`hud_limits_fy23` canonical, `hud_limits_fy24` diagnostic). Never swap the canonical input or redefine the canonical column. ## Stop and ask the user before - A parallel definition of a concept that already has a canonical source. - Changing a vintage, threshold, or geography rule. - Loosening, removing, or skipping a test. - Computing a one-off number for prose. - A "just for this script" override of an upstream rule. - A fix that silently changes an input. ## Deliver in progress, not at the end **Applies only when the project renders a document** — a QMD, a website, a PDF, a slide deck, anything a reader opens. Skip it for analysis that ends in a dataset, a model, or loose figures: there is no document to keep whole, and a `WORK_IN_PROGRESS` block inside a script helps nobody. The reader should be able to open the document at any moment and see what exists. - Render after each piece lands, not once at the end. Co-render every document that shares scalars. - Keep a `WORK_IN_PROGRESS` list at the top naming each thing being worked on right now. Delete an entry when it lands, and the block when the list empties. - The document must render even when the pipeline is incomplete. A missing number or figure is a visible gap, never a failed render. Make the accessors degrade instead of failing: a missing scalar returns a `**[name pending]**` marker, a missing figure a note saying it is being rebuilt, a missing table a one-row frame saying it is being built. Delete these fallbacks with the `WORK_IN_PROGRESS` block when the work is done. A gap must be visible. A scalar that silently renders as an empty string is worse than no update, because the reader cannot tell a missing number from a finished sentence. ## Put long work in the background The main thread stays responsive so the reader can review and redirect. - Send long, self-contained work to subagents: figure iteration, read-only briefs on other projects, audits. - Keep in the main thread the decision about what to do, and the edits the reader will review. - Two subagents must not write the same file. Read-only work never conflicts. - A subagent that must wait for a pipeline step waits on the artifact that step writes last, not on a fixed delay. - Never edit a script while it is running. On a network share this gives a torn read and a parse error partway through the run.