ai-research-workflow

name

ai-research-workflow

description

Apply the AEI Housing Center's rules for AI-assisted data-analysis projects — single source of truth, numbers flowing through code instead of prose, preserved names, consistent terminology, and output-level testing. Use this whenever work involves an analysis pipeline, whether it ends in a dataset, a model, or a rendered document; whenever a data pipeline feeds a document, report, memo, or deck; whenever two documents share a number; whenever a pipeline will be re-run or handed off; and whenever the output is external-facing. Also use when asked to review an analysis for reproducibility, number drift, hardcoded values, or terminology inconsistency. One-off exploratory scripts are exempt.

AI research workflow rules

Rules for AI-assisted data-analysis projects. They apply when documents share numbers, the output is external-facing, or the pipeline will be re-run. One-off exploratory scripts are exempt.

The rules are not independent: "numbers flow through code" only holds because there is a single source of truth, and testing the outputs is what makes both enforceable. Read all of it before starting.

"Deliver in progress" applies only when the project renders a document. Everything else applies to any analysis pipeline, including one whose output is a dataset, a model, or a set of figures.

Single source of truth

• Every value, threshold, vintage, path, and definition has exactly one defining location: helper modules, a paths module, one parser per external benchmark, one analysis-ready file.
• A magic number, duplicated path literal, re-implemented rule, or second derivation is a bug. Use the canonical source or ask.

Numbers flow through code, never prose

• Numbers derived from in-project work are never typed into prose. Compute them in a pipeline step, persist them, and inline them via templating (r scalars$x).
• Numbers from an external source are the exception; a citation is required and sufficient.

Text is a cost

• Code and text compound: each adds a place for error and drift, and makes an existing error harder to find — particularly for human readers. New work can be worth the cost, but the cost is never zero.

Preserve names end-to-end

• The source-data name is the name everywhere downstream. A mean of value is value_mean, not cityDollars.
• Rename only for figure/table labels, in one labels lookup.
• One term per concept, in prose as in code, reused exactly. Never let a concept drift into a synonym. Shorthand and abbreviations are fine if defined once, in a definitions section, and used everywhere after.
• Say Extremely Low Income (ELI) once and it is "ELI" in every later instance and in every related document — never an alias like "poor."

Test the outputs, not just the code

• Four layers: external benchmark agreement (within tolerance bands), internal consistency (identities, monotonicity, no-NA, row counts), document-vs-source (published number == persisted artifact), code invariants (scalar references exist, no orphan figures, no leftover placeholders).
• A failing test means a pipeline problem. Never edit the test or loosen a tolerance.
• Re-run the affected step plus tests after any code change.

Validate by reproduction

• Pick an external target the pipeline should reproduce (published report, public dataset, hand calculation). Pick it at project start, not after.

Pipeline structure

• Numbered scripts, one job and one canonical artifact each. Downstream reads the artifact, never re-derives it. The document is a downstream consumer.
• Add a shared helper at the third call site, not the second. Exception: tiny scalar-lookup functions shared across documents.
• When a script reads a column another document depends on, say so in a comment: # eli_worker_core from 11_classify_workers.R; tech report reads it.

Charts and tables

• Self-contained: a reader should not need the surrounding context to understand one.
• Carry brief substantive data information toward that end — at least one line, no more than three without approval.
• The caption's last line is "Source: [other sources if any, and] AEI Housing Center, aei.org/housing."

Alternates are siblings, never substitutions

• Comparing vintages, geographies, thresholds, or definitions: build named siblings of the canonical artifact (hud_limits_fy23 canonical, hud_limits_fy24 diagnostic). Never swap the canonical input or redefine the canonical column.

Stop and ask the user before

• A parallel definition of a concept that already has a canonical source.
• Changing a vintage, threshold, or geography rule.
• Loosening, removing, or skipping a test.
• Computing a one-off number for prose.
• A "just for this script" override of an upstream rule.
• A fix that silently changes an input.

Deliver in progress, not at the end

Applies only when the project renders a document — a QMD, a website, a PDF, a slide deck, anything a reader opens. Skip it for analysis that ends in a dataset, a model, or loose figures: there is no document to keep whole, and a WORK_IN_PROGRESS block inside a script helps nobody.

The reader should be able to open the document at any moment and see what exists.

• Render after each piece lands, not once at the end. Co-render every document that shares scalars.
• Keep a WORK_IN_PROGRESS list at the top naming each thing being worked on right now. Delete an entry when it lands, and the block when the list empties.
• The document must render even when the pipeline is incomplete. A missing number or figure is a visible gap, never a failed render.

Make the accessors degrade instead of failing: a missing scalar returns a **[name pending]** marker, a missing figure a note saying it is being rebuilt, a missing table a one-row frame saying it is being built. Delete these fallbacks with the WORK_IN_PROGRESS block when the work is done.

A gap must be visible. A scalar that silently renders as an empty string is worse than no update, because the reader cannot tell a missing number from a finished sentence.

Put long work in the background

The main thread stays responsive so the reader can review and redirect.

• Send long, self-contained work to subagents: figure iteration, read-only briefs on other projects, audits.
• Keep in the main thread the decision about what to do, and the edits the reader will review.
• Two subagents must not write the same file. Read-only work never conflicts.
• A subagent that must wait for a pipeline step waits on the artifact that step writes last, not on a fixed delay.
• Never edit a script while it is running. On a network share this gives a torn read and a parse error partway through the run.