openomb-spend-plan-fix

name	openomb-spend-plan-fix
description	Diagnose and fix a spend plan PDF that isn't parsing correctly. Takes a filename or URL and optional free-form agency/bureau hint, then diagnoses what's failing, explains the proposed fix, and applies it across load-file.ts, spend-plan-agency-match.ts, and the test file.

OpenOMB: Fix a spend plan parsing failure

The args will be free-form text containing: a filename or URL for a failing spend plan PDF, and optionally any agency or bureau name/acronym hints. Parse these out before starting.

---

Step 0: Optionally gather URL issues

If prompted to find any files that are failing to parse.

• Utilize the following command to run all Spend Plan collections. npm run dev:collect -- --no-archive --apportionment-type spend-plan
  • Save the output to a file and search for things like:
    • Agency not able to be parsed for spend plan — a genuine parsing failure. Continue with Steps 1-8 below.
    • Folder could not be determined from agency for spend plan — the agency did resolve; skip straight to "When the failure is 'Folder could not be determined from agency for spend plan'" below instead of Steps 3-8.
• Or, for a known URL, you can run the collect for that specific URL with something like: npm run dev:collect -- --no-archive --apportionment-type spend-plan URL

---

Step 1: Extract the filename

• If the args contain a URL (starts with http), decode it with decodeURIComponent, take the last path segment after /, and strip trailing .pdf. That is the filename.
• Otherwise, treat the non-URL portion of the args as the filename (strip .pdf if present).
• Any remaining text after removing the URL/filename is treated as an agency/bureau hint from the user.

---

Step 2: Read the relevant files

Read all four files before drawing any conclusions:

1. src/lib/server/load-file.ts — read the YEAR_EXTRACTORS array inside parseSpendPlanFilename() to understand the current extraction strategies
2. data/fixes/spend-plan-agency-match.ts — check whether a pattern already exists that would match this filename
3. data/agency-reference.ts — the agencies and bureaus arrays; each entry has short_name (one or more /-separated acronyms), budgetAgencyTitle, budgetAgencyTitleId, budgetBureauTitle, budgetBureauTitleId

These agencies/bureaus arrays are regenerated by bin/check-agencies.ts to only include agencies/bureaus backed by real (file_type = 'standard') apportionment data — being listed there guarantees a real folder exists for it. The leftoverAgencies/leftoverBureaus sections in the same file are Federal Register agencies with no real data in the system yet; parseSpendPlanFilename() does not consult them at all (that fallback was removed — it produced unconfirmed, coincidental agency matches, e.g. a spend plan about HHS's ACF program was once misattributed to "Administrative Office of United States Courts" purely because "AO" also appeared in the filename). Don't treat a leftoverAgencies/leftoverBureaus entry as usable for a fix.

---

Step 3: Write the failing test first

Before diagnosing anything, write the test that describes the desired behavior. This acts as the specification and gives a concrete red/green signal throughout.

Edit src/lib/server/load-file.test.ts. Inside describe('parseSpendPlanFilename()'), add the new case:

// <full original URL>
// <one-line note on why this filename is tricky — fill in after diagnosis if unclear now>
expect(parseSpendPlanFilename('<filename without .pdf>')).toMatchObject({
  fiscalYear: '<YYYY>',
  agency: '<budgetAgencyTitle>',
  bureau: '<budgetBureauTitle>' // omit if no bureau
});

Filling in the expected values: Use what's already known from the URL/filename and any agency/bureau hint. The fiscal year is often inferable even if the current code can't parse it (e.g., MAY25 → '2025'). If the agency or bureau is not clear from the hint or the filename itself, use AskUserQuestion to ask the user before writing the test — don't leave placeholder values.

Place the test near other entries covering the same failure class (near other month-year entries, near other ACF patterns, etc.), following the surrounding comment and assertion style.

Run just this test file to confirm it fails:

npm run test:unit -- --reporter=verbose src/lib/server/load-file.test.ts

The new assertion should fail. If it unexpectedly passes, the parsing is already working and no fix is needed — report that to the user and stop.

---

Step 4: Diagnose the fiscal year

Walk each extractor in YEAR_EXTRACTORS against the filename in order, exactly as the code does (first match wins). Note which strategy fires and what rawYear it would capture.

If a year is found: record the extracted year and move on.

If no year is found: look at the filename and ask — does it contain a year-like value that has a recognizable general structure a new regex could handle, such as:

• A month abbreviation immediately followed by digits (MAY25, JAN2026)
• A standalone 2-digit year after a consistent separator that the current extractors miss

If yes, plan a new YEAR_EXTRACTORS entry that would handle this class of filename. If no — the year is completely absent or genuinely ambiguous — ask the user to confirm the fiscal year before continuing. It will go into spend-plan-agency-match.ts.

---

Step 5: Diagnose the agency and bureau

Extract all consecutive uppercase runs of 2–6 letters from the filename (regex [A-Z]{2,6}). For each acronym:

1. Search agency-reference.ts agencies: does any entry's short_name (split on /) match the acronym? → that's the agency.
2. Search agency-reference.ts bureaus: does any entry's short_name match, and does its budgetAgencyTitle equal the agency found above? → that's the bureau.

Also check data/fixes/spend-plan-agency-match.ts — a pattern may already exist that would match, resolving the agency even if acronym matching fails.

If the user provided an agency/bureau hint: search agency-reference.ts by name, budgetAgencyTitle, and short_name. Report the exact budgetAgencyTitle / budgetBureauTitle that must be used — case, spacing, and articles ("the", "U.S.") matter. If the name isn't found in agency-reference.ts, run a grep over src/lib/server/db/ and src/lib/server/db/test-data/ to find how the agency appears in existing data. If still not found, note the discrepancy and ask the user to verify the exact name in the database.

If no agency can be determined and the user provided no hint: ask the user to identify the agency (suggest they check the PDF content or the OMB page). Do not guess.

If no acronym matches agencies/bureaus and no fixes-file pattern applies, do not fall back to leftoverAgencies/leftoverBureaus — that mechanism has been removed. Either confirm the real agency via the PDF content or OMB page and add a fixes entry for it (this is fine even if that agency has no real tafs data yet — see the section below for what happens next), or leave the file agency-unresolved (it will land in Unknown Folder with a graceful warning, not a crash).

---

When the failure is "Folder could not be determined from agency for spend plan"

This is a different problem from a parsing failure — parseSpendPlanFilename() already resolved an agency (the error message itself reports exactly which Agency / Budget Agency Title ID it used). The question isn't "how do we parse this filename" — it's "is that resolved agency actually right?"

1. Verify the resolved agency is actually correct for this file. Read the PDF content (the source PDF is linked from the error's URL) and confirm it's genuinely about the agency named in the error. Don't assume a plausible-looking acronym match is correct — this exact failure mode is how an HHS/ACF file was once misattributed to "Administrative Office of United States Courts" (the "AO" acronym happened to also appear in the filename).

2. If the resolved agency is wrong: find what actually matched it — an acronym match against agencies/bureaus, or a pattern in spend-plan-agency-match.ts — and fix that (a more specific/anchored fixes entry for the correct agency, placed above the incorrectly-matching pattern, following the "more specific patterns higher up" convention). Add/update the parseSpendPlanFilename() test the same way Step 3 describes, then verify per Step 8.

3. If the resolved agency is correct: check whether it appears in agency-reference.ts's main agencies/bureaus arrays. If it does, something else is wrong (worth a closer look — the folder-lookup query in src/lib/server/db/queries/agencies.ts may need investigation). If it does not appear there, this agency genuinely has no real (non-spend-plan) apportionment data collected yet — this is not a parsing bug, and there is no fixes-file entry that resolves it (pointing a fix at this same agency name would just keep hitting the same throw). Report this to the Developer as expected/by-design: loadPdfSpendPlan() intentionally hard-fails this case so it stays visible, rather than silently landing in Unknown Folder, until real apportionment data for that agency is collected or a Developer decides how to handle it. Do not "fix" this by adding a spend-plan-agency-match.ts entry — it won't change the outcome.

---

Step 6: Report the diagnosis and proposed fix

Before making any changes, output a clear summary:

Filename:     <decoded filename>
Fiscal year:  <year found, OR "NOT FOUND — [reason]">
Agency:       <resolved budgetAgencyTitle, OR "NOT FOUND — [reason]">
Bureau:       <resolved budgetBureauTitle, OR "none / NOT FOUND">

Proposed fix:
  [A] Update YEAR_EXTRACTORS in load-file.ts — new extractor for <pattern>
  [B] Add entry to spend-plan-agency-match.ts — pattern: /<regex>/, fiscalYear: '<YYYY>', agency: '<...>', bureau: '<...>'
  [C] Both A and B

If any required information is still missing (fiscal year, agency name), use AskUserQuestion to collect it before proceeding.

---

Step 7: Apply the fix

A. New YEAR_EXTRACTOR (if a general regex fix applies)

Edit src/lib/server/load-file.ts. Add the new entry to the YEAR_EXTRACTORS array at the correct position — more specific before less specific, consistent with the surrounding code style. Each entry is an arrow function: (name) => { const m = name.match(...); return m ? { rawYear: m[1], rest: name } : null; }. Add a brief comment above it describing the pattern and a concrete example filename.

For 4-digit years, guard against implausible matches with m[1][0] === '2' or m[1].length < 4. Follow the same guard logic already present in the file.

B. New entry in spend-plan-agency-match.ts (if a fix entry is needed)

Edit data/fixes/spend-plan-agency-match.ts. Add the new entry with:

• pattern — prefer an anchored exact match (/^...\$$/) for one-off filenames; use a general pattern only when this clearly represents a family of similar files
• fiscalYear — only if the year was not (and cannot be) extracted by YEAR_EXTRACTORS
• agency — the exact budgetAgencyTitle from agency-reference.ts
• bureau — the exact budgetBureauTitle, if applicable

Ordering matters: more specific patterns must appear above less specific ones. Place the new entry above any existing catch-all that would also match this filename. Add a comment with the full original URL above the entry, following the existing convention.

Do not invent agency or bureau names. Only use values found in agency-reference.ts or confirmed via database lookup.

Both A and B

If both a regex update and a fix entry are needed, apply A first (it may resolve the year, leaving only agency to be fixed by B).

---

Step 8: Verify

Run just the load-file test file to confirm the new test now passes:

npm run test:unit -- --reporter=verbose src/lib/server/load-file.test.ts

If the test still fails, diagnose the root cause and fix it before reporting success. Once it's green, confirm nothing else broke by running the full suite:

npm run test:unit

---

Notes

• The fixes fallback in parseSpendPlanFilename() only runs when either fiscal year or agency is still missing after the extractors and acronym scan. If a new extractor fires and supplies the year, acronym matching still runs before the fixes file is consulted.
• rawYear values of 2–3 digits are padded to 4 digits by rawYear.padStart(4, '20'). A 4-digit raw year is used as-is.
• Bureau matching requires the bureau's budgetAgencyTitle to equal the resolved agency — hierarchy alignment is enforced in the code.
• If a pattern already exists in spend-plan-agency-match.ts that matches this filename but is missing fiscalYear, adding a more specific entry above it is preferable to modifying the existing one (to avoid unintended side effects on other files that match the same pattern).
• leftoverAgencies/leftoverBureaus in agency-reference.ts exist only as bookkeeping for bin/check-agencies.ts (agencies not yet matched to anything in our system) — parseSpendPlanFilename() does not read them.