bulk-record-jobs

name

bulk-record-jobs

description

Plan and run bulk record jobs over the Skyrim load order via houseCARL — catalogues, audits, item/recipe/link graphs, conflict surveys, and batch patch rebuilds: any job that turns MANY records into ONE structured deliverable. Routes per-record loops to the bulk primitives (defined_in= scope, list-valued references=, group_by= counts, winner_fields=, resolve_names=, format=json, housecarl_resolve, housecarl_diff_record, CopyFrom/composes= batch writes) and standardizes on one canonical JSON shape. Use when the user wants a catalogue, index, spreadsheet, audit, or book of armors/weapons/NPCs/recipes/spells, asks what crafts or references what across plugins, who wins contested records, wants a compatibility patch rebuilt against a new mod version — and in ANY fan-out subagent task told to "extract X and return JSON/a table". Load BEFORE the first query or the first line of an output schema — per-record loops and schema drift are locked in by the first call.

Bulk Record Jobs

Overview

A bulk record job is any task shaped "many records in, one structured deliverable out": a gear catalogue, a crafting-graph, a conflict survey, a whole-mod audit, a compatibility-patch rebuild, a fan-out extraction returning JSON. The houseCARL tool surface has one primitive for every loop these jobs used to improvise — this skill maps job shapes to those primitives, carries the few game-level conventions the binary format doesn't (what a crafting-station keyword means, how a temper recipe differs structurally from a craft recipe), and defines the one canonical deliverable shape so ten agents extracting in parallel produce one schema, not ten.

Two costs dominate badly-planned bulk jobs, and both are locked in by the first call: per-record loops (a reverse lookup run 548 times instead of once; ~400 one-element list ops instead of ~40 batched ones) and schema drift (each extractor inventing its own output encoding). Plan the calls and fix the output shape before touching a record.

First step — size the job, then fix the shape

Before the first detail read:

1. Size the scope with a count, not a dump. group_by= on housecarl_cross_plugin_query aggregates ALL matches into a count table in one cheap call:

   housecarl_cross_plugin_query(type="ARMO", plugins=["SomeMod.esp"], group_by="defined_in")
   

   group_by="winner" opens a conflict survey ("who wins the contested records, by plugin"); "type" profiles a plugin's contents; "defined_in" splits definitions from overrides. The counts tell you whether the job is 40 records or 4,000 before you commit to a per-row plan.

2. Write the deliverable schema down (see "The canonical deliverable shape" below) before any extraction starts — especially before handing work to subagents. The schema is the contract; extraction fills it.

Scope truth — two flags that decide what your numbers mean

These two are the proven silent-wrong traps of bulk work. Neither errors when missed; both corrupt the deliverable.

• defined_in=true — definitions, not touches. plugins=["X.esp"] alone matches every record X.esp touches — its own definitions AND its overrides of other plugins' records. A catalogue scope ("the items this mod adds") means defined in. A replacer or patch plugin can return dozens of matches on a bare plugins= scope, every one an override that double-counts into your catalogue under the wrong mod. Pass defined_in=true whenever the question is "what does this plugin add"; leave it off only when overrides are genuinely part of the question.

• winner_fields=true — live values, not the scoped plugin's era. Under a plugins= scope, fields= renders the scoped plugin's own version of each value — the defining esp's original armor rating, not the number the game uses after later overrides. The output says so in a note, but a bulk consumer that never reads notes ships plausible stats from the wrong era. For any deliverable claiming live stats, pass winner_fields=true; the scoped-values default exists for the other question ("what did THIS plugin set").

Getting a big enumeration out — page it, or persist it

A bulk read stops for one of two independent reasons. They carry different accounting and different fixes; conflating them is how a "complete" catalogue silently ships short.

• Row cap — limit= → capped=true. limit= bounds how many rows render (default 500). group_by= counts are never capped by it — size with those.
• Output cap — max_chars → truncated=true. The document renders until it hits the per-call output budget, then stops mid-stream. On a whole-load-order sweep this is the cap that actually bites.

Primary lane — page with offset= (housecarl_cross_plugin_query). offset= skips the first N post-filter matches, so offset=0/500/1000… walks a big enumeration in exact windows; scan order is deterministic while the load order is unchanged, so windows tile with no gaps or overlaps, and the render prints the next offset to continue with. total always counts every match, so you know when you've walked them all. (Not valid with group_by= — a count table has no window.)

Complementary lane — raise max_chars, let the result persist to a file. When you want the entire result as one document to post-process with scripts — rather than walking windows into context — raise max_chars far past its default so it all renders in one call, let that oversized document persist to a file, and run scripts against the file. Never read a multi-MB document into context. One real run: a 7,479-NPC / ~15,000-path facegen scan that rendered ~117 rows/call at the default became one call per plugin at max_chars=4000000 — a ~3.6 MB, 5,118-row JSON document, complete and self-accounting. This is also the only lane for tools without offset= (e.g. housecarl_batch_record_detail — its only paging is to split its input formids= list; a size-cap stop there has no continuation, #254).

Guardrail — verify, never assume. A persisted document is trustworthy only once you've checked, in the file itself, that it did not truncate (truncated == false) and that its own accounting says it is whole — rendered == total on cross_plugin_query, or the count covering every input on the batch tools (housecarl_batch_record_detail / housecarl_resolve). A file that quietly hit even a raised cap is a silent short-ship — the exact failure this lane exists to prevent. format="json" keeps the accounting in-band for a one-line check.

Inputs cap too. A huge input array — multi-thousand FormIDs into references= / housecarl_resolve / housecarl_batch_record_detail — is its own failure mode: it must transit model context, and an oversized emission can stall the call. Batch inputs to a few hundred elements per call.

The loop-killer map

Reach for the primitive, never the loop. Each row below was a real improvisation in a real fleet run before the primitive existed:

About to…	Use instead
Run references= once per target record	ONE call — references= takes a LIST (OR semantics; each match reports which target(s) it hit via matches=)
Post-filter FormIDs by :Plugin.esp suffix	defined_in=true on the query
Dump all matches and tally winners by hand	group_by="winner" (counts ALL matches, not capped by limit=)
Read records one call each	housecarl_batch_record_detail with the whole FormID list
Read full records just to label FormIDs with names	housecarl_resolve — one identity line per FormID (type, editorid, name, winner), per-item error isolation
Maintain a FormID→name cache by hand	resolve_names=true on the read — every FormLink annotated inline, cached server-side across the batch
Parse path = token text back into JSON	format="json" on the read/query — same tokens, stable document, accounting in-band
Read two plugin versions and subtract by hand	housecarl_diff_record(formid, plugin_a, plugin_b) — either side may be an on-disk, not-enabled plugin
Re-type another version's field values into ops	verb:"CopyFrom" with from_plugin= in housecarl_bulk_apply — transplants the field, source may be off-order
Add list elements one op at a time	composes=[...] — one op appends N elements (verb:"Add") or replaces the whole list (verb:"ReplaceAll"; composes=[] clears it)

Recipe: the catalogue

"Enumerate everything of type T that mod set S adds, with live stats and resolved names."

1. Size: group_by="defined_in" over the scope (First step).

2. Enumerate + flat fields in one pass (scalar fields only — Name, ArmorRating, weights…):

   housecarl_cross_plugin_query(
       type="ARMO", plugins=["ModA.esp", "ModB.esp"], defined_in=true,
       fields=["Name", "ArmorRating"], winner_fields=true, format="json")
   

3. Expand list-bearing rows in the same scan. Pass depth= with fields= — fields=["Keywords"], depth=2 returns each element indexed (Keywords[0]…) across every match, and resolve_names=true adds the structured link sibling ({resolved, type, editorid, name}) beside the raw token. format="dense" stays depth-1 (its columnar cells are positional) — use format="json" for expanded scans, or hop the flagged subset to housecarl_batch_record_detail when only a few matches need expanding.

4. Respect the accounting. Every JSON document carries total / rendered / capped / truncated / notes in-band. capped=true means page with offset= or narrow the scope; truncated=true means raise max_chars (see "Getting a big enumeration out" above) — never ship a deliverable whose total exceeds its row count without saying so.

Recipe: the link graph

"For each record, what points at it / what does it point at, as names not hex." Works for any link-shaped question — crafting recipes, outfits, leveled lists, dialogue — because the reflection layer knows every FormLink on every type. One graph level = two batched calls:

1. Reverse edge (who points at these?): references=[...] with the whole target list, bounded by type= — e.g. every recipe involving a list of items is ONE call with type="COBJ".
2. Contents + identity: housecarl_batch_record_detail on the matches, depth= deep enough for the paths you need, resolve_names=true so every link is labeled inline.

Worked example — the crafting graph for one weapon (verified live; vanilla data):

housecarl_cross_plugin_query(type="COBJ", references=["012EB7:Skyrim.esm"])
→ TemperWeaponIronSword, RecipeWeaponIronSword, … (every recipe touching Iron Sword, one scan)

housecarl_batch_record_detail(
    formids=[…those COBJs…], depth=5, resolve_names=true, format="json",
    fields=["EditorID", "Items", "WorkbenchKeyword", "CreatedObject", "CreatedObjectCount", "Conditions"])

The load-bearing COBJ paths (only visible at depth ≥ 4–5):

• materials: Items[i].Item.Item (the ingredient link) + Items[i].Item.Count
• station: WorkbenchKeyword (a Keyword link — see conventions below)
• product: CreatedObject + CreatedObjectCount
• gates: Conditions[i].Data.Function names the condition kind; Conditions[i].Data.Perk carries the perk link when the function is HasPerk

resolve_names annotations resolve against the live load order: reading a vanilla body on a modded order can label a link with a mod's renamed identity for that FormID. That's the truthful in-game identity — but if the deliverable documents the original plugin's era, resolve names from that era's values instead of assuming the annotation matches the source.

Crafting-station conventions (what the format doesn't say)

The binary format stores WorkbenchKeyword as just a Keyword link; which station it is and what the recipe means there is Creation-Kit convention:

• Enumerate stations from data, don't hardcode: crafting-station keywords are vanilla KYWD records — housecarl_cross_plugin_query(type="KYWD", editorid_contains="Crafting", plugins=["Skyrim.esm"], defined_in=true) lists them live (forge, skyforge, sharpening wheel, armor table, smelter, tanning rack, cookpot…). Ignore the WICrafting* ones — those are radiant-story event keywords, not stations. Mods add their own station keywords; the same query without the plugin scope finds them.
• Craft vs temper is structural, not naming. A craft recipe (forge/smelter/tanning/cookpot): CreatedObject = the item produced from the materials. A temper recipe (sharpening wheel for weapons, armor table for armor): CreatedObject = the very item being improved, count 1, and the canonical vanilla condition pair is EPTemperingItemIsEnchanted != 1 (OR-flagged) + HasPerk(<arcane-smithing-class perk>). Classify by WorkbenchKeyword + this shape.
• Conventions are conventions — the data can be dirty. Vanilla itself ships TemperWeaponSkyforgeBow whose CreatedObject is a battleaxe. Classify from the fields you read, and let an EditorID that disagrees with the structure raise a flag in the deliverable rather than win the argument.
• A recipe that appears at no station (null/odd WorkbenchKeyword, or gated by an impossible condition) is a common mod-specific disablement idiom — whose meaning belongs to that mod's own skill lane, not here. Report the structure; don't guess the intent.

Recipe: the patch rebuild

"Re-derive an old compatibility patch against a new version of its target mod." The job is delta-analysis then batched re-application; the old patch is usually disabled, which is fine — the diff and copy primitives read on-disk plugins that aren't in the load order.

1. Survey the contest: group_by="winner" over the target mod's records — one call says how much of the mod is contested and by whom.
2. Extract the old delta per record: housecarl_diff_record(formid, plugin_a="OldPatch.esp", plugin_b="TargetMod.esp") — field-level deltas, each labeled by plugin_b's name; fields= scopes the comparison; a truncated read reports itself rather than claiming "identical".
3. Forward the new baseline (housecarl_forward_record), then re-apply the still-valid deltas with housecarl_bulk_apply into= on the forwarded copies — ops target the patch's own already-forwarded body, not the stale winner (the forward-then-edit precedence both tools' docs state).
4. Batch the re-application: whole-field transplants use verb:"CopyFrom" + from_plugin="OldPatch.esp" (works from a disabled plugin on disk); list rebuilds use composes=[...] — N perk placements in one op, ReplaceAll + composes to rebuild a modeled list wholesale, ReplaceAll + composes=[] to empty it. All-or-nothing per call: one bad element refuses the whole call with per-element reasons.
5. Verify before enabling: housecarl_check_errors and housecarl_compact_plugin accept the not-yet-enabled patch; housecarl_diff_record with the new patch as a pole confirms the deltas landed.

Recipe: the fan-out

When the job fans out to subagents, schema drift is the failure mode — N agents told "extract X, return JSON" will invent N schemas unless the orchestrator pins one. In every subagent prompt:

• Hand the agent the deliverable schema (below), filled with the job's field list — not a prose description of it. Agents hardwire against what you show them.
• Name the exact calls: the scope flags (defined_in, winner_fields), format="json", resolve_names=true, and the field paths with their depths. A recipe the orchestrator verified once beats eight agents re-deriving it.
• Require the accounting fields in each agent's return, so a truncated extraction surfaces at merge time instead of shipping silently incomplete.

The canonical deliverable shape

One blessed shape for "many records → one document". Use it as the return contract for subagents and the final deliverable alike; add job-specific keys rather than restructuring.

{
  "job": "one line: what this deliverable is",
  "scope": {"type": "WEAP", "plugins": ["ModA.esp"], "defined_in": true, "winner_fields": true},
  "total": 548,
  "complete": true,
  "notes": ["tool notes + job caveats carried here, never dropped"],
  "records": [
    {
      "formid": "012EB7:Skyrim.esm",
      "type": "Weapon",
      "editorid": "IronSword",
      "name": "Iron Sword",
      "winner": "SomePlugin.esp",
      "fields": {"Damage": "7"}
    }
  ]
}

The rules that make it a contract:

• formid is the full wire token (XXXXXX:Plugin.esp), verbatim from tool output — never bare hex (collides across plugins), never reformatted (a token read is a token a write can reuse).
• Identity is four keys: type, editorid, name, winner — straight from housecarl_resolve / the read header. name is null where the type has none; don't substitute the editorid.
• A resolved link is an object, not a replacement: {"formid": "…", "editorid": "…", "name": "…"} — keep the token AND the identity; a name-only column can't be queried again.
• Field values are wire tokens verbatim. Reformatting (unit conversion, rounding, splitting) happens in a separate presentation layer, never in the extraction rows.
• Accounting is mandatory: total (true match count), complete (every row present?), and notes travel with the document. A partial deliverable that says so is fine; one that doesn't is a silent wrong answer.
• Closed enums for classifications, declared in the deliverable (e.g. a recipe kind of craft / temper / other) — free-text classification is where eight agents drift eight ways.

For crafting-graph jobs specifically, the blessed per-recipe entry:

{
  "recipe": "0DA769:Skyrim.esm",
  "kind": "craft",
  "station": "CraftingSmithingForge",
  "creates": {"formid": "012EB7:Skyrim.esm", "editorid": "IronSword", "name": "Iron Sword", "count": 1},
  "materials": [{"formid": "05ACE4:Skyrim.esm", "editorid": "IngotIron", "name": "Iron Ingot", "count": 2}],
  "requires_perks": [{"formid": "…", "editorid": "…", "name": "…"}],
  "other_conditions": 1
}

Common mistakes

• Looping a primitive that takes a list. references=, housecarl_resolve, housecarl_batch_record_detail, composes= — all batch. The single-item call in a loop is the old world.
• Shipping scoped values as live stats. The plugins= + fields= default renders that plugin's own era; a catalogue built without winner_fields=true is plausibly-wrong everywhere a later plugin rebalanced.
• Counting overrides as content. A bare plugins= scope on a patch/replacer returns overrides; without defined_in=true they double-count into the catalogue under the wrong mod.
• Parsing text output when format="json" exists. Hand-parsing is where schema variance creeps in; the JSON document carries the same tokens plus the accounting.
• Dropping the accounting. capped, truncated, rendered < total, and notes exist to be propagated. A deliverable that silently ignores them claims completeness it doesn't have.
• Trusting names over structure. EditorIDs and display names are conventions; classify from fields (station keyword, CreatedObject relationship, condition functions) and flag disagreements — vanilla itself contains miswired records.
• Re-typing what CopyFrom can transplant. Reading a value out of one plugin and composing it back by hand re-introduces transcription risk the primitive exists to remove.

Sub-topic routing

The question is really…	Go to
"What fields does record type X have / what are the legal enum values?"	mutagen-reference
"Which armors sit on biped slot N?"	biped-slot-reference
What a specific mod's keywords/conventions mean (any per-mod semantics)	that mod's own skill lane (e.g. a Requiem companion skill), never encoded here
One record to read or edit	the tools directly — no bulk planning needed
Distributing spells/keywords/items at runtime instead of cataloguing them	spid-authoring / kid-authoring / skypatcher-authoring

Notes

• housecarl_cross_plugin_query takes depth= (with fields=; text/json formats) — list expansion rides the scan itself, no second call. housecarl_batch_record_detail remains the lever when only a subset of matches needs expanding, and under format="dense" (depth-1 by design — positional cells).
• where= accepts FormLink equality against a wire token (e.g. "WorkbenchKeyword = 088108:Skyrim.esm") — a station filter is one predicate, no post-filtering.
• Off-order (disabled, on-disk) plugins are first-class poles for housecarl_diff_record, sources for CopyFrom, and targets for housecarl_read_plugin_file / housecarl_check_errors / housecarl_compact_plugin — "the old patch is disabled" blocks nothing.
• Aggregation (group_by=) counts ALL matches regardless of limit=; only row rendering is capped. Use counts freely for sizing.