| name | model-calibration |
| description | Run, refresh, or diagnose the model's calibration pipeline (feed -> food_waste -> food_demand -> cost -> stability) that produces the per-config artefact sets under `data/curated/calibration/<source>/` (the `default` and `gbd-anchored` sets are git-tracked). Covers the dependency order, the `tools/calibrate` wrapper, realistic runtime expectations, when each kind of upstream change forces a re-run, and how to diagnose the most common failure mode: a hidden supply/demand mismatch that inflates the production-stability L1 cost. Use whenever calibration is relevant -- the user touches inputs/build logic that feed the calibration solves, calibration artefacts look off, or a refresh of the artefacts is needed after a model/data change. |
Model Calibration
The default workflow consumes five calibration artefact groups organized
in per-config sets under data/curated/calibration/<source>/, selected
by the calibration.source config key. Two sets are git-tracked, both
fit against the default GDD-IA diet source: default (fit against the
anchoring-off baseline diet of the health-off default config) and
gbd-anchored (fit against the GBD-anchored diet; consumed by the
health-enabled configs gsa, gsa_fixed_diet, validation and the doc
configs). No set is shipped for diet.source: fbs, which therefore
needs its own calibration run. tools/calibrate resolves the base config's
diet.anchor_groups_to_gbd sentinel once and pins it across all five
steps, and provenance stamps record the resolved anchoring. Each
artefact group is produced by a dedicated validation-mode solve and
absorbs a specific class of residual mismatch so that ordinary solves
don't have to. Without these files in place, production-stability,
costs, and food/feed accounting drift from observed 2020 reality.
Every set carries a provenance.yaml stamp of the structural config it
was fit against; workflow runs error at DAG time when their config
differs structurally from the consumed set's stamp (see "Artefact sets
and provenance" below).
Authoritative reference: docs/calibration.rst. This skill is the operational
companion: when to run, how to run, what to expect, what to watch out for.
The five steps at a glance
| Step | Config | Produces | Absorbs |
|---|
| feed | config/calibration/feed.yaml | grassland_yield.csv, fodder_conversion.csv, exogenous_forage.csv, exogenous_feed.csv | Per-country forage, and protein- and roughage-feed supply/demand gaps |
| food_waste | config/calibration/food_waste.yaml | food_waste.yaml | Per-food-group consumer-side waste multiplier (FBS supply vs GDD-IA intake) |
| food_demand | config/calibration/food_demand.yaml | food_demand.csv | Per-food residual mismatch left over after food_waste |
| cost | config/calibration/cost.yaml | crop_cost.csv, grassland_cost.csv, animal_cost.csv | Additive production-cost corrections from stability-constraint duals |
| stability | config/calibration/stability.yaml | deviation_penalty.yaml | The L1 penalty triple over the configured components (default: land + feed) that gives ~5% deviation on each axis |
The order is strict (next section explains why). The five configs each
carry name: "calibration", but tools/calibrate overrides it per step
with name: calibration-<source>-<step>, so each step gets its own
processing tree. That isolation is deliberate -- the steps disable
different calibration blocks, so a shared tree would thrash build_model
-- but it means the identical upstream prep is rebuilt five times
(~100 s and ~250 MB per step, so roughly 40% of chain wall-clock and
1.25 GB on disk for a full run).
Strict dependency order
Each step solves against a model whose earlier-step calibrations are
already applied. Re-running out of order leaks residual mismatch into
later artefacts and corrupts them in non-obvious ways:
feed first -- later solves rely on the feed slack already being closed.food_waste next -- food-bus slack must not be contaminated by feed-side mismatch.food_demand next -- the per-food residual must not be mis-attributed to waste.cost next -- without food_demand, per-food mismatch leaks into cost duals as spurious sign (olive-oil goes negative, coffee/tea peg at the slack ceiling).stability last -- the L1 Broyden iteration assumes all previous corrections are in place.
If in doubt, run the whole chain. Partial re-runs are only safe when every
earlier artefact is still semantically valid (see "When to re-run").
Entry point
tools/calibrate
tools/calibrate feed
tools/calibrate food_waste
tools/calibrate food_demand
tools/calibrate cost
tools/calibrate stability
tools/calibrate --check
tools/calibrate --record
tools/calibrate --base config/<name>.yaml [all|<step>|--check]
The wrapper defaults to pixi -e gurobi -- all calibration configs use
Gurobi. HiGHS is too slow here. Override with CALIBRATE_PIXI_ENV=<env>.
With --base, the base config must declare its own calibration.source
(refusing to overwrite the shared default set); a fresh set is seeded
from default and regenerated in order, and the all chain uses
name: calibration-<source>-<step> so processing trees don't thrash. After any
successful run the set is (re)stamped with provenance.yaml.
Pass extra flags through positionally:
tools/calibrate cost -j8
For local runs, SMK_MEM_MAX=30G is a safe ceiling on this workstation;
see local_machine_smk_defaults.md.
When to re-run calibration
Always probe first
tools/calibrate --check
Takes ~20 s for the default set, ~30 s for gbd-anchored. It compares
content hashes recorded in data/curated/calibration/<source>/fingerprint.yaml
against the step's external DAG leaves (curated/bundled/manually-downloaded
data, rule files, scripts), its config (YAML round-tripped, so comments don't
count), and the artefacts the step wrote. Everything inside the artefact set is
excluded, so the answer is stable: a completed chain reports all-green, and
mtime churn from git checkout or touch is invisible.
tools/smk also prints a one-line mtime-based reminder on every invocation.
That one is advisory and does produce false positives after git pull;
silence it with SMK_SKIP_CALIBRATION_HINT=1.
Editing any rule file or script in a step's DAG marks it stale, including
inert changes (refactors, comments, unrelated rules). When you have
established the artefacts are unaffected, tools/calibrate --record
re-stamps the set without solving. It asserts the artefacts are current, so
don't reach for it to make a genuine [STALE] go away.
Triggers for a full re-run (start from feed)
Default to a full re-run whenever something upstream of the calibration
solves changes materially. The chain is short and the artefacts are
internally consistent only as a set. Concretely:
- Feed supply/demand: GLEAM3 inputs, grassland yields, fodder/forage logic, feed-category routing, feed conversion factors, animal-product baselines.
- Food-bus mass balance: food group definitions, food-loss/waste defaults, processing pathways, FBS/QCL preparation, baseline-diet (GDD-IA / FBS) construction, within-group share logic.
- Crop/grassland/animal cost inputs: FAOSTAT prices, FADN animal costs, GAEZ/yield calibration touching the cost-baseline solve.
- Production-stability machinery:
workflow/scripts/build_model/ changes that touch stability links, constraint sign, slack semantics, or deviation accounting.
- LUC / land-carbon overhauls that change cropland or pasture baselines (recent precedent:
4cada27 LUC revert -> f3f86f3 refresh).
- Region or resource-class changes that move the baseline production map.
Triggers for a partial re-run
You can resume from a later step only if every earlier artefact is still
semantically valid. Rule of thumb:
- Anything upstream of cost in the feed/food chain -> start from
feed.
- Changes only to cost-extraction logic or to
cost.yaml/crop_cost* inputs -> start from cost.
- Changes only to the L1 mechanism or to the
stability.yaml config -> start from stability.
When in doubt, run the full chain. A few extra minutes is cheaper than
diagnosing a silently miscalibrated downstream solve.
What does NOT need a re-run
- Solve-time-only knobs: GHG price, value_per_yll, scenario overrides.
- Analysis, plotting, surrogate fitting, paper notebooks.
- Pure refactors that don't touch model construction, mass balance, or solve-time wiring.
Realistic runtime (local, gurobi)
End-to-end wall-clock of tools/calibrate ... all, measured on this
workstation at -j8 with the shared data/ and processing/shared/
trees already populated. Each step's time includes rebuilding its own
processing tree (see above), which is why no step comes in under a
minute even though the solves themselves are seconds.
| Step | Solves | default set | water study (750 regions, T=4) |
|---|
| feed | 1 validation solve + extract | 96 s | 145 s |
| food_waste | 1 validation solve + extract | 100 s | 145 s |
| food_demand | 1 validation solve + extract | 91 s | 151 s |
| cost | 2 paired solves (hard stability) + extract | 153 s | 198 s |
| stability | Broyden iterations | 193 s (1 iter) | 551 s (5 iters) |
| full chain | | 10.6 min | 19.9 min |
Of each step's time, only a small part is the solve. On the water study:
build_model is 16-18 s and a validation solve 23 s (cost's paired
solves 35 s each); the Broyden loop is the one genuinely long stage at
422 s for 5 iterations. Everything else is the per-step processing-tree
rebuild.
Stability dominates the variance: warm-starting from an existing
deviation_penalty.yaml converged in 1 iteration on the default set,
while the water set needed 5 from a seeded start. Budget 3-9 minutes for
that step alone.
These are unusually fast because every calibration config disables the
health objective (value_per_yll: 0), which makes the problem a pure LP.
A default-config solve carries the health term -- a piecewise-linear
approximation of log/exp products that introduces integer variables --
turning the problem into a MILP and slowing each solve noticeably.
Don't extrapolate these numbers to default-config solves.
They also assume the shared upstream data is already on disk. On a fresh
clone or after a deep model change, add the one-time build of the shared
prep -- 10-30 minutes depending on what has to be rebuilt and whether
retrieval rules fire (those need network access).
HPC offloading isn't worthwhile -- the feed/food/cost steps are single
solves with significant local prereqs, and stability is inherently
sequential (each Broyden iteration depends on the previous solve).
Output landing zones
data/curated/calibration/<source>/* -- one artefact set per base config, plus its provenance.yaml stamp and fingerprint.yaml input hashes; the default and gbd-anchored sets are git-tracked. Commit a set together as a refresh; mixed-vintage artefacts are the most common cause of confusing downstream solves.processing/calibration-<source>-<step>/* -- one upstream prep tree per step (~250 MB each), NOT committed.results/calibration-<source>-<step>/* -- per-iteration solve logs, NOT committed.results/calibration-<source>-stability/calibration/deviation_penalty_trace.csv -- per-iter Broyden trace (per-component lambda, achieved deviations, residual norm). Inspect when stability behaves oddly.
Artefact sets and provenance
- A config selects its set with
calibration.source (default: default); all artefact paths resolve through the {calibration_source} placeholder at config-load time.
- Structurally divergent configs must either calibrate their own set (
calibration.source: <name> + tools/calibrate --base config/<name>.yaml), point at a compatible set, or set calibration.accept_provenance_mismatch: true (test/tutorial-grade escape hatch: warning instead of error).
- The provenance check covers structural config drift only; input and code staleness is the fingerprint's job. Both run from
tools/calibrate --check.
- The stamp compares all non-solve-time leaves minus exempt machinery keys (see
PROVENANCE_EXEMPT_PREFIXES in workflow/validation/calibration_provenance.py). Solve-time knobs (GHG price, value_per_yll, deviation_penalty, scenario overrides) never trip it.
tests/test_calibration_provenance.py::TestDefaultStampConsistency fails when config/default.yaml changes structurally without a recalibration/restamp -- that is the intended forcing function.
The currently calibrated L1 centre lives in
data/curated/calibration/default/deviation_penalty.yaml under
l1_costs.<component>. Solves that set
deviation_penalty.{land,feed,diet}.l1_cost: "calibrated" resolve the
sentinel from this file at solve time. Per-component
l1_cost_factor lets scenarios scan around the calibrated value
without hard-coding absolute numbers.
Gotchas and pitfalls
The big one: inflated production-stability L1 cost as a mismatch signal
Symptom. After tools/calibrate stability finishes,
l1_costs.land and/or l1_costs.feed in deviation_penalty.yaml are
several-fold larger than the previously calibrated centre (current
centre: ~0.10 land, ~0.03 feed). Or Broyden refuses to converge at the
5 % deviation target and oscillates / hits max iterations.
Mechanism. A pure cost-minimisation solve is free to reorganise
production arbitrarily; the L1 penalty is what coerces the LP back into
observed allocation. When earlier calibration steps successfully absorb
their respective gaps, the LP can satisfy demand cheaply close to the
baseline pattern, and a modest L1 cost is enough to pin it there. But if
a residual supply/demand mismatch survives the feed / food_waste /
food_demand steps -- because of a new bug or a structural change
upstream -- the LP can only match observed production by being physically
coerced through every constraint. The L1 cost is the price of that
coercion, so it inflates.
Anti-pattern. Running tools/calibrate stability until something
finally converges and committing the inflated L1 cost. That hides the
real bug and silently changes the model's stiffness for every downstream
scenario.
Diagnosis recipe.
- Solve
config/validation.yaml with no stability and inspect food-bus and feed-bus slack:
tools/smk --configfile config/validation.yaml -- \
results/validation/solved/model_scen-default.nc
The validation scenario has enforce_baseline_diet: true,
enforce_baseline_feed: true, no stability penalty, and a modest
slack_marginal_cost. Any slack on the food or feed buses reveals a
supply/demand gap the calibration chain failed to close.
- Compare per-group / per-food / per-feed-category slack against the
previously expected post-calibration baseline. The food_waste and
food_demand intermediates
(
processing/calibration-<source>-food_waste/food_waste_uncal*,
processing/calibration-<source>-food_demand/food_demand_uncal*) from
the previous calibration run are the natural reference for what the gap
looked like before the change.
- A food group with >5 % net slack, or a feed category with a
shortage that wasn't there before, is the prime suspect. Trace the
upstream change before touching the calibration chain.
- Other plausible causes (less common, but real): broken
food->nutrient routing, a missing trade-hub edge, a mass-balance sign
flip in a new processing pathway, a unit-conversion bug in a
feed-conversion factor, an LUC sign error in a pasture flow.
Only after the mismatch is understood and either fixed or accepted should
the calibration chain be re-run. The L1 cost should land back near the
historical centre once the chain absorbs its gaps cleanly.
Other gotchas
-
Commit an artefact set whole. The five steps write into one
directory but each is fit against the others, so a set is only
meaningful as a single vintage. Committing one artefact requires either
all five to be consistent or a careful re-run from the step where
inconsistency starts. Don't commit them piecemeal.
-
Each step must disable the calibrations of every later step. The
chain is a single forward pass, not an iteration to a fixed point, so a
step that consumes a successor's artefact is fit against whatever
vintage happens to be on disk. That artefact is then an external leaf of
the step's DAG, so tools/calibrate --check goes stale every time the
successor is regenerated, and the chain never settles. The disable blocks in the step configs form
a lower triangle (feed disables the other four, food_waste the last
three, and so on); preserve it when adding a calibration.
The trap is that a step need not enable a successor's calibration to
depend on it -- inheriting l1_cost: "calibrated" from default.yaml is
enough. feed/food_waste/food_demand therefore set
deviation_penalty.enabled: false (production is pinned to actuals, so the
penalty has nothing to act on) and cost drives stability through hard
bounds. When adding a calibrated key to default.yaml, check what the
earlier step configs inherit.
-
Step configs must not override structural model keys. Anything a
step config pins that isn't calibration machinery, the validation
regime, or a solver budget silently defeats the point of a per-base
artefact set, because provenance.yaml stamps the base config. A
water.data.availability pin in cost.yaml did exactly this and moved
cost corrections by up to 30% on water-resolving configs.
-
Sentinel "calibrated" fails loudly when the yaml is stale or
missing. deviation_penalty.{land,feed,diet}.l1_cost: "calibrated"
resolves from deviation_penalty.yaml at solve time. Scenarios that
want a fixed numeric value should override the sentinel directly.
-
penalty_mode and the sentinel are coupled. The sentinel resolves
only under penalty_mode: "l1". If a scenario sets penalty_mode: "hard", drop the sentinel and pass an explicit numeric L1 cost (or
omit -- hard mode doesn't use L1).
-
Calibration components are configurable. The default subset is
deviation_penalty.calibration.components: [land, feed], reproducing
the historical 2D calibration. A separate config with
components: [land, feed, diet] adds diet as a third axis (used for
specific investigations where the priced optimum reshuffles the diet
while leaving land use approximately unchanged). The calibrator
iterates Broyden over the listed components; the output yaml only
contains keys for the calibrated subset, and the sentinel resolver
raises if a scenario uses "calibrated" on an uncalibrated component.
-
slack_marginal_cost: 7.5 in cost.yaml caps slack-driven duals.
Touch with care: raising it masks data errors; lowering it makes the
high-mismatch foods (buckwheat, plantain, coffee, tea, olive-oil)
diverge in cost calibration. See docs/costs.rst.
-
Step 1 of cost calibration uses hard stability at +/-5 % with
enable_slack: true -- not a plain validation solve. Step 2
tightens to +/-1 % and the dual on each tight constraint becomes the
per-group median cost correction.
-
Warm-starting stability is fine and encouraged. The previously
calibrated yaml is auto-detected as the Broyden seed. But if upstream
changed materially, the warm start can mislead the Jacobian. If
Broyden takes >5 iterations or oscillates, delete (or rename)
deviation_penalty.yaml and start cold with the seeds from
config/calibration/stability.yaml.
-
The canonical enabled: false, generate: true pattern. Every
calibration block has both flags. Generation always uses
enabled: false so the solve doesn't read the file it is about to
write. The combination enabled: true, generate: true is rejected by
workflow/validation/calibration.py.
Quick reference
tools/calibrate --check
tools/calibrate
tools/calibrate stability
tools/smk --configfile config/validation.yaml -- \
results/validation/solved/model_scen-default.nc
cat data/curated/calibration/default/deviation_penalty.yaml
cat results/calibration-default-stability/calibration/deviation_penalty_trace.csv
