| name | model-audit |
| description | Audit the LLM/Embedding models registered in SAS Model Manager for the attributes and lifecycle that mdb 1.6.0 populates (model family, provider/version, per-token/second costs, endpoint, modelStatus/approvalState) and for duplicated input/output variables, then report which models need `mdb register --update` (or `mdb pull` first). Use when migrating an existing deployment to mdb 1.6.0+ or to verify a registered model carries its full metadata. |
Audit registered models for the mdb 1.6.0 attributes
Models registered by the framework's legacy Python scripts — or by mdb before
1.6.0 — are missing the attributes SAS Model Manager expects (model family
llmodelType, deploymentId, inputTokenCount/outputTokenCount/hostingCosts,
endPoint, and the modelStatus/approvalState lifecycle), and may have
duplicated input/output variables from repeated --update runs. This skill
retrieves that information for every registered model and tells you exactly which
ones to backfill. It is read-only (GETs only).
See Model-Definition-Builder/MIGRATION.md for the full migration context.
Recipe
- Install mdb with the Viya extra (once), from
Model-Definition-Builder/cli:
pip install -e ".[viya]" (pulls in sasctl).
- Ensure the repo-root
.env has SAS_VIYA_URL, SAS_VIYA_USER,
SAS_VIYA_PASSWORD (and SAS_VIYA_VERIFY_SSL=false for a self-signed cert) —
the same settings the mdb commands use.
- Run the audit (companion script next to this file):
python Model-Definition-Builder/.claude/skills/model-audit/audit-models.py
Add --json for machine-readable output to parse and summarize.
- Read the report and, for each model flagged needs update, run the printed
remediation (see below). Re-run the audit; every model should read
ok.
What it reports
Per registered model (across the LLM Model Project and Embedding Model Project):
llmodelType (family), deploymentId (version), modelStatus/approvalState.- The variable count, flagged
dup! when the same (name, role) appears more
than once (the pre-1.6.0 --update duplication bug).
- A
missing list — any of: family (llmodelType), version (deploymentId),
endPoint, cost (none of inputTokenCount/outputTokenCount/hostingCosts set),
lifecycle (modelStatus/approvalState absent).
needs_update: true when anything is missing or variables are duplicated.
Remediation
- Model that still has its local definition folder:
mdb register <model_id> --update — backfills the attributes and clears the
duplicate variables. mdb register --all --update does the whole fleet.
- Model registered by the legacy scripts with no local
definition.yaml:
mdb pull <model_id> --import to recover the folder + a reverse-engineered
definition.yaml, review it, then mdb register <model_id> --update.
- Lifecycle:
mdb publish advances a model to deployed/approved;
mdb retire sets it to retired. --update never resets a model that is
already deployed/approved — it only backfills a lifecycle that is absent.
Gotchas
- The audit fetches each model's full detail (a GET per model) plus its
variables, because the models list-summary endpoint omits custom attributes
such as
llmodelType and returns deploymentId/endPoint only for some
models. Expect ~2 requests per registered model.
cost is reported missing only when a model has none of the three price
fields. A self-hosted model priced per second carries hostingCosts (no token
prices) and an API model carries token prices (no hostingCosts) — both count
as having cost.- A model with
modelStatus/approvalState absent (common on pre-1.6.0
embedding models) is flagged lifecycle; --update backfills it to
"ready for validation" / "awaiting approval" without touching a model that is
already deployed.
