Design, add, review, or modify AIC Collector operations and their case population. Use for new Collector ops, backend registry entries, collector/cases YAML, case generators/getters, pruning or deduplication, persisted perf keys, framework-version routing, and audits of whether generated cases match Python/Rust consumers.
Design, add, review, or modify AIC Collector operations and their case population. Use for new Collector ops, backend registry entries, collector/cases YAML, case generators/getters, pruning or deduplication, persisted perf keys, framework-version routing, and audits of whether generated cases match Python/Rust consumers.
AIC Collector OP Development
Build the smallest Collector change whose benchmark invocations are valid and
whose persisted rows satisfy the existing AIC consumer contract.
Required reading
Before editing, read:
.claude/rules/collector/layer_permissions.md,
.claude/rules/collector/failure_handling.md, and
.claude/rules/collector/case_authoring.md — the repository-owned policy.
It is authoritative over anything restated in this skill.
collector/README.md
collector/cases/README.md
docs/perf_database/collector-v2-population-design.md, especially
Three identities, Population flow, and Safe deduplication rules
The relevant collector/<backend>/registry.py, collector module, and model/base
case YAML
Every Python and Rust loader/query that consumes the op's perf file
If the task proceeds to GPU collection, also use $aic-auto-collect.
Non-negotiable rules
Treat Collector output as an existing-consumer contract. Do not modify SDK,
Rust, EngineSpec, or Dynamo Planner merely to make a generated case useful.
Consumer changes require separate explicit scope.
Do not call a case invalid merely because the synthetic collector cannot run
it. If the framework and consumer support the shape, fix or record the
Collector gap instead of pruning coverage.
Do not restore coverage with a different kernel under the same logical label.
Persisted quant/backend labels must describe the invocation that actually ran.
Do not modify historical snapshots to make an alignment test pass. Use
read-only comparisons as review evidence.
Do not add permanent legacy_* runtime concepts for a one-time migration.
Put production defaults in normal base-op YAML and keep migration comparisons
outside the runtime schema.
Do not ship a deduplication path unless current repository-owned YAML produces
at least one duplicate invocation. Record that stage's before/after counts
and prove that the unique invocation and persisted-key sets are unchanged.
Remove no-op deduplication instead of manufacturing duplicate-only fixtures
for it.
Treat the public getter and any subprocess or inner re-expansion as one
population contract. Full/raw and targeted entry paths must consume the same
resolved YAML quantization and SM gates; do not reconstruct the policy with a
second hard-coded hardware heuristic.
A case-population change may touch collector/collect.py only for model/op
plan selection required by the resolved case plan. Generic resume, retry,
checkpoint, logging, and output-finalization behavior require separate
explicit scope.
Anchor collector behavior to the requested framework version. Prefer one
verified __compat__/version route over speculative multi-version branches.
Step 1: Establish the consumer contract
Search from the perf filename and op name through both producers and consumers:
Record a temporary contract table; do not commit it unless it is useful product
documentation:
Axis
Recipe source
Changes invocation?
Persisted key?
Consumer query values
Evidence
Answer before coding:
Which columns form the exact lookup key?
Which axes interpolate, and which require an exact bucket?
Which TP, EP, head, window, quant, dtype, phase, and backend values can the
consumer request?
Does a missing key fail, use an empirical fallback, or silently select another
path?
Does the op have a production consumer? If not, keep it registry-only or
explicitly experimental instead of adding it to default model plans.
Never assume an adjacent key is a fallback. Verify it in the loader/query code.
Step 2: Define three identities
Keep these separate:
Recipe identity: why YAML requested the work.
Benchmark invocation identity: every value that can change the executed
model, kernel, runtime setup, or quantization.
Persisted physical key: the columns used by current consumers.
Deduplicate only when both invocation identity and persisted key are equivalent.
A persisted-key collision is a bug unless the invocations are proven identical.
If repository-owned cases map distinct invocation identities to one consumer
key, fail population with the conflicting owners and key. Do not silently use
first-wins or widen the SDK/Rust schema inside a Collector-only change.
Examples:
Model aliases may collapse for a shape-only synthetic benchmark.
Checkpoint paths must remain separate while native quantization or module
behavior is path-dependent.
W4A16 and W4A8 are distinct when activation precision changes.
NVFP4, MXFP4, FP8, and INT4 labels are not interchangeable because geometry
happens to match.
Step 3: Design cases without accidental Cartesian products
Keep independent workload axes such as batch or token count as lists.
Keep correlated structural axes in one profile. Typical correlated fields are
(heads, KV heads, head dimension, window, TP) and
(experts, top-k, hidden, intermediate, TP, EP, quant).
Put shared production defaults in collector/cases/base_ops/<op>.yaml.
Put model-native topology and artifact policy in
collector/cases/models/*_cases.yaml.
Make targeted structural population exact when a model profile exists; it
may still reuse shared workload sweeps. Full/raw collection may union
defaults and model profiles, then stably deduplicate.
Use stable first-wins deduplication on the real invocation/key identity.
When equivalent recipe representations collapse, document the canonical
representative (for example the smallest TP for a local-head key).
Add a generic synthetic default only when a consumer or interpolation need is
demonstrated.
For an existing op, compare current and candidate physical key sets during the
change. This is change-specific evidence, not a permanent compatibility mode.
Report kept, added, removed, and deduplicated separately; totals alone
cannot reveal coverage loss. Migration baselines, exact V1 totals, and
historical snapshots are review evidence, not permanent unit-test contracts
unless an unchanged consumer explicitly depends on that exact inventory.
Step 4: Implement one vertical slice
Touch only the layers the op needs:
Base/model YAML and model-plan selection
Case generator/getter when YAML cannot express the required correlation
Backend collector implementation
Backend registry and exact framework version route
Persisted schema/logging
Focused tests for generated cases and consumer-visible keys
Before adding a helper, schema field, or filter—and again before finalizing—find
its producer in current repository-owned YAML and its reader on a production
population path. If a later design decision removes either side, delete the
orphan. Do not retain speculative aliases, phase selectors, compatibility
modes, or one-time audit scaffolding.
Step 5: Narrow coverage only through the declared homes
A queued case has two legal collector outcomes: execute it, or raise
(layer_permissions.md). Coverage narrowing happens only in the auditable
declaration homes — capabilities.yaml positive floors, registry
unverified/unverified_sms markers, the hang denylist, declared model
correlations, and the sanctioned memory-feasibility filter — each backed by:
Exact-version framework source showing the path is unsupported, or
A minimal runtime repro on the target framework/GPU, or
A proven invocation/key duplicate.
An op with no production consumer stays registry-only or explicitly
experimental instead of joining default model plans. Keep shared generators
deterministic; avoid framework imports or runtime availability probes in
shared YAML population.
Pay special attention to boundaries:
EP>1 and TP>1, including combinations queried independently
Per-rank dimensions after TP/EP sharding
Global versus sliding-window attention
Hopper versus Blackwell quantization support
Native versus converted checkpoint artifacts
Context versus generation phases
If the framework supports a boundary but the collector harness does not, file a
TODO/issue and keep the missing coverage visible. Do not describe it as pruned
unsupported input.
Step 6: Validate in layers
Static population
Generate full/raw and representative targeted plans through each changed
op's registry/public getter; testing only a shared case generator is not
sufficient.
Count generator recipes, raw getter tasks, scheduled tasks (after
capability floors, registry maturity markers, and the denylist),
token-expanded benchmark invocations, and unique persisted keys separately.
For every deduplication, record stage-local before/after, unique invocation,
and unique persisted-key counts using repository-owned inputs. Name the
stage being counted (generator recipes, raw getter tasks, scheduled tasks,
or token-expanded invocations). If the count at the dedupe stage does not
decrease, remove the deduplication path.
When a runtime or subprocess expands an inner sweep, compare its quantization
and SM policy with the outer getter. Test both sides of every changed hardware
gate and assert that unsupported precision labels are absent.
Apply model plan selection, capability floors, registry maturity markers,
and the denylist before reporting a count as the final scheduled queue; raw
getter counts are not final plan counts.
Assert invocation IDs and persisted keys have no unexplained duplicates.
Assert required consumer query keys are covered.
Assert unrelated model dimensions never cross.
Assert targeted plans do not inherit unrelated defaults or ops.
Unit checks
Prefer behavior tests over copied migration inventories or AST tests of which
helper a function calls. Keep focused tests for:
Structural correlation and boundary TP/EP values
Quant/artifact policy
Alias versus path-sensitive behavior
Stable deduplication on the real invocation/key identity
Registry/version routing and plan selection
Persisted key names and local/global dimension semantics
Run inside the exact target framework image and record the installed package
version. Include representative cases for every material branch, not merely the
smallest cases:
Each quantization/kernel path
EP1 and EP>1 when supported
Low and high TP
Global and each supported window family
Context and generation
At least one model/artifact per path-sensitive branch
Then verify the produced rows can be loaded and queried by the unchanged AIC
consumer. A successful kernel timing alone is insufficient.
Definition of done
Do not call the op complete until the handoff states:
Exact framework version, GPU/SM, backend, and __compat__ route
Consumer lookup contract and fallback behavior
Case counts by backend/op: kept, added, removed, deduplicated, skipped
Evidence for every prune/skip
Unit/lint/runtime-smoke results
A real consumer query or support-matrix sanity check
Remaining gaps, especially required EP/TP/quant/window buckets
Whether the change touched Collector only; if not, why broader scope was
explicitly authorized
A final changed-file and changed-symbol audit that accounts for every
production Python change and finds no orphan schema/helper, no no-op
deduplication, and no unrelated orchestration behavior
