| name | validating-task-premises |
| description | Use when you are about to write an eval task, oracle, grader, or register against a generated world - the measure-first protocol that checks whether the pattern a task depends on actually exists, is evenly spread over time, is reachable by the agent through a tool, and discriminates. Load this before authoring a task from an idea of what a world "should" contain, whenever a measurement returns a suspiciously clean zero, whenever a graded field might be constant, and whenever a task's difficulty rests on a rate nobody has broken down by time. |
Measure the premise before you write the task
Every task starts as a guess about what a world contains. The guess feels
like knowledge because it is a reasonable picture of how the domain works
— people set deadlines, meetings concern projects, revisions describe
edits. Worlds generated by models are only sometimes that picture.
On one build, eight premises were written from that kind of confident
picture and then measured. Three died, one had to be inverted, and one
lost a graded field. Every death was cheap, because it happened before
the oracle, the grader and the rollout were built on top of it.
The failure this prevents is the worst kind: a task starved by its own
premise produces rows nobody can find, and a model that finds none of
them looks like a model that failed.
Six checks, cheapest first
Run them in order and stop as soon as one kills the premise.
1. Liveness — does the pattern occur at all?
Count the raw occurrences of every token, form, or relation the rule
depends on. Not the ones you would write; the ones the world wrote.
A date-handling task needed interval phrasings (within N days) and
calendar dates in one message. Both were zero across hundreds of
messages. Another needed each meeting to name the matter it concerned. Matching the
matter list against every transcribed meeting title gave zero of
ninety-three — every meeting was a standing one, a docket call or a
practice huddle, and standing meetings are named after the recurrence
rather than the work.
Re-run later on the same world, still recording, it gave 11 of 157, and
the difference is worth more than either number. Eight of the eleven are one
standing meeting that happens to carry a client's name in its title; three
are genuinely matter-specific, scheduled ad hoc by someone who needed a
meeting about a thing. The structural finding held. The absolute count did
not, because rare events accumulate.
Write the window into the claim. A corpus measurement is true of the
corpus you measured, and a world that is still growing will invalidate any
count stated as though it were a property. "Zero" became "2% of meetings,
ad hoc" without anything changing about the task. Neither task was hard. Both were impossible.
Expect a heavy skew. In one corpus a single deadline form carried 74% of
all occurrences and three of seven forms carried none. A rule listing all
seven scores three sevenths of its own vocabulary against nothing, and
the four that survive are not evenly matched either — one of them is three
quarters of every hit.
2. Read the rows, not the count
A count can agree with your premise while the rows refute it. This is the
check people skip because the number already looks right.
A task needed one sentence carrying two different dates. The detector
found eighteen. Reading them: fourteen were compound spellings of a
single deadline (by tomorrow EOD — two form-words, one date), and the
rest were multi-item lists sharing one due date. The true count was
approximately zero.
A detector that confirms what you already believe has told you
nothing until you read what it matched.
3. Distribution over time — is the rate an artifact?
A rate computed over a window can be dominated by what happens at one
edge of it, and a total will never show you. This is one line:
collections.Counter(day_of(row) for row in signal)
A scheduling-conflict task measured 54 real conflicts against 240
near-miss decoys — a healthy-looking 4.4:1 trap ratio. Grouped by date,
47 of the 54 fell on day one, where the world seeds far more events
than it creates on any later day. Outside that burst the world produced
one conflict in seventeen days. The ratio looked good precisely because
the artifact inflated the numerator.
Both edges do this, and they differ in what they cost you:
- Start-of-record artifacts (seeding bursts, everything created at
once) poison the signal. Usually fatal.
- End-of-record artifacts are excludable, and often improve the rule.
Questions asked on the final recorded day were 100% unanswered — nobody
had a chance to reply. Fixing it meant a stated response window and a
register that closes several units before the record does. The defect
became a design constraint.
4. Reachability — can the agent actually get it?
Three different things, and each can be true while the next is false:
- the fact is in the world log
- it is in the state served to the agent
- a tool the agent can call returns it
A task was designed on meeting transcripts, and built as far as a working
projection before anyone checked layer 2: the transcripts were in the log
and no served surface carried them. Not hard — impossible, silently.
Layer 3 bites too. A column present in a served table but never returned
by any tool is unreachable in practice. And if your surfaces mirror real
products, adding a tool to fix this is its own defect: an agent trained
against an invented tool learns a call that fails in the real product.
Put the data where the real product would put it.
Check the solver too. A reference solver reading the world log rather
than the served state computes an oracle the agent provably cannot
reproduce.
5. Degeneracy — does every graded field discriminate?
Compute the distribution of each graded field across the real rows. A
field that is constant grades nothing while looking like work.
One register carried a cross-surface boolean — had the author logged time
that day. Measured: true for 98 of 98 rows, because everyone who
touches a document logs time that day. It read like a reconciliation
check and was a free point.
6. Admission rate — is the rule worth applying?
What fraction of candidates does the rule admit?
| admission | what happens | fix |
|---|
| ~80%+ | "report everything" scores ~0.9 without reading | invert toward the minority |
| 15–40% | both precision and recall bite | good |
| <2% | needle hunt: bimodal, lands at 0 or 1 | bound it, or retire |
One rule admitted 98 of 119 revisions. Inverted — register the ones
declaring no change — it admitted 22, and over-admitting now destroys
precision while skimming destroys recall.
Read the schema; never guess a field name
A query against a field that does not exist returns empty, and empty
looks exactly like a finding. Twice in one session a guessed name
produced a clean, plausible, completely false zero: narrative and
hours did not exist where the real fields were note and minutes;
an event tag was guessed and reported "0 time entries" for the most
common record in the world.
Print the schema first, then assert the fields you need exist, so a
rename fails loudly instead of quietly:
fields = set(rows[0])
for need in ("note", "minutes", "billable"):
assert need in fields, f"{need} absent; have {sorted(fields)}"
The same discipline applies to any check that returns nothing. A check
that reports no problem needs proof it is capable of reporting one —
break the input deliberately and confirm it complains.
Retire rather than widen
When a premise fails, the tempting repair is to widen the rule until the
corpus fills it: admit two weeks alongside within N days, count
compound spellings as two dates, drop the boundary that excludes the
decoys.
Widening trades a rule the world does not write for a rule nobody
stated. The register then measures the author's vocabulary rather than
the model, and it will score in band for entirely the wrong reason.
Retire the task and take the mechanism from something you measured. Write
the numbers into the task file where the next person will make the same
decision, not into a report they will not read.
The helpers
scripts/premise_checks.py implements the generic half of the above —
liveness, concentration-by-bucket, field degeneracy, and admission-rate
banding — over plain lists and dicts, so it works against any world:
from premise_checks import liveness, concentration, degeneracy, admission
concentration(rows, key=lambda r: r["date"])
degeneracy(rows, fields=["author", "logged"])
Related: authoring-graded-tasks for turning a surviving premise into a
brief, iterating-task-difficulty for moving a task already in flight.
